# Realtime

Three tiers, one ladder. Same mutator shape at every rung — climbing is a config change, never a rewrite.

`As of 2026-08`. Stable API — semver from here ([Upgrading](/upgrading)). Tiers 1–2 ship. Tier 3 (local-first) has **not shipped**, `As of 2026-08`.

## Two entries: `.` is the browser's, `./server` is the node's

`@ultimat3/realtime` ships **two** entry points, and which one a name lives in is a mechanical fact
rather than a convention — the barrels are disjoint, and a test asserts it.

| Entry | Holds | Reaches |
|---|---|---|
| `@ultimat3/realtime` | `useQuery`, `useRecord`, `useMutation`, `useConnection`, `installRealtime`, `hasPageSocket`, `RecordStore`, `channel()` and `topic()`, the offline queue, the wire protocol, cursors | an island, a browser bundle |
| `@ultimat3/realtime/server` | `syncNode`, `ChannelHub`, `SocketRegistry`, `LiveQueryRegistry`, `NatsTransport`, `openNatsClient`, the replicator, the change feed, pg replication | a `sync` node, a worker, `server.ts` |

```ts
import { useQuery } from '@ultimat3/realtime';             // island
import { syncNode } from '@ultimat3/realtime/server'; // sync node
```

A server render is not a missing install. With no DOM, every read hook answers `pending` with no subscription, and `useMutation()` refuses with `X_LIVE_SERVER_RENDER`. `hasPageSocket()` answers `false` there, every time, so it is **not** a guard that makes a page-body read safe: a browser-only read in code no island imports is `X_LIVE_ROUTE_NO_ISLAND` at the `budgets` step, `hasPageSocket()` included. In a **browser**, a hook in a bundle that never called `installRealtime()` is `X_REALTIME_UNINSTALLED`.

A file that needs both writes both imports. **Nothing was deleted in the split** — if an import
stops resolving after upgrading to 8.0.0, the name moved to `./server`.

Why it is two entries and not one: a single barrel carried `useLive` beside `openNatsClient`, so
`bun build --target=browser` on an entry importing *only* the hook failed with *"Browser build
cannot require() Node.js builtin: `stream/web`"*, out of `nats` — the island this page tells you to
write could not be bundled at all. The package's `sideEffects` array is what lets the bundler shake
the bus out; the split is what makes "the client entry cannot reach the bus" something a build
cannot quietly undo, since a namespace import or an `export *` defeats tree-shaking.

## The ladder

| Tier | Name | You write | Server owns | Client owns | Cost |
|---|---|---|---|---|---|
| 1 | **Channels** | `channel('org-feed', { params, policy, catchUp, records?, events? })`, then `useChannel(decl, params)` | truth + fanout | subscription | ~0 — pubsub over WS |
| 2 | **Live queries** | `query({ live: true, sql })` | truth + change detection | a reactive result set | one replication slot + a matcher |
| 3 | **Local-first** | the same `mutator`, plus `entity(name, { persist: true })`, see [below](#tier-3-shipped-in-2100) | truth + rebase | a durable local store, offline writes | IndexedDB + the one outbox |

Tier 1 for presence, typing indicators, toasts, cursors. Tier 2 for "the list updates when someone else edits". Tier 3 for offline-capable apps.

**Row 3 is on the entity, not the query.** `persist: true` is an `entity()` option in 21.0.0, read by realtime's IndexedDB persister and one outbox. Read the row's own section before writing anything against it.

**Tier 1's `events` are at-most-once, by construction; its `records` frames are repaired.** A dropped `events` frame is counted and logged, never resent. A dropped `records` frame is marked on the server, which sends the socket a `replay-gap` frame once it drains, and the client re-runs the channel's `catchUp` read (21.0.0; [below](#why-delivery-needs-its-own-counter)). State that must arrive belongs on a live query or a `records` channel; ephemera belong on `events`.

Tier 2 covers what people almost always mean by "make it realtime": the list updates without a refresh, and my own click feels instant. It delivers both with **no client database, no client schema versioning, no conflict-resolution UX, and no offline-write semantics to design**. Tier 3 buys exactly one additional property — writes that survive being offline — and costs a durable local store, a rebase log, client migrations, and a conflict story per mutator. Charging every app for that is how realtime frameworks become slow frameworks.

## Same mutator at every rung

```ts
// mutator (action + optimistic local twin)
export const likePost = mutator({
  // REQUIRED (`X_MUTATOR_NOT_IDEMPOTENT`): a replay under the same Idempotency-Key answers the
  // first result instead of running `server` again.
  idempotent: true,
  // Convergent, not incremental: `local` replays on every rebase, so applying it N times has to
  // equal applying it once — `likedByMe` is what makes the second application a no-op.
  local(tx, { postId }) {
    tx.posts.update(postId, (p) =>
      p.likedByMe ? {} : { likedByMe: true, likeCount: p.likeCount + 1 });
  },
  async server(ctx, { postId }) { return ctx.posts.like(postId); },
  conflict: 'server-wins', // | 'last-write-wins' | custom(merge)
});
```

| Tier | What `local` does | What `server` does |
|---|---|---|
| 1 | not called | runs, publishes an event |
| 2 | applies to the page's record-store **overlay** immediately, visible in every island; dropped when the HTTP answer's records land, or on refusal | runs as an HTTP `POST` to the action's path, with an idempotency key; its records flow back into the store |
| 3 | applies to the overlay, and when the network takes nothing the write is queued in the page's outbox (the call resolves `undefined`) with the overlay kept on screen | runs on replay, over HTTP with the same idempotency key |

**A mutation the server *refuses* is rolled back, not retried.** The optimistic write goes, and so does every write made after it — undone newest first, then replayed without it, which is sound only because `local` is pure. The refused intent is dropped from the rebase log: a denial is a decision about that intent, and retrying it would put the write the server refused back on the screen. A refused key stays in the queue as `failed` for the UI to render, and re-issuing the same idempotency key is treated as a **new** intent with a new sequence at the back of the queue, not a collapse onto the denial.

`local` is a pure function of `(tx, input)`, therefore replayable. Hence the rule: **no I/O, no `Date.now()`, no `Math.random()` inside `local`.** Same input, same patch, every replay.

## The fluent surface

`interface Mutator extends Action` — a mutator **is** an action, so every action member is already on it (`likePost.openapi()`, `.client({ baseUrl })`, `.job()`, `.contract()`, `.as()`, `.describe()`, `.named()`, and the lifted `.input` `.output` `.policy` `.mcp`). The three names it was authored with — `.local`, `.server`, `.conflict` — project unchanged, plus a brand and its own descriptor. Both halves are reached through the mutator itself, `likePost.local(tx, input)`, never a declaration object. A mutator has no `.def`.

| Member | Is | Rule |
|---|---|---|
| `.local(tx, input)` | the optimistic twin | runs against the local store, synchronously, no I/O, returns `void`. Takes **already-parsed** input — nothing re-parses on this half |
| `.server(ctx, input)` | the authoritative half | takes **raw** input, like the callable and `.as()`, because this is where parsing happens |
| `.conflict` | the rebase strategy, lifted | `'server-wins'` \| `'last-write-wins'` \| `custom(merge)` — the declared value verbatim, merge function included |
| `.isMutator` | the brand | always `true`; `isMutator(value)` is the guard |
| `.describeMutator()` | the manifest row | the action descriptor plus `kind: 'mutator'` and the **resolved** strategy name — `custom(merge)` describes as `'custom'` |
| `.describe()` | the action descriptor | still reports `kind: 'action'`, with `mutator: true` — that flag is what puts the `mutator` count in `x.manifest.json`. `describeMutator()` is the one that says `kind: 'mutator'`, plus the resolved strategy |

**`.server()` routes through the action's own callable, never the declared `server`.** It calls `base(input, { ctx })`, and that callable *is* `invoke` — so the authoritative half cannot skip the input parse, the policy or the output parse. Reaching the declaration from there would be a second execution path, which is the one thing `@ultimat3/action` exists to prevent. Hence the guarantee this page rests on: the offline/realtime half of a mutator gets exactly the same actor gate → parse → policy → handle → parse core as an HTTP call, an MCP tool call and a job run, because it **is** that call. A denied `.server()` never reaches the declared half, and neither does one whose input fails the schema — `X_UNAUTHENTICATED`, `X_FORBIDDEN` and `X_INPUT_INVALID` come back from the same core that serves the HTTP route.

The parsed-vs-raw asymmetry between `.local()` and `.server()` is deliberate, not an inconsistency. `local` replays on every rebase and must stay a pure function of `(tx, input)` — no I/O, no clock, no randomness — so re-parsing on each replay would be wasted work on input that was already validated once.

`.local()` is the name. No alias: the old `.applyLocal` is gone, not deprecated.

`.named()` rewraps rather than dropping the twin — a renamed mutator keeps both halves, its `conflict`, and every inherited action member.

## Tier 3 shipped in 21.0.0

Opt-in per entity, and shipped in 21.0.0 (`CHANGELOG.md`). The 20.x shape (`store`, `queue` and `log` on
`LiveClientOptions`, `MemoryLocalStore`, a `createOpfsLocalStore()` that threw
`X_NOT_IMPLEMENTED`) is deleted with `LiveClient`. What replaces it:

| Piece | 21.0.0 |
|---|---|
| what persists | `entity(name, { persist: true })`, default `false`. The server renders the persisted record types into a private document's `<meta name="ultimate-persist">` (`packages/cli/src/page-sync.ts`, `persistedRecordTypes`) |
| where | IndexedDB, every entry keyed `[scope, type, key]`. Scope is `p:<principal>` or `anon`; an unscoped page persists nothing |
| what is written | synced rows only, never an overlay. Debounced 250 ms, flushed on `pagehide` and when the page is hidden |
| boot | the rows are restored **before** the socket connects, as provisional records; the first server row for a key wins outright |
| an answer that did not carry every row the overlay wrote | the overlay stays for those rows until a server row for them arrives, capped at 10 s (`DEFAULT_AWAIT_SERVER_MS`), after which server truth stands |
| a write with no response | only `X_CLIENT_TRANSPORT_FAILED` with `meta.failure: 'network'`: `useMutation` queues it in the page's one outbox under its idempotency key, keeps its overlay on screen, and resolves `undefined`. `'status'` rejects and drops the overlay; `'body'` (a 2xx that was not JSON, which may have landed) rejects and keeps the overlay until the next server row |
| replay | in order, over HTTP, each write with its original idempotency key: whenever the page socket (re)connects, on `online`, and on the service worker's `OUTBOX_DRAIN_MESSAGE` ([PWA and offline](/pwa-and-offline)). A retryable failure stops the pass; a refusal is final and rolls back its overlay. **At least once**: an entry leaves the disk when its ack is written, so a 2xx that reached a document already being replaced is sent again by the next one, under the same key — answered from the action's idempotency store (a `mutator()` is `idempotent: true` or it does not declare), never applied twice. A `409 X_IDEMPOTENCY_CONFLICT` for a first attempt still running stays queued and is asked again |
| a write queued offline | flushes the persisted records it touched first, so a reload inside the 250 ms persist debounce keeps the row the write changed. No replay runs while `navigator.onLine` is `false` (`page-outbox.ts`) |
| a principal change | **within one page** (`rescope()`): the previous principal's rows and queue are wiped, and writes still queued are lost, deliberately. A replay on the wire at that moment is abandoned before its next write: nothing queued by one principal is sent under the next one's session. A write being queued at that moment is refused — `mutate` rejects with `X_OFFLINE_QUEUE_ABANDONED` and its optimistic twin is taken back. **A sign-out that navigates** to a new document (the reference app's idiom: the `endSession` action at `POST /api/sessions/end`, posted by a native form) clears the browser first: its response carries `signOutHeaders()` from `@ultimat3/auth`, whose `Clear-Site-Data: "cache", "storage"` drops IndexedDB, local storage, the service worker and its cache, in a secure context. The next boot is the second line: it wipes every stored scope except the current principal's before restoring anything (`packages/realtime/src/boot.ts`, `wipeOthers`). An unscoped page wipes nothing |
| blocked storage | memory, plus one `X_LOCAL_STORE_UNAVAILABLE` warning; nothing survives a reload |

The boot runs as **one deferred classic script per page** (`/_x/assets/page-boot/<hash>.js`, from
`@ultimat3/realtime/boot`). It is rendered only on a scope-tagged (private) document that emitted
an island reaching `@ultimat3/realtime`. It wipes other scopes, restores this principal's rows,
and opens the outbox, so a reload that opens no live hook and writes nothing still replays what
the previous load queued.

**Trade-off: two principals in two tabs of one browser.** The newer page's boot wipes the other principal's rows and queued writes from disk. The older tab keeps its records and its queue in memory, and re-persists them on its next write; if it closes first, what it had queued offline is gone.

## Live query in the browser, end to end

Written `As of 2026-09-05` because four agents building one app each **polled** a query from an
island instead of subscribing. Rewritten for 21.0.0, which deletes the app's socket
adapter, its sync URL and its `LiveClient`: the page's one socket is the framework's.

| Step | Where | What |
|---|---|---|
| 1. the dependency | root `package.json` | `bun add @ultimat3/realtime`. `x new` installs `@ultimat3/query` and **not** `@ultimat3/realtime` |
| 2. the query | `app/<feature>/live.ts` | `query({ …, live: true, subscribes: ['posts'] })`, bounded and ordered; `x g query <name> --feature <slice> --live` writes it |
| 3. the ref | the island | `{ name: 'liveFeed', live: true }`, the query's registered **name**, never the query value: importing that drags its whole read path into the island. `live` is stated beside the name, not derived (see below) |
| 4. the install | nothing to write | `x build` builds every island whose own import graph reaches `@ultimat3/realtime` from an entry that calls `installRealtime({ signal: createSignal })` first, for about 100 B. An island not reaching realtime pays nothing. The sync target is read from the document's `<meta name="ultimate-sync">` ([Configuration](/configuration), `SYNC_URL`) |
| 5. the hook | the component | `useQuery(ref, input)` returns an `AsyncState` accessor. **In an island, never a page body**: a page component never runs in a browser, so the hook answers `pending` there forever |

```tsx
// app/feed/feed.island.tsx — the only module of /feed a browser downloads
import { useConnection, useQuery } from '@ultimat3/realtime';
import { For, Match, Switch } from 'solid-js';
import { render } from 'solid-js/web';

type FeedRow = { id: string; title: string };
const LIVE_FEED = { name: 'liveFeed', live: true } as const;

export function mount(el: HTMLElement, props: { orgId: string }) {
  el.textContent = '';                       // Solid's render APPENDS; the server's loading shell goes first
  render(() => <Feed orgId={props.orgId} />, el);
}

function Feed(props: { orgId: string }) {
  const feed = useQuery<FeedRow>(LIVE_FEED, { orgId: props.orgId }); // subscribe, once
  const connection = useConnection();
  return (
    <Switch>
      <Match when={feed().status === 'pending'}><p>loading…</p></Match>
      <Match when={feed().status === 'failed'}><p>could not load the feed</p></Match>
      <Match when={'data' in feed()}>
        {connection.offline && <p>offline — showing the last rows</p>}
        <ul><For each={(feed() as { data: readonly FeedRow[] }).data}>{(row) => <li>{row.title}</li>}</For></ul>
      </Match>
    </Switch>
  );
}
```

`feed()` is an `AsyncState`: `pending`, then `ready` (or `refreshing`) carrying the rows, or
`failed`. It re-renders on every patch the sync node sends, with no timer and no refetch. The list
holds **ids**; the rows come from the page's one `RecordStore`, so a write to a record through
any path re-renders every list showing it. A non-live query uses the same hook: drop `live`, and
name its entity (`{ name, entity: 'post' }`) so its rows are store records too.

**Two limits, `As of 2026-09-22`.** `x build` sees realtime only in the island's **own** graph. An
island that reaches it through a package, rather than importing it, calls
`installRealtime({ signal: createSignal })` in `mount` itself, or throws `X_REALTIME_UNINSTALLED`.
And `live` and `entity` on the ref are stated by hand, not derived from the query declaration. Both
are in the plan-101 DX ledger.

### Which islands wait for the page's restore

`As of 2026-10`. On a signed-in page with a sync node, the page boot restores persisted records and
the queued writes from disk. Server markup rendered before a queued write would paint the old value
first, so an island that can show such a value is **held**: hidden (`visibility:hidden`, its box
kept) until it mounts over the restored store.

| Island | Held | `mount` waits for the restore |
|---|---|---|
| takes `useQuery`, `useRecord`, `useRecords`, `useMutation`, `useMutationQueue`, `useOutbox`, or any other name not listed below, anywhere in its own graph | yes — until it mounts, 3 s at most | yes, 1 s at most (`FIRST_PAINT_HOLD_MS`) |
| takes `useQuery` or `useRecord` (and otherwise only the follow list below), in an app that declares no `persist: true` entity | **no** — the boot restores no record they could show | no |
| takes only `useChannel`, `usePresence`, `useConnection`, `hasPageSocket`, `useRecordChanges`, `pollEvery`, `readPresence`, `channel`, `channelRef`, `topic` | **no** — its server markup is on screen from the first byte | no — it mounts as soon as the page runtime is there |
| takes the barrel with no list of names (`import * as`, `export *`, `import()`) | yes | yes |
| reaches no realtime | no | no |

| Fact | Rule |
|---|---|
| who decides | `x build`, from the names each module of the island's own graph imports from `@ultimat3/realtime` (`packages/cli/src/island-realtime-state.ts`). Nothing to declare |
| whether the app persists | read off the entity registry the build loaded (`appPersistsRecords`). An empty registry is unknown, and keeps the held answer. `useMutation` is held either way: the outbox rebuilds a queued write's overlay |
| the events-only list | an island that adopts the server's markup and re-reads its own query when a channel event arrives is never hidden: nothing the restore brings back can contradict what the server drew |
| with scripting off | no island is ever hidden: `@media (scripting: none)` in `@ultimat3/ui`'s `global.scss` cancels the hold. A browser without that media feature, or a runtime that was blocked, shows the markup at the 3 s cap |
| a prebuilt image | each island's answer rides its chunk, and the store `x build --target docker` writes records it, so a pod that builds nothing renders the same page boot and the same holds as `x dev` |
| the blind spot | a **package** that reads the store for the island is not seen, as with the install above |

### In development, the feed is this process

`As of 2026-09-05`. The sync node's changes come from Postgres logical replication — a real
`DATABASE_URL` and the `replicator` role (`x dev --role replicator`). The embedded database has no
walsender, so under a plain `x dev` a subscription took its snapshot and then heard nothing: every
`--live` query in every scaffolded app was dead in development, which is where an author first
tries one. `x dev` now installs the in-process bridge — `startLiveReplicator`, on
`@ultimat3/realtime/server` since 2026-09-23 (`@ultimat3/testing` no longer exports it), the
row observer the framework's own live tests run on — whenever the database is embedded, and the
ready line says so:

| `db=` | `live=` | What reaches a subscriber |
|---|---|---|
| `embedded` | `in-process` | every repository write **this process** makes — a write made by another process is invisible, which is the bridge's honest bound and holds by construction under `x dev` |
| `external` | `replication` | what the WAL decoder delivers, in this process (`--role replicator`) or in another |
| no `sync` role | `none` | nothing; there is no node to feed |

Never both: with a real database the decoder already carries this process's own writes, and a
bridge beside it would deliver each of them twice. `--json` carries the same fact as `liveFeed`.

An entity the app publishes itself (`recordPublisher`, for a deployment with no replicator) is the
bridge's one exception: it stops carrying that entity's writes to channels, so each row reaches a
channel member once — from the app's publish, as in production. Live queries still hear the bridge.

## Live query pipeline

```
Postgres (logical replication slot)
    │  WAL decode, per-table
    ▼
replicator role  ──► change feed (ordered, per-table, with LSN)
    │
    ▼
incremental matcher  ──► for each registered live query: does this row enter/leave/update the result?
    │                      (predicate + order + limit evaluated against the changed row only)
    ▼
NATS subject per query-hash + tenant  ──► fanout
    │
    ▼
sync role (stateless)  ──► WS frame: {qid, op: insert|update|delete, row, lsn}
    │
    ▼
Solid signal patch — fine-grained, no re-render of the list
```

| Stage | Owned by | Guarantee |
|---|---|---|
| WAL decode | `replicator` (1 per DB) | ordered by LSN, at-least-once. A stream that ends is restarted by the same process from the last LSN it published, on a jittered backoff; `replicator.running` is `false` until it is back |
| matcher | `replicator` | a change touching no registered query costs one predicate check |
| fanout | NATS | subject = hash(query, params, tenant); no per-socket state on the bus |
| socket | `sync` (stateless, no sticky sessions) | client re-subscribes anywhere; scales on connection count |
| authz | `policy` on the `query` | evaluated at subscribe **and** re-checked on row delivery — a row that fails the policy is dropped, never sent |

The matcher is why this is affordable: membership is decided from the changed row plus the query's predicate, order, and limit — never by re-running the query. Which is exactly why `live: true` requires a deterministic, bounded `sql` ([Queries and live queries](/queries-and-live-queries)).

## One record per `type:key`

The page holds **one** `RecordStore` per tab, shared by every island through a `globalThis` handle.
It is an identity-mapped store: a record is one value per record type (the entity name) and record key,
however many lists, islands and paths hold it. It shipped in 21.0.0
(`packages/realtime/src/record-store.ts`).

| Consequence | Why it matters |
|---|---|
| A socket patch, an HTTP answer's records and an optimistic write all move the same record | two components rendering post #7 cannot disagree about it, in one island or across two |
| A list holds **ids**; the rows come from the store, for a live and a non-live query alike | a write to a record re-renders every list holding it, with no refetch |
| Two layers: **synced** is server truth, and the **overlay** is every pending optimistic write, replayed over it on every change | rolling back an optimistic write cannot delete a row the server has since sent |
| A patch that omits a field never clears it | two queries may project different columns; a narrower answer must not blank what a wider one renders |
| A row that is not an object, or has no key, is dropped and reported (`X_RECORD_REJECTED`) | a keyless row would overwrite another record |
| A record lives exactly as long as something holds it | the last release drops it, so an infinite scroll is not a leak |
| A list that must grow with the store is `useRecords(type, { where?, order? })` | a `useQuery` list holds its own answer's keys and `useRecords(type, keys)` the keys it was given, so a record a channel's `records` frame adopts reaches neither. The whole-type form shows every record of the type the store holds — `ready` from its first read, ordered by key unless `order` says otherwise, none evicted while it is held. The channel-records counterpart of a live window (25.0.0, #680) |
| An answer is the same object until what it shows moved | `useQuery`, `useRecord` and `useRecords` hand back the previous answer and its `data` array while the status and every row object match, so a poll or refetch that found nothing new re-runs nothing derived from it; a whole-type reader with `where` is not told of a record it neither shows nor would show (29.2.0) |
| A busy feed is throttled by its reader: `useRecords(type, { throttleMs })` | the first change at once, a burst as one notification at the end of the period, the last always landing; `release()` drops what is owed. Absent or `0`: every change. A non-finite or fractional value is `X_INVARIANT` (29.2.0) |
| A reader that only needs to know THAT records moved is `useRecordChanges(types, listener)` | it is called once per store update with the moved keys per type and holds no row and no type, so it pins nothing: re-rendering a server page on a channel's `records` costs no memory. Follow-only, so its island is never held |

**The key comes from the server, on every path.** A browser cannot compute a key without importing
the app's `entity()` declarations, so it never does.

| Path | Where the key is |
|---|---|
| an HTTP envelope | `records`: record type → record key → row, keyed by the entity's `recordProjection` |
| a live snapshot | `keys`, parallel to `rows`, sent only when some key is not its row's `id` |
| a live patch | `key`, sent only when it is not the row's `id` |

So an entity with a **composite primary key can be live**. A non-live `useQuery` lists its records
in the envelope's `records[type]` key order, which `rowsOf` fills in data order. An answer with no
envelope holds its rows itself, and they are not records. A live window whose node named no type
keeps its rows under `?query:<name>` (`unnamedType`), so two unnamed windows never merge two
entities' rows.

## Channels are declared

`As of 2026-09-22`, 21.0.0. A channel is a declaration, never a string. The hub API that
took raw topics (`hub.guard`, `subscribe(socket, topic)`, `publish(topic)`, `publishFrame`,
`channelFrame`, `TopicGuard*`, the `nodeId` option, and a `{ kind: 'topic' }` subscribe target) is
deleted.

A channel is written **once, in two halves on one ref**, the same split as a query's `QueryRef`: an
island holds the browser half and never imports an entity or a database package.

```ts
// app/feed/channel-ref.ts — the BROWSER half: name, params, catch-up read. No entity, no policy.
import { channelRef } from '@ultimat3/realtime';

export const ORG_FEED = channelRef('org-feed', {
  params: ['orgId'],
  catchUp: { name: 'orgFeed' }, // the query re-read on replay-gap or a new epoch
});
```

```ts
// app/feed/channels.ts — the SERVER half, on the same ref. Registers on import.
import { channel } from '@ultimat3/realtime';
import { Post } from '../posts/entity';
import { ORG_FEED } from './channel-ref';
import { feedRead } from './policies';

export const orgFeed = channel(ORG_FEED, {
  records: [Post],  // every param must be a column of each listed entity, or refused
  policy: feedRead, // decided at subscribe, with the params as input
  events: true,     // ephemeral payloads: typing, presence
});
```

```ts
// in an island: the ref, never the server half (which would bundle the entity and its driver)
import { useChannel } from '@ultimat3/realtime';
import { ORG_FEED } from './channel-ref';

declare const orgId: string;
const feed = useChannel(ORG_FEED, { orgId }, { onEvent: (event) => console.debug(event) });
```

`channel(name, { params, catchUp, policy, … })` in one call still exists for a channel no browser
holds. **`policy` is required** on either form: a declaration without one is refused
(`X_CHANNEL_DECLARATION_INVALID`), and a public channel says so with `policy: allow('public')`.
A `catchUp` passed as the query itself must decide with the channel's own `policy` object — the
read is served with the channel params as input, so a wider rule answers what the subscribe
refuses — or `channel()` throws `X_CHANNEL_CATCHUP_POLICY_MISMATCH` at load.
The reference app's idiom is `examples/dummy/apps/web/app/posts/channel-ref.ts` beside `channels.ts`. The catch-up
query's name is written by hand on the ref, like a `QueryRef`'s; `x build` emitting refs is
plan-101 DX ledger #21.

| Fact | Rule |
|---|---|
| a policy subject | `row: ({ params, ctx }) => …` loads what the policy decides about, the way an action's `row:` does |
| registration | `channel()` registers itself in the process's channel table. `new ChannelHub({ transport, sockets })` serves every declared channel |
| subscribe | by declared name + params. An undeclared name is `X_TOPIC_FORBIDDEN`, and so is a policy denial, latched per (socket, topic) until the session changes. Two more are denials before the policy runs: an anonymous socket on a channel that declares `row`, and a `row` loader the tenant guard refused |
| re-authorization | when a socket's grant is renewed every seat is decided again. Denied: dropped, and refused to the client under its sid. Undecided (the policy or loader raised): kept but **suspended** — nothing is delivered until a later pass succeeds, then a records channel is told `replay-gap` |
| the first join | a join with no cursor on a channel that carries records is answered with **one** `replay-gap`, so the client runs one catch-up read. Rows written between a page's render and its join are otherwise never seen. A resubscribe with `since` replays from the ring, or gets `replay-gap` when `since` fell out of it (`channel-logs.ts`, `resume`). A `since` that is already current is answered too — the `records` frame at that cursor, carrying nothing — so `useChannel()` leaves `joining`; an `events`-only channel leaves it on its first `events` frame, which is its presence roster |
| records | rows of the listed entities travel as `records` frames, carrying `seq` and `epoch`, into the record store; read them back as a list with `useRecords(type, { where?, order? })` |
| a missed change | a hole in a producer's `seq` on the change bus, or a bus that reconnected, tells every member of every open records topic `replay-gap` at a new epoch, beside the live windows' re-snapshot (24.0.0, #681) |
| records with no replicator | `recordPublisher({ transport, entities })` on `@ultimat3/realtime/server` puts an app's committed rows on the change bus, sequenced per producer; every node delivers them as `records`. Channels only — a publisher has no commit position to order a live window by. Under `x dev` a claimed entity is no longer also carried to channels by the in-process bridge, and a sync node carries each table from ONE producer kind — the first it saw — dropping the other's changes under `X_REALTIME_PRODUCER_CONFLICT` (25.0.0, #682). Rules: the package README, *Channel records with no replicator* |
| `useChannel()` states | `joining` → `live` on the node's first answer to every channel that carries `records` or `events` (an `events` channel's is its roster, empty on a node without presence); `catching-up` while a `replay-gap` re-read runs; `offline`; `failed` with the coded refusal — a **decision** (a policy denial, an undeclared channel, a cap) and nothing else. A node whose bus is away refuses with `X_TRANSPORT_UNAVAILABLE`, which is its condition and not its verdict: the membership reads `joining` and is `live` again on the node's next answer ([a bus outage, seen from an open page](#a-bus-outage-seen-from-an-open-page)). The table is in the package README, *What `useChannel()` reports* |
| events | on a channel declared with `events: true`; otherwise `X_CHANNEL_DECLARATION_INVALID`. **App code** — a job, an action — calls `publishChannelEvent(decl, params, event)` from `@ultimat3/realtime/server`: it publishes on the process's own bus, in every role, with no hub (27.5.0). The `sync` node's own code holds the hub and calls `hub.publishEvent(decl, params, event)`. Same frame, same subject. At-most-once, never stored, never replayed: an event says "re-read", never "here is the state". Call it **after COMMIT** — nothing here sees a transaction |
| events with no replicator | a channel that lists no `records` reads no committed change, so it needs no replicator, no replication slot and no `REPLICATION` grant: `NATS_URL` set for every role, a `sync` role, and the publishers. A node that serves no `live: true` query and no `records` channel boots on a real database with `live=none` and is never `X_REALTIME_TOPOLOGY` (27.5.0) |
| presence | a roster arrives as an `events` frame `{ presence: op, members, total? }`. Read it with `readPresence(frame.event)` in the channel's events handler. There is no separate presence frame |
| an events-only island | an island whose only realtime is `useChannel` / `usePresence` / `useConnection` is never hidden and never waits for the page's restore ([above](#which-islands-wait-for-the-pages-restore)) |
| the browser | `useChannel(decl, params, handlers?)`; `topic()` is the one way to spell a topic, and `bun run channel-literals` refuses any other |
| the manifest | lists every channel (`describeChannels`, on `@ultimat3/realtime/server`). In a contract diff these are **breaking**: a channel removed, its params changed, its policy changed, a record type no longer carried, `events` switched off |

## A channel event re-reads the page

`As of 30.0.0`. For a server-rendered page with nothing to push but "this changed": the channel
says **when**, the server's page says **what**. An event is at-most-once, so a slow safety re-read
sits beside it.

```ts
// in an island on the page: the event schedules, the router re-reads when the visitor is idle
import { visibleInterval } from '@ultimat3/core/page';
import { type ChannelRef, useChannel } from '@ultimat3/realtime';
import { scheduleRefresh } from '@ultimat3/render/client';
import { onCleanup } from 'solid-js';

declare const ORG_FEED: ChannelRef<'orgId'>; // the ref above, from `./channel-ref`
declare const orgId: string;
useChannel(ORG_FEED, { orgId }, { onEvent: () => scheduleRefresh() });
// the event is at-most-once: a safety re-read every 30 s, hidden tabs skipped
onCleanup(visibleInterval(scheduleRefresh, 30_000));
```

| Piece | Why it is that one |
|---|---|
| `scheduleRefresh()`, not `refresh()` | a burst of events is one re-read; nothing swaps under an open dialog, a focused field, a selection or a scroll ([Client navigation](/client-navigation#scheduling-a-refresh)) |
| `visibleInterval` from `@ultimat3/core/page` | the poll a hidden tab and a lost session stop; an island that only polls reaches no realtime ([Client data](/client-data)) |
| rows, not a page | `useQuery(ref, input, { live: true })` stays the idiom: this is for a page whose rows no island holds |

A page with no router does nothing on a `scheduleRefresh()`: it never reloads under a reader. Pause
toggles, a flash on change and a "live" dot are the app's.

## What each role needs from the bus

The bus is the `transport` adapter port: `NATS_URL` set builds the NATS transport, unset the in-process one, and `transport:` in `apps/<app>/runtime.ts` replaces that decision with any `Transport` from `@ultimat3/realtime/adapter` — already connected, and not closed by the boot ([Adapters](/adapters#transport)). There is no `setChannelTransport` and no `realtime.transport` config key (31.0.0).

With `NATS_URL` set every role holds a connection, and only two cannot work without one (`As of 2026-10`). The role decides, not the transport: a role that only **publishes** sends channel events and cache invalidations, and both mean "re-read".

| Role | Uses the bus to | At boot | NATS down while running | JetStream / the presence bucket | `transport` check |
|---|---|---|---|---|---|
| `sync` | serve sockets: subscribe to changes and channels, hold presence | **awaits** the dial, retried on backoff for 15 s (each attempt has a 4 s connect timeout, so a server that accepts and says nothing still fails attempts inside the wait), then refuses the boot with `X_TRANSPORT_UNAVAILABLE` naming the server, the wait, the last attempt's failure and the variable to check | keeps its sockets and reconnects on its own; a page's beat is refused `X_TRANSPORT_UNAVAILABLE` meanwhile ([below](#a-bus-outage-seen-from-an-open-page)); windows re-snapshot and members re-join on return | **required at the dial** — no `-js`, no boot | `failing` |
| `replicator` | publish committed changes to the nodes | **awaits** the dial, same 15 s, same refusal | a publish refused because the bus is away is retried in place — the same envelope, three more attempts over 2 s — so a NATS restart costs no stream restart; past that the run ends and the takeover loop resumes from the last published lsn | not asked for | `failing` |
| `web`, `worker`, `scheduler` | publish channel events (`publishChannelEvent`) and cache invalidations; receive peers' invalidations | **does not wait**: dials in the background on a jittered backoff (500 ms doubling to 30 s) and boots, serves and runs jobs meanwhile | stays up and ready; resumes on its own, no restart | not asked for — asserted only if something reads presence | `degraded` |
| `migrate` | nothing | never dials | — | — | — |

`x dev` is one process running several roles, and it never goes down for the bus. With `sync` in the set (the default) it waits **3 s** for NATS (`DEV_BUS_WAIT_MS`), then boots without it: the refusal a `ROLE=sync` container would have exited on is logged once at `warn` (`ultimate bus is away: x dev serves without live until it answers`, with its `code`, `cause` and `fix`), the dial goes on in the background, pages, jobs and tasks run, and the `sync` node stays mounted and **not ready** — an upgrade is shed with a 503 and a retry delay, so an open page keeps asking. When NATS answers the node starts (`sync node started: the bus answered`) and those pages go live with no restart. `checks.transport` is `degraded` there, never `failing`, so a bus lost under a running `x dev` leaves `/readyz` at 200. With `--role web,worker` nothing waits at all. A **container** is unchanged: `ROLE=sync` and the replicator keep the 15 s refusal, because they have nothing else to serve. `realtime.enabled: false` starts neither realtime role, so no process of such an app waits for the bus — the connection is still dialled in the background, because cache invalidation between replicas crosses it.

| While the bus is down, on a publishing role | What happens |
|---|---|
| `publishChannelEvent(…)` | rejects at once with `X_TRANSPORT_UNAVAILABLE` — never parked behind the dial. Catch it and log: the event said "re-read", and a subscriber that reconnects re-reads anyway |
| a publish made while the client is **reconnecting** | refused the same way, from the moment the client's socket closes. The `nats` client would otherwise buffer it without bound and replay it when the server returned, and it has no option to forbid that. One window remains: a connection whose packets are silently dropped — a server that accepts the socket and then says nothing — still reads as connected until two pings go unanswered (10 s apart; measured against a real server through a relay that stops forwarding: detected 30 s after the silence began, where the library's default is six minutes), and a publish made inside it is taken and lost. A KV **request** has an answer to wait for, so it is never buffered: inside the window it fails after its 5 s timeout, and once the client knows it is disconnected it is refused at once |
| an action's `invalidates` / `invalidateTags` | every local tier, Redis and the CDN are cleared as usual; the broadcast to the other replicas fails into `report.errors` (`tier: 'broadcast'`, logged as `cache.invalidate.partial`) and the write succeeds. **The refused tags are kept and published when the bus returns** — de-duplicated, at most 1,024 of them; past that the process owes one *flush-all* instead, so nothing grows with the outage. Until then another replica's in-process `lru` copy and its tag-revalidated ISR page are stale |
| when the bus returns | every process whose connection was down **drops its in-process tier and marks its tag-revalidated ISR pages stale** (`flushProcessTiers`): it heard no peer for a window nobody measured, so nothing in its own heap is known fresh. One cold `lru` per replica per outage — sized for by the database, not hidden. A replica that receives a peer's flush-all does the same. Redis and the CDN are not touched: whoever ran a bust cleared those |
| `/readyz` | 200 in both `health.readiness` modes, `checks.transport: 'degraded'` |
| `/readyz?deep=1` | 503 — the reading a monitor alerts on |
| the log | the boot line `{"msg":"ultimate started",…,"bus":"nats(connecting)"}` (`nats(up)` when it already is, `in-process` with no NATS); `{"msg":"ultimate bus","bus":"nats(up)"}` when the dial lands, and again after every recovery. Everything else is [one outage, said rarely](#what-an-outage-logs) |

### A bus outage, seen from an open page

A page that is open when NATS goes away keeps its socket: the `sync` node does not close it, and nothing is reloaded. What the page reads, `As of 2026-10`:

| Moment | The node | `useChannel()` | `usePresence()` | `useQuery({ live: true })` |
|---|---|---|---|---|
| the bus goes away | keeps every socket and every seat; cannot read or write presence, receives no change and no event | `live` until the page's next beat (≤ 10 s) | the last roster | `live`, with the rows it has; no patch arrives |
| each beat while it is away (every 10 s) | answers the membership's `subscribe` with an `ack` carrying `X_TRANSPORT_UNAVAILABLE` under its sid | **`joining`** — asked, unanswered. Never `failed`; `error()` stays `undefined` | the last roster | unchanged: the beat re-sends channels only, and a live subscribe never touches the bus |
| a page **opened** during the outage | seats it if it can; refuses the same way if it cannot | `joining` | empty | `live` once its snapshot lands — the read is the database's |
| the bus returns | re-binds its subscriptions, tells every records topic `replay-gap` at a new epoch and marks every live window for a re-snapshot (`live.bus_reconnected`) | a `records` channel: `catching-up` → `live` at once, on the gap | — | re-snapshotted on its next delivery |
| the next beat after it (≤ 10 s) | the member is gone from a restarted server's bucket, so it is joined again under the same socket id; the roster is sent | an `events` channel: **`live`** on that roster | the fresh roster | — |

Until 2026-10-10 the refusal marked the membership `failed` and nothing cleared it: events flowed again and the page read `failed` until a reload, while a new tab was `live` at once.

| Rule | Detail |
|---|---|
| what is transient | exactly one code, `X_TRANSPORT_UNAVAILABLE` (`client-channels.ts`, `isTransientRefusal`): the node's bus is away, or the node is draining |
| what is terminal | every other refusal: `X_TOPIC_FORBIDDEN` (a policy denial, an undeclared channel), a cap (`X_SUBSCRIPTION_LIMIT`), an auth failure. `failed`, with `error()` the node's own coded refusal |
| the retry | the beat itself. A membership is re-asked every heartbeat whatever its state — that is what the presence heartbeat already is — so an outage adds no timer and no load, and no backoff is needed: one frame per topic per 10 s is the healthy rate |
| a state the outage does not touch | `catching-up` (a read in flight), a `failed` catch-up read, a terminal `failed`, `offline` |
| a live query | not on this path: a live subscribe reads the database and never the bus, so no node refuses one with `X_TRANSPORT_UNAVAILABLE`, and the beat re-sends channels only. Its `ack` handling is unchanged — `failed` is a decision, and a rate limit is retried after the delay the node names |
| every tab | one socket per origin (the SharedWorker): the engine's own beat is the re-ask, and the refusal and the answer are routed to every tab holding the topic |
| a records-only channel on a seated socket | its beat touches nothing on the bus, so it is **not** refused and reads `live` through the outage, with no row arriving; the `replay-gap` on return repairs it |

### What an outage logs

One outage is one fact. Each condition below is written on its 1st, 2nd, 4th, 8th, … failure and once more when it ends (`outageLog`, `isOutageMilestone`, on `@ultimat3/core`), at a level chosen by what the process can still do: **`warn`** for one that is degraded and serving, **`error`** for one that cannot do its job.

| Line | Level | Where | Written |
|---|---|---|---|
| `sync.bus_unavailable` `{ operation, failures, error }` | `error` | `sync` | failures 1, 2, 4, 8, … of anything on the node that needed the bus — a page's refused beat (`operation: 'sync.frame'`), a presence sweep, a presence leave. One count for the node, and the error monitor is sent the same failures and no others |
| `sync.bus_unavailable recovered` `{ after }` | `info` | `sync` | once, when the bus answers |
| `nats transport error` `{ transport, failures, code, error }` | `error` on `sync` and the replicator, `warn` on `web`, `worker`, `scheduler` | every role | failures 1, 2, 4, 8, … of the connection itself: the drop, each refused dial. The library giving up on a client (`…reconnect budget was spent; re-dialling`) is always written — once per client |
| `nats transport recovered` `{ transport, after }` | `info` | every role | once per recovery |
| `cache.invalidate.partial` `{ …report, failures }` | `warn` | a publisher | refusals 1, 2, 4, 8, … **per failing tier** (`broadcast`, `redis`, …). Every bust's report still carries its `errors`, and `/_x` still records every bust |
| `cache.invalidate.partial recovered` `{ tier, after }` | `info` | a publisher | once, when that tier clears again |
| `cache.broadcast.subscribe-failed` `{ attempt, error }` · `… recovered` `{ after }` | `warn` · `info` | every role | silent for the first 10 s of the boot subscribe (`FIRST_DIAL_GRACE_MS`: the bus's first dial is still in progress, and the subscribe lands within it on a healthy pod); past that the first refusal and then attempts 2, 4, 8, … warn, and the landing is said once at info — only after a warning. `As of 2026-10-17` |
| `live.bus_reconnected` `{ desynced, channelGaps }` | `warn` | `sync` | once per recovery: how many windows and topics were told to re-read |

Until 2026-10-10 a three-minute outage wrote `presence.sweep failed` at `error` every 10 s on every `sync` node (19 lines on one), sent the monitor one event per refused beat per socket, and wrote `cache.invalidate.partial` once per bust on every publisher. A failure that is **not** the bus being away — a policy that threw inside a presence leave — is still written every time, as `<operation> failed`.

## LSN cursors

Every frame carries an LSN. The client's last-seen LSN is what makes reconnect a **delta** instead of a refetch.

| Property | Behavior |
|---|---|
| Cursor | the highest LSN the client has applied, per subscription. It advances on **every patch**, not only on a snapshot — a cursor whose `at` freezes at the last snapshot fails the lag check, and every client connected longer than `maxLagMs` re-snapshots instead of resuming |
| Reconnect inside the change buffer window | delta replay from the `sync` node's ring buffer — zero DB work. A ring is complete only from its floor (its first retained change, or the read its window was filled by): a cursor older than that, such as one minted on another node, takes the snapshot |
| Reconnect outside the window | one bounded snapshot query at a current LSN. Never WAL history traversal |
| Cursor unusable and no snapshot path supplied | `X_CURSOR_STALE` |
| Ordering | LSN is monotonic per DB, so a client can never apply an older change over a newer one |
| What a cursor carries | `{ qid, lsn, ids, at }` and nothing else, `As of 2026-08-24`. `digest` and `count` were written by every snapshot, encoded and validated on every `subscribe` frame, and **read by nothing** — `digest` cost a canonical-JSON render plus a hash over EVERY ROW of every snapshot, paid once per live query per reconnecting socket in the restart storm this package is measured on, and `count` would have been wrong had it ever gained a reader, because `advance` seeds its set from the already-truncated `ids`. Both are deleted, and that is what moved the wire version ([below](#wire-protocol-version)) |

## Inbound frame order

A frame is routed as soon as it arrives, so two frames from one socket can be in flight at once. What is ordered is narrow and deliberate `As of 2026-08` — a lane per socket would put every frame behind the slowest one, and the slowest one is a subscribe's snapshot read.

| Frames | Ordered against | Why that unit |
|---|---|---|
| `subscribe` on a query | the same `sid` | `add` then `drop` for one sid, or the drop finds nothing and the add strands the subscription it was meant to end |
| `subscribe` on a topic | the same topic name | one membership, same add/drop pair |
| `hello`, everything else | nothing | they read state and write none |

**Ordering is not what bounds the caps.** N sequential subscribes still pass a check-then-act limit N times, so every refusal a subscribe can answer with — the sid claim, `maxPerSocket`, `maxPerTenant`, `maxTopicsPerSocket`, `maxTopicsPerNode` — is decided **synchronously, before the first `await`**, against a count that already includes the subscribes still in flight. One WebSocket write carrying N subscribe frames used to pass each cap N times.

## Staying connected

A dead TCP connection that was never closed fires no `close` event. Only the client can end one, so it beats.

| Property | Behaviour |
|---|---|
| Interval | **the node's**: its `hello` reply names the beat (`heartbeatMs` — a third of its presence ttl, never slower than a quarter of its idle budget, floored at 1 s) and the page's one socket follows it. **10s** until it speaks, a third of the default ttl (30 s), so one lost beat is never a false leave. The client option that set it went with `LiveClient` in 21.0.0 |
| The node names it | the node's `hello` reply carries `heartbeatMs`: a third of the presence ttl that node really runs, never slower than a quarter of its idle budget (`clientHeartbeatMs`). Optional and additive — no protocol bump; a reply without it leaves the default above |
| One beat | a `hello`, plus one subscribe frame per topic held. A beat and an opening frame are **byte-identical** — `hello` carries no cursors — so a beat asks for nothing and resumes nothing |
| Silence | nothing received for **two** intervals ⇒ the socket is closed with code `4000` and the reconnect timer arms |
| Why re-sending topics | on the node, subscribing to a topic **is** joining its presence set, and repeating the frame is the presence heartbeat |
| No knob | `realtime.heartbeatMs` in `app.config.ts` was read by nothing and is **deleted** `As of 2026-08-19`; an app still setting it fails `typecheck` with `TS2353` |

**A reconnect re-announces everything, one frame at a time.** `hello`, then one `subscribe` per registration carrying that registration's cursor, then one per topic. `hello` itself carries **neither** cursors nor topic membership — resume is decided per subscription, by the frame that also names the query and its input, and topic membership is state on the node's socket. Without the topic half a channel stayed silent from the first reconnect onwards while its handler was still installed, and its presence membership was swept.

**Writes never ride the socket**, `As of 21.0.0`. The socket `mutate` path is deleted: `useMutation` posts the mutator's action over HTTP through `clientTransport`, with an idempotency key, so a resend after a lost answer is served from the action's idempotency store. Its answer's records are adopted before the overlay goes, so nothing flickers.

## The reconnect risk

**Reconnect is the expensive part of any sync engine, and it is where naive designs fall over.** A deploy or a network blip drops N sockets at once; each client then asks "what changed since LSN X?" If the answer requires replaying arbitrary WAL history or re-running every query, a rolling restart becomes a self-inflicted thundering herd that outlasts the deploy.

| # | Mitigation | Detail |
|---|---|---|
| 1 | **Prototype before locking topology** | the reconnect benchmark: 50k sockets, forced `sync` restart, recovery time and DB load. **Measured `As of 2026-08`** — the numbers are below, and both results are committed |
| 2 | **Bounded per-query change buffer** | the `replicator` keeps a ring buffer of recent changes per query-hash. Reconnect within the window = delta replay from the buffer, zero DB work |
| 3 | **Snapshot fallback, not WAL replay** | outside the window the client gets a fresh snapshot at a current LSN. Cost is one bounded query, never history traversal |
| 4 | **Jittered reconnect-with-backoff, server-directed** | draining `sync` nodes send a `reconnect` frame with a per-client delay so clients redistribute instead of stampeding. the page socket arms **one** timer per closed socket: the node's assigned delay when there is one, otherwise `browserBackoff`, which is base 500 ms, factor 2, `equal` jitter, **capped at 4 s** (`BROWSER_RECONNECT_MAX_MS`, `thundering-herd.ts`). It was the server-side `defaultBackoff`'s 30 s until 21.0.0: after a deploy that left the returning node, and its `update-available`, unreached for 27 s. `equal` keeps a floor, so a SIGKILLed node's herd redials across a 2–4 s window rather than at once, and the `AcceptBudget` sheds the excess before any query runs. `useConnection().reconnectAt` renders the wait. The committed benchmark runs predate the cap |
| 5 | **Per-tenant subscription caps** | a registered-query explosion is a load-shedding decision, made with a limit and a typed `X_SUBSCRIPTION_LIMIT`, not by falling over. **Reachable, not yet wired** `As of 2026-08`: the boot passes no caps, and the per-tenant scope needs both `maxPerTenant` and `tenantOf`. The **per-socket** cap applies today at its default of 128 |
| 6 | **Consider wrapping an existing protocol** | if the benchmark says our matcher is the bottleneck, adopting an existing open sync protocol beats inventing one |

## The forced-restart benchmark

One harness, two runs, two questions. **Reachability**: how long until a killed node's clients are back and receiving. **Delivery**: how many patches were lost getting there. A first-delivery timer answers the first and is blind to the second, which is why there are two.

[`scripts/bench/restart-bench.ts`](https://github.com/developerz-ai/ultimate/blob/main/scripts/bench/restart-bench.ts) produced both, each with its own transcript beside it in [`scripts/bench/results/`](https://github.com/developerz-ai/ultimate/tree/main/scripts/bench/results).

```bash
# reachability, 50,000 clients — 2026-08-11
bun run scripts/bench/restart-bench.ts --clients 50000 \
  --out scripts/bench/results/50k-restart.json

# delivery, 10,000 clients, a probe every 200ms — 2026-08-17
bun run scripts/bench/restart-bench.ts --clients 10000 --probe-interval-ms 200 \
  --out scripts/bench/results/10k-restart-seq.json
```

| Setup | Value |
|---|---|
| Clients | real WebSocket connections, split across client-shard OS processes — 50,000 over 10, 10,000 over 8 |
| Server | **one** `sync` node (the shipped `syncNode`) in its own process, over `InProcessTransport` |
| Admission | the shipped `AcceptBudget` at its defaults — 500/s, burst 2000 |
| Kill | `SIGKILL`, no drain, **no `reconnect` frame** — recovery is driven only by each client's own `backoffDelay` |
| Readiness | read from the server's own socket count, never the load generator's self-report |
| Subscription under test | a **channel** topic. Neither run subscribes to a live query, so no cursor, snapshot or gap-repair path is exercised |

### Reachability — 50,000 clients

Per client, the **first channel patch received on the reconnected socket**: reconnect *and* resubscribe *and* one delivery. It is not a consistency metric, and cannot be one — see below.

| Restart-phase result | Value |
|---|---|
| Reconnected | **50,000 / 50,000** |
| Received a channel patch inside the window | **49,981** |
| Time to first patch, p50 | **54.0s** |
| Time to first patch, p90 | **105.5s** |
| Time to first patch, p99 | 127.8s |
| Time to first patch, max | **145.7s** |
| Connect attempts shed before any query path | **156,851** — the DB-load proxy: none of them reached a query or snapshot |
| New server accepting | 2.3s after the kill |

**The timings are unchanged and still stand.** Only the name was wrong: this metric was published as "time-to-consistent" until 2026-08, and it never measured consistency. The harness recorded each client's last-seen sequence number and read it nowhere, so a patch the node dropped was invisible to it by construction. Nothing here is retracted — a number that timed reachability is now called reachability.

### Delivery — 10,000 clients

Every client counts **observed sequence gaps** in the probe stream it received, per connection. An observed gap is a break between two frames one connection actually received — the publisher numbers every probe, so a missing number *between* two arrivals is a frame that was published to a subscriber and never came.

| Restart-phase result | Value |
|---|---|
| Reconnected | **10,000 / 10,000** |
| Patches received | **1,666,882** |
| Observed sequence gaps | **0** — 0 gap events, 0 missing frames, 0 duplicates, 0 publisher rewinds, 0 malformed |
| Clients that observed a gap | **0 / 10,000** |
| Time to first patch, p50 / p90 / max | 10.9s / 22.3s / 43.5s |
| Connect attempts shed before any query path | 33,424 |

**Zero observed gaps is a lower bound on loss, not a proof of zero loss.** The counter can only see a hole with a received frame on each side of it, so three losses are invisible to it by construction:

| Invisible to the counter | Why |
|---|---|
| frames lost before a connection's first arrival | there is no lower anchor to measure the gap from |
| frames lost after a connection's last arrival | there is no upper anchor, and the connection may simply have ended |
| every frame, on a connection that received nothing at all | no anchors, so the connection contributes no sequence to check |

So the honest claim is **"no client observed a lost channel frame"**, not "no channel frame was lost". Every other statement of this result on the wiki is shorthand for this paragraph.

**Not evidence about 50,000.** The 50,000-client run predates the counter and carries no delivery number; `As of 2026-08` the 10,000-client run is the only one with delivery accounting, and it does not extrapolate.

### Why delivery needs its own counter

A dropped **`records`** channel frame is now **repaired**, `As of 2026-09-22` (21.0.0). `SocketRegistry.deliver` still counts every refusal (the series
`channel_frames_dropped_total`, the warn line `channel.frames_dropped` carrying
`{ topic, dropped, total }`, and `node.sockets.droppedChannelFrames` for a test that cannot scrape).
It also marks the (socket, topic) as gapped, and once the socket drains it sends one `replay-gap`
frame, counted in `channel_replay_gaps_total`. That series counts gaps announced, by design: the
repair happens in the browser. The client (`client-channels.ts`) keeps a cursor per channel. On
`replay-gap`, or a new epoch, it re-runs the channel's `catchUp` query and applies the frames that
arrived meanwhile after it. A dropped `events` frame is ephemeral by design and is only counted. The live-query path
already repairs the same drop, as it has since 2026-08: the subscriber is marked desynced, and the
next change re-snapshots it.

What it is **not**: a multi-node result — neither run crossed NATS, so this is **per-node recovery**, not fanout. Not a throughput figure either: no requests/sec, no message rate, no sustained-load number. Per-node socket capacity in the tables above this section is still a target derived from Bun's native WebSocket implementation, not a benchmark result. Long-running Bun processes are also less battle-proven than Node's; sustained-socket memory profiling is explicit roadmap work.

## `sync` drain

`sync` holds no durable state, so a restart is only dangerous in aggregate. Closing 50,000 sockets at once means 50,000 simultaneous reconnects, all resubscribing, all asking "what changed since my LSN?" — a self-inflicted DDoS landing during a deploy, when capacity is already reduced. Worse, it is fractal: surviving nodes overload, drop connections, and the herd re-forms.

So the drain is **server-directed**:

```json
{ "type": "reconnect", "v": 3, "afterMs": 1830, "reason": "drain" }
```

`ReconnectFrame` (`packages/realtime/src/sync-protocol.ts`) has those four fields and no cursor: `v` is `PROTOCOL_VERSION`, `reason` one of `drain`, `overload`, `rebalance`.

| Property | Effect |
|---|---|
| Per-client `afterMs`, jittered over a window | reconnects arrive spread out, not as a spike |
| The window is `syncNode({ drainSpreadMs })`, default 30s | each socket draws its own delay inside it (`drainPlan`) |
| The resume point is the client's own cursor | on reopen the client sends `hello`, then one `subscribe` per live query carrying that query's cursor, and re-announces each channel from its own; the node replays from its change buffer when the cursor is inside it, one snapshot when not (`live-resume.ts`) |
| Clients redistribute | the LB places them across remaining nodes; no sticky session to honour |
| Client-side backoff is a floor, not the mechanism | a client that loses the socket without a frame still backs off exponentially with jitter |
| Ordering in the drain sequence | `/readyz` → 503, stop new subscribes, send `reconnect` frames, close cleanly, flush spans, exit 0 |

Full drain sequence per role: [Deployment](/deployment).

## Wire protocol version

`PROTOCOL_VERSION` is **3** `As of 2026-09-22` (21.0.0; it was 2 from 2026-08-24) ([`packages/realtime/src/sync-protocol.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/realtime/src/sync-protocol.ts)). A mismatch is `X_PROTOCOL_VERSION` on the frame, with one instruction, rather than a per-field decode error nobody can act on.

**Clients and `sync` nodes must be redeployed together across this bump.** A cursor rides in BOTH directions — the client's `subscribe` and the node's `snapshot` — so a skew breaks resume from either side.

| Skew | What happens |
|---|---|
| new node ← old client (v2) | `decode` refuses the frame: `X_PROTOCOL_VERSION`. v3 deleted the `mutate` and `rebase` kinds (writes go over HTTP) and the `presence` kind (a roster is an `events` payload now), added `records`, `events` and `replay-gap` for channels, and narrowed the `ack` frame to a refusal: it carries the sid of a subscription the node refused, or the socket id for a frame it could not read, and is never a write receipt |
| old node ← new client (v3) | the same, at the other end |
| v1 ↔ v2 (2026-08-24) | the old node's `cursor()` reads `digest` through a `str()` that **throws on an absent field**, which is precisely why that was a version and not a silent widening |

**The version guards incompatibility, never novelty.** Two earlier realtime changes deliberately did not move it: an additive optional field (`snapshot.entity`) and a removed field read through `list()` (`hello.resume`) are both readable in either direction, because `decode` builds a whitelist object and `list()` answers `[]` for an absent key. `cursor()` is the other kind of reader — `str`/`num` throw — so removing a field it reads is the one shape that needs the number.

## Build skew

A client on build `A` connecting to a `sync` node on build `B` is **accepted**, then sent an `update-available` frame carrying the node's `buildId`; the socket is not killed. Skew is the build the client claims against the node's own: `?build=` on the dial, and the `hello` frame's `buildId`, which the node records on every hello and which is the later word — a dial without `?build=` starts as "not skewed" and the first hello settles it. Neither side's build moves while a socket is open, so the answer is a property of that socket for its lifetime — a client learns about a deploy on the socket it opens against the new node, never on one it is already holding. Every `hello` on that socket re-reports the same answer, the heartbeat's included. The client's `AppUpdateAvailable` signal flips and the app renders its own update affordance. See [PWA and offline](/pwa-and-offline).

## Device sockets

A long-lived socket for a client that is **not a browser** — a device agent that dials out and
receives work. `deviceSocket()` is a factory over `route`: the value is a websocket mount (`path`,
`fetch`, `websocket`) plus `request()`, which reaches one connected device from any process.
From `@ultimat3/realtime/server`, `As of 2026-11-06`.

```ts
import { userActor } from '@ultimat3/core';
import { deviceSocket } from '@ultimat3/realtime/server';

export const deviceLink = deviceSocket({
  path: '/link',
  resolveToken: (token) =>
    token === 'enrolled' ? { actor: userActor({ id: 'u1', orgId: 'o1' }), deviceId: 'desk-7' } : null,
  maxSocketsPerActor: 10,
  heartbeatMs: 15_000,
  onFrame: ({ frame, device }) => ({ ack: device.deviceId, received: frame }),
});

// From any action or job, on any node — scoped to the caller's tenant:
export const ask = (ctx: { readonly actor: ReturnType<typeof userActor> }) =>
  deviceLink.request(ctx, 'desk-7', { job: 'sync' }, { deadlineMs: 30_000 });
```

| Concern | Rule |
|---|---|
| Credential | `Authorization: Bearer <token>` on the upgrade, decided by `resolveToken` before any frame. None: `401` + `www-authenticate: Bearer`; refused or expired grant: `401` + `Bearer error="invalid_token"` (`X_SOCKET_UNAUTHENTICATED`). A grant's `expiresAt` also closes an open socket (`1008`) |
| Slow token service | `resolveToken` is cut at `DEVICE_AUTH_DEADLINE_MS` (5 s, `503 X_SOCKET_AUTH_UNAVAILABLE`); past `DEVICE_MAX_PENDING_UPGRADES` (256) parked upgrades, and past the accept budget, the dial is shed `503` with a jittered `retry-after-ms` |
| Origin | a device sends no `Origin`. An upgrade that carries one is a browser: `403 X_SOCKET_ORIGIN_REFUSED` unless `allowedOrigins` lists it exactly |
| Caps | `maxSocketsPerActor` (default 10) per node, reserved before `server.upgrade`: `429 X_SOCKET_LIMIT`. One socket per device: a newer one, on any node, closes the older (`1008 replaced`) |
| Frames | JSON text. `DEVICE_FRAME_MAX_BYTES` (256 KiB) is the framework's and is not an option: Bun's `maxPayloadLength` and the handler both enforce it (`1009 X_DEVICE_FRAME_TOO_LARGE`); a `request()` over it throws before sending. `onFrame` runs one frame at a time per socket, as the device's actor; past 32 queued the socket closes `X_FRAME_RATE_LIMIT` |
| Heartbeat | the node names `heartbeatMs` in its `hello`; a device silent for two beats is closed `4001` |
| Addressing | the holder registers the device in the bus's shared set (TTL three beats) under `(tenant, deviceId)`; `request()` looks it up and publishes to that node. Nobody holds it, or another tenant does: `X_DEVICE_OFFLINE` at once. No answer in `deadlineMs` (default 30 s, max 600 s): `X_DEVICE_REQUEST_TIMEOUT`. A socket that closes answers its open requests `X_DEVICE_OFFLINE` |
| Drain | SIGTERM: stop accepting, then the sync node's own `drainPlan` + `reconnect` frame per socket, then close `1001` |
| Reconnect | `deviceAgent()` (a Bun client) redials on realtime's one jittered backoff, or at the slot a `reconnect` frame assigned |

**Wire.** Device → node: `{ type: 'beat' }`, `{ type: 'frame', frame }`, `{ type: 'response', id, frame }`.
Node → device: `{ type: 'hello', heartbeatMs, maxFrameBytes }`, `{ type: 'request', id, frame }`,
`{ type: 'frame', frame }` (what `onFrame` returned), `{ type: 'error', code }`, and the sync
protocol's `reconnect`.

**Where it runs.** Every role attaches every declaration to its bus (`hostDeviceSockets`), so a
worker's job can `request()`; only `web` holds sockets. Across replicas every role needs `NATS_URL`
set, or one `transport` in `apps/<app>/runtime.ts` — on the in-process bus a device is reachable from its own process only.

**Where it is served.** The `web` role's own listener, beside the sync node's `/_x/sync`: the boot
hands `httpServer` one mount per declared path beside `syncMount`, every path is checked free of a
route (`X_ROUTE_CONFLICT`), and each upgrade goes to the path's CURRENT declaration, its frames to
the declaration that upgraded it. The same in the container (`ROLE=web`) and under `x dev` — no
port, no option. A declaration is found by the boot's module scan, so it lives in any module under
`apps/web/app/` (`examples/dummy/apps/web/app/devices/link.ts`). `As of 2026-11-06`.

**Under `x dev`.** A save re-evaluates the module that declares a socket (and anything it imports),
so its `deviceSocket()` runs again. Each rescan is a new generation (`dev-rebuild.ts`): the path's
new value replaces the old one, is attached, and serves the next upgrade; the old one's open sockets
get a `reconnect` frame and a `1001` close, and `deviceAgent()` redials onto the edited handler. A different module declaring a path an earlier scan holds is `X_CONFIG_INVALID`, naming both files — the holder was not re-evaluated; moving a declaration is `x stop && x dev`. A
path first declared by a save is not served until a restart (`X_DEV_RESTART_REQUIRED` on `/_x`):
the listener's websocket paths are fixed when it binds. A container scans once, so never replaces.

| Rule | Error |
|---|---|
| an app module calls `server.upgrade(`, hands `Bun.serve` a `websocket` key, or writes an inline `websocket: { … }` handler set | `X_SOCKET_HAND_ROLLED`, `x verify`'s `boundaries` step, with the file and `meta.line` — declare `deviceSocket({ path, resolveToken, onFrame })` instead |
| two declarations at one path in one scan | `X_CONFIG_INVALID` at the second `deviceSocket()` |
| a declaration at a path a route already answers | `X_ROUTE_CONFLICT` at boot |

## Errors

| Code | Cause | Fix |
|---|---|---|
| `X_TOPIC_FORBIDDEN` | an undeclared channel name, or the declaration's policy denied the actor | `x policy list --json`, then widen the policy on the `channel()` declaration, or subscribe as an actor it allows |
| `X_SUBSCRIPTION_LIMIT` | a socket, tenant or node reached a cap; the error names which scope refused, and which knob | `raise maxPerSocket / maxPerTenant / maxEntries on the LiveQueryRegistry (per socket, default 128), or unsubscribe unused live queries` — a **channel topic** cap answers `maxTopicsPerSocket` (64) / `maxTopicsPerNode` (10,000) on the `ChannelHub`. All constructor options, none an `app.config.ts` field |
| `X_PROTOCOL_VERSION` | client and server disagree on the wire format, or a malformed frame. The wire is at **3** `As of 2026-09-22` ([above](#wire-protocol-version)) — a v2 client and a v3 node cannot talk in either direction, so the two are redeployed together | `x build && redeploy the client; the sync node sends 'update-available' before it drains` |
| `X_LIVE_QUERY_UNKNOWN` | a `subscribe` frame named a live query this node does not have | `x queries list --json` |
| `X_CURSOR_STALE` | resume cursor cannot be honoured and no snapshot path was supplied | `pass 'snapshot' to resumeFrom() so the fallback path can re-snapshot instead of failing` |
| `X_REBASE_CONFLICT` | a `custom(merge)` returned something other than a row with a string `id` | `set conflict: 'server-wins' on the mutator, or return a row from custom(merge)` |
| `X_REALTIME_UNINSTALLED` | a hook ran in a browser island whose bundle never called `installRealtime()` | `x build`, which installs it on every island whose own graph imports realtime. An island that reaches realtime only through a package calls `installRealtime({ signal: createSignal })` in `mount` |
| `X_SYNC_UNCONFIGURED` | a live hook needed the page socket, and neither `installRealtime({ sync })` nor `<meta name="ultimate-sync">` named a target. The meta is not rendered at all when `app.config.ts` says `realtime: { enabled: false }`: no node is started, so there is nothing to dial (22.2.2) | serve the page through `x dev` or the container, which render the meta, or pass `installRealtime({ signal, sync: { url, buildId } })` |
| `X_RECORD_REJECTED` | a row reached the record store with no key, or not as an object | `x entities show <entity> --json`, then return whole rows |
| `X_MUTATOR_CLOCK_MISSING` | `conflict: 'last-write-wins'` with no number `updatedAt` on the entity | add the clock, or declare `conflict: 'server-wins'` |
| `X_TRANSPORT_UNAVAILABLE` | the fanout bus is down. At boot only for `sync` and the replicator, after a bounded wait; on `web`, `worker` and `scheduler` it is what a publish rejects with while the bus is away ([what each role needs](#what-each-role-needs-from-the-bus)); on a socket it is the one refusal a client treats as transient ([a bus outage, seen from an open page](#a-bus-outage-seen-from-an-open-page)) | `x doctor — then check NATS_URL points at a reachable nats-server` |
| `X_TRANSPORT_PROTOCOL` | the bus answers in a protocol this build does not speak | `the bus must be nats-server >= 2.11 with JetStream enabled (nats-server -js)` |
| `X_DEVICE_OFFLINE` | `deviceLink.request()` named a device no node holds a socket for under the caller's tenant | `x doctor --json`, then retry once the device redials |
| `X_DEVICE_REQUEST_TIMEOUT` | the device took the request and sent no `response` with its id before `deadlineMs` | a longer `deadlineMs`, or answer every request the node sends the device |
| `X_DEVICE_FRAME_TOO_LARGE` | a device frame, or a `request()` frame, past `DEVICE_FRAME_MAX_BYTES` | split the payload, or store the bytes and send their key |
| `X_BUILD_SKEW` | client build's contract is incompatible with the server's | reload the client; see the fix line on the error |

Verbatim shapes: [`packages/realtime/src/errors.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/realtime/src/errors.ts). Full index: [Error codes](/error-codes).

## Rules

- Truth is always the server. A client is never the merge authority.
- One authz system — the `query`'s `policy`, re-evaluated per delivered row.
- `sync` is stateless. Any state that must survive a restart lives in Postgres or NATS.
- A live query must be deterministic and bounded (`orderBy` + `limit`), or `x verify` rejects it.
- Presence and typing indicators are tier 1 forever — never model ephemeral state as rows.
- `local` does no I/O, no `Date.now()`, no `Math.random()`. It must be replayable.
- Offline writes go through the tier-3 mutator queue, never Background Sync guesswork.
- Climb the ladder for a property you need, not for a tier you like. Tier 3 costs a durable client store forever.