Ultimate

Realtime

Three tiers, one ladder. Same mutator shape at every rung — climbing is a config change, never a rewrite.

As of 2026-08. Stable API — semver from here (Upgrading). Tiers 1–2 ship. Tier 3 (local-first) has not shipped, As of 2026-08.

Two entries: . is the browser's, ./server is the node's

@ultimat3/realtime ships two entry points, and which one a name lives in is a mechanical fact rather than a convention — the barrels are disjoint, and a test asserts it.

EntryHoldsReaches
@ultimat3/realtimeuseQuery, useRecord, useMutation, useConnection, installRealtime, hasPageSocket, RecordStore, channel() and topic(), the offline queue, the wire protocol, cursorsan island, a browser bundle
@ultimat3/realtime/serversyncNode, ChannelHub, SocketRegistry, LiveQueryRegistry, NatsTransport, openNatsClient, the replicator, the change feed, pg replicationa sync node, a worker, server.ts
import { useQuery } from '@ultimat3/realtime';             // island
import { syncNode } from '@ultimat3/realtime/server'; // sync node

A server render is not a missing install. With no DOM, every read hook answers pending with no subscription, and useMutation() refuses with X_LIVE_SERVER_RENDER. hasPageSocket() answers false there, every time, so it is not a guard that makes a page-body read safe: a browser-only read in code no island imports is X_LIVE_ROUTE_NO_ISLAND at the budgets step, hasPageSocket() included. In a browser, a hook in a bundle that never called installRealtime() is X_REALTIME_UNINSTALLED.

A file that needs both writes both imports. Nothing was deleted in the split — if an import stops resolving after upgrading to 8.0.0, the name moved to ./server.

Why it is two entries and not one: a single barrel carried useLive beside openNatsClient, so bun build --target=browser on an entry importing only the hook failed with "Browser build cannot require() Node.js builtin: stream/web", out of nats — the island this page tells you to write could not be bundled at all. The package's sideEffects array is what lets the bundler shake the bus out; the split is what makes "the client entry cannot reach the bus" something a build cannot quietly undo, since a namespace import or an export * defeats tree-shaking.

The ladder

TierNameYou writeServer ownsClient ownsCost
1Channelschannel('org-feed', { params, policy, catchUp, records?, events? }), then useChannel(decl, params)truth + fanoutsubscription~0 — pubsub over WS
2Live queriesquery({ live: true, sql })truth + change detectiona reactive result setone replication slot + a matcher
3Local-firstthe same mutator, plus entity(name, { persist: true }), see belowtruth + rebasea durable local store, offline writesIndexedDB + the one outbox

Tier 1 for presence, typing indicators, toasts, cursors. Tier 2 for "the list updates when someone else edits". Tier 3 for offline-capable apps.

Row 3 is on the entity, not the query. persist: true is an entity() option in 21.0.0, read by realtime's IndexedDB persister and one outbox. Read the row's own section before writing anything against it.

Tier 1's events are at-most-once, by construction; its records frames are repaired. A dropped events frame is counted and logged, never resent. A dropped records frame is marked on the server, which sends the socket a replay-gap frame once it drains, and the client re-runs the channel's catchUp read (21.0.0; below). State that must arrive belongs on a live query or a records channel; ephemera belong on events.

Tier 2 covers what people almost always mean by "make it realtime": the list updates without a refresh, and my own click feels instant. It delivers both with no client database, no client schema versioning, no conflict-resolution UX, and no offline-write semantics to design. Tier 3 buys exactly one additional property — writes that survive being offline — and costs a durable local store, a rebase log, client migrations, and a conflict story per mutator. Charging every app for that is how realtime frameworks become slow frameworks.

Same mutator at every rung

// mutator (action + optimistic local twin)
export const likePost = mutator({
  // REQUIRED (`X_MUTATOR_NOT_IDEMPOTENT`): a replay under the same Idempotency-Key answers the
  // first result instead of running `server` again.
  idempotent: true,
  // Convergent, not incremental: `local` replays on every rebase, so applying it N times has to
  // equal applying it once — `likedByMe` is what makes the second application a no-op.
  local(tx, { postId }) {
    tx.posts.update(postId, (p) =>
      p.likedByMe ? {} : { likedByMe: true, likeCount: p.likeCount + 1 });
  },
  async server(ctx, { postId }) { return ctx.posts.like(postId); },
  conflict: 'server-wins', // | 'last-write-wins' | custom(merge)
});
TierWhat local doesWhat server does
1not calledruns, publishes an event
2applies to the page's record-store overlay immediately, visible in every island; dropped when the HTTP answer's records land, or on refusalruns as an HTTP POST to the action's path, with an idempotency key; its records flow back into the store
3applies to the overlay, and when the network takes nothing the write is queued in the page's outbox (the call resolves undefined) with the overlay kept on screenruns on replay, over HTTP with the same idempotency key

A mutation the server refuses is rolled back, not retried. The optimistic write goes, and so does every write made after it — undone newest first, then replayed without it, which is sound only because local is pure. The refused intent is dropped from the rebase log: a denial is a decision about that intent, and retrying it would put the write the server refused back on the screen. A refused key stays in the queue as failed for the UI to render, and re-issuing the same idempotency key is treated as a new intent with a new sequence at the back of the queue, not a collapse onto the denial.

local is a pure function of (tx, input), therefore replayable. Hence the rule: no I/O, no Date.now(), no Math.random() inside local. Same input, same patch, every replay.

The fluent surface

interface Mutator extends Action — a mutator is an action, so every action member is already on it (likePost.openapi(), .client({ baseUrl }), .job(), .contract(), .as(), .describe(), .named(), and the lifted .input .output .policy .mcp). The three names it was authored with — .local, .server, .conflict — project unchanged, plus a brand and its own descriptor. Both halves are reached through the mutator itself, likePost.local(tx, input), never a declaration object. A mutator has no .def.

MemberIsRule
.local(tx, input)the optimistic twinruns against the local store, synchronously, no I/O, returns void. Takes already-parsed input — nothing re-parses on this half
.server(ctx, input)the authoritative halftakes raw input, like the callable and .as(), because this is where parsing happens
.conflictthe rebase strategy, lifted'server-wins' | 'last-write-wins' | custom(merge) — the declared value verbatim, merge function included
.isMutatorthe brandalways true; isMutator(value) is the guard
.describeMutator()the manifest rowthe action descriptor plus kind: 'mutator' and the resolved strategy name — custom(merge) describes as 'custom'
.describe()the action descriptorstill reports kind: 'action', with mutator: true — that flag is what puts the mutator count in x.manifest.json. describeMutator() is the one that says kind: 'mutator', plus the resolved strategy

.server() routes through the action's own callable, never the declared server. It calls base(input, { ctx }), and that callable is invoke — so the authoritative half cannot skip the input parse, the policy or the output parse. Reaching the declaration from there would be a second execution path, which is the one thing @ultimat3/action exists to prevent. Hence the guarantee this page rests on: the offline/realtime half of a mutator gets exactly the same actor gate → parse → policy → handle → parse core as an HTTP call, an MCP tool call and a job run, because it is that call. A denied .server() never reaches the declared half, and neither does one whose input fails the schema — X_UNAUTHENTICATED, X_FORBIDDEN and X_INPUT_INVALID come back from the same core that serves the HTTP route.

The parsed-vs-raw asymmetry between .local() and .server() is deliberate, not an inconsistency. local replays on every rebase and must stay a pure function of (tx, input) — no I/O, no clock, no randomness — so re-parsing on each replay would be wasted work on input that was already validated once.

.local() is the name. No alias: the old .applyLocal is gone, not deprecated.

.named() rewraps rather than dropping the twin — a renamed mutator keeps both halves, its conflict, and every inherited action member.

Tier 3 shipped in 21.0.0

Opt-in per entity, and shipped in 21.0.0 (CHANGELOG.md). The 20.x shape (store, queue and log on LiveClientOptions, MemoryLocalStore, a createOpfsLocalStore() that threw X_NOT_IMPLEMENTED) is deleted with LiveClient. What replaces it:

Piece21.0.0
what persistsentity(name, { persist: true }), default false. The server renders the persisted record types into a private document's <meta name="ultimate-persist"> (packages/cli/src/page-sync.ts, persistedRecordTypes)
whereIndexedDB, every entry keyed [scope, type, key]. Scope is p:<principal> or anon; an unscoped page persists nothing
what is writtensynced rows only, never an overlay. Debounced 250 ms, flushed on pagehide and when the page is hidden
bootthe rows are restored before the socket connects, as provisional records; the first server row for a key wins outright
an answer that did not carry every row the overlay wrotethe overlay stays for those rows until a server row for them arrives, capped at 10 s (DEFAULT_AWAIT_SERVER_MS), after which server truth stands
a write with no responseonly X_CLIENT_TRANSPORT_FAILED with meta.failure: 'network': useMutation queues it in the page's one outbox under its idempotency key, keeps its overlay on screen, and resolves undefined. 'status' rejects and drops the overlay; 'body' (a 2xx that was not JSON, which may have landed) rejects and keeps the overlay until the next server row
replayin order, over HTTP, each write with its original idempotency key: whenever the page socket (re)connects, on online, and on the service worker's OUTBOX_DRAIN_MESSAGE (PWA and offline). A retryable failure stops the pass; a refusal is final and rolls back its overlay. At least once: an entry leaves the disk when its ack is written, so a 2xx that reached a document already being replaced is sent again by the next one, under the same key — answered from the action's idempotency store (a mutator() is idempotent: true or it does not declare), never applied twice. A 409 X_IDEMPOTENCY_CONFLICT for a first attempt still running stays queued and is asked again
a write queued offlineflushes the persisted records it touched first, so a reload inside the 250 ms persist debounce keeps the row the write changed. No replay runs while navigator.onLine is false (page-outbox.ts)
a principal changewithin one page (rescope()): the previous principal's rows and queue are wiped, and writes still queued are lost, deliberately. A replay on the wire at that moment is abandoned before its next write: nothing queued by one principal is sent under the next one's session. A write being queued at that moment is refused — mutate rejects with X_OFFLINE_QUEUE_ABANDONED and its optimistic twin is taken back. A sign-out that navigates to a new document (the reference app's idiom: the endSession action at POST /api/sessions/end, posted by a native form) clears the browser first: its response carries signOutHeaders() from @ultimat3/auth, whose Clear-Site-Data: "cache", "storage" drops IndexedDB, local storage, the service worker and its cache, in a secure context. The next boot is the second line: it wipes every stored scope except the current principal's before restoring anything (packages/realtime/src/boot.ts, wipeOthers). An unscoped page wipes nothing
blocked storagememory, plus one X_LOCAL_STORE_UNAVAILABLE warning; nothing survives a reload

The boot runs as one deferred classic script per page (/_x/assets/page-boot/<hash>.js, from @ultimat3/realtime/boot). It is rendered only on a scope-tagged (private) document that emitted an island reaching @ultimat3/realtime. It wipes other scopes, restores this principal's rows, and opens the outbox, so a reload that opens no live hook and writes nothing still replays what the previous load queued.

Trade-off: two principals in two tabs of one browser. The newer page's boot wipes the other principal's rows and queued writes from disk. The older tab keeps its records and its queue in memory, and re-persists them on its next write; if it closes first, what it had queued offline is gone.

Live query in the browser, end to end

Written As of 2026-09-05 because four agents building one app each polled a query from an island instead of subscribing. Rewritten for 21.0.0, which deletes the app's socket adapter, its sync URL and its LiveClient: the page's one socket is the framework's.

StepWhereWhat
1. the dependencyroot package.jsonbun add @ultimat3/realtime. x new installs @ultimat3/query and not @ultimat3/realtime
2. the queryapp/<feature>/live.tsquery({ …, live: true, subscribes: ['posts'] }), bounded and ordered; x g query <name> --feature <slice> --live writes it
3. the refthe island{ name: 'liveFeed', live: true }, the query's registered name, never the query value: importing that drags its whole read path into the island. live is stated beside the name, not derived (see below)
4. the installnothing to writex build builds every island whose own import graph reaches @ultimat3/realtime from an entry that calls installRealtime({ signal: createSignal }) first, for about 100 B. An island not reaching realtime pays nothing. The sync target is read from the document's <meta name="ultimate-sync"> (Configuration, SYNC_URL)
5. the hookthe componentuseQuery(ref, input) returns an AsyncState accessor. In an island, never a page body: a page component never runs in a browser, so the hook answers pending there forever
// app/feed/feed.island.tsx — the only module of /feed a browser downloads
import { useConnection, useQuery } from '@ultimat3/realtime';
import { For, Match, Switch } from 'solid-js';
import { render } from 'solid-js/web';

type FeedRow = { id: string; title: string };
const LIVE_FEED = { name: 'liveFeed', live: true } as const;

export function mount(el: HTMLElement, props: { orgId: string }) {
  el.textContent = '';                       // Solid's render APPENDS; the server's loading shell goes first
  render(() => <Feed orgId={props.orgId} />, el);
}

function Feed(props: { orgId: string }) {
  const feed = useQuery<FeedRow>(LIVE_FEED, { orgId: props.orgId }); // subscribe, once
  const connection = useConnection();
  return (
    <Switch>
      <Match when={feed().status === 'pending'}><p>loading…</p></Match>
      <Match when={feed().status === 'failed'}><p>could not load the feed</p></Match>
      <Match when={'data' in feed()}>
        {connection.offline && <p>offline — showing the last rows</p>}
        <ul><For each={(feed() as { data: readonly FeedRow[] }).data}>{(row) => <li>{row.title}</li>}</For></ul>
      </Match>
    </Switch>
  );
}

feed() is an AsyncState: pending, then ready (or refreshing) carrying the rows, or failed. It re-renders on every patch the sync node sends, with no timer and no refetch. The list holds ids; the rows come from the page's one RecordStore, so a write to a record through any path re-renders every list showing it. A non-live query uses the same hook: drop live, and name its entity ({ name, entity: 'post' }) so its rows are store records too.

Two limits, As of 2026-09-22. x build sees realtime only in the island's own graph. An island that reaches it through a package, rather than importing it, calls installRealtime({ signal: createSignal }) in mount itself, or throws X_REALTIME_UNINSTALLED. And live and entity on the ref are stated by hand, not derived from the query declaration. Both are in the plan-101 DX ledger.

Which islands wait for the page's restore

As of 2026-10. On a signed-in page with a sync node, the page boot restores persisted records and the queued writes from disk. Server markup rendered before a queued write would paint the old value first, so an island that can show such a value is held: hidden (visibility:hidden, its box kept) until it mounts over the restored store.

IslandHeldmount waits for the restore
takes useQuery, useRecord, useRecords, useMutation, useMutationQueue, useOutbox, or any other name not listed below, anywhere in its own graphyes — until it mounts, 3 s at mostyes, 1 s at most (FIRST_PAINT_HOLD_MS)
takes useQuery or useRecord (and otherwise only the follow list below), in an app that declares no persist: true entityno — the boot restores no record they could showno
takes only useChannel, usePresence, useConnection, hasPageSocket, useRecordChanges, pollEvery, readPresence, channel, channelRef, topicno — its server markup is on screen from the first byteno — it mounts as soon as the page runtime is there
takes the barrel with no list of names (import * as, export *, import())yesyes
reaches no realtimenono
FactRule
who decidesx build, from the names each module of the island's own graph imports from @ultimat3/realtime (packages/cli/src/island-realtime-state.ts). Nothing to declare
whether the app persistsread off the entity registry the build loaded (appPersistsRecords). An empty registry is unknown, and keeps the held answer. useMutation is held either way: the outbox rebuilds a queued write's overlay
the events-only listan island that adopts the server's markup and re-reads its own query when a channel event arrives is never hidden: nothing the restore brings back can contradict what the server drew
with scripting offno island is ever hidden: @media (scripting: none) in @ultimat3/ui's global.scss cancels the hold. A browser without that media feature, or a runtime that was blocked, shows the markup at the 3 s cap
a prebuilt imageeach island's answer rides its chunk, and the store x build --target docker writes records it, so a pod that builds nothing renders the same page boot and the same holds as x dev
the blind spota package that reads the store for the island is not seen, as with the install above

In development, the feed is this process

As of 2026-09-05. The sync node's changes come from Postgres logical replication — a real DATABASE_URL and the replicator role (x dev --role replicator). The embedded database has no walsender, so under a plain x dev a subscription took its snapshot and then heard nothing: every --live query in every scaffolded app was dead in development, which is where an author first tries one. x dev now installs the in-process bridge — startLiveReplicator, on @ultimat3/realtime/server since 2026-09-23 (@ultimat3/testing no longer exports it), the row observer the framework's own live tests run on — whenever the database is embedded, and the ready line says so:

db=live=What reaches a subscriber
embeddedin-processevery repository write this process makes — a write made by another process is invisible, which is the bridge's honest bound and holds by construction under x dev
externalreplicationwhat the WAL decoder delivers, in this process (--role replicator) or in another
no sync rolenonenothing; there is no node to feed

Never both: with a real database the decoder already carries this process's own writes, and a bridge beside it would deliver each of them twice. --json carries the same fact as liveFeed.

An entity the app publishes itself (recordPublisher, for a deployment with no replicator) is the bridge's one exception: it stops carrying that entity's writes to channels, so each row reaches a channel member once — from the app's publish, as in production. Live queries still hear the bridge.

Live query pipeline

Postgres (logical replication slot)
    │  WAL decode, per-table
    ▼
replicator role  ──► change feed (ordered, per-table, with LSN)
    │
    ▼
incremental matcher  ──► for each registered live query: does this row enter/leave/update the result?
    │                      (predicate + order + limit evaluated against the changed row only)
    ▼
NATS subject per query-hash + tenant  ──► fanout
    │
    ▼
sync role (stateless)  ──► WS frame: {qid, op: insert|update|delete, row, lsn}
    │
    ▼
Solid signal patch — fine-grained, no re-render of the list
StageOwned byGuarantee
WAL decodereplicator (1 per DB)ordered by LSN, at-least-once. A stream that ends is restarted by the same process from the last LSN it published, on a jittered backoff; replicator.running is false until it is back
matcherreplicatora change touching no registered query costs one predicate check
fanoutNATSsubject = hash(query, params, tenant); no per-socket state on the bus
socketsync (stateless, no sticky sessions)client re-subscribes anywhere; scales on connection count
authzpolicy on the queryevaluated at subscribe and re-checked on row delivery — a row that fails the policy is dropped, never sent

The matcher is why this is affordable: membership is decided from the changed row plus the query's predicate, order, and limit — never by re-running the query. Which is exactly why live: true requires a deterministic, bounded sql (Queries and live queries).

One record per type:key

The page holds one RecordStore per tab, shared by every island through a globalThis handle. It is an identity-mapped store: a record is one value per record type (the entity name) and record key, however many lists, islands and paths hold it. It shipped in 21.0.0 (packages/realtime/src/record-store.ts).

ConsequenceWhy it matters
A socket patch, an HTTP answer's records and an optimistic write all move the same recordtwo components rendering post #7 cannot disagree about it, in one island or across two
A list holds ids; the rows come from the store, for a live and a non-live query alikea write to a record re-renders every list holding it, with no refetch
Two layers: synced is server truth, and the overlay is every pending optimistic write, replayed over it on every changerolling back an optimistic write cannot delete a row the server has since sent
A patch that omits a field never clears ittwo queries may project different columns; a narrower answer must not blank what a wider one renders
A row that is not an object, or has no key, is dropped and reported (X_RECORD_REJECTED)a keyless row would overwrite another record
A record lives exactly as long as something holds itthe last release drops it, so an infinite scroll is not a leak
A list that must grow with the store is useRecords(type, { where?, order? })a useQuery list holds its own answer's keys and useRecords(type, keys) the keys it was given, so a record a channel's records frame adopts reaches neither. The whole-type form shows every record of the type the store holds — ready from its first read, ordered by key unless order says otherwise, none evicted while it is held. The channel-records counterpart of a live window (25.0.0, #680)
An answer is the same object until what it shows moveduseQuery, useRecord and useRecords hand back the previous answer and its data array while the status and every row object match, so a poll or refetch that found nothing new re-runs nothing derived from it; a whole-type reader with where is not told of a record it neither shows nor would show (29.2.0)
A busy feed is throttled by its reader: useRecords(type, { throttleMs })the first change at once, a burst as one notification at the end of the period, the last always landing; release() drops what is owed. Absent or 0: every change. A non-finite or fractional value is X_INVARIANT (29.2.0)
A reader that only needs to know THAT records moved is useRecordChanges(types, listener)it is called once per store update with the moved keys per type and holds no row and no type, so it pins nothing: re-rendering a server page on a channel's records costs no memory. Follow-only, so its island is never held

The key comes from the server, on every path. A browser cannot compute a key without importing the app's entity() declarations, so it never does.

PathWhere the key is
an HTTP enveloperecords: record type → record key → row, keyed by the entity's recordProjection
a live snapshotkeys, parallel to rows, sent only when some key is not its row's id
a live patchkey, sent only when it is not the row's id

So an entity with a composite primary key can be live. A non-live useQuery lists its records in the envelope's records[type] key order, which rowsOf fills in data order. An answer with no envelope holds its rows itself, and they are not records. A live window whose node named no type keeps its rows under ?query:<name> (unnamedType), so two unnamed windows never merge two entities' rows.

Channels are declared

As of 2026-09-22, 21.0.0. A channel is a declaration, never a string. The hub API that took raw topics (hub.guard, subscribe(socket, topic), publish(topic), publishFrame, channelFrame, TopicGuard*, the nodeId option, and a { kind: 'topic' } subscribe target) is deleted.

A channel is written once, in two halves on one ref, the same split as a query's QueryRef: an island holds the browser half and never imports an entity or a database package.

// app/feed/channel-ref.ts — the BROWSER half: name, params, catch-up read. No entity, no policy.
import { channelRef } from '@ultimat3/realtime';

export const ORG_FEED = channelRef('org-feed', {
  params: ['orgId'],
  catchUp: { name: 'orgFeed' }, // the query re-read on replay-gap or a new epoch
});
// app/feed/channels.ts — the SERVER half, on the same ref. Registers on import.
import { channel } from '@ultimat3/realtime';
import { Post } from '../posts/entity';
import { ORG_FEED } from './channel-ref';
import { feedRead } from './policies';

export const orgFeed = channel(ORG_FEED, {
  records: [Post],  // every param must be a column of each listed entity, or refused
  policy: feedRead, // decided at subscribe, with the params as input
  events: true,     // ephemeral payloads: typing, presence
});
// in an island: the ref, never the server half (which would bundle the entity and its driver)
import { useChannel } from '@ultimat3/realtime';
import { ORG_FEED } from './channel-ref';

declare const orgId: string;
const feed = useChannel(ORG_FEED, { orgId }, { onEvent: (event) => console.debug(event) });

channel(name, { params, catchUp, policy, … }) in one call still exists for a channel no browser holds. policy is required on either form: a declaration without one is refused (X_CHANNEL_DECLARATION_INVALID), and a public channel says so with policy: allow('public'). A catchUp passed as the query itself must decide with the channel's own policy object — the read is served with the channel params as input, so a wider rule answers what the subscribe refuses — or channel() throws X_CHANNEL_CATCHUP_POLICY_MISMATCH at load. The reference app's idiom is examples/dummy/apps/web/app/posts/channel-ref.ts beside channels.ts. The catch-up query's name is written by hand on the ref, like a QueryRef's; x build emitting refs is plan-101 DX ledger #21.

FactRule
a policy subjectrow: ({ params, ctx }) => … loads what the policy decides about, the way an action's row: does
registrationchannel() registers itself in the process's channel table. new ChannelHub({ transport, sockets }) serves every declared channel
subscribeby declared name + params. An undeclared name is X_TOPIC_FORBIDDEN, and so is a policy denial, latched per (socket, topic) until the session changes. Two more are denials before the policy runs: an anonymous socket on a channel that declares row, and a row loader the tenant guard refused
re-authorizationwhen a socket's grant is renewed every seat is decided again. Denied: dropped, and refused to the client under its sid. Undecided (the policy or loader raised): kept but suspended — nothing is delivered until a later pass succeeds, then a records channel is told replay-gap
the first joina join with no cursor on a channel that carries records is answered with one replay-gap, so the client runs one catch-up read. Rows written between a page's render and its join are otherwise never seen. A resubscribe with since replays from the ring, or gets replay-gap when since fell out of it (channel-logs.ts, resume). A since that is already current is answered too — the records frame at that cursor, carrying nothing — so useChannel() leaves joining; an events-only channel leaves it on its first events frame, which is its presence roster
recordsrows of the listed entities travel as records frames, carrying seq and epoch, into the record store; read them back as a list with useRecords(type, { where?, order? })
a missed changea hole in a producer's seq on the change bus, or a bus that reconnected, tells every member of every open records topic replay-gap at a new epoch, beside the live windows' re-snapshot (24.0.0, #681)
records with no replicatorrecordPublisher({ transport, entities }) on @ultimat3/realtime/server puts an app's committed rows on the change bus, sequenced per producer; every node delivers them as records. Channels only — a publisher has no commit position to order a live window by. Under x dev a claimed entity is no longer also carried to channels by the in-process bridge, and a sync node carries each table from ONE producer kind — the first it saw — dropping the other's changes under X_REALTIME_PRODUCER_CONFLICT (25.0.0, #682). Rules: the package README, Channel records with no replicator
useChannel() statesjoining → live on the node's first answer to every channel that carries records or events (an events channel's is its roster, empty on a node without presence); catching-up while a replay-gap re-read runs; offline; failed with the coded refusal — a decision (a policy denial, an undeclared channel, a cap) and nothing else. A node whose bus is away refuses with X_TRANSPORT_UNAVAILABLE, which is its condition and not its verdict: the membership reads joining and is live again on the node's next answer (a bus outage, seen from an open page). The table is in the package README, What useChannel() reports
eventson a channel declared with events: true; otherwise X_CHANNEL_DECLARATION_INVALID. App code — a job, an action — calls publishChannelEvent(decl, params, event) from @ultimat3/realtime/server: it publishes on the process's own bus, in every role, with no hub (27.5.0). The sync node's own code holds the hub and calls hub.publishEvent(decl, params, event). Same frame, same subject. At-most-once, never stored, never replayed: an event says "re-read", never "here is the state". Call it after COMMIT — nothing here sees a transaction
events with no replicatora channel that lists no records reads no committed change, so it needs no replicator, no replication slot and no REPLICATION grant: NATS_URL set for every role, a sync role, and the publishers. A node that serves no live: true query and no records channel boots on a real database with live=none and is never X_REALTIME_TOPOLOGY (27.5.0)
presencea roster arrives as an events frame { presence: op, members, total? }. Read it with readPresence(frame.event) in the channel's events handler. There is no separate presence frame
an events-only islandan island whose only realtime is useChannel / usePresence / useConnection is never hidden and never waits for the page's restore (above)
the browseruseChannel(decl, params, handlers?); topic() is the one way to spell a topic, and bun run channel-literals refuses any other
the manifestlists every channel (describeChannels, on @ultimat3/realtime/server). In a contract diff these are breaking: a channel removed, its params changed, its policy changed, a record type no longer carried, events switched off

A channel event re-reads the page

As of 30.0.0. For a server-rendered page with nothing to push but "this changed": the channel says when, the server's page says what. An event is at-most-once, so a slow safety re-read sits beside it.

// in an island on the page: the event schedules, the router re-reads when the visitor is idle
import { visibleInterval } from '@ultimat3/core/page';
import { type ChannelRef, useChannel } from '@ultimat3/realtime';
import { scheduleRefresh } from '@ultimat3/render/client';
import { onCleanup } from 'solid-js';

declare const ORG_FEED: ChannelRef<'orgId'>; // the ref above, from `./channel-ref`
declare const orgId: string;
useChannel(ORG_FEED, { orgId }, { onEvent: () => scheduleRefresh() });
// the event is at-most-once: a safety re-read every 30 s, hidden tabs skipped
onCleanup(visibleInterval(scheduleRefresh, 30_000));
PieceWhy it is that one
scheduleRefresh(), not refresh()a burst of events is one re-read; nothing swaps under an open dialog, a focused field, a selection or a scroll (Client navigation)
visibleInterval from @ultimat3/core/pagethe poll a hidden tab and a lost session stop; an island that only polls reaches no realtime (Client data)
rows, not a pageuseQuery(ref, input, { live: true }) stays the idiom: this is for a page whose rows no island holds

A page with no router does nothing on a scheduleRefresh(): it never reloads under a reader. Pause toggles, a flash on change and a "live" dot are the app's.

What each role needs from the bus

The bus is the transport adapter port: NATS_URL set builds the NATS transport, unset the in-process one, and transport: in apps/<app>/runtime.ts replaces that decision with any Transport from @ultimat3/realtime/adapter — already connected, and not closed by the boot (Adapters). There is no setChannelTransport and no realtime.transport config key (31.0.0).

With NATS_URL set every role holds a connection, and only two cannot work without one (As of 2026-10). The role decides, not the transport: a role that only publishes sends channel events and cache invalidations, and both mean "re-read".

RoleUses the bus toAt bootNATS down while runningJetStream / the presence buckettransport check
syncserve sockets: subscribe to changes and channels, hold presenceawaits the dial, retried on backoff for 15 s (each attempt has a 4 s connect timeout, so a server that accepts and says nothing still fails attempts inside the wait), then refuses the boot with X_TRANSPORT_UNAVAILABLE naming the server, the wait, the last attempt's failure and the variable to checkkeeps its sockets and reconnects on its own; a page's beat is refused X_TRANSPORT_UNAVAILABLE meanwhile (below); windows re-snapshot and members re-join on returnrequired at the dial — no -js, no bootfailing
replicatorpublish committed changes to the nodesawaits the dial, same 15 s, same refusala publish refused because the bus is away is retried in place — the same envelope, three more attempts over 2 s — so a NATS restart costs no stream restart; past that the run ends and the takeover loop resumes from the last published lsnnot asked forfailing
web, worker, schedulerpublish channel events (publishChannelEvent) and cache invalidations; receive peers' invalidationsdoes not wait: dials in the background on a jittered backoff (500 ms doubling to 30 s) and boots, serves and runs jobs meanwhilestays up and ready; resumes on its own, no restartnot asked for — asserted only if something reads presencedegraded
migratenothingnever dials———

x dev is one process running several roles, and it never goes down for the bus. With sync in the set (the default) it waits 3 s for NATS (DEV_BUS_WAIT_MS), then boots without it: the refusal a ROLE=sync container would have exited on is logged once at warn (ultimate bus is away: x dev serves without live until it answers, with its code, cause and fix), the dial goes on in the background, pages, jobs and tasks run, and the sync node stays mounted and not ready — an upgrade is shed with a 503 and a retry delay, so an open page keeps asking. When NATS answers the node starts (sync node started: the bus answered) and those pages go live with no restart. checks.transport is degraded there, never failing, so a bus lost under a running x dev leaves /readyz at 200. With --role web,worker nothing waits at all. A container is unchanged: ROLE=sync and the replicator keep the 15 s refusal, because they have nothing else to serve. realtime.enabled: false starts neither realtime role, so no process of such an app waits for the bus — the connection is still dialled in the background, because cache invalidation between replicas crosses it.

While the bus is down, on a publishing roleWhat happens
publishChannelEvent(…)rejects at once with X_TRANSPORT_UNAVAILABLE — never parked behind the dial. Catch it and log: the event said "re-read", and a subscriber that reconnects re-reads anyway
a publish made while the client is reconnectingrefused the same way, from the moment the client's socket closes. The nats client would otherwise buffer it without bound and replay it when the server returned, and it has no option to forbid that. One window remains: a connection whose packets are silently dropped — a server that accepts the socket and then says nothing — still reads as connected until two pings go unanswered (10 s apart; measured against a real server through a relay that stops forwarding: detected 30 s after the silence began, where the library's default is six minutes), and a publish made inside it is taken and lost. A KV request has an answer to wait for, so it is never buffered: inside the window it fails after its 5 s timeout, and once the client knows it is disconnected it is refused at once
an action's invalidates / invalidateTagsevery local tier, Redis and the CDN are cleared as usual; the broadcast to the other replicas fails into report.errors (tier: 'broadcast', logged as cache.invalidate.partial) and the write succeeds. The refused tags are kept and published when the bus returns — de-duplicated, at most 1,024 of them; past that the process owes one flush-all instead, so nothing grows with the outage. Until then another replica's in-process lru copy and its tag-revalidated ISR page are stale
when the bus returnsevery process whose connection was down drops its in-process tier and marks its tag-revalidated ISR pages stale (flushProcessTiers): it heard no peer for a window nobody measured, so nothing in its own heap is known fresh. One cold lru per replica per outage — sized for by the database, not hidden. A replica that receives a peer's flush-all does the same. Redis and the CDN are not touched: whoever ran a bust cleared those
/readyz200 in both health.readiness modes, checks.transport: 'degraded'
/readyz?deep=1503 — the reading a monitor alerts on
the logthe boot line {"msg":"ultimate started",…,"bus":"nats(connecting)"} (nats(up) when it already is, in-process with no NATS); {"msg":"ultimate bus","bus":"nats(up)"} when the dial lands, and again after every recovery. Everything else is one outage, said rarely

A bus outage, seen from an open page

A page that is open when NATS goes away keeps its socket: the sync node does not close it, and nothing is reloaded. What the page reads, As of 2026-10:

MomentThe nodeuseChannel()usePresence()useQuery({ live: true })
the bus goes awaykeeps every socket and every seat; cannot read or write presence, receives no change and no eventlive until the page's next beat (≤ 10 s)the last rosterlive, with the rows it has; no patch arrives
each beat while it is away (every 10 s)answers the membership's subscribe with an ack carrying X_TRANSPORT_UNAVAILABLE under its sidjoining — asked, unanswered. Never failed; error() stays undefinedthe last rosterunchanged: the beat re-sends channels only, and a live subscribe never touches the bus
a page opened during the outageseats it if it can; refuses the same way if it cannotjoiningemptylive once its snapshot lands — the read is the database's
the bus returnsre-binds its subscriptions, tells every records topic replay-gap at a new epoch and marks every live window for a re-snapshot (live.bus_reconnected)a records channel: catching-up → live at once, on the gap—re-snapshotted on its next delivery
the next beat after it (≤ 10 s)the member is gone from a restarted server's bucket, so it is joined again under the same socket id; the roster is sentan events channel: live on that rosterthe fresh roster—

Until 2026-10-10 the refusal marked the membership failed and nothing cleared it: events flowed again and the page read failed until a reload, while a new tab was live at once.

RuleDetail
what is transientexactly one code, X_TRANSPORT_UNAVAILABLE (client-channels.ts, isTransientRefusal): the node's bus is away, or the node is draining
what is terminalevery other refusal: X_TOPIC_FORBIDDEN (a policy denial, an undeclared channel), a cap (X_SUBSCRIPTION_LIMIT), an auth failure. failed, with error() the node's own coded refusal
the retrythe beat itself. A membership is re-asked every heartbeat whatever its state — that is what the presence heartbeat already is — so an outage adds no timer and no load, and no backoff is needed: one frame per topic per 10 s is the healthy rate
a state the outage does not touchcatching-up (a read in flight), a failed catch-up read, a terminal failed, offline
a live querynot on this path: a live subscribe reads the database and never the bus, so no node refuses one with X_TRANSPORT_UNAVAILABLE, and the beat re-sends channels only. Its ack handling is unchanged — failed is a decision, and a rate limit is retried after the delay the node names
every tabone socket per origin (the SharedWorker): the engine's own beat is the re-ask, and the refusal and the answer are routed to every tab holding the topic
a records-only channel on a seated socketits beat touches nothing on the bus, so it is not refused and reads live through the outage, with no row arriving; the replay-gap on return repairs it

What an outage logs

One outage is one fact. Each condition below is written on its 1st, 2nd, 4th, 8th, … failure and once more when it ends (outageLog, isOutageMilestone, on @ultimat3/core), at a level chosen by what the process can still do: warn for one that is degraded and serving, error for one that cannot do its job.

LineLevelWhereWritten
sync.bus_unavailable { operation, failures, error }errorsyncfailures 1, 2, 4, 8, … of anything on the node that needed the bus — a page's refused beat (operation: 'sync.frame'), a presence sweep, a presence leave. One count for the node, and the error monitor is sent the same failures and no others
sync.bus_unavailable recovered { after }infosynconce, when the bus answers
nats transport error { transport, failures, code, error }error on sync and the replicator, warn on web, worker, schedulerevery rolefailures 1, 2, 4, 8, … of the connection itself: the drop, each refused dial. The library giving up on a client (…reconnect budget was spent; re-dialling) is always written — once per client
nats transport recovered { transport, after }infoevery roleonce per recovery
cache.invalidate.partial { …report, failures }warna publisherrefusals 1, 2, 4, 8, … per failing tier (broadcast, redis, …). Every bust's report still carries its errors, and /_x still records every bust
cache.invalidate.partial recovered { tier, after }infoa publisheronce, when that tier clears again
cache.broadcast.subscribe-failed { attempt, error } · … recovered { after }warn · infoevery rolesilent for the first 10 s of the boot subscribe (FIRST_DIAL_GRACE_MS: the bus's first dial is still in progress, and the subscribe lands within it on a healthy pod); past that the first refusal and then attempts 2, 4, 8, … warn, and the landing is said once at info — only after a warning. As of 2026-10-17
live.bus_reconnected { desynced, channelGaps }warnsynconce per recovery: how many windows and topics were told to re-read

Until 2026-10-10 a three-minute outage wrote presence.sweep failed at error every 10 s on every sync node (19 lines on one), sent the monitor one event per refused beat per socket, and wrote cache.invalidate.partial once per bust on every publisher. A failure that is not the bus being away — a policy that threw inside a presence leave — is still written every time, as <operation> failed.

LSN cursors

Every frame carries an LSN. The client's last-seen LSN is what makes reconnect a delta instead of a refetch.

PropertyBehavior
Cursorthe highest LSN the client has applied, per subscription. It advances on every patch, not only on a snapshot — a cursor whose at freezes at the last snapshot fails the lag check, and every client connected longer than maxLagMs re-snapshots instead of resuming
Reconnect inside the change buffer windowdelta replay from the sync node's ring buffer — zero DB work. A ring is complete only from its floor (its first retained change, or the read its window was filled by): a cursor older than that, such as one minted on another node, takes the snapshot
Reconnect outside the windowone bounded snapshot query at a current LSN. Never WAL history traversal
Cursor unusable and no snapshot path suppliedX_CURSOR_STALE
OrderingLSN is monotonic per DB, so a client can never apply an older change over a newer one
What a cursor carries{ qid, lsn, ids, at } and nothing else, As of 2026-08-24. digest and count were written by every snapshot, encoded and validated on every subscribe frame, and read by nothing — digest cost a canonical-JSON render plus a hash over EVERY ROW of every snapshot, paid once per live query per reconnecting socket in the restart storm this package is measured on, and count would have been wrong had it ever gained a reader, because advance seeds its set from the already-truncated ids. Both are deleted, and that is what moved the wire version (below)

Inbound frame order

A frame is routed as soon as it arrives, so two frames from one socket can be in flight at once. What is ordered is narrow and deliberate As of 2026-08 — a lane per socket would put every frame behind the slowest one, and the slowest one is a subscribe's snapshot read.

FramesOrdered againstWhy that unit
subscribe on a querythe same sidadd then drop for one sid, or the drop finds nothing and the add strands the subscription it was meant to end
subscribe on a topicthe same topic nameone membership, same add/drop pair
hello, everything elsenothingthey read state and write none

Ordering is not what bounds the caps. N sequential subscribes still pass a check-then-act limit N times, so every refusal a subscribe can answer with — the sid claim, maxPerSocket, maxPerTenant, maxTopicsPerSocket, maxTopicsPerNode — is decided synchronously, before the first await, against a count that already includes the subscribes still in flight. One WebSocket write carrying N subscribe frames used to pass each cap N times.

Staying connected

A dead TCP connection that was never closed fires no close event. Only the client can end one, so it beats.

PropertyBehaviour
Intervalthe node's: its hello reply names the beat (heartbeatMs — a third of its presence ttl, never slower than a quarter of its idle budget, floored at 1 s) and the page's one socket follows it. 10s until it speaks, a third of the default ttl (30 s), so one lost beat is never a false leave. The client option that set it went with LiveClient in 21.0.0
The node names itthe node's hello reply carries heartbeatMs: a third of the presence ttl that node really runs, never slower than a quarter of its idle budget (clientHeartbeatMs). Optional and additive — no protocol bump; a reply without it leaves the default above
One beata hello, plus one subscribe frame per topic held. A beat and an opening frame are byte-identical — hello carries no cursors — so a beat asks for nothing and resumes nothing
Silencenothing received for two intervals ⇒ the socket is closed with code 4000 and the reconnect timer arms
Why re-sending topicson the node, subscribing to a topic is joining its presence set, and repeating the frame is the presence heartbeat
No knobrealtime.heartbeatMs in app.config.ts was read by nothing and is deleted As of 2026-08-19; an app still setting it fails typecheck with TS2353

A reconnect re-announces everything, one frame at a time. hello, then one subscribe per registration carrying that registration's cursor, then one per topic. hello itself carries neither cursors nor topic membership — resume is decided per subscription, by the frame that also names the query and its input, and topic membership is state on the node's socket. Without the topic half a channel stayed silent from the first reconnect onwards while its handler was still installed, and its presence membership was swept.

Writes never ride the socket, As of 21.0.0. The socket mutate path is deleted: useMutation posts the mutator's action over HTTP through clientTransport, with an idempotency key, so a resend after a lost answer is served from the action's idempotency store. Its answer's records are adopted before the overlay goes, so nothing flickers.

The reconnect risk

Reconnect is the expensive part of any sync engine, and it is where naive designs fall over. A deploy or a network blip drops N sockets at once; each client then asks "what changed since LSN X?" If the answer requires replaying arbitrary WAL history or re-running every query, a rolling restart becomes a self-inflicted thundering herd that outlasts the deploy.

#MitigationDetail
1Prototype before locking topologythe reconnect benchmark: 50k sockets, forced sync restart, recovery time and DB load. Measured As of 2026-08 — the numbers are below, and both results are committed
2Bounded per-query change bufferthe replicator keeps a ring buffer of recent changes per query-hash. Reconnect within the window = delta replay from the buffer, zero DB work
3Snapshot fallback, not WAL replayoutside the window the client gets a fresh snapshot at a current LSN. Cost is one bounded query, never history traversal
4Jittered reconnect-with-backoff, server-directeddraining sync nodes send a reconnect frame with a per-client delay so clients redistribute instead of stampeding. the page socket arms one timer per closed socket: the node's assigned delay when there is one, otherwise browserBackoff, which is base 500 ms, factor 2, equal jitter, capped at 4 s (BROWSER_RECONNECT_MAX_MS, thundering-herd.ts). It was the server-side defaultBackoff's 30 s until 21.0.0: after a deploy that left the returning node, and its update-available, unreached for 27 s. equal keeps a floor, so a SIGKILLed node's herd redials across a 2–4 s window rather than at once, and the AcceptBudget sheds the excess before any query runs. useConnection().reconnectAt renders the wait. The committed benchmark runs predate the cap
5Per-tenant subscription capsa registered-query explosion is a load-shedding decision, made with a limit and a typed X_SUBSCRIPTION_LIMIT, not by falling over. Reachable, not yet wired As of 2026-08: the boot passes no caps, and the per-tenant scope needs both maxPerTenant and tenantOf. The per-socket cap applies today at its default of 128
6Consider wrapping an existing protocolif the benchmark says our matcher is the bottleneck, adopting an existing open sync protocol beats inventing one

The forced-restart benchmark

One harness, two runs, two questions. Reachability: how long until a killed node's clients are back and receiving. Delivery: how many patches were lost getting there. A first-delivery timer answers the first and is blind to the second, which is why there are two.

scripts/bench/restart-bench.ts produced both, each with its own transcript beside it in scripts/bench/results/.

# reachability, 50,000 clients — 2026-08-11
bun run scripts/bench/restart-bench.ts --clients 50000 \
  --out scripts/bench/results/50k-restart.json

# delivery, 10,000 clients, a probe every 200ms — 2026-08-17
bun run scripts/bench/restart-bench.ts --clients 10000 --probe-interval-ms 200 \
  --out scripts/bench/results/10k-restart-seq.json
SetupValue
Clientsreal WebSocket connections, split across client-shard OS processes — 50,000 over 10, 10,000 over 8
Serverone sync node (the shipped syncNode) in its own process, over InProcessTransport
Admissionthe shipped AcceptBudget at its defaults — 500/s, burst 2000
KillSIGKILL, no drain, no reconnect frame — recovery is driven only by each client's own backoffDelay
Readinessread from the server's own socket count, never the load generator's self-report
Subscription under testa channel topic. Neither run subscribes to a live query, so no cursor, snapshot or gap-repair path is exercised

Reachability — 50,000 clients

Per client, the first channel patch received on the reconnected socket: reconnect and resubscribe and one delivery. It is not a consistency metric, and cannot be one — see below.

Restart-phase resultValue
Reconnected50,000 / 50,000
Received a channel patch inside the window49,981
Time to first patch, p5054.0s
Time to first patch, p90105.5s
Time to first patch, p99127.8s
Time to first patch, max145.7s
Connect attempts shed before any query path156,851 — the DB-load proxy: none of them reached a query or snapshot
New server accepting2.3s after the kill

The timings are unchanged and still stand. Only the name was wrong: this metric was published as "time-to-consistent" until 2026-08, and it never measured consistency. The harness recorded each client's last-seen sequence number and read it nowhere, so a patch the node dropped was invisible to it by construction. Nothing here is retracted — a number that timed reachability is now called reachability.

Delivery — 10,000 clients

Every client counts observed sequence gaps in the probe stream it received, per connection. An observed gap is a break between two frames one connection actually received — the publisher numbers every probe, so a missing number between two arrivals is a frame that was published to a subscriber and never came.

Restart-phase resultValue
Reconnected10,000 / 10,000
Patches received1,666,882
Observed sequence gaps0 — 0 gap events, 0 missing frames, 0 duplicates, 0 publisher rewinds, 0 malformed
Clients that observed a gap0 / 10,000
Time to first patch, p50 / p90 / max10.9s / 22.3s / 43.5s
Connect attempts shed before any query path33,424

Zero observed gaps is a lower bound on loss, not a proof of zero loss. The counter can only see a hole with a received frame on each side of it, so three losses are invisible to it by construction:

Invisible to the counterWhy
frames lost before a connection's first arrivalthere is no lower anchor to measure the gap from
frames lost after a connection's last arrivalthere is no upper anchor, and the connection may simply have ended
every frame, on a connection that received nothing at allno anchors, so the connection contributes no sequence to check

So the honest claim is "no client observed a lost channel frame", not "no channel frame was lost". Every other statement of this result on the wiki is shorthand for this paragraph.

Not evidence about 50,000. The 50,000-client run predates the counter and carries no delivery number; As of 2026-08 the 10,000-client run is the only one with delivery accounting, and it does not extrapolate.

Why delivery needs its own counter

A dropped records channel frame is now repaired, As of 2026-09-22 (21.0.0). SocketRegistry.deliver still counts every refusal (the series channel_frames_dropped_total, the warn line channel.frames_dropped carrying { topic, dropped, total }, and node.sockets.droppedChannelFrames for a test that cannot scrape). It also marks the (socket, topic) as gapped, and once the socket drains it sends one replay-gap frame, counted in channel_replay_gaps_total. That series counts gaps announced, by design: the repair happens in the browser. The client (client-channels.ts) keeps a cursor per channel. On replay-gap, or a new epoch, it re-runs the channel's catchUp query and applies the frames that arrived meanwhile after it. A dropped events frame is ephemeral by design and is only counted. The live-query path already repairs the same drop, as it has since 2026-08: the subscriber is marked desynced, and the next change re-snapshots it.

What it is not: a multi-node result — neither run crossed NATS, so this is per-node recovery, not fanout. Not a throughput figure either: no requests/sec, no message rate, no sustained-load number. Per-node socket capacity in the tables above this section is still a target derived from Bun's native WebSocket implementation, not a benchmark result. Long-running Bun processes are also less battle-proven than Node's; sustained-socket memory profiling is explicit roadmap work.

sync drain

sync holds no durable state, so a restart is only dangerous in aggregate. Closing 50,000 sockets at once means 50,000 simultaneous reconnects, all resubscribing, all asking "what changed since my LSN?" — a self-inflicted DDoS landing during a deploy, when capacity is already reduced. Worse, it is fractal: surviving nodes overload, drop connections, and the herd re-forms.

So the drain is server-directed:

{ "type": "reconnect", "v": 3, "afterMs": 1830, "reason": "drain" }

ReconnectFrame (packages/realtime/src/sync-protocol.ts) has those four fields and no cursor: v is PROTOCOL_VERSION, reason one of drain, overload, rebalance.

PropertyEffect
Per-client afterMs, jittered over a windowreconnects arrive spread out, not as a spike
The window is syncNode({ drainSpreadMs }), default 30seach socket draws its own delay inside it (drainPlan)
The resume point is the client's own cursoron reopen the client sends hello, then one subscribe per live query carrying that query's cursor, and re-announces each channel from its own; the node replays from its change buffer when the cursor is inside it, one snapshot when not (live-resume.ts)
Clients redistributethe LB places them across remaining nodes; no sticky session to honour
Client-side backoff is a floor, not the mechanisma client that loses the socket without a frame still backs off exponentially with jitter
Ordering in the drain sequence/readyz → 503, stop new subscribes, send reconnect frames, close cleanly, flush spans, exit 0

Full drain sequence per role: Deployment.

Wire protocol version

PROTOCOL_VERSION is 3 As of 2026-09-22 (21.0.0; it was 2 from 2026-08-24) (packages/realtime/src/sync-protocol.ts). A mismatch is X_PROTOCOL_VERSION on the frame, with one instruction, rather than a per-field decode error nobody can act on.

Clients and sync nodes must be redeployed together across this bump. A cursor rides in BOTH directions — the client's subscribe and the node's snapshot — so a skew breaks resume from either side.

SkewWhat happens
new node ← old client (v2)decode refuses the frame: X_PROTOCOL_VERSION. v3 deleted the mutate and rebase kinds (writes go over HTTP) and the presence kind (a roster is an events payload now), added records, events and replay-gap for channels, and narrowed the ack frame to a refusal: it carries the sid of a subscription the node refused, or the socket id for a frame it could not read, and is never a write receipt
old node ← new client (v3)the same, at the other end
v1 ↔ v2 (2026-08-24)the old node's cursor() reads digest through a str() that throws on an absent field, which is precisely why that was a version and not a silent widening

The version guards incompatibility, never novelty. Two earlier realtime changes deliberately did not move it: an additive optional field (snapshot.entity) and a removed field read through list() (hello.resume) are both readable in either direction, because decode builds a whitelist object and list() answers [] for an absent key. cursor() is the other kind of reader — str/num throw — so removing a field it reads is the one shape that needs the number.

Build skew

A client on build A connecting to a sync node on build B is accepted, then sent an update-available frame carrying the node's buildId; the socket is not killed. Skew is the build the client claims against the node's own: ?build= on the dial, and the hello frame's buildId, which the node records on every hello and which is the later word — a dial without ?build= starts as "not skewed" and the first hello settles it. Neither side's build moves while a socket is open, so the answer is a property of that socket for its lifetime — a client learns about a deploy on the socket it opens against the new node, never on one it is already holding. Every hello on that socket re-reports the same answer, the heartbeat's included. The client's AppUpdateAvailable signal flips and the app renders its own update affordance. See PWA and offline.

Device sockets

A long-lived socket for a client that is not a browser — a device agent that dials out and receives work. deviceSocket() is a factory over route: the value is a websocket mount (path, fetch, websocket) plus request(), which reaches one connected device from any process. From @ultimat3/realtime/server, As of 2026-11-06.

import { userActor } from '@ultimat3/core';
import { deviceSocket } from '@ultimat3/realtime/server';

export const deviceLink = deviceSocket({
  path: '/link',
  resolveToken: (token) =>
    token === 'enrolled' ? { actor: userActor({ id: 'u1', orgId: 'o1' }), deviceId: 'desk-7' } : null,
  maxSocketsPerActor: 10,
  heartbeatMs: 15_000,
  onFrame: ({ frame, device }) => ({ ack: device.deviceId, received: frame }),
});

// From any action or job, on any node — scoped to the caller's tenant:
export const ask = (ctx: { readonly actor: ReturnType<typeof userActor> }) =>
  deviceLink.request(ctx, 'desk-7', { job: 'sync' }, { deadlineMs: 30_000 });
ConcernRule
CredentialAuthorization: Bearer <token> on the upgrade, decided by resolveToken before any frame. None: 401 + www-authenticate: Bearer; refused or expired grant: 401 + Bearer error="invalid_token" (X_SOCKET_UNAUTHENTICATED). A grant's expiresAt also closes an open socket (1008)
Slow token serviceresolveToken is cut at DEVICE_AUTH_DEADLINE_MS (5 s, 503 X_SOCKET_AUTH_UNAVAILABLE); past DEVICE_MAX_PENDING_UPGRADES (256) parked upgrades, and past the accept budget, the dial is shed 503 with a jittered retry-after-ms
Origina device sends no Origin. An upgrade that carries one is a browser: 403 X_SOCKET_ORIGIN_REFUSED unless allowedOrigins lists it exactly
CapsmaxSocketsPerActor (default 10) per node, reserved before server.upgrade: 429 X_SOCKET_LIMIT. One socket per device: a newer one, on any node, closes the older (1008 replaced)
FramesJSON text. DEVICE_FRAME_MAX_BYTES (256 KiB) is the framework's and is not an option: Bun's maxPayloadLength and the handler both enforce it (1009 X_DEVICE_FRAME_TOO_LARGE); a request() over it throws before sending. onFrame runs one frame at a time per socket, as the device's actor; past 32 queued the socket closes X_FRAME_RATE_LIMIT
Heartbeatthe node names heartbeatMs in its hello; a device silent for two beats is closed 4001
Addressingthe holder registers the device in the bus's shared set (TTL three beats) under (tenant, deviceId); request() looks it up and publishes to that node. Nobody holds it, or another tenant does: X_DEVICE_OFFLINE at once. No answer in deadlineMs (default 30 s, max 600 s): X_DEVICE_REQUEST_TIMEOUT. A socket that closes answers its open requests X_DEVICE_OFFLINE
DrainSIGTERM: stop accepting, then the sync node's own drainPlan + reconnect frame per socket, then close 1001
ReconnectdeviceAgent() (a Bun client) redials on realtime's one jittered backoff, or at the slot a reconnect frame assigned

Wire. Device → node: { type: 'beat' }, { type: 'frame', frame }, { type: 'response', id, frame }. Node → device: { type: 'hello', heartbeatMs, maxFrameBytes }, { type: 'request', id, frame }, { type: 'frame', frame } (what onFrame returned), { type: 'error', code }, and the sync protocol's reconnect.

Where it runs. Every role attaches every declaration to its bus (hostDeviceSockets), so a worker's job can request(); only web holds sockets. Across replicas every role needs NATS_URL set, or one transport in apps/<app>/runtime.ts — on the in-process bus a device is reachable from its own process only.

Where it is served. The web role's own listener, beside the sync node's /_x/sync: the boot hands httpServer one mount per declared path beside syncMount, every path is checked free of a route (X_ROUTE_CONFLICT), and each upgrade goes to the path's CURRENT declaration, its frames to the declaration that upgraded it. The same in the container (ROLE=web) and under x dev — no port, no option. A declaration is found by the boot's module scan, so it lives in any module under apps/web/app/ (examples/dummy/apps/web/app/devices/link.ts). As of 2026-11-06.

Under x dev. A save re-evaluates the module that declares a socket (and anything it imports), so its deviceSocket() runs again. Each rescan is a new generation (dev-rebuild.ts): the path's new value replaces the old one, is attached, and serves the next upgrade; the old one's open sockets get a reconnect frame and a 1001 close, and deviceAgent() redials onto the edited handler. A different module declaring a path an earlier scan holds is X_CONFIG_INVALID, naming both files — the holder was not re-evaluated; moving a declaration is x stop && x dev. A path first declared by a save is not served until a restart (X_DEV_RESTART_REQUIRED on /_x): the listener's websocket paths are fixed when it binds. A container scans once, so never replaces.

RuleError
an app module calls server.upgrade(, hands Bun.serve a websocket key, or writes an inline websocket: { … } handler setX_SOCKET_HAND_ROLLED, x verify's boundaries step, with the file and meta.line — declare deviceSocket({ path, resolveToken, onFrame }) instead
two declarations at one path in one scanX_CONFIG_INVALID at the second deviceSocket()
a declaration at a path a route already answersX_ROUTE_CONFLICT at boot

Errors

CodeCauseFix
X_TOPIC_FORBIDDENan undeclared channel name, or the declaration's policy denied the actorx policy list --json, then widen the policy on the channel() declaration, or subscribe as an actor it allows
X_SUBSCRIPTION_LIMITa socket, tenant or node reached a cap; the error names which scope refused, and which knobraise maxPerSocket / maxPerTenant / maxEntries on the LiveQueryRegistry (per socket, default 128), or unsubscribe unused live queries — a channel topic cap answers maxTopicsPerSocket (64) / maxTopicsPerNode (10,000) on the ChannelHub. All constructor options, none an app.config.ts field
X_PROTOCOL_VERSIONclient and server disagree on the wire format, or a malformed frame. The wire is at 3 As of 2026-09-22 (above) — a v2 client and a v3 node cannot talk in either direction, so the two are redeployed togetherx build && redeploy the client; the sync node sends 'update-available' before it drains
X_LIVE_QUERY_UNKNOWNa subscribe frame named a live query this node does not havex queries list --json
X_CURSOR_STALEresume cursor cannot be honoured and no snapshot path was suppliedpass 'snapshot' to resumeFrom() so the fallback path can re-snapshot instead of failing
X_REBASE_CONFLICTa custom(merge) returned something other than a row with a string idset conflict: 'server-wins' on the mutator, or return a row from custom(merge)
X_REALTIME_UNINSTALLEDa hook ran in a browser island whose bundle never called installRealtime()x build, which installs it on every island whose own graph imports realtime. An island that reaches realtime only through a package calls installRealtime({ signal: createSignal }) in mount
X_SYNC_UNCONFIGUREDa live hook needed the page socket, and neither installRealtime({ sync }) nor <meta name="ultimate-sync"> named a target. The meta is not rendered at all when app.config.ts says realtime: { enabled: false }: no node is started, so there is nothing to dial (22.2.2)serve the page through x dev or the container, which render the meta, or pass installRealtime({ signal, sync: { url, buildId } })
X_RECORD_REJECTEDa row reached the record store with no key, or not as an objectx entities show <entity> --json, then return whole rows
X_MUTATOR_CLOCK_MISSINGconflict: 'last-write-wins' with no number updatedAt on the entityadd the clock, or declare conflict: 'server-wins'
X_TRANSPORT_UNAVAILABLEthe fanout bus is down. At boot only for sync and the replicator, after a bounded wait; on web, worker and scheduler it is what a publish rejects with while the bus is away (what each role needs); on a socket it is the one refusal a client treats as transient (a bus outage, seen from an open page)x doctor — then check NATS_URL points at a reachable nats-server
X_TRANSPORT_PROTOCOLthe bus answers in a protocol this build does not speakthe bus must be nats-server >= 2.11 with JetStream enabled (nats-server -js)
X_DEVICE_OFFLINEdeviceLink.request() named a device no node holds a socket for under the caller's tenantx doctor --json, then retry once the device redials
X_DEVICE_REQUEST_TIMEOUTthe device took the request and sent no response with its id before deadlineMsa longer deadlineMs, or answer every request the node sends the device
X_DEVICE_FRAME_TOO_LARGEa device frame, or a request() frame, past DEVICE_FRAME_MAX_BYTESsplit the payload, or store the bytes and send their key
X_BUILD_SKEWclient build's contract is incompatible with the server'sreload the client; see the fix line on the error

Verbatim shapes: packages/realtime/src/errors.ts. Full index: Error codes.

Rules

  • Truth is always the server. A client is never the merge authority.
  • One authz system — the query's policy, re-evaluated per delivered row.
  • sync is stateless. Any state that must survive a restart lives in Postgres or NATS.
  • A live query must be deterministic and bounded (orderBy + limit), or x verify rejects it.
  • Presence and typing indicators are tier 1 forever — never model ephemeral state as rows.
  • local does no I/O, no Date.now(), no Math.random(). It must be replayable.
  • Offline writes go through the tier-3 mutator queue, never Background Sync guesswork.
  • Climb the ladder for a property you need, not for a tier you like. Tier 3 costs a durable client store forever.

Edit this page on GitHubThis page as markdown