Ultimate

Tutorial 4 — jobs, tasks and live queries

Three primitives, three rules. A job is durable work with memoized steps. A task is a cron trigger that only ever enqueues. A query with live: true is a bounded, totally-ordered read that patches per subscriber.

As of 2026-11-03. Every command, generated file and output on this page was re-run in the app tutorials 1–3 leave, built from this tree's packages (scripts/scaffold-smoke-overrides.ts), carrying the tutorial 2 todo slice. The capacity figures are the committed benchmark results, quoted, not re-run.

Series: 1 — first app · 2 — first feature · 3 — auth and admin · 4 · 5 — deploy free · 6 — growing up

A durable job

x g resource already wrote one. Its shape is the whole contract:

export const reindexTodo = job({
  input: t.object({ id: t.uuid, orgId: t.uuid }),
  tenant: (input) => input.orgId,              // the org the body reads as, from its own input
  idempotencyKey: ({ id }) => `reindex-todo:${id}`,
  retry: { attempts: 5, backoff: 'exponential' },
  async run({ input, step }) {
    const row = await step.run('load', () => repo.byId(input.id));
    if (row === null) return { skipped: true };
    await step.run('process', async () => {
      await repo.list(1);
    });
    return { skipped: false };
  },
});
FieldRule
step.run('<name>', fn)each step's result is stored under its name. A retry replays completed steps from storage and re-executes only from the failure. Step names are stable identifiers, not labels
idempotencyKeyan at-least-once caller may enqueue twice; the work happens once
retryper-attempt backoff, independent per step
.enqueue(input)the only queue path. A job is never run inline

The generated reindex-todo.job.test.ts pins the distant invariant rather than the happy path:

const first = await reindexTodo.enqueue({ id });
expect(first.deduped).toBe(false);
const again = await reindexTodo.enqueue({ id });
expect(again.deduped).toBe(true);

Its .job.test.ts suffix is what runs it in the gate's job step rather than unit — see tutorial 2.

Enqueue from an action and the outbox makes it transactional: the row and the job commit together, or neither does. ctx.jobs is the facade <job>.enqueue() resolves through. Full model: Jobs and workflows.

A scheduled task

bunx x g task daily-digest --feature todo
  + apps/web/app/todo/tasks/daily-digest.ts
  + apps/web/app/todo/tasks/daily-digest.job.test.ts
  + apps/web/app/todo/tasks/daily-digest.test.ts
  + apps/web/app/todo/jobs/daily-digest-job.ts
  + apps/web/app/todo/jobs/daily-digest-job.job.test.ts
  + apps/web/app/todo/jobs/daily-digest-job.test.ts
  + apps/web/api/index.ts
  + x.manifest.json
  + openapi.json
✓ wrote 9 file(s) for task daily-digest

--feature todo puts the task in the existing slice; without it the generator opens a daily-digest slice of its own. Either way the task and its job are added to apps/web/api/index.ts, which is what names them.

export const dailyDigest = task({
  cron: '0 3 * * *',
  tz: 'UTC',
  // The org rides in the payload: the job declares its tenant from its own input, and a task has
  // no request behind it whose org could be read instead.
  enqueue: () => [[dailyDigestJob, { id: '…', orgId: '…' }]],
});
RuleWhy
a task only enqueuesthe work is durable and retryable; a cron that did the work itself loses it on a restart
tz is required, IANA0 3 * * * in America/New_York is a different instant before and after a DST boundary. There is no ambient default anywhere in the framework
one leaderleadership is an expiring lease row in x_scheduler_leader, never an advisory lock — that grant belongs to the session, not to the process: it outlives every transaction on the connection and is released only by an explicit unlock, the pool's reset on release, or the connection dying — so a node can neither renew it nor prove it still holds one. A second scheduler is a warm standby, not a duplicate

Registration names it

Export names become primitive names, and only defineApi stamps them. A task no defineApi lists keeps a positional name (anonymous-task-1, enqueuing anonymous-job-2); x verify's manifest step refuses that job as X_JOB_UNREGISTERED. With the generator's edit to apps/web/api/index.ts in place:

bunx x tasks list
  name         cron       tz   catchUp  jobs            last  next
  dailyDigest  0 3 * * *  UTC  skip     dailyDigestJob  -     2026-11-04T03:00:00Z
✓ 1 task(s)
bunx x tasks show dailyDigest --count 3
  kind: task
  name: dailyDigest
  cron: 0 3 * * *
  tz: UTC
  catchUp: skip
  maxCatchUp: 10
  jobs: ["dailyDigestJob"]
  last: -
  at 03:00 every day
    2026-11-04T03:00:00Z
    2026-11-05T03:00:00Z
    2026-11-06T03:00:00Z
✓ dailyDigest — 0 3 * * * (UTC), next …

Every instant renders in the task's own zone, never a machine-local default. Full model: Scheduled tasks.

Inspecting the queue

bunx x jobs list
  0 job(s)
✓ 0 ready · 0 running · 0 delayed · 0 dead across 0 queue(s)

A dead job is never filtered out of view.

SubcommandDoes
listqueue depth, matching rows, and the dead-letter list
show <id>state, attempt, every step's result, remaining retry delays
retry <id> --from-step <name>drops that step so it re-executes; everything before it replays from storage
drain --to <driver>planned — no target ships. Would move ready/delayed/suspended jobs to another durable driver; enqueues on the target before acking the source. --to memory is refused by name (X_CLI_BAD_FLAG), As of 2026-09: it acked durable rows into a Map that died with the command

There is no Redis or NATS job driver, As of 2026-10 — 25.0.0 deleted both all-throw stubs. So drain has no target: it exits X_NOT_IMPLEMENTED before the queue boots, having moved nothing. Postgres is the shipped driver, and it is the one x dev boots.

A live query

The columns this read has an order for, then the read:

export const todoListSorts = ['createdAt', 'title'] as const;
export const todoList = query({
  input: t.object({
    limit: t.number.default(50),
    // A closed set, so a sort is a column this read has an order for, never a caller's string.
    sort: t.enum(todoListSorts).default('createdAt'),
    dir: t.enum(['asc', 'desc']).default('desc'),
  }),
  policy: canTodoRead,
  live: true,
  // Opt-in, unlike an action's tool: a read hands rows to an agent, so silence exposes nothing.
  mcp: { expose: true, description: 'todoList — edit this description' },
  sql: ({ limit, sort, dir }) =>
    // `feature.table`, not the kebab plural: `from()` quotes the identifier into the SQL text,
    // and the entity created the table as snake_case.
    from<Todo>('todos', () => repo.list(limit, sort, dir))
      // `repo.list`'s own order: the thunk is what the in-memory driver answers and this chain is
      // the SQL Postgres runs, so a sort that differs here is a page holding different rows.
      .orderBy(sort, dir)
      // The primary key last, in the direction the handle adds it, makes the order TOTAL.
      .orderBy('id', dir)
      .limit(limit),
});

live: true is not free. The gate rejects it without all four:

RequirementWhy
orderBy on a total orderthe matcher decides enters / leaves / moves within the result from the changed row alone
limitan unbounded result has no bounded change buffer and no bounded reconnect snapshot
no now() / random()the same (input, row) must always yield the same membership answer
no cross-tenant predicatetenancy comes from ctx, not from input — otherwise X_FORBIDDEN at subscribe

The generated todo-list.live.test.ts asserts on the SQL text, because that is the contract an agent reads to self-correct:

const source = await sourceFor(
  target,
  { limit: 50 },
  {
    actor: null,
    unenforced: 'a scaffolded test asserts the SQL text; the policy is asserted separately',
  },
);
const text = source.toSQL().sql.toLowerCase();
expect(text).toContain('order by "createdat" desc');
expect(text).toContain('limit');
// "ordered" is not enough — dropping the id tiebreak leaves an ORDER BY while going
// non-deterministic under ties.
expect(text.slice(text.lastIndexOf('order by'))).toContain('id');

Policy runs per row, per subscriber

Not a subscribe-time gate that then trusts the stream. The same policy object is re-evaluated for each delivered patch — which is why policy predicates are synchronous: an await there would be a database round trip per row per connected client.

toolFrom(todoList) from @ultimat3/mcp is a read tool (annotations.readOnlyHint: true) that reads through todoList's own policy. A read hands rows to an agent, so MCP exposure on a query is opt-in — silence exposes nothing.

Five projections from the one declaration: HTTP GET /_x/query/todo-list?limit=50 (x routes lists it as query todoList · todo:read), a typed client hook, the live subscription, a cache entry keyed by tenant and policy scope, and the MCP read tool. Full table: Queries and live queries.

Which role runs what

bunx x dev --once --port 3100
  roles web, sync, worker, scheduler

The same run logs sync reachable with the node on ws://localhost:3101/_x/sync and the app's own http://localhost:3100/_x/sync in front of it.

RoleRunsScales on
webHTTP: pages, actions, queriesRPS
synclive-query websockets, on $PORT + 1concurrent connections — no sticky sessions
workerthe job queuequeue demand: jobs held plus jobs due
schedulercron dispatch, enqueue onlyfixed 1, lease-row leader (x_scheduler_leader)
replicatorlogical replication → change feed1 per database; opt in with --role, needs a real Postgres with wal_level=logical

x dev co-locates the first four in one process. Isolation is simulated, not skipped: separate ALS contexts, a real Postgres queue, a real SIGTERM drain.

Debugging, without grep

QuestionPanel at /_xCLI
which step failed, what is queuedjobsx jobs show <id>
why did this subscriber not get the rowlive—
which clause decided, for which actorpolicyx policy explain <subject>
which tags would this write bustcache—
where did this request spend its millisecondstimeline—

/_x/<panel>?json=1 returns exactly what the tab draws. live lists the registered live queries and states plainly that no subscriber list is attached — @ultimat3/realtime does not retain a subscriber's matcher trace. A panel whose source is unwired says which half is missing rather than rendering an empty tab as an answer.

Capacity, measured

Reachability. 50,000 real WebSocket clients against a single sync node over the in-process transport, SIGKILLed with no drain: all 50,000 reconnected, 49,981 received a channel patch inside the window, first patch on the reconnected socket at p50 54.0s / p90 105.5s / max 145.7s, and 156,851 connect attempts shed by the accept budget before any query path. Committed at scripts/bench/results/50k-restart.json.

Delivery. A second run, 10,000 clients and a probe every 200ms, counting gaps in each client's received sequence: 1,666,882 channel patches received, 0 observed sequence gaps (10k-restart-seq.json). A first-patch timer cannot see a lost patch, which is why the second run exists — As of 2026-08 it is the only one with delivery accounting. It says nothing about 50,000, and its zero is a lower bound: a gap needs a received frame on each side of it, so "no client observed a loss" is the claim, not "nothing was lost" (Realtime).

Both are per-node recovery — neither run crossed NATS, so neither is a multi-node result nor a throughput figure. Detail and limits: Realtime.

Realtime tier 3 (local-first) shipped in 21.0.0, and it is declared on the entity, not the query: entity(name, { persist: true }), default false, keeps its records in IndexedDB beside the one outbox (Realtime). persist is not a query() field — writing it there is a TS2353 excess property.

Next

Tutorial 5 — deploy on a free tier: the scaffolded Dockerfile, $PORT, 0.0.0.0, the health probes, and release-phase migrations — plus what a sleeping free instance does to the worker and scheduler roles.

Related: Jobs and workflows · Scheduled tasks · Queries and live queries · Realtime

Edit this page on GitHubThis page as markdown