# Tutorial 4 — jobs, tasks and live queries Three primitives, three rules. A `job` is durable work with memoized steps. A `task` is a cron trigger that only ever enqueues. A `query` with `live: true` is a bounded, totally-ordered read that patches per subscriber. `As of 2026-11-03`. Every command, generated file and output on this page was re-run in the app tutorials 1–3 leave, built from this tree's packages (`scripts/scaffold-smoke-overrides.ts`), carrying the [tutorial 2](/tutorial-02-first-feature) `todo` slice. The [capacity figures](#capacity-measured) are the committed benchmark results, quoted, not re-run. Series: [1 — first app](/tutorial-01-first-app) · [2 — first feature](/tutorial-02-first-feature) · [3 — auth and admin](/tutorial-03-auth-and-admin) · **4** · [5 — deploy free](/tutorial-05-deploy-free) · [6 — growing up](/tutorial-06-growing-up) ## A durable job `x g resource` already wrote one. Its shape is the whole contract: ```ts export const reindexTodo = job({ input: t.object({ id: t.uuid, orgId: t.uuid }), tenant: (input) => input.orgId, // the org the body reads as, from its own input idempotencyKey: ({ id }) => `reindex-todo:${id}`, retry: { attempts: 5, backoff: 'exponential' }, async run({ input, step }) { const row = await step.run('load', () => repo.byId(input.id)); if (row === null) return { skipped: true }; await step.run('process', async () => { await repo.list(1); }); return { skipped: false }; }, }); ``` | Field | Rule | |---|---| | `step.run('', fn)` | each step's result is stored under its **name**. A retry replays completed steps from storage and re-executes only from the failure. Step names are stable identifiers, not labels | | `idempotencyKey` | an at-least-once caller may enqueue twice; the work happens once | | `retry` | per-attempt backoff, independent per step | | `.enqueue(input)` | the **only** queue path. A job is never run inline | The generated `reindex-todo.job.test.ts` pins the distant invariant rather than the happy path: ```ts const first = await reindexTodo.enqueue({ id }); expect(first.deduped).toBe(false); const again = await reindexTodo.enqueue({ id }); expect(again.deduped).toBe(true); ``` Its `.job.test.ts` suffix is what runs it in the gate's `job` step rather than `unit` — see [tutorial 2](/tutorial-02-first-feature#name-the-tests-after-their-step). Enqueue from an action and the outbox makes it transactional: the row and the job commit together, or neither does. `ctx.jobs` is the facade `.enqueue()` resolves through. Full model: [Jobs and workflows](/jobs-and-workflows). ## A scheduled task ```bash bunx x g task daily-digest --feature todo ``` ```text + apps/web/app/todo/tasks/daily-digest.ts + apps/web/app/todo/tasks/daily-digest.job.test.ts + apps/web/app/todo/tasks/daily-digest.test.ts + apps/web/app/todo/jobs/daily-digest-job.ts + apps/web/app/todo/jobs/daily-digest-job.job.test.ts + apps/web/app/todo/jobs/daily-digest-job.test.ts + apps/web/api/index.ts + x.manifest.json + openapi.json ✓ wrote 9 file(s) for task daily-digest ``` `--feature todo` puts the task in the existing slice; without it the generator opens a `daily-digest` slice of its own. Either way the task and its job are added to `apps/web/api/index.ts`, which is what names them. ```ts export const dailyDigest = task({ cron: '0 3 * * *', tz: 'UTC', // The org rides in the payload: the job declares its tenant from its own input, and a task has // no request behind it whose org could be read instead. enqueue: () => [[dailyDigestJob, { id: '…', orgId: '…' }]], }); ``` | Rule | Why | |---|---| | a task **only** enqueues | the work is durable and retryable; a cron that did the work itself loses it on a restart | | `tz` is required, IANA | `0 3 * * *` in `America/New_York` is a different instant before and after a DST boundary. There is no ambient default anywhere in the framework | | one leader | leadership is an expiring lease row in `x_scheduler_leader`, never an advisory lock — that grant belongs to the **session**, not to the process: it outlives every transaction on the connection and is released only by an explicit unlock, the pool's reset on release, or the connection dying — so a node can neither renew it nor prove it still holds one. A second `scheduler` is a warm standby, not a duplicate | ### Registration names it Export names become primitive names, and only `defineApi` stamps them. A task no `defineApi` lists keeps a positional name (`anonymous-task-1`, enqueuing `anonymous-job-2`); `x verify`'s `manifest` step refuses that job as `X_JOB_UNREGISTERED`. With the generator's edit to `apps/web/api/index.ts` in place: ```bash bunx x tasks list ``` ```text name cron tz catchUp jobs last next dailyDigest 0 3 * * * UTC skip dailyDigestJob - 2026-11-04T03:00:00Z ✓ 1 task(s) ``` ```bash bunx x tasks show dailyDigest --count 3 ``` ```text kind: task name: dailyDigest cron: 0 3 * * * tz: UTC catchUp: skip maxCatchUp: 10 jobs: ["dailyDigestJob"] last: - at 03:00 every day 2026-11-04T03:00:00Z 2026-11-05T03:00:00Z 2026-11-06T03:00:00Z ✓ dailyDigest — 0 3 * * * (UTC), next … ``` Every instant renders in the task's **own** zone, never a machine-local default. Full model: [Scheduled tasks](/scheduled-tasks). ## Inspecting the queue ```bash bunx x jobs list ``` ```text 0 job(s) ✓ 0 ready · 0 running · 0 delayed · 0 dead across 0 queue(s) ``` A dead job is never filtered out of view. | Subcommand | Does | |---|---| | `list` | queue depth, matching rows, and the dead-letter list | | `show ` | state, attempt, every step's result, remaining retry delays | | `retry --from-step ` | drops that step so it re-executes; everything before it replays from storage | | `drain --to ` | **planned** — no target ships. Would move `ready`/`delayed`/`suspended` jobs to another **durable** driver; enqueues on the target **before** acking the source. `--to memory` is refused by name (`X_CLI_BAD_FLAG`), `As of 2026-09`: it acked durable rows into a `Map` that died with the command | There is no Redis or NATS **job** driver, `As of 2026-10` — 25.0.0 deleted both all-throw stubs. So `drain` has no target: it exits `X_NOT_IMPLEMENTED` before the queue boots, having moved nothing. Postgres is the shipped driver, and it is the one `x dev` boots. ## A live query The columns this read has an order for, then the read: ```ts export const todoListSorts = ['createdAt', 'title'] as const; ``` ```ts export const todoList = query({ input: t.object({ limit: t.number.default(50), // A closed set, so a sort is a column this read has an order for, never a caller's string. sort: t.enum(todoListSorts).default('createdAt'), dir: t.enum(['asc', 'desc']).default('desc'), }), policy: canTodoRead, live: true, // Opt-in, unlike an action's tool: a read hands rows to an agent, so silence exposes nothing. mcp: { expose: true, description: 'todoList — edit this description' }, sql: ({ limit, sort, dir }) => // `feature.table`, not the kebab plural: `from()` quotes the identifier into the SQL text, // and the entity created the table as snake_case. from('todos', () => repo.list(limit, sort, dir)) // `repo.list`'s own order: the thunk is what the in-memory driver answers and this chain is // the SQL Postgres runs, so a sort that differs here is a page holding different rows. .orderBy(sort, dir) // The primary key last, in the direction the handle adds it, makes the order TOTAL. .orderBy('id', dir) .limit(limit), }); ``` `live: true` is not free. The gate rejects it without all four: | Requirement | Why | |---|---| | `orderBy` on a **total** order | the matcher decides *enters / leaves / moves within* the result from the changed row alone | | `limit` | an unbounded result has no bounded change buffer and no bounded reconnect snapshot | | no `now()` / `random()` | the same `(input, row)` must always yield the same membership answer | | no cross-tenant predicate | tenancy comes from `ctx`, not from `input` — otherwise `X_FORBIDDEN` at subscribe | The generated `todo-list.live.test.ts` asserts on the **SQL text**, because that is the contract an agent reads to self-correct: ```ts const source = await sourceFor( target, { limit: 50 }, { actor: null, unenforced: 'a scaffolded test asserts the SQL text; the policy is asserted separately', }, ); const text = source.toSQL().sql.toLowerCase(); expect(text).toContain('order by "createdat" desc'); expect(text).toContain('limit'); // "ordered" is not enough — dropping the id tiebreak leaves an ORDER BY while going // non-deterministic under ties. expect(text.slice(text.lastIndexOf('order by'))).toContain('id'); ``` ### Policy runs per row, per subscriber Not a subscribe-time gate that then trusts the stream. The same `policy` object is re-evaluated for each delivered patch — which is why policy predicates are **synchronous**: an `await` there would be a database round trip per row per connected client. `toolFrom(todoList)` from `@ultimat3/mcp` is a read tool (`annotations.readOnlyHint: true`) that reads through `todoList`'s own policy. A read hands rows to an agent, so MCP exposure on a query is opt-in — silence exposes nothing. Five projections from the one declaration: HTTP `GET /_x/query/todo-list?limit=50` (`x routes` lists it as `query todoList · todo:read`), a typed client hook, the live subscription, a cache entry keyed by tenant and policy scope, and the MCP read tool. Full table: [Queries and live queries](/queries-and-live-queries). ## Which role runs what ```bash bunx x dev --once --port 3100 ``` ```text roles web, sync, worker, scheduler ``` The same run logs `sync reachable` with the node on `ws://localhost:3101/_x/sync` and the app's own `http://localhost:3100/_x/sync` in front of it. | Role | Runs | Scales on | |---|---|---| | `web` | HTTP: pages, actions, queries | RPS | | `sync` | live-query websockets, on `$PORT + 1` | concurrent connections — no sticky sessions | | `worker` | the job queue | queue demand: jobs held plus jobs due | | `scheduler` | cron dispatch, enqueue only | fixed 1, lease-row leader (`x_scheduler_leader`) | | `replicator` | logical replication → change feed | 1 per database; opt in with `--role`, needs a real Postgres with `wal_level=logical` | `x dev` co-locates the first four in one process. Isolation is simulated, not skipped: separate ALS contexts, a real Postgres queue, a real SIGTERM drain. ## Debugging, without grep | Question | Panel at `/_x` | CLI | |---|---|---| | which step failed, what is queued | `jobs` | `x jobs show ` | | why did this subscriber not get the row | `live` | — | | which clause decided, for which actor | `policy` | `x policy explain ` | | which tags would this write bust | `cache` | — | | where did this request spend its milliseconds | `timeline` | — | `/_x/?json=1` returns exactly what the tab draws. `live` lists the registered live queries and states plainly that no subscriber list is attached — `@ultimat3/realtime` does not retain a subscriber's matcher trace. A panel whose source is unwired says which half is missing rather than rendering an empty tab as an answer. ## Capacity, measured **Reachability.** 50,000 real WebSocket clients against a **single** `sync` node over the in-process transport, `SIGKILL`ed with no drain: all 50,000 reconnected, 49,981 received a channel patch inside the window, first patch on the reconnected socket at p50 **54.0s** / p90 **105.5s** / max **145.7s**, and 156,851 connect attempts shed by the accept budget before any query path. Committed at [`scripts/bench/results/50k-restart.json`](https://github.com/developerz-ai/ultimate/blob/main/scripts/bench/results/50k-restart.json). **Delivery.** A second run, 10,000 clients and a probe every 200ms, counting gaps in each client's received sequence: **1,666,882 channel patches received, 0 observed sequence gaps** ([`10k-restart-seq.json`](https://github.com/developerz-ai/ultimate/blob/main/scripts/bench/results/10k-restart-seq.json)). A first-patch timer cannot see a lost patch, which is why the second run exists — `As of 2026-08` it is the only one with delivery accounting. It says nothing about 50,000, and its zero is a lower bound: a gap needs a received frame on each side of it, so "no client observed a loss" is the claim, not "nothing was lost" ([Realtime](/realtime)). Both are per-node recovery — neither run crossed NATS, so neither is a multi-node result nor a throughput figure. Detail and limits: [Realtime](/realtime). Realtime tier 3 (local-first) shipped in 21.0.0, and it is declared on the **entity**, not the query: `entity(name, { persist: true })`, default `false`, keeps its records in IndexedDB beside the one outbox ([Realtime](/realtime#tier-3-shipped-in-2100)). `persist` is not a `query()` field — writing it there is a `TS2353` excess property. ## Next [Tutorial 5 — deploy on a free tier](/tutorial-05-deploy-free): the scaffolded Dockerfile, `$PORT`, `0.0.0.0`, the health probes, and release-phase migrations — plus what a sleeping free instance does to the `worker` and `scheduler` roles. Related: [Jobs and workflows](/jobs-and-workflows) · [Scheduled tasks](/scheduled-tasks) · [Queries and live queries](/queries-and-live-queries) · [Realtime](/realtime)