Deployment

One image, N roles. Build once; the ROLE env var selects behavior. No role-specific Dockerfile, no per-role dependency set, no drift between what you tested and what runs.

As of 2026-08. Stable API — semver from here (Upgrading). All three artifact targets ship — x build --target docker, x build --target binary, x build --target static — plus x build --target prebuilt, the step the app image runs on itself (As of 2026-10), and so do the compose files and the Helm chart. Milestone 11 is 🚧 on one thing (roadmap): the two-platform proof — the demo app on Compose and on K8s from one image, with a rolling restart invisible to connected clients.

docker build -t myapp .          # once
ROLE=web       myapp             # ← same image
ROLE=sync      myapp
ROLE=worker    myapp
ROLE=scheduler myapp
ROLE=migrate   myapp             # pre-deploy hook
ROLE=replicator myapp

Roles

• No role holds durable state. Everything survivable is in Postgres, NATS, or object storage.
• Only migrate exits non-zero with a typed error when its lock is held (X_MIGRATE_CONCURRENT), and x dev --role replicator refuses (X_REPLICATOR_SLOT_HELD). A scheduler or replicator container that does not hold its lease or lock stays up as a standby — unready, retrying — rather than running degraded or double-delivering.
• There is no ROLE=all. Those six names are the whole set; anything else is X_ROLE_UNKNOWN at boot. For dev, x dev co-locates web, sync, worker and scheduler in one process — role isolation is simulated, not skipped, and --role opts the replicator in.

PORT selects the bind port, default 3000. Empty or whitespace falls back to the default; anything else must be an integer in 0–65535 or the process refuses with X_PORT_INVALID rather than quietly binding 3000 and failing the platform's health probe with nothing in the log that names the cause. The production entry is the scaffolded apps/web/server.ts → CLI reference.

One repository, one image

Rule: one repository is one app and one image. As of 2026-11-09. Every apps/* workspace (web, admin, api, …) boots into the one registry the root app.config.ts configures, and one image serves them all; ROLE chooses the process, never the app.

x verify --changed gates only what a diff reaches: a docs-only or test-only diff runs a handful of steps; drift runs only when an entity's import closure, packages/db/**, a generated client or openapi.json moved; each suite runs only the tests the diff's import closure reaches (CLI reference).

Health endpoints

Every serving role answers both. web and sync on their HTTP server; worker, scheduler and replicator construct none and answer them on the metrics listener (METRICS_PORT, default 9090, packages/cli/src/metrics-endpoint.ts) with the verdict only — { state, ready, role }, never the check names. There /readyz is 503 until the role has started and from the first instant of a drain, which is what the chart's readiness probe on those roles reads. Both endpoints return a body, never a bare 200 OK.

The body is { state, ready, role } for everyone, and { state, ready, uptimeMs, inflight, buildId, checks, registered, role } — checks a map of name → ok / failing — for a peer configureHttp({ healthDetailPeers }) lists (default ['loopback']: kubectl exec, a port-forward, the container's own healthcheck). The status code is the same for both, and is all a probe reads. A published container port is NOT loopback: the caller arrives from the bridge gateway, so list 'private' to read the detail through -p. The sync role's own port answers by the same list, and tells a request carrying Forwarded or X-Forwarded-For the verdict only.

health: { readiness: 'process' } in app.config.ts (As of 22.5; default 'dependencies'). Every replica shares the database, so one blip fails database everywhere at once and the ingress answers "no available server" for the whole site — pages that never touch the database included. In 'process' mode /readyz is 503 only while starting, draining or stopped; a failing check stays 200 with ready: true and the check still named failing in checks. /readyz?deep=1 always answers in 'dependencies' mode — point monitoring there. Web and sync roles alike; liveness is unchanged in both modes.

A check has three answers: ok, degraded, failing (As of 2026-10). degraded is a dependency the role serves without — today, the bus on a role that only publishes to it. It never turns /readyz into a 503, in either mode, so a NATS outage cannot pull every web pod from the ingress; /readyz?deep=1 is strict and answers 503 for it. An app registers its own with registerReadinessCheck(name, check, { onFailure: 'degraded' }).

Which build answers is a header, on every health answer (As of 27.2.3). /healthz, /readyz and /readyz?deep=1 send x-ultimate-build — the same header every page sends — on the web port, the sync port and the metrics listener alike, whoever asks. The body does not change: a stranger still reads { state, ready, role }. A deploy check that waits for the new build runs x deploy verify --url https://<host> (As of 30.0.0): it samples this header until the build holds, then grades the queues. Two cases send none: a web role whose buildId is null (no BUILD_ID and none from the boot), and the metrics listener in the moments before its boot has computed the id — a container opens it first, so a cold pod's probe gets the status without the header.

registered: 0 is a real answer, not an error. An empty registry is still ready, so a 200 means no more than "the socket is bound" — which is what the chart's and compose's healthchecks route traffic on. Read registered before you trust checks: {}.

Graceful drain on SIGTERM

Identical in every role. Framework behavior, not a deployment guide.

SIGTERM
  1. /readyz → 503                    (web, sync: the listener stays open for drain.readinessGraceMs —
                                       5 s outside development/test — so the LB stops routing here
                                       first; every other role: no grace)
  2. stop accepting new work          (HTTP: close listener; worker: stop claiming; sync: refuse new upgrades)
  3. finish in-flight work            (bounded by drain.deadlineMs in app.config.ts, default 25 s)
  4. role-specific handoff            (see table)
  5. flush OTel spans + logs
  6. close pools, release advisory locks, exit 0

onShutdown has three phases, accept → inflight → close. accept and inflight run hooks in registration order. close runs them last-registered first, As of 2026-11: resources are released in reverse of acquisition, and the boot starts the OTLP exporter before it imports the app, so the flush is the last hook and a teardown's own spans still reach the collector.

Both numbers are app.config.ts's drain section, applied to every role by the production boot (Configuration). The budget is what a running job gets on a deploy, so an app with long jobs raises it there: drain: { deadlineMs: 600_000 }. The platform's kill timer must outlast grace + budget — the Helm chart derives each role's terminationGracePeriodSeconds from the values x deploy --method helm passes it; on Compose, stop_grace_period is yours.

A hook or request still running at drain.deadlineMs (the worker's: drain.workerDeadlineMs when declared) is abandoned — logged as X_SHUTDOWN_TIMEOUT naming it, never thrown, and the process exits clean; requests arriving during the drain get X_DRAINING.

Retiring a worker before SIGTERM

The SIGTERM path above already finishes held jobs — inside the worker's drain budget: drain.deadlineMs (at most 1 h), or drain.workerDeadlineMs (at most a day, the worker alone) when declared. A rollout or a scale-down on Kubernetes does not start there: As of 30.0.0 the worker is retired first, for up to drain.workerRetireMs (an hour by default, 0 off, at most a day). The signal is SIGUSR2 (As of 25.0.0, ROLE=worker only, packages/cli/src/serve-retire.ts).

It exits rather than holds: the only thing a preStop can observe is PID 1 going away, and a worker that claims nothing has nothing left to serve. A second SIGUSR2 joins the first. One during boot waits for the worker, then retires it. A SIGTERM that lands mid-retire is the ordinary drain: the wait is bound to the worker's budget (drain.workerDeadlineMs, else drain.deadlineMs) and cut off at the margin, as in the table — so a worker whose jobs outlast the hour declares drain.workerDeadlineMs. isDraining() stays false until that drain; the app's own code (a heartbeat, a status row) reads isRetiring() to say "finishing, about to exit" — x dev's restart, which retires the same way, sets it too. Outside a pod deletion an exit 0 is restarted by the Deployment like any other: a fresh worker.

The Helm chart wires it behind one chart value, roles.worker.retireSeconds in values.yaml (3600, on, by default). x deploy --method helm derives it from the app's drain.workerRetireMs, 0 included. The worker's preStop sends its PID 1 SIGUSR2 and waits for it to exit, for at most those seconds; then the SIGTERM lands with the drain budget left, so a job past the retire is cut off and handed back rather than SIGKILLed. The same seconds are added to its terminationGracePeriodSeconds, which the kubelet counts the preStop against. An idle worker exits at once, so the retire costs only while a job really holds a pod, and a Terminating pod does not count against rollout completion: new pods claim at once, and the rollout shows an old worker Terminating while it holds a job. drain.workerDeadlineMs is a different knob: it bounds a SIGTERM drain, then aborts; the retire aborts nothing. The hook is bun -e, the one binary every app image carries — the image's ENTRYPOINT ["bun", "apps/web/server.ts"] is exec form, so bun is PID 1. Outside the chart, the target is the same: kill -USR2 1 inside the container, or docker kill --signal USR2 <container>.

helm upgrade app docker/helm --set roles.worker.retireSeconds=7200 \
  --set drain.workerDeadlineSeconds=7500   # 2 h worker; `x deploy --method helm` sets both itself

x verify holds the job to it: a job whose declared timeout exceeds the worker's budget (drain.workerRetireMs when above 0, else drain.workerDeadlineMs, else drain.deadlineMs) reds the manifest step with X_JOB_OUTLASTS_DRAIN, naming the job, its limit and the budget. A job with no timeout is not graded.

Compose stays SIGTERM with stop_grace_period: it has no preStop, and a SIGKILL mid-job is worse than a handed-back claim, so size stop_grace_period to the drain as below. It is the single-node rung, and makes no such claim → scale ladder.

Windows has no SIGUSR2: there the retire installs nothing and the worker stops only through the drain above (Ctrl-C, Ctrl-Break). A native Windows process is for development and CI.

Which signals start the drain

Windows never sends SIGTERM to a console process, so the set is per platform — drainSignals(platform) in @ultimat3/core, installed by every role As of 2026-10-05:

Stop a native Windows process with Ctrl-C or Ctrl-Break, and the drain log line names the signal ({"msg":"draining","signal":"SIGBREAK"}). Closing the console window also drains, inside the few seconds Windows allows before it kills the process. A native Windows binary is for development and CI. Deploying on a Windows host means the Linux image under Docker Desktop, where SIGTERM works as on any other host.

Why sync sends server-directed jittered reconnect

Closing 50,000 sockets at once means 50,000 simultaneous reconnects, all resubscribing, all asking "what changed since my LSN?" — a self-inflicted DDoS landing during a deploy when capacity is already reduced, and it is fractal: surviving nodes overload, drop connections, and the herd re-forms.

{ "type": "reconnect", "v": 3, "afterMs": 1830, "reason": "drain" }

ReconnectFrame in packages/realtime/src/sync-protocol.ts — four fields, built by reconnectFrame() (thundering-herd.ts); v is PROTOCOL_VERSION, reason one of drain, overload, rebalance.

sync takes both shutdown phases, and they answer different questions As of 2026-08. The accept phase calls stopAccepting(): /readyz flips to 503 and an upgrade arriving anyway is shed with retry-after-ms, while every socket the node already holds keeps its patch stream — sockets are untouched and no reconnect frame has been sent yet. The close phase is the drain below, then stop(). Registered with no phase, the whole thing landed in close, and until that last phase ran the node went on upgrading new websockets onto a process that was going away.

There is no realtime.drain config key — the spread window is syncNode({ drainSpreadMs }), default 30s, and the grace is drain({ graceMs }), default 5s (Configuration). The reconnect benchmark that gated topology now exists: 50,000 sockets, a SIGKILLed sync node with no drain and no reconnect frame — all 50,000 reconnected on their own backoff, 49,981 received a channel patch inside the window at p50 54.0s / p90 105.5s, 156,851 connect attempts shed before any query path. That is time to the first patch on the reconnected socket — reachability, not consistency — and it is the floor this section's frame is meant to beat. Measured on one node (Realtime).

x build

x build --target docker     # one image, all roles (default)
x build --target binary     # a Bun-compiled launcher: no Bun on the host, but the app tree beside it
x build --target static     # site/ output only: HTML, assets, sitemap, feeds
x build --target prebuilt   # the line the Dockerfile runs INSIDE the image build

What a pod boots from

A pod builds nothing at boot. As of 2026-10-01. The scaffolded Dockerfile carries, after COPY . .:

RUN bun node_modules/@ultimat3/cli/src/bin.ts build --target prebuilt

An image without the line still serves. Every boot then runs Babel over every island and Sass over every stylesheet, and logs one error line saying so — X_IMAGE_NOT_PREBUILT, with the counts, the milliseconds and the line above. Measured on the demo app's image, ROLE=web, four boots each:

An app scaffolded before 23.0.0 does not have the line — its docker/Dockerfile is the app's own file. Add it; the boot log names it until you do (Upgrading).

The prebuilt module registry. As of 2026-11-09. The same RUN also writes node_modules/.cache/ultimate/registry/: every module the app scan imports, in scan order, as chunk modules of 256 static imports each, plus registry.json (format, Bun version, file list). The web, sync and replicator roles boot from it instead of scanning; worker and scheduler keep their document-graph scan.

Every scan without a registry — tools, x dev's first load, x build — imports in batches of 256 through one generated entry module per batch, in the serial scan's evaluation order (packages/cli/src/app-load-order.test.ts). Measured cold module load, 5,000 synthetic modules: serial 3,258 ms, batched 1,259 ms (boot-vs-files.json).

All targets share one build ID (content hash), stamped into the image, the HTML, the assets, x.manifest.json — and sw.js, As of 2026-08: the worker's cache names carry the build id, so a deploy retires the previous build's caches instead of serving them (#390). Every target that serves or writes a document emits manifest.webmanifest, sw.js and x-sw-register.js alike (PWA and offline).

$ x build --target docker
  ✓ typecheck + boundaries           ✓ site/  12 routes  static   0kb js
  ✓ app/   31 routes  stream         ✓ static assets     avif+webp, 214 files
  ✓ manifest + openapi emitted       ✓ image  myapp:8f2a1c9  118MB
                                       build id 8f2a1c9

x build runs six of x verify's steps first — typecheck, lint, boundaries, filesize, package-shape, errors — and produces no artifact if any fail. It also refuses before spawning the builder when the target's entry file is missing (X_BUILD_ENTRY_MISSING) → CLI reference.

Dev compose

# docker/docker-compose.dev.yml — only needed for parity checks; `x dev` needs none of this
services:
  db:   { image: postgres:17-alpine, ports: ['127.0.0.1:5432:5432'] }
  nats: { image: nats:2-alpine, command: ['-js'], ports: ['127.0.0.1:4222:4222'] }
  s3:   { image: versity/versitygw:v1.8.0, ports: ['127.0.0.1:9000:9000'], volumes: ['s3bucket:/data/myapp'] }

x dev uses embedded Postgres, in-process NATS, and a local directory for S3 — Docker is not required to develop. This file exists for parity debugging: three backing services and no app service, because the app in development is x dev on the host, pointed at them by environment (the file's header is the line to paste).

Prod compose

# docker/docker-compose.prod.yml
x-app: &app
  image: myapp:${BUILD_ID}
  env_file: [../.env.production]   # what the CONTAINERS see — not what `${VAR:?}` reads
  restart: unless-stopped

services:
  migrate:    { <<: *app, environment: { ROLE: migrate },    restart: 'no' }
  web:        { <<: *app, environment: { ROLE: web, PORT: 3000 }, ports: ['3000:3000'],
                deploy: { replicas: 1 },
                depends_on: { migrate: { condition: service_completed_successfully } } }
  # PORT is the WEB port even here: `sync` binds PORT + 1, so 3000 is a listener on 3001.
  sync:       { <<: *app, environment: { ROLE: sync, PORT: 3000 }, ports: ['3001:3001'],
                deploy: { replicas: 1 } }
  # WORKER_QUEUES is the EXACT set a worker claims (25.0.0): `integrations` runs on its own
  # service, so the other worker names every remaining queue itself. Unset, a worker claims
  # jobs.queues plus every queue a registered job is enqueued on; set, `jobs.worker.queue-unserved`
  # names the registered queues it leaves that NO live worker serves — none, once `integrations` is up.
  worker:     { <<: *app, environment: { ROLE: worker, WORKER_QUEUES: default },
                deploy: { replicas: 4 } }
  worker-integrations: { <<: *app, environment: { ROLE: worker, WORKER_QUEUES: integrations },
                deploy: { replicas: 1 } }
  scheduler:  { <<: *app, environment: { ROLE: scheduler },  deploy: { replicas: 1 } }
  replicator: { <<: *app, environment: { ROLE: replicator }, deploy: { replicas: 1 } }

x deploy --method compose applies this against the committed docker/docker-compose.prod.yml; it is a plain compose file you can read, diff, and run by hand — from the app root, as docker compose --env-file .env.production -f docker/docker-compose.prod.yml up -d.

.env.production has one writer, x env init --production (As of 30.0.0): it fills every ${VAR:?} above, a DATABASE_URL naming the compose db (role postgres, database the app name, the generated POSTGRES_PASSWORD), and every secret the boot refuses to start without — then x deploy reads it before the first step and names what is missing (X_DEPLOY_ENV_MISSING). The three commands, executed on every push by the compose-deploy CI job: x env init --production, x build --target docker, x deploy (Deploy recipes).

The /app/.x tmpfs is owned by the image's user (As of 30.0.0): '/app/.x:uid=1000,gid=1000' in the scaffold (its USER 1000:1000), uid=65532 in the framework's specimen (distroless nonroot). A tmpfs mounts root-owned under read_only: true, so before this every role refused at boot with X_STORAGE_UNWRITABLE … EACCES mkdir '/app/.x/storage'. An app scaffolded earlier edits that one line.

web and sync are one replica each, and the file says so As of 2026-08. Both publish a host port, one host port has exactly one binder, so both declare replicas: 1 — a ceiling that is declared rather than discovered when the second container dies on port is already allocated. worker, scheduler and replicator publish nothing; worker scales freely.

Want	Do
more web/sync on the same box	delete their ports: lines, add a reverse proxy of your choosing to the compose file, point it at the service names — compose DNS resolves each to every replica
more web/sync, full stop	the Helm chart below, which carries a per-role HPA and an Ingress

The framework ships neither proxy: a proxy image in docker-compose.prod.yml is a dependency every app inherits and a second answer to "how does traffic reach a role" beside the chart's Ingress. This is the rung-1 ceiling → docs/idea/17-scale-ladder.md.

Helm

x new writes it, As of 2026-08-19 — docker/helm, one Deployment per role, 8 files: Chart.yaml, values.yaml and 6 templates. There is still no --helm flag on x build; the chart is a scaffold artifact like docker/Dockerfile, not a build target.

CPU autoscaling is wrong for sync and worker: a node holding 80k idle sockets is near-zero CPU and near-capacity, and a worker blocked on a slow HTTP call is idle CPU with a growing backlog. The framework declares both series — connections and queue_demand, with rps derived from the monotonic http_requests_total — and SCALING_METRICS maps each role's signal to its series so the chart and the role table cannot drift. As of 2026-08 every role serves /metrics on METRICS_PORT (default 9090) and http/realtime/jobs call the recorders, so the signals exist. The chart's half is closed in 2.0.0: values.yaml declares metricsPort: 9090, _helpers.tpl emits a container port named metrics on every role but migrate, service.yaml publishes it by name and templates/servicemonitor.yaml ships the scrape target. serviceMonitor.enabled is auto: the object renders wherever the cluster serves the Prometheus operator's kind and is skipped where it does not, so helm install never fails on an unknown kind — and templates/alert-rules.yaml ships the mechanism alerts the same way (alerts.enabled: auto). One thing remains, and it is not the chart's: turning scraped series into the Pods metrics an HPA reads needs a custom-metrics adapter. Do not hand-add a metrics container port — the chart already emits one and a duplicate is rejected by the API server → Observability.

x deploy --method helm works in a fresh app As of 2026-08-19: the command implements helm completely and its X_NOT_IMPLEMENTED branch — which claimed the build did not implement it, over a build that did — is deleted. An app that deleted its chart now gets helm's own error. The image goes in as --set-string image.repository=… --set-string image.tag=… As of 2026-10: a plain --set typed an all-digit tag (a build number) as an integer and rendered app:%!s(int64=…). As of 2026-10-05 it also passes the chart values --set drain.deadlineSeconds=… --set drain.readinessGraceSeconds=…, read off the app's own drain config, so the chart's grace periods follow a raised budget. Every role carries a readiness probe on /readyz (worker and scheduler on the metrics port), every Deployment minReadySeconds: 10, and every rolling role keeps maxUnavailable: 0.

The framework repo's own docker/helm carries one template the scaffold does not — pdb.yaml, which is cluster policy. It ships in no npm tarball, so taking it is a git clone of this repo. The monitoring pair (servicemonitor.yaml, alert-rules.yaml) is in both charts; an app chart written before them gets both from x g chart-alerts. Its image.repository is a placeholder (registry.example.com/your-app) — the framework publishes no app image — and an emptied one is a render error naming the key. On 3.0.0 and below, x new writes no chart at all and --method helm exits X_NOT_IMPLEMENTED: copy the chart in, or use --method compose.

Routing a proxy to the sync role

As of 26.1.0. A proxy in front of both roles (an Ingress, a Cloudflare tunnel, nginx, an ALB) sends the sync role exactly one path family, and everything else to web:

Anchor the rule. ^/_x/sync with no end anchor is the rule an operator writes first, and it is safe as of 26.1.0 only because nothing else the framework serves starts with that string. Through 26.0.0 the sync worker lived at /_x/sync-worker/<hash>.js: that rule sent it to the sync role, which serves only the socket, so no page got its shared worker (no realtime), and the 404 came back with no cache-control, which a CDN pinned for its default TTL (Cloudflare: four hours).

Every framework-served, content-addressed script lives under one asset prefix, /_x/assets/ — the Rails rule: <prefix>/<name>/<sha>.js, served immutable, a new URL per build:

Island chunks (/islands/…), surface stylesheets and asset() files have prefixes of their own; no live endpoint shares any of them. A miss on any content-addressed path is 404 with cache-control: no-store — a CDN never keeps it — and so is every answer of the sync role that is not the socket (a stray path, a plain GET of the socket path, a shed upgrade). Old /_x/sync-worker/, /_x/page-boot/ and /_x/navigation/ URLs are not served: each named one build's file, which no other build serves either, and a tab mid-roll that already started its worker keeps it.

Behind a proxy — TRUSTED_PROXY_HOPS

TRUSTED_PROXY_HOPS counts the proxies that append to x-forwarded-for between the client and web/sync → Configuration. Unset trusts no proxy header. Proxies treat the two headers differently, and the reader follows that:

As of 2026-10, two hops with an overwriting proto proxy read https, so HSTS goes out. Before, a one-entry proto header behind TRUSTED_PROXY_HOPS=2 counted as "not the chain": ctx.https was false and no HSTS was sent.

Make sure at least one trusted hop writes x-forwarded-proto. If none does, the client's own value reaches the app, exactly as it always has with TRUSTED_PROXY_HOPS=1. The edge that ends TLS is the one to set it: nginx proxy_set_header X-Forwarded-Proto $scheme;. An inner proxy reached over plain http must pass the edge's value on rather than writing its own $scheme — on ingress-nginx, use-forwarded-headers: "true".

Through a connection pooler

DATABASE_POOLER=session|transaction when a pooler sits between the app and Postgres, unset when none does. As of 2026-11-02. The client then sends neither bound as libpq options, and the bounds become the server's:

-- direct to the server, per database user; numbers per role: docs/ops/04-datastores.md
alter role <user> set statement_timeout = '10s';
alter role <user> set idle_in_transaction_session_timeout = '30s';

Then reopen the pooler's server connections: a role setting applies at login.

x doctor --json reports the bound as the session reads it (data.statementBound). A bound that is gone is X_DB_STATEMENT_BOUND_LOST, one error line at boot and a finding, never a refusal. The table per mode: docs/ops/04-datastores.md.

Compression

The container gzips a public text answer of 1 KiB to 256 KiB with a declared length. Nothing to configure, and nothing to add in front.

Rule and figures: docs/architecture/16-build-pipeline.md.

Static deploys independently

x build --target static        # dist in .x/static — upload it with your CDN's own tool

Targets

The only requirement: something that runs containers, plus Postgres. NATS and object storage are optional in small deployments — Postgres covers queue and pubsub; a local volume covers files.

Not supported, by design: vendor edge runtimes, serverless-function-per-route, vendor KV / queue / cron primitives, proprietary image loaders. Each would need a second implementation of a framework primitive, and the second implementation is where behavior diverges.

Release checklist

x verify                       # the gate — green means shippable
x build --target docker
ROLE=migrate <image>           # pre-deploy, must exit 0
<roll web + sync>              # drain-aware; clients reconnect with backoff
x build --target static        # independently, whenever copy changes
x doctor --json                # env, versions, drift, ports

Running it for real — docs/ops/

The framework depends on none of this; it is the operations manual for the app you deploy with it. Recommendations, not contracts — nothing in packages/ reads a word of it.

Two design-only companions — specification, not shipped behaviour: docs/idea/16-app-targets.md (three targets, one backend, two view layers) and docs/idea/17-scale-ladder.md (why the app code is identical at rung 0 and rung 4).

Rollback

Redeploy the previous image tag. Inside an assets.retain window (below) a rollback 404s nobody mid-session; without one, a tab holding the other build's document reloads.

The older image's migrate role decides whether the schema allows it (As of 2026-11). Over ledger rows the newer build applied it applies nothing and logs them (ahead). Over a row the newer build RECORDED as destructive (x_migrations.destructive, written at apply) it refuses before anything serves:

The refusal is the ROLLED-BACK image's code, so it protects from the release after the one that ships it; a rollback to an older build cannot refuse. x deploy verify --json then proves the rolled-back build serves → runbooks.

Hashed assets across a deploy — assets.retain

A web process serves only the hashed files it built: /styles/<hash>.css, /islands/<island>-<hash>.js, /_x/assets/<script>/<hash>.js, /assets/<name>.<hash>.<ext>. With no window, an open tab holding the previous build's document gets a 404 for its stylesheet, or for its island chunk on the first click, after every deploy, one replica or many. During a rolling update both directions fail: a new document that reaches an old pod also gets no stylesheet. Declare a window over a disk every web pod can read:

// app.config.ts — `deploys` is a disk from the app's defineStorage({ disks })
import { defineConfig } from '@ultimat3/core';

export const config = defineConfig({
  name: 'my-app',
  assets: { retain: { disk: 'deploys', builds: 3 } },
});

disk naming no declared disk, or builds that is not a whole number ≥ 1, is X_CONFIG_INVALID. x dev never publishes. An S3-compatible bucket or a volume shared on one box is the whole requirement: no CDN, no platform primitive. The prefix sits outside org/, so /_storage and /media refuse it. A static export needs none of this: a host keeps earlier exports' hashed files because nothing deletes them. Version-skew handling: Upgrading. Failure symptoms: Troubleshooting.