Testing

bun test. N workers, N real Postgres databases, sealed network. No mocking the database, no rollback hacks, no shared-state flakes.

Tests live next to their source as <file>.test.ts.

Parallel by database clone

x test unit --workers 4        # one bun test --parallel=4: Bun's workers 1..4
  worker 1 → ultimate_test_template_w1   (CREATE DATABASE … TEMPLATE ultimate_test_template)
  worker 2 → ultimate_test_template_w2
  ...

--workers is x test's flag, spent as Bun's --parallel; bun test itself has no --workers.

Rejected approaches

Real databases, truly parallel, is the only combination that is both fast and honest.

Database state, and what a run leaves behind

As of 27.3.0. The framework's DatabaseCleaner and tmp:clear in one place: every test starts from a known database by construction, and what a run leaves on disk or on the server is swept by the framework, never by the app.

The migrated database. migratedDatabase({ root }) from @ultimat3/cli is a fresh embedded Postgres holding the framework schema (x_jobs, x_users, …) and every migration under packages/db/migrations — exactly what x db migrate applies. It boots from a template (the migrated data directory, dumped once per PGlite version, framework schema and migration set) cached at <app root>/.x/test-db/pglite-<key>.tar: a restore is ~0.5 s where initdb plus the migrations is 4-7 s. A changed migration is a new key, so a stale template is never loaded. The recipe, once per app, in the app's own test-support module:

import { migratedDatabase } from '@ultimat3/cli';
import { setDbClient } from '@ultimat3/db';
import { afterAll, beforeAll, reusableDatabase } from '@ultimat3/testing';

const workerDatabase = reusableDatabase(() => migratedDatabase({ root: import.meta.dir }));

export function useMigratedDatabase(): void {
  beforeAll(async () => setDbClient(await workerDatabase()), 60_000);
  afterAll(() => setDbClient(undefined));
}

The handle follows the installed client, As of 30.0. The scaffolded packages/db/src/client.ts picks testDriver() under test (storeMode(env) === 'memory' ? testDriver() : postgresDriver()): memory until setDbClient installs one, that client while it is, decided per repository call. So db.<table> and a repo's raw sql join read and write one database in a file that runs the recipe, and a file that never runs it is the memory driver exactly: no PGlite loaded, no WASM cost. An app scaffolded before 30.0 makes that one-line edit in its own client.ts; until then its handle stays in memory while raw SQL reaches the installed client. Worked example: examples/dummy/apps/web/app/posts/repo-join.test.ts.

What a run leaves behind is swept — before every x test and every test step of x verify (once per process), and on demand by x clean:

Every .x path a framework module writes resolves from the APP ROOT (appStatePath in @ultimat3/core), whichever folder the process started in. A temp directory a framework test makes is removed by the file that made it; scripts/temp-dir-cleanup.test.ts refuses one that is not.

Determinism

Any test that can pass twice and fail the third time is worse than no test — it trains people to ignore red.

Sealed network is the highest-value rule: it converts "the suite is slow and occasionally fails" into "you forgot to mock Stripe, here is the line".

X_TEST_NETWORK_SEALED: unmocked network call
  cause: POST https://api.stripe.com/v1/charges from app/billing/service.ts:42
  fix:   mockFetch('https://api.stripe.com/v1/charges', () => new Response('{}')) — or allowHost('api.stripe.com') if it must be real

mockFetch(match, answer) holds what a retry test needs in one registration:

import { expect, mockFetch, requestLog } from '@ultimat3/testing';

declare function charge(): Promise<void>; // yours: the code under test

mockFetch('POST https://api.vendor.test/charge', [
  new Response('busy', { status: 503 }),
  new Response('{"id":"ch_1"}'),
]);
await charge();
expect(requestLog()).toHaveLength(2);
const [first, second] = requestLog();
expect(second?.headers['authorization']).toBe(first?.headers['authorization']);
expect(second?.body).toBe(first?.body);

• match: an exact URL, a prefix ending in *, or a RegExp, every verb. A string may name one verb first, 'POST https://…'. An address-literal URL carrying a host header also matches a route on that name, so a pinned dial answers the mock of its host.
• answer: a handler, or a list of Response | Error answered in order with the last repeated. An Error is thrown. An empty list is X_INVARIANT.
• requestLog(): { method, url, headers, body }[], in call order, refused calls and socket dials included, unredacted. Header names are lowercase. body is UTF-8 text or null; a ReadableStream on its way to a real server is never read and logs null. A stream body or a Request's own is captured beside the call, never ahead of the answer: it is in the entry once it ends, null until then.

A vendor's answers, recorded once — cassette(dir, name, options), As of 2026-11. Replay is mockFetch: each recorded URL becomes one route answered in call order, so anything else is still X_TEST_NETWORK_SEALED. Recording is by hand.

import { cassette, expect } from '@ultimat3/testing';

declare function refund(): Promise<{ status: number }>; // yours: the code under test

await using _tape = cassette(import.meta.dir, 'refund-flow', {
  record: { hosts: ['api.vendor.test'] },
});
expect((await refund()).status).toBe(200);

Locale and zone are declared per test when the behavior under test depends on them — see Timezones and dates and I18n.

The six test types

Each helper takes { timeoutMs } — unitTest(name, body, { timeoutMs: 30_000 }), the same on contractTest, liveTest, jobTest, e2eTest and test, and a key of evalTest's options. Handed to bun:test at this machine's pace; absent is Bun's 5 s default, paced the same way. Never test(testName('unit', …), body, 30_000) to buy a deadline.

unitTest, contractTest, liveTest and jobTest take the fixture bag, exactly as test does: unitTest('…', async ({ runJobs, clock }) => …). A body that destructures nothing builds nothing. Never describe(testName('unit', …)) around a bare test to reach a fixture. As of 2026-10.

The e2e step drives a real browser, As of 2026-09-22 (21.0.0). When Chrome is on the machine, x verify's e2e step spawns the app with startE2eApp (reset, seed, boot, on its own throwaway ULTIMATE_STATE_DIR) and runs the suite with a real page (packages/cli/src/verify-e2e.ts). Before this, the reference app's e2e suite ran with no browser at all, and its browser-backed cases skipped under a green gate; this page described them as running. The session API (openE2eBrowser(), cdpE2eSession) can open a second tab in the same profile (newTab) and add an init script. It can take the page offline, workers included, and set cookies. It lists the page's sockets and requests, reads IndexedDB database names, and waits for an expression (waitFor). That is what the two-tab and offline cases rest on. The spawned app runs with APP_URL set to its own origin, NODE_ENV stripped and ULTIMATE_ENV=development: a NODE_ENV=test inherited from bun test made it resolve as test, and a missing APP_URL answered 500 with X_ENV_MISSING. DATABASE_URL and S3_ENDPOINT are stripped too (25.0.0, #674): the state directory owns the database and the disk, and an exported integration-test Postgres made x db reset refuse an external database and the step go red. A run that wants an external one passes it in startE2eApp({ env }). Readiness is polled over node:http, because the e2e preload seals fetch against egress. Each e2e test gets 60 s (E2E_TEST_TIMEOUT_MS). A spawned app that will not come up is X_E2E_APP_FAILED.

Each type is a first-class runner with its own fixture shape — not a naming convention on top of one runner. Every command supports --json.

Evals

An eval declares its cases beside the prompt (<name>.evals.ts) and gates on the drop from a committed baseline, never on an absolute score — models drift, prompts should not.

export const summarizeEval = defineEval({
  name: 'summarize',
  prompt: summarizePrompt,
  cases: [{ name: 'refund', vars: { ticket: 'I want my money back' }, expected: 'billing' }],
  scorers: [exact, jsonSchemaValid(['category', 'summary'])],
  baseline: import.meta.resolve('./summarize.baseline.json'),
  tolerance: 0.05,
});

ULTIMATE_EVAL_RECORD=1 x test eval re-records every baseline, so accepting a new number is a reviewable diff instead of an edited threshold. It is refused inside the gate: x verify with that variable set is X_EVAL_RECORDING and runs no eval suite, because recording passes by definition and would overwrite the committed baselines during the run.

An agent's prompt takes the agent arm — one case is one whole agent.run(), its tools and its turns, under the gateway handed to run / assert. The prompt arm sends one request with no tools, which measures something an agent never runs. As of 2026-11:

import { calledTool, contains, defineEval, type EvaluatedAgent } from '@ultimat3/ai';
import type { Actor } from '@ultimat3/core';

declare const supportAgent: EvaluatedAgent<{ question: string }>; // yours: an agent() action
declare const reviewer: Actor;

export const supportEval = defineEval({
  name: 'support.answer',
  agent: supportAgent,                       // an agent() action, registered
  cases: [{ name: 'where-is-o-1', input: { question: 'Where is o-1?' }, expected: 'shipped' }],
  tools: { lookupOrder: () => ({ status: 'shipped' }) },   // a double per tool the model may call
  scorers: [calledTool('lookupOrder'), contains],
  actor: reviewer,                           // optional: default is the `eval` service actor
  baseline: import.meta.resolve('./support.baseline.json'),
  tolerance: 0.05,
});

Recorded runs come from scriptedProvider({ turns }); harvesting production failures into cases is the app's script — transcriptOf(error) hands over what happened.

// contract test — generated as a scaffold with the action
test('publishPost denies a non-owner', async ({ seed, actorFor }) => {
  const { post, stranger } = await seed('two-orgs');
  await expect(publishPost.as(actorFor(stranger), { postId: post.id }))
    .rejects.toBeUltimateError('X_FORBIDDEN');
});

// job test — the step guarantee, not the happy path
test('onboardOrg retries only the failed step', async ({ seed, clock, mail }) => {
  const { org } = await seed('fresh-org');
  mail.failOnce(nudgeEmail);
  await runJobs(onboardOrg, { orgId: org.id });
  clock.advance('3d');
  const trace = await runJobs.drain();
  expect(trace.steps.provision.executions).toBe(1);       // replayed from storage
  expect(trace.steps['nudge'].executions).toBe(2);        // only this one retried
});

The job example is the one that matters: it asserts the durability guarantee, not that mail was sent. See Jobs and workflows and Policies and authz.

The accessibility audit

Every e2eTest audits the browser's accessibility tree, As of 2026-11 (30.0.0 entry 88). After each settled goto, reload and gotoStreamed, and once more after the body (so a dialog the body opened counts), the driver reads the tree the browser computed for a screen reader and fails the test once, at the end, with X_E2E_A11Y_VIOLATION listing every element. There is no switch.

Name what it lists: visible text, aria-label, <label for>, alt (alt="" for a decorative image, which the browser then leaves out of the tree), a <title>, <html lang>. A deliberate exception is per test and per element, with a reason:

import { e2eTest, expect } from '@ultimat3/testing';

e2eTest(
  'the gallery paints',
  async ({ page }) => {
    await page.goto('/gallery');
    await expect(page.getByRole('heading', { name: 'Gallery' })).toBeVisible();
  },
  {
    a11y: {
      allow: [
        { role: 'image', selector: '#vendor-map img', because: 'the vendor embed sets no alt' },
      ],
    },
  },
);

And every raw tab, As of 2026-11-05 (entry 88). A tab a suite drives itself (browser.session.newTab() off openE2eBrowser(), or the browser's first page) is read after each settled goto and reload, and once more at the test's end, and the findings fail that test from the preload's one afterEach — no suite opts in. A raw-tab test has no options slot, so its exceptions are a call, first in the body, before the navigation that lands on them:

import { test } from 'bun:test';
import { allowA11y, openE2eBrowser } from '@ultimat3/testing';

test('the vendor map renders', async () => {
  allowA11y([{ role: 'image', selector: '#vendor-map img', because: 'the vendor embed sets no alt' }]);
  const browser = await openE2eBrowser();
  try {
    const tab = await browser.session.newTab();
    await tab.goto('http://localhost:3000/map');
  } finally {
    await browser.close();
  }
});

An entry covers the reads after the call and ends with the test; a blank because is refused, as in e2eTest. Each goto now spends five CDP deadlines, not three (E2E_GOTO_MS), for the audit's two reads. Source: packages/testing/src/e2e-a11y.ts (rules), e2e-a11y-run.ts (one test's run), e2e-a11y-tabs.ts (raw tabs), ax-node.ts (the tree reader).

Matchers

Installed by the preload on expect.

toEqualRow exists because toEqual walks enumerable properties and a repository row does not enumerate a sealed column (Sealed columns):

const row = await db.connections.insert({ label: 'primary', password: 'hunter2' });

expect(row).toEqual({ id: row.id, label: 'primary', password: 'hunter2' });    // fails, empty diff
expect(row).toEqualRow({ id: row.id, label: 'primary', password: 'hunter2' }); // passes

A failure names each property that differs. A server-only one is named and never printed — not its value, not its length.

Rendering a page in a unit test

No server, no browser, no cast. renderRoute takes the route module whole and resolves it the way a request does; renderView renders one component with its own props.

import { expect, renderRoute, renderView, unitTest } from '@ultimat3/testing';
import * as page from './page';

unitTest('the dashboard counts the posts of the org that is looking', async () => {
  const view = await renderRoute(page, { url: 'https://example.test/dashboard', actor: viewer });
  expect(view.data.rows).toHaveLength(2);
  expect(view.meta.title).toBe(t('app.dashboard.title'));
  expect(view.text).toContain(t('app.dashboard.tableTitle'));
  expect(view.islands.map((island) => island.moduleId)).toEqual(['shared-theme-toggle']);
});

actor runs load and the page inside a request context holding that actor. renderView refuses a component that renders an island() — its hydration timing is a route's. As of 2026-10.

Asserting on what an element carries

A string cannot say whether a node carries aria-live, or what its onKeyDown does. Walk the tree instead, imported from the package that declares it, @ultimat3/core/render-nodes:

import { fire, one, renderNodes, withAttr } from '@ultimat3/core/render-nodes';
import { Menu, type MenuItem } from '@ultimat3/ui';

declare const items: readonly MenuItem[];
const menu = one(withAttr(renderNodes(Menu, { items }), 'role', 'menu'), 'the menu');
fire(menu, 'onKeyDown', { key: 'ArrowDown', preventDefault: () => undefined });

nodesOf(value, { components }): 'expand' (default) calls components and keeps host nodes only; 'keep' keeps each component node beside its output, so byComponent(nodes, 'DateTime') reads the props a view handed it; 'stop' never calls one. An island's server shell is walked; a hole is its first flush, fallback and error, and its deferred children are never called. one(), attachRef() and fire() refuse with X_INVARIANT, naming what they sought. The worked example: examples/dummy/apps/web/app/settings/page.test.ts. As of 2026-11.

Running a mutator's local half

const store = memoryLocalTx({ posts: { [id]: { id, title: 'before', pending: false } } });
renamePost.local(store.tx, { id, orgId, title: 'after' });
expect(store.rows('posts')).toEqual({ [id]: { id, title: 'after', pending: true } });

memoryLocalTx(seed?) is the client store under test: keyed rows, update a no-op for a key it does not hold, undefined leaving a column alone, rows frozen. Apply local twice and compare — a rebase replays it. As of 2026-10.

Seeing what you built

An assertion says a component behaved. It does not say what it looked like while it did. x shot is the other half: a real browser, a PNG, and a verdict.json that says whether the picture is of what you think it is.

x shot /dashboard --json              # one route
x shot --island post-form --json      # one island, every state it declares, both themes
x shot --island post-form --state refused --json
x shot --all-islands --json           # every island in the app, plus an index

x shot is deliberately not a step of x verify. It needs a real browser, and a gate that goes red because a machine has no Chrome fails for reasons unrelated to the change. x shot launches Chrome itself over raw CDP — no browser library in the app. It takes the first of --browser, PUPPETEER_EXECUTABLE_PATH, CHROME_PATH, then a probe of the usual install paths; or it attaches to a running browser with --cdp-url (or SCRAPE_CDP_URL). No browser at all is X_SHOT_BROWSER_MISSING, raised before the dev server boots: export CHROME_PATH=<binary>.

As of 2026-09-23.

States are declared, not clicked

A state a running app will not produce on request — a save the server refused, a read that came back empty, a label three times as long in the next locale — is unreachable to a reviewer and to a model. Declare it instead, in a <name>.island.states.ts beside the island:

import { defineIslandStates } from '@ultimat3/testing';

export const postFormStates = defineIslandStates({
  island: 'apps/web/app/post/post-form.island.tsx',
  states: [
    { id: 'idle', title: 'the first paint', props: {} },
    {
      id: 'refused',
      title: 'the server refused the save',
      note: 'unreachable by clicking — the action only fails for an actor without the capability',
      props: { error: 'You cannot publish in this org.' },
    },
    {
      id: 'slow',
      title: 'the save is in flight',
      note: 'a request that never settles; a real one resolves faster than a screenshot',
      props: {},
      routes: [{ match: 'POST /api/post', respond: { kind: 'pending' } }],
    },
  ],
});

Rules the file is held to:

x g island <name> and x g resource <name> write the states file with the island — two states, one of them a translation three times as long. You are editing a generated file, never creating one.

The index is the artifact

A run writes .x/shot/island/index.md — one Markdown file an agent opens instead of guessing at forty PNGs. Written for a single-island run too: the file that says what a picture is cannot be a property of how many islands you asked for.

Pictures land at .x/shot/island/<name>/<state>-<theme>.png. Flat and mechanical, because the reader guesses the path.

The blind spots are in the artifact rather than in this page: the picture is the crop target and a thin margin, so a component's own fault sitting further out is outside the frame, and a toLocaleString() on a Date resolves its zone inside the engine where only an explicit timeZone is pinned. The index prints both under "What this capture cannot see".

What the rendered result is then held to: Interface rules.

The fixture bag

test and the typed helpers pass a bag as the first argument and build only what the body destructures — a test that never names runJobs never starts a queue. The framework owns the whole bag; an app registers only what the framework cannot know.

waitForServiceWorker({ timeoutMs? }) hands the first page over (25.0.0, #678). The framework's register script runs on load, so the first page of a run is committed before its worker activates and clients.claim() can miss it: ready resolves and no controllerchange ever fires. The wait answers ready, gives the claim a second (CLAIM_GRACE_MS), and reloads an uncontrolled page ONCE — that navigation is answered by the worker — within one budget (serviceWorkerTimeoutMs, 10 s, or this call's timeoutMs). A page with no worker is refused without a reload, and a reload still uncontrolled is refused saying so (X_E2E_SERVICE_WORKER_ABSENT). Proven against a real worker that never claims: packages/testing/e2e/service-worker-handover.e2e.test.ts.

The last five are declared, not built. The framework does not bundle a browser, so the name resolves and asking for it without a driver fails as X_TEST_FIXTURE_UNAVAILABLE naming what is missing — different from X_TEST_FIXTURE_UNKNOWN, whose fix ("register it") would have the app inventing its own idea of what a page is. A driver installs through the same registry:

// scripts/test-setup.ts
defineFixtures({ page: () => openBrowserPage(), seed: () => loadSeed });

defineFixtures merges and the last registration wins, so a driver replaces the declaration it was waiting on. There is no second seam.

network.offline() fails every request ahead of the mocks, so the app's own offline path runs instead of a branch written for the test; drop() is the same for a request but tells a subscriber its connection was cut rather than closed, which is what separates a resume from a resubscribe.

A job under runJobs

unitTest('a row deleted before the run is skipped, not failed', async ({ runJobs }) => {
  const trace = await runJobs(reindexPost, { id, orgId });
  expect(trace.executions.map((run) => run.outcome)).toEqual(['completed']);
  expect(trace.executions[0]?.result).toEqual({ skipped: true });
  expect(Object.keys(trace.steps)).toEqual(['load']);
});

Enforced through it, keyed or not: concurrency and whenBusy — each pass is a real worker's tick(), so a run over its slot waits for the next pass, or is refused under whenBusy: 'fail'. The lease is renewed on the test clock — every millisecond of it — so a body in flight learns of a cancel at the next clock.advance(): cancel, clock.advance(1), await the drain. A clock.advance() past the visibility timeout while it runs is X_JOB_LEASE_LOST, as in production. As of 2026-10.

A job killed at every boundary, and raced from a seed

crashProof(handle, input) and simulateJobs({ … }), both from @ultimat3/testing, judge a job by its effects — row writes, mail, calls out, enqueues — never its return value. Reads are not effects. As of 2026-11.

A compensates: true job is also forced to fail for good and killed while it unwinds: each compensate:* once, last in first out. A kill inside step.run(…, { once: true }) ends X_STEP_OUTCOME_UNKNOWN and is green. Postgres is not simulated. The reference: examples/dummy/apps/web/app/posts/jobs.job.test.ts.

A model call under test

echoProvider() answers text; it cannot call a tool. A test that drives an agent() through its tools scripts the model with scriptedProvider({ turns }): each provider call answers the next turn, and .seen keeps every request in order.

import { meteredRun, providerGateway, scriptedProvider } from '@ultimat3/ai';
import { installAiGateway } from '@ultimat3/ai/adapter';

declare const draft: { id: string; orgId: string };
declare function reviewDraft(input: { postId: string; orgId: string }): Promise<unknown>;

const model = scriptedProvider({
  turns: [
    { call: 'summarize', input: { postId: draft.id, orgId: draft.orgId } },
    { answer: { summary: 'Minor units and a currency.', tags: ['money'] } }, // the tool's own llm()
    { answer: { verdict: 'revise', notes: 'Say which rounding rule the body defends.' } },
  ],
});
const release = installAiGateway(providerGateway({ providers: [model] })); // release() in the afterEach

// The agent's turns and its tool's model call, as one bill: three calls.
const { billed } = await meteredRun(() => reviewDraft({ postId: draft.id, orgId: draft.orgId }));
console.assert(billed?.calls === 3 && model.seen.length === 3);

One call past the last turn is X_AI_SCRIPT_EXHAUSTED, never the last turn again: a loop that never answers fails the test instead of spinning to maxTurns on a script that looked like it passed. usage fixes every turn's token counts; absent, they are estimated from the request, as echo does. As of 2026-11.

Testing an island

One block per state the island's manifest declares. describeIslandState owns the lookup, the mount (beforeAll, 60 s unless timeoutMs says otherwise) and the teardown; the island is built once per file.

const island = { build: buildIslands, root: join(import.meta.dir, '..', '..', '..') };

describeIslandState(counterStates, 'idle', island, (mounted) => {
  test('a click reaches the DOM through the signal', () => {
    expect(mounted().fire('button', 'click')).toBe(true);
    expect(mounted().text('[data-role="count"]')).toBe('1');
  });
});

The props are the manifest's, so the picture x shot --island takes and the test are of one component. mountIsland stays the call for props no state declares. As of 2026-10.

A realtime island's page outlives its mount: the socket, the record store and the outbox sit on globalThis, shared by every bundle. Between files (or cases that need a fresh page) call resetPageRealtime() from @ultimat3/realtime in afterAll — it closes the socket the CHUNK's copy of realtime built and drops every page global. Never Reflect.deleteProperty(globalThis, Symbol.for('ultimate.realtime')): that leaves the socket open. As of 2026-11.

Testing an authenticator

unitTest('a session cookie resolves the member it was issued to', async () => {
  const actor = await authenticate(...(await authRequest({ cookies: { session: token } })));
  expect(actor?.id).toBe(member.id);
});

authRequest({ url?, method?, headers?, cookies?, bearer? }) answers the two arguments configureAuthenticator()'s function is handed — a real request over a real context, no server, no cast. cookies and bearer win over the same header spelled in headers. As of 2026-10.

Testing a passkey ceremony

import type { PasskeyLogin } from '@ultimat3/auth';
import { softAuthenticator } from '@ultimat3/auth/soft-authenticator';

declare const passkeys: PasskeyLogin; // passkeyLogin(defineAuth({ adapter, passkeys: { rpId, origins } }))
declare const userId: string;

const key = await softAuthenticator({ origin: 'https://app.test' });
const created = await key.create(await passkeys.registerOptions({ userId }));
await passkeys.register({ userId, response: created });
const session = await passkeys.login({ response: await key.get(await passkeys.loginOptions()) });

softAuthenticator is a passkey with real WebCrypto keys (ES256 or RS256): create(options) answers what newPasskey would in a browser, get(options) what getPasskey would, so the test runs the server's real verification. One implementation and one path, @ultimat3/auth/soft-authenticator, the one auth's own suites use. As of 2026-11.

Testing a sign-in against an identity provider

As of 2026-11. fakeIdentityProvider({ users, issuer?, clientId?, clock? }) is an OpenID provider served through mockFetch, no socket: /.well-known/openid-configuration, a JWKS from a key it generates, /authorize approving as the chosen user, /token (PKCE checked) and /userinfo.

import { discoverOAuthProvider, registerOAuthProvider } from '@ultimat3/auth';
import { expect, fakeIdentityProvider, unitTest } from '@ultimat3/testing';

unitTest('the provider registers from its own discovery document', async () => {
  const idp = await fakeIdentityProvider({ users: [{ sub: 'ada', email: 'ada@corp.test' }] });
  const provider = registerOAuthProvider(
    await discoverOAuthProvider({ id: 'corp', issuer: idp.issuer }),
  );
  expect(provider.jwksUri).toBe(idp.jwksUri);
  idp.tamper({ wrongKey: true }); // every token from here on fails the signature check
});

Each tamper is refused by @ultimat3/auth's real verifyIdToken with the code a real provider's broken token gets (packages/testing/src/fake-identity-provider.test.ts). An unknown sub and an empty users are X_INVARIANT.

Log lines under test

LOG_LEVEL=info bun test <file> is how to read the lines. A level above info filters the terminal and never the logger, so a test asserting on an info audit line passes under LOG_LEVEL=error. As of 2026-10.

An N+1 fails the test it happened in

x dev warns about a query loop; CI is where nobody is watching. Destructuring statements installs the same detector in strict mode for the length of one test:

test('the feed reads its authors once', async ({ statements }) => {
  await renderFeed();
  //   X_N_PLUS_ONE_QUERY: members.findById ran 5 times in one request — one read per row
  //   fix: db.posts.preload('author')   # one statement for the whole page
  expect(statements.count('posts.findMany')).toBe(1);
});

The loop runs to its end and the verdict is raised when the fixture is disposed — the bus never lets an observer fail the read it observes — naming the shape and its count; statements.verdicts() reads it earlier. Opting in is naming the fixture: there is no strict: true to remember and no switch left on for the next file.

A deliberate loop is declared where it is written, never silenced at the test:

await expectedQueryLoop('one indexed lookup per searchable field', () => searchEachField(term));

Factories

One row shape per entity, deterministic, with named variations, FactoryBot-grade (As of 31.0.0). defineFactory is the only entry point — there is no bare factory().

import { belongsTo, defineFactory, hasMany } from '@ultimat3/testing';
import { entity, text, timestamp, uuid } from '@ultimat3/entity';

const orgs = entity('orgs', {
  columns: { id: uuid().primaryKey(), slug: text(), name: text() },
});
const posts = entity('posts', {
  columns: {
    id: uuid().primaryKey(),
    orgId: uuid().references(() => orgs.id).tenant(),
    slug: text(),
    title: text(),
    status: text().default('draft'),
    publishedAt: timestamp().nullable(),
  },
});
const comments = entity('comments', {
  columns: {
    id: uuid().primaryKey(),
    orgId: uuid().references(() => orgs.id).tenant(),
    postId: uuid().references(() => posts.id),
    body: text(),
  },
});

const orgFactory = defineFactory(orgs, {
  defaults: (n, ids) => ({ id: ids.uuid(), slug: `org-${n}`, name: `Org ${n}` }),
});
const postBase = defineFactory(posts, {
  defaults: (n, ids) => ({ id: ids.uuid(), orgId: '', slug: `post-${n}`, title: `Post ${n}` }),
  traits: { published: { status: 'published', publishedAt: new Date('2026-01-05T09:00:00Z') } },
  transient: { shout: false },
  belongsTo: { org: belongsTo(orgFactory, 'orgId') },
  afterBuild: (post, { transient }) => {
    if (transient.shout) post.title = post.title.toUpperCase();
  },
});
const commentFactory = defineFactory(comments, {
  defaults: (n, ids) => ({ id: ids.uuid(), orgId: '', postId: '', body: `Comment ${n}` }),
  belongsTo: { post: belongsTo(postBase, 'postId') },
});
const postFactory = postBase.extend({ hasMany: { comments: hasMany(commentFactory, 'postId') } });

postFactory.build(); // memory: org built alongside, nothing written
const post = await postFactory.with('published').create({ comments: 2 }); // the STORED row
const org = await orgFactory.create();
await postFactory.createList(3, { org }); // reuses that org, makes none
postFactory.attributesFor(); // an action input: no id, no orgId

Determinism

seed defaults to a hash of the table name, so every table has its own uuid stream while staying a pure function of the schema. As of 2026-08 that is the 1.1.0 fix: every registry factory used to default to seed: 1, so two tables minted the same uuid and a join assertion could pass for the wrong reason.

ids.uuid() and ids.number() come from the seeded generators; index starts at 1 for the first row.

factoriesFor(registry, seed?) derives a factory per entity, each value read off the column's declaration, never its name (As of 2026-11, 30.0.0 entry 89): the declared default first, else an enumerated value, else what the kind offers (uuid, text, integer, boolean, timestamp, money as { minor, currency }, …), and null only where the column is nullable. A text column held to a matches(/…/) invariant is also offered strings built from that pattern, and those come first; a json(schema) column is offered values its schema accepts (As of 2026-11-07). Every candidate goes through the column's own $parse, so a built row is one the table stores. A registry entry whose $columns are not entity() columns, or a column that refuses every generated value (a json() whose schema carries a pattern its sampler cannot meet), throws X_SEED_GRAPH_UNSATISFIED at build(), naming the defineFactory to write. Source: packages/testing/src/factory-registry.ts, seed-graph-values.ts.

Persisting

create() writes through the entity's own repository on defaultDriver() — no setup for an app on the default driver. An app that names its own driver installs it once, in its test preload:

// scripts/test-setup.ts
import { entityPersister, usePersister } from '@ultimat3/testing';
import { driver } from '@app/db';

usePersister(entityPersister({ driver })); // or { client }

build() needs no persister. clearPersister() and persisterInstalled() round out the seam.

A whole graph: seedGraph

Rows for every table, with no factory written. Pure: it returns a plan, insertSeedGraph writes it.

import { entity, enumerated, text, uuid } from '@ultimat3/entity';
import { insertSeedGraph, type SeedWriter, seedGraph } from '@ultimat3/testing';

const orgs = entity('orgs', { columns: { id: uuid().primaryKey(), name: text({ max: 80 }) } });
const posts = entity('posts', {
  columns: {
    id: uuid().primaryKey(),
    orgId: uuid()
      .references(() => orgs.id)
      .tenant(),
    status: enumerated(['draft', 'published'] as const),
  },
});

const graph = seedGraph({ orgs, posts }, { tenants: 2, perTable: 3, seed: 7 });
// graph.order is ['orgs', 'posts']: one org per tenant, three posts in each

declare const writer: SeedWriter; // insert, update and snapshot, over your driver
await insertSeedGraph(graph, writer);

X_SEED_GRAPH_UNSATISFIED names the table, the columns to supply and the defineFactory call. As of 2026-11-05, seed-graph.live.test.ts inserts both tracked apps' schemas into Postgres with no factory, and seed-graph-pglite.test.ts the same into PGlite, in the unit step. Source: packages/testing/src/seed-graph*.ts.

Shared examples

One contract, asserted against many subjects, with the subject passed as a thunk so a beforeAll-built value is read at test time rather than at declaration time.

const anAuthenticatedAction = sharedExamples<Denier>('an authenticated action', (subject) => {
  test('denies an anonymous actor', () => {
    expect(subject().denies('anonymous')).toBe(true);
  });
});

describe(testName('unit', 'publishPost'), () => {
  behavesLike(anAuthenticatedAction, () => publishPost);
});

behavesLike wraps the body in describe('behaves like <name>'), so a failure reads publishPost > behaves like an authenticated action > denies an anonymous actor — the subject and the contract both named. Because it calls describe, it belongs at declaration scope and never inside a test body. The body runs once per behavesLike call.

Every adapter port's contract ships as shared examples on @ultimat3/testing/conformance — never . (31.0.0, Upgrading #46), one suite per ADAPTER_PORTS row, named as its conformance field: behavesLike(jobDriverConformance, () => myDriver) holds a custom JobDriver to claim / ack / nack, the claim fence, lease expiry and the burial of a row whose lease lapsed on its final attempt (onExhausted, dropExhausted); behavesLike(budgetStoreConformance, () => myStore) races 24 concurrent takes on one key against an @ultimat3/ai BudgetStore. Every suite and its subject: the README's table; per port, with the reference to copy: Adapters.

Generated scaffolds

Every generator writes the tests beside what it wrote, and the unit half RUNS the code — the coverage floor counts that suite alone, so a body only a typed suite reaches is uncovered source. As of 2026-10.

A stored-row case is written only into a slice whose entity still has the columns x g entity scaffolds; one an author reshaped gets the cases that need no row.

The exercise assertion

.contract()'s fourth assertion calls the handler with generated input its own schema accepts. A legal input that crashes the handler is red in contract, the day the action is written. As of 2026-11.

A query has the same assertion, read off the same file: orgFeed.contract({ exercise }), named <name>: read survives generated input. "Reached" is sql() building a source, and a read writes nothing, so no transaction wraps it; it refuses a deploy the same way. Both are one loop, @ultimat3/core's runExercise: the verdicts above are the same words for a read and a write. The generators declare an actor (x g action, x g resource), holding the slice's grant: a hand-written policy.ts's one literal can('<perm>'), or { off } when its grant cannot be read. As of 2026-11-08 the call is target.contract({ exercise }) and the actor is the ENTRY the generator inserts into apps/web/exercise.ts (userActor({ id: 'member', orgId, permissions: [grant] }), its draw's orgId bound under a tenancy rule; x new writes its example slice's the same way). The file is created from x new's when absent; a file the generator cannot edit — no single export const exercise = defineExercise({ … }) — is X_EXERCISE_FILE_UNEDITABLE, the entry to paste in its fix:. x g query emits no .contract() call: x exercise picks a reader by role. @ultimat3/schema/sample's arbitrary(schema, { seed, run }) is the draw, off the barrel.

The exercise pass

The exercise assertion asks of one action, in a test someone wrote. x exercise asks of the whole app, with no test: every action, query and job, run once against generated rows. Same option shape, same unreached code. As of 2026-11.

x exercise --json                 # the whole pass
x exercise publishPost --json     # one primitive, after pasting its override

The tenant proof

Every action, query and job is attacked from another tenant, around its own call, As of 2026-11. Tenant B, holding the same role, sends exactly A's first input — A's seeded ids bound in. Source: packages/testing/src/tenant-proof.ts, tenant-canary.ts, tenant-proof-report.ts.

B's actor is chosen as A's was, in B's tenant, with tenancy:cross stripped from its scopes: an attack holding the escape would only prove the escape works. B being refused is never judged — only what came back and what changed.

off and refused records carry no tenant. A job has no policy, so it is public only as a declared sweep or a tenant: 'none' job handed nothing of A's. crossTenant() is the one declaration the proof reads (@ultimat3/entity tells a witness each scope opened, after its capability is proven); it excuses nothing on an action or a query, which answer B directly, nor on a job B handed one of A's ids — that is B choosing A's row, crossed as ever. tenant: 'none' excuses only the write half, and only with no id of A's handed: an action or a query runs as B's actor, so a write to A's row with no id in the input is a statement no tenant predicate scopes, crossed. A finding prints paths, tables and seeded ids, never a bound value — one per leg and code, its places listed once, a list index that only repeats collapsed (effect.enqueue[*].input.orgId (3 each)). The fix on each: x exercise <name> --json, then read or write the row through the repository scoped to ctx.actor; for a tenant: 'none' job handed nothing of A's, whose read has no org to scope to, wrap it in crossTenant('<why <name> reads every org>', async () => …). What this does not walk — a route load, a task — is Policies and authz.

The crash proof

Every job the pass reached is killed at every step boundary, As of 2026-11: crashProof (above) with the control's own input and actor, after the tenant proof. Source: packages/testing/src/exercise-crash.ts, packages/cli/src/exercise-findings.ts.

Where the time went. Every record carries proofMs: { tenant, crash, scale } beside ms (the control's own call); the report carries phases: { seed, primitives, scale, plans, subject }, in milliseconds. No per-record plan time: one statement is planned once for every primitive that sent it. The table's tenant ms / crash ms / scale ms columns, then took 44704 ms — seed 818 · primitives 41099 · … — slowest: primitives and the five slowest proofs by record, slowest: job billing.plan-catalog · crash 35692 ms. As of 2026-11.

The human output prints one line per killed boundary, job chargeOrder · killed at mail · X_JOB_CRASH_DIVERGED row:payments:insert 2× (clean 1×). The fix reruns the one job and names the effect to move:

x exercise chargeOrder --json   # in job("chargeOrder"): wrap the "row:payments:insert" made after step "create" in step.run('<name>', async () => …)

The plan and scale sweeps

After every primitive ran, two sweeps, with no flag to turn either off, As of 2026-11. Source: packages/testing/src/scale-sweep*.ts, plan-oracle.ts, plan-sweep*.ts.

With the planner's scans and sorts disabled it picks one only when no index can answer, so the verdict reads the schema on empty tables — no volume, no statistics. Never red: scale.verdict flat (a declared loop listed in expected with its reason) or unproven (a call that did not answer at both sizes); a statement whose equality pins a unique key; one a partial index serves once its value is bound; a whole read beside a join whose join column leads a declared index on that side, while the other side is bound by its own key — on empty tables a partial index the statement implies looks like one row, so the planner walks it rather than probe it per row (As of 2026-11-08; with no such index the read is red and its indexes: entry is the join column); one under expectedSeqScan('<why>', fn) or on an entity with scan: '<why>' (plans.exempt); one generic planning refuses (plans.unplannable). A server below Postgres 12: X_QUERY_PLAN_UNAVAILABLE.

x db gen posts-orgId-createdAt-id-index   # after indexes: [{ on: ['orgId', 'createdAt', 'id'] }] in entity("posts")

Memory, width and isolation

As of 22.7. The gate is built for the machines it runs on — 8-16 GB, 2-8 cores, often two agents' gates at once — and it is held to a memory budget, not to whatever looks free.

• Width. max(2.75 GiB, 25% of total RAM) divided by 1.25 GiB a worker, clamped to 1..cores: 3 workers on a 16 GB box, 12 on a 62 GB 16-core one, 2 on an 8 GB one (the 2.75 GiB floor). Never more workers than cores. The step line says what it chose and why — 3 workers (budget 4.0 GB). ULTIMATE_TEST_MEMORY_BUDGET=3g replaces the budget, ULTIMATE_TEST_MAX_WORKERS=2 caps the width, --workers N wins over both. Total RAM, never "free": page cache counts as free, and two gates each planning on it took a 45 GB box down.
• One budget per machine. Each batch of workers leases that many slots from a pool of lock files under the OS temp dir (one per worker the budget allows; a dead holder's slot is taken over). A second x test or x verify on the same box gets what is left and runs narrower, or waits for one slot, instead of doubling the memory. ULTIMATE_TEST_SLOTS=0 turns it off.
• No per-file isolation by default. A worker keeps one global and one module registry across the files it runs (bun test --parallel=N --no-isolate) — measured 2-5x faster than --isolate at the same width. Between two files the testing preload hands the next file the process it would have had alone: undisposed island mounts are disposed, globalThis and process.env go back to the first file's baseline, the permission/role/catalog registries are restored (as a union — a module imported once per worker declares once), tasks exactly (an anonymous-task-<n> or anonymous-job-<n> name is never re-minted in a process — resetTasks()/resetJobs() clear the registry, not the counter — so a later file's task cannot take an earlier one's seat), every mock.module (vi.mock, jest.mock) the file made is put back to the real module — Bun keeps one past mock.restore() and past the file — and .tsx always compiles with the app's JSX factory. The real module is loaded for its snapshot before the fake goes in, so its top-level code runs; one that throws there cannot be put back, and the next file on the worker fails X_TEST_MODULE_MOCK_UNRESTORABLE naming the file that mocked it. A repository whose tests need a fresh global per file says "isolate": true in x.verify.json, or passes --isolate to x test / x verify. The framework repository itself still says true, As of 2026-11: ai mcp in one shared process went from 66 failures to 0 and auth mcp ai testing reads 0 (a test file's top-level definePermissions is its worker's environment — declare in beforeAll); cli is unmeasured.
• One long-lived process per worker. Without isolation a worker's memory is set by its live database, not by how many files it ran, so a pass is ONE bun test --parallel=N (recycled only past 256 files a worker; 24 under --isolate, whose heap grows per file). Every parallel run reads and refreshes .x/test-timings.json, so Bun starts the slowest files first.
• One database per worker. reusableDatabase(open) from @ultimat3/testing opens an embedded database once per worker and resets its DATA to the template between files (truncate, re-insert the snapshot, reset sequences, triggers off meanwhile) — a PGlite boot from a migrated template costs 2.4-7 s, which a per-file database pays in every file.

Measured, As of 2026-09-27, 12-core box, whole process tree sampled every 200 ms:

The framework repository itself opts back into "isolate": true: its suites exercise the process-global registries on purpose (fixture apps per test, empty permission sets), and without isolation a handful of files fail depending on which worker ran what before them. Apps do not need it — one adopter's unit, contract and job tiers pass without isolation.

A worker on the embedded database (PGlite) cannot go much below 1 GB: one booted PGlite holds 0.9-1.1 GB RSS even after close() and a full GC.

For CI, where wall time matters more than one box's memory, the gate splits across jobs: CI: the gate across parallel jobs.

Coverage

95% of lines and 95% of functions — one bar, for the framework's packages and for every app. Held by the unit step: lint and tests alone are not green.

{
  "steps": ["unit"],
  "coverage": {
    "lines": 95,
    "funcs": 95,
    "exclude": [{ "glob": "apps/*/server.ts", "why": "the container entry point" }]
  }
}

• The floor only rises. Under 95 it may be nothing but what the tree measures: more than 1.5 points over it is red until the floor moves up. No history is read — an app with no git is held the same.
• exclude is for code a unit test cannot execute — an island's browser mount (its test runs a built chunk), a container entry point. Each entry carries its why; x doctor prints them all.
• A floor, not the goal. Coverage counts execution, not validation: 100% is reachable with zero assertions. A test added to raise it is proven by mutation — break the source, watch it go red, restore. A covered branch whose test cannot fail is worse than an uncovered one.
• Cost: none measured. The step on examples/dummy (53 unit files): 7.5 s before, 7.5 s after (As of 2026-10-01).
• Same number on every machine. The suite runs as fixed slices of 16 neighbouring test files, one plain bun test each; the lcov parts are folded here. Not bun test --parallel --coverage: Bun's own cross-worker merge read one tree as 59.9% at four workers and 65.0% at two.
• Functions are counted per source file as the best one slice reached — Bun's lcov carries no function names to union. A directory's tests share a slice, so tests beside their source count together.
• Under --shard a slice cannot hold a floor: it is green on coverage, carries its facts in data.coverage.unit.facts, and x verify merge judges the fold → CI.
• In --json a judged run carries data.coverage.unit — { "lines", "funcs", "files" }. The human render prints the same line under the step with --verbose, and always when it is red.
• A .tsx is counted in its COMPILED lines — a Bun inaccuracy, As of 2026-10 (Bun 1.4.0). Bun reports a plugin-loaded file under its source path with the output's line numbers and totals, and every rendered .tsx is compiled by a plugin. So Uncovered Line #s for a .tsx is not a place in the source, and a view whose tests assert every branch can still read under 100%. Read the percentage, ignore the line list, and keep logic that needs a line-accurate answer in a .ts module beside the view. The minimal case: docs/history/bun-coverage-plugin-lines.md.

The framework holds itself to the same constant: bun run coverage gates each package and scripts/ alone, one process each (scripts/coverage-gate.ts).

x verify

The single gate. Green means shippable.

21 steps, one list, in cost order. The gate is this command with no flag — "green" has to mean the same thing for everyone. x verify --only <step>[,<step>…] runs the named steps — several in one process — for an iteration loop and says NOT A GATE RUN in the summary and in --json (data.notAGateRun), writing no floor file; there is no --skip, because a knob that removes a step from a run still calling itself the gate is the one thing this command must not offer. A step with nothing to check reports as skipped (-), never as passed.

The list is defined once, as VERIFY_STEP_NAMES in packages/cli/src/verify-step.ts. This table is a hand-synced copy of it (Contributing).

Any failure fails the gate; flakes are failures.

$ x verify
  ✓ typecheck  ✓ lint  ✓ boundaries  ✓ filesize  ✓ package-shape  ✓ errors
  ✓ unit  ✓ contract  ✓ live  ✓ job  ✓ exercise  ✓ e2e  ✓ eval
  ✗ drift
      X_DB_DRIFT: schema differs from migrations
        cause: table "posts" has column "publish_at" not present in any migration
        fix:   x db gen "add publish_at"

A step has a deadline, and a hung one fails by name.

The declared numbers are a 4-core CI runner's. x verify measures the machine once, before any step — its cores against 4, a short single-threaded probe against the runner's — and multiplies every declared deadline by that pace: each step's above, Bun's 5 s per test (--timeout), and a test's own { timeoutMs } or paced(ms) (ULTIMATE_TEST_PACE). A faster machine is pace 1, never a tighter number. A number in x.verify.json is taken as written. --json carries the reading as data.pace — { factor, cpus, probeMs } — and a timed-out step whose deadline was stretched says by how much.

The gate says where it is. In either render, one line per finished step on stderr — {"step":"lint","ok":true,"ms":21987} — so a cancelled CI job's log ends on the last step that finished, and a still running: unit (12m 3s) heartbeat every minute a step runs, so a harness that kills on silence never kills a slow but healthy step. Under --json stdout stays the one document.

x verify --json emits the same content machine-readably → MCP and AI. CI runs exactly x verify — no bespoke pipeline steps, because a check that lives only in CI is a check developers cannot run.

Errors

Full list: Error codes.

Rules

• Never mock the database. Clone it.
• Never assert on wall-clock time. Advance the frozen clock.
• Never let a test reach the network unmocked — it fails by design.
• A flaky test is deleted or fixed the day it flakes. There is no retry: 3.
• Hunting an order flake: x test unit --filter <file> -- --rerun-each=30 --randomize — the -- hands the rest to bun test.
• Every framework package ships at least 2 tests that would catch a real regression. No expect(true).toBe(true).
• Tests live next to their source as <file>.test.ts.
• A denial test per policy branch, not one happy-path test per action.
• Every island declares its states, and the states worth declaring are the ones you cannot click to. What the pictures are then judged against: Interface rules.
• Assert on error codes, never on error message text.
• CI runs x verify and nothing else.