# MCP and AI

The differentiator. Not a chat widget, not an "AI SDK integration" — the framework is built so an agent can read it, drive it, and verify its own work, and so the apps it generates have the same property.

`As of 2026-08`. Stable API — semver from here ([Upgrading](/upgrading)). The MCP registry, wire protocol, dev-tool catalog, read-only SQL guard, and action projection are built, and so are the four that used to be contracted: `llm()` is an action factory ([`packages/ai/src/llm.ts`](https://github.com/developerz-ai/ultimate/blob/main/packages/ai/src/llm.ts)), prompts are versioned, `postgresVectorStore()` fuses pgvector cosine with Postgres FTS via RRF, and evals gate on a committed baseline inside `x verify`'s `eval` step.

## Built-in MCP dev server

`x mcp serve` is the dev-tool server, over stdio or HTTP; `x dev` does not start it. Point any MCP client at it and the agent stops guessing. A new app's `.mcp.json` already names it — [Client config](#client-config).

The whole catalog, spelled exactly as each tool must be called; `bunx x mcp tools --json` reads it off the server rather than a count here. No aliases; renaming one is a major.

| Tool | Introspects / does | Replaces the agent's usual guess |
|---|---|---|
| `routes.list` | the `x routes --json` table: pages (render mode, hydrate, offline, budget), then every action, query and `runtime.ts` route | grepping a router directory |
| `schema.describe` | entities with columns, types and invariants | reading migration files in order |
| `policies.list` | every `policy`: permission, subject, where it is enforced | "is this endpoint protected?" |
| `actions.describe` | every action **and query**: input/output schema, policy, cache tags, MCP exposure | reading `api/` by hand |
| `jobs.inspect` | job definitions, retry policy and steps; omit `name` for all | reading `jobs.ts` and guessing the retry |
| `queue.depth` | pending, running and failed counts per queue | tailing a worker to see if it keeps up |
| `manifest.read` | the whole `x.manifest.json`, as text | ten separate reads |
| `errors.explain` | `X_*` code → cause, exact fix command, docs URL | web search |
| `graph.query` | `{ node }` (a file path, a primitive name or a node id) → every route, action, query, job, MCP tool, published operation, test and budget a change to it reaches, each with the provenance of the edges that reached it — `x impact`'s answer. `{ from, to }` → the chain of edges linking two nodes — `x graph --from --to`'s. An unknown name is `X_GRAPH_NODE_UNKNOWN` naming the nearest three; any other combination of arguments is an `isError` naming the two shapes. Rebuilt per call, so it answers for the code as it is now | grepping importers before an edit, and missing the route that reads it through a query |
| `db.query` | **read-only** SQL, 100-row default and 1000-row maximum, `EXPLAIN` on request | inventing a query and hoping |
| `db.migrate` | apply pending migrations **in a branch DB only** | mutating the dev database |
| `tests.run` | run the suite or a substring filter, structured results | parsing terminal output |
| `verify.run` | the whole gate; `fix: true` applies safe autofixes | guessing whether the work is shippable |
| `logs.tail` | last N structured log lines, filterable by runtime role | scrollback archaeology |
| `ui.shot` | photograph one route at a named viewport (`phone`/`tablet`/`desktop`, or `width`+`height`) in `light` or `dark` (`theme`: `prefers-color-scheme` is emulated **and** the scheme is stored as the visitor's choice under `THEME_STORAGE_KEY` before navigation, so an app with `theme.defaultMode: 'dark'` still captures light when asked — `As of 2026-09-19`), against the running `x dev`; returns the PNG **path** and the same verdict `x shot` writes — console, page errors, refused requests, whether every island mounted. Refuses a route with no `budget.js` | a hand-written puppeteer script, and a picture nobody judged |
| `ui.island` | `x shot --island <name> [--state]` as a tool: every declared state photographed and judged | the same, per component |
| `ui.inspect` | DOM, computed-style and accessibility facts for up to 20 selectors in **one** navigation of a route: per match the tag, text (≤200 chars), box, visibility, attributes (≤20), the computed `styles` you name (≤32, kebab-case; the rest land in `droppedStyles`) and with `a11y: true` the browser-computed role and name; plus `title`, `data-theme`, the focused element, the island count, console and page errors. ≤25 matches per selector, 64 KB on the wire (matches are dropped from the LAST selectors first, `truncated: true`); an unparsable selector answers `valid: false`. Takes the same PNG and verdict as `ui.shot` under `inspect/`; same budget gate, same `theme` semantics (the reported `data-theme` is the one asked for, on a dark-default app too) | `page.evaluate` scripts written per question, and a browser launched per question |
| `ui.interact` | drive one route through at most 12 steps — `{click}`, `{type: {selector, text}}` (≤500 chars), `{press}` (a key chord), `{focus}`, `{wait: ms ≤ 5000 \| selector}` (waited visible) — then photograph it (`fullPage` defaults to **false**: a dialog is judged on the fold) and, with an `inspect` block, read the same facts `ui.inspect` reads, in **one** navigation. Same `theme` semantics as `ui.shot`. After every step the islands settle again and one poll interval passes for the CSS transition; per step you get `{ index, kind, ms, navigated, url }`. Refuses whole, never trims: a list over the bounds (`X_UI_INTERACT_STEPS_INVALID`), typing into `<input type="password">` before any keystroke (`X_UI_INTERACT_SECRET_FIELD`), a step that leaves the dev server's origin (`X_UI_INTERACT_LEFT_APP`; a same-origin navigation is allowed and reported `navigated: true`), a step the driver refused (`X_UI_INTERACT_STEP_FAILED`, `meta.step` names it). PNG and verdict under `interact-<hash of the steps>/`, never over `ui.shot`'s; `ok` is the verdict's, except that a navigation a step caused is not the redirect the verdict fails a capture for | a puppeteer script per scene, and a picture of the closed state |
| `ui.diff` | compare two PNGs the other `ui.*` tools wrote — `before`/`after` relative to the app root and confined to `.x/shot/` (`X_UI_DIFF_PATH_OUTSIDE` otherwise, symlinks resolved) — pixel by pixel: `changedPixels`, `changedPercent` (two decimals), `changedBox` (the bounding box, or `null`) and the path of a diff PNG written beside `after` (`diff-<hash8 of before>.png`, or `out`): the `after` capture faded to a quarter over grey, changed pixels solid red. A pixel is changed when any channel moved by more than `threshold × 255` (default `0.1`); no anti-alias detection. Two sizes are `X_UI_DIFF_SIZE_MISMATCH`, a missing file `X_UI_DIFF_FILE_MISSING`. **No browser, no dependency** | a pixelmatch install, or two pictures eyeballed |

| Class | Tools | Exposure |
|---|---|---|
| read | `routes.list`, `schema.describe`, `policies.list`, `actions.describe`, `jobs.inspect`, `queue.depth`, `manifest.read`, `errors.explain`, `graph.query`, `ui.diff` | scope `dev:read`, unrestricted in dev — `ui.diff` reads files, and sits here because it reads only `.x/shot/` (resolve + prefix check, symlinks resolved) and launches nothing |
| gated read | `db.query`, `logs.tail` | scope `db:read` / `dev:logs` |
| executes code | `tests.run`, `verify.run`, `ui.shot`, `ui.island`, `ui.inspect`, `ui.interact` | scope `dev:test`; all six declare `destructive: true`, so none is metered as read chatter — the four `ui.*` tools launch a browser |
| write | `db.migrate` | scope `db:migrate`, **branch environments only** |

None of them is exposed in `ROLE=web`. `db.query` accepts one statement, whose leading keyword must be `SELECT`/`WITH`/`EXPLAIN`/`SHOW`/`TABLE`/`VALUES` — necessary, never sufficient. Batches, any write keyword at statement level (a data-modifying CTE included), locking clauses (`FOR UPDATE`/`FOR SHARE`), `EXPLAIN ANALYZE`, and whole function families matched by prefix of the called name, quoted and schema-qualified spellings included — file access (`pg_read_*`, `pg_ls_*`, `lo_*`, `dblink`), locks (`pg_advisory_*`), session settings (`set_config`), sleeps (`pg_sleep*`), catalog writes (`pg_import_*`) and the functions that run a query handed to them as text (`query_to_xml*`, `cursor_to_xml*`, `table_to_xml*`, `schema_to_xml*`, `database_to_xml*`, `ts_stat`, `ts_rewrite`) — are **refused**, not discouraged — `X_MCP_QUERY_REJECTED`, enforced before the host sees the string. What the scan cannot read is refused, never decoded: a Unicode-escaped quoted identifier (`U&"…"`) is `X_MCP_QUERY_REJECTED` (`As of 2026-10`). A keyword is matched as Postgres lexes an identifier, so a column `set2` is a column, not `SET`. Its Postgres SELECT-only role is conditional on the connection's own rights; the answer's `guards` array names the defences that engaged. `db.migrate` refuses a target that is not a branch database — `X_MCP_NOT_BRANCH_DB`.


### Client config

`x new` writes one client config, `.mcp.json` at the app root — the format several MCP clients read — and no per-vendor directory: a second copy is a second thing to drift.

```json
{
  "mcpServers": {
    "myapp": {
      "command": "bunx",
      "args": ["@ultimat3/cli", "mcp", "serve", "--transport", "stdio"]
    }
  }
}
```

| Client wants | Give it |
|---|---|
| `.mcp.json` | nothing: it reads this file |
| a command and its arguments | `bunx @ultimat3/cli mcp serve --transport stdio`, started in the app's root — the command needs an app (`X_NOT_IN_APP` otherwise) |
| a URL | `bunx x mcp serve --transport http --port 9229`, then `http://127.0.0.1:9229/mcp` and the bearer token it prints |
| its own config file | the same command, in that file's own syntax; the framework generates none |

**`bunx @ultimat3/cli`, never `bunx x`, in a client config.** A client may spawn the command outside the installed app, and there `bunx x` fetches an unrelated npm package named `x`. Inside an app's shell, `bunx x` resolves the app's own CLI. `scaffold-mcp-json.test.ts` pins the spelling, and CI's `scaffold-smoke` spawns the file's own command and asks it `initialize` and `tools/list` on every push.

### Rate limits on the HTTP transport

**Enforced `As of 2026-08-24`.** `mcpHttpRoute` meters itself, per caller, per class, per minute. It has to: all MCP traffic is one `POST /mcp`, so the HTTP pipeline's own buckets — which key on the route — cannot tell an `initialize` handshake from a migration.

| | Default | Spent by |
|---|---|---|
| `read` | 120 / minute | every method that is not `tools/call`, plus any tool that does not declare `destructive: true` |
| `write` | 20 / minute | a `tools/call` naming a `destructive: true` tool, and **any call this server cannot resolve** — fail-closed, so a probing client never gets the cheap bucket |
| `unauthenticated` | 20 / minute, per **address** | a missing token, or one `resolveToken` answers `null`. Past it the address is refused `429` **before** `resolveToken` runs — a valid guess included, so the answer says nothing about the token (`As of 2026-10`). Keyed on `handle(request, { address })` — the host's resolved client address; a host that passes none is not metered here |

| Rule | Detail |
|---|---|
| Metered after the parse, before the tool | the class comes out of the body, so it cannot move above the JSON parse. An unauthenticated caller is answered `401` four lines earlier and can never spend an actor's allowance |
| The key is the ACTOR | `mcp:<class>\|actor:<id>`, and it never reaches the caller: a 429 is provokable by anyone holding a valid token, so an actor or org id in one is a leak wearing a throttle's clothes |
| Refusal | `X_MCP_RATE_LIMITED`, 429, with `Retry-After`. Its **own** code and not `X_RATE_LIMITED`, because that one's `fix:` names the HTTP pipeline's buckets, which do not govern this route — a fix line that runs and changes nothing is worse than none |
| The knob | `mcpHttpRoute({ rateLimits: { read, write, unauthenticated } })`, or `defineAppMcp({ rateLimits })` |
| Behind N replicas | pass `rateLimitStore: postgresRateLimitStore({ executor })` as well. The default is a per-process memory store — honest for `x mcp serve`, and a lie for N replicas behind one URL, each enforcing the full allowance on its own |


### Protocol revisions: modern and legacy, one endpoint

`As of 2026-11`, read from the spec on 2026-11-02. Every server is **dual-era**: the era is decided per request, never per connection.

| Request | Era | Answered |
|---|---|---|
| `params._meta["io.modelcontextprotocol/protocolVersion"]` in `MCP_MODERN_VERSIONS` (`2026-07-28`) | modern, stateless | `server/discover` and the seven shared methods. Every result carries `resultType: 'complete'` and `_meta["io.modelcontextprotocol/serverInfo"]`; the lists, `resources/read` and `server/discover` add `ttlMs: 0`, `cacheScope: 'private'`. No `initialize`, no `ping` |
| no `_meta` version, or one in `MCP_LEGACY_VERSIONS` (`2025-11-25`, `2025-06-18`, `2025-03-26`) | legacy | `initialize` echoes an offered legacy revision, else answers `2025-11-25`; `ping`; the seven shared methods |
| any other `_meta` version | — | `-32022` `X_MCP_PROTOCOL_UNSUPPORTED`, `data: { supported, requested }` |

`MCP_PROTOCOL_VERSION` is gone (`As of 2026-11`): name the era — `MCP_MODERN_VERSIONS[0]`, or `MCP_LEGACY_VERSIONS`. `resources/templates/list` is `{ resourceTemplates: [] }` on both eras. A resource not found, hidden or absent, is one answer: `-32601` legacy, `-32602` modern. Not served and not advertised: `subscriptions/listen`, multi-round-trip input, the tasks extension, `x-mcp-header`, SSE responses; every `listChanged` is `false`.

Over HTTP:

| Request | Answer |
|---|---|
| `Origin` present and core's `proveSameOrigin` cannot show it is this app: not the route's public origin, not listed in `http.cors.origins` (CSRF's, the sync upgrade's and `/_x`'s rule) | `403` `X_MCP_ORIGIN_REFUSED`, **before** `resolveToken` runs. No `Origin` (a client process) passes. A page rebound to this server's address IS same-origin and passes here: `x mcp serve` refuses a non-loopback Host first (`421` `X_MCP_HOST_REFUSED`), and `x new`'s development token is honoured only with no `Origin` at all |
| modern, `MCP-Protocol-Version` / `Mcp-Method` / `Mcp-Name` (`tools/call`, `resources/read`, `prompts/get`) absent or unequal to the body (`=?base64?…?=` decoded first) | `400`, `-32020` `X_MCP_HEADER_MISMATCH` |
| `MCP-Protocol-Version` naming no served revision | `400`, `-32022` |
| modern, a method this server does not answer | `404`, `-32601` |
| `GET` or `DELETE` on the endpoint — `x mcp serve` and an app's own mounted `/mcp` alike | `405` `X_METHOD_NOT_ALLOWED`, `Allow: POST` — `@ultimat3/http`'s, the pipeline's own |
| `x mcp serve`: a Host that is not `localhost`, `*.localhost` or a loopback address | `421` `X_MCP_HOST_REFUSED`, before anything else — `/_x`'s rule |

## Every action is an MCP tool

```ts
mcp: { expose: true, description: 'Publish a draft post' },
```

That line is the entire integration. From the existing declaration:

| MCP requirement | Source |
|---|---|
| tool name | the action's export name, **verbatim** — `publishPost`, never `publish_post`. The one name `tools/call` accepts, `scopes:` is keyed on, and every published catalog spells. `As of 2026-08`: `openapi.json`'s `x-ultimate.mcpTool` and `describe().mcp.tool` published a snake_case name until then, and an agent that trusted either called a tool the server answers ToolNotFound for |
| JSON Schema for input | the `input` schema (Standard Schema → JSON Schema, via `introspect()`) |
| output schema | `output` — published as the tool's `outputSchema` when its root is an object ([below](#what-a-client-is-told-beyond-the-schema)) |
| description | `mcp.description`, ending `Declared refusals: <codes>.` when the primitive declares `errors:` — the same sentence on the MCP tool and the in-app agent's tool, on both protocol eras, so an agent plans around a refusal before it calls ([Actions](/actions#declared-refusals-errors)). `As of 2026-11` |
| title | `mcp.title` — omitted, none is published |
| annotations | derived from the kind, `mcp.annotations` overrides key by key ([below](#what-a-client-is-told-beyond-the-schema)) |
| **authorization** | the action's `policy` — unchanged, unwrapped, identical |
| idempotency | `idempotent: true` adds one optional argument, `idempotencyKey` (string, 1–255), to the tool's `inputSchema`. `tools/call` removes it from the arguments and passes it to `invoke` as the key — the `Idempotency-Key` header's MCP twin, filed under action, caller and key — so a retry with the same key replays the first result and the same key with other arguments is `X_IDEMPOTENCY_CONFLICT`. The action's input never sees it. An argument, not `params._meta`: a model-driven client cannot set `_meta`, and the retry that needs the key is the agent's |
| audit trail | the same OTel span and log line as an HTTP call |

The projected tool's `run` **is** `invoke` — the same entry point the HTTP route calls (`packages/mcp/src/projectable.ts`). Policy runs inside it, so there is nothing to keep in sync. The projection itself adds **no** MCP scope: a second gate hard-coded into the projection would sit in front of the only gate that matters, and the two would eventually disagree. `defineAppMcp`'s `scopes:` map may still attach one from outside — a property of the connection's token, never invented by the projection.

No MCP-specific permission table, no service account with broad rights. Exposure is opt-in; silence exposes nothing.

The user's own agents can therefore operate the user's product — refund an order, re-run an import, publish a post — with the exact permissions that user has in the UI. See [Admin dashboard](/admin-dashboard) and [Actions](/actions).

### What a client is told beyond the schema

`As of 22.10`, MCP 2025-06-18. Every piece is optional for the app; the defaults come from the
declaration already written.

| Field | Where it comes from | Default |
|---|---|---|
| `initialize` → `instructions` | `defineAppMcp({ instructions })` — a string, or `(caller) => string \| undefined` for per-population advice (the same shape as `surface`) | none sent. A function that throws or answers blank sends none; the handshake still answers |
| tool `title` | `mcp: { title }` on the action/query; `title` on a hand-written tool | none |
| `annotations.readOnlyHint` | the kind | query `true`, action `false` |
| `annotations.destructiveHint` | the kind | action `true` — the spec's own default made explicit: the framework cannot tell an insert from a delete. Declare `false` for an additive write |
| `annotations.idempotentHint` | `idempotent: true` on the action | `false` |
| `annotations.openWorldHint` | `mcp.annotations` only | not published (only the author knows a write sends mail) |
| `outputSchema` | an action's `output` whose root is an object; a query's declared `rows` as `{ rows: [<row>] }` — or the row itself for a `single: true` read, which answers one row or `X_NOT_FOUND` as its route does (`As of 2026-10`) | none — text only |
| `structuredContent` | every successful call of a tool that publishes `outputSchema`: the serialized answer read back (a `Date` is its string). A LIST query's rows sit under `rows`; a `single: true` query's answer is the row object itself, with no wrapper | absent |

```ts
mcp: {
  expose: true,
  description: 'Tag a post. Adds, never removes.',
  title: 'Tag post',
  annotations: { destructiveHint: false },
},
```

An `outputSchema` is **structure only** — `type`, `properties`, `required`, `items`, `enum`,
`const`, `anyOf`. A client validates `structuredContent` against it and refuses the call on a
miss, so a bound, a pattern or `additionalProperties: false` on rows read from the database would
be a promise nothing enforces on the way out. The meta tools publish their own hints:
`list_resources` / `describe_resource` read-only, `manage_resource` a write (it reaches every
grouped tool); `manage_resource` answers the inner tool's `structuredContent` byte for byte.

**Split by effect** (`surface: 'meta-split'`, `As of 2026-10-18`): `read_resource` (queries,
`readOnlyHint: true`), `write_resource` (`mcp: { effect: 'write' }`) and `delete_resource`
(`mcp: { effect: 'delete' }`, `destructiveHint: true`; `{ resource, action, params, live,
confirmed_by_user }` — without `live: true` it answers a preview (the framework's runs nothing; an action declaring `live` answers its own), with it it runs
only on `confirmed_by_user: true`) replace `manage_resource`, for clients and directories that
refuse a dispatcher which both reads and writes. A grouped write without an effect fails boot
(`X_MCP_EFFECT_UNDECLARED`); an action called on the wrong dispatcher is `X_MCP_WRONG_TOOL`
naming the right one; an unconfirmed commit is `X_MCP_USER_CONFIRMATION_REQUIRED`. Full contract:
[`packages/mcp/README.md`](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/README.md).

**What the model reads is compact.** A tool's text block is one-line JSON (the 2-space form until
22.10 spent a third of a large answer on indentation). `list_resources` answers **plain text**, one
line per resource and one per action, each fact said once where it is cheapest (`As of 2026-10`):

| Rule | Rendered |
|---|---|
| a scope every action of a resource shares | once, on the resource line: `(scope posts)` |
| a scope every action of one KIND shares | once, per kind: `(query scope posts:read; action scope posts:write)` |
| any other scope, or an unscoped action among scoped ones | on the action itself: `(scope …)` |
| kind | `(action)` only, from `destructive`; untagged = read-only query — never inferred from the name |
| `confirms: true` | `(confirms)`, always on the action |
| params | every required field, then optional ones to 4 fields / 100 chars, then `…`; an enum past 8 literals ends `\|…` |

```text
2 resource(s). Run one with manage_resource({resource, action, params}); describe_resource({resources:["<name>"]}) has the full input schemas. Untagged = read-only query; (action) may write; (confirms) waits for a human; … = more in describe_resource.

org (scope app:use) — The organisation
  transferOrg (action) {} — Transfer the org

posts (query scope app:use; action scope posts:write) — Blog posts
  listPosts {status_eq?: string|number|boolean, title_cont?: string, sort?: "createdAt"|"-createdAt", …} — Posts, filtered
  publishPost (action; confirms) {postId: string} — Publish a draft post
```

**A catalog too big for that: `catalogDetail: 'compact'`**, once, in `defineAppMcp` (`As of 2026-11`). Each resource line stays; its actions become ONE line of names, grouped by kind tag, with `confirms` and an unhoisted scope still on the action. Descriptions and parameter hints move to `describe_resource`, which the header tells the model to read before a resource's first call. Measured on a 46-resource, 322-action catalog: 95,318 characters full, 13,085 compact. Refused on a server no caller reads `list_resources` from (`X_MCP_SURFACE_INVALID`).

```text
posts (query scope app:use; action scope posts:write) — Blog posts
  listPosts; action: publishPost (confirms), archivePost
```

The same catalog as data, for a test: `mcp.server.catalog(caller)` — every action keeps its own
`scope` there. `describe_resource` stays compact JSON with the full schema — a schema is JSON.

**Hold the standing surface to a number.** `tools/list` rides every step and `list_resources` sits
in the transcript all session:

```ts
import { assertMcpSurfaceBudget } from '@ultimat3/mcp';

test('the staff surface stays small', async () => {
  // The derivation beside the number: the size measureMcpSurface reported, × a margin.
  await assertMcpSurfaceBudget(mcp.server, staffCaller, { toolsList: 4_000, listResources: 12_000 });
});
```

`measureMcpSurface(server, caller)` returns `{ toolsList, listResources, instructions }` in
characters, off the wire; the assert refuses every surface over its ceiling in one
`X_MCP_SURFACE_OVER_BUDGET`.

**Refusals speak to the caller.** `defineAppMcp` renders an error's `callerFix` — what a remote
agent can do — where the framework's own `fix` names a command only the app's developer can run:
`X_FORBIDDEN` says to ask the account owner for the permission, not `x policy explain`; a missing
scope says to ask for a token that carries it; an invalid argument says to correct the field
against the published schema. The developer's `fix` stays in the log line, `--json`, and on
`mcpServer` (the dev server), whose default is `errorAudience: 'developer'`;
`defineAppMcp({ errorAudience: 'developer' })` restores it for an app. **A 5xx cause is the
server's own business** (`As of 2026-10`): for the caller audience, a thrown 5xx code core's
`hasPublicCause` does not list (`X_DB_STATEMENT_FAILED` carries the Postgres message and the
statement) renders a fixed sentence as its `cause` and `x errors explain <code> --json` (or its
`callerFix`) as its `fix` — in a tool result and in a `resources/read` error's `data` alike, the
verdict a production problem document gives. The developer audience keeps both. An app's own error takes the
same two fields, plus `docs` — declared once with `defineError`:

```ts
import { defineError } from '@ultimat3/core';

export const RefundWindowClosed = defineError<{ readonly ageDays: number }>({
  code: 'X_REFUND_WINDOW_CLOSED',
  title: 'the refund window has closed',
  status: 403,
  cause: (p) => `the payment is ${p.ageDays} days old; refunds are limited to 90 days`,
  fix: 'x policy explain refund:create --json',
  callerFix: 'issue a credit note instead',
  docs: 'docs://recipes/issue-a-credit-note',
});
```

A `docs` other than the framework's one Error-Codes page is a fourth line of the tool result —
`  docs:  docs://recipes/issue-a-credit-note` — and is the `docs` member of the problem document
over HTTP. A production problem document carries `callerFix` as its `fix`; a dev one keeps the
developer's.

### The app's own endpoint is mounted by the web role

`As of 2026-09-05`. `defineAppMcp({ …, resolveToken })` builds `mcp.route`, and `app.config.ts`
declares `ai: { mcp: { expose: true } }` **by default** (the path is `defineAppMcp`'s own, `/mcp` by default) — and until this date nothing
between the two served it: `POST /mcp` was `X_ROUTE_NOT_FOUND` under `x dev` and in every
container. The contract is one file per app directory:

| Piece | Where | Rule |
|---|---|---|
| the declaration | `apps/<app>/mcp.ts` | `export const mcp = defineAppMcp({ include: 'exposed', resolveToken })` |
| the switch | `app.config.ts` → `ai.mcp` | `expose` (default `true`). The path is each endpoint's own `defineAppMcp({ path })` (default `/mcp`) |
| several app directories | every `apps/<app>/mcp.ts` mounts (`As of 2026-11`) | `apps/web/mcp.ts` first — its first endpoint is the default — then the rest sorted. An admin surface's `apps/admin/mcp.ts` stands beside the web one; until 2026-11 the first sorted file mounted ALONE, so it silently took `/mcp` away. Two files claiming one route: `X_MCP_PATH_DUPLICATE` naming both |
| when it is built | at the top level of `apps/<app>/mcp.ts` | the boot scan evaluates that file after every `apps/*/{site,app,api,shared}` module has registered, so `include: 'exposed'` sees every exposed primitive. An endpoint projected earlier — in a module under `apps/<app>/app/` that `mcp.ts` re-exports — is `X_MCP_INCLUDE_EARLY` at the mount, naming the tools it lacks, never a short catalog. Never import the API index to fill the registry (`X_BOUNDARY_APP_TO_API`) |
| the mount | `x dev` **and** `runRole`, through one call | `POST <path>` → `mcp.route.handle(request)`; the boot log says `app mcp mounted`, `x dev`'s summary prints `mcp POST /mcp` |
| auth | the route's own | `auth: 'public'`, `enforcedBy: 'handler'` on the http route — the bearer token is read by `resolveToken`, never pre-judged by the pipeline |
| exposed and unmountable | `X_MCP_APP_UNMOUNTED`, logged once | no file exports `mcp`, or it was built without `resolveToken` (no `route`); the fix names the file to write |
| a new app | `x new` writes it | `apps/web/mcp.ts` over the `@<app>/mcp` catalog, `resolveToken` from `apps/web/app/auth/agent-token.ts`: an API key (`issueApiKey`, `x_api_keys`) is an agent in every environment; `dev-agent.<role>` is one under `x dev` alone, and only from a loopback address with no `Origin` (a page rebound to 127.0.0.1 sends one). No token, or one it does not know: 401 |
| the admin surface | `x new` writes it too | `apps/admin/mcp.ts`: `adminMcp({ app: admin, path: '/admin/mcp', include: 'exposed', actor })` — the admin's own tools and the app's exposed ones, the actor from the same `resolveAgentToken` (handed the request's `facts`). A key with scopes is held to its `admin:*` ones, a key with none is as strong as its actor → [Admin dashboard](/admin-dashboard#an-admin-surface-serves-the-apps-tools) |

**A token from an external issuer** (`As of 2026-11-05`): an app whose agents present an IdP's JWT
rather than an API key hands `defineAppMcp` `jwtBearerResolver` from `@ultimat3/auth` — the twin of
`apiKeyResolver`. The signature, issuer, audience and time checks are `verifyWorkloadToken`'s; the
app keeps only its claim check and its actor:

```ts
// apps/web/app/auth/agent-token.ts
import { actorFromService, jwksClient, jwtBearerResolver } from '@ultimat3/auth';

const ISSUER = 'https://login.example.com/tenant-a/v2.0';
const TENANT = 'tenant-a';
const ORG_ID = '00000000-0000-4000-8000-000000000001';
const keys = jwksClient({ provider: ISSUER, jwksUri: `${ISSUER}/discovery/v2.0/keys` });

export const resolveAgentToken = jwtBearerResolver({
  keys: () => keys,                       // read per token: no network at import
  issuers: [ISSUER],
  audience: 'api://ledger',
  requiredScope: 'ledger.agent',          // optional; the scope name is the app's
  accept: (raw) => raw['tid'] === TENANT, // the app's own claim, after the signature held
  actorOf: (token) => actorFromService({ ...token.identity, orgId: ORG_ID }),
});
```

Every refusal is `null` — the mount's `401` — and one `auth.jwt_bearer.refused` line naming the
check (`signature`, `issuer`, `audience`, `expiry`, `not-before`, `scope`, `accept`, `key-set`),
never a claim's value. A key set that cannot be read and an unknown `kid` both fail closed.

#### Several endpoints, one per population

`As of 22.14.0`. `mcp` may be a non-empty **array** of `defineAppMcp` values — one endpoint per
population, each with its own catalog, `instructions`, `groups`, `scopes` and `oauth`. Same export
name, same file; the order is the contract:

```ts
// apps/web/mcp.ts
export const mcp = [
  defineAppMcp({ name: 'acme', include: 'exposed', scopes: CUSTOMER_SCOPES, resolveToken: customerToken, oauth }),
  defineAppMcp({ name: 'acme-admin', path: '/mcp/admin', tools: staffTools, surface: 'meta', groups, scopes: STAFF_SCOPES, resolveToken: staffToken, oauth }),
  defineAppMcp({ name: 'acme-afiliados', path: '/mcp/afiliados', tools: affiliateTools, scopes: AFFILIATE_SCOPES, resolveToken: affiliateToken, oauth }),
];
```

| Rule | Detail |
|---|---|
| endpoint #0 | mounts at its own `defineAppMcp({ path })` (default `/mcp`), exactly as a single export does, and owns the root `/.well-known/oauth-protected-resource` |
| every other endpoint | mounts at its own `defineAppMcp({ path })` (default `/mcp`), route name `mcp:<path>` |
| isolation | each endpoint is its own `McpServer`: a tool, resource or prompt of one is absent from another's `tools/list` and answers `-32601` there |
| two endpoints on one route | `X_MCP_PATH_DUPLICATE`, **thrown** at boot — a second endpoint that forgot `path` collides with the default `/mcp` |
| one endpoint without `resolveToken` | `X_MCP_APP_UNMOUNTED` names it; the others still mount |
| boot report | one `app mcp mounted` log line and one `mcp POST <path>` summary line per endpoint; `x dev --json` carries `mcp` (endpoint #0) and `mcpPaths` (all) |

The same discovery serves an app's `RuntimeOverrides`: `apps/<app>/runtime.ts` exporting `runtime`
reaches `x dev` (its middleware, rate-limit store and plain `routes`) and `runRole` when
`apps/web/server.ts` passes none — one middleware chain in development and in the container, not two.
`runtime.routes` (`As of 22.6.0`) is the one escape hatch for a URL no primitive projects to and a
wire format none speaks — an OAuth token endpoint answering RFC 6749 JSON to a form-encoded POST —
mounted after the framework's routes and before the pages, through the whole pipeline.

### OAuth discovery for remote connectors

`defineAppMcp({ oauth })` makes the endpoint an OAuth 2.1 protected resource per the MCP
authorization spec (`As of 2026-11`, served at modern `2026-07-28` and the legacy handshake
revisions `2025-11-25` / `2025-06-18` / `2025-03-26`) and RFC 9728 (`As of 22.6.0`). Discovery is
not the same in each: RFC 9728 protected-resource metadata is required from `2025-06-18` on;
`2025-11-25` made the 401's `WWW-Authenticate` optional with a `.well-known` fallback (SEP-985)
where `2025-06-18` required it; `2025-03-26` defines none, its discovery being RFC 8414
authorization-server metadata at the origin:

```ts
export const mcp = defineAppMcp({
  include: 'exposed',
  scopes: MCP_SCOPES,
  resolveToken,
  oauth: {
    authorizationServers: ['https://www.example.com'], // issuer(s); https, or http on localhost
    resourceName: 'Example',
    // resource: 'https://www.example.com/mcp',   // omitted: the request's PUBLIC origin + path
    // scopesSupported: [...],                     // omitted: Object.keys(scopes)
  },
});
```

| What | Answer |
|---|---|
| 401 (no token) | `WWW-Authenticate: Bearer realm="ultimate-mcp", resource_metadata="<origin>/.well-known/oauth-protected-resource/mcp"` |
| 401 (token did not resolve) | the same plus `error="invalid_token"` |
| `GET /.well-known/oauth-protected-resource/mcp` and `GET /.well-known/oauth-protected-resource` | `{ resource, authorization_servers, bearer_methods_supported: ['header'], scopes_supported, resource_name }`, public, `access-control-allow-origin: *` — mounted by both boots beside `POST /mcp` |
| several endpoints | each serves its own path-inserted document (RFC 9728 §3.1) — `/.well-known/oauth-protected-resource/mcp/admin` for `POST /mcp/admin` — with its own `resource` and `scopes_supported`, and its 401 names that document; the root document stays endpoint #0's |
| origin | `ctx.https` + host as the pipeline resolved them, so a TLS-terminating ingress yields `https://…` |

The authorization server is `mcpAuthorizationServer` (`As of 2026-11`): `@ultimat3/auth`'s
`authorizationServer` ([Auth](/auth#authorization-server)) bound to this endpoint. One call, one
consent page:

```ts
// apps/web/mcp-oauth.ts
import { postgresAuthServerStore } from '@ultimat3/auth';
import type { Actor, PgExecutor } from '@ultimat3/core';
import type { McpScopes } from '@ultimat3/mcp';
import { defineAppMcp, mcpAuthorizationServer } from '@ultimat3/mcp';

declare const executor: PgExecutor;
declare const MCP_SCOPES: McpScopes;             // the scope → tool names map, as above
declare function agentFor(subject: string): Promise<Actor | null>;

export const as = mcpAuthorizationServer({
  issuer: 'https://www.example.com',
  resource: 'https://www.example.com/mcp',
  store: postgresAuthServerStore({ executor }),
  consentPath: '/oauth/consent',
  lifetimes: { accessSeconds: 600, refreshIdleDays: 30 },
  scopes: Object.keys(MCP_SCOPES),
  registration: 'dynamic',                    // omitted: metadata-document client ids only
  actor: (grant) => agentFor(grant.subject),  // per call: a removed member is refused at once
});
export const mcp = defineAppMcp({ include: 'exposed', scopes: MCP_SCOPES, oauth: as.oauth, resolveToken: as.resolveToken });
// apps/web/runtime.ts: routes: [...as.routes]
```

| What | Answer |
|---|---|
| `resource` | every grant binds to it, a request naming none included; `resolveToken` refuses a token bound to anything else, or to nothing |
| `oauth` | `{ authorizationServers: [issuer], resource, scopesSupported: scopes }` for `defineAppMcp` |
| audit | each server decision is one `mcp.auth.<event>` line — `issued`, `refused`, `revoked` at `info`; `code-replayed`, `refresh-replayed`, `grant-disabled` at `warn` |
| the protocol | none of it here: every rule is `authorizationServer`'s, so a hand-written server and plain `oauth:` keep working |

Without `oauth`, the 401 is `Bearer realm="ultimate-mcp"` as before. Bad block:
`X_MCP_OAUTH_INVALID` at definition.

### Per-caller tools and one token across tenants

`As of 2026-11`. Two `defineAppMcp` options, both resolved per request and both behind the same
gates as every other tool:

| Option | What it does | Refusals |
|---|---|---|
| `catalog: (caller) => ({ tools, groups? })` | tools that exist per caller (a tenant's connectors), each a `runtimeTool({ name, description, inputSchema, permission, run })` borrowing a **declared** permission; asked on every `tools/list` and `tools/call`, never cached | `X_MCP_CATALOG_TOOL_DUPLICATE` (a name taken twice), `X_MCP_CATALOG_UNRESOLVED` (the resolver threw, or answered something unservable) — the request fails, never a smaller list |
| `tenant: { arg, resolve, list }` | the token resolves to a person; every call names its tenant under `arg`, stripped before validation; `resolve` builds the call's actor, so policy and tenancy guards see one org per call; `list_tenants` lists them | `X_MCP_TENANT_REQUIRED` (no tenant named — fix `list_tenants({})`), `X_MCP_TENANT_UNRESOLVED` (the lookup threw), `X_MCP_TENANT_INVALID` (boot: a tool declares the arg) |

A tenant the person is not in (`resolve` answers `null`) gets exactly the answer a hidden tool gets.
Another caller's runtime tool is an absent tool. The audit line carries `tenant`.

**`tools/list` is person-scoped, and not a promise.** MCP's `tools/list` names no tenant, so `visibleTo` (function form), `can()` and `catalog` run there as the person; `tools/call` runs them as the tenant actor and is authoritative. A tool visible to the person but not the tenant actor is listed and answers as a hidden tool; one visible only through a tenant role or a tenant's catalog row is unlisted and still runs when called by name.

```ts
import { agentActor } from '@ultimat3/core';
import { defineAppMcp, runtimeTool } from '@ultimat3/mcp';

declare function roleIn(personId: string, org: string): Promise<string | undefined>;

defineAppMcp({
  include: 'exposed',
  tenant: {
    arg: 'org',
    resolve: async ({ person, value }) => {
      const role = await roleIn(person.id, value);
      return role === undefined ? null : agentActor({ id: person.id, orgId: value, roles: [role] });
    },
    list: () => [],
  },
  catalog: (caller) => ({
    tools: caller.actor.orgId === undefined ? [] : [
      runtimeTool({
        name: 'org_ping',
        description: 'Ping the acting org. Read-only.',
        inputSchema: { type: 'object', properties: {} },
        permission: 'org:read',
        destructive: false,
        run: ({ ctx }) => ({ org: ctx.actor.orgId }),
      }),
    ],
  }),
});
```

Worked example: Postly's `quote_<plan>` tools and its multi-org token —
`examples/dummy/packages/mcp/src/{quotes,tenants}.ts`, proven over its own endpoint by
`tenant.contract.test.ts`. Full rules: `packages/mcp/README.md`.

### Long tools: `jobTool()` and `job_status`

A tool whose work outlives a client's timeout. `tools: { reindex: jobTool({ job, policy, description, result, input?, jobInput?, inlineMs? }) }`.

| Fact | Rule |
|---|---|
| Durable | enqueues at once (`job.as(caller)`); never an in-process promise |
| Input | the tool's `input` (default: the job's); `jobInput(input, ctx)` builds the job's from it and the caller — a member id read off the request, never an argument an agent fills in |
| Inline | polls `introspect.job(id)` every 250 ms up to `inlineMs` (default 5,000, ceiling 25,000), honours `ctx.signal`. Also bounded by the poll count, so a clock that does not move still ends the wait. `inlineMs: 0` answers `running` at once — right for a model run |
| Answers | `done` + the `result` query's rows, read **as the caller** · `running` + `poll: { tool: 'job_status', arguments: { jobId } }` · `failed` + the code it was thrown with (`X_INTERNAL` for an uncoded throw, as HTTP answers one), never the row's text |
| `job_status` | projected once beside any job tool; no scope (gated per row); another actor, another org and an unknown id are the same `X_JOB_NOT_FOUND` |
| Dedupe | the job's own `idempotencyKey`: a repeated call attaches to the live run |
| Refused | `X_MCP_JOB_TOOL_INVALID`: not a `job()` handle, `inlineMs` out of range, a driver with no `introspect` |

The reference app's `draftReview` (`examples/dummy/apps/web/mcp-tools.ts`) queues `reviewDraftLater` for the caller's own member and reads `postReview` back; `mcp-tools.contract.test.ts` drives it over the wire.

### Recipes: `docs://recipes/<name>` and the `docs` tool

`defineAppMcp({ recipes: [{ name, title, description, steps: [{ tool, why, example? }], notes? }] })`.

| Fact | Rule |
|---|---|
| Checked at boot | every `step.tool` is in the final catalog (`X_MCP_RECIPE_TOOL_UNKNOWN`), every `example` passes that tool's input (`X_MCP_RECIPE_EXAMPLE_INVALID`) — all offenders in one boot |
| Served | an `McpResource` (`text/markdown`, rendered from the structure) and a read-only `docs({ name? })` tool |
| Visibility | only to a caller who can see every step's tool and holds its scope; hidden = unknown, byte for byte |
| The prefix | with `recipes:`, `resources:` may not hold `docs://recipes/` (`X_MCP_RECIPE_URI_RESERVED`) |
| Error links | `defineError({ docs: 'docs://recipes/<name>' })` must name a recipe some endpoint serves — `X_MCP_RECIPE_UNKNOWN` at mount; a throw-site link to none is dropped from the render and logged |

**From a markdown folder.** `recipes: await markdownRecipes({ dir: join(import.meta.dir, 'docs') })` reads one `*.md` per recipe and serves its body verbatim. The file name is the recipe name. `title:` and `description:` come from `---` frontmatter, else from the first `# ` heading and the paragraph under it. A fence tagged `call` holds one step, `tool {"arg": "value"}`, and is checked at boot like any `step.example`. A whole `"<postId>"` value is a placeholder, so any value passes at boot. A `docs://recipes/<name>` link in the text that names no recipe is `X_MCP_RECIPE_LINK_DANGLING`.

**Searching.** `docs({})` lists what this caller can follow. `docs({ name })` and `docs({ uri })` read one recipe. `docs({ q })` inlines the best match, scoring 3 for each word in the name or title, 2 in the description and 1 in the body. On a miss it names the nearest recipes, so a caller with any visible recipe never gets an empty answer.

**Per caller.** `extraRecipes: (caller) => recipes` adds the caller's own recipes, such as a tenant's runbooks. They are listed as `source: 'caller'` on `docs`, and on `resources/list` and `resources/read` to that caller only. They are not checked at boot. A recipe that names a tool the caller cannot call is not shown.

**Replayed.** In a contract test, `await replayRecipes(server, recipes, { caller, placeholders })` calls every step's example through the real server as `caller` and reads every link. Three answers fail it with `X_MCP_RECIPE_EXAMPLE_INVALID`: no such tool, refused arguments, and a policy that refuses that caller. A dangling link fails it with `X_MCP_RECIPE_LINK_DANGLING`. A tool's own refusal (`X_NOT_FOUND` for a sample id) counts as the tool working. The reference app replays its recipes as an org owner: `examples/dummy/packages/mcp/src/recipes-replay.contract.test.ts`.

### Consuming another server: `mcpTools()`

Remote MCP tools as this app's actions — the app's `policy`, rate limit, confirmation gate and manifest row; one tool list for an agent.

| Fact | Rule |
|---|---|
| Declared | `mcpTools({ name, url, auth, allow, policy, lock })` → `tools: Map<remoteName, Action>`, action names `<name>_<tool>`. `policy` is an app `resource:verb` permission, as a hand-written tool takes: the input is the remote's schema, so a predicate over it would decide nothing |
| Reviewed | `allow` is required; `lock` is the committed `tools.lock.json` written by `x mcp lock <name>`; `x mcp lock <name> --check` writes nothing and fails on drift |
| Drift | a missing, re-described or re-shaped tool is `X_MCP_REMOTE_TOOL_CHANGED` — at declaration against the lock, before the first call against the live server |
| Wire | `outbound()` (`dial: 'public'` by default, deadline, size cap); modern era first, then legacy `initialize` + `Mcp-Session-Id`; JSON or SSE answers |
| Untrusted | the answer arrives as `{ untrusted, isError }`, fenced `<untrusted source="mcp:<name>/<tool>">`, middle-cut past `maxResultChars`; invisible format characters stripped, then every `<` (and full-width `＜`, small `﹤`) written `&lt;` — no spelling of `</untrusted>` survives |
| Exposed | `expose` defaults to `false`: the actions are this app's, not its `/mcp`'s, so `include: 'exposed'` leaves them out; they are declared `agents: true`, so this app's own `agent()`s are offered them anyway. `expose: true` opts one remote server in, and this app's `/mcp` then answers for a third party's tools |
| Re-checked | the live list is held to the lock before the first call, on every `list_changed`, and again once a verdict is `reverifyMs` old (default 3,600,000) — a remote that changes silently is caught within the hour |
| Read-only | only when a reviewer sets `readOnly: true` in the lock — the remote's hint is never trusted |
| Failure | `X_MCP_REMOTE_REFUSED` (502), the cause names the status |

**Five outcomes of a call.** `meta.failure` tells them apart without reading a cause:

| Outcome | What the caller gets |
|---|---|
| success, text only | `{ isError: false, text }`. No `structuredContent` and no output schema is still a success |
| the tool's own `isError: true` | a RESULT `{ isError: true, text }`, handed to the model like any tool result |
| no answer (DNS, connect, deadline) | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'transport'`. An abort from the caller's own signal passes through as `X_ABORTED` |
| an HTTP status | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'status'`, `meta.status` |
| a JSON-RPC `error` | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'rpc'`, `meta.rpcCode` |
| a body that is no JSON-RPC answer | `X_MCP_REMOTE_REFUSED`, `meta.failure: 'malformed'` |
| `401` | `X_MCP_REMOTE_AUTH_REQUIRED`, `meta.resourceMetadata` from the `WWW-Authenticate` challenge (RFC 9728) when the server sent one |

**Streaming.** A server may answer any POST as JSON or as `text/event-stream`. The client reads the answer with its own id out of the stream, and acts on any `notifications/tools/list_changed` riding with it. A legacy (2025-era) session also gets one long-lived `GET` stream for the server's own messages. It is opened after `initialize`, detached from the request that opened it, and lives `listenMs` (default 5 minutes) before the next call opens another. A `405` means the server offers no such stream, and the client never asks again. `client.close()` ends it. A modern server is stateless, so its changes ride response streams only. Either way, a change makes the next call re-list and hold the list to the lock again.

**OAuth to the upstream.** `mcpOAuthClient({ clientId, clientName, redirectUri, scopes? })` is the client half of the MCP authorization spec:

1. `begin(serverUrl)` reads the server's RFC 9728 protected-resource document, then its authorization server's RFC 8414 metadata. It answers `{ authorizeUrl, handshake }`, with PKCE S256 from `@ultimat3/auth` and `resource` per RFC 8707.
2. Redirect the person to `authorizeUrl`. Keep `handshake` server-side, in the person's session.
3. On the callback, `exchange(handshake, { code, state })` answers `{ accessToken, refreshToken, expiresIn }`. `refresh(handshake, refreshToken)` answers the same shape.
4. Store the tokens per connection. Return the access token from `mcpTools({ auth })`, which is read on every call, so a rotated token is never cached.

`clientId` is the HTTPS URL at which this app serves `metadata()` (a client ID metadata document), or an id the authorization server issued when the app was registered by hand. A `401` mid-session is `X_MCP_REMOTE_AUTH_REQUIRED`: run the handshake again.

**Proved on itself.** `examples/dummy/packages/mcp/src/remote-self.contract.test.ts` points `mcpTools()` at the reference app's own MCP route. It locks the live list, calls `seatReport`, and meets three refusals: the remote policy's `X_FORBIDDEN` as a result, a stolen token as `X_MCP_REMOTE_AUTH_REQUIRED`, and a schema moved since the lock as `X_MCP_REMOTE_TOOL_CHANGED`.

## Three outcomes, deliberately different

Role, scope and policy refuse in three distinguishable ways. The difference is the security property, not an implementation detail.

| Situation | Response | Wire |
|---|---|---|
| The actor's role can never invoke the tool | absent from `tools/list`; a direct call answers ToolNotFound | JSON-RPC `-32601`, message `tool not found: <name> — call tools/list to read the catalog this caller may use`, no `data` at all. The hint is the same sentence on both branches — absent and hidden — so it instructs without saying which |
| The role could invoke it, but the connection's scope does not include it | explicit refusal naming the missing scope | JSON-RPC `-32600`, `data: { code: 'X_MCP_SCOPE_DENIED', scope, fix }` |
| The tool was invoked and the policy denied this input | `X_FORBIDDEN` with the denial reason | a normal `result` with `isError: true` — identical to the HTTP answer for the same call |
| The caller may see and call it, and the arguments fail its published schema | `X_INPUT_INVALID`, each issue addressed by path | a normal `result` with `isError: true` (`As of 22.10`; a `-32602` until then, which clients hide from the model). `-32602` remains for a call that is not one — no `params`, a non-string `name` |

Hidden means hidden: `Forbidden` on a hidden tool is an enumeration oracle — an agent, or an attacker driving one, walks a name list and reads the org's feature set, entity names and internal operations off the difference between "not found" and "forbidden". A scope refusal is the opposite case: a well-behaved client can legitimately fix it, so hiding it would only strand the caller.

| Rule | Detail |
|---|---|
| `visibleTo` takes two forms | a **role allowlist**, or a **predicate over the caller**. A tool that declares neither is visible to everyone |
| Both forms are **fail-closed** | a role list admits only the roles it names, so a caller whose role is not in it — including a caller with no role at all — is refused; a caller with no role sees only tools that declare no `visibleTo` |
| `tools/list` is per connection | answered per caller, never a static catalog |
| Visibility is input-independent | the predicate takes the caller and nothing else — it structurally cannot read call arguments, so existence cannot be probed by varying them |
| Gate order is fixed | visibility → scope → the policy's actor half → arguments → policy. Scope runs before the policy, so a refusal never depends on evaluating a policy against attacker-supplied input; a caller the policy refuses whatever they send gets the policy's `X_FORBIDDEN` (the tool's `admit`) before their arguments are checked, so an issue list never describes a tool they may not call; the rest of the policy — predicates over the arguments — runs after them |
| Every outcome is audited | one structured log line per `tools/call`: `surface: 'mcp'`, tool name, actor id, outcome. ToolNotFound, scope denials and policy denials log at `warn`, a successful call at `info` — ToolNotFound is `warn` on purpose, because an enumeration attempt is a detectable pattern |
| Audit lines carry no payload | tool name, outcome and error code only — never call arguments, never row data |
| No trusted-tool mode | there is no flag that skips policy evaluation, on any MCP surface |
| The actor cannot exceed the human | the actor is the signed-in user's session; an agent inherits exactly those permissions |

Where the first two outcomes are declared:

| Outcome | Declared | Property |
|---|---|---|
| Hidden (role) | `mcp: { visibleTo: [...] }`, on the action or query itself | `readonly string[]` — a role allowlist. A primitive declares the list form only: a declared fact stays static and serialisable. The predicate form of `McpVisibility` is for a surface that builds its catalog programmatically (`@ultimat3/admin` derives visibility from the actor's admin permissions) and hands `@ultimat3/mcp` a tool directly. Both are fail-closed: an unnamed role — including a caller carrying no role at all — gets ToolNotFound, never Forbidden. A catalog audience, not an authz rule; the primitive's `policy` still decides every call |
| Hidden (prompt) | `visibleTo` on an `McpPrompt` passed to `defineAppMcp({ prompts })` | the same `McpVisibility`, the same fail-closed evaluation: absent from `prompts/list`, and `prompts/get` answers exactly as for a prompt that does not exist (`As of 22.14.0`). Resources take the same field |
| Scope | `scopes:` on `defineAppMcp` | `Readonly<Record<string, readonly string[]>>` — scope name → tool names. A capability of the connection's token, so it is declared once per app rather than beside every primitive. A name the catalog does not contain, or one claimed by two scope entries, refuses at boot: `X_MCP_SCOPE_UNKNOWN`, `X_MCP_SCOPE_CONFLICT` |

Rationale for each: [`docs/architecture/11-ai-surface.md`](https://github.com/developerz-ai/ultimate/blob/main/docs/architecture/11-ai-surface.md).

## Generated facts, hand-written conventions

| Artifact | Author | Contents | Rule |
|---|---|---|---|
| `x.manifest.json` | **generated**, every build | routes, entities, actions, mutators, queries, jobs, tasks, policies, cache tags, MCP tools, budgets, build ID | never hand-edited; drift is a `x verify` failure |
| `openapi.json` | **generated** | HTTP surface from action/query declarations | contract diff in `x verify` |
| `AGENTS.md` | **human-authored**, short | project-specific conventions an agent cannot infer | never generated, never auto-appended |
| `CLAUDE.md` | **human-authored**, short | same, compressed-config style, <600 lines | never generated, never auto-appended |

LLM-generated context files measurably reduce task success. A model writing "here is what this codebase does" produces confident, plausible, partly-wrong prose, and the next agent treats it as ground truth — errors compound and cannot be distinguished from facts. So: facts come from code (structured, verifiable, regenerated every build), conventions come from a human (short, opinionated, stable). Ultimate never generates prose documentation at runtime, and `x new` scaffolds `AGENTS.md` as a terse human-editable stub, not an essay.

## LLM gateway

One typed entry point for model calls — provider-agnostic, observable, cached, evaluated.

```ts
export const summarize = llm({
  model: 'claude-sonnet-5-5',
  input:  t.object({ postId: t.uuid }),
  output: t.object({ summary: t.string, tags: t.string.array() }),
  prompt: summarizePrompt,                       // versioned artifact
  cache:  { semantic: { threshold: 0.97, ttl: '7d' } },
  budget: { tokensIn: 8000, costPerCall: { minor: 5, currency: 'USD' } },
  policy: can('post:read'),
});
```

| Feature | Behavior |
|---|---|
| Structured output | `output` schema drives tool-use/JSON mode; a parse failure retries once, then throws `X_LLM_OUTPUT_INVALID` |
| Streaming | `.stream()` on the returned action; policy, budget, cache and span unchanged; the `done` chunk carries the validated value |
| Cost + token accounting | per call, per tenant, per prompt version; exceeding `budget` throws before spending |
| Retries | typed on provider errors; rate limits back off, content refusals do not retry |
| Breaker | one per provider, on by default (`{ failures: 5, cooldownMs: 30_000 }`): a provider whose retries ran out five passes running is skipped, no call and no sleep, until one probe after the cooldown answers; a 400 never opens it. `providerGateway({ breaker: (provider) => breaker({ name: provider.name, failures: 3, cooldownMs: 10_000 }) })` chooses your own, `breaker: 'off'` declines. [Outbound HTTP](/outbound-http#breaker-stop-paying-for-a-dependency-that-is-down) |
| Caching | semantic cache from [Caching and invalidation](/caching-and-invalidation) |
| Provider prompt caching | the provider keeps a request's stable prefix; on by default for `agent()`, off for `llm()` → [Agents](/agents#provider-prompt-caching) |
| Tracing | one OTel span per call with model, tokens, cache hit, prompt version |
| Fallback | ordered provider list per model; the provider that answered is on the span as `gen_ai.provider.name`, never silent. A refusal names a more capable model for the declaration to adopt — models are never swapped at runtime |
| Money | `Money = { minor, currency }` — never a float → [Money](/money) |

**A stream is the same action over a different transport, with one difference that is forced rather than chosen: there is no repair turn.** The non-streaming call retries once on a parse failure; a stream cannot, because the tokens are already on the reader's screen and a second answer is two answers to one question. So a stream yields **unvalidated** text increments plus one final `done` carrying the value that did satisfy `output`, and a schema disagreement is `X_LLM_STREAM_INVALID` naming the non-streaming call as the fix. A structured schema wants the non-streaming call.

**Showing what a model wrote is `Markdown` from `@ultimat3/ui`**, never `innerHTML`: the text is
parsed to a tree and rendered as elements, so a `<script>` or a `javascript:` link in an answer is
text, and a streaming answer re-parses per token while finished blocks keep their nodes. Tool calls
and usage records are `JsonView`; a model's proposed edit against the original is `DiffView`
→ [UI components](/ui-components).

Long or multi-call chains are `job`s with steps, so a model call that fails on step 4 retries step 4 only. See [Jobs and workflows](/jobs-and-workflows).

## Agents, hives and agent jobs → [Agents](/agents)

`llm()` is one model call. A **tool-using loop** is `agent()`, and it is the same kind of thing — an action factory, not a ninth primitive. Everything the loop needs is a factory over a primitive already in the vocabulary:

| Factory | Over | Gives you |
|---|---|---|
| `llm()` | `action()` | one model call, streamable, semantically cacheable |
| `agent()` | `action()` | the turn loop: real actions as tools, forced structured output, `maxTurns`, a run budget, `ctx.signal`, `onTurn`. **No `.stream()`** |
| `hive()` | `action()` | one action fanned out over many inputs: bounded concurrency, split-order results, three-way `ok`/`failed`/`skipped` member outcomes |
| `agentJob()` | `job()` | an agent as durable, retried, cancellable queue work — `As of 2026-08` the only way an agent reaches a queue |
| `backfill()` | `job()` | the resumable, paced sweep those run over → [Migrations and backfills](/migrations-and-backfills) |

A tool is one of two things. A real `action()` carrying `mcp: { expose: true }` — the **same** predicate an external MCP client is filtered by, so an in-app agent and an external one are offered exactly the same catalogue of actions and authorize identically. Or an `inProcessTool({ name, description, input, execute })`: a tool that is this process's alone, with no `mcp` flag, in no MCP catalogue, route or `openapi.json`, running as `ctx.actor` under the agent's own policy → [In-process tools](/agents#in-process-tools). An `agent()` returns an action, so an agent is a tool of another agent with no supervisor primitive anywhere.

A tool in `mcpConfirmations({ tools })` waits for a person on **both** surfaces: `X_MCP_CONFIRMATION_PENDING` over `tools/call`, on a server handed `confirmations` or not, and as an agent's `tool_result`. A name no tool answers to is `X_MCP_CONFIRMATION_TOOL_UNKNOWN` at the `manifest` step. One list, no second option → [A tool that waits for a person](/agents#a-tool-that-waits-for-a-person).

**Quorum and signed decisions** (`As of 2026-11`) are two more options on the same factory, never a second gate:

| Option | What it holds |
|---|---|
| `approvers: ({ args }) => ({ quorum, from })` | N of M distinct approvers matching `'role:<name>'` / `'actor:<id>'`, resolved per call from the validated arguments and stored on the row. The requesting actor never counts (nor an impersonated decider): `X_FORBIDDEN`. One approver counts once. Short of the quorum, the agent's repeat answers `X_MCP_CONFIRMATION_QUORUM_PENDING`, whose `meta.outstanding` names the selectors still able to vote (never the requester or one who already voted); a bar nobody can meet is `X_MCP_CONFIRMATION_QUORUM_INVALID` before a row opens |
| `evidence: { verify }` | every vote carries `signature` over the `signingPayload` the view returns (confirmation id, tool, keyed argument digest), checked by the app's `verify` against keys registered to the approver. A missing or failing one stores no vote; the stored row is re-verified before the call runs, so a row written around the decision runs nothing. Both are `X_MCP_CONFIRMATION_EVIDENCE_INVALID` |

`verifyConfirmationEvidence(row, { evidence, quorum, from, arguments })` is exported so a separate process that executes the action re-checks the stored evidence with public keys alone. `ttlMs` expires a partly approved row like any other; one rejection rejects. The full table: [`@ultimat3/mcp` README](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/README.md#quorum-and-signed-decisions).

Full reference, including the at-least-once trap that makes every mutating tool's idempotency your obligation: [Agents](/agents).

## Prompts as versioned artifacts

```
apps/web/app/posts/prompts/summarize.v3.md      # the prompt, plain markdown + typed slots
apps/web/app/posts/prompts/summarize.evals.ts   # evals attached to it
```

| Rule | Why |
|---|---|
| A prompt is a file with a version, not a string literal | diffable, reviewable, attributable in traces |
| Editing a prompt requires a version bump | invalidates the semantic cache; keeps A/B honest |
| Slots are typed | a missing variable is a compile error, not a `{{undefined}}` in production |
| **Every prompt has an evals file** — no evals is a `x verify` failure | an unevaluated prompt is untested code |
| Old versions retained | traces stay interpretable; rollback is a config line |

## Vectors and hybrid search

pgvector in the same Postgres. No second datastore.

| Piece | Detail |
|---|---|
| Embeddings | an `Embedder` value, never an entity option: `remoteEmbedder({ name, dimension, baseUrl? })` speaks the `/v1/embeddings` shape (`EMBEDDINGS_API_KEY`), `hashEmbedder()` is the offline default |
| Writing | `indexDocument({ store, embedder, document })` chunks, embeds, upserts and prunes the stale tail; a sweep over many rows is the app's `backfill()` calling it per page |
| Index | HNSW: `postgresVectorStore({ name, dimension }).ddl()` prints the table and both indexes, pasted into a migration by the app |
| Hybrid search | one `query` primitive fusing pgvector cosine + Postgres FTS with Reciprocal Rank Fusion; weights are config |
| Filtering | tenant + policy filters applied **in SQL**, so vector search cannot leak across tenants; a store with no tenant bound, read inside a request acting for an org, is `X_VECTOR_UNSCOPED` — a backfill opts in with `scope: UNSCOPED` |
| Re-embed | `indexDocument` again: every chunk is embedded on each call, so a job decides when text changed |

## Evals as a test type

`eval` is one of the six test types in [Testing](/testing).

| Aspect | Detail |
|---|---|
| Shape | fixture set + assertions: exact, schema, rubric (LLM judge), or regression-vs-baseline |
| Agents | `defineEval({ agent, cases, tools })` runs the whole loop with each tool a double; an undoubled call is `X_EVAL_TOOL_UNDOUBLED` → [Testing](/testing#evals) |
| Determinism | temperature 0 where possible; judge model and prompt version pinned |
| Gate | `x verify` fails on a score drop beyond the declared tolerance, not on absolute score |
| Cost | reported per run; `x test eval --sample 20` for the fast local loop |
| Output | `--json` with per-case scores, so an agent iterating on a prompt sees which case it broke |

## Branch environments

The shipped surface is `x db branch`, with three verbs. `x branch` (no `db`) is **planned** and exits `X_NOT_IMPLEMENTED`; the build-id and MCP-socket rows below are what it will add.

```bash
x db branch create feat-new-billing --json
x db branch list --json
x db branch delete feat-new-billing --json
```

| Property | Detail | `As of 2026-08` |
|---|---|---|
| DB | `CREATE DATABASE "<source>_branch_<slug>" TEMPLATE "<source>"` copy-on-write clone — cheap, isolated, disposable. `<slug>` is the name with every character outside `[A-Za-z0-9_]` replaced by `_` — a hyphen is not legal in an unquoted Postgres identifier — so `create feat-new-billing` clones into `myapp_branch_feat_new_billing`. Embedded: a copied `pgdata-<name>` directory, which keeps the name **as typed** | **shipped** |
| Migrations | `db.migrate` applies here, never to the shared dev DB (`X_MCP_NOT_BRANCH_DB`) | **shipped** |
| Preview URL | `http://<name>.localhost:<PORT>`, reported on `data.preview` | **the URL is computed**; nothing routes that subdomain for you |
| Teardown | `x db branch delete <name>` — it may only delete what `list` shows | **shipped** |
| **Build ID scopes the SW** | a per-branch build id giving the branch its own SW scope and cache namespace, so a preview can never poison prod cache → [PWA and offline](/pwa-and-offline) | **planned**, part of `x branch` |
| Scoped MCP socket | `ws://localhost:9229/<name>` | **planned**, part of `x branch` |
| Agent use | an agent can migrate, seed, test, and browse a preview without risking anything shared | |

## `--json` everywhere

Every command and every error has a machine-readable form. Same content, different encoding.

```
$ x verify --json
{"ok":false,"checks":[{"name":"budgets","ok":false,"failures":[
  {"route":"site/pricing","metric":"js","actual":"61kb","limit":"40kb",
   "cause":"chart.js via shared/ui/button.tsx",
   "fix":"x fix boundary site/pricing/page.tsx"}]}]}
```

| Surface | Machine form |
|---|---|
| CLI | `--json` on every subcommand |
| Errors | `UltimateError` serializes to `{ code, cause, fix, docs }` |
| HTTP errors | same JSON body, same codes |
| Dev overlay | the identical string a terminal shows |
| MCP | tool errors carry the same code + fix |

## Errors

| Code | Cause | Fix |
|---|---|---|
| `X_MCP_TOOL_UNKNOWN` | no visible tool answers that name (role-hidden and absent are indistinguishable) | `tools/list` to read the catalog this caller may use |
| `X_INPUT_INVALID` | arguments failed the tool's published `inputSchema` — an `isError` result naming each field (`As of 22.10`) | correct the named fields against `inputSchema` (`tools/list`, or `describe_resource` on the meta surface) and resend |
| `X_MCP_ARGS_INVALID` | not raised on the wire since 22.10 — invalid arguments answer `X_INPUT_INVALID` as a tool result; `McpArgsInvalidError` stays exported | — |
| `X_MCP_SURFACE_OVER_BUDGET` | `assertMcpSurfaceBudget` measured `tools/list`, `list_resources` or `instructions` over the budget declared for them | shorten what it names, group tools behind `surface: 'meta'`, or raise the budget with the measured size and the reason in the same diff |
| `X_MCP_IDEMPOTENCY_KEY_SHADOWED` | an `idempotent: true` action's input declares `idempotencyKey`, the tool argument reserved for the retry key — refused at boot | rename the action's input field |
| `X_MCP_SCOPE_DENIED` | the connection's token does not carry the tool's scope | reconnect with a token whose scopes include the one `cause` names — the app's `resolveToken(token)` is what returns them — or drop that scope from `defineAppMcp({ scopes })`. Scopes are fixed for the life of a connection, so a grant takes effect on the next one. **Not** `x token grant`: that command is `PLANNED` and exits `X_NOT_IMPLEMENTED` |
| `X_MCP_SCOPE_UNKNOWN` | `defineAppMcp`'s `scopes:` names a tool the server does not project | spell the name as one of the tools the server actually projects, or drop it from that `scopes` entry |
| `X_MCP_SCOPE_CONFLICT` | two `scopes:` entries claim the same tool | keep the tool under the single scope a token must hold for it, and remove the other entry |
| `X_MCP_QUERY_REJECTED` | `db.query` was not given one read-only statement | send exactly one **read-only** `SELECT`/`WITH`/`EXPLAIN`/`SHOW`/`TABLE`/`VALUES` — a data-modifying CTE is not a read |
| `X_MCP_NOT_BRANCH_DB` | `db.migrate` pointed at a database that is not a branch | `x db branch create <name>   # then retry db.migrate` — point the host at the database the create reported (`DATABASE_URL=…/<source>_branch_<slug>`). The target is read from the database's own name, so a shared one can never pass |
| `X_MCP_PROTOCOL` | malformed envelope, a JSON-RPC **batch** (an array — this server answers one request per message and never walks one), or an unsupported method — a client bug, not an authz outcome | send a JSON-RPC 2.0 body, one request per `POST /mcp` (one per line over stdio); the `-32600` carries `data: { code, fix }` naming which |
| `X_MCP_BODY_TOO_LARGE` | one message over `bodyLimitBytes` (HTTP, `413`) or `lineLimitBytes` (stdio) — on a JSON-RPC `-32600` with `data: { code, cause, fix, limit }` | send less in one message, or raise the cap where the route is built: `mcpHttpRoute({ bodyLimitBytes })` / `defineAppMcp({ bodyLimitBytes })` / `serveStdio({ lineLimitBytes })` |
| `X_MCP_RATE_LIMITED` | this caller spent its per-minute allowance for the request's class ([above](#rate-limits-on-the-http-transport)) | wait out the `Retry-After`, or raise it where the route is built: `mcpHttpRoute({ rateLimits })` / `defineAppMcp({ rateLimits })`. Never `X_RATE_LIMITED`'s buckets — they do not govern this route |
| `X_FORBIDDEN` | the action's policy refused this actor — identical to the HTTP denial | over an app's MCP: the caller's fix — ask the account owner or an administrator for the permission; a retry is refused the same way. As the app's developer: `policies.list` for the permission this tool enforces, then grant it to the actor's role in `apps/web/shared/policies.ts` |
| `X_LLM_OUTPUT_INVALID` | model output failed the `output` schema twice | tighten the prompt or widen the schema; bump the prompt version |
| `X_AGENT_TOOL_UNEXPOSED` | an `agent()` lists an action that is not MCP-exposed and is not an `inProcessTool()` — refused at **declaration** | `inProcessTool({ … })` for a tool only the agent may call, or add `mcp: { expose: true, description }` to the action, or drop it from `tools` |
| `X_AGENT_MAX_TURNS` | an `agent()` used every turn without answering | tell the template when to stop and answer through the respond tool, then bump its version |
| `X_HIVE_EMPTY` | a `hive()` split into zero members | return at least one member input, or guard the call site |
| `X_NOT_IMPLEMENTED` | a remote driver stub was reached | configure the local/PGlite driver, or wait for the release named in `fix` |

Full list: [Error codes](/error-codes). CLI surface: [CLI reference](/cli-reference).

## Rules

- One authz system. An MCP call and an HTTP call reach the same `policy` with the same actor resolution.
- One name. A projected tool is called the primitive's export name, verbatim, in `tools/list`, in `scopes:`, in the LLM tool list, in `openapi.json` and in `describe()`. There is no second spelling to derive.
- Exposure is opt-in per action; the projection carries no scope of its own — `defineAppMcp`'s `scopes:` map may still attach one, from outside the primitive.
- Visibility is fail-closed and computed per connection. A hidden tool answers ToolNotFound, never Forbidden.
- Gate order is visibility → scope → arguments → policy, and every outcome is audited. There is no trusted-tool mode.
- Write tools are branch-scoped. The dev server is never reachable in `ROLE=web`.
- Facts are generated every build; conventions are hand-written and short.
- Never generate prose documentation at runtime.
- Every command and every error has a `--json` form; budgets throw before spending, in `Money`.
- Every prompt is a versioned file with an evals file.

Source: [`packages/mcp/src`](https://github.com/developerz-ai/ultimate/blob/main/packages/mcp/src)