Observability

One instrumentation bus, OpenTelemetry semantic conventions, and OTLP spans, logs and metrics — all in @ultimat3/core, all dependency-free.

As of 2026-11 (31.0). Stable API — semver from here (Upgrading).

Every observer — the OTLP exporters, the semconv histograms, x dev's recorder, the /_x panels, an app's own code — is a subscriber on one bus. No package keeps a set*Observer, set*Sink, single-exporter slot or recent*() ring buffer of its own: the boundaries step refuses one (X_INSTRUMENT_SLOT_FORBIDDEN).

The instrumentation bus

import { logger, subscribe } from '@ultimat3/core';

// Every slow statement, sampled trace or not — the "subscribe to every slow query" recipe.
const off = subscribe('db.statement', ({ payload }) => {
  if (payload.durationMs > 250) logger.warn('db.slow', { sql: payload.text, ms: payload.durationMs });
});
off(); // the unsubscribe; calling it twice is a no-op

The events

INSTRUMENT_EVENTS is the table as a value. Payload types are exported from @ultimat3/core; every timed event carries durationMs, and error on failure (the value the caller receives). An emitter reports a failed operation too.

One render.isr.regenerate per generation, never per request that joined it. One outbound.request per call, retries inside it. A storage.* call a driver makes on itself is not counted twice; a driver held directly, never resolved through defineStorage, reports nothing.

Semantic-convention names

Span attribute keys come from SEMCONV (no string literal), metric names from SEMCONV_METRICS. The legacy http.method, http.status_code, http.target, db.statement and llm.* attributes are gone (As of 31.0). Framework-specific attributes are ultimate.*.

An app's own events

Augment the map, then emit with the same instrument() — one bus, typed end to end:

import { counter, instrument, subscribe } from '@ultimat3/core';
import type { Money } from '@ultimat3/money';

declare module '@ultimat3/core' {
  interface InstrumentEventMap {
    'billing.charged': { readonly orgId: string; readonly amount: Money };
  }
}

const charges = counter('billing_charges_total', { unit: '{charge}' });
subscribe('billing', ({ payload }) => charges.add(1, { currency: payload.amount.currency }));
instrument('billing.charged', { orgId: 'org_1', amount: { minor: 1200, currency: 'EUR' } });

A name is lowercase dot-segments; the framework's first segments (INSTRUMENT_EVENTS) are taken.

Exporting: OTLP spans, logs and metrics

The container boot (startOtlpExport) installs one exporter per signal an endpoint names, each a bus subscriber with its own close-phase flush, and releases them on stop:

Every boot, collector or not, also subscribes subscribeSemconvMetrics(): db_client_operation_duration_seconds (from db.statement), job_duration_seconds (from job.perform) and gen_ai_client_token_usage (from ai.llm) — on /metrics and over OTLP alike. OTLP/HTTP JSON over fetch, no dependency; _HEADERS and _PROTOCOL are read too. gRPC on :4317 is refused with X_OTLP_PROTOCOL_UNSUPPORTED and will stay refused — HTTP/2 plus protobuf is a dependency and a second wire format, so point the collector's HTTP receiver at :4318. Scraping metricsText() remains equally supported.

Proof, As of 2026-11-09: packages/cli/src/otlp-export.live.test.ts sends one request through an action that queries Postgres and reads, off the collector's wire, an HTTP server span with the db.select client span beneath it in one trace, semconv names only.

Span cost. An unsampled span costs 3.3 µs (root) / 2.0 µs (child), against a 1 µs target (scripts/bench/results/span-overhead.json, As of 2026-11-09): ids and clock reads are still paid before the sampling decision is known. Known gap, pinned by scripts/bench/span-overhead.bench.test.ts.

Instruments

import { counter, gauge, histogram } from '@ultimat3/core';

const signups = counter('signups_total', { unit: '{signup}', description: 'completed signups' });
const inflight = gauge('inflight_requests');
const latency = histogram('external_call_seconds', { unit: 's', bounds: [0.1, 0.5, 1, 5] });

signups.add();                            // value defaults to 1
signups.add(3, { plan: 'pro' });
inflight.record(12);
inflight.add(-1);
latency.record(0.42, { host: 'stripe' });

DEFAULT_HISTOGRAM_BOUNDS is 0.005 0.01 0.025 0.05 0.075 0.1 0.25 0.5 0.75 1 2.5 5 7.5 10 — seconds, so an HTTP or RPC duration needs no bounds of its own.

Defaults when omitted: unit: '1', description: ''.

Dotted OTel names survive OTLP but not a Prometheus scrape, and the autoscaler reads the scrape — which is why the name rule is ^[a-z_][a-z0-9_]*$ and not a suggestion.

Attributes

Readonly<Record<string, string | number | boolean>>. Every distinct attribute set is its own series, so an attribute carrying an id or a raw path is an unbounded-cardinality bug, not a richer metric.

Reading them

import type { MetricExporter } from '@ultimat3/core';
import { collectMetrics, exportMetrics, startMetricExport, subscribe } from '@ultimat3/core';

declare const myExporter: MetricExporter;

collectMetrics(); // a MetricCollection: { at, resource, metrics }
const release = subscribe('metric.export', (event) => myExporter.export(event.payload)); // the seam
exportMetrics(); // one `metric.export` tick; nothing is collected while nobody subscribes
const stop = startMetricExport(60_000); // periodic; the timer is unref'd. Default 60s

Temporality is cumulative. An exporter is a metric.export subscriber on the instrumentation bus, like the span and log exporters, released by its own unsubscribe; configureMetrics({ clock, enabled }) is configuration only — its exporter option left in 31.0.0 (Upgrading). memoryMetricExporter() is the test double. resetMetrics() restores the defaults and clears the series while keeping declarations — test-only.

The Prometheus body

import { METRICS_CONTENT_TYPE, METRICS_PATH, metricsText } from '@ultimat3/core';

METRICS_PATH;          // '/metrics'
METRICS_CONTENT_TYPE;  // 'text/plain; version=0.0.4; charset=utf-8'
metricsText();         // the whole scrape body, from a fresh collectMetrics()

The body always opens with a target_info gauge carrying the service identity, then per metric a # HELP, a # TYPE, and its points. A non-trivial unit is appended to the help text as <description> (<unit>). Histograms render cumulative _bucket series including le="+Inf", then _sum and _count. Labels are sorted alphabetically; \`, "and newline are escaped;+Inf/-Inf/NaN` are spelled out.

/metrics is mounted by every role, on its own port — METRICS_PORT, default 9090 — not on the role's HTTP port. startMetricsEndpoint in packages/cli/src/metrics-endpoint.ts starts first and stops last, so x dev and production behave identically.

The series autoscaling reads

Declared in runtime-metrics.ts, so the role table, the Helm chart and the scrape cannot drift apart.

SCALING_METRICS maps each ScalingSignal to its series: rps → http_requests_total, ws-connections → connections, queue-demand → queue_demand; singleton, run-once and per-database map to null, because a role pinned at one replica has nothing to scale on. rps is derived from the monotonic counter — a rate is not a series, so nothing exports one. queue_demand is what the worker must hold: running on a serving worker, plus due rows on an unpaused queue. queue_depth is what waits, and stays for the backlog alert: 10 jobs held read queue_depth 0 and queue_demand 10, and an HPA on the first shrank the fleet mid-job. Every serving worker publishes the same queue_demand, so the adapter rule is max by (queue) (queue_demand), never sum; a draining worker publishes 0, since its loop has stopped and a frozen value would win that max for its whole retire. JobDriver.demand() is required of every driver; Redis keeps no worker registry, so it counts every running row.

Recorders: recordRequest({ method, route, status, durationMs }), recordConnection(delta), recordQueueDepth(queue, depth), recordJob(queue, outcome), recordLeaseLost(queue).

All five are wired As of 2026-08, each in the package that owns the event:

What every process reports about itself

Declared in process-metrics.ts and packages/db/src/pool-gauge.ts, started by the scrape listener — so every role that can be scraped answers them, worker and scheduler included. As of 2026-10.

• Cost: one unref'd timer wake a second — 0.12 millicore measured, on top of the 1.8 an idle Bun process spends — and ~12 µs per reading (process.memoryUsage() + process.cpuUsage()); a scrape reads the gauges at collection time.
• No GC count, no heap object count. Bun exposes neither cheaply: bun:jsc's heapStats() runs a full collection to answer — 9 ms on a process holding only the framework.
• The pool series are derived from demand. Bun.SQL publishes no occupancy, so client.ts counts every statement from send to settle and every pin from ask to release: demand up to max is in use, the rest is waiting. db_pool_max - db_pool_in_use is the headroom; there is no idle count.
• queue_wake_live (gauge, worker only): 1 once a notification has crossed the worker's LISTEN session, 0 while jobs start on the poll alone — a transaction-pooling proxy disables the wake without an error.
• What grows? process_resident_memory_bytes rising with process_heap_used_bytes flat is memory outside the heap (process_external_memory_bytes, or the allocator); both rising is a retained structure.

The one series a sync node adds

Declared in packages/realtime/src/socket.ts, not in runtime-metrics.ts: that file is the series every Ultimate process emits and the chart scales on, and this one exists only where channels do.

No labels, on purpose. A topic is client-chosen — topic() admits any [A-Za-z0-9_-]+ segment — so a per-topic label is unbounded series one socket can mint. The topic goes in the log line instead: channel.frames_dropped at warn, with { topic, dropped, total }.

It is a data-loss counter, not a saturation one. Nothing replays a channel frame: a topic has no cursor and no re-snapshot, so a frame backpressure refused is gone → Realtime. Alert on any non-zero rate. The live-query path is the one that repairs, and its drops are invisible here by design. SocketRegistry.droppedChannelFrames is the same number in process, for a test or a benchmark that cannot scrape.

CPU autoscaling is wrong for sync and worker: a node holding 80k idle sockets is near-zero CPU and near-capacity, and a worker blocked on a slow HTTP call is idle CPU with a growing backlog. That is why the two custom series exist → Deployment.

The database, on demand

x db diagnostics --json (As of 2026-11) answers what the series cannot: who blocks whom (data.blockers.rows[].blockedBy), what each session is doing, the slowest statements (pg_stat_statements, when an operator installed it), unused and invalid indexes, NOT VALID constraints, dead tuples and the last autovacuum. Facts only, exit 0; /readyz stays the one-select 1 probe. Read-only, 2 s per statement, no user table read, literals printed as ?; whoever runs it sees the redacted SQL of every session on the app's database role → CLI reference. The same report is dbDiagnostics() from @ultimat3/db.

Spans and logs, in detail

Browser failures

app.config.ts clientErrors: { enabled: true } (As of 30.0, off by default) sends a failure on an app/ page to the same ErrorReporter a server failure reaches, as source: 'browser' and code X_CLIENT_ERROR. site/ documents never carry it; the admin ships no script.

Real-user timings

clientErrors.vitals: { enabled: true } (As of 2026-11, off by default) measures what a user waited for, with no library: PerformanceObserver in the browser, one beacon per page view.

Error groups

errorGroupsSink (@ultimat3/admin/error-groups, As of 2026-11) is an ErrorReporter that folds every report into its errorFingerprint group in x_error_groups — first seen, last seen, count, a redacted sample — and /admin/errors lists them (errorGroupsPages, permission errors:read).

// apps/web/runtime.ts
import { errorGroupsSink, postgresErrorGroupsStore } from '@ultimat3/admin/error-groups';
import type { RuntimeOverrides } from '@ultimat3/cli/serve';
import { job, t } from '@ultimat3/jobs';

// The app's own: which tracker, which title. Enqueued once per NEW group, fleet-wide.
export const openErrorIssue = job({
  input: t.object({
    fingerprint: t.string,
    code: t.string,
    source: t.string,
    operation: t.nullable(t.string),
    message: t.string,
    firstSeen: t.string,
  }),
  idempotencyKey: ({ fingerprint }) => `error-issue:${fingerprint}`,
  tenant: 'none',
  retry: { attempts: 3, backoff: 'exponential' },
  async run({ input }) {
    console.log('new error group', input.fingerprint);
  },
});

export const errorGroups = postgresErrorGroupsStore();
export const runtime = {
  errorReporter: errorGroupsSink({ store: errorGroups, onNewGroup: openErrorIssue }),
} satisfies RuntimeOverrides;

Running an app for real — the PaaS → Compose → Kubernetes ladder, secrets, dashboards, datastore sizing, DR and runbooks — is the operations manual: docs/ops/, and specifically docs/ops/03-observability.md. Recommendations only; the framework depends on none of it.