Observability

Undercroft ships an opt-in observability layer: structured logs, a Prometheus /metrics endpoint, and OpenTelemetry (OTLP) trace export. It is built to preserve the project’s stance:

• Off by default. A standard build carries none of the telemetry dependencies and no runtime overhead — the layer only exists when you compile with --features telemetry.
• Local-first / no phone-home. Nothing leaves the process unless you explicitly point it somewhere: /metrics is served only when you ask, and OTLP export happens only when UNDERCROFT_OTLP_ENDPOINT is set. Set means set to an endpoint — a declaration that names none refuses to start rather than exporting nothing silently, since a collector you configured and never receive spans from is the harder failure to notice.
• Metadata only. Every signal is a count, a rate, a latency, or an aggregate gauge. Drawer content, drawer names beyond what stats already exposes, and key material are never emitted. The security LEVEL does not narrow that further: a sealed vault’s wing and room names travel like any other, to the authorized subscriber that asked for them (see the live stream below).

The full opt-in pipeline — every edge exists only when its gate is set, and every signal is metadata/counts only:

flowchart LR
    e["undercroft engine<br/><i>--features telemetry</i>"]
    e -- "UNDERCROFT_METRICS=1<br/>/metrics behind the palace bearer<br/>whenever one is declared" --> prom["Prometheus"]
    prom --> am["Alertmanager<br/><i>PalaceTamperDetected, chain stalls,<br/>latency, engine down, 5xx, auth spikes,<br/>embed / rerank / late-interaction failures</i>"] --> hook["webhook sink"]
    e -- "UNDERCROFT_LOG_FORMAT=json<br/>stdout" --> promtail["promtail"] --> loki["Loki"]
    e -- "UNDERCROFT_OTLP_ENDPOINT<br/><i>metadata-only spans, on the policed<br/>agent, root pinned by UNDERCROFT_OTLP_CA</i>" --> tls["tempo-tls<br/><i>Caddy terminator</i>"] --> tempo["Tempo"]
    e -- "SSE /v1/vaults/{id}/stream<br/><i>bearer + assertion</i>" --> monitor["Palace Monitor<br/><i>GET /monitor</i>"]
    prom --> graf["Grafana"]
    loki --> graf
    tempo --> graf

The control plane

The control plane has its own telemetry since 1.1.0 (undercroft-orchestrator --features telemetry), on a separate listener declared by UNDERCROFT_ORCH_METRICS_ADDR: its serving port must be reachable by tenants, so a /metrics path there would be exposed in every real fleet. Loopback needs no token; any other address refuses to start without UNDERCROFT_ORCH_METRICS_TOKEN. It exports four undercroft_orch_* counters — requests by route class, refused credentials by kind, rate-screen firings, engine-call outcomes — and a request-duration histogram by route class, and carries no tenant, vault or tenant-name label; per-tenant figures live on the admin plane. No scrape job or alert rules ship for it yet.

Building with telemetry

cargo build -p undercroft-cli --release --features telemetry

Without the feature the same binary runs identically, and hitting /metrics (if enabled) returns 503 with a hint to rebuild.

Structured logs

With the feature on, diagnostics become tracing events.

Prometheus metrics

UNDERCROFT_METRICS=1 undercroft serve-http --host 127.0.0.1 --port 8765
curl -H "Authorization: Bearer $UNDERCROFT_MCP_HTTP_TOKEN" \
     http://127.0.0.1:8765/metrics

/metrics is opt-in (UNDERCROFT_METRICS=1), served on the bind address (loopback unless you deliberately expose the server), and sits behind the same bearer token as the rest of the server. It is absent (404) when the flag is unset.

Exposed series (all undercroft_*):

• Counters — search_total{fusion}, search_prefiltered_total, search_wings_probed_total (how many per-wing indexes served one query’s candidates — the honest cost metric for anything fan-out shaped; a count, never a wing name), drawer_writes_total{outcome} (created / deduped / quarantined — a third VALUE on its one outcome label since 1.0.0, because a diverted write was counted as created on every write arm, which is a durable signal that is wrong rather than merely missing; the counter and the live frame are now emitted from one function so they cannot be classified differently), drawer_deletes_total, embed_failures_total{backend} (http / onnx / ort — an embed the embedder degraded to a ZERO VECTOR rather than failing the write; the drawer is stored verbatim and lexically findable but semantically invisible until re-embedded. The live count is embed_failures on every stats surface; this is its durable half, so a server nobody polls still has a series to alert on. A kind, never a model name. Since ROADMAP O150 a PANIC inside the in-process model counts here too — on all three model roles — where it used to end the process; its log line reads inference panicked: …), rerank_failures_total{backend} and late_failures_total{backend,side} (the other two model roles, ROADMAP O131 — a cross-encoder score degraded to 0.0, which SINKS that candidate in the reranked window rather than merely losing it, and a ColBERT encode degraded to an empty matrix, where side=doc is a durable hole at rest and side=query retires the late stage for one search. Separate series rather than one stage label, because the three failures cost different things), kg_writes_total{kind}, chain_commits_total (audit-chain RECORDS, not manifest anchors — a 256-drawer bulk transaction anchors once and advances this by 256, and records appended without an anchor, such as read-audit records, are counted by the next anchor), chain_replays_total (ROADMAP O250 — FULL audit-chain replays by the label guard, which authenticates every label a reader decides from. It is designed to run once per handle, re-running only when another connection commits, so a sustained rate here is not routine: it is a second writer moving PRAGMA data_version under a long-lived server, and each move costs the next guarded read a walk of the entire audit table. That was a +213% regression for a whole release, found by a reviewer reading code because this count lived in test builds only. The live half is chain_replays on every stats surface. No labels — a vault-shaped one has a value set created by use, and the per-vault figure is on /v1/…/stats), hmac_verify_failures_total{surface}, vault_opens_total, http_requests_total{route,status}, auth_rejections_total{kind}.
• Histograms — search_duration_seconds, search_hits, http_request_duration_seconds{route}.
• Gauges (per vault — and suppressed entirely when UNDERCROFT_ASSERTION_SECRET is declared, since /metrics addresses no single vault and would otherwise carry one vault’s counts to a caller who can assert only another’s; the per-vault detail is on /v1/…/stats, which is assertion-gated, and no alert depends on these) — drawers, audit_chain_height, plus kg_triples / kg_entities / store_bytes where sampled, and the five codebook generation counters — codebook_generation_pq_codebook, …_pq_ivf, …_fde_codebook, …_fde_ivf, …_tok_codebook. A step means every row coded against the previous generation was re-coded (or, for the IVF pairs, re-partitioned: the code bytes are unchanged and the candidate set moved). They sit outside HMAC coverage, so they are evidence about ambiguity in a retrieval result, never about tampering.

A gauge name must appear in undercroft_obs::GAUGE_NAMES or the value is dropped without a trace — write-only telemetry that looks live at the call site and never reaches /metrics. The list is public so a producer can pin the names it emits against the names actually registered.

hmac_verify_failures_total is the headline signal: any non-zero value means a record, KG triple, tunnel, or vault manifest failed HMAC verification — i.e. tamper was detected on read.

OpenTelemetry (OTLP)

Set an endpoint to export traces over OTLP/HTTP. Metrics stay on the Prometheus /metrics pull endpoint above — there is no OTLP metric push:

# Loopback cleartext is allowed — the collector never leaves the machine.
UNDERCROFT_OTLP_ENDPOINT=http://localhost:4318 \
UNDERCROFT_SERVICE_NAME=undercroft \
undercroft serve-http

For a collector on another host, TLS is required and there is no override — the headers this exporter sends are documented to carry a bearer token, and the spans carry vault ids and route labels:

UNDERCROFT_OTLP_ENDPOINT=https://collector.internal \
UNDERCROFT_OTLP_CA=/etc/undercroft/collector-ca.crt \
UNDERCROFT_SERVICE_NAME=undercroft \
undercroft serve-http

Spans cover each inbound request (a request root span per /v1 request and per MCP method call) and the search, save/dedup and KG-write operations, which nest under it when a request drives them. Export is synchronous and thread-based — the server itself stays fully synchronous, with no async runtime introduced.

The full stack (Grafana)

A ready-to-run stack lives in deploy/observability/ — a telemetry-built Undercroft server wired to the full operability picture: metrics (Prometheus), logs (Loki), distributed traces (Tempo), and alerting (Alertmanager), all rendered in Grafana.

cd deploy/observability
docker compose -f docker-compose.observability.yml up --build
# Grafana → http://localhost:3000  (dashboard: "Undercroft — Palace")

undercroft (telemetry) ──/metrics──▶ Prometheus ──rules──▶ Alertmanager ──▶ alert-sink
          │  │                          │                                    (webhook)
          │  └──JSON logs──▶ promtail ──▶ Loki ──┐
          └──OTLP traces────────────────▶ Tempo ─┤
                                                 └──▶ Grafana (+ image-renderer)

The dashboard surfaces request rate by route, search rate and p95/p50 latency, drawer writes by outcome (created / deduped / quarantined), audit-chain commit rate, HTTP 5xx and auth rejections, tamper broken out by surface, recent logs and traces, active alerts, and — front and centre — the HMAC-verify-failures stat that turns red the instant tamper is detected.

Alerting (Prometheus + Alertmanager)

Prometheus evaluates alerts.yml and pushes firing alerts to Alertmanager, which routes them to a receiver. The demo stack ships a tiny alert-sink webhook that logs every delivery, so the whole path is visible without external credentials — swap in Slack/email/PagerDuty in alertmanager/alertmanager.yml.

A firing tamper alert links straight to the tamper runbook — where it happened, and how to confirm, mitigate, fix, and prevent it.

Every rule preserves instance (each aggregation keeps it in its by (…) list, and the two rules that do not aggregate carry it through), so an alert names the process that is slow or erroring rather than reporting that somebody, somewhere, is — and Alertmanager’s inhibition (a critical silences warnings on that instance) has a label to compare on. That detail is load-bearing: a label missing from both sides of an equal: counts as equal, so scoping an inhibition by a label no rule emits silences the entire fleet instead of one host. The shipped config did exactly that, and the only symptom was an alert that never arrived. docker compose run --rm obs-config now evaluates the rules with Prometheus’s own promtool, asserts the exact label set each one emits, and fails if the inhibition equals on anything they do not all carry.

Logs & traces (metadata only)

With UNDERCROFT_LOG_FORMAT=json, promtail ships Undercroft’s structured logs to Loki; with UNDERCROFT_OTLP_ENDPOINT set, request/search/save/kg spans export to Tempo. Both carry only metadata — operation names, routes, the surface label, vault ids, counts and durations. Query text, drawer content, wing/room names, and key material are never emitted, so you get full traceability without leaking what’s in the palace.

Here the logs even carry the tamper signal: integrity failure — HMAC verification failed on drawer, tagged with the operation span — traceable, but content-free.

See deploy/observability/README.md for ports, the tamper-demo commands, and the security notes.

Live stream (SSE)

Prometheus is pull-based; for a live view the multi-tenant server also pushes an SSE stream per vault — a periodic sample of aggregate counts plus discrete event pings as they happen. This is what the Palace Monitor UI below consumes. Telemetry build + bearer required. Wing and room names travel on every security level, sealed included: a subscription is only created after the bearer and the per-vault assertion are verified, and that same caller reads those names from GET /v1/vaults/<id>/stats. Drawer content, offsets into it and key material never travel — which is what the suite pins.

# live event stream (Ctrl-C to stop)
curl -N -H "Authorization: Bearer $TOKEN" \
     http://127.0.0.1:8765/v1/vaults/<id>/stream

# recent samples for backfill
curl -H "Authorization: Bearer $TOKEN" \
     "http://127.0.0.1:8765/v1/vaults/<id>/stats/history?window=100"

Frames:

• event: sample — {ts, vault, sealed, drawers, rooms, wings, kg_triples, kg_entities, kg_active, tunnels, chain_height, db_bytes}, where wings is a list of [name, drawers] pairs. Emitted on the sampler tick (default 2s, UNDERCROFT_SAMPLE_INTERVAL_MS), and only for vaults with an active subscriber.
• Discrete pings as they happen, on every security level, each with its own payload: drawer-saved {vault, wing, room, deduped}, drawer-quarantined {vault, intended_wing, room, signals}, drawer-deleted {vault}, search {vault, wing, room, hits} (the search’s declared scope, null where it declared none — never its query), kg-triple {vault} and chain-commit {vault, records}. drawer-quarantined is a write the admission screen DIVERTED: it carries the intended wing/room and the tier-1 signal codes (a closed vocabulary — never the flagged text, never its offsets), and it is deliberately not a drawer-saved into a wing named quarantine-pending. chain-commit’s records is how many chain records that anchor committed.
• event: hmac-fail — {vault, surface, id, wing, room, unverified}, the tamper signal the monitor’s beacon fires on. id, wing and room are what the failing row says about itself — it has just failed its own HMAC — and unverified is always true, so the payload says so.

A comment heartbeat (: ping) every 15s keeps the connection detectably alive.

Each connection is served on its own thread (the request is handed off so the single-threaded server keeps serving), reading only from an in-process broker — never a vault store — so streaming can never touch content.

Palace Monitor UI

A telemetry build also serves a self-contained pixel-art dashboard at GET /monitor (unauthenticated static page — no secrets in it):

http://127.0.0.1:8765/monitor

Enter the palace bearer token, pick a vault (from GET /v1/vaults, or type the id), and connect. An archivist files drawers into wings as writes land, searches pulse the wings, the audit chain stamps on each commit, and the ambulance beacon fires on a real HMAC-verify failure (tamper) — the same hmac_verify_failures signal, live. Until you connect it runs in demo mode with synthetic events. A sealed vault draws its wings like any other — the names travel to the subscriber that proved per-vault authorization — and the beacon lands on the wing the failing row CLAIMS rather than flooding all of them. Against a server that sends no names (an older engine), the page falls back to one locked ◈ sealed block and keeps working.

The beacon is not decorative. Corrupt a single drawer’s bytes on disk and the next read fails its HMAC; a genuine hmac-fail stream event floods the vault red. It fires only on real integrity failure — never a synthetic alarm.

The page uses fetch() streaming (not EventSource, which can’t send an Authorization header) and is fully self-contained — no external requests, same-origin only. It targets bearer-only servers; with per-vault assertions enabled the stream is rejected (the UI shows it) since a browser can’t mint an assertion.