Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Tamper runbook

When Undercroft raises PalaceTamperDetected (or the Palace Monitor’s ambulance beacon lights, or undercroft verify reports a non-zero hmac failures count), a stored record failed its HMAC integrity tag on read. Treat it as on-disk tampering until proven otherwise. This page is what the alert’s runbook_url points to.

Integrity is cryptographic, not advisory: every drawer, KG triple, tunnel, and vault manifest carries an HMAC-SHA256 tag, and every write joins a tamper-evident audit chain. A verify failure means the bytes on disk no longer match what Undercroft sealed.

The whole procedure at a glance — each step is detailed below:

flowchart TB
    alert["PalaceTamperDetected<br/><i>alert / monitor beacon / verify count</i>"] --> loc["1 · Where?<br/><i>surface label on the counter,<br/>vault · id · wing · room on the event</i>"]
    loc --> conf["2 · Confirm + pinpoint<br/><i>undercroft verify --vault —<br/>names the exact record(s), chain state</i>"]
    conf --> mit["3 · Mitigate<br/><i>preserve evidence copy FIRST ·<br/>freeze writes (--read-only) · isolate vault</i>"]
    mit --> fix{"4 · Fix — verbatim restore,<br/>never repair-in-place"}
    fix -- "known-good backup" --> restore["backup restore →<br/>verify must report 0 failures"]
    fix -- "single MINED record,<br/>source document available" --> refile["re-file it —<br/><i>source-derived id ⇒ idempotent re-seal</i>"]
    restore --> clean["repair (housekeeping) →<br/>read-write only once verify is clean"]
    refile --> clean
    clean --> prev["5 · Prevent<br/><i>scheduled backups ·<br/>owner-only permissions ·<br/>OS-level FIM · alerting on ·<br/>per-vault assertions</i>"]

1. Where did it happen?

The alert carries two labels that localize the failure:

  • surface — which structure failed: drawer, kg, tunnel, or manifest.
  • instance — which server process counted it.

There is no vault label on this alert: the integrity counter is emitted with surface alone. (One rule does carry one — AuditChainHeightHigh, whose expression is an unaggregated gauge, and gauges have always been per-vault. This sentence said “or any other” until ROADMAP O250 added it.) The vault is on the live event stream instead — the hmac-fail frame the Palace Monitor reads names it, beside the id, wing and room the failing row claims (marked unverified, since that row has just failed its own HMAC). The stream is live only, so it names the vault only to a subscriber connected when the failure happened; otherwise run step 2’s verify against each vault the process serves.

In Grafana, the “Tamper by surface” panel and the HMAC verify failures stat show the same signal; the Logs panel shows the integrity failure — HMAC verification failed on <surface> line.

2. Confirm and pinpoint the record

Run a full verification of the affected vault — it re-checks every record’s HMAC and replays the audit chain, naming the exact bad record(s):

undercroft verify --vault <vault>
# records checked: 1284
# hmac failures:   1
#   TAMPERED: 5a2fc91d…
# unreadable:      0
# audit chain:     ok
# orphan labels:   0
# mirror drift:    0
# policy drift:    0
# version replay:  0
# storage:         0
# identity:        0
# field shapes:    0
# VERIFY FAILED

The named id is the tampered record, and VERIFY FAILED exits 2. Expect audit chain: ok beside it: the chain is replayed from the audit trail’s own tags against the committed head and the manifest anchor, so editing the tampered record’s bytes does not move it. audit chain: BROKEN is a separate finding — the audit trail itself was edited or truncated, or the database was rolled back relative to the anchor.

unreadable: counts drawers whose HMAC VERIFIES while their covered meta does not parse (ROADMAP O231), each named UNREADABLE: <id>: …. No read can return one, it fails the verdict, and backup create refuses while it stands — but a key rotation is not refused over it, since a rotation cannot launder it. It can be made without the key (ROADMAP O406), so treat it as evidence: preserve the directory first, then restore an archive that holds the drawer intact, or drawer delete it if its content is not needed. Since 1.7.0 it also counts a row a column of which holds what no read takes — the wrong storage class, or text that is not UTF-8 — named UNREADABLE: <id>: column <name> holds … (or by seq where its id does not decode), where verify used to stop with a conversion error; that one DOES block a rotation, which cannot read it (ROADMAP O269).

storage: is SQLite’s own integrity_check of the file the rows live in, run first (ROADMAP O269). STORAGE: integrity_check: row 11 missing from index idx_drawers_room names an index that disagrees with its table: an index hides rows from every read that finds one through it — forget included — so a vault can answer every other leg clean and still serve wrong answers. Restore the newest archive whose restore is accepted; a restore refuses one carrying the same damage. undercroft repair does not mend it, on purpose: its closing VACUUM copies an index as it stands, and rebuilding the index from its table would conform it to the table — while one byte changed on the TABLE side of an indexed column gives exactly this line, so a rebuild could turn an edit into a clean verify (ROADMAP O434 asks whether a rebuild that first rules that out should exist). STORAGE: <leg> not run: database disk image is malformed is a leg that met damaged pages partway; and a file damaged at its first pages refuses to open at all, exit 2 with the vault’s database is damaged. All of them mean restore. One exception is not damage: a read-only handle that had to open its database with immutable=1 — a write-protected mount, a snapshot — answers the storage leg cannot be verified on this handle, exit 1, rather than a storage line, because beside a process still writing the vault its reads can tear; run verify where the directory is writable, or on a copy.

A broken chain also stops work before you get here, and that is deliberate (ROADMAP O237). Every read that DECIDES from an audit label — a trust-floored search, trust list, retention list, a retention sweep, forget, and the version check every returning read rides — refuses with the audit chain does not authenticate its own labels, exit 2 and 409 class: "integrity", rather than acting on a relabelled record until somebody runs this command. So an operator usually arrives here because a read refused, not because an alert fired. The remedy is the same: preserve the evidence, then restore a backup that verifies. A vault this binary has not yet opened writable keeps serving, because its labels were never bound to the chain.

The next five lines are further legs, and a non-zero count on any of them fails the verdict too; a vault holding supersession links or fact receipts prints a line for each of those legs as well, where only a tampered count fails.

3. Mitigate now (stop the bleeding)

  1. Preserve evidence first. Copy the vault directory before anything else touches it — the DB, its -wal/-shm, vault.json, and vault.json.next if one is there:

    cp -a "$UNDERCROFT_HOME/vaults/<vault>" "/tmp/<vault>.evidence.$(date +%s)"

    Since 1.2.1 a read-only open (see step 2) writes nothing to the database, vault.json, vault.json.next or a hot -wal. In a writable directory it may still create the -shm wal-index and a zero-length -wal — SQLite’s scaffolding for reading a WAL database, carrying no database content. From 1.0.0 through 1.2.0 an embedder lookup ran before the posture took effect — on a /v1 request from 1.0.0, and on the CLI’s own read-only open from 1.2.0 — and could create a missing database or checkpoint a crashed writer’s hot -wal into it (ROADMAP O91), so on those versions the copy has to come before any process opens the vault. Take the copy on every version: it is the only thing that survives a writable process someone else starts, and a forensic copy costs seconds. backup create is not this copy: it archives only a vault that verifies, prunes the vault’s oldest archive once ten exist — perhaps the one taken before the incident — and a writable one first heals a lagging anchor and promotes a deferred key rotation. Since 1.7.0 --read-only backup create refuses before it writes anything under backups/ or into the vault’s database or manifest (ROADMAP O212) — its read-only open, like every one, may still leave the scaffolding described above, which is why the copy comes first.

  2. Freeze writes. Restart the server read-only so nothing new is written on top of a compromised store while you investigate:

    undercroft serve-http --read-only …

    --read-only is a posture on the whole process, not a filter on one port: both stores the server opens take it, the gate sits in front of route dispatch and fails closed (everything is a mutation unless explicitly named otherwise), and the read-audit record and the embedder migration — both writes — are suppressed. POST …/verify is allowed and is a genuine read: it walks every record’s HMAC and replays the chain, and it does not fast-forward the manifest anchor (an earlier version of this step said it did).

    The open is a read too — the open itself since 1.0.0, and the embedder lookup that ran ahead of it since 1.2.1 (step 1). It used to be the one write --read-only did not bound, and the worst of it ran on the very path this step recommends: rotation reconciliation happened before the read-only/read-write split, so the first request against a cold handle either promoted a staged vault.json.next over vault.json — adopting a new key generation — or deleted it outright, with an fsync. That was potential evidence destruction on the path chosen to avoid touching the vault (ROADMAP R4/A32). Now the connection itself is opened SQLITE_OPEN_READ_ONLY under PRAGMA query_only=ON, the schema is checked rather than created, the anchor is reported rather than healed, a staged rotation is honoured in memory only and its file left exactly where it is — and, since 1.7.0, verified against: a rotation whose promote was deferred (or a writer that crashed between its rotation’s commit and its promote) opens read-only, verifies against the STAGED manifest, and says so on unhealed. From 1.1.0 until then that open answered ManifestTampered, exit 2 — if you met it right after a rotation, a writable open would have promoted and verified. The note is re-checked against the files once the open has its database (ROADMAP O288): if a promote, a discard or a restore lands between the open’s unlock and its database open, the command is refused with the reopen class and retried once, so the note is true at the moment the open checks it. It is not re-checked after that: a promote landing later — even within one command, and for the whole life of a long-lived server — leaves the note describing the vault as it was at that open, and the note says so (“at this open”); since ROADMAP O289 that server answers a vault.json forged afterwards with the tamper verdict and its alert, as any server does. Do not delete vault.json.next while it is there: until a writable open promotes it, it is the only file holding the vault’s current keys, and a read-only open refuses as an integrity verdict once it is gone. A live server whose vault.json is deleted, or replaced by anything that is not a manifest file, answers the integrity verdict on verify, the witness and a backup (ROADMAP O277) — it used to answer VERIFY OK over a vault no process could reopen. A writable open over a deferred rotation whose vault.json went missing, or was edited, between its unlock and its database open refuses — the integrity verdict for a missing, torn or forged file, the age refusal for a too-new one, an I/O error for one it cannot read — and leaves both files exactly as found (ROADMAP O290): it used to write the staged manifest back from memory, which turned a database restored beneath it into a crash heal and overwrote a forged manifest with no alert. The same holds for an edit landing while that open replays the audit chain, just before it writes (ROADMAP O304): it asks again, and refuses with both files kept. A manifest of the new key generation put there meanwhile — another promote, or one copied by hand — is followed when it is the staged one; at any other head the open answers this handle must be reopened, which the CLI and /v1 retry once, and the reopen judges it. And a writable open whose DATABASE does not answer to the manifest it would adopt — its key-generation marker deleted or edited beside a deferred rotation, or an older copy of the database restored behind the manifest’s anchor — refuses before it removes vault.json.next, writes vault.json or re-seeds the marker, and leaves all three as found (ROADMAP O296): it used to delete the staged file, the only copy of the new keys, and only then refuse. An older copy restored exactly AT the anchor is indistinguishable from a rotation that crashed before its commit: the open discards the staged file and serves the older state, as a crash would — keep a copy of vault.json.next before restoring a database by hand beside it. A database whose chain_meta holds no committed head while its audit trail holds records — or a head with no committed height — is refused by a writable open with nothing written and reported by a read-only one; an audit trail erased beside a later anchor is the tamper verdict (ROADMAP O303). No release since 1.0.0 writes either state, so treat it as an edit and restore a backup that verifies: before 1.7.0 the open adopted it from the manifest and its first write moved the anchor over the evidence. A prefilter loads an index but never builds one. What the open declined to repair is printed as a warning and readable afterwards on undercroft stats (and GET /v1/vaults/{id}/stats) as unhealed — during an incident, read it: “vault.json.next does not authenticate … and was left in place” names a staged manifest this build cannot read — a torn write from a build before 1.7.0 or a file something else put there, which since 1.7.0 no open deletes on either posture (ROADMAP O257) — and since 1.7.0 (ROADMAP O246) a WRITABLE open reports on the same list the manifest anchor heal it performed and how far behind the anchor was — a crash, or another writer committing while the handle opened (ROADMAP O253), is the ordinary cause, and a genuine older vault.json restored beside a current database looks identical, so treat it as evidence to read during an incident rather than as an alarm. Since 1.2.0 the admin console at GET /ui shows it too, as an UNHEALED panel that appears only when there is something to say, beside a POSTURE gauge naming the role the handle was opened under. Before that the console showed a clean, complete- looking stats page for a replica with both conditions live.

    Two conditions refuse instead, both 409, because serving through them would answer a question wrongly rather than partially: a manifest whose vault.db is absent (a half-copied backup or a snapshot taken mid-write — “empty” is not “absent”, and this one exits 2, an integrity verdict), and a schema this build would have had to migrate (open it once with a writable process, then retry).

    The database is vault.db since 1.5.0, beside vault.json; before that it was palace.db. A vault created earlier keeps that name until the first WRITABLE open that finds nothing else holding it, which checkpoints the WAL and renames it in place; beside a running replica or server the name is kept and that open says so on unhealed (ROADMAP O281 — renaming beneath a holder left it on a file nobody else shared). A read-only open serves the file where it is and reports the pending rename on unhealed, so a replica of a not-yet-migrated primary says so rather than failing. A directory holding BOTH files is refused on either posture (409, exit 2 — one of them is a stray copy, and an open that picked one would serve the wrong vault silently): move the stray aside and reopen. Any script of yours that names the file — backups, the tamper demo in the observability README, a monitor — must name vault.db, or check for both.

    If your incident needs a byte-frozen vault, stop the server rather than restarting it. If the vault lives on a write-protected mount or a snapshot, the read-only open escalates to SQLite’s immutable=1 mode and says so in a warning — correct there, and wrong if anything is still writing, which is why it is reached only after the ordinary open has failed.

  3. Isolate. If this is a multi-tenant server, the vault named by the hmac-fail event (or by verify) scopes the blast radius — other vaults have independent HKDF-derived keys, so one vault falling tells an attacker nothing about its siblings.

4. Fix (restore verbatim)

Undercroft never lossily transforms your data, so the fix is a verbatim restore, not a repair-in-place of forged bytes:

  1. Restore from the most recent good backup. backup refuses to run if the source failed verification, so a listed backup is known-good at capture time — and restore proves it again before it touches the live vault: the archive is copied into a stage, opened with this installation’s key and the vault’s embedder, verified over every leg — SQLite’s integrity_check among them, as the storage leg (ROADMAP O269) — and only then swapped in. An archive that fails any of it exits 2, prints what its verification found, and the live vault is left exactly as it was, which on this path is also your evidence (ROADMAP O268): 
    undercroft backup list                          # names are <vault>-<stamp>
undercroft backup restore <vault>-<stamp> --force   # stop the server first; --force to overwrite the live vault
undercroft verify --vault <vault>   # must now report 0 hmac failures, chain ok
    The restore prints the chain height and head the archive held — compare them with what backup create printed for that archive — and whether the vault it replaced had been rotated since (it brings the older keys back: rotate again if that rotation answered a compromise). Emit a new external witness afterwards; an older one reads the restored vault as rolled back, which it is. If the live vault’s directory holds a manifest and no database, restore refuses (O270): stop every process, move that directory aside, then restore. If it holds a symbolic link or anything but regular files — a link planted at its -wal or vault.json.next included — or is a link itself, restore refuses and names the entry (O283, O301): move the entry out, keeping it with your evidence, then restore. If a restore is interrupted between its two renames, vault create, init and another restore refuse and print the mv that puts the replaced vault back.
  2. If a single mined record was hit and you have the source document, re-mine it with the same path, --wing and --mode it was mined with: a mined drawer’s id is derived from (wing, room, source, chunk index, normalize version), so re-mining rewrites that row under a fresh seal. Re-verify afterwards. This does not hold for drawers filed by sweep, which skips any message whose content fingerprint is already stored — and an edit to the content bytes leaves the fingerprint column in place, so a re-sweep counts the tampered message as already filed and never rewrites it. Nor does it hold for drawers written through remember / the API, which have no source and carry a unique append index instead — re-saving those creates a new drawer beside the tampered one rather than replacing it. Restore from backup is the only verbatim fix for both.
  3. Housekeeping after a clean restore: 
    undercroft repair --vault <vault>  # backfill fingerprints, re-embed every drawer + drop PQ/IVF (a served embedder receives the corpus, recorded as egress/embed/repair), vacuum, re-verify

Only return the server to read-write once verify is clean.

5. Prevent (before the next time)

  • Back up on a schedule. undercroft backup create --vault <vault> is the recovery path above; without a good backup, a verbatim restore isn’t possible. Only the ten most recent snapshots per vault are kept — older ones are pruned on each create, so a schedule needs its own off-box retention. Since 1.7.0 an archive is exactly the state its verify judged, even beside a running server, and backup create prints the chain height it holds: record it beside the name. An archive taken by 1.6.1 or earlier while anything held the vault may be torn — restore it into a scratch data directory and run undercroft verify there before restoring it over a vault, because a restore replaces the vault before it knows the archive opens (ROADMAP O268).
  • Lock down the store. master.key should be 0600 and the vault directory 0700 (owner-only). Anything that can write the vault DB out-of-band can tamper; anything that can read master.key can forge.
  • Back up the key material too. backup create copies the vault, not master.key or kdf.salt, and a backup opens only under this installation’s key. Never delete either file to silence a message: the engine refuses a key source the installation contradicts rather than writing a new key (ROADMAP O204), and an installation an older release left holding both files may have vaults under each.
  • Add OS-level file-integrity monitoring (auditd / a tripwire) on the vault directory — Undercroft catches tamper on read; FIM catches the write.
  • Keep telemetry alerting on. PalaceTamperDetected fires within a scrape interval — that early signal is the point.
  • Use per-vault assertions for multi-tenant deployments so a compromised client can’t reach another tenant’s vault.
  1. Witness the chain off-machine, on a cadence (ROADMAP O245). The one rollback the anchor cannot see is a genuine earlier (vault.db, vault.json) pair restored together — verify is green on it by construction. undercroft witness emit --out <file> (or GET /v1/vaults/{id}/witness on a served vault) writes a small document that binds the audit row count and an unkeyed digest over the rows’ preserved bytes; keep it somewhere the machine cannot write (a commit in another repository, an append-only log, mail to yourself — never under the data directory, which a backup restores with the vault), one file per emit, and check before you emit: undercroft witness check <file> (or POST …/witness with the document as the body) answers WITNESS OK when the chain extends it and WITNESS FAILED, exit 2, when the vault has been rolled back below it. A key rotation does not fail the check — it retires the head’s corroboration and the check says so. Sign with --sign and the bundle sign-keygen identity if others can write to the witness store, and keep that key off the machine too, since it defaults to living beside master.key.

The guarantee

Tamper-evidence only works if the alarm is trustworthy — so Undercroft only ever raises it on a real HMAC-verify failure. There are no synthetic or demo tamper alarms anywhere in the system: metrics, the live event stream, and the Palace Monitor beacon all read the same hmac_verify_failures signal.