Architecture

The palace

Palace (data dir, one master key)
└── Vaults (isolation boundary: own DB file, own derived keys)
    ├── Wings   (people / projects)          ── connected by Tunnels
    │   └── Rooms (topics)
    │       └── Drawers (verbatim chunks, ~800 chars)
    ├── Knowledge graph (temporal triples with validity windows)
    ├── Audit chain (append-only, HMAC-chained writes)
    └── Hallways (entity co-occurrence, computed on demand — never persisted)

Components and dependencies

Thirteen crates. Solid arrows are Cargo.toml dependencies; the dashed arrow is the one deliberate non-dependency in the design — the orchestrator talks to engines only over HTTP (/v1), so the engine stays tree-blind and portable.

flowchart TB
    subgraph engine["Engine (ships in the box)"]
        core["undercroft-core<br/><i>domain, chunking, ids,<br/>hash embedder, FDE, MaxSim</i>"]
        vault["undercroft-vault<br/><i>HKDF keys, AEAD sealing,<br/>HMAC tags, audit chain</i>"]
        store["undercroft-store<br/><i>per-vault SQLite, hybrid search,<br/>PQ/IVF, ColBERT stage, FDE index, KG</i>"]
        index["undercroft-index<br/><i>remote vector backends<br/>(untrusted accelerators)</i>"]
        llm["undercroft-llm<br/><i>local LLM runtimes<br/>(refine → KG, served embedder)</i>"]
        net["undercroft-net<br/><i>outbound transport policy:<br/>TLS or loopback, no override</i>"]
        config["undercroft-config<br/><i>declaration resolvers the engine<br/>and the control plane share</i>"]
        obs["undercroft-obs<br/><i>observability shim<br/>(no-op by default)</i>"]
        cli["undercroft-cli<br/><b>undercroft</b> binary<br/><i>CLI + MCP + HTTP /v1</i>"]
    end
    subgraph optional["Opt-in inference backends"]
        onnx["undercroft-embed-onnx<br/><i>tract: embedder, reranker, ColBERT</i>"]
        ort["undercroft-embed-ort<br/><i>ONNX Runtime: same trio, faster</i>"]
    end
    bench["undercroft-bench<br/><i>LongMemEval / LoCoMo / synthetic<br/>scale + screen instruments</i>"]
    orch["undercroft-orchestrator<br/><b>undercroft-orchestrator</b> binary<br/><i>multi-tenant control plane</i>"]

    vault --> core
    vault --> obs
    store --> core
    store --> vault
    store --> index
    store --> net
    store --> config
    store --> obs
    index --> net
    llm --> core
    llm --> net
    llm --> obs
    obs -. "feature telemetry<br/>(OTLP hop)" .-> net
    cli --> core
    cli --> vault
    cli --> store
    cli --> index
    cli --> llm
    cli --> obs
    cli --> net
    cli -. "features onnx / ort" .-> onnx
    cli -. "features onnx / ort" .-> ort
    onnx --> core
    onnx --> obs
    ort --> core
    ort --> obs
    bench --> core
    bench --> vault
    bench --> store
    bench --> index
    bench --> llm
    bench -. "features onnx / ort" .-> onnx
    bench -. "features onnx / ort" .-> ort
    orch --> obs
    orch --> net
    orch --> config
    orch -. "HTTP /v1 only —<br/>never linked BY the engine" .-> cli

Key hierarchy and AAD domains

Isolation is cryptographic, not logical. One master key; every vault derives its own keys via HKDF, and every sealing operation binds the vault id (and an artifact-specific label) into the AAD — ciphertext moved across vaults, rows, or artifact kinds fails to open rather than decrypting wrongly.

flowchart TB
    master["Master key<br/><i>file or Argon2id passphrase</i>"]
    master -- "HKDF-SHA256(vault A salt, label)" --> ka["vault A subkeys<br/>enc · mac · manifest · sample<br/><i>fingerprints = truncated HMAC under mac</i>"]
    master -- "HKDF-SHA256(vault B salt, label)" --> kb["vault B subkeys<br/>enc · mac · manifest · sample"]
    ka --> doms["AAD domains (vault A)<br/><br/>{id} — drawer content<br/>{id}/emb — embeddings<br/>{id}/tok — token matrices<br/>fde/{id}/tok — FDE rows<br/>pqrow/…/pq — PQ index artifacts<br/>kg/{id} — fact objects<br/>kgterms/{id} — subject + predicate<br/>kgname/{blind} — entity names"]
    ka --> kgs["kg blind secret<br/><i>32 random bytes sealed in meta —<br/>STORED, re-sealed on rotation,<br/>never re-derived: ids must not move</i>"]
    kb -. "vault B ciphertext under<br/>vault A keys ⇒ fails to open" .-> ka

Sealed vaults never persist plaintext or plaintext-derived data in clear: embeddings, PQ code rows and codebooks, ColBERT token matrices, and FDE rows are all AEAD-sealed under their distinct domains, and search runs from decrypt-once RAM caches.

Write path

Every write is verbatim (never summarized), deterministic (same logical drawer ⇒ same id ⇒ idempotent re-mining), and atomic with its audit entry — the chain head lives in SQLite and advances inside the same transaction as the data it covers.

sequenceDiagram
    participant C as Caller (CLI / MCP / REST)
    participant S as store
    participant V as vault
    participant DB as SQLite (one transaction)
    C->>C: normalize (verbatim-preserving) → chunk → deterministic id at construction
    C->>S: save(drawer — content, wing, room)
    S->>S: validate the declaration (names, kind, id shape, content length)
    S->>S: embed (hash / onnx / ort / http / external vector)
    S->>S: validate again with the vector, then Screen (admission tier 1 + rate)
    Note over S: a flagged write is DIVERTED into the reserved review wing<br/>and re-enters this path with Bypass(AlreadyDiverted) — never dropped
    S->>DB: BEGIN IMMEDIATE
    S->>V: seal content + embedding (sealed vaults — AAD binds vault id + record id)
    S->>V: HMAC tag over id ␟ meta_at_rest ␟ sealed content
    DB->>DB: drawer row (sealed blobs + tag)
    DB->>DB: audit row + chain_append → chain_meta head advances
    DB->>DB: COMMIT  — data and chain move together or not at all
    S->>V: anchor manifest (lagging rollback anchor, post-commit)
    Note over S: derived artifacts, advisory, from plaintext in hand:<br/>PQ code row → token matrix (ColBERT) → FDE → FTS entry (hmac-only)

Crash between COMMIT and the manifest anchor? The next open replays the audit rows: an anchor inside the replayed chain is a crash artifact (silent fast-forward); an anchor outside it is a rollback or fork (ManifestTampered). A power cut is never a false alarm; a restored old database still alarms.

Search pipeline

Candidate generation is pluggable; everything downstream is identical on every path, and every candidate’s HMAC is verified before its content is returned.

flowchart LR
    q["query"] --> scope["scope resolution<br/><i>wing · room · kind · trust floor ·<br/>quarantine fence → seq filter,<br/>BEFORE any candidate is drawn</i>"]
    scope --> cand{{"candidate stage"}}
    cand -- "UNDERCROFT_RETRIEVAL=fde" --> fde["FDE dot product<br/><i>token-aware, PQ-coded cache</i>"]
    cand -- "=pq" --> pq["PQ / IVF ADC scan<br/><i>bounded RAM, per-wing tier</i>"]
    cand -- "=hnsw (feature)" --> hnsw["in-memory HNSW<br/><i>experimental</i>"]
    cand -- "default" --> fts["FTS5 BM25 prefilter<br/><i>hmac-only, ≥2k drawers</i><br/>or full cosine scan"]
    fde --> hyd
    pq --> hyd
    hnsw --> hyd
    fts --> hyd
    hyd["hydrate candidates<br/>+ <b>HMAC verify each</b><br/>+ decrypt (sealed)"] --> fuse["fusion score<br/><i>cosine + BM25 + recency</i>"]
    fuse --> gate["relevance gate<br/><i>lexical exact / morph channels,<br/>or cosine above the embedder's<br/>measured admission floor</i>"]
    gate --> second{{"second stage"}}
    second -- "UNDERCROFT_RERANKER=onnx | ort" --> ce["cross-encoder rerank<br/><i>top-N forwards</i>"]
    second -- "=colbert | colbert-ort" --> ms["MaxSim rescore<br/><i>stored token matrices,<br/>PQ-LUT, one query forward</i>"]
    second -- "unset" --> out
    ce --> out["verbatim hits"]
    ms --> out

The FDE and MaxSim stages share one query forward per search; sealed vaults serve all of this from decrypt-once RAM caches. Measured numbers for every stage live in RETRIEVAL_SCALING.md.

Multi-tenant deployment

One engine hosts many cryptographically isolated vaults; fleets add the optional orchestrator — topology, request routing, and the migration sequence are diagrammed in MULTI_TENANCY.md.