Retrieval, scoring & scaling

Undercroft’s search is a configurable pipeline, not a fixed stack. This page documents how it works, what was measured (full datasets, inside Docker, on real hardware), and which options to pick for which deployment — from a 4-core edge box to a many-core server.

Every measurement below is reproducible with the harnesses in the repo; recall figures and exact commands are in benchmarks/RESULTS.md. The engineering rationale is in docs/RETRIEVAL_SCALING.md.

The pipeline

1. Candidate generation — shortlist drawers for a query.
2. Fusion — hybrid rank of the candidates (semantic cosine + Okapi BM25 + recency).
3. Scoring (optional) — a second stage that re-orders the top candidates for accuracy.

flowchart TB
    subgraph c["Candidate tier — pick one (UNDERCROFT_RETRIEVAL)"]
        scan["full cosine scan<br/><i>default, small palaces</i>"]
        ftsx["FTS5 BM25 prefilter<br/><i>hmac-only, ≥2k drawers</i>"]
        pqx["PQ / IVF ADC<br/><i>48 B/vector, RAM code cache,<br/>sealed rows AEAD</i>"]
        fdex["MUVERA FDE dot<br/><i>token-aware; 256 B PQ codes,<br/>sealed rows AEAD</i>"]
        hnswx["HNSW (feature)<br/><i>RAM-only, ef scales with N</i>"]
    end
    c --> fusion["Fusion — cosine + BM25 + recency<br/><i>HMAC-verified, decrypted candidates</i>"]
    fusion --> r
    subgraph r["Rescore tier — optional (UNDERCROFT_RERANKER)"]
        cex["cross-encoder<br/><i>top-N forwards, many-core</i>"]
        msx["ColBERT MaxSim<br/><i>stored token matrices → tok-PQ LUT,<br/>one query forward, core-independent</i>"]
    end
    r --> hits["verbatim hits"]

The two dominant costs — candidate generation at scale and scoring — are independent, and each has its own purpose-built option.

Measured results

All on LoCoMo (1,982 evaluable QA, session-recall @10) unless noted; synthetic corpora for the pure scaling curves.

Fusion is a free accuracy win

Hash embedder, no reranker, all fusion modes measured:

BM25 buys +1.9 pts at zero latency cost — it re-ranks already-verified candidates and is embedder-independent. The rrf mode measured below both score blends (rank fusion discards score magnitude) and has been removed; its row stays as the record of why.

MiniLM is a wash under BM25 — a modern embedder is not

On LoCoMo, MiniLM adds ~128 ms/query and ~24× ingest for no accuracy gain under BM25. This page used to generalise that row into “the embedder is a wash” — it was a fact about MiniLM, not about model embedders as a class, and four served models measured on the same corpus overturned it (separate run, own k and pool, so read it against its own hash baseline rather than against the table above):

+3.2 to +4.2pp of turn all-gold over hash — comparable to ColBERT’s +4.9pp, at no storage cost and no ONNX export. The second reading matters as much: the four modern models span 1.0pp, so the lever is using a real embedder at all, not picking the best one, and public leaderboard order does not transfer here. No winner is claimed — one run per model, and the served path has not been shown run-to-run deterministic. The cost is 11–29× ingest (one HTTP call per drawer) and +20–57% search.

Serve one with UNDERCROFT_EMBEDDER=http + UNDERCROFT_EMBED_URL. The transport is TLS or loopback, nothing else — cleartext http to a non-loopback host is refused at construction with no override, and UNDERCROFT_EMBED_CA pins a self-signed root (a garbage file refuses rather than falling back to the public roots). Two hazards are stated rather than hidden: the endpoint reads drawer text in plaintext, so TLS protects the wire and not the destination — only the in-process onnx/ort backends close that — and a failed embed cannot fail a write, so it degrades to a counted zero vector: lexically findable, semantically invisible until re-embedded.

Cross-lingual retrieval is the embedder’s job, and the default cannot do it. HashEmbedder is feature hashing over surface forms, so texts meet only on shared literal tokens and trigrams: measured, an EN/AR translation pair scores below an unrelated sentence, and car/automobile do not match either. With a multilingual model served, FLORES-200 cross-script pairs read 95–100% R@5 at the shipped defaults — reached by two calibrations rather than by tuning: the semantic map’s neutral is the embedder’s own measured unrelated floor, and a (query, candidate) pair sharing no letter script takes the blend at the weight ceiling. Both are pairwise byte-readable evidence, never language identification, and the hash default stays bit-identical.

The reranker: big accuracy, big cost — then tamed

A cross-encoder re-scores the top candidates by the full (query, passage) pair. It lifts LoCoMo R@10 to ~98% (+3 pts) but naively costs one forward per candidate:

Parallelizing the independent passes and capping the pool at top_n takes it from unusable to ~24–43× faster at full accuracy. Latency scales as ⌈top_n / cores⌉ — see Scaling to few cores.

Candidate generation at scale (synthetic, hash embedder)

Full-scan is O(n) per query; an ANN index (HNSW prototype) stays flat:

The speedup is real and grows without bound. The recall fall-off in this table was a fixed search beam (ef_search=100 vs the ≥256 candidates the store requests) — since fixed by scaling ef with the corpus: R@5 93→98.8% at 20k and 72→96.3% at 50k, at 126–186 q/s (accuracy now degrades gently instead of collapsing). The in-memory HNSW still costs O(corpus) RAM, though. The durable, bounded-RAM design is the on-disk PQ prefilter (shipped for hmac-only vaults, mirroring the on-disk FTS5 rule): Product Quantization compresses each vector ~32× (1.5 KB → 48 B), the codes live on disk, and only a ~400 KB codebook stays resident. Measured at N=20,000 (hmac-only):

PQ’s recall is flat in N (98.6% → 98.9% — it scans every code, so the only error is quantization), where the graph-based HNSW collapsed without per-size tuning in this run (93% → 72%; fixed since by corpus-scaled ef — see above).

Sealed vaults now get the index too — encrypted at rest. Every code row, the codebook, and the IVF centroids are AEAD-sealed (list ids never stored in clear — they would leak semantic clustering); search decrypts the rows once per open into a ~52 B/drawer RAM cache and scans there. Measured: sealed search went from 2.1 → 33.4 q/s at N=20k (×16) and 1.1 → 11.8 at 50k (×11), at parity with the plaintext hmac-only index — encryption stops being a query-time cost. An offline attacker sees fixed-size sealed blobs: the drawer count it already knows.

A research spike (undercroft-bench pqpage-synth) priced the multi-million follow-up — sealing one AEAD page per IVF list and decrypting only probed lists: at 10⁷ synthetic drawers pages cut at-rest size 2.1×, drop the 22 s open-time decrypt-all to zero, and run warm at 630 MB vs ~1 GB. Both landed: slab-grouping the existing RAM cache by IVF list (no format change) is on by default, and the sealed page tier ships behind UNDERCROFT_PQ_PAGE_MIN — one AEAD page per list, lazily decrypted per probed list, default off because the flat cache is faster until the corpus makes the open-time decrypt hurt. Those pages are sealed but deliberately not compressed: a 4096-row page is InnoDB’s geometry, and compressing-then-encrypting page-shaped data is exactly DBREACH’s precondition (measured details in docs/RETRIEVAL_SCALING.md).

IVF inverted lists now sit on top of the codes: a coarse quantizer (√N centroids) partitions the corpus, codes are physically clustered by list on disk, and a query ADC-scans only the quarter of lists nearest it — recall tracks the probed fraction, and a quarter is exactly recall parity (measured: 99.6% at N=20k, 99.1% at 50k, identical to the flat scan). Benchmarking IVF exposed three structural costs in the scan path — a random-access row layout, a per-search coherence check, and a per-row join — and fixing them lifted flat PQ itself ~45% (within-run: 23.9 → 34.4 q/s at N=20k, 10.1 → 14.8 at 50k). IVF’s marginal gain on top is +7–11% at these sizes and grows with the corpus, since the probed scan is the only query cost that scales with N. On by default above UNDERCROFT_IVF_MIN (8192) whenever PQ is enabled (UNDERCROFT_RETRIEVAL=pq, now wired through the CLI and the multi-tenant /v1 server, not just the bench harness).

Settled at a million drawers

The tables above stop at 50k because that is where the instruments stopped. They no longer do. undercroft-bench pqscale and scopescale grow one cumulative vault through four checkpoints from 131k to 1M drawers, and the shipped defaults hold R@5 100.0% in every column at every checkpoint:

Room-scoped queries run 13–17 ms and wing+room 13–15 ms, flat across all four checkpoints. Only the unscoped row grows with the corpus; every scoped shape is flat, because a declared filter is resolved into the candidate draw rather than applied to it afterwards — room used to be a plain WHERE over globally generated candidates, which is the wing-starvation defect one level down. A scope that fits the hydration budget is scanned exactly; a larger one gets membership-filtered candidates and a pool sized by the scope.

Three findings worth carrying away, because each cost a belief:

• The per-wing index tier’s query-latency benefit is dead. pqscale’s unscoped PQ curve shows no break anywhere from 131k to 1M, so the tier’s real value is the build economics (wing-shaped rather than corpus-shaped) and the starvation fix — a global top-k can miss a scoped wing entirely, leaving candidates ∩ wing empty while the wing holds the answer. The 913 s/query figure that once motivated the tier was the full-scan path, which the global PQ tier answers on its own.
• Recall leaks are a pool-sizing problem, not an index problem. Unscoped R@5 drifted 100.0 → 96.8% by 1M against a fixed 256-candidate pool while the competitor set grew. Closed by a two-stage pool sized in the corpus, and scoped queries by a pool sized in the scope — which read 89.6% until the scope-sized policy closed it at 100.0%. The stage-2 cut is deliberately floored: a sealed vault has no lexical prefilter, so hydration is the only door through which BM25 evidence reaches fusion, and cutting by pure cosine measurably regressed 1M to 98.9%.
• The hotspot was not where anyone thought. Parallel candidate hydration — the queued lever — changed nothing when built. An opt-in phase trace (UNDERCROFT_SEARCH_TRACE=1) then found the cost in BM25’s serial per-candidate scan, ~70 µs each and dominant at every scope. Fanning that out (order-preserving, byte-identical) is what produced the numbers above, from 39.4/66.1/132.8/269.3 unscoped and ~85 ms/q wing-scoped. The instrument that refutes a belief is cheaper than the optimization that encodes it.

Remote vector backends are untrusted accelerators, not a store swap

Undercroft can push sealed content + embeddings to Qdrant / Weaviate / pgvector / Milvus / Chroma, but they only return candidate ids — every candidate is re-verified (HMAC) and re-scored locally. Measured on LoCoMo, the remote backends sat at ~0.5% CPU while the client did all the work, and were slower than the local full-scan for corpora this size (network + a bounded local decrypt per candidate outweigh ANN when the palace is small). They earn their keep only on very large corpora — and even then the scoring stays local. Accuracy and integrity never depend on the untrusted index.

Retrieval policy on that path is the local path’s, verbatim. The closed vocabularies, the deployment trust floor and the quarantine fence all come from one shared resolver and are applied to each candidate’s HMAC-verified metadata. They were absent here until 2026-08-04, which made index push --backend qdrant a route around admission control — closed with a shared required step, not a second copy of the logic. index_push still mirrors quarantined rows deliberately: an untrusted mirror can offer any id, so a push-side filter would not be a boundary, and dropping them would make a reviewer’s explicit --wing quarantine-pending scope answer an empty page instead of the truth. The residue is stated rather than hidden — remotely the floor bounds what came back, not what was generated, which is an availability cost, never an integrity one.

Inference runtime: tract vs ONNX Runtime

Per-forward latency, same ONNX models, seq 256, on a CPU with avx512_vnni (no GPU):

ONNX Runtime is ~2.5× faster than tract at the same precision, and int8 (VNNI) more again — validated in Rust via the ort crate (undercroft-embed-ort, opt-in; tract stays the pure-Rust default). The CLI wires it end to end: build with --features ort, then UNDERCROFT_EMBEDDER=ort / UNDERCROFT_RERANKER=ort / UNDERCROFT_RERANKER=colbert-ort select it at runtime (same model files and env variables as tract). fp32 accuracy is runtime-invariant (identical weights); int8 is within noise. Measured end-to-end on LoCoMo, the ORT backend with a session pool (independent forwards fanned across single-thread sessions; pool=1 = one batched all-core forward for few-core boxes) and int8 models (a 4× smaller file — no code change, just point the env at the quantized model):

with R@10 at 98.3 / 98.0 / 98.0% — and ingest embed ~4–5× faster (24 s → 5 s). End to end, the reranker went 16.6 s → ~101–171 ms (~100–160×) at ~98% accuracy. On a GPU, ORT-CUDA puts each forward at ~1–5 ms.

Scaling to few cores

The reranker’s parallel strategy is ⌈top_n / cores⌉ waves of one forward each. On 24 cores top_n=20 is one wave; on 4 cores it is 5 waves (~270 ms). More cores buy headroom, not a lower floor; the floor is one forward. So on constrained devices the answer isn’t more parallelism — it’s doing fewer query-time forwards:

• ColBERT late interaction (shipped, UNDERCROFT_RERANKER=colbert) encodes passage tokens once at ingest (PQ-compressed on disk; sealed vaults AEAD-seal every matrix — the first encrypted-at-rest derived store) and, per query, does one forward + a cheap MaxSim (no transformer per candidate). Measured on LoCoMo (full 1,982 QA): 94.6 → 96.77% R@10 at a flat 92.7 ms/query on pure-Rust tract, 70.3 ms/query with the opt-in ONNX Runtime forwards + token-PQ LUT (recall identical across runtimes; ingest 3.3× faster too) — the same on 4 cores or 24, while the cross-encoder’s 97.68% costs 101–327 ms on 24 cores and ~5× that on 4.
• A stronger bi-encoder with no reranker is also one forward, core- and top_n-independent, at some accuracy cost.

So the cross-encoder + rayon path is a many-core optimization; ColBERT is the portable, core-independent option for constrained boxes.

MUVERA FDE candidates (UNDERCROFT_RETRIEVAL=fde) extend token-awareness to the candidate stage: each stored token matrix compresses into one fixed-dimensional vector (arXiv:2405.19504) whose dot product approximates MaxSim — sealed at rest, built with zero extra transformer forwards, one shared query forward per search. Measured: LoCoMo recall question-for-question identical to the fusion pipeline at 52.9 vs 70.3 ms/query (−25%); on synthetic corpora up to N=200,000 the exact MaxSim top-10 survived the FDE top-100 100% of the time at 38–40× below exact-scan cost. Above a few hundred drawers the FDEs PQ-compress 32× (256 B each, 51 MB at N=200k) with containment still perfect and the scan ~8× faster — the same bounded-RAM story as every other index tier.

Configurable — choose per deployment

Retrieval, scoring, and runtime are independent, user-selectable axes. Defaults are local-first and pure-Rust; every faster option is opt-in.

Retrieval

Scoring

Inference runtime

A 4-core edge box picks IVF-PQ + ColBERT + int8; a many-core server can add the cross-encoder + rayon fast path; a GPU box turns on ort-CUDA. Same engine, config-selected — never a rewrite.

Scenario recipes

Concrete configurations with the measured expectations:

Rules of thumb from the measurements: BM25 fusion is always on (free +1.9 pts); MiniLM is not worth 20× latency under BM25, but a modern served embedder is (+3.2–4.2pp of turn all-gold, and the only way to retrieve across languages at all); the reranker is the accuracy lever (+3 pts) and is now affordable (top_n=20, ort+int8); PQ is the bounded-RAM index whose recall holds at scale — 100.0% R@5 measured at every checkpoint from 131k to 1M drawers; remote vector DBs never make a small palace faster — they are for corpora too large to scan locally, and all trust (and all retrieval policy) stays local regardless.

Invariants preserved throughout

Every option obeys the vault rules: sealed vaults never persist a plaintext-derived index to disk (in-memory ANN is RAM-only; on-disk indexes for sealed vaults are encrypted at rest, mirroring drawer sealing). Remote backends are untrusted — content is sealed before upload and every result re-verified locally. Faster never means less safe.