Don’t Trust Our Benchmark. Audit It.

The benchmark nobody asked us to prove

We publish a retrieval benchmark for Uteke. 98.4% recall@5 on LongMemEval-S. The number gets attention. The reaction underneath it is always the same question, phrased five different ways: why should I believe you?

Fair question. Vendor benchmarks have a bad reputation, and the honest answer used to be a shrug. This post is the answer that replaced the shrug.

What we committed

Everything. The repo now carries the raw per-question results of every run we cite:

benchmarks/longmemeval/results/default-500q-v017.jsonl carries 500 lines, one per question: the question ID, the question type, the retrieved session ranking, and the strict and any metrics for that question. Same for the fts5-only ablation. Same for the contradiction-resolution segment. If a number appears anywhere in our README, there is a line in a committed file that produced it.

One of those lines is question 0977f2af, knowledge-update, two gold sessions. In the published run it sat at rank 5. In the independent re-run on a different CPU architecture, rank 6. That single rank swap moved its strict recall from 1.0 to 0.5, and it is the entire difference between the published aggregate and the re-run one. We documented it instead of averaging it away, because a benchmark that hides its one exception is a benchmark you cannot check.

The re-validation nobody paid us for

v0.17.0 shipped with explain mode, a supersede command, and a provenance ledger. None of that touches the retrieval scoring path. But "we did not change the scoring code" is a claim, and claims are cheap.

So we rebuilt the binary from the release SHA, ran all 500 questions again on Modal, and compared. 238 rankings identical. 254 with the same retrieved set in a slightly different order. Eight questions with a different top-10 set. Strict recall@5: 0.8800 before, 0.8800 after. Zero regressions, all deltas within floating-point noise on near-ties.

The number you cite is now the number that survived a rebuild. Different adjective, same accuracy.

How to audit us in about four minutes

Four steps, no trust required:

1. Clone the repo. git clone https://github.com/codecoradev/uteke && cd uteke/benchmarks/longmemeval

2. Find the number you doubt. Every headline in the README names the file that produced it. Doubt the 98.4%? That is results/default-500q-v017.jsonl.

3. Count. Here is the whole audit for the strict number, straight from the README:

python3 -c "import json; e=[json.loads(l) for l in open('results/default-500q-v017.jsonl')]; s=[x['retrieval_results']['metrics']['session'] for x in e]; print(sum(m['recall_all@5']==1.0 for m in s)/len(s))"

That prints 0.880. Same 500 rows we published. No filter, no threshold, no averaging across sub-runs.

4. If you want the full pipeline: run_eval.py runs the harness, modal_fanout.py fans it across ten shards on Modal, and REPRODUCING.md walks the whole path including the exact binary SHA. The fan-out is resume-safe; when Modal preempted three of our shards mid-run, the rerun picked up where they stopped.

What the audit found before anyone else could

Transparency is not charity toward readers. It caught our own mistakes first.

Preparing our benchmark documentation, we found that a baseline in our results file was mislabeled: the numbers we called "v0.15.0 hybrid" were actually the vector-only run. The audit trail made the error findable and fixable in an afternoon. A results table without raw artifacts behind it would have carried that error into every comparison we publish.

The re-run also surfaced a race condition waiting to happen in our benchmark infra: resuming a run against shards produced by a different binary version would have silently mixed results. Now the harness isolates each run's shards, and the fix is in the repo.

Why this matters more than the number

Anyone can print 98.4%. The expensive part is making the number checkable. Every claim in our README traces to a committed artifact, every artifact is small enough to read, and the harness to regenerate them is in the same repository.

That is the whole pitch. The benchmark number is not the trust. The audit path is.