We published our benchmark. Then we tried to break it.
We re-ran our LongMemEval results on completely different hardware. 107 of 108 questions reproduced exactly. What that taught us about reading AI memory benchmark numbers, including ours.
Last week we published a number: 98.2% recall on LongMemEval-S: 500 questions, zero config, one Rust binary, no LLM anywhere in the retrieval path.
It felt good for about a day. Then the question showed up, the one that should follow every benchmark post on the internet:
How do you know that number survives outside the machine that produced it?
This post is about how we tried to break our own number, what survived, what didn't, and what we learned about reading benchmark claims in the AI memory space. Including ours.
Why we tried to break it
Our published run was produced on cloud x86 infrastructure (Modal). The re-run happened on a 4-core ARM desktop, a completely different CPU architecture.
That difference matters more than it sounds. Our retrieval pipeline fuses two rankings with Reciprocal Rank Fusion, which is floating-point arithmetic all the way down. Cross an architecture boundary and the last bits of precision can round differently. When two sessions score almost identically, a rounding difference can swap their order.
If our 98.2% was secretly an artifact of one machine, we wanted to find out before someone else did.
The experiment
Published run: Modal x86, full 500 questions, uteke v0.16.0, default strategy, August 29.
Re-run: a 4-core ARM (Ampere A1) box, same v0.16.0 binary, same public harness from the repo, a 108-question subset.
We compared per question, ranking against ranking, rather than trusting the aggregate.
107 of 108 questions produced identical rankings.
The one question that differed is worth understanding. Both runs retrieved the exact same top-10 set of sessions. One gold session sat at rank 5 in the published run and rank 6 in the re-run. On the strict metric we publish, that one swap moves that question's recall from 1.0 to 0.5.
That's the entire difference between the two runs. Same sessions found, one near-tie flipped order across a CPU architecture boundary. The aggregates moved from 96.7% to 96.7% and from 100.0% to 99.4%.
A number that survives that trip is a number. We're comfortable putting ours on it.
A field guide to reading memory benchmark numbers
While staring at these results we kept a list of everything that makes two "98%" claims incomparable. Five checks, in the order we'd apply them to any memory benchmark, including the ones in this post.
1. Which metric family? recall_any@K passes a question when at least one correct session appears in the top K. recall_all@K requires every correct session. The first is the de-facto vendor metric; the second matches reality, because an answer spread across three sessions is only truly solved when all three come back. From our own published run: 98.2% recall_any@5, but 88.0% recall_all@5 (the strict number has a mathematical ceiling of 99.4%, because three questions carry six gold sessions each). Same run, same data, ten honest points apart. Most published numbers you'll see are from the first family only.
2. What's the K? Recall@15 is easier than Recall@5, because a bigger window catches more. Two systems reporting different K values are not in a competition.
3. How big was the sample? 100 questions is a smoke test; 500 is a validation run. On a 100-question set, one question is a full percentage point.
4. When was it measured, on which version? Benchmarks are point-in-time snapshots of moving software. Mnemosyne's README, refreshingly, asks readers to check version labels before quoting their numbers, because their published results predate current features. That's the right instinct, and more projects should copy it.
5. Is the harness public? A table in a PDF is marketing. A harness you can clone and run is a benchmark.
The whole field, read honestly
The wider picture, with every number traced to its published source and its metric labeled, because that's the whole point of this post:
| System | Number | What it measures |
|---|---|---|
| Mnemosyne | 98.9% | strict Recall@All@5, 100 questions, Apr 2026 |
| Uteke | 98.2% | recall_any@5, 500 questions, re-run verified |
| MemPalace | 96.6% | recall_any@5 |
| agentmemory | 95.2% | recall_any@5 |
| Supermemory | 95.0% | Recall@15 |
Read that table as a map, not a podium. Different metrics, sample sizes, embedding models, and dates. Mnemosyne's own README says it best, and we'll quote them: "the metrics are not identical and the ordering should not be read as a strict ranking."
One thing we noticed while collecting these: almost every system publishes a single number, and it's almost always from the easier recall_any family. To our knowledge, Mnemosyne is the only other system in this field that publishes a strict-family metric at all. We publish both families from the same run, plus the mathematical ceiling, because a strict score without its ceiling is just a smaller number with no context.
And because trust is symmetric: when we re-ran a competitor's baseline harness as a control during this exercise, it reproduced their published numbers exactly. That's what a healthy benchmark looks like. The ecosystem gets better when reproduction is normal.
What we're not claiming
We are not claiming we beat Mnemosyne. The metrics and sample sizes differ, and we won't pretend a chart makes them comparable. We are not claiming these numbers transfer to your embedding model, your hardware, or your data. And we're not claiming the 500-question run is the last word; it's the current word: dated, versioned, and as of this week reproduced.
Run your own
The full harness, dataset pipeline, and per-question results live in the repo:
# retrieval eval, zero-config default (fusion)
python3 run_eval.py --data data/longmemeval_s_cleaned.json --strategy default- Harness + reproduction results: benchmarks/longmemeval/RESULTS.md
- Published numbers: docs/benchmarks.md
Don't trust our benchmark. Run your own. That's not a tagline. This week it's literally what we did.