Why our benchmark numbers are lower than Mem0's (and why that's the point)
Mem0 publishes 92.5 on LoCoMo; we publish 72.2. Swapping only the judge model moved the score by 8 points on identical answers. Our grading is stricter on purpose, and every number ships with its receipts.
Two numbers, one benchmark.
Mem0 publishes 92.5 on LoCoMo. We publish 72.2, and ours comes with a named judge model (the same one that graded the answers, and as you'll see, graded them twice as strictly), plus raw outputs you can re-score in about a minute.
If you read those two numbers as a leaderboard, we lose by twenty points. We'd argue the comparison itself is broken, and that the twenty points are mostly the measurement, not the memory. Here is the evidence, then our actual numbers with the recipe attached, so you can decide for yourself.
The judge is most of the score
LoCoMo accuracy is not produced by the memory system alone. A language model reads each answer, compares it to a reference, and decides correct or wrong. That judge has its own mood: swap the grading rubric and the same answers get a different grade.
This is not a hypothetical. An independent analysis (Mnemoverse, June 2026) took one system's LoCoMo answers and re-graded them with two rubrics and the same judge model: the lenient rubric scored 0.901, the strict one 0.480. A 42-point swing with zero changes to the system being graded. They repeated the control on Mem0's own published answers: 0.741 lenient, 0.343 strict. Swapping the judge model moved the score by one or two points. The prompt did the rest.
The two controls agree within about two points. Generation cannot be the cause, because generation never changed. The only variable is the grading prompt, which means the canonical LoCoMo headline, scored under the lenient rubric, is judge-inflated for everyone who runs that eval, us included.
We then ran the control on ourselves with the judge model instead of the prompt. Same answerer (Claude Sonnet 5.5), same 1,540 questions, same retrieval, only the judge swapped from Haiku 4.5 to Sonnet 5.5. The headline moved from 80.4% to 72.2%: eight points on identical generation. The answerer's own measured behavior drifted too: hedge-style answers the judge credited fell from 104 to 27 between runs. The judge is not a footnote of the methodology; it is most of the headline.
So when a vendor publishes a LoCoMo number, the honest question is not "how good is the memory?" but "who graded the exam, and how forgiving was the answer key?"
Every published number in this space has moved
We are not picking on Mem0. Everyone's number has moved, because everyone reruns the benchmark on their own stack:
- Mem0: 66.9 in its own April 2025 paper (arXiv:2504.19413); 92.5 on its research page a year later (self-reported, different setup). Their paper also graded Zep at 65.99.
- Zep: objected to that grading, re-ran with its own implementation, and published 75.14 (May 2025); its research page now shows 94.7 (self-reported, August 2026).
- Hindsight: 92 on the neutral Agent Memory Benchmark (vectorize-io's board: same answerer and judge for every provider), the closest thing this space has to an apples-to-apples axis.
Three vendors, four numbers, at least four evaluation pipelines. The structural problem is that every follower runs its own judge, prompt, reader, and question subset, then presents the result as comparable. A single absolute LoCoMo number is the weakest signal you can read.
What we do differently
Our number is lower because our grading is harder, and we think that's the right trade:
- The judge is named and strict. Every answer is graded by a specific model, recorded in the run artifacts, under a rubric that does not give partial credit for vague answers.
- Abstaining on an answerable question counts as wrong. In our first end-to-end run, the judge credited 91 answers that were actually refusals on questions whose evidence was in the store. We caught it, reclassified them, and published both numbers: 80.4% as run, 74.5% corrected. The canonical figure is the stricter re-judged run, 72.2%. The correction note lives next to the number it corrects, permanently.
- Retrieval quality is measured without a judge at all. On LongMemEval-S (500 hand-curated questions, ~115k-token conversations), Uteke places the needed evidence in the top 5 for 98.2% of questions. That is checked against the answer key directly: no LLM in the loop, deterministic on one CPU.
- Every number ships with its ruler. Raw per-question outputs are committed to the repo. Verifying them takes a minute and no embedding model; re-running the full suite costs about $10 on commodity infrastructure. If you can't reproduce a vendor's number, it's a claim. Ours are measurements.
None of this makes 72.2 comparable to 92.5, and that is exactly the point. We label the judge, the rubric, and the abstention handling so nobody mistakes the grading for the memory.
What we actually claim
Strip the judge out and the claims get boring and solid, which is how claims should be:
- LongMemEval-S: recall@5 98.2%, strict (all gold evidence in top 5) 88.0%, zero misses in the top 50. Full 500 questions, receipts committed, re-validated after our most recent regression fix with the results published alongside the fix.
- LoCoMo retrieval: evidence in top 5 for 84.0% of questions, strict 71.6%, about 23 minutes end to end on CPU-only infrastructure, no LLM call in the retrieval path.
- Latency stays flat from 100 to 10k stored memories, because retrieval is local embeddings plus search: no network, no third-party API in the hot path.
If your agent needs memory that works offline, on cheap hardware, with numbers you can check yourself, that's the product. The benchmark theater is just there to keep us honest, publicly.
Sources: Mem0 paper (arXiv:2504.19413, Apr 2025), mem0.ai/research (accessed Nov 2026, self-reported), Zep paper (arXiv:2501.13956), Zep blog "Lies, Damn Lies & Statistics" (May 2025), getzep.com/research (Aug 2026, self-reported), vectorize-io Agent Memory Benchmark leaderboard, Mnemoverse judge-variance analysis (Jun 2026). Uteke numbers: benchmarks/longmemeval/RESULTS.md and benchmarks/locomo/RESULTS.md in the codecoradev/uteke repository, with committed raw outputs.