LongMemEval and Agent Memory Benchmarks, Read Honestly

Every memory vendor has a number now

Ask three memory providers how good their recall is and you will get three percentages, all plausible, all attached to the same handful of benchmark names: LongMemEval, LoCoMo, and whatever the vendor built in-house. 2026 made this noisier in a productive way. A second-generation LongMemEval shipped in May, an independent multi-dataset harness went public in March, and vendor benchmark reports now compete for the same search results as the academic papers. The numbers look comparable. Most of them are not.

This post maps what each benchmark measures, why two scores carrying the same name can sit twenty points apart for legitimate reasons, and the questions worth asking before you repeat any of these numbers, including ours.

Who measures what

LongMemEval (ICLR 2025, from UCLA and Tencent AI Lab researchers) put agent memory evaluation on a shared footing: 500 questions, each answered over a long history of prior chat sessions, with facts that update, contradict, and demand temporal reasoning. The official metric is end-to-end question answering. A system reads its retrieved memories, generates an answer, and the answer gets graded. Retrieval quality and answer quality travel together in that score.

LongMemEval-V2 (May 2026) points the same idea at a different subject. Instead of chat histories it scores web-agent trajectories: up to 500 trajectories, 115 million tokens of accumulated environment experience, 451 questions across five memory abilities. The question shifts from "did the assistant remember the user" to "did the agent turn what it did into reusable experience." The dataset is on Hugging Face, the paper is on arXiv (2605.12493), and the authors run a public leaderboard.

The Agent Memory Benchmark takes a third approach: one open harness, seven datasets, providers evaluated under identical settings, with scoring logic and raw results in a public repository. Its datasets span personal conversations (LoCoMo among them), long-context agent trajectories (beam, up to 10 million tokens), and a coding benchmark whose most honest entry is a vanilla agent with no memory system at all, scoring the same 100 percent as the memory-equipped one on that task.

Vendor reports complete the picture. Redis published a LongMemEval report in September quoting 86.5 percent task-averaged accuracy. Reports like this one are useful and self-selected at the same time: the vendor chose the protocol, the models, and the comparison set, then reported against its own choices.

Why same-name scores don't compare

Two teams can both say "92 percent on LongMemEval" and both be right. The phrase hides at least four decisions that move the number.

Retrieval versus answers. A retrieval-only score asks whether the right memories surfaced in the top-k. The official LongMemEval metric grades final answers. The same system can look strong on one and ordinary on the other, because a model reading slightly wrong memories still sometimes writes the right answer, and a model reading perfect memories can still fumble the response.

Configuration. Which model reads the memories, how many memories get injected, which embedding model runs, and which split of the dataset you use all change the outcome. Papers like Zep's temporal knowledge graph architecture (arXiv 2501.13956) showed a year ago that the LLM doing the reading moves end-to-end scores by whole grade levels on identical retrieval.

Modes. The open harness runs a plain RAG mode, an agentic mode where the model chooses when to recall, and a retrieval mode with no model in the scoring path at all. A provider can legitimately report a different number for each, and all three can be honest.

Who ran it. Self-reported numbers dominate this field. On the open harness's LongMemEval split, the top entry scores 94.6 percent against 74.0 for a plain hybrid-search baseline. That gap is real and worth knowing about, and so is this: the harness is maintained by a company that also ships a memory product, which currently tops most of its own tables. Neutrality of the code is not neutrality of incentives, and a careful reader prices both.

A field guide for reading any memory number

Five questions separate a claim you can use from one you should scroll past.

1. Which metric? Retrieval recall and answer accuracy are different claims. Ask which one the number measures before comparing it to anything.
2. Which protocol? Same dataset, different model reading the memories, different top-k, different split. If the config is not published, the number is a mood, not a measurement.
3. Where are the raw scores? A leaderboard you can re-run beats a chart you can only screenshot. Check whether per-question results are committed somewhere.
4. What is the baseline? The most useful columns in any memory comparison are the boring ones: keyword search, grep, a vanilla agent. When plain retrieval matches the fancy pipeline, the fancy pipeline owes you an explanation.
5. Who paid for the run? Not a disqualifier. Just something to know.

Our own row in the table

We publish Uteke, a local-first memory engine, and we hold our own numbers to the standard above. On LongMemEval-S, the short split of the 2025 benchmark, our committed harness scores 98.2 percent recall_any@5 (the needed evidence lands in the top 5 results) and 88.0 percent on the stricter recall_all@5 (every needed piece, against a 99.4 percent ceiling imposed by the dataset itself). Warm queries return in roughly 45 milliseconds with zero LLM tokens in the retrieval path, on CPU, fully offline.

Note the honest edges of that claim. It is a retrieval score, not an end-to-end answer score. It is LongMemEval-S, not V2, which targets web-agent trajectories we have not run. We publish the full harness, the raw per-question outputs, and two posts about trying to break our own results (here and here), plus a head-to-head comparison against ChromaDB and Mem0 on our own benchmark (link) and the offline-memory comparison across Mem0, Letta, and Zep (link). Audit us the same way you would audit anyone in this field. That is the point of writing this post.

Frequently asked questions

Is LongMemEval-V2 a replacement for LongMemEval?

No. V2 evaluates memory for web agents working in environments, using trajectories instead of chat histories. The original remains the standard for conversational assistant memory. They measure neighboring things, and both matter as agents move from chat into tool use.

What is the Agent Memory Benchmark?

An open, reproducible harness that runs memory and retrieval providers across seven datasets under identical settings, from personal conversations to million-token agent trajectories. Datasets, prompts, and scoring logic are public, so any provider, including ours, can be added and compared.

Why do vendors report different LongMemEval scores?

Because "a LongMemEval score" is not one measurement. Metric type (retrieval versus answers), the model reading the memories, the number of memories injected, the split, and the run mode all move the result. A score without its protocol is an anecdote.

Agent memory became the feature of 2026 for a reason, and the evaluation tooling finally catching up is good news for everyone building it. The numbers will keep multiplying. The discipline of asking what a number measures, who produced it, and whether you can re-run it costs nothing, and it is the only reliable filter we know of (why memory became the feature of the year).