Semantic Search Alone Found 5 of 9. With Keywords: 9 of 9.

Last week an open-source Mac app called DigUp hit the front page of r/LocalLLaMA with 1,791 upvotes. It runs Google's EmbeddingGemma 2 embedding model entirely on-device and lets you search your own files by meaning: "the clause about pets in the lease" jumps to the right PDF page, "where they talk about sleep" lands on the right minute of a podcast.

The thread is worth reading for more than the demo. In the comments, the developer published an evaluation number most search marketing would never volunteer, and it is the most honest lesson about retrieval we have seen a community validate in months.

The developer's own numbers

A commenter who had built something similar raised the obvious concern: semantic search misses when humans remember the abstract, and exact lookups like error codes tend to fail. The developer's answer, verbatim:

"It isn't semantic-only. Keyword search over file names, OCR text and page and document text runs alongside the model, and the scores are combined. On my evals, the model alone found 5 of 9 exact lookups like error codes; with keywords, all 9."

Five out of nine, then nine out of nine. The difference between a frustrating search experience and a useful one was not a bigger model. It was a keyword index running alongside the vectors, with the scores fused.

Why semantic-only fails when it should shine

Embeddings are built to blur. That is the trick: "deploy v2.1" and "ship the release" land near each other in vector space, which is what you want for meaning-based lookup and exactly what you don't want when someone needs the string ERR_0x4F2. Error codes, part numbers, function names, dates: these are token-exact queries, and a 768-dimensional meaning space smooths them into noise.

This is not an edge case. Exact-token lookups are a large share of real developer searches, and a retrieval system that aces "how do I avoid double charges" while flunking "find the idempotency key note" will feel broken in daily use.

Fusion, not mode-picking

The fix is not a settings toggle. Users cannot and should not classify their own queries. The fix is running both retrievals on every query and merging the rankings: vectors catch paraphrase, full-text (SQLite FTS5 in practice) catches exact tokens, and a fusion step such as Reciprocal Rank Fusion merges both lists into one ordering.

That is how we built recall in Uteke, our agent memory engine: vector and FTS5 run in parallel and fuse into a single ranking, the default since v0.16.0, because "no mode to pick" is a feature. On LongMemEval-S that design scores 98.2% recall@5 with zero LLM calls in the retrieval path, raw outputs committed to the repo. We are the builders, so treat the benchmark as our claim; the DigUp numbers are the independent version of the same lesson.

A four-point checklist for your own retrieval eval

1. Include exact-token queries in your eval set. Error codes, identifiers, names, numbers. Paraphrase-only evals measure the easy half.
2. Never ship semantic-only. A keyword index over the same content is cheap (FTS5 ships inside SQLite) and covers what embeddings structurally miss.
3. Fuse at the ranking layer, not the UI. One query, one merged result list, no mode switch.
4. Report the misses. "5 of 9, then 9 of 9" builds more trust than any hero number.

The next time a tool advertises "AI-powered semantic search," the useful question is the one this thread answered for free: what happens on the queries where meaning is not the point?