How Uteke Embeds 10K Memories in 5ms on CPU

How Uteke Embeds 10K Memories in 5ms on CPU

One of the numbers that surprises people when they first try Uteke is the recall latency. At 10,000 memories, semantic recall consistently returns results in under 5 milliseconds on a single CPU core. No GPU required. Here is how that works.

The Embedding Model

Uteke uses all-minilm-L6-v2 as its default embedding model — a 384-dimensional sentence transformer optimized for speed. It generates embeddings entirely on CPU using ONNX Runtime, which provides optimized inference without PyTorch overhead.

The model was chosen for its strong balance of quality and speed. At 384 dimensions, it is compact enough for fast distance computations while maintaining competitive retrieval quality against larger models.

HNSW Indexing

The real speed comes from HNSW (Hierarchical Navigable Small World) indexing. HNSW builds a layered graph structure where each memory node connects to nearby nodes. Queries traverse this graph from the top layer down, quickly narrowing the search space.

At 10K entries with 384-dimensional vectors, a well-tuned HNSW index can answer nearest-neighbor queries in microseconds. The 5ms measurement includes embedding the query text, traversing the index, and serializing the response.

Query Pipeline Breakdown

• Embed query text with all-minilm-L6-v2 via ONNX Runtime: ~3ms
• Traverse HNSW index for top-K neighbors: ~1ms
• Fetch full memory records from SQLite: ~0.5ms
• Serialize JSON response: ~0.5ms

Memory Footprint

The entire index for 10K memories fits in roughly 15MB of RAM — the HNSW graph plus the vector data. This means the index stays hot in L3 cache on most modern CPUs, eliminating main memory latency for most queries.

Scaling Considerations

As you grow beyond 10K memories, latency increases gradually rather than linearly. HNSW's logarithmic scaling means 100K memories might see 8-12ms recall, and 1M memories around 15-20ms. Still fast enough for real-time use.

CPU vs GPU

For the embedding step, a GPU would cut the 3ms to under 1ms. But for most agent workflows where recall happens once per conversation turn, the CPU performance is more than sufficient. The complexity and cost of GPU deployment is hard to justify at these latencies.

The embedding and indexing logic is open source — check the Uteke repository for implementation details.