How we built Uteke's hybrid memory architecture

AI agents need memory. Without it, every conversation starts from zero. The question is how to build it so it works in production without a cloud dependency, a heavy database, or a billing surprise.

We built Uteke to solve this. It is a local-first semantic memory engine written in Rust, currently at v0.10.1 with 150 stars and 40+ releases shipped in 58 days. This post covers the architecture decisions, the trade-offs, and a few things that did not work.

The problem

Hermes agents run continuously. They handle research, write code, coordinate across profiles, and manage long-running projects. None of that works without persistent memory that goes beyond simple key-value storage.

We needed three things. First, structured recall: look up by ID, list by tag, filter by namespace. Second, semantic recall: find memories related to a query even when no keywords match. Third, isolation: multiple agents on the same machine must maintain separate memory spaces.

Existing options fell short. Cloud vector databases like Pinecone, Weaviate, and Qdrant Cloud introduced network latency and external dependencies. Running a full Qdrant instance locally worked, but the resource overhead was excessive. Plain SQLite gave us structured queries but no semantic search. We needed both, in a single binary, with no external services.

Architecture overview

Uteke runs as a single Rust binary with an HTTP API. In production it runs as a container at http://uteke:8767, authenticated via UTEKE_TOKEN. There are two operational modes.

Pure vector mode embeds the query and runs a vector similarity search against stored embeddings. Fast, simple, good for cases where you only need "find me things related to this."

Full pipeline mode runs the full recall chain. It parses the query, checks for exact ID matches, runs structured filters (tag, namespace, entity), then falls through to vector search for the remaining results. This is the default and what most agents use.

The full pipeline is what makes Uteke useful in practice. Pure vector search answers fuzzy questions well but cannot do "get me all memories tagged decision in the project-x namespace." The full pipeline handles both.

SQLite plus vector search

The hybrid approach is the core design decision. Uteke stores all memory metadata in SQLite: IDs, timestamps, tags, namespaces, entities, content text, and room assignments. Vector embeddings live in a separate structure optimized for similarity search.

When a recall request arrives in full pipeline mode, the engine runs structured filters first. Tag and namespace constraints become SQL WHERE clauses, eliminating irrelevant records before the expensive vector computation. The remaining candidates get embedded and compared against the query embedding.

This ordering matters. Early in development we tried vector search first, then filtering. It was simpler to implement but produced worse results. Vector similarity does not respect hard constraints. A semantically similar memory in the wrong namespace would rank highly and push out a correctly scoped one. Filtering first guarantees that vector search only considers eligible candidates.

The SQLite database at 10,000 memories is roughly 4.5MB, including all metadata, content, and indexes. The small footprint means the entire database fits in the OS page cache.

Room-based organization

Rooms are Uteke's isolation primitive. A room is a named partition within a single Uteke instance. Each room has its own memory set, its own namespace, and its own search scope.

This exists because we run multiple Hermes agents on the same machine. A coding agent and a research agent should not see each other's memories unless explicitly configured to do so. Rooms solve this without running separate Uteke instances.

We initially considered running a separate Uteke container per agent. That would have given stronger isolation but would have multiplied memory usage for the embedding model. One model, one process, partitioned by rooms, was the better trade-off.

Why local-first

The embedding model is 188MB and runs entirely on-device. No API calls, no network round-trips for inference, no rate limits, no token billing.

This was a hard requirement. Hermes agents run on a single Oracle Cloud ARM instance. Adding a dependency on an external embedding API would have introduced a network latency floor, an external failure mode, and a per-query cost that scales with usage.

Running locally also means Uteke works on laptops, on air-gapped machines, and in environments without reliable internet. The trade-off is that the embedding model is fixed and cannot be swapped for a larger one without rebuilding. A single good-enough model that always works is better than a theoretically better model that requires a network connection.

Performance numbers

Average recall latency across the full pipeline is about 45ms. That includes query parsing, SQL filtering, query embedding, vector comparison, and result ranking.

The model loads once at startup and stays resident. The SQLite database at 10K memories is small enough that structured filters run in microseconds. We have not yet benchmarked the scaling ceiling, but structured filters should keep performance stable as the database grows since they prune candidates before vector search runs.

What did not work

Early prototypes used an in-memory vector store. Fast, but lost all data on restart. We moved to persistent storage quickly.

We also tried filtering vector results after search rather than before. Semantic similarity does not respect structural constraints, so a relevant memory in the wrong namespace would rank highly and push out a correctly scoped one. Switching to filter-first was one of the most impactful changes.

Tag-based-only recall was another dead end. Agents do not consistently tag their memories, so many relevant results were invisible to recall. Vector search catches those untagged memories by content similarity.

What is next

The near-term roadmap focuses on three areas. First, consolidation tooling to merge duplicate memories and resolve contradictions automatically. Second, namespace hierarchies to support deeper organizational structures. Third, improved benchmarking at scale to find the practical limits of the current architecture.

Uteke is open source under Apache-2.0. The code is on GitHub, and you can install it with Homebrew:

brew install codecoradev/tap/uteke

Or run it as a container. Either way, you get a local-first semantic memory engine that runs in a single binary, keeps your data on your machine, and recalls memories in under 50 milliseconds.