4-Bit Vectors on the Edge: Where Recall Holds and Where It Gives

Ternary models had a good week. On September 16 a paper titled "Breaking the 1.58-bit Barrier for Ternary LLMs" hit Hacker News and collected 242 points, and a few days earlier Ternary Bonsai 2 showed a 27B model running from a file under 6GB. The pitch in both cases is the same: drop the bits, keep the behavior.

Vector search is converging on the same idea, and with less ceremony. vecq is our training-free quantization library: it compresses 768-dimension embeddings from 3,072 bytes to 642 bytes at 5-bit, or 514 bytes at 4-bit, with no training pass and zero dependencies beyond the Rust standard library. It shipped as v0.3.1 on crates.io on September 14.

The new release added something we consider more interesting than a feature: a 100K-vector benchmark profile, and an honest result inside it. At server scale, recall decays. This post walks through the numbers, because the shape of that decay tells you exactly where a 4-bit index belongs and where it does not.

The budget math

Compression numbers only mean something once you attach them to a device. Here is what storing dense embeddings costs in practice, computed straight from the byte counts (3072 bytes per vector at f32, 642 at 5-bit, 514 at 4-bit):

A Raspberry-class board or a phone has room for the 5MB end of that table. It does not have room for 300MB of a single index, next to the OS, your app, and everything else. Compression is what moves vector search from "runs on a server" to "ships inside the binary."

There is a second constraint people miss: determinism. vecq writes a single file and the same file plus the same query produces bit-identical results on any platform. On an edge device you cannot debug "the index answered differently last night." Either the result is reproducible or you cannot trust it at all.

What v0.3.1 measured

The release added a seeded 100K-vector dataset generator and a server_scale benchmark suite that measures three modes against exact f32 cosine ground truth: plain 5-bit, Matryoshka truncation to 256 dimensions, and the 2-bit cascade. Three findings came out of it.

First, compression holds. At 100K vectors the ratio stays at 4.78x, matching the README numbers measured at 2K. The bytes you save do not depend on how many vectors you store.

Second, recall is N-dependent. On the 2K-vectors profile that most README numbers use, 5-bit recall@10 is 0.979. At 100K it drops to 0.850. The 4-bit width on the small profile is 0.958; residual mode trades some bytes back for 0.984. Nothing changed in the algorithm. What changed is that as the dataset grows, the margin between the true nearest neighbor and its closest impostor collapses, so any lossy code has less room to be wrong before it ranks the wrong vector.

Third, the measured sweet spot is the local profile: up to around 10K vectors, where 5-bit holds r@10 0.932 at 18ms per query on a single thread and 4-bit stays above 0.95 at 2K. The release notes say the 2-bit cascade is not a single-threaded throughput win at server N, and we agree with the measurement rather than defending the feature.

Where 4-bit belongs

That benchmark line, up to ~10K vectors, is not a limitation buried in a footnote. It is the actual product boundary, and it happens to match where edge workloads live.

A personal agent's memory: thousands of notes, decisions, and facts, not millions. A photo library's semantic index for one user. A voice assistant's wake-word context. A config file explaining itself. These are all 1K to 10K vector problems, the exact range where a 514-byte vector with 0.958 recall fits in RAM with room to spare.

One file, mmap-readable

At that scale you also get properties server engines cannot offer. The index is one file you can ship inside an app bundle, back up by copying, and read back through a zero-copy mmap view that is ready about 76x faster than a full load at 12K vectors. vecq is the search layer below uteke, our local-first memory engine, where it is available as an optional backend for mobile and embedded deployments.

The trade-offs, stated plainly

You give things up, and the README says so before we have to. Recall at 4-bit is 0.958, not 0.995; if every last point matters, use f32 or a residual mode. There is no payload filtering and no server-scale throughput. vecq is deliberately small. It is a good SQLite profile, and it is not a Qdrant replacement.

The honest version of the pitch: if your vectors live on a server and number in the millions, run a real vector database. If your vectors live on a device, in the thousands, and they need to fit inside a binary you control, the 4-bit file format is hard to beat.

That boundary is also why the 100K decay number matters more than any headline recall figure. It is the difference between marketing a library and describing one.

Try it

The crate is vecq-core on crates.io (0.3.1), Apache-2.0, with docs and the full benchmark methodology in the repo. Everything in this post reproduces from that suite: cargo run --release -p vecq-bench --bin real for recall and latency, vs_usearch for the head-to-head. Repo: github.com/codecoradev/vecq. Star count as of writing: 1. It would like more.