vecq: 4-Bit Vector Search That Fits in Your Pocket

Today we are releasing vecq-core 0.1.1 on crates.io. It is a training-free 4-bit vector quantization and search library written in pure Rust with zero dependencies. We built it for one reason: vector indexes on mobile and edge devices are too big, and the usual fixes pull in C++ toolchains that mobile builds do not want.

The pitch is simple. vecq is the SQLite profile for edge vector storage. Not a server database. Not a replacement for Qdrant or usearch on a beefy machine. It is the thing you reach for when your index has to live on a phone.

The problem

On-device AI keeps shipping embeddings. A 768-dimensional float32 vector is 3072 bytes. Ten thousand memories later, you are carrying a 30 MB index in app storage, and that is before the HNSW graph overhead. Compress with product quantization and you usually inherit a C++ dependency, a training step, and a build pipeline that fights your cross-compilation setup.

For on-device AI apps this shows up as a concrete question: can an app keep semantic recall without shipping a large index and without linking a C++ search library into the build?

What vecq does

vecq quantizes each float32 vector to 4-bit codes using a training-free scheme. No calibration dataset, no training pass, no model artifacts. Add vectors, search, persist, load. That is the whole API surface.

The numbers from our benchmark suite (768-dim EmbeddingGemma embeddings, 2000 index vectors, 100 queries, ARM64):

Read that honestly. vecq is not faster than usearch's HNSW. It is roughly 4x slower per query. The value is elsewhere: a 6x smaller index, zero dependencies, deterministic bit-identical results across every ARM64 device (we avoid FMA contraction so rankings never drift between phones), and a build that compiles anywhere Rust does.

On ARM64 it uses NEON 4-vector batch scoring with a bounded-heap top-k, so the brute-force scan stays flat and predictable. If brute force ever becomes the bottleneck at your scale, you do not need vecq, you need a server index.

Design decisions worth knowing

Training-free over accurate. 4-bit codes lose precision. Recall@10 of 0.958 is fine for on-device memory recall where you fetch a shortlist and re-rank. If you need exact search, keep float32.

Zero dependencies, period. The crate has no dependency tree. cargo add vecq-core pulls exactly one crate. That was a hard constraint from day one because dependency audits on mobile projects are painful enough already.

Determinism as a feature. Same index, same query, same ranking on every device. No floating point contraction, no platform-dependent reordering. When a user reports "result 3 changed", platform variance is never the suspect.

Format v1.1 is frozen. Index files written today will load tomorrow. Recovery path for anything lossy is re-embedding from the float32 source of truth, which stays in SQLite anyway.

Where it fits

Use vecq when the index lives on-device, dimensions are a few hundred to a thousand, the corpus is in the tens of thousands, and you value small files and simple builds over last-millisecond latency.

Do not use it as your server-side vector database. Qdrant, usearch, or a full ANN index will beat it on throughput at scale and keep recall near 1.0. vecq is the edge piece of that picture: the server keeps the master float32 vectors, and devices carry a compact .vecq projection for offline search.

Integration status

The first consumer is uteke, our local-first memory engine. We have opened an issue to add vecq as an optional search backend behind a cargo feature flag, keeping usearch as the default so self-hosted installs see zero change (uteke#1098). Non-breaking, opt-in, measured before it becomes a default anywhere.

Repository: github.com/codecoradev/vecq (Apache-2.0) Crate: crates.io/crates/vecq-core

If you are building on-device retrieval and the index size hurts, the benchmark methodology is in the repo. Reproduce the numbers yourself rather than trusting ours.