An 87GB model on a CPU: what the quantization math buys you

A llama.cpp benchmark runs Qwen3.8-Flash-Next from CPU-only inference to 96GB of VRAM. The scaling curve has a lesson for embedding search too.

The run that shouldn't work

A post on r/LocalLLaMA this week benchmarks Qwen3.8-Flash-Next in llama.cpp across every VRAM tier from CPU-only to a full 96GB RTX PRO 6000. The GGUF file weighs 87.2 GiB, and that is already the compressed number: the model runs as a 4-bit quant (UD-IQ4_XS). CPU-only decode lands at 8.34 tok/s. With all 96GB available, it reaches 109.07.

Eight tokens per second on a CPU sounds like a party trick. It is closer to a proof. An 87GB model doing interactive chat on a Ryzen 9 9950X, no GPU in the loop, exists for one boring reason: the weights are 4-bit.

What the scaling curve says

The author scaled usable VRAM on one card by capping the pool a helper process left available. Decode at a 2K prompt:

Usable VRAM	Decode
CPU-only	8.34 tok/s
8GB	35.69
24GB	39.01
48GB	51.73
96GB	109.07

Two findings worth keeping.

Capacity matters less as context grows. At a 2K prompt, 96GB beat 24GB by 2.80x. At 245K tokens of context the advantage shrank to 1.45x (21.61 vs 14.89 tok/s). Every configuration slows down at long context and the curve flattens. Plan memory for the context you actually run, not the one in the headline.

Placement can beat capacity. Forcing the model's 27.2 GiB per-layer embedding table onto the GPU dropped decode from 108.5 to 1.95 tok/s in this build. Leaving the table in system RAM was 55.6x faster. The author verified the tensor moved and ruled out test-order effects, and is careful to say the root cause is unproven. The operational lesson stands anyway: more VRAM is not automatically a faster model. Where a tensor lives matters as much as how much room you have.

One honest caveat, stated by the author himself: the small-VRAM tiers simulate capacity on one big card. A real 8GB GPU has less bandwidth than a capped 96GB one. Treat the tier table as an offload study, not a shopping guide.

The domino everyone skips

It is tempting to read this as a hardware scaling story. The first domino is the quant. At 16-bit weights the same model would sit around four times larger, north of 300GB. No consumer setup loads that, CPU or GPU. The CPU-only run at 8.34 tok/s is downstream of a compression decision made before the first benchmark started.

The MoE architecture helps too: only about 6B parameters activate per token, which is why CPU decode is tolerable rather than miserable. But active parameters change compute, not storage. The file still has to fit somewhere. That is the quant's job.

Embeddings hit the same wall, just lower

If you run local AI, the model file gets the headlines and the vector index quietly eats your storage. Every embedding at f32 with 768 dimensions costs 3,072 bytes. A year of agent memory at 100k vectors is roughly 300MB before metadata. On a server that rounds to zero. On a phone it is a real number.

vecq, our Apache-2.0 Rust crate (vecq-core on crates.io), applies the same fix to embeddings. Rotate the vector, quantize against fixed Lloyd-Max tables, store packed codes with per-vector scales. No training, no calibration set. Numbers from our benchmark on real EmbeddingGemma vectors, 2,000 base + 100 queries at dim 768:

mode	bytes/vector	compression	recall@10	ms/query
f32 (usearch HNSW)	3,072	1x	0.995	0.23
vecq 5-bit (default)	642	4.78x	0.979	3.21
vecq 4-bit	514	5.98x	0.958	0.89
vecq 4-bit + residual	1,028	2.99x	0.984	1.76

The shape matches the LLM story: a 4 to 6x smaller footprint with recall you can reason about, and a documented trade somewhere else.

What we give up, on the record

vecq search is brute force, and HNSW is about 14x faster at our default width (3.21 vs 0.23 ms/q on the same dataset). On a server with millions of vectors, use HNSW. vecq targets the other surface: on-device indexes in the thousands to low tens of thousands of vectors, where 1 to 3 ms per query is interactive and 5x smaller storage decides whether the feature ships. Building the index is also 14x faster than HNSW construction (75 ms vs 893 ms), and scoring is deterministic: the same file produces identical results on any platform, which matters when a phone and a server need to agree on the top 10.

The takeaway

Quantization keeps paying for everything downstream. It put an 87GB model on a CPU. It puts a year of agent memory in about 64MB. The next time a local AI feature looks impossible, check which numbers are still stored at full precision.

vecq-core is on crates.io, source and benchmark docs on GitHub: