A 10 million document corpus takes 31 GB of RAM as float32. turbovec fits it in 4 GB - and searches it faster than FAISS.
turbovec is a Rust vector index with Python bindings, built on Google Research's TurboQuant algorithm — a data-oblivious quantizer with near-optimal distortion and no separate training phase.
- Online ingest. Add vectors, they're indexed — no train step, no parameter tuning, no rebuilds as the corpus grows.
- Fast SIMD search. Hand-written kernels — NEON SDOT/SMMLA on ARM, AVX-512 VNNI and
vpermb on x86, with AVX2 and scalar fallbacks — beat FAISS IndexPQFastScan in every measured config, averaging 3.4× at 4-bit and 23% at 2-bit across the eight cells of each width, on both architectures.
- Incremental saves.
sync(path) persists just what changed since the last sync — one fsync per call, crash-safe at any byte, and a removal or a small append costs milliseconds however large the index. write/load stay for whole-file snapshots.
- Filter at search time. Pass an id allowlist (or a slot bitmask) to
search() and the kernel honours it directly. You always get up to k results from the allowed set — no over-fetching, no recall hit on selective filters.
- Pure local. No managed service, no data leaving your machine or VPC. Pair with any open-source embedding model for a fully air-gapped RAG stack.
Building RAG where privacy, memory, or latency matters? You're in the right place.
Python
pip install turbovec
from turbovec import TurboQuantIndex
index = TurboQuantIndex(dim=1536, bit_width=4)
index.add(vectors)
index.add(more_vectors)
scores, indices = index.search(query, k=10)
index.write("my_index.tv")
loaded = TurboQuantIndex.load("my_index.tv")
index.sync("my_index.tv") # after more changes: durable incremental save
vectors and query are 2-D float32 arrays of shape (n, dim) — other dtypes are rejected rather than silently converted, so cast with np.asarray(x, dtype=np.float32) first if needed.
Need stable ids that survive deletes? Use IdMapIndex:
import numpy as np
from turbovec import IdMapIndex
index = IdMapIndex(dim=1536, bit_width=4)
index.add_with_ids(vectors, np.array([1001, 1002, 1003], dtype=np.uint64))
scores, ids = index.search(query, k=10) # ids are your uint64 external ids
index.remove(1002) # O(1) by id
index.write("my_index.tvim")
loaded = IdMapIndex.load("my_index.tvim")
index.sync("my_index.tvim") # durable incremental save, ids included
Hybrid retrieval (filtered search)
Restrict results to a candidate set produced by another system (SQL, BM25, ACL, time window, …):
import numpy as np
from turbovec import IdMapIndex
idx = IdMapIndex(dim=1536, bit_width=4)
idx.add_with_ids(vectors, ids)
# Stage 1: external system narrows to candidate ids.
allowed = np.array(db.execute("SELECT id FROM docs WHERE tenant=?", (t,)).fetchall(),
dtype=np.uint64)
# Stage 2: dense rerank within the candidate set.
scores, ids = idx.search(query, k=10, allowlist=allowed)
Filtering happens inside the SIMD kernel at 32-vector block granularity: blocks with no allowed slots are short-circuited before any LUT lookup or scoring work, and individual non-allowed slots inside scored blocks are dropped at heap-insert. Selective allowlists (small fraction of the index allowed) therefore avoid most of the SIMD cost rather than paying it and discarding the result afterwards.
The output length is min(k, n_allowed), where n_allowed counts distinct allowed vectors — when fewer vectors are allowed than k you get exactly that many results rather than padded fallbacks.
See docs/api.md for the full reference.
Framework integrations
Drop-in replacements for the in-tree reference vector / document stores in each framework. Same public surface, same persistence semantics, same retriever and pipeline wiring — swap the import and keep your pipeline.
- LangChain —
pip install turbovec[langchain] · replaces langchain_core.vectorstores.InMemoryVectorStore
- LlamaIndex —
pip install turbovec[llama-index] · replaces llama_index.core.vector_stores.SimpleVectorStore
- Haystack —
pip install turbovec[haystack] · replaces haystack.document_stores.in_memory.InMemoryDocumentStore
- Agno —
pip install turbovec[agno] · replaces agno.vectordb.lancedb.LanceDb
Rust
cargo add turbovec
use turbovec::TurboQuantIndex;
let mut index = TurboQuantIndex::new(1536, 4).unwrap();
index.add(&vectors);
let results = index.search(&queries, 10);
index.write("index.tv").unwrap();
let loaded = TurboQuantIndex::load("index.tv").unwrap();
For stable external ids that survive deletes:
use turbovec::IdMapIndex;
let mut index = IdMapIndex::new(1536, 4).unwrap();
index.add_with_ids(&vectors, &[1001, 1002, 1003]).unwrap();
let (scores, ids) = index.search(&queries, 10);
index.remove(1002);
index.write("index.tvim").unwrap();
let loaded = IdMapIndex::load("index.tvim").unwrap();
Recall
TurboQuant vs FAISS IndexPQ (LUT256, nbits=8) — the paper's Section 4.4 baseline. 100K vectors, k=64. FAISS PQ sub-quantizer counts sized to match TurboQuant's bit rate (m=d/4 at 2-bit, m=d/2 at 4-bit).
The charts plot calibrated TurboQuant (TQ+). Across OpenAI d=1536 and d=3072, TQ+ beats FAISS at R@1 on three of four cells (by 0.9–2.9 points; d=1536 4-bit trails by 0.7), and both reach 1.0 by k=8 (≥0.997 already at k≤4). GloVe d=200 is the harder regime — at low dim the asymptotic Beta assumption is looser. TQ+ lands ahead of FAISS at R@1 at both bit widths (+1.9 at 4-bit, +0.8 at 2-bit), with FAISS keeping a slim edge at 2-bit from k≈8. Uncalibrated numbers are in the JSONs (tq_recalls).
A note on baselines. We compare against FAISS IndexPQ (LUT256, nbits=8, float32 LUT) because it's the default production-grade PQ most users would reach for. This is a stronger baseline than the custom u8-LUT PQ in the TurboQuant paper — FAISS uses a higher-precision LUT at scoring time and k-means++ for codebook training. We reproduce the paper's TurboQuant numbers on OpenAI d=1536 / d=3072 and hit similar numbers to other community reference implementations on low-dim embeddings (see turboquant-py at d=384). On GloVe (d=200) — the low-dim regime where the asymptotic Beta assumption is loosest — TurboQuant lands ahead of FAISS at 4-bit but trails it at 2-bit; TQ+ calibration recovers the 2-bit deficit at R@1 (0.572 vs FAISS's 0.564), with FAISS keeping a slim edge at deeper k.
Full results: d=1536 2-bit, d=1536 4-bit, d=3072 2-bit, d=3072 4-bit, GloVe 2-bit, GloVe 4-bit.
Compression
Search Speed
All benchmarks: 100K vectors, 1K queries, k=64, median of 5 runs.
ARM (GCP c4a-standard-8, Google Axion, 8 vCPUs)
On ARM, TurboQuant beats FAISS FastScan in every config, averaging 3.5× at 4-bit (3.4–3.7× across cells — the SDOT/SMMLA dot-product kernels score the vector-major layout directly) and 26% at 2-bit (22–29%).
x86 (Intel Xeon Platinum 8481C / Sapphire Rapids, 8 vCPUs)
On x86, TurboQuant wins every config, averaging 3.4× at 4-bit (3.2–3.5× across cells — the AVX-512 VNNI dot-product kernel on the vector-major layout) and 20% at 2-bit (5–32%), where the vpermb LUT scan carries the short 2-bit accumulate loop.
Insertion & Removal Latency
Same corpus as the search cells: 100K OpenAI vectors, median of 5 runs, timed loops including the Python-call overhead a caller actually pays per op. Insertion measures per-vector add() latency on a warm, populated index (built untimed) at n=1 — a single-vector add() — and n=100 — a 100-vector batch, showing how far batching amortizes the per-call overhead — against add() into the trained, populated FAISS IndexPQFastScan (training untimed). A single add() lands in 6.3–19.7 µs depending on the cell (7.6–13.9× faster than a FAISS single add), and a 100-vector batch amortizes TurboQuant to 4.6–16.3 µs/vector (4.6–15.1× faster than the same batch into FAISS). Removal measures per-op remove-by-id latency at n=1 (the steady per-op rate over 1000 removes) and n=100 (the first 100 removes on a fresh index): IdMapIndex.remove(id) — O(1) swap-and-pop plus the id-map bookkeeping — lands at 0.44–1.22 µs and 0.59–1.37 µs per op across the cells. The FAISS column is the same user-visible operation, remove_ids on an IndexIDMap over IndexPQFastScan, which repacks the stored codes on every call: 0.19–1.02 s per single remove at 100K, with cost doubling alongside code size — which is why the removal charts use a log-scale axis. Charts show the single-threaded cells (RAYON_NUM_THREADS=1); the _mt cells are measured too and match at n=1, since a single add is serial. Scripts: benchmarks/suite/.
ARM (GCP c4a-standard-8, Google Axion, 8 vCPUs)
Full results: d=1536 2-bit insert, d=1536 4-bit insert, d=3072 2-bit insert, d=3072 4-bit insert, and the matching speed_remove_* and _mt files.
x86 (Intel Xeon Platinum 8481C / Sapphire Rapids, 8 vCPUs)
Full results: d=1536 2-bit insert, d=1536 4-bit insert, d=3072 2-bit insert, d=3072 4-bit insert, and the matching speed_remove_* and _mt files.
Save & Load
Same corpus as the search cells: 100K OpenAI vectors, median of 5 runs. TurboQuant serializes to a single .tv file with an fsync + atomic rename; FAISS is write_index / read_index on the precision-matched IndexPQFastScan (sub-quantizer count matched to TurboQuant's bit rate, as in the search cells). Save (warm) is a write after a search has run, so the blocked layout cache is populated. Load → first search opens a fresh index and times the first query — separating bare deserialization (the page cache is warm throughout, so this is layout work, not cold-storage I/O) from the first-query cost. Round-trip chains the checkpoint/resume cycle an embedding store actually pays — mutate 1K vectors → save → reopen → serve the first query; FAISS has no measured equivalent for this path, so it is shown for TurboQuant only. On the smaller payloads the round-trip can come in below the isolated post-mutation ("dirty") write: the two are timed in separate suite steps, and at small file sizes the standalone fsync in the dirty-write step dominates and inflates it — a measurement artifact of the harness, not a repack win in the combined path. Single-threaded cells pin RAYON_NUM_THREADS=1. Scripts: benchmarks/suite/.
ARM (GCP c4a-standard-8, Google Axion, 8 vCPUs)
Full results: d=1536 2-bit persist ST, MT, d=1536 4-bit persist ST, MT, d=3072 2-bit persist ST, MT, d=3072 4-bit persist ST, MT.
x86 (Intel Xeon Platinum 8481C / Sapphire Rapids, 8 vCPUs)
Full results: d=1536 2-bit persist ST, MT, d=1536 4-bit persist ST, MT, d=3072 2-bit persist ST, MT, d=3072 4-bit persist ST, MT.
How it works
Each vector is a direction on a high-dimensional hypersphere. TurboQuant compresses these directions using a simple insight: after applying a random rotation, every coordinate follows a known distribution -- regardless of the input data.
1. Normalize. Strip the length (norm) from each vector and store it as a single float. Now every vector is a unit direction on the hypersphere.
2. Random rotation. Multiply all vectors by the same random orthogonal matrix. After rotation, each coordinate independently follows a Beta distribution that converges to Gaussian N(0, 1/d) in high dimensions. This holds for any input data -- the rotation makes the coordinate distribution predictable.
3. Per-coordinate calibration (TQ+). The Beta distribution from step 2 is asymptotic — at finite dimensions, individual coordinates drift from the canonical shape (especially low-bit and word-vector-style embeddings). TQ+ fits two scalars per coordinate — a shift and a scale — mapping each coordinate's empirical quantiles onto the codebook's outermost centroids. The probability level comes from the codebook, so it tracks the bit width (0.933 at 2-bit, ~0.996 at 4-bit) rather than being fixed. The Lloyd-Max codebook then quantizes against the target distribution it was designed for. The fit is explicit: call index.calibrate(sample) once with a random, representative sample of your vectors (1024 rows is enough — a draw of that size matches fitting on the whole corpus) before adding; afterwards the calibration is committed and reused by every add — no retraining, no rebuilds, no separate train phase. An index you never calibrate is plain TurboQuant. index.calibration_state reports "uncalibrated" or "calibrated". Recall gain: up to +2.2pp at @1 on the cells that drift most (e.g. GloVe at 2-bit).
4. Lloyd-Max scalar quantization. Since the distribution is known, we can precompute the optimal way to bucket each coordinate. For 2-bit, that's 4 buckets; for 4-bit, 16 buckets. The Lloyd-Max algorithm finds bucket boundaries and centroids that minimize mean squared error. These are computed once from the math, not from the data.
5. Bit-pack. Each coordinate is now a small integer (0-3 for 2-bit, 0-15 for 4-bit). Pack these tightly into bytes. A 1536-dim vect
一个向量索引库，用于在嵌入向量上做最近邻搜索，用 Rust 编写并带有 Python 绑定。它基于 Google Research 的 TurboQuant 量化技术，面向注重内存、延迟和隐私的 RAG 场景。