turboquant_plus README

The project's README from github.com/TheTom/turboquant_plus. As one markdown file: README.md.

TurboQuant+

🚀 TurboQuant KV cache compression is now in vLLM (PR #38479, merged April 2026): --kv-cache-dtype turboquant_k8v4 and friends, with fused Triton store/decode kernels. The PR discussion drew on the asymmetric K/V findings from this repo. Upstream llama.cpp has merged the core idea too: Hadamard KV cache rotation (#21038, citing TurboQuant directly) with fast WHT kernels on CPU (#22631), CUDA (#23615), and Vulkan (#23687). Rotation + the stock q4_0 cache is essentially turbo4's rotation stage; the PolarQuant codebook, norm extraction, and asymmetric policies remain here and in the fork.

Getting Started Guide | Configuration Recommendations | Benchmarks | Commercial Support

Implementation of TurboQuant (ICLR 2026) with implementation work, experiments, and follow-on findings beyond the base paper. Compresses transformer KV cache 3.8-6.4x using PolarQuant + Walsh-Hadamard rotation, at near q8_0 prefill speed and ~0.9x decode throughput at long context. Validated end-to-end from 1.5B to 104B at 128K context on a MacBook (turbo3, PPL 4.024, 74 GB peak memory).

This repository is the research home: the Python reference implementation, the validation papers, and the benchmark data. To run TurboQuant in an inference engine, pick from the table below. Pieces that prove useful and stable get upstreamed incrementally as small, reviewable patches.

Run It Today

Engine Platform Status Notes
vLLM CUDA / ROCm, datacenter Upstream, merged --kv-cache-dtype turboquant_k8v4 and friends (PR #38479)
llama.cpp All backends Upstream, rotation merged Hadamard KV cache rotation (#21038) + fast WHT kernels (CPU #22631, CUDA #23615, Vulkan #23687). Rotation + q4_0 cache approximates turbo4; full PolarQuant codec is in the fork below
llama-cpp-turboquant Metal, CUDA, HIP, CPU Production fork turbo2/3/4 KV cache + TQ3_1S/TQ4_1S weight formats; prebuilt binaries for Mac (Metal) and Windows (CUDA)
mlx-swift-lm Apple Silicon, Swift Upstream, merged Merged into Apple MLX (PR #232): full asymmetric family (turbo0v*/turbo8v*) + symmetric turbo4/3/2 with per-dimension key calibration, behind kvScheme. turbo8v3: 2.7x KV at affine8-class KLD across 6 families (1.7B to 32B, incl. 30B MoE); decode 0.7-0.8x fp16
vllm-swift Apple Silicon, Swift Active OpenAI-compatible serving built on mlx-swift-lm; no Python in the inference hot path
Atlas Rust, Metal Integrated turbo4 KV cache append/decode kernels
mlxcel Rust, MLX Community port Turbo KV cache docs. Thanks to @inureyes and the Lablup team for the careful port and upstream attribution
MLX Python fork Apple Silicon, Python Experimental TurboKVCache drop-in for mlx-lm and mlx-vlm; see docs/mlx-port.md
turboquant-pytorch PyTorch Independent implementation From-scratch PyTorch TurboQuant for KV cache; applies this repo's layer-adaptive compression and Sparse V findings
turboquant-vllm CUDA, vLLM package Community package TQ+ KV cache for vLLM with fused CUDA kernels; builds on this repo's block_size=128 and K-dominance research
Quansloth Local server product Community product Air-gapped local AI server built on the TurboQuant+ stack

Key Findings

Three follow-on findings, independently validated by multiple researchers across different hardware and backends:

  1. V compression is free. Compressing the value cache (even down to 2 bits) has zero measurable effect on attention quality when key precision is maintained. Confirmed on Metal (M5 Max), CUDA RTX 4090 (@sztlink), and CUDA RTX 3090 (@HyperionMS2040). See asymmetric K/V paper.
  2. All quality degradation comes from K compression. This is why asymmetric configs (q8_0-K + turbo-V) rescue models where symmetric fails. Validated across Qwen, Llama, Mistral, and Command-R+ families. See M5 Max stress test.
  3. Boundary layers are disproportionately sensitive. Protecting the first 2 + last 2 layers at higher precision recovers 37-91% of the quality gap. See Boundary V paper.

Additional experiments and writeups: Sparse V dequant (+22.8% decode at 32K, not TurboQuant-specific), block size optimization (5.12x compression), turbo4 resurrection (QJL hurts, PolarQuant works), EDEN optimal-S response (rotation is first-order, scale is second-order).

Quality at a Glance (M5 Max 128GB)

Cache Type Bits/val Compression PPL (wikitext-2, 512c) vs q8_0
f16 16.0 1.0x 6.121 -0.16%
q8_0 8.5 1.9x 6.111 baseline
turbo4 4.25 3.8x 6.125 +0.23%
q4_0 4.5 3.6x 6.142 +0.52%
turbo3 3.5† 4.6x† 6.176 +1.06%
turbo2 2.5 6.4x 6.507 +6.48%

turbo4 (4-bit PolarQuant) has the best quality after q8_0 — closer to q8_0 than q4_0, at better compression. turbo3 trades quality for maximum compression. turbo2 (2-bit) trades more quality for extreme compression — best used asymmetrically.

†turbo3 at default block_size=32. At block_size=128, turbo3 achieves 3.125 bits/val and 5.12x compression with identical PPL. See block size study.

Important: choosing the right config for your model. TurboQuant quality depends on your base weight quantization. Models with Q8_0+ weights work well with symmetric turbo (e.g., -ctk turbo3 -ctv turbo3). Some low-bit models with Q4_K_M weights may benefit from asymmetric K/V: use -ctk q8_0 -ctv turbo4 to keep K precision high while compressing V. K precision is the dominant quality factor because it controls attention routing via softmax. Bigger models absorb quantization stacking better (104B: +3.6% vs 70B: +11.4% for turbo3). Validate on your specific model. See Configuration Recommendations for the full tested matrix.

Everything else lives in docs/benchmarks.md: asymmetric K/V and Boundary V tables, prefill context scaling, MoE and dense decode speed, NIAH retrieval, KL divergence, 70B/104B stress tests, community results on RTX 3090, M1 Max, and AMD RX 9070 XT, and the speed optimization journey.

Getting Started

Python Reference Implementation

git clone https://github.com/TheTom/turboquant_plus.git
cd turboquant_plus
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

# Verify — should print "141 passed"
python3 -m pytest tests/ -v

# Quick compression demo (no model needed)
python3 benchmarks/demo.py

# Validate on real model KV tensors (downloads Qwen3-1.7B, ~4GB)
pip install transformers torch accelerate
python3 benchmarks/validate_real_model.py

Requires Python >= 3.10, NumPy >= 1.24, SciPy >= 1.10. torch / transformers / accelerate are optional (real-model validation only).

Run Inference

The fastest path is a prebuilt binary, or build from the fork — see the llama-cpp-turboquant quick start for build instructions, supported backends, and usage details.

# Server mode with TurboQuant KV cache
./build/bin/llama-server -m models/your-model.gguf \
  --jinja -ngl 99 -c 262144 -fa on \
  --cache-type-k turbo3 --cache-type-v turbo3 \
  -np 1 --host 0.0.0.0 --port 8080
Flag Bits/val Compression vs fp16 Description
turbo3 3.5† 4.6x† 3-bit PolarQuant + WHT rotation. Best compression, q8_0 speed.
turbo4 4.25 3.8x 4-bit PolarQuant (16 centroids). Best quality.
q8_0 8 2.0x llama.cpp default quantized cache.
q4_0 4 4.0x llama.cpp 4-bit cache.

For per-model configuration guidance (symmetric vs asymmetric, Boundary V, large-model memory caps), see the Getting Started Guide and Configuration Recommendations.

Architecture

Input: KV cache vector x ∈ R^d (one attention head)
    │
    ├── Extract norm: γ = ||x||, x̂ = x/γ
    │
    ├── Random rotation: WHT + random sign flips
    │   coordinates ~ N(0, 1/d) after rotation
    │
    ├── Optimal scalar quantization (Lloyd-Max)
    │   turbo4: 16 centroids (4-bit), turbo3: 8 centroids (3-bit), turbo2: 4 centroids (2-bit)
    │
    └── Output: quantized indices + norm per block
        Compression: 3.8x (turbo4), 5.1x (turbo3), 7.5x (turbo2)

Note on QJL: reference only, not used in production.

The original TurboQuant paper (Zandieh et al. 2024, arXiv 2406.03482) includes a 1-bit QJL error-correction stage. The Python qjl.py here implements it for paper reproducibility.

Production drops QJL on both K and V. QJL eliminates reconstruction bias but amplifies variance, which softmax turns into attention noise. Five independent groups confirmed (buun, scos-lab, Arclabs001, +2). See turbo4-resurrection.md for the full ablation and mechanism.

If you're building on this repo: use TurboQuantMSE (V cache), or implement straight 4 to 8 bit PolarQuant on K. Only enable the QJL / TurboQuant (with QJL) classes if you are reproducing the original paper or doing K-side research below 8-bit.

Project Structure 
turboquant/
├── rotation.py        # Walsh-Hadamard Transform + random sign flips
├── codebook.py        # Lloyd-Max optimal centroid computation
├── polar_quant.py     # PolarQuant — norm extraction + WHT rotation + scalar quantization
├── qjl.py            # QJL 1-bit quantizer (paper-faithful reference, see README §QJL). Not used in production.
├── turboquant.py      # Full TurboQuant pipeline
├── kv_cache.py        # KV cache integration layer
├── outlier.py         # Outlier channel strategy (2.5-bit, 3.5-bit)
├── lloyd_max.py       # Lloyd-Max quantizer implementation
├── utils.py           # Bit packing, memory measurement
├── isoquant.py        # IsoQuant (quaternion SO(4)) experimental comparison
└── rotorquant.py      # RotorQuant experimental comparison

tests/                 # 14 test files, 500+ tests
benchmarks/
├── demo.py                       # Quick compression demo
├── run_benchmark.py              # Server-based benchmark runner
├── benchmark_results.md          # Full benchmark report
├── benchmark_llama.sh            # llama.cpp benchmark script
├── benchmark_norm_correction.py  # Norm correction validation
├── benchmark_ppl_tq_vs_rq.py    # TurboQuant vs RotorQuant PPL comparison
├── temporal_decay_prototype.py   # Temporal decay experiment
├── test_with_llama.py            # Integration test at Qwen 3.5 dimensions
├── test_outlier_comparison.py    # Outlier strategy comparison
└── validate_real_model.py        # Real model KV tensor validation

docs/
├── benchmarks.md                 # Full benchmark and validation data
├── mlx-port.md                   # MLX framework port (Python)
├── changelog.md                  # v1 milestone history
├── turboquant-recommendations.md # Configuration guide (tested matrix)
├── windows-rdna4-setup.md        # Windows + AMD RDNA 4 build guide
├── papers/                       # Validation papers and experiment writeups
└── (25+ engineering docs, investigations, experiment logs)

Roadmap

Phase Status Details
Core algorithms (NumPy) ✅ 500+ tests across 14 test files
Distortion validation ✅ Matches paper bounds (Table 2)
Real model validation ✅ Rotation validated on Qwen3 KV tensors (kurtosis 900→2.9)
llama.cpp C port ✅ Metal GPU inference working on M1 through M5
Metal shader optimization ✅ q8_0 speed parity: prefill matches or beats q8_0
CUDA backend ✅ Community-tested on RTX 3080 Ti/3090/4090/5090, DGX Spark Blackwell
HIP/AMD backend ✅ RX 9070 XT (RDNA 4) validated, gfx1201 native
Asymmetric K/V ✅ q8_0-K + turbo-V rescues Q4_K_M models
Boundary V ✅ Layer-aware V compression, 37-91% quality recovery
Sparse V ✅ Attention-gated dequant skip, +22.8% decode on MoE. Upstream PR #21119
Block size optimization ✅ 32→128, 12% better compression, zero quality cost
vLLM upstream ✅ Merged as the TurboQuant attention backend (PR #38479)
llama.cpp upstream (rotation) ✅ Hadamard KV rotation + FWHT kernels merged upstream (#21038, #22631)
Upstream coordination 🔄 llama.cpp PR preparation for the full codec (#27)
TurboQuant+ extensions ⏳ Adaptive bits, temporal decay, MoE-aware compression
MLX Swift upstream ✅ Merged into Apple mlx-swift-lm (PR #232, 2026-07-20): JIT Metal kernels, asymmetric + calibrated symmetric schemes, 63 tests

Paper Reference

Docs

Contributing

Issues and PRs welcome. The main areas where help is needed:

  1. Upstream PR — prepare llama.cpp contribution (CONTRIBUTING.md requirements)
  2. CUDA kernel optimization — fused FA kernels, decode speed parity
  3. MLX memory recovery — implement FP16 KV drop + compressed-only attention for memory-constrained long context
  4. Quality metrics — multi-run statistics, additional task benchmarks (GSM8K, code gen, reasoning)
  5. Long context validation — 64K+ testing across architectures

Support

If you find this work useful, you can support it via GitHub Sponsors or BTC:

BTC: bc1qsfaaf6mkz2yxx2vavg2n0zgsf3qj25uh94t83rwuq7de67dey05sc3tgjx

Commercial support: For inference optimization and KV cache tuning engagements, DM @no_stp_on_snek on X.

License

Apache License 2.0 — see LICENSE.

Copyright 2026 Tom Turney.

Based on Google Research's TurboQuant paper (arXiv 2504.19874).

turboquant_plus (TheTom/turboquant_plus)

Searching Database…

7K ·GitHub
Turboquant_plus
这是什么？TurboQuant+ 是 TurboQuant（ICLR 2026）的研究项目主页，TurboQuant 是一种面向 Transformer 大语言模型的 KV 缓存压缩方法。它借助 PolarQuant 和 Walsh-Hadamard 旋转，把 KV 缓存缩小 3.8-6.4 倍。这个仓库里包含 Python 参考实现、验证论文和基准测试数据。
那这是要加到我项目里的库，还是别的东西？这是研究代码，不是面向最终用户的应用：它是一个 Python 参考实现，外加论文和基准测试。要做真正的推理部署，你需要使用它表格中列出的某个引擎——vLLM、llama.cpp（上游版本或生产分支）或 Apple MLX——TurboQuant 的支持就内置在这些引擎里。
我怎么把它跑起来？最快的路径：直接下载预编译的 llama-cpp-turboquant 二进制文件（Mac Metal 或 Windows CUDA），或者从这个分支自己编译。在 vLLM 中它已经进入上游主线——只需加上 --kv-cache-dtype turboquantk8v4 参数。这个仓库里的 Python 代码是用来做实验的，不是用来部署服务的。
Python 参考实现需要什么环境？Python 3.10+、NumPy 1.24+ 和 SciPy 1.10+。torch、transformers 和 accelerate 是可选的——只有要在真实模型上做验证时才需要它们。
做压缩的话，质量会损失多少？在中等压缩档位下几乎不损失：turbo4 在相对于 q80 基线困惑度只增加 0.23% 的情况下实现 3.8 倍压缩，turbo3 在增加 1.06% 的情况下实现 4.6 倍。只有极端的 2 比特 turbo2（6.4 倍）才会带来明显的质量损失，大约 +6.5%。
它能在哪些硬件上运行？支持范围很广：Mac 上的 Metal（M1 到 M5）、NVIDIA 上的 CUDA（社区在 RTX 3080 Ti 到 5090 以及 DGX Spark 上测试过）、AMD 上的 HIP（RX 9070 XT 已验证），另外还有 CPU 内核。预编译二进制文件提供 Mac（Metal）和 Windows（CUDA）版本。
除了原论文，作者还发现了什么？有三个发现，并经多位研究者独立验证：只要 K 的精度保持住，压缩 V 缓存基本没有代价；所有质量损失都来自 K 的压缩，因为键（key）决定了注意力的路由；把最前面 2 层和最后面 2 层这些边界层用更高精度保护起来，可以挽回 37-91% 的质量差距。
TurboQuant 已经进入主流推理引擎了吗？是的。vLLM 已把它合并为一种 KV 缓存 dtype（PR #38479，2026 年 4 月），llama.cpp 合并了 Hadamard 旋转以及 CPU、CUDA 和 Vulkan 上的快速内核，Apple 也把整套方案合并进了 mlx-swift-lm。稳定的部分会以小而易于审查的补丁形式进入上游主线。
QJL 是什么？为什么 README 里对它有警告？QJL 是原始 TurboQuant 论文中的一种 1 比特纠错环节，这个仓库保留它只是为了复现结果。生产环境在 K 和 V 上都不再使用它：它虽然消除了重建偏差，却会放大方差，而 softmax 会把这些方差变成注意力噪声——已有五个独立团队证实了这一点。
我怎么为我的模型选一套配置？要和你权重的量化方式匹配。权重是 Q80+ 的模型适合对称的 turbo 配置（比如 -ctk turbo3 -ctv turbo3）；低比特的 Q4KM 模型往往用非对称配置效果更好——键用 q80、值用 turbo4——因为键的精度对质量影响最大。越大的模型越能吸收量化叠加的影响，所以请在你的具体模型上验证。
What are similar projects?pd-bridge, solo_jev, tilelang, MiniCPM, backburner, Hyperloom, StartLux-Decision, Model-Optimizer
Which topics is it in?long-context, llamacpp, mlx, amd, metal, numpy, optimization, llm-inference, vllm, quantization, apple-silicon, inference
Creator
TheTom
Size
5.4 MB
Stars
7K
Updated
CommitJul 20
Age
6mo
Commits · 90 Days