PrismQuant README

项目的 README,来自 github.com/ForeverBlue816/PrismQuant。合并为一个 markdown 文件:README.md。

PrismQuant blue triangular logo

PrismQuant

Optimal Null-Space Rotations for Grouped Quantizers

Paper · Hugging Face · Models · Indexed results · Quick start · Method · Figures · Reproduction

arXiv

Official code and pretrained checkpoints for PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers (arXiv:2609.32429), by Yanlong Chen, Yining Chen, Song Zhang, Amirhossein Habibian, and Yawei Li. See the released models for all Hugging Face checkpoints.

PrismQuant method overview

PrismQuant aligns high-energy activation directions with the constant directions of quantization groups. It combines calibrated orthogonal rotations, compact-WY representations and groupwise asymmetric INT4 quantization. This repository contains the core implementation, released model checkpoints and paper figures.

Quick start

Use Python 3.10+ and a PyTorch installation appropriate for your CUDA version.

git clone --depth 1 https://github.com/ForeverBlue816/PrismQuant.git
cd PrismQuant
pip install -e .
prismquant models

Start with the smallest released model:

from prismquant import load_model

model = load_model("qwen3_0.6b_base")
print(model.generate("The key idea behind quantization is", max_new_tokens=64))

Or use the command line:

prismquant generate --model qwen3_0.6b_base \
  --prompt "The key idea behind quantization is" --max-new-tokens 64

The loader downloads only the selected checkpoint and its required rotation factors, then obtains the matching base model and tokenizer. The default PrismQuant configuration is selected automatically. These are base-model completions, not chat-tuned assistants. See model selection, memory and loading options.

Checkpoint format. The released GPTQ weights are dequantized INT4 values in floating-point tensors. The reference runtime simulates activation and KV quantization while retaining floating-point storage; it is not a packed INT4 serving engine. Use this package's loader, rather than passing the artifact repo to AutoModelForCausalLM.from_pretrained. The separate fused R4 kernels are provided for implementation work.

Released models

Family Sizes Hugging Face
Qwen3 Base 0.6B, 1.7B, 4B, 8B Checkpoints and rotation factors
Qwen3 MoE Base 30B-A3B Checkpoint and expert rotations
Llama 3.2 3B Checkpoints and rotation factors
Llama 3.1 8B Checkpoints and rotation factors
Llama 3.1 70B Checkpoints and rotation factors

The model catalog pins the model artifacts and base models to exact revisions. The MoE release includes all expert weights and per-expert rotations. Upstream model licenses apply.

Method

  1. Construct: estimate dominant uncentered activation directions and map them to group anchors with Householder reflectors and a balanced permutation.
  2. Represent: apply block Hadamard transforms so aligned directions become constant within groups; store the group scale and real affine offset in fp16.
  3. Deploy: fold static transforms into weights and use compact-WY factors for the remaining online rotations.

Alignment and group quantization

The core rotation can also be used independently of a language model:

import torch
from prismquant import fit_rotation, dynamic_asym_int4

calibration = torch.randn(512, 256)  # replace with your calibration activations
rotation = fit_rotation(calibration, rank=2, group_size=128)
signs = torch.ones(256)
rotated = rotation.apply(calibration, signs)
quantized, scale, offset, codes = dynamic_asym_int4(rotated, 128)
reconstructed = rotation.transpose(quantized, signs)

This example fits a rotation from user-provided activation samples using exact second moments. See the technical guide for conventions and the kernel guide for the fused path.

Results and figures

Papers with Code: WikiText-2 result, 3.85 PPL

Seven paper results are indexed on Papers with Code. As checked on 2026-10-14, PrismQuant on Llama-3.1-70B, W4A4KV4, k=8 has 3.85 WikiText-2 perplexity and ranks first among that site's indexed WikiText-2 results. This rank is specific to the indexed comparison set; base models, tokenizers, precision budgets and evaluation protocols differ. The badge shows the published score, not a live rank.

All seven scores are unchanged, and the catalog now includes the checkpoint links and clarified evaluation protocols. See verified evaluation metadata for evaluation settings and reproduction guidance. Catalog verification is not an independent experimental replication.

Layerwise activation range and INT4 error

Geometry, energy coverage and range law

The figure gallery includes the author's final Figure 1–6, method overview and activation matrices, with PDF originals and PNG/SVG previews. Figure 5 reports the dedicated deployment implementation; its throughput should not be attributed to the floating-point reference loader above.

Repository layout

prismquant/       Rotation, quantization, GPTQ, folding, loading and CUDA kernels
examples/         Text completion and standalone rotation examples
Figures/          Author-provided paper figures and shareable previews
docs/             Model format, technical guide and reproduction pointers
tests/            Algebra, metadata, loading and generation checks

This repository provides the reference implementation, released model catalog, tests and paper figures. See reproduction guidance for model revisions, evaluation settings and the scope of the public package.

Development

pip install -e '.[dev]'
pytest -q

See release validation for the test scope and numerical checks.

Paper and citation

PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers

Yanlong Chen, Yining Chen, Song Zhang, Amirhossein Habibian, and Yawei Li.

@article{chen2026prismquant,
  title={PrismQuant: Optimal Null-Space Rotations for Grouped Quantizers},
  author={Chen, Yanlong and Chen, Yining and Zhang, Song and Habibian, Amirhossein and Li, Yawei},
  journal={arXiv preprint arXiv:2609.32429},
  year={2026},
  doi={10.48550/arXiv.2609.32429},
  url={https://arxiv.org/abs/2609.32429}
}

License and acknowledgments

Code: Apache-2.0. Model weights retain their respective upstream licenses. Built with Llama for the Llama-derived checkpoints. The GPTQ core is adapted from QuaRot; see third-party notices.

PrismQuant (ForeverBlue816/PrismQuant)

正在搜索数据库…

这是什么?PrismQuant 是一篇关于大语言模型 INT4 量化研究论文的官方代码和预训练检查点。它利用校准后的正交旋转,将高能量的激活方向与量化分组的常量方向对齐。
这是可以直接用于服务的 INT4 推理引擎,还是研究参考实现?它是参考实现和模型目录,不是生产级推理引擎。已发布的 GPTQ 权重是以浮点张量存储的反量化 INT4 数值,运行时模拟的是激活和 KV 量化,而不是运行打包的 INT4 运算。
我该如何试用已发布的模型?使用 Python 3.10+ 和与你的 CUDA 版本匹配的 PyTorch 版本,然后通过本包自带的加载器加载最小的已发布检查点(可通过 Python 或命令行使用)。加载器会下载检查点、对应的旋转因子以及匹配的基础模型和分词器。
使用之前需要安装什么?运行加载器和模型需要 Python 3.10 或更高版本,以及与你的 CUDA 版本匹配的 PyTorch。
加载检查点时有什么需要注意的坑吗?必须使用本包自带的加载器,不能直接把检查点仓库传给 'AutoModelForCausalLM.from_pretrained',因为这些权重需要配套的旋转因子。另外,已发布的模型是基础模型的续写模型,不是经过对话微调的助手。
哪些模型系列发布了检查点?Qwen3 Base(0.6B-8B)、Qwen3 MoE Base(30B-A3B)、Llama 3.2(3B),以及 Llama 3.1(8B 和 70B),全部托管在 Hugging Face 上,并附带旋转因子。
实际的量化效果如何?在 Llama-3.1-70B 上使用 W4A4KV4 和 k=8 时,PrismQuant 报告的 WikiText-2 困惑度为 3.85,截至核对日期,在 Papers with Code 的已收录结果中排名第一。
我能否在不使用完整语言模型的情况下使用旋转方法?可以。核心旋转构造可以单独利用用户提供的激活样本及其精确二阶矩进行拟合,无需加载任何大语言模型。
量化内核的实现来自哪里?融合的 R4 内核单独提供,供实现相关的工作使用;GPTQ 核心改编自 QuaRot 项目,相关致谢见第三方声明。
有哪些类似的项目?jeeves, OraRL, litgpt, airllm, Model-Optimizer, pointdit, ncm, NeoHorse
大小
17.6 MB
星标
158
更新
提交Oct 14
时长
2个月
提交 · 90 天