ktransformers README

The project's README from github.com/kvcache-ai/ktransformers. As one markdown file: README.md.

KTransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference/Fine-tune Optimizations

🎯 Overview | 🚀 Inference | 🎓 SFT | 🔥 Citation | 🚀 Roadmap(2026Q2)

🎯 Overview

KTransformers is a research project focused on efficient inference and fine-tuning of large language models through CPU-GPU heterogeneous computing. The project now exposes two user-facing capabilities from the kt-kernel source tree: Inference and SFT.

🔥 Updates

📦 Capabilities

🚀 Inference - High-Performance kt-kernel Serving

CPU-optimized kernel operations for heterogeneous LLM inference.

image

Key Features:

Quick Start:

cd kt-kernel
pip install .

Use Cases:

Performance Examples:

Model Hardware Configuration Total Throughput Output Throughput
DeepSeek-R1-0528 (FP8) 8×L20 GPU + Xeon Gold 6454S 227.85 tokens/s 87.58 tokens/s (8-way concurrency)

👉 Full Documentation →

🎓 SFT - Fine-Tuning with LlamaFactory

KTransformers × LlamaFactory integration for ultra-large MoE model fine-tuning. The new Cookbook provides an easy-to-use path from hardware checks and installation to BF16/FP8/INT8 configuration, LoRA/full training, resource planning, and troubleshooting.

LlamaFactory and KTransformers MoE fine-tuning architecture

Key Features:

Model GPU Memory Training Speed Hardware
DeepSeek-V3 ~80GB total 3.7 it/s 4x RTX 4090
DeepSeek-R1 ~80GB total 3.7 it/s 4x RTX 4090
Qwen3-30B-A3B ~24GB total 8+ it/s 1x RTX 4090

Quick Start:

cd /path/to/LLaMA-Factory
python -m pip install -e .
python -m pip install "ktransformers[sft]==0.7.0"
python -m pip install "sglang-kt==0.7.0"
CUDA_VISIBLE_DEVICES=0,1,2,3 accelerate launch \
  --config_file examples/ktransformers/accelerate/fsdp2_kt_int8.yaml \
  src/train.py \
  examples/ktransformers/train_lora/qwen3_5moe_lora_sft_kt.yaml

👉 Easy-to-Use Fine-Tuning Cookbook → 👉 Quick Start → 👉 Full Documentation →

🔥 Citation

If you use KTransformers in your research, please cite our paper:

@inproceedings{10.1145/3731569.3764843,
  title = {KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models},
  author = {Chen, Hongtao and Xie, Weiyu and Zhang, Boxin and Tang, Jingqi and Wang, Jiahao and Dong, Jianwei and Chen, Shaoyuan and Yuan, Ziwei and Lin, Chen and Qiu, Chengyu and Zhu, Yuening and Ou, Qingliang and Liao, Jiaqi and Chen, Xianglin and Ai, Zhiyuan and Wu, Yongwei and Zhang, Mingxing},
  booktitle = {Proceedings of the ACM SIGOPS 31st Symposium on Operating Systems Principles},
  year = {2025}
}

👥 Contributors & Team

Developed and maintained by:

We welcome contributions! Please feel free to submit issues and pull requests.

💬 Community & Support

📦 KT original Code

The original integrated KTransformers framework has been archived to the archive/ directory for reference. The project now organizes the two capabilities above from the kt-kernel source tree for clearer documentation and maintenance.

For the original documentation with full quick-start guides and examples, see:

ktransformers (kvcache-ai/ktransformers)

Searching Database…

19.6K ·GitHub
Ktransformers
这是什么？KTransformers 是一个研究框架，通过 CPU-GPU 异构计算，实现大语言模型的高效推理和微调，让超大规模 MoE 模型能在显存不足以完整装入 GPU 的硬件上运行。
这是一个独立的聊天应用，还是需要我接入自己服务栈的组件？它是一个框架/内核库，而不是应用：你需要把它的 “kt-kernel” Python API 集成到 SGLang 等推理服务器中用于服务，或集成到 LlamaFactory 中用于微调。
我该如何开始用它进行推理或微调？该仓库为推理和 SFT 分别提供了快速入门入口：推理通过 kt-kernel Python API 使用，常与 SGLang 搭配；SFT 则通过 LlamaFactory 集成以及 MoE Fine-Tuning Cookbook 完成。
要真正使用它，需要什么硬件？量化内核需要支持 AMX 或 AVX512/AVX2 指令集的 CPU，另外还需要一块 GPU（NVIDIA、通过 ROCm 支持的 AMD、Intel Arc 或昇腾 NPU）；运行 DeepSeek-R1 等超大模型时，可能需要高达 382GB 内存或多块 GPU。
在尝试之前，有哪些硬件限制需要了解？部分功能最初只限于支持 AMX 的 Intel CPU；LoRA 微调最近才新增支持普通的 AVX512 x86 CPU（包括 AMD 服务器），无需 AMX，因此较旧的配置可能无法用上较新的优化。
它能在 Windows 上运行，还是只能在 Linux 服务器上运行？它可以原生运行在 Windows 和 Linux 上，并支持多种 GPU 后端，包括 NVIDIA、AMD（ROCm）、Intel Arc 和昇腾 NPU。
DeepSeek-R1 的推理速度大概是多少？在 8×L20 GPU 加 Xeon Gold 6454S CPU 的配置下，KTransformers 在 8 路并发时，对 DeepSeek-R1-0528（FP8）的总吞吐量约为 227.85 tokens/s，输出吞吐量约为 87.58 tokens/s。
我能否只用一块消费级 GPU 微调超大 MoE 模型？可以，Qwen3-30B-A3B 在单块 RTX 4090 上使用约 24GB 显存即可微调，而 DeepSeek-V3/R1 在 4x RTX 4090 上共使用约 80GB 显存即可微调。
哪些模型能获得 day-0 支持？近期的 day-0 发布包括 GLM-5.3-flash、MiniMax-M3、GLM-5.2、Kimi-K2.5 和 GLM-5，同时还持续支持 DeepSeek-V3/R1、Qwen3 以及实验性的 LLaMA 4。
What are similar projects?vllm, verl, sglang, MoE4All, decider, airllm, ms-swift, unsloth
Which topics is it in?cpu-inference, intel, sglang, rocm, cpu, amd-gpu, mixture-of-experts, moe, llama, lora, llm-inference, high-performance
Creator
kvcache-ai
Size
73 MB
Stars
19.6K
Updated
CommitOct 15Releasev0.7.2Oct 15
Age
2y
Commits · 90 Days