PaddleFormers README

The project's README from github.com/PaddlePaddle/PaddleFormers. As one markdown file: README.md.

最新更新 | 特性 | 安装 | 快速体验 | 社区交流

PaddleFormers

📝简介

PaddleFormers 是基于百度深度学习框架 PaddlePaddle 搭建的 Transformers 库，旨在为 PaddlePaddle 生态提供与 Hugging Face Transformers 项目对等的模型接口与功能体验，支持大语言模型（LLM）与视觉语言模型（VLM）的训练能力。PaddleFormers 充分发挥 PaddlePaddle 在高性能训练方面的内置优势，全面支持包括张量并行、流水线并行和专家并行在内的主流大模型分布式训练策略，以及自动混合精度等加速技术，在 DeepSeek-V4、GLM-4.5-Air 等重点模型上，训练性能明显超越 Megatron-LM ，实现了高效的预训练与后训练性能。

结合业界主流优化方法与飞桨在业务实践中积累的高效特性，PaddleFormers 致力于打造高性能、低资源占用的训练体验，帮助用户高效便捷地完成大模型训练，而无需关注底层复杂的优化细节。

🆕最新更新

[!IMPORTANT] PaddleFormers v1.2 发布 — 业界率先实现对 DeepSeek-V4 大规模分布式训练的全面支持！ 查看训练实践 →

✨特性

📋模型列表

模型类型 模型系列 模型名称 Chat Template
LLM DeepSeekv3 deepseek-ai/DeepSeek-V3-Base、deepseek-ai/DeepSeek-V3、deepseek-ai/DeepSeek-V3-0324 deepseek3
DeepSeekv3.2 deepseek-ai/DeepSeek-V3.2 deepseek3
DeepSeekv4 deepseek-ai/DeepSeek-V4-Pro、deepseek-ai/DeepSeek-V4-Flash deepseek3
🏛️ERNIE-4.5 baidu/ERNIE-4.5-0.3B-Base-PT、baidu/ERNIE-4.5-0.3B-PT、baidu/ERNIE-4.5-21B-A3B-Base-PT、baidu/ERNIE-4.5-21B-A3B-PT、baidu/ERNIE-4.5-300B-A47B-Base-PT、baidu/ERNIE-4.5-300B-A47B-PT、baidu/ERNIE-4.5-21B-A3B-Thinking ernie、ernie_nothink
gemma3 google/gemma-3-270m、google/gemma-3-270m-it、google/gemma-3-1b-pt、google/gemma-3-1b-it、google/gemma-3-4b-pt、google/gemma-3-4b-it、google/gemma-3-12b-pt、google/gemma-3-12b-it、google/gemma-3-27b-pt、google/gemma-3-27b-it gemma
GLM-4.5 zai-org/GLM-4.5-Air-Base、zai-org/GLM-4.5-Air、zai-org/GLM-4.5-Base、zai-org/GLM-4.5 glm4_moe
gpt-oss openai/gpt-oss-20b、openai/gpt-oss-120b gpt
Granite ibm-granite/granite-3.2-2b-base、ibm-granite/granite-3.2-2b-instruct、ibm-granite/granite-3.2-8b-base、ibm-granite/granite-3.2-8b-instruct granite
Llama-3 meta-llama/Meta-Llama-3-8B、meta-llama/Meta-Llama-3-8B-Instruct、meta-llama/Meta-Llama-3-70B、meta-llama/Meta-Llama-3-70B-Instruct、meta-llama/Llama-3.1-8B、meta-llama/Llama-3.1-8B-Instruct、meta-llama/Llama-3.1-70B、meta-llama/Llama-3.1-70B-Instruct、meta-llama/Llama-3.1-405B、meta-llama/Llama-3.1-405B-Instruct、meta-llama/Llama-3.2-1B、meta-llama/Llama-3.2-1B-Instruct、meta-llama/Llama-3.2-3B、meta-llama/Llama-3.2-3B-Instruct、meta-llama/Llama-3.3-70B-Instruct llama3
OLMo2 allenai/OLMo-2-0425-1B、allenai/OLMo-2-0425-1B-SFT、allenai/OLMo-2-0425-1B-DPO、allenai/OLMo-2-0425-1B-Instruct、allenai/OLMo-2-1124-7B、allenai/OLMo-2-1124-7B-SFT、allenai/OLMo-2-1124-7B-DPO、allenai/OLMo-2-1124-7B-Instruct、allenai/OLMo-2-1124-13B、allenai/OLMo-2-1124-13B-SFT、allenai/OLMo-2-1124-13B-DPO、allenai/OLMo-2-1124-13B-Instruct、allenai/OLMo-2-0325-32B、allenai/OLMo-2-0325-32B-SFT、allenai/OLMo-2-0325-32B-DPO、allenai/OLMo-2-0325-32B-Instruct jinja
phi-4 microsoft/phi-4 phi4
Qwen2 Qwen/Qwen2-0.5B、Qwen/Qwen2-0.5B-Instruct、Qwen/Qwen2-1.5B、Qwen/Qwen2-1.5B-Instruct、Qwen/Qwen2-7B、Qwen/Qwen2-7B-Instruct、Qwen/Qwen2-57B-A14B、Qwen/Qwen2-57B-A14B-Instruct、Qwen/Qwen2-72B、Qwen/Qwen2-0.5B-Instruct qwen
Qwen3 Qwen/Qwen3-0.6B-Base、Qwen/Qwen3-0.6B、Qwen/Qwen3-1.7B-Base、Qwen/Qwen3-1.7B、Qwen/Qwen3-4B-Base、Qwen/Qwen3-4B、Qwen/Qwen3-4B-Instruct-2507、Qwen/Qwen3-4B-Thinking-2507、Qwen/Qwen3-8B-Base、Qwen/Qwen3-8B、Qwen/Qwen3-14B-Base、Qwen/Qwen3-14B、Qwen/Qwen3-32B、Qwen/Qwen3-30B-A3B-Base、Qwen/Qwen3-30B-A3B、Qwen/Qwen3-30B-A3B-Instruct-2507、Qwen/Qwen3-30B-A3B-Thinking-2507、Qwen/Qwen3-235B-A22B、Qwen/Qwen3-235B-A22B-Instruct-2507、Qwen/Qwen3-235B-A22B-Thinking-2507 qwen3、qwen3_nothink
Qwen3-Next Qwen/Qwen3-Next-80B-A3B-Instruct、Qwen/Qwen3-Next-80B-A3B-Thinking qwen3、qwen3_nothink
Qwen3.5 Qwen/Qwen3.5-MoE qwen3_5、qwen3_5_nothink
VLM 🏛️ERNIE-4.5-VL baidu/ERNIE-4.5-VL-28B-A3B-Base-PT、baidu/ERNIE-4.5-VL-28B-A3B-PT、baidu/ERNIE-4.5-VL-424B-A47B-Base-PT、baidu/ERNIE-4.5-VL-424B-A47B-PT、baidu/ERNIE-4.5-VL-28B-A3B-Thinking ernie_vl、ernie_vl_nothink
🏛️PaddleOCR-VL PaddlePaddle/PaddleOCR-VL paddleocr_vl
Qwen2.5-VL Qwen/Qwen2.5-VL-3B-Instruct、Qwen/Qwen2.5-VL-7B-Instruct、Qwen/Qwen2.5-VL-32B-Instruct、Qwen/Qwen2.5-VL-72B-Instruct qwen2_vl
Qwen3-VL Qwen/Qwen3-VL-2B-Instruct、Qwen/Qwen3-VL-2B-Thinking、Qwen/Qwen3-VL-4B-Instruct、Qwen/Qwen3-VL-4B-Thinking、Qwen/Qwen3-VL-8B-Instruct、Qwen/Qwen3-VL-8B-Thinking、Qwen/Qwen3-VL-32B-Instruct、Qwen/Qwen3-VL-32B-Thinking、Qwen/Qwen3-VL-30B-A3B-Instruct、Qwen/Qwen3-VL-30B-A3B-Thinking、Qwen/Qwen3-VL-235B-A22B-Instruct、Qwen/Qwen3-VL-235B-A22B-Thinking qwen3_vl、qwen3_vl_nothink

💾安装

环境依赖

安装依赖（GPU）

基于 Docker 容器的方式（推荐）

为了避免本地环境存在较多冲突，我们建议使用 PaddleFormers 的预置镜像来准备环境，容器中已经拉取了 PaddleFormers 仓库并完成了安装：

# 以cuda12.6为例
docker run --gpus all --name paddleformers-work -v $(pwd):/work  \
    -w=/work --shm-size=512G --network=host -it \
    ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddle:3.3.0-gpu-cuda12.6-cudnn9.5 /bin/bash

# cuda12.9镜像：ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddle:3.3.0-gpu-cuda12.9-cudnn9.9
# cuda13.0镜像：ccr-2vdh3abv-pub.cnc.bj.baidubce.com/paddlepaddle/paddle:3.3.0-gpu-cuda13.0-cudnn9.13
基于 pip/源码的安装方式

我们推荐使用 conda / venv / uv 等虚拟环境工具管理 python 环境。

# conda
conda create -n paddleformers-work python=3.10 #支持python3.10～3.13
conda activate paddleformers-work
# venv
python -m venv .paddleformers-work
source .paddleformers-work/bin/activate
# uv
uv venv .paddleformers-work
source .paddleformers-work/bin/activate

安装方案一： 拉取源码安装

# Install development version
git clone https://github.com/PaddlePaddle/PaddleFormers.git
cd PaddleFormers
# cuda12.6
python -m pip install -e '.[paddlefleet]' --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu126/ --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
# cuda12.9
# python -m pip install -e '.[paddlefleet]' --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu129/ --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu129/
# cuda13.0
# python -m pip install -e '.[paddlefleet]' --extra-index-url https://www.paddlepaddle.org.cn/packages/nightly/cu130/ --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu130/

安装方案二： 如果您不想拉取源码，可以基于下面的命令安装 PaddleFormers 和 PaddleFleet。

# Install via pip
# cuda12.6
python -m pip install "paddleformers[paddlefleet]" --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
# cuda12.9
# python -m pip install "paddleformers[paddlefleet]" --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu129/
# cuda13.0
# python -m pip install "paddleformers[paddlefleet]" --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu130/

安装方案三： 如果您只需使用 tokenizer 或者 processor，可以通过以下命令安装，这种情况下不会安装训练相关的依赖，安装速度更加快。

python -m pip install paddleformers

安装依赖（XPU & ILUVATAR-GPU & Metax GPU）

⚡快速体验

PaddleFormers 在 API 设计上与 Hugging Face Transformers 保持了高度一致，使用示例如下：

使用 tokenizer

from paddleformers.transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")
print(tokenizer.encode("中华人民共和国"))
# 中华人民共和国将会被编码为两个token：
# [105492, 104773]

文本生成

from paddleformers.transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-0.6B-Base", dtype="bfloat16").eval()

input_features = tokenizer("请给我一段大模型的简短介绍：", return_tensors="pd")
outputs = model.generate(**input_features, max_new_tokens=256)
output_ids = outputs[0].tolist()[0]

print(tokenizer.decode(output_ids, skip_special_tokens=True))

模型训练

paddleformers-cli train ./examples/config/sft/full.yaml

📊数据处理

🚀模型训练 & 部署

💻多硬件使用

🔍最佳实践

➕其他

💬社区相关

贡献代码

和我们交流

qrcode

🙏致谢

我们借鉴了 Hugging Face 的Transformers🤗关于预训练模型使用的优秀设计，在此对 Hugging Face 作者及其开源社区表示感谢。

📜许可证

PaddleFormers 遵循Apache-2.0开源协议。

PaddleFormers (PaddlePaddle/PaddleFormers)

Searching Database…

13K ·GitHub
PaddleFormers
What is this?PaddleFormers is a Transformers-style library built on Baidu's PaddlePaddle deep learning framework, giving the PaddlePaddle ecosystem the same kind of model interface and training workflow as Hugging Face Transformers, with full training support for 100+ LLMs and vision-language models.
Is this a framework or a library I import into my code?It's a Python library whose API design intentionally mirrors Hugging Face Transformers, so if you know 'transformers' you can use this the same way, just running on PaddlePaddle instead of PyTorch.
What do I need before installing it?You need Python 3.12 or newer and CUDA 13.0 or newer, plus PaddleFleet 0.3+ which is only required if you want to use the GPU training features.
Is a lightweight install possible if I only need tokenization?Yes — there's a minimal install option that skips all training dependencies if you only need the tokenizer or processor, which installs faster but can't be used for actual model training.
Which hardware beyond NVIDIA GPUs is supported?PaddleFormers has dedicated adaptation for several Chinese domestic AI accelerators, including Kunlunxin P800, Iluvatar Tiangai 150, Moxin C550, and Moore Threads chips.
Does it support fine-tuning methods like LoRA and DPO?Yes, it covers the full post-training pipeline including CPT, SFT, SFT-LoRA, DPO, and DPO-LoRA, letting you go from pretrained checkpoint to an aligned model.
Can models trained here be deployed with vLLM or SGLang?Yes, trained models are saved in the standard Safetensors format compatible with Hugging Face weights, so they can be served directly via FastDeploy, vLLM, or SGLang without conversion.
What's new in the latest v1.2 release?v1.2 adds full support for training DeepSeek-V4 at 128K+ context length, including its CSA+HCA hybrid attention, mHC residual connections, and Muon optimizer, plus new FP8 MoE kernels and expert-parallel communication libraries.
How does its training performance compare to Megatron-LM?On flagship models like DeepSeek-V3 and GLM-4.5-Air, PaddleFormers reports training performance that clearly surpasses Megatron-LM thanks to its fused operators and communication-compute overlap optimizations.
What are similar projects?Qwen3.8-Flash-Next, Hy4-preview, Hy-MT2, open-r1, nanochat, verl, jeeves, Meshy
Which topics is it in?baidu, pretraining, llm-training, model, distributed-training, mixture-of-experts, vllm, gpu-acceleration, fine-tuning, transformer, large-language-models, nlp
Creator
PaddlePaddle
Size
230.5 MB
Stars
13K
Updated
CommitOct 13Releasev1.2.0Jun 19
Age
7y
Commits · 90 Days