What is this?
TRL (Transformers Reinforcement Learning) is a Hugging Face library for post-training foundation models — fine-tuning them with techniques like SFT, GRPO and DPO on top of the Transformers stack.
TRL (Transformers Reinforcement Learning) is a Hugging Face library for post-training foundation models — fine-tuning them with techniques like SFT, GRPO and DPO on top of the Transformers stack.So is it a Python library I import, or an app I run?It's a Python library: each trainer (SFTTrainer, DPOTrainer, GRPOTrainer and others) is a light wrapper around the standard Transformers Trainer, so it slots into an existing Transformers workflow.
It's a Python library: each trainer (SFTTrainer, DPOTrainer, GRPOTrainer and others) is a light wrapper around the standard Transformers Trainer, so it slots into an existing Transformers workflow.How do I install it and start a first training run?Install it with pip install trl (or from source for pre-release features); clone the repo if you want the examples, then pick a trainer and point it at your dataset.
Install it with pip install trl (or from source for pre-release features); clone the repo if you want the examples, then pick a trainer and point it at your dataset.Does training run locally on my machine, or does it need cloud API keys?It runs locally on your own hardware — no API keys or accounts are involved. It builds on the Hugging Face stack: Transformers, Accelerate for distributed runs, plus optional PEFT and Unsloth.
It runs locally on your own hardware — no API keys or accounts are involved. It builds on the Hugging Face stack: Transformers, Accelerate for distributed runs, plus optional PEFT and Unsloth.Is everything in the library stable enough to depend on?Not quite: code under trl.experimental is an incubation area for unstable, fast-evolving features — anything there may change or be removed in any release without notice.
Not quite: code under trl.experimental is an incubation area for unstable, fast-evolving features — anything there may change or be removed in any release without notice.Do I need a multi-GPU cluster to actually use it?Not necessarily: it scales from a single GPU up to multi-node clusters, and PEFT integration (quantization, LoRA/QLoRA) lets you fine-tune large models on modest hardware.
Not necessarily: it scales from a single GPU up to multi-node clusters, and PEFT integration (quantization, LoRA/QLoRA) lets you fine-tune large models on modest hardware.What is GRPO, and why do people pick it over PPO?GRPO (Group Relative Policy Optimization) is a reinforcement learning method that is more memory-efficient than PPO — it's what DeepSeek used to train R1, and TRL ships it as GRPOTrainer.
GRPO (Group Relative Policy Optimization) is a reinforcement learning method that is more memory-efficient than PPO — it's what DeepSeek used to train R1, and TRL ships it as GRPOTrainer.What is KTO and when would I use it?KTO (Kahneman-Tversky Optimization) aligns models from simple binary desirable/undesirable feedback instead of requiring paired preference data, which makes it handy when you only have thumbs-up/thumbs-down labels. TRL provides KTOTrainer for it.
KTO (Kahneman-Tversky Optimization) aligns models from simple binary desirable/undesirable feedback instead of requiring paired preference data, which makes it handy when you only have thumbs-up/thumbs-down labels. TRL provides KTOTrainer for it.Have any well-known models been post-trained with these trainers?Yes — the README notes GRPO was used to train DeepSeek R1, and DPO was used to post-train Llama 3 and many other models.
Yes — the README notes GRPO was used to train DeepSeek R1, and DPO was used to post-train Llama 3 and many other models.Does it handle very long context training?A new long-context guide covers the four things that break as sequences grow — loss, positions, activations and single-GPU memory — and ends with an example training Qwen3-8B on million-token sequences on a single 8-GPU node.
A new long-context guide covers the four things that break as sequences grow — loss, positions, activations and single-GPU memory — and ends with an example training Qwen3-8B on million-token sequences on a single 8-GPU node.What are similar projects?halo, ms-swift, train-llm-from-scratch, verl, unsloth, Soup, airllm, jeeves
halo, ms-swift, train-llm-from-scratch, verl, unsloth, Soup, airllm, jeevesWhich topics is it in?dpo, grpo, rlhf, lora, training, transformers, reinforcement-learning, quantization, fine-tuning, huggingface, pytorch, deep-learning
dpo, grpo, rlhf, lora, training, transformers, reinforcement-learning, quantization, fine-tuning, huggingface, pytorch, deep-learning