Halo
Frontier-Lab Training for Everyone
Blog post ·
Docs ·
Examples ·
Benchmarks
What is Halo
Halo is an open-source framework built by White Circle for training large language and multimodal models. It supports pre-training, SFT, preference optimization, fully asynchronous multi-turn RL, and much more.
The same codebase runs on one GPU or across multiple nodes, with EP, CP, TP, ETP, FSDP2, fused kernels, BF16 training, and distributed rollouts.
Halo trains Hugging Face models directly. Checkpoints still load with from_pretrained, and supporting a new model family typically takes under 140 lines of integration code.
On 8× B300, Halo delivers up to ~2.8× the training throughput of stock TRL (2.7× at 25% less peak memory when both sides shard ZeRO-3), with larger margins over the other frameworks benchmarked.
Pre- & Post-Training · EP / CP / TP / ETP ·
Multi-Node · Verifiable & Multi-Turn RL · FA4 + Liger + Grouped GEMM · Full BF16
Why Halo
Halo trains Hugging Face models directly. Trainers subclass Hugging Face or TRL trainers, checkpoints remain standard SafeTensors, and from_pretrained continues to work.
Parallelism is added to the existing model. EP wraps MoE blocks, CP wraps attention, and TP/ETP shard weights in place. There is no separate distributed implementation of the model.
The distributed stack is built on PyTorch. FSDP2, DTensor, and DeviceMesh provide the underlying primitives, while DeepEP handles the all-to-all communication for Expert Parallelism. EP, CP, TP, and ETP can be configured independently and combined across GPUs and nodes.
Training methods share the same infrastructure. Pre-training, SFT, preference optimization, distillation, and RL use the same parallelism and checkpointing code, so new methods don't need their own distributed implementation.
Performance optimizations are integrated. Halo includes DeepEP V2, FlashAttention 4, Liger, Grouped GEMM, padding-free packing, and AdamWBF16, with implementations and fallbacks for Blackwell, Hopper, and older GPUs.
RL is asynchronous, with a clean Transformers ↔ vLLM/SGLang split. Multi-turn environment rollouts run as Ray actors against a vLLM or SGLang server and overlap training through a prefetch queue; the trainer pushes updated weights to the server over native NCCL. There is no Megatron backend and no veRL dependency — the training side stays plain Transformers, with the same parallelism and checkpointing stack. See Async GRPO with Environments.
Latest updates
- 2026-08-20 — Halo 1.0.0. First public release: EP / CP / TP / ETP on native HuggingFace models
(LLM and VLM), 15 MoE families, pre-training through multi-turn RL with vLLM or SGLang rollouts,
FlashAttention-4, DeepEP V2, DeepGEMM, and the bf16
AdamW optimizer.
Quick start
Installation
Halo ships as separate Docker images for Blackwell and Hopper, with PyTorch, Flash Attention, and
DeepEP V2 already built in — pull anonymously, no account required:
# B200/B300 — FA4 + FA2 + DeepEP
docker pull public.ecr.aws/whitecircle/halo:blackwell
# H100/H200 — FA2 + FA3 + DeepEP
docker pull public.ecr.aws/whitecircle/halo:hopper
Versioned tags (:blackwell-1.0.0, :hopper-1.0.0) pin the release. There is
no latest tag, since the images are architecture-specific. The RL
inference images are published alongside them (:vllm-0.26.0, :sglang-0.5.17).
To build locally instead:
# B200/B300 — FA4 + FA2 + DeepEP
make build-blackwell
# H100/H200 — FA2 + FA3 + DeepEP
make build-hopper
No host Python environment is required. See Installation for image details and build options.
Launch a training run
halo launch takes a training method and a YAML config:
# SFT + LoRA (fits into a single GPU)
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml
# SFT + Expert Parallelism (MoE, 8 GPUs → torchrun)
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml -n 8
# FSDP-2 + Accelerate
halo launch sft examples/sft/qwen3/qwen3-4b-ultrachat.yaml \
-a launcher-configs/accelerate/fsdp2_gradop_config.yaml -n 8
# all training methods (SFT, SMPO, DPO, online-GRPO, environmental-GRPO, ...)
halo launch --list
halo selects python, torchrun, or accelerate launch based on the arguments. Options after the
config that are not the launcher's own are passed to the trainer and override the YAML config:
halo launch sft examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml \
-n 8 \
--expert_parallel_size=8 \
--learning_rate=1e-5 \
--max_length=32000
Use --dry-run to print the command without running it, and --list to see the available methods:
halo launch --list
halo launch sft config.yaml --dry-run
Checkpoint, inference, data, and environment utilities use halo run:
halo run merge-ep-shards --input_dir <ep-checkpoint-dir> --output_dir <merged-dir>
halo run quantize-to-lowp --input_dir <bf16-model-dir> --output_dir <nvfp4-out-dir> --format nvfp4
# all tools (merge-*, quantize-to-lowp, rm-scoring, run-env, …)
halo run --list
The underlying scripts can also be run directly:
python scripts/training/sft.py \
examples/sft/qwen3/qwen3-4b-ultrachat-lora.yaml
torchrun --nproc_per_node=8 scripts/training/sft.py \
examples/sft/qwen3_5/qwen3.5-35b-a3b-ultrachat-ep.yaml
Coding agents
The training images ship repo-specific agent skills (discoverable by Claude Code and Codex; add
--build-arg INSTALL_CLAUDE_CODE=1 to bake the CLI into the image) for launching jobs,
choosing parallelism settings, debugging distributed runs, adding models, managing checkpoints,
preparing data, optimizing throughput, and setting up RL.
See skills/ for the available skills.
Documentation
We keep two versions of the docs, depending on whether or not you're AI:
human-docs/ — shorter, task-oriented guides for installing, configuring,
and running Halo.agent-docs/ — the full technical reference, written for coding agents:
model integrations, parallelism, optimization internals, infrastructure, and troubleshooting.
The full reference is also perfectly readable if you're human and want the details.
The human guide:
Installation ·
Quickstart ·
Choosing a Method ·
The halo CLI ·
Writing a Config ·
Datasets ·
Training Methods ·
Rollout Servers ·
Parallelism ·
Clusters & Multi-Node ·
Performance ·
Supported Matrix ·
Supported Models ·
Model Integration Cost ·
Checkpoints & Export ·
Monitoring ·
Troubleshooting ·
Environment Variables ·
AI Tooling ·
Contributing
Repository layout
A map of the repository:
| Path |
What's inside |
src/ |
Trainers, parallelism, optimizers, kernels, collators, and callbacks |
scripts/ |
Training entry points, inference, data-prep, and checkpoint tools |
examples/ |
Ready-to-run YAML configs, grouped by method and model |
launcher-configs/ |
Accelerate, SkyPilot, and Nomad launch configs |
human-docs/ |
Short, task-oriented documentation |
agent-docs/ |
Full technical reference |
skills/ |
Agent skills (linked from .claude/skills and .agents/skills) |
tests/ |
CPU and distributed GPU tests |
Dockerfile · Makefile |
Image builds, tests, and training commands |
AGENTS.md |
Contributor and agent instructions |
A typical run is: pick an examples/ config → halo launch <method> <config> (or python scripts/training/<method>.py <config>), inside the Docker image.
Benchmarks
Benchmarks below were run on 8× B300. The stock TRL baseline uses trl.SFTTrainer with
Transformers v5 and FSDP2 ZeRO-3, with the same model, data, bf16 precision, FlashAttention 4,
Liger kernels, and grouped GEMM.
| Result |
Configuration |
| 2.3–2.8× stock TRL throughput |
gpt-oss-20b, 4k–16k; EP2 / dense EP1. Loss matches the baseline to ~1% by step 100. |
| 24,456 tok/s/GPU |
gpt-oss-20b, 4k, batch 4, GC off — ~196k tok/s across 8 GPUs. |
| 12,584 tok/s/GPU at 1,410 TFLOPS |
Qwen3.5-35B-A3B, 4k, batch 4, EP2 — the highest achieved TFLOPS of the MoE rosters. |
| Up to 256k context |
gpt-oss-20b; dense EP1 is 2.1× faster than TRL at 64k and 1.28× at 256k. CP configurations cut per-GPU memory to roughly half the baseline. |
| 23–76 GB/GPU on the same 16k workload |
EP8+CP8: 23 GB at 5.4k tok/s/GPU. Dense EP1: 76 GB at 18.3k tok/s/GPU. |
| 2.12× Grouped GEMM |
Qwen3-30B-A3B, EP2; 3.43× at batch 1. |
| 2.1–3.7× FA4 kernel throughput |
FA4 vs FA2; up to 2.3× end-to-end on dense long-context training. |
Full results and methodology: Performance
Parallelism
EP, CP, TP, and ETP are configured independently:
--expert_parallel_size=N # MoE experts across GPUs (DeepEP)
--context_parallel_size=N # long sequences (Ulysses attention)
--tensor_parallel_size=N # attention weights (DTensor)
--expert_tensor_parallel_size=N # expert-FFN sharding (MoE, experimental)
| Mode |
Best for |
| EP |
Distribute MoE experts across GPUs |
| TP |
Shard model weights across GPUs |
| CP |
Split long sequences across GPUs |
| ETP |
Shard individual expert FFNs |
| EP+CP |
MoE + long-context training |
| EP+TP |
MoE + weight sharding |
| EP+ETP |
MoE + sharded experts |
Pass them directly to torchrun, or after the config with halo launch:
halo launch sft config.yaml -n 8 --expert_parallel_size=8
Multi-node runs support shared (NFS/Lustre) and node-local filesystems.
See Parallelism to pick a mode, Supported Matrix
for the per-model matrix, and Clusters & Multi-Node for the multi-node recipes.
Optimizations
Enabled by default where supported:
Attention and kernels — Flash Attention 4 on Blackwell, FA2/FA3 on Hopper, Liger fused
CE/SwiGLU/RMSNorm, and Grouped GEMM for MoE expert compute.
Precision and optimizers — AdamWBF16 with stochastic rounding uses 6 bytes/parameter of
optimizer state instead of 12 with FP32 master weights. Muon and FlashAdamW are also supported.
FP8/FP4 MoE training is available through fake-quant QAT and DeepGEMM, with mxfp8/nvfp4 export.
Memory and PEFT — padding-free, boundary-aware packing with cu_seq_lens, plus LoRA and QLoRA.
QLoRA fits a 24 GB consumer GPU: Qwen3-4B peaks at 7.9 GB (batch 1, 4k) and Qwen3-8B at 32k fits in
20.9 GB with the fused loss. Ampere and Ada cards run single-GPU LoRA/QLoRA from halo:blackwell (FA2/SDPA).
Checkpoints — large gathered checkpoints are automatically sharded, with tools for merging
EP/TP shards and PEFT adapters back into a standard HuggingFace checkpoint.
→ Checkpoints
Data — offline tokenization, packing, and sharding, plus native s3:// dataset streaming.
→ Performance — what each lever buys, and the bottlenecks they attack
Built with AI
We use GPT and Claude extensively while developing Halo, through
PI-Agent,
Claude Code, and
Codex.
AGENTS.md contains the instructions and conventions used by both coding agents and
human contributors.
AI-assisted contributions are welcome. We just ask that contributors understand and stand behind
the code they submit. See CONTRIBUTING.md.
Acknowledgments
Halo builds on a lot of great open-source work, in particular:
- Hugging Face — Transformers, TRL, Accelerate, Datasets, and PEFT.
- Liger Kernel — fused cross-entropy, SwiGLU, and RMSNorm.
- FlashAdamW by Databricks AI Research — memory-efficient
AdamW with quantized optimizer states and compact master weights.
- DeepEP and DeepGEMM
by DeepSeek-AI — expert communication and grouped GEMM. Halo's aux-loss-free router balancing also
follows the approach introduced with DeepSeek-V3.
- Flash Attention by Tri Dao and contributors —
FA2, FA3, and FA4.
- vLLM — the rollout/serving engine behind online and
environmental RL.
- PyTorch — DTensor, FSDP2, and
torch.distributed.
- SGLang — generation and NCCL weight synchronization for
online and multi-turn RL.
- SDPG, led by researchers at UCLA — self-distilled policy
gradient for online reinforcement learning.
DeepEP and Flash Attention are vendored as git submodules under vendors/ and retain
their upstream licenses; the build pins live in the Dockerfile.
See pyproject.toml for the full dependency set.
Halo is built and maintained by White Circle.
License
Halo is released under the Halo License — the Apache License 2.0
(APACHE-2.0.txt) as modified by the Supplemental Terms in LICENSE.
Copyright © 2026 White Circle, PBC.
In plain terms: free to use, modify, self-host, and to train your own models and ship them in your
own products — at any scale. The one commercial condition is offering Halo's training functionality
to third parties as a service: that's free up to US $20M in such revenue over any 12 months, and
above that requires a commercial agreement with White Circle. Models trained with Halo by anyone
over $20M in annual revenue must carry a "Trained with Halo" credit.
Questions or commercial licensing: [email protected].
Halo is an open-source framework from White Circle for training large language and multimodal models: pre-training, SFT, preference optimization, and fully asynchronous multi-turn RL in one codebase, from a single GPU up to multi-node clusters.