README проекта с github.com/qlabs-eng/noise. Одним файлом markdown: README.md.

Noise is a pretraining speedrun for learning algorithms that never run a backward pass. We think replacing backprop with a much better credit assignment algorithm is inevitable. Deep learning's architectures, optimizers and hardware have co-evolved with backprop, but as compute grows, the bitter lesson suggests that more general, brute-force learning algorithms based on search will win. Dust is a big step forward: it rivals backprop under SGD and is 1,000 to 10,000x more efficient than EGGROLL, the state-of-the-art weight-space ES, but still trails backprop under Adam. These methods have also had a small fraction of the research that went into backprop. We made Noise to close the gap: better estimators, and optimizers co-designed for them instead of for backprop gradients.
The model, the data and the score are fixed: the 37.7M-parameter transformer of Dust, a 30M-token FineWeb
training set, and the cross-entropy on Dust's validation set. What changes is the learning algorithm, which may
only run the model forward; track 3 also frees the architecture. Until October 22, 2026, the training set was
Dust's 10M tokens; those tracks and records are archived in archive/10M/.
The 10M-token tracks are archived in archive/10M/ and no longer accept submissions.
train/arch/model.py, which entries may change.utils/model.py with a
strict state-dict match. How training computes the same function, and how it initializes it, is free. On
track 3 the model is train/arch/model.py instead, a copy of utils/model.py that
entries change in place; it must keep the interface utils/eval.py uses (GPT(GPTConfig()), init_weights()
and the summed loss from forward).data/train.pt (30M tokens: 14,648 sequences of 2,048 predictions), for at most one
epoch: each training sequence feeds at most one update. Batch size and order are free.utils/eval.py on the final weights (on track 3 with --model train/arch/model.py): mean next-token cross-entropy over the 1,114,112 predictions of data/val.pt. No
checkpoint selection on validation loss.Requires Linux, an NVIDIA GPU with bfloat16 support and uv.
uv sync
uv run utils/prepare_data.py
utils/prepare_data.py streams FineWeb sample-10BT, tokenizes it with the fixed BPE-4096 tokenizer in
tokenizer/, and writes data/train.pt and data/val.pt. val.pt is the
Dust repo's; train.pt is Dust's ~20.5M-token training pool followed by the
documents after its validation and test documents, so its first 4,880 sequences are Dust's train_10M.pt. Both
are checked against pinned hashes.
uv run torchrun --standalone --nproc_per_node=8 train/val/run.py # track 1
uv run torchrun --standalone --nproc_per_node=8 train/speedrun/run.py # track 2
uv run torchrun --standalone --nproc_per_node=8 train/arch/run.py # track 3
uv run torchrun --standalone --nproc_per_node=8 train/val/dust.py # Dust w/ Adam
Each run writes config.json, metrics.jsonl, final.pt and result.json (validation loss and training time)
to logs/. uv run utils/eval.py logs/val/final.pt rescores a checkpoint.
| # | Val loss | Description | Time | Script | Contributors |
|---|---|---|---|---|---|
| 1 | 3.767 | Dust w/ Adam, 24k population, lr cooldown | 5 h 17 min | Script | - |
| 2 | 3.409 | 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge | 1 h 43 min | Script | @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned |
| 3 | 3.338 | A forward readout of cross-token credit along each token's path; a fixed Boost, lower ANVIL II weight decay | 56 min | Script | @CharlesDDDD |
| 4 | 3.300 | #3 with two-hop credit through next-token twins and a transit decode; head transplants, self-distillation, a planted previous-token head, a linear decay | 43 min | Script | @CharlesDDDD |
| # | Val loss | Description | Time | Script | Contributors |
|---|---|---|---|---|---|
| 1 | 3.409 | 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge | 1 h 43 min | Script | @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned |
| 2 | 3.499 | The identity path first, then Cut-lite at 8 rows per update with faster chain reruns | 8 min 20 s | Script | @RobertTLange |
| 3 | 3.494 | A shorter identity phase, then Cut with paired MLP roots and a difficulty-balanced row order | 2 min 3 s | Script | @conglu1997 |
| 4 | 3.498 | #3's identity phase, then Cut with two single roots, fused Gluon chain kernels and overlapped bf16 collectives | 22.9 s | Script | @RobertTLange |
| 5 | 3.496 | #4 with root 4 on importance-drawn tokens (root 6 as its control), root 6 along its routed span, 16 rows per update in the second half, overhead in CUDA graphs, a fixed end merge, word-mix row pruning, warp-specialized attention kernels | 15.9 s | Script | @CharlesDDDD |
| 6 | 3.497 | #4 with 16-row Cut updates from 10% of Cut, root 4 on every 2nd update early, per-head q/k polar, an in-run per-part end merge on held-out rows, tensor-core and Gluon credit kernels | 14.2 s | Script | @RobertTLange |
New track where any architecture is allowed!
| # | Val loss | Description | Time | Script | Contributors |
|---|---|---|---|---|---|
| 1 | 3.767 | Dust w/ Adam, 24k population, lr cooldown | 5 h 17 min | Script | - |
| 2 | 3.409 | 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge | 1 h 43 min | Script | @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned |
| 3 | 3.378 | A quarter-turn previous-token tap in the embedding; exact future-token probes | 3 h 50 min | Script | @dylantirandaz |
| 4 | 3.358 | 6 blocks with hashed n-gram tables; future and input-key probes by moments | 27 min | Script | @Euraba |
| 5 | 3.211 | 12 × 1024 GPT with hashed n-gram value embeddings; Cut-lite with input-key probes | 2 h 57 min | Script | @RobertTLange |
The same model, data and epoch trained with backprop, scored the same way (seed 0, a few learning rates each):
| Optimizer | Val loss |
|---|---|
| Adam w/ lr cooldown | 3.419 |
| ANVIL II on the block matrices and Adam elsewhere | 3.340 |
The goal is to beat backprop by a large margin.
Open a PR that changes its track's training script in place, train/val/run.py,
train/speedrun/run.py or train/arch/run.py (with
train/arch/model.py on track 3), with the run's log (config.json, metrics.jsonl and result.json from logs/) in the log/ folder next to it. A record must beat the current one
in its track under the rules above; the training script is the record.
@misc{noise_2026,
title = {Noise: Speedrun for pretraining without backpropagation},
author = {G{\"u}lbahar, Serdar and Mandal, Bishwas and Vegesna, Akshay and Dahal, Samip},
year = {2026},
url = {https://github.com/qlabs-eng/noise}
}
Core contributors: Serdar Gülbahar and Bishwas Mandal.
Released under the MIT License.