noise README

项目的 README,来自 github.com/qlabs-eng/noise。合并为一个 markdown 文件:README.md。

Noise: Speedrun for pretraining without backpropagation

Noise baselines

Noise is a pretraining speedrun for learning algorithms that never run a backward pass. We think replacing backprop with a much better credit assignment algorithm is inevitable. Deep learning's architectures, optimizers and hardware have co-evolved with backprop, but as compute grows, the bitter lesson suggests that more general, brute-force learning algorithms based on search will win. Dust is a big step forward: it rivals backprop under SGD and is 1,000 to 10,000x more efficient than EGGROLL, the state-of-the-art weight-space ES, but still trails backprop under Adam. These methods have also had a small fraction of the research that went into backprop. We made Noise to close the gap: better estimators, and optimizers co-designed for them instead of for backprop gradients.

The model, the data and the score are fixed: the 37.7M-parameter transformer of Dust, a 30M-token FineWeb training set, and the cross-entropy on Dust's validation set. What changes is the learning algorithm, which may only run the model forward; track 3 also frees the architecture. Until October 22, 2026, the training set was Dust's 10M tokens; those tracks and records are archived in archive/10M/.

Tracks

The 10M-token tracks are archived in archive/10M/ and no longer accept submissions.

  1. Validation loss. The lowest validation loss after one epoch on one 8xH100 node, with no time limit.
  2. Speedrun. The shortest training time on one 8xH100 node to reach validation loss 3.5.
  3. Open architecture. Track 1 with the architecture free: the model is train/arch/model.py, which entries may change.

Rules

Setup

Requires Linux, an NVIDIA GPU with bfloat16 support and uv.

uv sync
uv run utils/prepare_data.py

utils/prepare_data.py streams FineWeb sample-10BT, tokenizes it with the fixed BPE-4096 tokenizer in tokenizer/, and writes data/train.pt and data/val.pt. val.pt is the Dust repo's; train.pt is Dust's ~20.5M-token training pool followed by the documents after its validation and test documents, so its first 4,880 sequences are Dust's train_10M.pt. Both are checked against pinned hashes.

Running the records

uv run torchrun --standalone --nproc_per_node=8 train/val/run.py         # track 1
uv run torchrun --standalone --nproc_per_node=8 train/speedrun/run.py    # track 2
uv run torchrun --standalone --nproc_per_node=8 train/arch/run.py        # track 3
uv run torchrun --standalone --nproc_per_node=8 train/val/dust.py        # Dust w/ Adam

Each run writes config.json, metrics.jsonl, final.pt and result.json (validation loss and training time) to logs/. uv run utils/eval.py logs/val/final.pt rescores a checkpoint.

Records

Track 1: validation loss (30M)

# Val loss Description Time Script Contributors
1 3.767 Dust w/ Adam, 24k population, lr cooldown 5 h 17 min Script -
2 3.409 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge 1 h 43 min Script @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned
3 3.338 A forward readout of cross-token credit along each token's path; a fixed Boost, lower ANVIL II weight decay 56 min Script @CharlesDDDD
4 3.300 #3 with two-hop credit through next-token twins and a transit decode; head transplants, self-distillation, a planted previous-token head, a linear decay 43 min Script @CharlesDDDD

Track 2: speedrun to 3.5 (30M)

# Val loss Description Time Script Contributors
1 3.409 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge 1 h 43 min Script @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned
2 3.499 The identity path first, then Cut-lite at 8 rows per update with faster chain reruns 8 min 20 s Script @RobertTLange
3 3.494 A shorter identity phase, then Cut with paired MLP roots and a difficulty-balanced row order 2 min 3 s Script @conglu1997
4 3.498 #3's identity phase, then Cut with two single roots, fused Gluon chain kernels and overlapped bf16 collectives 22.9 s Script @RobertTLange
5 3.496 #4 with root 4 on importance-drawn tokens (root 6 as its control), root 6 along its routed span, 16 rows per update in the second half, overhead in CUDA graphs, a fixed end merge, word-mix row pruning, warp-specialized attention kernels 15.9 s Script @CharlesDDDD
6 3.497 #4 with 16-row Cut updates from 10% of Cut, root 4 on every 2nd update early, per-head q/k polar, an in-run per-part end merge on held-out rows, tensor-core and Gluon credit kernels 14.2 s Script @RobertTLange

Track 3: open architecture (30M)

New track where any architecture is allowed!

# Val loss Description Time Script Contributors
1 3.767 Dust w/ Adam, 24k population, lr cooldown 5 h 17 min Script -
2 3.409 10M record #14: Boost → Cut with own queries, exact values, future-token probes and an end merge 1 h 43 min Script @JaydenTeoh, @spicylemonade, @hoo0681, @Lazarus-931, @CharlesDDDD, @NotDesigned
3 3.378 A quarter-turn previous-token tap in the embedding; exact future-token probes 3 h 50 min Script @dylantirandaz
4 3.358 6 blocks with hashed n-gram tables; future and input-key probes by moments 27 min Script @Euraba
5 3.211 12 × 1024 GPT with hashed n-gram value embeddings; Cut-lite with input-key probes 2 h 57 min Script @RobertTLange

What backprop gets

The same model, data and epoch trained with backprop, scored the same way (seed 0, a few learning rates each):

Optimizer Val loss
Adam w/ lr cooldown 3.419
ANVIL II on the block matrices and Adam elsewhere 3.340

The goal is to beat backprop by a large margin.

Submitting

Open a PR that changes its track's training script in place, train/val/run.py, train/speedrun/run.py or train/arch/run.py (with train/arch/model.py on track 3), with the run's log (config.json, metrics.jsonl and result.json from logs/) in the log/ folder next to it. A record must beat the current one in its track under the rules above; the training script is the record.

Citation

@misc{noise_2026,
  title  = {Noise: Speedrun for pretraining without backpropagation},
  author = {G{\"u}lbahar, Serdar and Mandal, Bishwas and Vegesna, Akshay and Dahal, Samip},
  year   = {2026},
  url    = {https://github.com/qlabs-eng/noise}
}

Core contributors: Serdar Gülbahar and Bishwas Mandal.

License

Released under the MIT License.

noise (qlabs-eng/noise)

正在搜索数据库…

451 ·GitHub
这是什么?Noise 是一个基准测试与速通挑战,用于在完全不运行反向传播的情况下预训练 Transformer,检验其他信用分配算法能否匹敌或超越反向传播。
这是库还是竞赛仓库?这是一个竞赛仓库,模型、数据集和评分脚本都是固定的。你需要通过提交拉取请求(pull request)来直接修改对应赛道的训练脚本,从而提交成绩记录,而不是安装某个软件包。
我该怎么自己运行这个基准测试?运行 utils/preparedata.py,流式下载并分词 FineWeb 样本,生成 data/train.pt 和 data/val.pt,然后执行某个赛道的 run.py;之后可用 utils/eval.py 对任意检查点重新评分。
参加需要什么硬件和工具?你需要 Linux 系统、支持 bfloat16 的 NVIDIA GPU,以及 uv Python 工具;官方记录均在单台 8xH100 节点上计时。
规则下什么算作作弊?任何自动微分(前向或反向模式)或手写的反向传播都会取消参赛资格;学习信号必须仅来自前向计算,并且在赛道 1 和 2 中,模型权重必须严格符合固定架构。
Dust 是什么?Dust 是 Noise 所基于的参考项目:它提供了 37.7M 参数的 Transformer 架构、源自 FineWeb 的训练与验证数据,以及一个无需反向传播的估计器,当前 Noise 的记录都在尝试超越它。
现在非反向传播方法离击败反向传播还有多远?使用 Adam 和学习率冷却(lr cooldown)时,普通反向传播的验证损失为 3.419;目前最好的非反向传播记录以 3.409 略微超过它,而基线种群搜索条目只达到 3.767。
三个赛道有什么区别?赛道 1 追求最低验证损失且不限时间,赛道 2 追求最快达到 3.5 的验证损失,赛道 3 沿用赛道 1 的规则,但还允许参赛者修改模型架构本身。
为什么要避开反向传播?作者认为,随着算力持续增长,更通用的暴力搜索式、基于搜索的学习算法最终会在规模上超越具有反向传播结构的架构和硬件,因此他们希望尽早缩小这一差距。
作者
qlabs-eng
大小
2.3 MB
星标
451
更新
提交Nov 03
时长
1个月
提交 · 90 天