parameter-golf README

The project's README from github.com/openai/parameter-golf. As one markdown file: README.md.

1920x640-discord

OpenAI Model Craft Challenge: Parameter Golf is a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte).

This challenge is heavily inspired by the NanoGPT Speedrunning challenge, where participants compete to train a model that reaches 3.28 FineWeb validation loss as quickly as possible. We're excited to see how optimizing for a parameter-constrained setting pushes people toward unique architectures (test-time compute, aggressive parameter tying, depth recurrence, low-rank training, ...), compression schemes (low precision, QAT, bitnets, novel tokenizers, ...), and other creative submissions (test-time training, long context, megakernels ...).

If you're familiar with neural scaling laws, you can consider this challenge a form of L(N) optimization, where the objective is to optimize the lowest loss given a fixed number of parameters (N) unconstrained by data, compute, steps, or architecture. Challenges like the NanoGPT Speedrun, which optimizes for a form of L(T) (~lowest time given constrained loss) or the NanoGPT Slowrun, which optimizes for L(D) (lowest loss given constrained dataset size), can be thought of as equivalent challenges in this family.

Ideally, we'd allow for submissions to use arbitrary computational resources. But in order to make the challenge not inaccessibly expensive, we're limiting leaderboard submissions to 10 minutes on 8xH100s. However, we'd still love to see submissions that don't meet the compute limitation requirements in our 'Non-record Submissions' section: We're excited to see people push the infinite frontier of parameter limited performance as well.

We also know compute is expensive, so OpenAI is sponsoring $1,000,000 in compute credits to help people get started training their models. To request a compute grant, use this form: Request a Compute Grant. When requesting compute, please make sure you choose the appropriate level, write sufficient justification, and submit with an email tied to a OpenAI / ChatGPT account.

Participant Form

If you enjoy solving very difficult technical problems, please introduce yourself via the Challenge Participant Form. It helps us attribute challenge submissions and reach out about opportunities with OpenAI. Completing the form is not required to participate.

Many researchers at OpenAI first distinguished themselves through elite mathematics and programming competitions. The Model Craft Challenge is designed in that spirit: testing the ability to tackle unfamiliar problems with creativity and rigor, qualities we believe are essential for frontier AI research.

In June, we plan to hire a small cohort of early-career researchers, targeting current undergraduate students and recent graduates, including Olympiad medalists and elite competitors. For exceptional participants, the challenge may also serve as a way to stand out to OpenAI researchers and recruiters.

The challenge runs from March 18th to April 30th.

Happy training!

Leaderboard

Run Score Author Summary Date Info
Calib32 Token-Only N-gram + AsymLogit Stack 1.0565 codemath3000 On PR #2135: pre-cutoff PR #2130 architecture rerun on clean canonical CaseOps data with GPTQ_CALIBRATION_BATCHES=32; 3-seed mean 1.05651 under grace policy (p=0.014 vs PR #2014) 2026-05-01 info
Progressive Context Growth + Short-Doc Score-First TTT 1.0576 simonbissonnette On PR #2014: PR #1855/#1953 CaseOps stack with progressive context growth to 3k plus short-doc score-first TTT on the AWQ-lite/AsymLogit lineage; 3-seed mean 1.05759 (p=0.011 vs PR #1953) 2026-04-30 info
Long-Context No-Q/V TTT + QK-Gain 5.25 1.0586 andrewbaggio1 On PR #1953: PR #1945 V21 base with 2560 eval/TTT context, no-Q/V TTT mask, TTT LR 0.75, and QK_GAIN_INIT=5.25; 3-seed mean 1.05855 (p=0.063 vs PR #1945 V21 v2) 2026-04-30 info
AWQ-Lite GPTQ + AsymLogit on PR1855 Stack 1.0594 alertcat On PR #1945 commit 70067534: PR #1855 stack plus PR #1908 AWQ-lite mixed GPTQ and PR #1923 AsymLogit; V21 v2 3-seed mean 1.05943 after strict seed-42 rerun (p=0.034 vs PR #1855) 2026-04-29 info, commit
BOS-Fixed SmearGate + LQER + SparseAttnGate + 9-Hparam Stack 1.0611 codemath3000 On PR #1855: BOS-fixed #1797-derived stack with LQER, PR #1787 SparseAttnGate/PolarNS/FusedCE base, per-group lrzip compression, and 9 greedy hyperparameter overrides; submitted 3-seed mean 1.06108 with broader reproduction support (p=0.188 vs PR #1868 latest rerun) 2026-04-27 info, repro
BOS-Fixed SmearGate + LQER Asymmetric + PR1787 SparseAttn + Phased TTT 1.0614 aquariouseworkman On PR #1851 with 3-seed compliance-rerun support from PR #1868: BOS-boundary fix from PR #1851 applied to dexhunter's PR #1797 SmearGate + LQER stack, using the PR #1787 SparseAttnGate/PolarNS/FusedCE base plus CaseOps and phased score-first TTT 2026-04-27 info, 3-seed
PR1736 + PolarNS + MIN_LR + SparseAttnGate + FusedCE + Warm-A TTT 1.0634 nprime06 On PR #1787: PR #1736 CaseOps stack plus Polar Express Newton-Schulz coefficients, MIN_LR=0.1, SparseAttnGate, fused softcapped CE, and PR #1767-style warm-start-A TTT 2026-04-23 info
CaseOps + MLPClip12 + SmearGate/LoRA-TTT 1.0645 dexhunter On PR #1769: CaseOps stack with SmearGate/LoRA-TTT refinements and MLPClip12; 5-seed mean improves the accepted CaseOps frontier (p=0.063 vs #1736) 2026-04-22 info
SP8192 + CaseOps + GatedAttn + QuantGate + Loop45 + Phased TTT 1.0655 dexhunter On PR #1736: adopts romeerp's lossless CaseOps transform from PR #1729 with byte-sidecar BPB accounting, then adds gated attention and quant-gate scaling on the PR #1530 SP8192 phased-TTT stack 2026-04-19 info
CaseOps Tokenizer + Tapered WD + Phased TTT 1.0678 romeerp On PR #1729: lossless CaseOps bijective case transform with validation byte sidecars, plus mild late Muon weight-decay taper on the PR #1626 legal phased-TTT stack 2026-04-19 info
SmearGate + Attention Output Gate + Legal TTT 1.0714 MarioPaerle On PR #1667: SmearGate, attention output gate, depth recurrence, parallel residuals, QK-Gain 5.25, quantization, and score-first TTT 2026-04-16 info
VarLen Attention + Fused MLP + Multi-Phase Global SGD TTT 1.0719 dexhunter On PR #1626: VarLen attention, fused MLP, multi-phase global SGD TTT, trimmed GPTQ, MLR 0.026, int7 embeddings, and adaptive clip 2026-04-14 info
VarLenAttn + PhasingTTT 1.0728 romeerp On PR #1610: #1530-style VarLen/fused stack plus phased TTT over already-scored validation chunks 2026-04-13 info
VarLen Attention + Fused MLP + Doc-Independent Legal TTT 1.0734 samacqua On PR #1530: variable-length FA3 attention, fused Triton MLP, grouped small-parameter all-reduces, and doc-independent score-first LoRA TTT 2026-04-11 info
Improved Parallel Residuals + CUTLASS EVT + Legal TTT 1.0758 msisovic On PR #1529: PR #1523 SP8192 baseline with fuller two-lane parallel residual routing, PARALLEL_RESIDUAL_START=8, inline CUTLASS EVT/Triton fused kernels, and legal score-first TTT; corrected 3-seed mean after GPTQ reserve/seed fix (p=0.001 vs #1514) 2026-04-11 info
SP8192 + Muon 0.97 + Legal Score-First TTT 1.0798 dexhunter On PR #1514: SP8192 with Muon 0.97 and legal score-first TTT; 3-seed sweep beats #1493 (p=0.020) 2026-04-09 info
SP8192 + 3-Layer Recurrence + Parallel Residuals + Legal TTT 1.0810 bigbag On PR #1493: 3-layer recurrence, parallel residuals, QK-Gain 5.25, and legal score-first TTT on the PR #1394 stack 2026-04-09 info
SP8192 + Parallel Residuals + Score-First TTT 1.0822 aryanbhosale On PR #1477: parallel residuals on the PR #1413 SP8192 + legal score-first TTT stack 2026-04-08 info
SP8192 + QK-Gain 5 + Legal Score-First TTT 1.0828 dexhunter On PR #1413: QK-Gain 5.0 + legal score-first TTT on the PR #1394 SP8192 stack 2026-04-06 info
SP8192 + Parallel Residuals + Hessian-Aware SDClip 1.0835 Robby Sneiderman On PR #1412: parallel residuals, Hessian-aware SDClip, and progressive recurrence on the PR #1394 stack 2026-04-06 info
SP8192 + GPTQ Embeddings + Depth Recurrence + SDClip 1.0856 Kevin Clark On PR #1394: SP8192, GPTQ embeddings, looped layers 4-5, MuonEq-R, and std-based GPTQ clipping 2026-04-05 info
SP4096 + Depth Recurrence + Parallel Residuals + MuonEq-R 1.0897 aryanbhosale On PR #1334: SP4096 + depth recurrence + parallel residuals + MuonEq-R + QK-Gain 5.0 2026-04-04 info
MuonEq-R + Depth Recurrence + WD=0.090 + All-Int6 GPTQ 1.0912 dexhunter On PR #1285: MuonEq-R + layers 4-5 recurrence + higher weight decay + all-int6 GPTQ 2026-04-03 info
4096-Vocab + Larger Model + High WD + Simplifications 1.0979 Kevin Clark On PR #1218: SP4096 + 4x MLP + high weight decay, with TTT, hash embeddings, SmearGate, and value residuals removed 2026-04-01 info
Parallel Residuals + Mini Depth Recurrence 1.1063 Marko Sisovic On PR #1204: mini recurrence on layers 4-5 + parallel attention/MLP residual lanes + AR self-generated GPTQ calibration 2026-03-31 info
Rascal 1.1099 newjordan On PR #1120: XSA-all + Parallel Muon + coprime loader + Bigram2048/RoPE16 + SWA/late QAT without GPTQ 2026-03-30 info
Coprime-Stride Loader + Full GPTQ + XSA-all 1.1122 dexhunter On PR #1060: coprime multi-shard loader + Full Hessian GPTQ + XSA on all layers + BigramHash(2816x112) 2026-03-29 info
11L AR Self-Gen GPTQ + XSA 1.1147 abaybektursun On PR #1019: Self-Generated GPTQ Calibration Data + all-layer XSA on the PR #549 stack 2026-03-25 info
LeakyReLU² + Legal Score-First TTT + Parallel Muon 1.1194 abaybektursun On PR #549: LeakyReLU(0.5)^2 + TTT + Parallel Muon on the PR #414 stack 2026-03-23 info
11L EMA + GPTQ-lite + warmdown3500 1.1228 signalrush On PR #374: GPTQ-lite clip search + EMA, plus warmdown3500 and [email protected] 2026-03-22 info
11L Partial RoPE + LN Scale + EMA + XSA4 1.1248 jfprincz On PR #287: Partial RoPE (16/64) + layerwise LN scale 2026-03-21 info
11L XSA4 + EMA + Int6 MLP3x 1.1271 jfprincz On PR #198: XSA on the last 4 layers + EMA replacing SWA 2026-03-20 info
11L Efficient Partial XSA 1.1307 unnir On PR #198: Efficient Partial XSA on the deepest 3 layers 2026-03-20 info
10L Int5-MLP + BigramHash(10240) 1.1428 thwu1 10 layers, mixed int5/int6 quantization, BigramHash(10240), SWA(0.4), WD=0.04 2026-03-20 info
Int6 MLP3x + SmearGate + BigramHash 1.1458 Raahil Shah 3x MLP + SmearGate + BigramHash + OrthoInit + Muon WD + SWA 2026-03-20 info
11L MLP3x + Int6 QAT 1.1502 aruniyer 11 layers, 3x MLP, int6 QAT, zstd-22, WD=0.04, sliding eval 2026-03-20 info
SmearGate + OrthoInit + Muon WD 1.1556 aquariouseworkman SmearGate + BigramHash + 3x MLP + int6 STE QAT + sliding eval 2026-03-19 info
Ternary Quantization 1.1570 Ciprian-Florin Ifrim 73.7M params quantized to 1 0 -1 + misc arch changes 2026-03-24 info
10L Int6 QAT + Zstd MLP2.6x 1.1586 yahya010 10 layers, int6 QAT + zstd-22, MLP 1344, Muon 0.99, sliding eval 2026-03-19 info
Mixed Quant + Sliding Window Eval 1.1630 aquariouseworkman Int6 block weights + int8 embeddings + 3x MLP + sliding eval 2026-03-19 info
Muon WD + 10 layer 1.1748 notapplica Includes prev. wins + Spectral embed init + resid mix 2026-03-19 info
Sliding Window Eval 1.1925 Matthew Li Sliding window evaluation at stride=64, increasing context for eval 2026-03-19 info
Lora TTT 1.1928 samacqua Test-time training with LORAs 2026-03-19 info
4k seq length 1.2014 Spokane Way 4k seq length + better hypers 2026-03-19 info
2048 seq length 1.206 Spokane Way 2048 seq length (train + val) 2026-03-18 info
int6 mixed precision 1.2147 Nan Liu 10 layers, mixed int8/int6 2026-03-18 info
fp16 Embed 1.2197 Renier Velazco FP16 Tied Embedding + LR/Warmdown Tuning 2026-03-18 info
Naive Baseline 1.2244 Baseline 9layer 512dim 1024vocab TiedEmbeddings 4 KV heads 2026-03-18 info

Unlimited Compute Leaderboard & Non-record Submissions

Run Score Author Summary Date Info
1 Bit Quantization 1.1239 CiprianFlorin-Ifrim 106M params quantized to 1 bit + misc arch changes + 2hr training 2026-03-24 info
MDLM Text Diffusion 1.1465 agalimova On PR #1106: masked diffusion LM with absorbing-mask ELBO-style eval, validated on 2xH100 as a discrete diffusion language-model entry 2026-03-29 info
Hymba-8L + Sliding Attention at 32K 1.1467 mkenney2 On PR #1245: hybrid Mamba SSM + sliding-window attention with 32K context, 3-seed under-16MB result, and score-first TTT 2026-04-01 info
Mamba-3 Hybrid SSM + SP8192 + Legal TTT 1.1473 mradassaad On PR #1644: 7-layer Mamba-3 hybrid with 5 SSM blocks, 2 attention layers, SP8192, AR GPTQ, chunk score-first TTT, and stateful-overlap eval 2026-04-15 info
Differential-Gated Attention 1.1898 ddavidgao On PR #542: alternative attention mechanism carrying novelty/differential payloads in deep layers, with analysis of depth-dependent redundancy 2026-03-23 info
Learned Adapters on Random Linear Maps 1.1971 pranavxiyer On PR #2058: random-seeded adapter MLPs with rank-160 learned LoRA-style adapters, 12 layers, 3x MLP, mixed int6/int8 compression, and 3-seed 10-minute evidence 2026-04-30 info
JEPA + Mamba2 LeWorldModel 1.2064 CiprianFlorin-Ifrim On PR #903: ambitious SSM + JEPA latent-prediction submission with long-compute evidence and 10-minute logs 2026-03-26 info
4-Hour Baseline 1.2074 Will DePue Testing unlimited compute, 4 hours on 8xH100 2026-03-18 info
Universal Transformer with Iteration Embeddings 1.2249 gowtham0992 On PR #1110: 3 unique blocks looped 4 times with iteration embeddings, giving 12 effective layers in a 4.95MB artifact 2026-03-30 info
LegendreGPT Depth Parameterization 1.2266 sergimichi On PR #1337: generates transformer weights as smooth Legendre-polynomial functions of layer depth, producing 24 virtual layers in 15.7MB after post-hoc quantization 2026-04-04 info
ByteJEPA 1.3496 hardik-bhadani-git On PR #1443: byte-level JEPA with latent prediction, SIGReg anti-collapse, no tokenizer, and an auxiliary CE head for BPB evaluation 2026-04-07 info
Byte-Level H-Net Dynamic Chunking 1.3595 DariusFeher On PR #1104: systematic H-Net byte-vs-subword study with learned whitespace/word-like boundaries, plus 10-minute and 4-hour evidence 2026-03-30 [info](records/

parameter-golf (openai/parameter-golf)

Searching Database…

5.2K ·GitHub
Parameter-golf
这是什么？这是 OpenAI 的「Model Craft Challenge: Parameter Golf」，一场比赛：训练出在 16MB 的产物大小以内、在 8 块 H100 GPU 上 10 分钟内完成训练的最强语言模型，并以 FineWeb 验证集上的每字节比特数（bits-per-byte）压缩效果来评分。
这是训练框架还是公开比赛？这是一场公开比赛，带有实时排行榜，并不是可以反复使用的框架。参赛者提交 pull request，附上模型架构和训练代码以及结果，灵感来自 NanoGPT Speedrunning 挑战。
我该如何向排行榜提交作品？先 fork 仓库，在大小和时间限制内训练你的模型，然后发起一个包含你的运行结果的 pull request；通过审核的作品会出现在排行榜表格中，并标注对应的 PR 编号。
参赛需要自己有 GPU 吗？要完成符合规则的运行，你需要能使用 8 块 H100 GPU。但 OpenAI 赞助了 100 万美元的算力额度，如果你没有这样的硬件，可以通过表单申请算力资助。
提交正式记录的硬性限制是什么？有资格上榜的运行必须生成小于 16MB 的模型产物，并在 8 块 H100 上于 10 分钟内完成训练；超出此预算的运行仍可以提交到单独的「Non-record Submissions」（非记录提交）类别中。
OpenAI 为什么要举办这场挑战？它被定位为一项技术人才发掘活动：OpenAI 计划在六月招募一小批早期职业研究人员，包括奥林匹克竞赛获奖者和顶尖竞赛选手，表现出色的提交作品可以帮助参赛者在招聘人员面前脱颖而出。
作品具体是如何评分的？模型通过 FineWeb 验证集上的压缩率（每字节比特数）进行评估，该指标与分词器无关，因此不同的分词方案可以得到公平比较。
排行榜上的作品用了哪些技巧？排名靠前的作品会把多种技术叠加使用，例如自定义分词器（CaseOps）、量化感知训练、GPTQ 校准、稀疏/可变长度注意力、深度循环和测试时训练（TTT），并在连续的 PR 中逐步叠加。
Parameter Golf 的灵感来自哪里？它以 NanoGPT Speedrunning 挑战为蓝本，但不是优化达到损失所需的时间 L(T) 或数据效率 L(D)，而是在固定参数量下优化损失 L(N)。
What are similar projects?lectures, train-llm-from-scratch, DeepGEMM, InstinctFlash, stanford_alpaca, llama, MSA, decider
Which topics is it in?triton, tokenizer, compression, attention-mechanism, language-model, dataset, quantization, large-language-models, nlp, nvidia, cuda, benchmark
Creator
openai
Size
18.7 MB
Stars
5.2K
Updated
CommitMay 04
Age
8mo
Commits · 90 Days