5.2K ·GitHub
What is this?
This is OpenAI's 'Model Craft Challenge: Parameter Golf' - a competition to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100 GPUs, scored by bits-per-byte compression on the FineWeb validation set.
This is OpenAI's 'Model Craft Challenge: Parameter Golf' - a competition to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100 GPUs, scored by bits-per-byte compression on the FineWeb validation set.Is this a training framework or an open competition?It is an open competition with a live leaderboard, not a reusable framework. Participants submit pull requests with their model architecture/training code and results, inspired by the NanoGPT Speedrunning challenge.
It is an open competition with a live leaderboard, not a reusable framework. Participants submit pull requests with their model architecture/training code and results, inspired by the NanoGPT Speedrunning challenge.How do I submit an entry to the leaderboard?You fork the repo, train your model under the size and time limits, and open a pull request with your run; accepted entries show up on the leaderboard table referencing the PR number.
You fork the repo, train your model under the size and time limits, and open a pull request with your run; accepted entries show up on the leaderboard table referencing the PR number.Do I need my own GPUs to compete?You need access to 8xH100 GPUs for a compliant run, but OpenAI is sponsoring $1,000,000 in compute credits and provides a form to request a compute grant if you don't have the hardware.
You need access to 8xH100 GPUs for a compliant run, but OpenAI is sponsoring $1,000,000 in compute credits and provides a form to request a compute grant if you don't have the hardware.What are the hard limits for a record submission?A leaderboard-eligible run must produce a model artifact under 16MB and complete training in under 10 minutes on 8xH100s; runs that exceed this budget can still be submitted under a separate 'Non-record Submissions' category.
A leaderboard-eligible run must produce a model artifact under 16MB and complete training in under 10 minutes on 8xH100s; runs that exceed this budget can still be submitted under a separate 'Non-record Submissions' category.Why is OpenAI running this challenge?It's framed as a technical talent scouting effort: OpenAI plans to hire a small cohort of early-career researchers in June, including Olympiad medalists and elite competitors, and strong submissions can help participants stand out to recruiters.
It's framed as a technical talent scouting effort: OpenAI plans to hire a small cohort of early-career researchers in June, including Olympiad medalists and elite competitors, and strong submissions can help participants stand out to recruiters.How exactly is a submission scored?Models are evaluated by compression (bits per byte) on the FineWeb validation set, and the metric is tokenizer-agnostic so different tokenization schemes can be fairly compared.
Models are evaluated by compression (bits per byte) on the FineWeb validation set, and the metric is tokenizer-agnostic so different tokenization schemes can be fairly compared.What kinds of tricks are leaderboard entries using?Top entries combine techniques like custom tokenizers (CaseOps), quantization-aware training, GPTQ calibration, sparse/variable-length attention, depth recurrence, and test-time training (TTT) stacked on top of each other across successive PRs.
Top entries combine techniques like custom tokenizers (CaseOps), quantization-aware training, GPTQ calibration, sparse/variable-length attention, depth recurrence, and test-time training (TTT) stacked on top of each other across successive PRs.What inspired Parameter Golf?It's modeled on the NanoGPT Speedrunning challenge, but instead of optimizing time-to-loss (L(T)) or data efficiency (L(D)), it optimizes loss for a fixed parameter count (L(N)).
It's modeled on the NanoGPT Speedrunning challenge, but instead of optimizing time-to-loss (L(T)) or data efficiency (L(D)), it optimizes loss for a fixed parameter count (L(N)).What are similar projects?lectures, train-llm-from-scratch, DeepGEMM, InstinctFlash, stanford_alpaca, llama, MSA, decider
lectures, train-llm-from-scratch, DeepGEMM, InstinctFlash, stanford_alpaca, llama, MSA, deciderWhich topics is it in?triton, tokenizer, compression, attention-mechanism, language-model, dataset, quantization, large-language-models, nlp, nvidia, cuda, benchmark
triton, tokenizer, compression, attention-mechanism, language-model, dataset, quantization, large-language-models, nlp, nvidia, cuda, benchmark