What is this?
A collection of PyTorch implementations of modern LLM architectures (GPT-2, Llama 3/3.2, Llama 4, OLMo 2, Phi-4, DeepSeek v3, Gemma 3, xLSTM, Mistral 3.1) built from scratch to understand how they work internally.
A collection of PyTorch implementations of modern LLM architectures (GPT-2, Llama 3/3.2, Llama 4, OLMo 2, Phi-4, DeepSeek v3, Gemma 3, xLSTM, Mistral 3.1) built from scratch to understand how they work internally.Is this a usable library or more of a study repo?It is a study/reference codebase, not a packaged library or API: each model is reimplemented in PyTorch purely for learning its internals, not for production inference.
It is a study/reference codebase, not a packaged library or API: each model is reimplemented in PyTorch purely for learning its internals, not for production inference.How do I use this?Clone the repository and read/run the per-model PyTorch implementation code for whichever architecture you want to study, since each entry in the table is a separate reimplementation.
Clone the repository and read/run the per-model PyTorch implementation code for whichever architecture you want to study, since each entry in the table is a separate reimplementation.Do I need PyTorch to run any of this?Yes, every architecture is implemented in PyTorch, so you need a working PyTorch environment.
Yes, every architecture is implemented in PyTorch, so you need a working PyTorch environment.Can I actually run the big models like DeepSeek v3 or Llama 4 Maverick locally?These are architecture reimplementations, not pretrained weight downloads, and models like DeepSeek v3 (671B) and Llama 4 Maverick (400B) are far too large to train or run on typical hardware; the code is meant for understanding the architecture, not for practical deployment.
These are architecture reimplementations, not pretrained weight downloads, and models like DeepSeek v3 (671B) and Llama 4 Maverick (400B) are far too large to train or run on typical hardware; the code is meant for understanding the architecture, not for practical deployment.Who is this repo for?Developers and researchers who want to learn how modern LLM internals (attention variants, MoE, normalization, positional encoding) actually work by reading and running from-scratch implementations.
Developers and researchers who want to learn how modern LLM internals (attention variants, MoE, normalization, positional encoding) actually work by reading and running from-scratch implementations.Which model architectures are already implemented?GPT-2 XL, Llama3-8B, Llama 3.2-3B, OLMo 2 7B, Phi-4, DeepSeek v3, Gemma 3, xLSTM, Mistral 3.1 Small, and Llama 4 Maverick, with Qwen3 (32B and 235B-A22B) listed as in progress.
GPT-2 XL, Llama3-8B, Llama 3.2-3B, OLMo 2 7B, Phi-4, DeepSeek v3, Gemma 3, xLSTM, Mistral 3.1 Small, and Llama 4 Maverick, with Qwen3 (32B and 235B-A22B) listed as in progress.Does the repo cover Mixture-of-Experts models?Yes, DeepSeek v3 and Llama 4 Maverick are implemented with Mixture of Experts enabled, while the rest of the table are dense models.
Yes, DeepSeek v3 and Llama 4 Maverick are implemented with Mixture of Experts enabled, while the rest of the table are dense models.Are any of the implemented models multimodal?Gemma 3 and Mistral 3.1 Small are implemented as Text + Image models, while the other architectures are text-only.
Gemma 3 and Mistral 3.1 Small are implemented as Text + Image models, while the other architectures are text-only.What attention and positional encoding techniques are covered?The table spans causal multi-head self-attention, grouped-query attention, and sliding-window + GQA attention, paired with either learned absolute positions or RoPE depending on the model.
The table spans causal multi-head self-attention, grouped-query attention, and sliding-window + GQA attention, paired with either learned absolute positions or RoPE depending on the model.What are similar projects?OpenArch, llama3-from-scratch, train-llm-from-scratch, LLMs-from-scratch, vllm, minGPT, Engram, DMI
OpenArch, llama3-from-scratch, train-llm-from-scratch, LLMs-from-scratch, vllm, minGPT, Engram, DMIWhich topics is it in?from-scratch, qwen3, mixture-of-experts, attention-mechanism, moe, transformers, neural-network, multimodal, large-language-models, transformer, deepseek, learning-resources
from-scratch, qwen3, mixture-of-experts, attention-mechanism, moe, transformers, neural-network, multimodal, large-language-models, transformer, deepseek, learning-resources