OpenArch README

The project's README from github.com/anuj0456/OpenArch. As one markdown file: README.md.

OpenArch

Python implementations of modern open-source LLM architectures — written from scratch, one model at a time.

This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery. Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery.

The goal is not to compete with transformers or other production libraries. The goal is clarity and learning: a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side.

Why this repo?

Modern LLM architectures share a common skeleton but differ in dozens of small, important choices:

Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading.

What's implemented (so far)

Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction.

Modality Model Status Model Size Normalization Positional Encoding Attention Mixture of Experts
Text GPT-2 XL ✅ 1.5B - Absolute Multihead Attention No
Llama 2 ✅ 7B RMS Norm RoPE Multihead Attention No
Llama 3 ✅ 8B RMS Norm RoPE Grouped Query Attention No
OLMo 2 ✅ 7B RMS Norm & QK-Norm RoPE Multihead Attention No
DeepSeek R1 ✅ 671B RMS Norm & QK-Norm RoPE Multihead Latent Attention Yes
Gemma 3 ✅ 27B RMS Norm & QK-Norm RoPE Grouped Query Attention with Sliding Window No
Mistral 3 ✅ 24B RMS Norm RoPE Grouped Query Attention with Sliding Window No
Llama 4 Maverick ✅ 400B RMS Norm RoPE Grouped Query Attention Yes
Qwen 3 ✅ 4B RMS Norm & QK-Norm RoPE Grouped Query Attention No
30B-A3B RMS Norm & QK-Norm RoPE Grouped Query Attention Yes
Kimi K2 ✅ 1T RMS Norm RoPE Multihead Latent Attention Yes
GLM 4.5 ✅ 355B RMS Norm & QK-Norm RoPE Grouped Query Attention & Multi-Token Prediction Yes
GPT-OSS ✅ 20B RMS Norm RoPE Grouped Query Attention with Sliding Window Yes
Grok-2.5 ✅ 270B RMS Norm RoPE Grouped Query Attention Yes
Qwen3 Next 🚧 80B-A3B RMS Norm RoPE Gated Attention Yes
Multimodal PaliGemma ✅ 3B RMS Norm RoPE Multihead Attention No
Qwen3-VL 🚧 3B RMS Norm RoPE Multihead Attention No
Image Dall-e 🚧 - - - Transformer -

The full target list mirrors the 103 architectures in the Architecture Gallery. Contributions toward any of them are welcome.

Repository layout

OpenArch/
├── text/
│   ├── gpt2/
│   │   ├── model.py
│   │   └── README.md
│   ├── llama3/
│   ├── qwen3/
|   ├── grok2.5/
│   └── deepseek_v3/
├── multimodal/
│   └── pali-gemma/
│       ├── model.py
│       └── README.md
├── README.md
└── requirements.txt

Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used.

Contributing

I am actively looking for contributors. If you enjoy reading model papers, comparing config.json files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start.

Good first contributions:

Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first.

See CONTRIBUTING.md for more details.

Acknowledgements

This repository would not exist without the work of two outstanding educators:

Any errors in the implementations here are entirely my own.

License

This project is licensed under the Apache License 2.0 — see LICENSE for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics.

Disclaimer

These implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a learning resource and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or transformers.

OpenArch (anuj0456/OpenArch)

Searching Database…

426 ·GitHub
OpenArch
What is this?A set of hand-written PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi and more), written from scratch for readability and learning.
Is this a library I can build on, or a collection meant for reading code?A learning resource, not a production library: one readable file per architecture, with design choices like attention type, normalization and MoE routing made explicit and easy to compare side-by-side.
How do I get the code and start exploring a model?Clone the repo and open a model's folder: each architecture lives in its own folder with a model.py and a short README describing the choices and references behind it.
What do I need installed to run one of these models?Python and PyTorch: each architecture is a single hand-written PyTorch model file you can open and run, with no other dependencies named in the repo.
Can I train models with it, or just run them?It's a read-and-run resource, not production code: entries marked ✅ support forward passes, ones marked 🚧 are still under construction, and the README points production users to the official implementations or transformers.
Can I use the code freely in my own projects?Open source — the README states Apache License 2.0 (GitHub metadata lists MIT), and individual model implementations follow the original models' licenses where applicable.
Why reimplement these instead of reading the official model code?Production repos optimize for speed, sharding and backward compatibility, which makes them hard to read. OpenArch optimizes for reading, so the structural decisions are visible and comparable side-by-side.
How big are the models it covers?From GPT-2 XL at 1.5B parameters up to Kimi K2 at 1T, including DeepSeek R1 (671B) and Llama 4 Maverick (400B). The repo implements the architectures; it does not ship the weights.
Is the project finished?No — 18 of the 103 architectures in Raschka's gallery are implemented so far, three entries are still marked under construction, and the author is actively looking for contributors.
What are similar projects?llm-architecture-from-scratch, LLMs-from-scratch, LlamaFactory, llama3-from-scratch, minGPT, vllm, train-llm-from-scratch, ollama
Which topics is it in?llama3, gemma, mistral, from-scratch, mixture-of-experts, attention-mechanism, language-model, moe, llms, grok, transformers, architecture
Creator
anuj0456
Size
0.1 MB
Stars
426
Updated
CommitSep 26
Age
4mo
Commits · 90 Days