OpenArch README

The project's README from github.com/anuj0456/OpenArch. As one markdown file: README.md.

OpenArch

Python implementations of modern open-source LLM architectures — written from scratch, one model at a time.

This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery. Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery.

The goal is not to compete with transformers or other production libraries. The goal is clarity and learning: a single readable file per architecture, with the structural choices (attention type, normalization, layer mix, MoE routing, positional encoding) made explicit and easy to compare side-by-side.

Why this repo?

Modern LLM architectures share a common skeleton but differ in dozens of small, important choices:

Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading.

What's implemented (so far)

Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction.

Modality Model Status Model Size Normalization Positional Encoding Attention Mixture of Experts
Text GPT-2 XL ✅ 1.5B - Absolute Multihead Attention No
Llama 2 ✅ 7B RMS Norm RoPE Multihead Attention No
Llama 3 ✅ 8B RMS Norm RoPE Grouped Query Attention No
OLMo 2 ✅ 7B RMS Norm & QK-Norm RoPE Multihead Attention No
DeepSeek R1 ✅ 671B RMS Norm & QK-Norm RoPE Multihead Latent Attention Yes
Gemma 3 ✅ 27B RMS Norm & QK-Norm RoPE Grouped Query Attention with Sliding Window No
Mistral 3 ✅ 24B RMS Norm RoPE Grouped Query Attention with Sliding Window No
Llama 4 Maverick ✅ 400B RMS Norm RoPE Grouped Query Attention Yes
Qwen 3 ✅ 4B RMS Norm & QK-Norm RoPE Grouped Query Attention No
30B-A3B RMS Norm & QK-Norm RoPE Grouped Query Attention Yes
Kimi K2 ✅ 1T RMS Norm RoPE Multihead Latent Attention Yes
GLM 4.5 ✅ 355B RMS Norm & QK-Norm RoPE Grouped Query Attention & Multi-Token Prediction Yes
GPT-OSS ✅ 20B RMS Norm RoPE Grouped Query Attention with Sliding Window Yes
Grok-2.5 ✅ 270B RMS Norm RoPE Grouped Query Attention Yes
Qwen3 Next 🚧 80B-A3B RMS Norm RoPE Gated Attention Yes
Multimodal PaliGemma ✅ 3B RMS Norm RoPE Multihead Attention No
Qwen3-VL 🚧 3B RMS Norm RoPE Multihead Attention No
Image Dall-e 🚧 - - - Transformer -

The full target list mirrors the 103 architectures in the Architecture Gallery. Contributions toward any of them are welcome.

Repository layout

OpenArch/
├── text/
│   ├── gpt2/
│   │   ├── model.py
│   │   └── README.md
│   ├── llama3/
│   ├── qwen3/
|   ├── grok2.5/
│   └── deepseek_v3/
├── multimodal/
│   └── pali-gemma/
│       ├── model.py
│       └── README.md
├── README.md
└── requirements.txt

Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used.

Contributing

I am actively looking for contributors. If you enjoy reading model papers, comparing config.json files, or just want to deepen your understanding of how modern LLMs are built, this is a friendly place to start.

Good first contributions:

Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first.

See CONTRIBUTING.md for more details.

Acknowledgements

This repository would not exist without the work of two outstanding educators:

Any errors in the implementations here are entirely my own.

License

This project is licensed under the Apache License 2.0 — see LICENSE for details. Individual model implementations follow the licenses of the original models where applicable; see each model's folder for specifics.

Disclaimer

These implementations are written to the best of my knowledge based on publicly available papers, technical reports, configuration files, and educational material. They are intended as a learning resource and are not affiliated with or endorsed by the original model authors. For production use, please use the official implementations or transformers.

OpenArch (anuj0456/OpenArch)

Searching Database…

426 ·GitHub
OpenArch
这是什么？一组手写的现代开源大模型架构 PyTorch 实现（Llama、Qwen、DeepSeek、Gemma、GPT-OSS、Kimi 等），全部从零写起，主打易读、适合学习。
这是一个可以拿来开发项目的库，还是主要用来阅读代码的集合？这是一个学习资源，而不是生产级库：每种架构对应一个可读的文件，注意力类型、归一化、MoE 路由这类设计选择都写得明明白白，方便并排对比。
我怎么获取代码并开始研究一个模型？克隆仓库，打开某个模型的文件夹：每种架构都在自己的文件夹里，包含一个 model.py 和一份简短的 README，说明背后的设计选择和参考资料。
要运行其中一个模型，我需要安装什么？Python 和 PyTorch：每种架构都是单个手写的 PyTorch 模型文件，打开就能运行，仓库没有提到其他任何依赖。
我能用它训练模型，还是只能运行它们？这是一个用来阅读和运行的学习资源，不是生产代码：标记 ✅ 的条目支持前向传播，标记 🚧 的还在建设中，README 会引导生产用户去看官方实现或 transformers。
我能在我自己的项目里自由使用这些代码吗？开源——README 写明采用 Apache License 2.0（GitHub 元数据显示为 MIT），各个模型实现在适用的情况下遵循原模型的许可证。
为什么要重新实现这些模型，而不是直接读官方代码？生产代码仓库为了速度、分片和向后兼容做了大量优化，结果就是很难读。OpenArch 专为阅读优化，所以结构性决策一目了然，可以并排比较。
它覆盖的模型规模有多大？从 15 亿参数的 GPT-2 XL 一直到 1 万亿参数的 Kimi K2，中间还包括 DeepSeek R1（671B）和 Llama 4 Maverick（400B）。这个仓库实现的是架构本身，并不附带模型权重。
这个项目已经完成了吗？还没有——Raschka 图鉴里的 103 种架构目前实现了 18 种，还有三个条目标记为建设中，作者正在积极寻找贡献者。
What are similar projects?llm-architecture-from-scratch, how-to-train-your-gpt, LLMs-from-scratch, LlamaFactory, llama3-from-scratch, minGPT, vllm, train-llm-from-scratch
Which topics is it in?llama3, gemma, mistral, from-scratch, mixture-of-experts, attention-mechanism, language-model, moe, llms, grok, transformers, architecture
Creator
anuj0456
Size
0.1 MB
Stars
426
Updated
CommitSep 26
Age
4mo
Commits · 90 Days