tiny-llm README

项目的 README,来自 github.com/skyzh/tiny-llm。合并为一个 markdown 文件:README.md。

tiny-llm

CI (main)

tiny-llm is a hands-on course for systems engineers who want to understand LLM inference end to end. You can think of it as an LLM-serving counterpart to CMU's Needle project: build the path that loads a Qwen3 model, turns tokens into logits, and generates text.

The course begins with array and matrix operations, then introduces kernels and serving machinery as the running model needs them. Keeping the implementation small enough to read end to end makes it possible to connect the equations to memory traffic, kernel occupancy, KV-cache growth, batching, and request scheduling.

The course is built on MLX arrays and the MLX extension runtime, without using high-level neural-network layers. When a chapter teaches an operator, your solution implements that operator in Python, C++, or Metal rather than calling the corresponding optimized MLX operation. MLX remains the correctness oracle and performance baseline.

The Learning Path

The course follows a four-week learning path:

Why MLX and Qwen3?

Apple silicon provides a practical local environment with one shared memory space and direct access to Metal kernels. Students can inspect the complete path on one machine instead of depending on an expensive CUDA GPU setup.

Qwen3-4B is large enough to expose real weight-bandwidth, attention, and cache costs, but small enough to iterate on locally. Its grouped-query attention, QK normalization, BF16 activations, and 4-bit weights also keep the exercises close to current model-serving work.

Start Here

The book is published at skyzh.github.io/tiny-llm. Begin with the environment setup, or verify an existing checkout with:

pdm install -v
pdm run check-installation
pdm run test-refsol -- -- -k week_1

The tiny_llm package is where students implement the exercises. tiny_llm_ref contains the reference solution used by the tests and benchmark appendix. The book summary lists the chapter order; implementation, test, and publication readiness is tracked below.

Roadmap

The table tracks implementation (Code), tests (Test), rendered chapters (Doc), and Chi's review of learner-facing material (Audit). Week 4 is publishing one reviewed day at a time; Days 1 through 9 are currently available to learners. The Audit column reflects Chi's personal editorial pass on the published course content and is independent of code/test/doc readiness.

Day 3 can send file contents to the model, modify files after approval, and run one exact configured command. Use a disposable workspace without secrets and read the Week 4 overview before running the loop. Day 4 checkpoints a complete tool-observation boundary with the scripted model's fake cache metadata, then restores a fresh model without replaying the completed edit or command. Day 5 compacts older completed effects in the model-visible transcript while their exact receipts retain the full action, result, and changed artifacts. Day 6 inspects one complete-observation checkpoint, appends one visible operator instruction, and resumes a fresh model without replaying the completed effect. Day 7 evaluates one completed run from declared final, file, result, and receipt facts without grading hidden reasoning or exact transcript shape. Day 8 reuses one real tokenizer/KV checkpoint for two differently steered, effect-isolated continuations, evaluates both with Day 7's harness, and makes one explicit passing selection without pretending completed effects were rewound. Day 9 stores exact oversized tool-result bytes outside the model prompt, shows a bounded identity/digest/head-tail observation, and lets the model retrieve one explicit byte range through the existing loop.

Week + Chapter Topic Code Test Doc Audit
1.1 Attention ✅ ✅ ✅ ✅
1.2 RoPE ✅ ✅ ✅ ✅
1.3 Grouped Query Attention ✅ ✅ ✅ ✅
1.4 RMSNorm and MLP ✅ ✅ ✅ ✅
1.5 Load the Model ✅ ✅ ✅ ✅
1.6 Generate Responses (aka Decoding) ✅ ✅ ✅ ✅
1.7 Sampling ✅ ✅ ✅ ✅
2.1 KV Cache (kv-cache) 🚧 🚧 ✅ 🚧
2.2 Packed W4 Weights (packed-weights) 🚧 🚧 ✅ 🚧
2.3 Decode Matvec (quantized-matvec) 🚧 🚧 ✅ 🚧
2.4 SIMD Prefill (simd-matmul) 🚧 🚧 ✅ 🚧
2.5 Fused Kernels (rmsnorm, rope, swiglu) 🚧 🚧 ✅ 🚧
2.6 Tiled Attention (tiled-prefill, selected) 🚧 🚧 ✅ 🚧
3.1 Continuous Batching ✅ ✅ ✅ 🚧
3.2 Chunked Prefill ✅ ✅ ✅ 🚧
3.3 Paged KV Cache ✅ ✅ ✅ 🚧
3.4 Direct Paged Attention ✅ ✅ ✅ 🚧
3.5 Paged FlashAttention ✅ ✅ ✅ 🚧
3.6 (optional) MoE (Mixture of Experts) ✅ ✅ ✅ 🚧
3.7 (optional) Speculative Decoding ✅ ✅ ✅ 🚧
4.1 Validated Agent Loop ✅ ✅ ✅ 🚧
4.2 Inspect a Workspace ✅ ✅ ✅ 🚧
4.3 Edit, Validate, and Record ✅ ✅ ✅ 🚧
4.4 Checkpoint and Resume ✅ ✅ ✅ 🚧
4.5 Compact Completed Work ✅ ✅ ✅ 🚧
4.6 Inspect and Steer a Paused Agent ✅ ✅ ✅ 🚧
4.7 Evaluate Observable Outcomes ✅ ✅ ✅ 🚧
4.8 Fork, Steer, and Select ✅ ✅ ✅ 🚧
4.9 Bound Tool Evidence ✅ ✅ ✅ 🚧

Earlier Week 2 chapter URLs not reused by the active six days still serve their historical pages, which are hidden from the book's sidebar. The five-day Day 3 to 5 addresses redirect to their renumbered chapters.

Other topics not covered include quantized or compressed KV caches, cross-request prefix caching, fine-tuning, and long-context techniques.

Community

Join skyzh's Discord server to study with the tiny-llm community.

Join skyzh's Discord Server

Community port: amagilabs/tiny-llm-cuda is an unofficial CUDA (NVIDIA GPU) backend for the reference solution kernels, via MLX's CUDA backend.

tiny-llm (skyzh/tiny-llm)

正在搜索数据库…

这是什么?tiny-llm 是一门动手实践的课程,让系统工程师在 Apple Silicon 上使用 MLX,构建一个小型 Qwen3 模型和类似 vLLM 的迷你服务栈,从头到尾理解大语言模型(LLM)推理的工作原理。
这是我可以导入的库,还是我要跟着学习的课程?这是一本已出版的类书籍课程(skyzh.github.io/tiny-llm),其中有一个名为 tinyllm 的学员代码包,你需要在里面实现每章的练习;另有一个 tinyllmref 包,存放测试所用的参考答案。
我该如何开始学习这门课程?你可以从已出版书籍中描述的环境搭建开始;也可以使用提供的命令,验证已有的本地代码副本,然后再开始第 1 周的内容。
学习这门课程必须要有 CUDA GPU 吗?不需要。整门课程都在 Apple Silicon 的 Mac 上运行,使用 MLX 和 Metal 内核,而不是 CUDA。
整个四周的课程都已经完成了吗?第 1 周和第 3 周已完全实现、测试并附有文档,但第 2 周的几个章节仍标记为编写中;第 4 周的编程智能体内容则是经过审核后每次发布一天,目前已发布到第 9 天。
这门课必须使用 Apple Silicon 吗?Intel 版 Mac 行不行?本课程围绕 Apple Silicon 的统一内存架构和直接的 Metal 内核访问而构建,因此它面向 Apple Silicon 的 Mac,而不支持 Intel 版 Mac 或 Linux/CUDA 机器。
这门课程以哪个模型为核心?Qwen3-4B。它足够大,能体现真实的权重带宽、注意力和缓存开销;同时又足够小,便于在本地反复迭代。其分组查询注意力(grouped-query attention)、QK 归一化、BF16 激活值和 4 位权重,让练习贴近当前的模型服务实践。
为什么用 MLX,而不用 PyTorch 或 CUDA 环境?MLX 为 Apple Silicon 提供了统一内存环境和直接的 Metal 内核访问,让学员无需昂贵的 CUDA GPU,就能在一台机器上看到完整的推理流程;MLX 同时也是每个练习的正确性参照标准和性能基线。
第 4 周为课程增加了什么内容?第 4 周构建了一个有边界、经过验证的编程智能体循环,并连接到一个小型工作区,逐步加入检查点与恢复、上下文压缩、暂停与引导控制,以及基于证据的评估,内容分布在已发布的各天中。
有没有可以一起学习这门课程的社区?有。作者运营着一个 Discord 服务器,学习者可以在那里一起学习 tiny-llm。
作者
skyzh
大小
7.9 MB
星标
5.8K
更新
提交Nov 16
时长
1年
提交 · 90 天