1.5K ·GitHub
What is this?
The official SGLang x Datawhale course on LLM inference: it teaches how inference engines work, has you build a mini-sglang from scratch, and then digs into the real SGLang source code.
The official SGLang x Datawhale course on LLM inference: it teaches how inference engines work, has you build a mini-sglang from scratch, and then digs into the real SGLang source code.Is this a course I read, or a codebase I clone and run?Mostly a course: written chapters served as a docs site (built with VitePress), plus a reference mini-sglang implementation that you code along with in Part II.
Mostly a course: written chapters served as a docs site (built with VitePress), plus a reference mini-sglang implementation that you code along with in Part II.What does the learning path look like if I start today?Four steps: read Part I for inference fundamentals (no GPU needed), build a mini-sglang from zero in Part II, study real SGLang optimizations in Part III, then finish Part IV's profiling, trace analysis and PR workflow.
Four steps: read Part I for inference fundamentals (no GPU needed), build a mini-sglang from zero in Part II, study real SGLang optimizations in Part III, then finish Part IV's profiling, trace analysis and PR workflow.What do I need to know before starting?Proficient Python with basic software-engineering habits, PyTorch and neural-network basics, plus linear algebra and probability. CUDA knowledge is optional — the course fills in what you need as it goes.
Proficient Python with basic software-engineering habits, PyTorch and neural-network basics, plus linear algebra and probability. CUDA knowledge is optional — the course fills in what you need as it goes.Is the course finished, or can I read it all now?It's still being written. Work started Aug 24, 2026 at about one chapter per week: Part 0 is done and Part I is nearly there, but most Part II chapters are still in preparation, with Parts III-IV due by Dec 2026.
It's still being written. Work started Aug 24, 2026 at about one chapter per week: Part 0 is done and Part I is nearly there, but most Part II chapters are still in preparation, with Parts III-IV due by Dec 2026.Can I reuse the materials for my own workshop or company training?Careful: it's licensed CC BY-NC-SA 4.0, so you must give attribution, share adaptations under the same license, and non-commercial use only — company training would need separate permission.
Careful: it's licensed CC BY-NC-SA 4.0, so you must give attribution, share adaptations under the same license, and non-commercial use only — company training would need separate permission.What will I actually have built by the end of Part II?A working mini-sglang engine: forward pass and autoregressive generation, KV Cache, an HTTP server handling concurrent requests, continuous batching with a scheduler, Paged KV Cache, RadixAttention with prefix caching, multi-process tensor parallelism and speculative decoding — added one step at a time.
A working mini-sglang engine: forward pass and autoregressive generation, KV Cache, an HTTP server handling concurrent requests, continuous batching with a scheduler, Paged KV Cache, RadixAttention with prefix caching, multi-process tensor parallelism and speculative decoding — added one step at a time.Is the course available in English?Yes — the project ships in both English and Chinese. Translated chapters go under course-material/eng/, and helping with translation is one of the listed ways to contribute.
Yes — the project ships in both English and Chinese. Translated chapters go under course-material/eng/, and helping with translation is one of the listed ways to contribute.What does Part 0 cover before the main course starts?Two starter chapters: coding ethics and the open-source spirit, and deploying your first SGLang server — you set up the environment and run Qwen3-0.6B on your own GPU with real SGLang before touching any theory.
Two starter chapters: coding ethics and the open-source spirit, and deploying your first SGLang server — you set up the environment and run Qwen3-0.6B on your own GPU with real SGLang before touching any theory.What are similar projects?Model-Optimizer, jeeves, TypeLLM, TensorFold, llm-inference-engineering, Hy4-preview, ds4, text-generation-inference
Model-Optimizer, jeeves, TypeLLM, TensorFold, llm-inference-engineering, Hy4-preview, ds4, text-generation-inferenceWhich topics is it in?kv-cache, attention, from-scratch, sglang, bilingual, speculative-decoding, inference-engine, llm-inference, quantization, transformer, cuda, pytorch
kv-cache, attention, from-scratch, sglang, bilingual, speculative-decoding, inference-engine, llm-inference, quantization, transformer, cuda, pytorch