llm-inference-engineering README

The project's README from github.com/amitshekhariitbhu/llm-inference-engineering. As one markdown file: README.md.

LLM Inference Engineering

LLM Inference Engineering

Learn LLM Inference Engineering step by step - from KV cache, PagedAttention, and continuous batching to vLLM, SGLang, and GPUs.

Prepared and maintained by the Founder of Outcome School: Amit Shekhar

Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.

Part 1: The Basics

Everything in inference optimization builds on one fact: an LLM generates text one token at a time. Let's understand that generation loop and the two phases behind it first.

Autoregressive Models

In this blog, we will learn about Autoregressive Models, the family of models that generate one piece at a time by predicting the next step from the past.

We will cover the following:

Let's get started: Autoregressive Models

Prefill vs Decode: LLM Inference Optimization

In this blog, we will learn about Prefill vs Decode, the two phases of LLM inference, and how understanding them helps us optimize the speed of an LLM. We will also see how the KV cache connects the two phases and how we optimize each phase to make an LLM faster.

We will cover the following:

Let's get started: Prefill vs Decode: LLM Inference Optimization

Part 2: Memory and Attention Optimizations

Now that we understand the generation loop, let's attack its biggest bottlenecks: repeated computation, wasted GPU memory, and the cost of attention itself.

KV Cache in LLMs

In this blog, we will learn about KV Cache - where K stands for Key and V stands for Value. We will also see why it is used in Large Language Models (LLMs) to speed up text generation.

Let's get started: KV Cache in LLMs

Paged Attention in LLMs

In this blog, we will learn about Paged Attention, a technique that solves the memory waste problem of KV Cache. It allows LLMs to serve many more users at the same time.

Let's get started: Paged Attention in LLMs

Decoding Flash Attention in LLMs

In this blog, we will learn about Flash Attention by decoding it piece by piece - understanding why standard attention is slow, what makes Flash Attention fast, how it uses GPU memory cleverly, and why it is used in almost every modern Large Language Model (LLM).

We will cover the following:

Let's get started: Decoding Flash Attention in LLMs

Grouped Query Attention

In this blog, we will learn about Grouped-Query Attention (GQA) and how it differs from Multi-Head Attention (MHA).

We will cover the following:

Let's get started: Grouped Query Attention

Part 3: Throughput and Latency

With memory under control, the next goal is to serve more users per GPU and make every response feel faster.

Continuous Batching in LLMs

In this blog, we will learn about Continuous Batching, a technique that lets LLM servers handle many more users at the same time by keeping the GPU busy at every single step of generation.

We will cover the following:

Let's get started: Continuous Batching in LLMs

Speculative Decoding

In this blog, we will learn about Speculative Decoding - what it is, why LLM generation is slow without it, how a small draft model and a big target model work together to produce tokens faster, the rejection sampling math that guarantees no quality loss, real numbers showing the 2x to 3x speedup, where it is used in production, and the trade-offs to watch out for.

We will cover the following:

Let's get started: Speculative Decoding

Decoding Medusa

In this blog, we will learn about Medusa, a simple way to make a language model generate text 2 to 3 times faster by giving it several extra heads that guess multiple future tokens at once.

We will cover the following:

Let's get started: Decoding Medusa

Decoding EAGLE

In this blog, we will learn about EAGLE, a state-of-the-art way to speed up language model generation by drafting tokens at the feature level instead of the token level.

We will cover the following:

Let's get started: Decoding EAGLE

How does Token Streaming work?

In this blog, we will learn about how Token Streaming works. We will also see why we need it, how the server and the browser talk to each other to make it happen, and where it is used in real systems like ChatGPT and Claude.

We will cover the following:

Let's get started: How does Token Streaming work?

How does Prompt Caching work?

In this blog, we will learn about how Prompt Caching works. We will also see why we need it, how it actually works inside a large language model, and where it is used in real systems like AI assistants and agents.

We will cover the following:

Let's get started: How does Prompt Caching work?

Part 4: Engines and Formats

These optimizations come together inside real serving engines and model formats. Let's see how the most important ones work.

How does vLLM work?

In this blog, we will learn about how vLLM works. We will also see why we need it, how it manages memory so cleverly, and where it is used in the real world to serve large language models to many users at once.

We will cover the following:

Let's get started: How does vLLM work?

How does SGLang work?

In this blog, we will learn about how SGLang works. We will also see what problem it solves, how it makes serving large language models faster, and the clever ideas that make it special.

We will cover the following:

Let's get started: How does SGLang work?

How does TensorRT-LLM work?

In this blog, we will learn about how TensorRT-LLM works, NVIDIA's own engine that squeezes the highest possible speed out of an NVIDIA GPU when it runs a large language model. We will also see what inference means, what a GPU kernel is, why a normal model run wastes a lot of the GPU's time, how TensorRT-LLM prepares the model ahead of time instead of figuring things out on the fly, how kernel fusion, quantization, the paged KV cache, in-flight batching, CUDA graphs, and speculative decoding each add speed, how one model is spread across many GPUs, and where it works well and where it fails.

We will cover the following:

Let's get started: How does TensorRT-LLM work?

How does GGUF work?

In this blog, we will learn about how GGUF works. We will also see what problem it solves, what is stored inside a GGUF file, how quantization makes big models fit on a normal laptop, and where it is used in real tools.

We will cover the following:

Let's get started: How does GGUF work?

Part 5: Architecture-Level Optimizations

Beyond serving techniques, the model itself can be designed, picked, or shrunk for cheaper and faster inference.

Mixture of Experts Explained

In this blog, we will learn about the Mixture of Experts (MoE) architecture - understanding what experts are, how the router picks them, why MoE makes large models faster and cheaper, and why it powers many of today's most powerful Large Language Models (LLMs).

We will cover the following:

Let's get started: Mixture of Experts Explained

Small Language Models (SLMs)

In this blog, we will learn about Small Language Models (SLMs), what counts as small, why they matter, where they shine, and the trade-offs we must keep in mind.

We will cover the following:

Let's get started: Small Language Models (SLMs)

LLM Routing

In this blog, we will learn about LLM Routing, why it matters, and how to send each user query to the right LLM based on cost, latency, and quality.

We will cover the following:

Let's get started: LLM Routing

How does Knowledge Distillation work?

In this blog, we will learn about how Knowledge Distillation works. We will also see why we need it, how a small model learns from a big model, and how this lets us run powerful AI on a phone, on an edge device, and at low cost.

We will cover the following:

Let's get started: How does Knowledge Distillation work?

Part 6: Hardware

Finally, all of this runs on hardware. Understanding how GPUs and TPUs actually work explains why every optimization above matters.

How does a GPU work for Deep Learning?

In this blog, we will learn about how a GPU works for Deep Learning. We will also see why the GPU is perfect for deep learning, how they do so much math at the same time, and why companies like NVIDIA power almost all of modern AI.

We will cover the following:

Let's get started: How does a GPU work for Deep Learning?

How does a Google TPU work?

In this blog, we will learn about how a Google TPU works. We will also see what a TPU is, why Google built it, how it is different from a CPU and a GPU, and how it makes machine learning fast.

We will cover the following:

Let's get started: How does a Google TPU work?

How does an LPU work?

In this blog, we will learn about how an LPU works, the chip that was built for one single job, running a large language model and printing words on our screen as fast as possible. We will also see what an LPU actually is, how a language model writes text one token at a time, why memory and not math is the real bottleneck, how an LPU keeps the model right next to the compute, how the compiler plans every single cycle in advance, how hundreds of chips work together like an assembly line, and where it works well and where it fails.

We will cover the following:

Let's get started: How does an LPU work?

More blogs and videos coming soon!

License

   Copyright (C) 2026 Outcome School

   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
   You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.

llm-inference-engineering (amitshekhariitbhu/llm-inference-engineering)

Searching Database…

348 ·GitHub
Llm-inference-engineering
这是什么？一个免费且不断扩充的博客系列，循序渐进地讲解 LLM 推理工程，涵盖 KV 缓存、PagedAttention、连续批处理、投机解码，以及 vLLM、SGLang、TensorRT-LLM、GGUF 等服务引擎。
这是动手编程课程，还是一组概念性博客文章？它是一系列文字博客文章，分为五个部分，每部分用图示和带具体数值演算的示例讲透一个概念（如 KV Cache 或 vLLM），而不是可运行的代码实验。
我应该从哪里开始读？从第 1 部分“基础篇”开始，先读讲自回归模型和 Prefill 与 Decode 区别的那篇文章，然后按顺序读完第 2 到第 5 部分，因为每一部分都建立在上一部分之上。
这个指南写完了吗，还是在继续更新？它是一个持续更新的系列，作者会随着时间不断补充新的博客文章和视频，所以并不是一份固定、封闭的课程。
这个指南是为谁写的？想弄懂 LLM 在生产环境中究竟如何被服务的开发者和机器学习工程师，内容从缓存机制一直讲到 vLLM 和 TensorRT-LLM 这样的完整服务引擎。
它涵盖哪些生产级推理引擎？它深入讲解了 vLLM、SGLang、TensorRT-LLM 和 GGUF 格式的内部工作原理，并比较了它们服务 LLM 的不同方式。
它涵盖了哪些加速 token 生成的技术？它讲解了投机解码、Medusa、EAGLE、连续批处理和提示缓存等技术，在不损失质量的前提下让 LLM 生成更快。
它也讲模型架构，而不只是服务吗？讲，第 5 部分深入架构级优化，例如专家混合（MoE）——路由器为每个 token 只激活少数几个专家，从而实现更便宜、更快的推理。
What are similar projects?lectures, vllm, ai-system-design, dgx-spark-qwen38, HyperQwen, qwen3.8-Flash-DGX, GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark, ai-engineering-interview-questions-company-wise
Which topics is it in?inference-optimization, sglang, llm-serving, speculative-decoding, mixture-of-experts, model-serving, attention-mechanism, gguf, llms, caching, curriculum, llm-inference
Creator
amitshekhariitbhu
Size
0.1 MB
Stars
348
Updated
CommitOct 05
Age
2mo
Commits · 90 Days