What is this?
InfraTech is a collection of Jupyter notebooks and articles sharing AI Infra knowledge: training/inference frameworks (PyTorch, vLLM/SGLang, slime/vime), performance acceleration, deep learning fundamentals and AI hardware/software basics.
InfraTech is a collection of Jupyter notebooks and articles sharing AI Infra knowledge: training/inference frameworks (PyTorch, vLLM/SGLang, slime/vime), performance acceleration, deep learning fundamentals and AI hardware/software basics.Is this a tutorial series or a ready-to-use framework?It is a learning repository of standalone Jupyter notebooks and write-ups grouped by topic (inference, PyTorch, vLLM, SGLang, RL), not an installable framework or library you import into your own project.
It is a learning repository of standalone Jupyter notebooks and write-ups grouped by topic (inference, PyTorch, vLLM, SGLang, RL), not an installable framework or library you import into your own project.How do I start going through the material?Clone the repo and open the notebooks under folders like ./llminfer (inference exercises), ./models (model intros) or ./docs (shared AI Infra materials) in Jupyter, then run the code cells.
Clone the repo and open the notebooks under folders like ./llminfer (inference exercises), ./models (model intros) or ./docs (shared AI Infra materials) in Jupyter, then run the code cells.Do I need PyTorch and vLLM/SGLang installed to run the notebooks?Yes, the exercises are written in Python as notebooks, and specific notebooks additionally require PyTorch, vLLM or SGLang depending on the topic (e.g. vLLM memory-management or SGLang RadixAttention notebooks).
Yes, the exercises are written in Python as notebooks, and specific notebooks additionally require PyTorch, vLLM or SGLang depending on the topic (e.g. vLLM memory-management or SGLang RadixAttention notebooks).Is the content a sequential course or loose collection of topics?It is a loose, topic-grouped collection rather than a strict step-by-step curriculum; each notebook has its own difficulty rating shown with lightning-bolt icons (1 to 3 bolts).
It is a loose, topic-grouped collection rather than a strict step-by-step curriculum; each notebook has its own difficulty rating shown with lightning-bolt icons (1 to 3 bolts).Who is this aimed at?Engineers and researchers who already know some deep learning and want to go deeper into LLM inference internals, parallelism strategies and frameworks like vLLM/SGLang rather than complete beginners.
Engineers and researchers who already know some deep learning and want to go deeper into LLM inference internals, parallelism strategies and frameworks like vLLM/SGLang rather than complete beginners.What is vime mentioned in the RL section?vime is an RL framework that combines slime and vLLM, covered here with notebooks on Ascend co-location, MoE-RL training consistency (R3), and training an Agent that calls tools via RL (tau-bench).
vime is an RL framework that combines slime and vLLM, covered here with notebooks on Ascend co-location, MoE-RL training consistency (R3), and training an Agent that calls tools via RL (tau-bench).What is Nano-vLLM used for in this repo?Nano-vLLM is used as a minimal inference framework to teach the core concepts of building an LLM inference engine from scratch before diving into the full vLLM/SGLang codebases.
Nano-vLLM is used as a minimal inference framework to teach the core concepts of building an LLM inference engine from scratch before diving into the full vLLM/SGLang codebases.What inference topics does it cover beyond basic serving?It covers chunked prefill, flash decoding, speculative decoding, KV cache strategies (including prefix cache and linear attention), quantization, PD disaffinity/disaggregation, and MoE expert-parallel load balancing (EPLB).
It covers chunked prefill, flash decoding, speculative decoding, KV cache strategies (including prefix cache and linear attention), quantization, PD disaffinity/disaggregation, and MoE expert-parallel load balancing (EPLB).Can I run these notebooks without a GPU?Not stated explicitly, but many notebooks cover CUDA-specific topics like CUDA graphs, GPU/NPU profiling and vLLM memory snapshots, so a GPU is effectively needed to reproduce the practical examples.
Not stated explicitly, but many notebooks cover CUDA-specific topics like CUDA graphs, GPU/NPU profiling and vLLM memory snapshots, so a GPU is effectively needed to reproduce the practical examples.What are similar projects?halo, ms-swift, verl, lectures, autumn-recruitment, ktransformers, DMI, self-llm
halo, ms-swift, verl, lectures, autumn-recruitment, ktransformers, DMI, self-llmWhich topics is it in?ai-infrastructure, performance-optimization, sglang, distributed-training, attention-mechanism, moe, lora, llm-inference, vllm, reinforcement-learning, jupyter notebook, quantization
ai-infrastructure, performance-optimization, sglang, distributed-training, attention-mechanism, moe, lora, llm-inference, vllm, reinforcement-learning, jupyter notebook, quantization