What is this?
This is the supplementary material repository for the GPU MODE lectures YouTube channel — a growing collection of notebooks, slides and code from each numbered lecture on GPU and CUDA programming.
This is the supplementary material repository for the GPU MODE lectures YouTube channel — a growing collection of notebooks, slides and code from each numbered lecture on GPU and CUDA programming.Is this a structured course or just a collection of individual talks?It is organized as numbered lectures, each tied to a YouTube recording and a speaker, with its own materials folder; the first few lectures recap chapters of the PMPP textbook, while later ones become free-standing specialized talks.
It is organized as numbered lectures, each tied to a YouTube recording and a speaker, with its own materials folder; the first few lectures recap chapters of the PMPP textbook, while later ones become free-standing specialized talks.How is the material organized, and where should I start as a beginner?Each lecture lives in its own lectureNNN folder linked from the README, so you open the folder for the topic you want; new learners should start with Lectures 1-5, which cover CUDA profiling, PyTorch kernel integration, and getting started with CUDA.
Each lecture lives in its own lectureNNN folder linked from the README, so you open the folder for the topic you want; new learners should start with Lectures 1-5, which cover CUDA profiling, PyTorch kernel integration, and getting started with CUDA.Do I need an NVIDIA GPU and CUDA toolkit to follow along?Yes — almost all lectures are about writing and profiling CUDA kernels in PyTorch and Triton, so a CUDA-capable NVIDIA GPU is expected for hands-on exercises; a few later lectures cover Apple Metal and ARM CPU kernels instead.
Yes — almost all lectures are about writing and profiling CUDA kernels in PyTorch and Triton, so a CUDA-capable NVIDIA GPU is expected for hands-on exercises; a few later lectures cover Apple Metal and ARM CPU kernels instead.Are all lectures fully documented with notebooks and slides?Coverage is uneven — several lecture numbers are skipped entirely in the README, and some entries list only a title, speaker and a slides link with no further description or code.
Coverage is uneven — several lecture numbers are skipped entirely in the README, and some entries list only a title, speaker and a slides link with no further description or code.Who is this aimed at — complete beginners or experienced GPU engineers?It spans both ends: early talks target Python programmers new to CUDA, while later lectures cover advanced systems topics like CUTLASS, NCCL collectives, Triton internals and LLM serving for experienced engineers.
It spans both ends: early talks target Python programmers new to CUDA, while later lectures cover advanced systems topics like CUTLASS, NCCL collectives, Triton internals and LLM serving for experienced engineers.What topics do the lectures cover overall?They range from CUDA fundamentals and PyTorch kernel integration to advanced subjects like Flash Attention, quantized training, Triton/Gluon internals, CUTLASS, NCCL collectives, and LLM serving systems such as VLLM and SGLang.
They range from CUDA fundamentals and PyTorch kernel integration to advanced subjects like Flash Attention, quantized training, Triton/Gluon internals, CUTLASS, NCCL collectives, and LLM serving systems such as VLLM and SGLang.Is each lecture tied to the PMPP book?Only the early lectures recap chapters of 'Programming Massively Parallel Processors: A Hands-on Approach'; later lectures move well beyond the book into specialized, speaker-contributed topics.
Only the early lectures recap chapters of 'Programming Massively Parallel Processors: A Hands-on Approach'; later lectures move well beyond the book into specialized, speaker-contributed topics.Who presents these lectures?Speakers are practitioners and researchers from projects like PyTorch, Unsloth, SGLang, Hugging Face and NVIDIA, including Jeremy Howard, Thomas Viehmann, Mark Saroufim and Daniel Han.
Speakers are practitioners and researchers from projects like PyTorch, Unsloth, SGLang, Hugging Face and NVIDIA, including Jeremy Howard, Thomas Viehmann, Mark Saroufim and Daniel Han.Is the lecture series still active?Yes, lecture numbers go well past 100 in the README (e.g. a CuTe layout algebra talk at Lecture 103 and a PyCuTe talk at Lecture 114), indicating the series is still being added to.
Yes, lecture numbers go well past 100 in the README (e.g. a CuTe layout algebra talk at Lecture 103 and a PyCuTe talk at Lecture 114), indicating the series is still being added to.What are similar projects?llm-inference-engineering, Model-Optimizer, DMI, LeetCUDA, InfraTech, AISystem, vdn-minimax-h3, TensorRT-LLM
llm-inference-engineering, Model-Optimizer, DMI, LeetCUDA, InfraTech, AISystem, vdn-minimax-h3, TensorRT-LLMWhich topics is it in?triton, flash-attention, sglang, gpu-programming, llm-serving, arm, speculative-decoding, attention-mechanism, metal, youtube, vllm, course
triton, flash-attention, sglang, gpu-programming, llm-serving, arm, speculative-decoding, attention-mechanism, metal, youtube, vllm, course