What is this?
LLM101n is a planned course by Andrej Karpathy that walks through building a 'Storyteller' AI language model end-to-end, from basics all the way to a ChatGPT-like web app, implemented from scratch in Python, C and CUDA.
LLM101n is a planned course by Andrej Karpathy that walks through building a 'Storyteller' AI language model end-to-end, from basics all the way to a ChatGPT-like web app, implemented from scratch in Python, C and CUDA.Is this a working course I can enroll in right now?No, right now it is only a syllabus. The repo is archived because the actual course content is still being produced.
No, right now it is only a syllabus. The repo is archived because the actual course content is still being produced.How is the curriculum structured?It is laid out as 17 sequential chapters starting from a bigram language model and micrograd, moving through attention, transformers, tokenization, optimization, speed/distributed training, inference tricks like KV-cache and quantization, finetuning (SFT, RLHF), deployment, and finally multimodal generation.
It is laid out as 17 sequential chapters starting from a bigram language model and micrograd, moving through attention, transformers, tokenization, optimization, speed/distributed training, inference tricks like KV-cache and quantization, finetuning (SFT, RLHF), deployment, and finally multimodal generation.What programming languages and tools will it use?The course is planned to be built from scratch using Python, C and CUDA, with an appendix also mentioning PyTorch and JAX as frameworks to be worked in.
The course is planned to be built from scratch using Python, C and CUDA, with an appendix also mentioning PyTorch and JAX as frameworks to be worked in.Is the course finished, or still being built?It's still being built by Eureka Labs and the repo has been archived in the meantime, so there is no runnable content here yet, just the syllabus.
It's still being built by Eureka Labs and the repo has been archived in the meantime, so there is no runnable content here yet, just the syllabus.Do I need a strong background in math or CS to follow along?No, the course is designed to require only minimal computer science prerequisites, building understanding from basics up to deep learning concepts.
No, the course is designed to require only minimal computer science prerequisites, building understanding from basics up to deep learning concepts.What is the final project supposed to look like?A functioning web app similar to ChatGPT that can generate, refine, and illustrate little stories, built entirely from scratch across the 17 chapters.
A functioning web app similar to ChatGPT that can generate, refine, and illustrate little stories, built entirely from scratch across the 17 chapters.Who is developing this course?Eureka Labs, the company behind Andrej Karpathy's teaching projects, is building out the full LLM101n curriculum.
Eureka Labs, the company behind Andrej Karpathy's teaching projects, is building out the full LLM101n curriculum.What advanced topics does the later part of the syllabus cover?The later chapters move into mixed precision and distributed training, KV-cache and quantization for inference, LoRA/PEFT finetuning, RLHF-style reinforcement learning, deployment as a web app, and multimodal generation with VQVAE and diffusion transformers.
The later chapters move into mixed precision and distributed training, KV-cache and quantization for inference, LoRA/PEFT finetuning, RLHF-style reinforcement learning, deployment as a web app, and multimodal generation with VQVAE and diffusion transformers.What are similar projects?nlp_course, train-llm-from-scratch, halo, llm_interview_note, ms-swift, jeeves, llama3-from-scratch, open-r1
nlp_course, train-llm-from-scratch, halo, llm_interview_note, ms-swift, jeeves, llama3-from-scratch, open-r1Which topics is it in?attention, tokenization, rlhf, jax, distributed-training, neural-networks, lora, curriculum, gpt, reinforcement-learning, course, quantization
attention, tokenization, rlhf, jax, distributed-training, neural-networks, lora, curriculum, gpt, reinforcement-learning, course, quantization