What is this?
Nano-vLLM is a lightweight, from-scratch reimplementation of the vLLM inference engine for running large language models, built in roughly 1,200 lines of readable Python.
Nano-vLLM is a lightweight, from-scratch reimplementation of the vLLM inference engine for running large language models, built in roughly 1,200 lines of readable Python.Is this a standalone server or a library I call from my own code?It is a Python library: you create an LLM object and call generate() in your own script, mirroring vLLM's interface almost exactly.
It is a Python library: you create an LLM object and call generate() in your own script, mirroring vLLM's interface almost exactly.How do I get it running?Install the package following the README's Installation section, download the model weights manually with the command given under Model Download, then run example.py to see a working generate() call.
Install the package following the README's Installation section, download the model weights manually with the command given under Model Download, then run example.py to see a working generate() call.Do I need a CUDA GPU to run this?Yes - it relies on Tensor Parallelism and CUDA graph capture for speed, and its own benchmark was run on an NVIDIA RTX 4070 Laptop GPU.
Yes - it relies on Tensor Parallelism and CUDA graph capture for speed, and its own benchmark was run on an NVIDIA RTX 4070 Laptop GPU.How does it actually compare to full vLLM?In the project's own benchmark with a Qwen3-0.6B model and 256 requests, Nano-vLLM reached 1434 tokens/s versus vLLM's 1362 tokens/s, so it matches or slightly beats vLLM despite being a tiny fraction of the code.
In the project's own benchmark with a Qwen3-0.6B model and 256 requests, Nano-vLLM reached 1434 tokens/s versus vLLM's 1362 tokens/s, so it matches or slightly beats vLLM despite being a tiny fraction of the code.What optimizations does Nano-vLLM include out of the box?It implements prefix caching, tensor parallelism, torch compilation, and CUDA graph capture to speed up inference.
It implements prefix caching, tensor parallelism, torch compilation, and CUDA graph capture to speed up inference.Why would someone pick Nano-vLLM over regular vLLM?It delivers comparable or slightly better throughput while being readable and hackable at only about 1,200 lines of Python, versus vLLM's much larger codebase.
It delivers comparable or slightly better throughput while being readable and hackable at only about 1,200 lines of Python, versus vLLM's much larger codebase.What model and setup was used for its benchmark?The README benchmarks it with a Qwen3-0.6B model on an RTX 4070 Laptop GPU (8GB), running 256 requests with input and output lengths randomly sampled between 100 and 1024 tokens.
The README benchmarks it with a Qwen3-0.6B model on an RTX 4070 Laptop GPU (8GB), running 256 requests with input and output lengths randomly sampled between 100 and 1024 tokens.What are similar projects?jeeves, DMI, StartLux-Decision, Model-Optimizer, airllm, DeepGEMM, TypeLLM, decider
jeeves, DMI, StartLux-Decision, Model-Optimizer, airllm, DeepGEMM, TypeLLM, deciderWhich topics is it in?inference-engine, llm-inference, vllm, large-language-models, transformer, nlp, artificial-intelligence, offline, inference, nvidia, cuda, pytorch
inference-engine, llm-inference, vllm, large-language-models, transformer, nlp, artificial-intelligence, offline, inference, nvidia, cuda, pytorch