14.8K ·GitHub
What is this?
TensorRT LLM is NVIDIA's Python framework for defining large language models (and now visual generation models) and compiling them into highly optimized inference engines that run on NVIDIA GPUs, with specialized kernels and Python/C++ runtimes.
TensorRT LLM is NVIDIA's Python framework for defining large language models (and now visual generation models) and compiling them into highly optimized inference engines that run on NVIDIA GPUs, with specialized kernels and Python/C++ runtimes.Is this a standalone app or a library I build on top of?It's a framework/library: you define your model with its Python API and it provides the runtimes (Python and C++) that execute the compiled inference engine, rather than a ready-made end-user application.
It's a framework/library: you define your model with its Python API and it provides the runtimes (Python and C++) that execute the compiled inference engine, rather than a ready-made end-user application.Do I need an NVIDIA GPU and CUDA to use this?Yes — TensorRT LLM is built specifically around NVIDIA GPU architectures (e.g. Hopper, Blackwell) and CUDA-based optimizations like CUDA graphs and NVLink communication.
Yes — TensorRT LLM is built specifically around NVIDIA GPU architectures (e.g. Hopper, Blackwell) and CUDA-based optimizations like CUDA graphs and NVLink communication.Do all the performance gains apply to any NVIDIA GPU, or only the newest ones?Many of the headline optimizations (sparse attention, DeepSeek-V4 tuning, 1,000+ TPS/user) are specific to the newest Blackwell GPUs like the B200; older GPU generations are supported but won't see the same numbers.
Many of the headline optimizations (sparse attention, DeepSeek-V4 tuning, 1,000+ TPS/user) are specific to the newest Blackwell GPUs like the B200; older GPU generations are supported but won't see the same numbers.Does it support image or video generation, not just text LLMs?Yes, since April 2026 TensorRT LLM added support for diffusion models used in visual generation (images/video), alongside its original LLM focus.
Yes, since April 2026 TensorRT LLM added support for diffusion models used in visual generation (images/video), alongside its original LLM focus.How much faster is inference with this compared to standard setups?NVIDIA reports major speedups, such as Llama 4 running at over 40,000 tokens per second on B200 GPUs and up to 3x throughput gains from speculative decoding.
NVIDIA reports major speedups, such as Llama 4 running at over 40,000 tokens per second on B200 GPUs and up to 3x throughput gains from speculative decoding.Is TensorRT LLM actually open source now?Yes, as of March 2026 development moved fully to this public GitHub repo, making it fully open-source.
Yes, as of March 2026 development moved fully to this public GitHub repo, making it fully open-source.Does it support Mixture-of-Experts and speculative decoding?Yes, it includes MoE-specific communication optimizations (like one-sided AllToAll over NVLink) and multiple speculative decoding techniques including N-Gram-based decoding.
Yes, it includes MoE-specific communication optimizations (like one-sided AllToAll over NVLink) and multiple speculative decoding techniques including N-Gram-based decoding.Which topics is it in?blackwell, llm-serving, speculative-decoding, moe, llama, diffusion-models, high-performance, gpu-acceleration, quantization, large-language-models, transformer, artificial-intelligence
blackwell, llm-serving, speculative-decoding, moe, llama, diffusion-models, high-performance, gpu-acceleration, quantization, large-language-models, transformer, artificial-intelligence