What is this?
vLLM-Omni is an inference and serving framework that extends vLLM beyond text-only LLMs to omni-modality models, handling text, image, audio, video, and action outputs.
vLLM-Omni is an inference and serving framework that extends vLLM beyond text-only LLMs to omni-modality models, handling text, image, audio, video, and action outputs.Is this a deployable framework or a hosted service?It is a self-hosted serving framework: you run it yourself and it exposes an OpenAI-compatible API server, rather than being a cloud service you sign up for.
It is a self-hosted serving framework: you run it yourself and it exposes an OpenAI-compatible API server, rather than being a cloud service you sign up for.How do I get it running?You follow the project's Installation and Quickstart docs; nightly CUDA Docker images built from the latest main branch are also provided for quick deployment.
You follow the project's Installation and Quickstart docs; nightly CUDA Docker images built from the latest main branch are also provided for quick deployment.What hardware do I need to run it?You need a GPU for model inference; the framework targets CUDA primarily but also has broad ROCm, XPU, and NPU support, and relies on PyTorch under the hood.
You need a GPU for model inference; the framework targets CUDA primarily but also has broad ROCm, XPU, and NPU support, and relies on PyTorch under the hood.How stable are releases?Releases are tightly coupled to upstream vLLM: starting from 0.14.0 it ships a stable version aligned to every even-numbered vLLM minor release, so you need to track vLLM's own versioning cadence.
Releases are tightly coupled to upstream vLLM: starting from 0.14.0 it ships a stable version aligned to every even-numbered vLLM minor release, so you need to track vLLM's own versioning cadence.What kinds of models can it serve?It supports omni-modality models like Qwen3-Omni, MiniCPM-o, Cosmos3, HunyuanImage, and BAGEL, plus TTS models (Qwen3-TTS, CosyVoice3), diffusion image/video models (LTX-2, Wan2.2), and robot-policy/action models such as π0 and GR00T-N1.7.
It supports omni-modality models like Qwen3-Omni, MiniCPM-o, Cosmos3, HunyuanImage, and BAGEL, plus TTS models (Qwen3-TTS, CosyVoice3), diffusion image/video models (LTX-2, Wan2.2), and robot-policy/action models such as π0 and GR00T-N1.7.Why not just use plain vLLM?Vanilla vLLM only handles autoregressive text generation; vLLM-Omni adds support for non-autoregressive Diffusion Transformers and heterogeneous multimodal pipelines (text, image, audio, video, action) on top of vLLM's KV-cache efficiency.
Vanilla vLLM only handles autoregressive text generation; vLLM-Omni adds support for non-autoregressive Diffusion Transformers and heterogeneous multimodal pipelines (text, image, audio, video, action) on top of vLLM's KV-cache efficiency.Can it handle real-time voice conversations?Yes, it implements full-duplex realtime serving with streaming audio input and output, used for example with the MiniCPM-o series.
Yes, it implements full-duplex realtime serving with streaming audio input and output, used for example with the MiniCPM-o series.What are similar projects?GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, Qwen3.8-Flash-Next-Dual-DGX-Sparks, ZDTaichu5.0-9B, vllm, dgx-spark-qwen38, text-generation-inference, GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark, Model-Optimizer
GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, Qwen3.8-Flash-Next-Dual-DGX-Sparks, ZDTaichu5.0-9B, vllm, dgx-spark-qwen38, text-generation-inference, GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark, Model-OptimizerWhich topics is it in?world-model, world-models, diffusion, model-serving, audio-generation, robotics, llm-inference, tts, vllm, quantization, multimodal, video-generation
world-model, world-models, diffusion, model-serving, audio-generation, robotics, llm-inference, tts, vllm, quantization, multimodal, video-generation