Topics / #model-serving

Model-serving projects on GitHub

Serving trained models as low latency APIs.

As of , the most-starred Model-serving projects on GitHub are vllm (94K stars), exo (47.9K stars) and ray (44.1K stars). GitHub.Locker tracks 46 Model-serving repositories with 400.4K stars combined. The fastest-growing on October 16 was solo_jev, with +149 stars in a day.

46Projects
400.4KStars in total
+725Stars in a day
Browse in the catalog →github.com/topics/model-serving ↗

Hot right now

Most stars gained in a day · Oct 16

+149

solo_jev

0814wdwd/solo_jev · ★ 622

SOLO reorders the rows and fields of a dataset so an LLM decision backend with prefix caching can reuse computation across requests, running a natural-language decision over every row while returning results in the original row order.

+135

magnitude

magnitudedev/magnitude · ★ 7K

Magnitude is an open source inference server that profiles your machine, recommends local models that fit it, then downloads, tunes, and runs them for your coding agent.

+73

vllm

vllm-project/vllm · ★ 94K

vLLM is a library for fast LLM inference and serving, built around PagedAttention for efficient memory management of attention keys and values.

+58

project-maya

mw00/project-maya · ★ 954

Project Maya lets you run GLM-5.3-Flash, a 321-billion-parameter mixture-of-experts AI model, on your own NVIDIA GPU(s) by streaming the model's experts across VRAM, RAM and NVMe SSD as the conversation needs them.

The all-time top

46 projects · by stars

  1. #1 · vllm-project/vllm

    vllm

    vLLM is a library for fast LLM inference and serving, built around PagedAttention for efficient memory management of attention keys and values.

    94K
  2. #2 · exo-explore/exo

    exo

    exo turns a group of your own devices (Macs, Linux boxes) into a single AI cluster so you can run large AI models that would never fit on one machine, pooling their memory and compute together.

    47.9K
  3. #3 · ray-project/ray

    ray

    Ray is an open-source framework for scaling AI and Python applications: a core distributed runtime plus a set of AI libraries that simplify ML compute.

    44.1K
  1. 4

    Detectron

    26.4Kfacebookresearch/Detectron · Python

    Detectron is Facebook AI Research's software system implementing state-of-the-art object detection algorithms like Mask R-CNN and RetinaNet, built as a research platform for computer vision.

  2. 5

    omlx

    22.9Kjundot/omlx · Python

    oMLX is a local LLM inference server for Apple Silicon Macs. It serves text LLMs, vision models, OCR, embeddings and rerankers with continuous batching and a tiered KV cache, managed from the macOS menu bar.

  3. 6

    serve

    21.9Kjina-ai/serve · Python

    Jina is an open-source Python framework for building multimodal AI applications on a cloud-native stack — you wrap models into gRPC microservices, chain them into flows, and scale from local to cloud.

  4. 7

    FreeToken

    14.5KFlashML-org/FreeToken · Python

    FreeToken is an edge-native Mixture-of-Experts (MoE) serving engine built to run frontier-scale open-weight models, including 290B+ parameter models, on consumer hardware like gaming PCs.

  5. 8

    TensorRT

    13.4KNVIDIA/TensorRT · C++

    TensorRT is NVIDIA's SDK for high-performance deep learning inference on NVIDIA GPUs. This repository holds the open source components: plugins, the ONNX parser, and sample applications that demonstrate TensorRT usage.

  6. 9

    LMCache

    12.1KLMCache/LMCache · Python

    LMCache is a KV cache management layer for LLM inference that turns the KV cache from a temporary, per-process state into reusable, persistent 'AI-native knowledge' that can be shared across serving engines, reducing time-to-first-token and boosting throughput for long-context, multi-turn, and RAG workloads.

  7. 10

    server

    11.1Ktriton-inference-server/server · Python

    Triton Inference Server is an open source serving system for deploying trained AI models from frameworks like TensorRT, PyTorch, ONNX, OpenVINO, Python and RAPIDS FIL, running inference requests over HTTP/REST or gRPC.

  8. 11

    openvino

    11Kopenvinotoolkit/openvino · C++

    OpenVINO is Intel's open-source toolkit for optimizing and deploying AI inference — it takes trained deep learning models for vision, speech, NLP, and generative AI and runs them faster, from edge devices to cloud servers.

  9. 12

    text-generation-inference

    10.9Khuggingface/text-generation-inference · Python

    Text Generation Inference (TGI) is a toolkit for deploying and serving large language models — a Rust, Python and gRPC server that powers Hugging Chat, the Inference API and Inference Endpoints at Hugging Face.

  10. 13

    petals

    10.6Kbigscience-workshop/petals · Python

    Petals lets you run big open LLMs at home, BitTorrent-style: you join a public swarm that serves models like Llama 3.1 (405B), Mixtral (8x22B), Falcon (40B+) and BLOOM (176B) for inference and fine-tuning right from your desktop or Google Colab, up to 10x faster than offloading.

  11. 14

    kev

    9.3Kjaredpalmer/kev · Python

    Kev is an open family of small decision models (0.8B to 27B parameters) built on Qwen3.5/3.8 that answer yes/no, multiple-choice and rating questions about text with calibrated probabilities, acting as a self-hosted drop-in replacement for TypeSafe's hosted Jev model.

  12. 15

    vllm-omni

    7.2Kvllm-project/vllm-omni · Python

    vLLM-Omni is an inference and serving framework that extends vLLM beyond text-only LLMs to omni-modality models, handling text, image, audio, video, and action outputs.

  13. 16

    magnitude

    7Kmagnitudedev/magnitude · Rust

    Magnitude is an open source inference server that profiles your machine, recommends local models that fit it, then downloads, tunes, and runs them for your coding agent.

  14. 17

    turbo-fieldfare

    6.9Kdrumih/turbo-fieldfare · Swift

    TurboFieldfare is a custom Swift and Metal runtime that runs the instruction-tuned Gemma 4 26B-A4B model on Apple Silicon Macs using only about 2 GB of RAM, even on 8 GB machines, by streaming only the experts needed for each token from SSD instead of loading the full 14.3 GB model.

  15. 18

    dflash

    6.2Kz-lab/dflash · Python

    DFlash is a lightweight block diffusion model built for speculative decoding, drafting tokens in parallel blocks so a larger target LLM can verify them faster than generating one token at a time.

  16. 19

    CLM

    3.1KContrastive-LM/CLM · Python

    CLM (Contrastive Language Models) is a 'System One' model trained with a contrastive learning objective that connects states and actions, served behind a TypeSafe-compatible API as CLM-8B. It is meant to be plugged into agents as a fast scorer/verifier for decisions, tool calls, and best-of-N candidates.

  17. 20

    warp

    2.5Ksqliteai/warp · C

    WARP (Weight-Aware Runtime and Paging, formerly WASTE) is a dependency-free, embeddable C inference engine that runs huge mixture-of-experts models on consumer hardware by streaming each token's activated weights straight from NVMe — the full models, not distilled versions.

  18. 21

    uzu

    2.4Ktrymirai/uzu · Rust

    uzu is a high-performance inference engine for AI models that lets you run LLMs directly inside your app with zero latency, full data privacy, and no per-request inference costs.

  19. 22

    HyperQwen

    2Ksyv-ai/HyperQwen · Python

    HyperQwen is a self-hosted serving setup that runs the Qwen3.8-27B model on a single 24 GB consumer GPU (like an RTX 3090) using vLLM, exposing it through an OpenAI-compatible API.

  20. 23

    TensorFold

    1.4Kashhart/TensorFold · Zig

    TensorFold is an LLM inference engine that serves open-weight language models through an OpenAI-compatible API, running them on Apple Silicon via MLX or on NVIDIA GPUs via CUDA.

  21. 24

    decider

    1.2KMapika/decider · Python

    Decider is a family of language models fine-tuned from Qwen3.5 that don't generate text; instead they read a state plus a set of typed questions and return calibrated probability distributions for every question in one forward pass.

  22. 25

    MiniMax-Music3

    961MiniMax-AI/MiniMax-Music3

    MiniMax Music 3 is a music generation model that creates complete songs up to five minutes long from lyrics and a music description, producing 32 kHz 16-bit stereo WAV audio with coherent structure and expressive vocals.

  23. 26

    TypeLLM

    956TypeLLM/TypeLLM · Python

    TypeLLM adds type-safe, schema-guaranteed output generation to existing open autoregressive LLMs, without changing their weights or architecture, letting models still think freely while returning typed JSON Schema-constrained answers.

  24. 27

    project-maya

    954mw00/project-maya · C++

    Project Maya lets you run GLM-5.3-Flash, a 321-billion-parameter mixture-of-experts AI model, on your own NVIDIA GPU(s) by streaming the model's experts across VRAM, RAM and NVMe SSD as the conversation needs them.

  25. 28

    GLM-5.3-Flash-EXL3-2x-DGX-Sparks

    734MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks · Python

    A deployment recipe and Docker image that serves the GLM-5.3-Flash model as an OpenAI-compatible vLLM endpoint across 2-4 NVIDIA DGX Spark (GB10) kits, using a custom EXL3/TR3 4-bit quantization of the weights.

  26. 29

    solo_jev

    6220814wdwd/solo_jev · Python

    SOLO reorders the rows and fields of a dataset so an LLM decision backend with prefix caching can reuse computation across requests, running a natural-language decision over every row while returning results in the original row order.

  27. 30

    Qwen3.8-Flash-Next-Single-DGX-Spark

    609MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark · Python

    A self-contained recipe for serving the Qwen3.8-Flash-Next-NVFP4 checkpoint from a single NVIDIA DGX Spark machine's unified memory, using vLLM with the PLE table offloaded and memory-mapped.

  28. 31

    djev-run

    584taeold/djev-run · HTML

    djev-run serves DiffusionGemma-Jev (djev) behind a TypeSafe AI compatible API on Google Cloud Run with an NVIDIA RTX PRO 6000 Blackwell or L4 GPU, or runs it directly inside Google Colab Pro via colab.ipynb; it is built on mmastrac/djev.

  29. 32

    GLiClass

    563Knowledgator/GLiClass · Python

    GLiClass is an efficient zero-shot model for sequence classification, inspired by GLiNER: give it a text and a list of candidate labels, and it classifies the text with accuracy comparable to cross-encoders.

  30. 33

    Qwen3.8-27B-SGLang-DGX-Spark

    474MiaAI-Lab/Qwen3.8-27B-SGLang-DGX-Spark · Shell

    A set of opinionated, ready-to-run Docker scripts that serve the Qwen3.8-27B model through SGLang on an NVIDIA DGX Spark (GB10, aarch64), offering three swap-in speculative-decoding modes — EAGLE/MTP, DSpark, and DFlash2 — each tuned with on-device measurements rather than guesses.

  31. 34

    qwen3.8-Flash-DGX

    453blazux/qwen3.8-Flash-DGX · Python

    A recipe (scripts plus a patched Docker image) for serving the ~176B-parameter Qwen3.8-Flash-Next model with vLLM on a single NVIDIA DGX Spark (GB10). It solves the problem that the 126 GiB NVFP4 checkpoint doesn't fit next to a usable KV cache in the Spark's 128 GB unified memory, by streaming a 48 GiB lookup table from NVMe via mmap instead of keeping it resident.

  32. 35

    GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold

    444MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold · Python

    This project serves the GLM-5.3-Flash language model as an OpenAI-compatible API, with vision and video input, running across two linked NVIDIA DGX Spark machines (GB10, 128GB each) using TensorFold.

  33. 36

    awesome-minimax-h3-integration

    442MiniMax-AI/awesome-minimax-h3-integration

    A community-maintained index of checkpoints, tools, and workflows for MiniMax H3, a video generation model that produces clips with native stereo audio from text, image, video, and audio inputs.

  34. 37

    jeeves

    425PostHog/jeeves · Python

    Jeeves is a reasoning-enhanced 'Jev-like' decision model built on Qwen3.5-9B with LoRA. It thinks through a reasoning chain before giving calibrated yes/no, multiple-choice, or rating answers, and ships with a diffusion drafter for faster inference.

  35. 38

    Qwen3.8-Flash-Next-Dual-DGX-Sparks

    410MiaAI-Lab/Qwen3.8-Flash-Next-Dual-DGX-Sparks · HTML

    A multi-node inference launcher that runs the Qwen3.8-Flash-Next-NVFP4 language model across two NVIDIA DGX Spark nodes using vLLM with tensor parallelism, expert parallelism and speculative decoding.

  36. 39

    DeepSeek-v4-Flash-One-DGX-Spark

    346MiaAI-Lab/DeepSeek-v4-Flash-One-DGX-Spark · Python

    A single-node Docker launcher that serves the DeepSeek V4 Flash 0731 model (EXL3 3.0bpw quantization) with DSpark speculative decoding, tuned to run entirely on one NVIDIA DGX Spark machine.

  37. 40

    llm-inference-engineering

    344amitshekhariitbhu/llm-inference-engineering · Markdown

    A free, continuously growing blog series that teaches LLM inference engineering step by step, covering KV cache, PagedAttention, continuous batching, speculative decoding, and serving engines like vLLM, SGLang, TensorRT-LLM, and GGUF.

  38. 41

    GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark

    236tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark · Python

    A deployment recipe and set of patched Docker images for serving GLM-5.3-Flash (320B total / 18B active MoE) as an OpenAI-compatible API across two NVIDIA DGX Spark nodes, using vLLM tensor-parallel 2, NVFP4 quantization, and DFlash2 speculative decoding.

  39. 42

    Qwen3.8-27B-DFlash2-EXL3-5.0bpw

    232MiaAI-Lab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw · Python

    A deployment kit for serving Qwen3.8-27B (quantized to EXL3 3.5bpw) through an OpenAI-compatible server, using speculative decoding to speed up generation. It bundles the launcher, server, and config; the model weights live on Hugging Face and inference runs on the team's exllamav3 fork.

  40. 43

    dgx-spark-qwen38

    227hasso5703/dgx-spark-qwen38 · Python

    A one-command installer that sets up a hardened, boot-persistent SGLang serving stack for the Qwen3.8 model family on a single NVIDIA DGX Spark (GB10), reaching 50 tok/s greedy median and up to 1M context on the 27B NVFP4 config, with a 176B Flash-Next MoE model also fitting on one box.

  41. 44

    InstinctFlash

    200General-Instinct/InstinctFlash · Python

    InstinctFlash is a high-performance serving runtime for robotics foundation models - it loads checkpoints like GR00T, Cosmos3, pi0.5, LingBot-VA/VLA and DreamZero, applies acceleration (FP8, fused Triton kernels, caching) and serves them over Python or WebSocket, reporting up to 33.78x speedup on Jetson Thor.

  42. 45

    nari-qwen3-tts

    187nari-labs/nari-qwen3-tts · Python

    Nari Qwen3-TTS is a high-performance serving implementation of the Qwen3-TTS 1.7B CustomVoice model, built to run on a single NVIDIA H100 and exposed as an API with streaming and non-streaming speech generation.

  43. 46

    v100-skinny

    172dnv2003/v100-skinny · Python

    v100-skinny provides hand-written NVFP4 (W4A16) and FP8 (W8A16) CUDA kernels that let four Tesla V100-SXM2-16GB GPUs run Qwen3.8-27B's published mixed NVFP4/FP8 weights directly, serving the model through an OpenAI-compatible API.

Often tagged together