Use it for structured workloads with repeated policies, account context, catalog attributes, or correlated fields. 5-9B, through vLLM. It returns binary, categorical or scored decisions with probabilities.
Global
A high-throughput and memory-efficient inference and serving engine for LLMs
Open source inference engine that tunes its kernels on your hardware to find the fastest way to run each model. 2x faster than Ollama. Works on Apple Silicon, NVIDIA, AMD, or just a CPU.
#623 · +47 · 1d ·GitHub
3-Flash (321B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.
Jev-like family of decision models built on top of Qwen3.5/3.8 you can train and run on your own
FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
exo connects all your devices into an AI cluster. Not only does exo enable running models larger than would fit on a single device, but with day-0 support for RDMA over Thunderbolt, makes models run faster as you add more devices.
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar
A high-performance inference engine for AI models. It allows you to deploy AI directly in your app with zero latency, full data privacy, and no inference costs. Key features:
#1462 · +17 · 1d ·GitHub
Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.
A framework for efficient model inference with omni-modality models
Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
⭐ If LMCache helps you serve LLMs faster and cheaper, give us a star — it helps more teams discover the project.
Questions may be Noul / Choice / Score objects or plain wire-format dicts, so a request written for TypeSafe replays as client.systemone(state, questions).
#2544 · +8 · 1d ·GitHub
3-Flash from two NVIDIA DGX Sparks (GB10, 128 GB each, linked by their ConnectX-7 ports) through an OpenAI-compatible API, with 4 concurrent requests, the model's full 1,048,576-token context and image and video input.
3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.
sh as its gate, so latest is always the current stack. /models), and serves on port 18020. Pick a mode — one GPU serves one at a time:
#3313 · +5 · 1d ·GitHub
Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.
#4135 · +3 · 1d ·GitHub
3 GB model into memory. 35 GB core and FP16 KV cache in memory, then streams only the experts needed for each token from SSD. This is what lets the model run on Macs with 8 GB of RAM.
DFlash is a lightweight block diffusion model designed for speculative decoding. It enables efficient and high-quality parallel drafting.
TypeLLM brings type-safe generation to existing autoregressive LLMs without changing their architecture or weights. Inspired by TypeSafe AI's Jev, it lets models retain their native thinking and free-form generation while producing schema-guaranteed outputs through JSON Schema.
#4234 · +3 · 1d ·GitHub
8-Flash-Next — a 176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill speed, with MTP speculative decoding, working prefix caching, deterministic greedy decoding, and up to 500k tokens of context.
#4238 · +3 · 1d ·GitHub
This navigation guide is not a complete compatibility list.
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.
The Triton Inference Server provides an optimized cloud and edge inferencing solution.
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
#4891 · +2 · 1d ·GitHub
MiniMax Music 3 is a high-performance music generation model for creating complete songs up to five minutes long.
#4903 · +2 · 1d ·GitHub
This is EXL3 weights + fp8 KV on GB10. Do not pass --moe-backend marlin. The Hub card on brandonmusic (TP2/EP2/DCP2 + calibrated NVFP4 MLA KV) is the SM120 B12X image (verdictai/glm53-flash-exl3-k4:…-v84-dflash2), not this overlay.
GLiClass is an efficient, zero-shot sequence classification model inspired by the GLiNER framework.
#5003 · +2 · 1d ·GitHub
8-27B's published mixed NVFP4/FP8 weight representation unchanged on four Tesla V100-SXM2-16GB. m8n8k4. No FP8→FP4 requantization.