Global

8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. 8-Flash-Next (125B) on 6GB, and DeepSeek-V3 (671B) on 12GB. 8-Flash-Next (125B) under 6GB.

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev stats / lib llama API / llama-server REST API

8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. cpp and MLX.

FreeVideo is a local inference engine for MiniMax H3 on consumer GPUs, built on OpenVDN's 8-step VDN-H3 model with Video DeltaNet's hybrid attention.

SGLang is a high-performance serving framework for large language models and multimodal models.

To download the model weights manually, use the following command:

1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.

An open-source AI accelerator, developed by AI: RTL, ISA, simulator, compiler and profiler in one repo. 5 on a Kintex-7 PCIe card.

24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

A high-performance inference engine for AI models. It allows you to deploy AI directly in your app with zero latency, full data privacy, and no inference costs. Key features:

\janas-chat. 8 times as fast with the dense model (Qwen3-4B 346 tok/s against 193), whose whole blocks go to the GPU, through Intel's own Windows driver there.

TokenSpeed is a speed-of-light LLM inference engine designed for agentic workloads, with TensorRT-LLM-level performance and vLLM-level usability. The goal is to be the most performant inference engine for production agentic workloads.

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

The Open Source Strix Halo AI engine. 56tg tok/s single user with DFlash2

MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering high-performance on-device LLMs and Edge AI.

The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

OpenVINO™ is an open source toolkit for optimizing and deploying AI inference

8-Flash-Next — a 176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill speed, with MTP speculative decoding, working prefix caching, deterministic greedy decoding, and up to 500k tokens of context.

An educational project made to understand how LLM inference works. The full inference path is implemented in one file without external libraries.

3-Flash beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

MiniMax Music 3 is a high-performance music generation model for creating complete songs up to five minutes long.

AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术

NobodyWho is an inference engine that lets you run LLMs locally and efficiently on any device.