Global

A high-throughput and memory-efficient inference and serving engine for LLMs

预览地址为 ，简体中文位于根目录，英文与繁體中文分别位于 en/ 和 zh-tw/。生成文件位于 build/，详细说明见网站构建与发布。

SGLang is a high-performance serving framework for large language models and multimodal models.

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.

8-27B bf16 (130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.

⭐ If LMCache helps you serve LLMs faster and cheaper, give us a star — it helps more teams discover the project.

3-Flash from two NVIDIA DGX Sparks (GB10, 128 GB each, linked by their ConnectX-7 ports) through an OpenAI-compatible API, with 4 concurrent requests, the model's full 1,048,576-token context and image and video input.

sh as its gate, so latest is always the current stack. /models), and serves on port 18020. Pick a mode — one GPU serves one at a time:

Material for gpu-mode lectures

The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)

Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

8-Flash-Next — a 176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill speed, with MTP speculative decoding, working prefix caching, deterministic greedy decoding, and up to 500k tokens of context.

The AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.

ROCm™ Hyperloom is a multi-agent harness that autonomously optimizes inference on AMD Instinct™ GPUs.

8-27B's published mixed NVFP4/FP8 weight representation unchanged on four Tesla V100-SXM2-16GB. m8n8k4. No FP8→FP4 requantization.

8-Flash-Next-NVFP4 checkpoint (99 GiB) from a single DGX Spark's 121 GiB unified memory, via vLLM with the PLE table offloaded and memory-mapped. This is a vision-language model: text, images and video all work out of the box (see below).

00bpwH3V3). 8-27B-exl3 by revision. Which one you get is decided by your VRAM, at setup, and you can change it any time.

0 bpw EXL3) via the sparkinfer (formerly b12x) kernel stack — a complete, self-contained Docker recipe tuned for speed and KV-cache headroom on a single device.

Both demos support terminal interaction and an OpenAI-compatible HTTP server.

Jina-serve is a framework for building and deploying AI services that communicate via gRPC, HTTP and WebSockets. Scale your services from local development to production while focusing on your core logic.

Run any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.

\ No Kev-9B JevBench result is published. These are Kev-8B (Qwen3).

All numbers in KV cache budget and Default runtime below were read from the running server (docker logs vllm-fn, docker inspect vllm-fn) — they are measurements, not estimates. e. sample ship today.

Inference Autopilot (inferopt) is an optimization CLI for single-host and cooperative multi-host SGLang deployments. Give it a model, GPU resources, a representative workload, optional latency SLOs, and an experiment budget.