Global

8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. 32 GB.

A high-throughput and memory-efficient inference and serving engine for LLMs

3-Flash (321B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python.

Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.

We are releasing MiniCPM5-2B, the second model in the MiniCPM5 series, following MiniCPM5-1B. It is a dense 2B Transformer that scales up the same training recipe, built for on-device, local deployment, and resource-constrained scenarios, reaching 2B-class open-source SOTA.

sh as its gate, so latest is always the current stack. /models), and serves on port 18020. Pick a mode — one GPU serves one at a time:

8-27B bf16 (130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

📍 Use GLM-5.3 & GLM-5.3-Flash API services on Z.ai API Platform.

TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc.

Official SGLang x Datawhale course on LLM inference: understand inference, build a mini-sglang from scratch, then read the real SGLang source and land your first PR. Available in English and Chinese.

The Open Source Strix Halo AI engine. 56tg tok/s single user with DFlash2

The fastest way to run Qwen3.8-Flash-Next on Strix Halo (gfx1151)

Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.

Material for gpu-mode lectures

8-Flash-Next — a 176B-parameter model (125B main + 51B n-gram, 6B active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill speed, with MTP speculative decoding, working prefix caching, deterministic greedy decoding, and up to 500k tokens of context.

> AI Engineering Interview Questions Company Wise - Your Cheat Sheet For AI Engineering Interviews at Top AI Companies > > Real interview questions asked in AI Engineering interviews at 35 companies, organized company by company, with answers linked wherever we have them.

0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable tool calling for coding agents. Up to 4× faster than Apple's MLX (mlx-lm) on the same weights.

Run a 105 GB AI model on a Mac that can't hold it. 8-Flash-Next (125B mixture of experts) from your SSD and caches the busiest experts in memory, so it runs on Macs with 16 to 64 GB. One native Swift binary on MLX and Metal, no Python, offline.

8 family on a single DGX Spark, with seven switchable targets and zero quality loss on each (NVFP4 is the quantization floor, Qwen's own FP8 is available above it; every speculative path is lossless by construction).

DFlash is a lightweight block diffusion model designed for speculative decoding. It enables efficient and high-quality parallel drafting.

Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable

This is EXL3 weights + fp8 KV on GB10. Do not pass --moe-backend marlin. The Hub card on brandonmusic (TP2/EP2/DCP2 + calibrated NVFP4 MLA KV) is the SM120 B12X image (verdictai/glm53-flash-exl3-k4:…-v84-dflash2), not this overlay.

Peer-to-peer LLM inference in the browser: pool your devices to run big open models, for chat and coding agents.

\ No Kev-9B JevBench result is published. These are Kev-8B (Qwen3).

3-Flash-NVFP4 and the abliterated variants) emit intermittent corrupted token IDs (vLLM #54150). Nearly invisible in English, but when a corrupted token lands inside a tool-call block the parser desyncs and generation can spiral into a repetition lock.

Easy-to-use and powerful LLM and SLM library with awesome model zoo.