capy-flash README

README проекта с github.com/ValerioDolci/capy-flash. Одним файлом markdown: README.md.

capy-flash

Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on one or two 16 GB RTX 50-series GPUs, built for agents.

capy-flash is an inference server. It runs two models on consumer GPUs with 16 GB of VRAM and exposes an OpenAI- and Anthropic-compatible HTTP API, so agent tools such as Claude Code, Codex CLI and OpenCode can use it like a hosted model.

capy-flash is ninfer-tp2, renamed at v0.5.0: the two-GPU tensor-parallel fork of Neroued/ninfer (v0.1-v0.4.9 under the old name); NOTICE lists every piece of prior work it builds on. Status: v0.10.0. Built, tested and measured on one machine (below); users report other setups. Not affiliated with Qwen or NVIDIA.

Pick a model

16 GB of VRAM per GPU is the minimum. RAM matters for Flash-Next only.

what you want to run GPUs (RTX 50 series, sm_120) RAM install.sh
Qwen3.8-27B, 3-bit one with 16 GB of VRAM (the minimum) nothing special --model 27b-16gb
Qwen3.8-27B, NVFP4 two with 16 GB nothing special (8 GiB of context cache) --model 27b
Flash-Next, 64 GB edition one or two with 16 GB a 64 GB machine (~56 GiB free) --model flash-next-64gb
Flash-Next, NVIDIA NVFP4 (default) two with 16 GB a 96 GB machine (~75 GiB free) (default)

What you need

Quick start

Default model (Flash-Next NVFP4, two 16 GB cards); for another row of the table add its --model:

curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash -s -- --check   # can this machine run it?
curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash               # release + model into ~/capy-flash
~/capy-flash/current/serve-flash-next.sh ~/capy-flash/models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"capy-flash","messages":[{"role":"user","content":"What is 17*23?"}]}'                # from another shell

Connect a client

The server listens on http://127.0.0.1:8080 (NINFER_HOST / NINFER_PORT change it): OpenAI Chat Completions and Responses under /v1, Anthropic Messages at /v1/messages. --api-key KEY makes it require Authorization: Bearer KEY or x-api-key: KEY; --api-key-file FILE or the NINFER_API_KEY environment variable set the key without putting it on the command line.

Details, including authentication and what each client sends: docs/serving.md.

Structured output. response_format (json_object, json_schema) on Chat Completions, text.format on Responses, output_config.format on Messages, and the structured_outputs extension (grammar, regex, choice) constrain the answer: the reply is valid JSON for the schema (or matches the grammar, regex or choice) whenever the model ends it. With thinking on, only the answer after </think> is constrained.

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "capy-flash", "messages": [{"role": "user", "content": "A cat, as JSON."}],
  "response_format": {"type": "json_schema", "json_schema": {"name": "cat", "schema": {"type": "object",
    "properties": {"name": {"type": "string"}, "age": {"type": "integer"}}, "required": ["name", "age"]}}}}'

Schemas and grammars come from the client, so they are bounded: size, nesting and item limits before compilation, a compilation timeout, and a time and memory budget while generating (--constraint-* flags). A request over a limit fails alone with a 400 that names the flag; the other requests keep their speed. Fields, errors, limits and cost: docs/serving.md.

Run it other ways

Build

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a \
  -DNINFER_BUILD_APPS=ON -DNINFER_BUILD_BENCHMARKS=OFF -DBUILD_TESTING=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc
ninja -C build apps/ninfer-serve
build/apps/ninfer-serve <dir>/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --model-id capy-flash \
  --chat-template tools/chat_templates/qwen3_8.jinja

Or take the prebuilt tarball of a release (Ubuntu, sm_120a), which has the three launchers (tools/release).

Profiles

--profile NAME applies the tested configuration of a model; flags given on the command line override it, in any order. ninfer-serve --help and docs/serving.md list the full expansions.

profile for in short
flash-next Flash-Next on two GPUs (the launcher's default) --tp 2, INT8 KV, elastic KV / expert slots, 262,144 tokens, MTP 3, 4 GiB of idle KV and 8 conversation states in RAM, embedding in RAM
flash-next-1gpu the 64 GB edition on one GPU (NINFER_DEVICES=0) the same on one device, 1 GiB of idle KV and 8 states in RAM
27b Qwen3.8-27B NVFP4 on two GPUs --tp 2, INT8 KV, 196,608 tokens, MTP 3, Vision, 8 GiB of context cache in RAM
27b-16gb Qwen3.8-27B 3-bit on one GPU INT8 KV, 65,536 tokens, MTP 3, Vision, embedding in RAM (13.9 GB of VRAM)

For the 64 GB edition on two GPUs add --host-kv-mib 1024 --host-state-slots 2 to --profile flash-next (smaller RAM caches); install.sh --model flash-next-64gb prints that command.

For long contexts on Flash-Next, add --kv-ram-active to either Flash-Next profile: the KV cache stays in pinned RAM, taken as the conversations grow (about 2.5 GiB per GPU for 262k tokens, 4.2 GiB on one GPU), and the VRAM it frees goes to expert slots. Same tokens; within a few per cent of the default speed (costs). For several long conversations at once add --max-concurrency N --kv-capacity T (T tokens for all of them together; the RAM follows what they use). Prompts are prefilled one at a time: a request waits at most --pending-timeout-ms (default 10 minutes) for the others.

Docker

docker/Dockerfile.release packs a release tarball (builds in about a minute; the root Dockerfile compiles from source instead). It needs the NVIDIA Container Toolkit:

docker build -f docker/Dockerfile.release -t capy-flash .
docker run --gpus all -p 8080:8080 -v ~/capy-flash/models:/models capy-flash \
  /models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --host 0.0.0.0 \
  --model-id capy-flash --chat-template /opt/capy-flash/chat_templates/qwen3_8.jinja

Tested with rootless Podman and the two GPUs passed through (Flash-Next loads its 64.6 GiB of page-locked experts without extra limits).

Model weights

Ready to serve on Hugging Face (install.sh --model downloads one of these; each has its SHA256SUMS):

How the engine runs these models, every server flag, the RAM tiers, the chat template and the CPU threads: docs/serving.md. Environment variables and diagnostics: docs/capy-flash/diagnostics.md. All documentation: docs/README.md.

Benchmarks

Our machine: AMD Ryzen 9 9900X (12 cores), 96 GB DDR5-5600, 2x RTX 5070 Ti 16 GB on PCIe 5.0 x8 + x8 (no P2P), Ubuntu, CUDA 13.1, full clocks. Decode is the server-side rate. v0.8.0 runs at the speed of v0.7.7 (the release notes have the comparison). The agentic replay is bench/capy_flash: 3 synthetic coding-agent sessions, 53 turns, prompts growing to ~81k tokens with ~90 % prefix reuse, 256 output tokens per turn, temperature 0. LocalMaxxing is the reasoning-v1 benchmark (short prompt, 512 tokens).

Ours (one GPU = one RTX 5070 Ti):

model, edition GPUs version, date measured result
Flash-Next NVFP4 2 v0.6.0, 2026-10-18 agentic replay: decode median / total time / time to first token, median (p90) 155 tok/s / 177 s / 1.48 s (2.62 s)
Flash-Next NVFP4 2 v0.6.0, 2026-10-18 prefill, cold, ~3.5k / ~15k / ~55k-token prompts 2,422 / 4,615 / 4,828 tok/s
Flash-Next NVFP4 2 v0.6.0, 2026-10-18 LocalMaxxing reasoning-v1 178 tok/s
Flash-Next NVFP4 2 v0.7.0, 2026-10-22 agentic replay decode median 155 tok/s (as v0.6)
Flash-Next 64 GB 2 v0.7.0, 2026-10-22 agentic replay decode median 182 tok/s
Flash-Next 64 GB 2 v0.7.0, 2026-10-22 LocalMaxxing reasoning-v1, verified run 234 tok/s
Flash-Next 64 GB 2 v0.7.2, 2026-10-23 prefill 4k / 16k / 64k 2,168 / 3,924 / 3,994 tok/s
Flash-Next 64 GB 1 v0.7.0, 2026-10-22 LocalMaxxing reasoning-v1, verified run 121 tok/s
Flash-Next 64 GB 1 v0.7.1, 2026-10-22 agentic replay: decode median / total time / time to first token, median / prompt tokens from cache 104 tok/s / 276 s / 1.9 s / 83 %
Flash-Next 64 GB 1 v0.7.2, 2026-10-23 prefill 4k / 16k / 64k 1,671 / 2,838 / 2,921 tok/s
Qwen3.8-27B 3-bit 1 v0.7.0, 2026-10-22 decode, 8k-token real-text prompt, MTP 139 tok/s
Qwen3.8-27B 3-bit 1 v0.7.0 prefill 4k / 16k 3,401 / 3,211 tok/s
Qwen3.8-27B 3-bit 1 v0.7.0, 2026-10-22 LocalMaxxing reasoning-v1, verified run 174 tok/s
Qwen3.8-27B 3-bit 2 v0.7.0, 2026-10-22 decode, 8k-token real-text prompt, MTP 230 tok/s
Qwen3.8-27B 3-bit 2 v0.7.0 prefill 4k / 16k / 64k 5,787 / 5,567 / 4,304 tok/s
Qwen3.8-27B 3-bit 2 v0.7.0, 2026-10-22 LocalMaxxing reasoning-v1, verified run 288 tok/s
Qwen3.8-27B NVFP4 (QUASAR) 2 v0.6.0 prefill 4k / 16k / 64k, --profile 27b 9,343 / 9,234 / 6,287 tok/s

More on the 27B NVFP4 (--profile 27b, MTP; 280 prompts in three languages, v0.6): the same tokens as v0.5.4 on 280 of 280 answers (and with DFlash2), decode +5 % (Italian 168 → 176 tok/s), prefill at 4k / 16k / 64k 5,777 / 5,585 / 4,352 → 9,343 / 9,234 / 6,287 tok/s with MTP and 5,719 / 5,530 / 4,316 → 9,205 / 9,052 / 6,198 with DFlash2; concurrent decode (NInfer's decode-saturation suite, MTP3, 8,192 tokens per request) 175 / 326 / 560 / 927 tok/s aggregate at 1 / 2 / 4 / 8 requests. On an agentic replay of the 27B (3 interleaved sessions and 11 subagents, 4 lanes), the v0.5.4 context cache took the median time to first token from 2.14 to 0.98 s and the prompt tokens served from cache from 67 to 78 %, decode unchanged. Flash-Next decode stays flat with context: 168 / 165 / 171 / 165 tok/s with 0 / 14k / 61k / 118k tokens of prompt (v0.6), and a cold 382-token prompt answers in 1.35 s. A cold start of the Flash-Next server takes 26-29 s here (v0.5.3), most of it reading the 64.6 GiB of experts. The 64 GB edition peaks at 55.2 GiB of RSS inside a 64 GiB memory cgroup without swap (57 GiB used with one GPU); its speeds were measured on the 96 GB machine.

Reported by users (not reproduced by us):

setup model, edition version measured result source
AMD EPYC 7F52 16 cores (Zen 2, AVX2), 2x RTX 5080 (overclocked) Flash-Next NVFP4 v0.7.2 built with CUDA 13.4 agentic replay (53 turns, greedy): decode median 141.3 tok/s with automatic threads; 144.4 with --expert-cpu-threads 15 and taskset -c 0-15; release tarball (CUDA 13.1) 137.6; v0.7.1 tarball 121.8 issue #11
same same same own measurement, not greedy prefill at 131k 5,541 tok/s; time to first token 22.1 s at 131k, 1.15 s at 1k issue #11
AMD Threadripper PRO 5975WX (AVX2), 2x RTX PRO 4000 Blackwell 24 GB (145 W), Proxmox LXC Flash-Next 64 GB (GSQ-RCO) v0.7.1 built from source, CUDA 13.3 decode / prefill / local aider coding benchmark decode ~100-114 tok/s on prose, ~140-155 on reasoning; prefill at 40k ~3,300 tok/s; aider 145-146 of 148 issue #22

Against Strata. Strata 0.1.45 on our machine, same day (2026-10-18), two GPUs, capy-flash v0.6.0:

capy-flash, NVFP4 Strata, same NVFP4 weights Strata, 3-bit IQ3_S (tuned)
agentic replay, 53 turns up to 81k tokens: total time 177 s 268 s 197 s
agentic replay: decode, server side (median) 155 tok/s 98 tok/s 151 tok/s
agentic replay: time to first token, median (p90) 1.48 s (2.62) 2.35 s (4.17) 1.60 s (3.69)
prefill, ~3.5k-token prompt 2,422 tok/s 1,519 tok/s 2,216 tok/s
prefill, ~15k-token prompt 4,615 tok/s 3,513 tok/s 3,811 tok/s
prefill, ~55k-token prompt 4,828 tok/s 4,849 tok/s 5,233 tok/s
LocalMaxxing reasoning-v1 (short prompt, 512 tokens) 178 tok/s 119 tok/s 195 tok/s

Strata's IQ3_S experts are 26 % smaller than NVIDIA's NVFP4 ones (lower quality: KL 0.144 vs 0.070 for NVFP4, as Strata reports), which is why it leads on the short benchmark. v0.6 against the build before the v0.6 work (2026-10-17): replay total 212 → 177 s, decode 118 → 155 tok/s.

Comparing decode with other engines. capy-flash computes every routed expert the model selects; an expert that misses the VRAM cache is computed on the CPU, never skipped. Strata's "route tail skip" skips low-ranked experts that miss the cache, which changes the output slightly and saves CPU work. A decode-speed gap against Strata, especially on a slower CPU, most likely comes from that. Measured here with the same GSQ-RCO IQ3_S experts on both (one RTX 5070 Ti, LocalMaxxing decode, 2026-10-23): Strata 138 tok/s with the skip, 116.6 with STRATA_ROUTE_TAIL_SKIP=0, capy-flash 129.8. On two GPUs (capy-flash v0.7.0 against Strata 0.2.2, 2026-10-22) the agentic replay decode median was 200.9 against 171.3 tok/s; Strata finished that replay sooner overall (189 against 200 s) thanks to its faster prefill.

Quality against the GB300 reference (NVIDIA's NVFP4 checkpoint served by vLLM on a GB300, same prompts, paired):

model, version GSM8K (reference) MMLU-Pro (reference)
Flash-Next NVFP4, v0.6.0 0.9750 (0.9810), 1,319 questions; 9 / 1 discordant, McNemar p 0.02 0.8413 (0.8452), 1,008 questions: non-inferior (difference -0.40 pt, lower bound -1.85 pt, margin -2.5 pt)
Flash-Next NVFP4, v0.7.0 0.985 (0.985), 330 questions 0.829 (0.849), 252 questions; paired with v0.6: no difference
Flash-Next 64 GB, v0.7.0 0.982 (0.985), 330 questions 0.837 (0.849), 252 questions
Qwen3.8-27B 3-bit, v0.7.0 0.968 (0.968), 1,319 questions 0.69-0.71, about 0.72 with the empty-reply guards (0.78), 252 questions

The v0.6.0 release binary was checked on the first 330 / 252 questions (GSM8K 0.979, MMLU-Pro 0.841). Output is not bit-reproducible run to run (which experts sit in VRAM depends on timing), so two runs of the same build differ by a few questions. The 3-bit 27B loses MMLU-Pro points mostly because more answers run out of their 8,192 tokens while reasoning; the v0.7 empty-reply guards bring back the ones that stopped inside their reasoning (9 of 9, all correct: about +3.6 points, estimated on those questions). Agent tasks (2026-10-14 build): 5 multi-step coding tasks of a private task bank, 4 runs each, OpenCode with the model's sampling: 20/20 solved, 42.9k generated tokens per run (Strata 0.1.43 IQ3_S on the same tasks: 52.7k, 20/20; the GB300 reference: 26.5k), no malformed tool call in 1,124.

Known limits

Tests

ctest --test-dir build runs the unit tests. The Engine integration tests run when their artifacts are set: NINFER_TEST_QWEN4_ARTIFACT (a small synthetic Flash-Next artifact with the real layer geometry, made by tools/capy_flash/make_test_artifact.sh <checkpoint dir> <out dir>, CPU only) and NINFER_TEST_ARTIFACT (a Qwen3.8-27B artifact, for the tests inherited from ninfer-tp2). ctest -j4 is supported; tests that need a model too large for one 16 GB board skip with the sizes. For the 27B, tools/tp2/gate.sh is the reference run: it sets the artifacts, runs the integration tests (GATE_CTEST_EXCLUDE excludes more), then checks golden ids, perplexity and greedy texts against a recorded reference (tools/tp2/README.md).

What's new

v0.10.0 (release notes)

v0.9.0 (release notes)

v0.8.2 (release notes)

v0.8.1 (release notes)

v0.8.0 (release notes)

v0.7.7

v0.7.6

v0.7.5

v0.7.4

v0.7.3

v0.7.2: pick up where you left off.

v0.7.1: --profile flash-next-1gpu keeps 8 conversation states in RAM (+0.66 GiB): on the agentic replay with one GPU 83 % of the prompt tokens come from the cache, 1.9 s to the first token (median) and 276 s for the whole replay (v0.7.0: 34 %, 13.3 s, 858 s).

v0.7.0: start small, grow later, same engine.

Older releases: GitHub releases and the development log, which records the work behind the Tnn tags in the sources.

License

Apache License 2.0 (LICENSE); attribution in NOTICE. The original ninfer-tp2 README is the one of v0.4.9 (historical).

capy-flash (ValerioDolci/capy-flash)

Ищем в базе…

Что это такое?capy-flash — это самостоятельно развёртываемый сервер инференса, который запускает Qwen3.8-Flash-Next (большую MoE-модель) и Qwen3.8-27B на одной или двух видеокартах RTX 50-й серии с 16 ГБ памяти каждая. Эксперты, которые не помещаются в VRAM, выгружаются в оперативную память (CPU RAM). Сервер предоставляет HTTP API, совместимый с OpenAI и Anthropic, поэтому агенты для программирования, такие как Claude Code, Codex CLI и OpenCode, могут использовать его как локальную модель.
Как это запустить?Запустите установщик install.sh. Он проверит видеокарты, драйвер, поддержку AVX2/AVX-512, оперативную память и место на диске, скачает последний релиз и репозиторий модели с Hugging Face, проверит оба и выведет точную команду для запуска сервера, а также тестовый запрос.
Какое железо нужно?Linux x86_64 (проверено на Ubuntu; пользователи сообщают о работе через WSL2/Docker на Windows) с драйвером NVIDIA, который поддерживает CUDA 13.1, одна или две видеокарты sm120 с 16 ГБ VRAM каждая (RTX 5060 Ti/5070 Ti/5080/5090), а для Flash-Next — процессор с AVX2+FMA+F16C или AVX-512 и свободные 64–96 ГБ оперативной памяти для размещения выгруженных экспертов.
Поддержка Windows уже готова к использованию?Официально нет: проект собирается и тестируется под Linux. Пользователи запускают 27B через WSL2 или Docker на Windows, а нативная сборка под Windows существует только как неофициальный порт от сообщества и пока не входит в релизы.
В чём практическая разница между Flash-Next и моделью 27B?Flash-Next — большая MoE-модель (до 126 ГБ), эксперты которой частично работают из оперативной памяти для лучшего качества, но для этого нужно 64–96 ГБ RAM. Qwen3.8-27B — меньшая плотная модель, которая полностью помещается в видеокартах и не требует оперативной памяти, в обмен на некоторое снижение качества и более простые требования к железу.
Какие агенты для программирования могут подключиться к capy-flash?Claude Code (через свою конечную точку Anthropic Messages), Codex CLI (через пользовательского провайдера OpenAI Responses) и OpenCode или любой другой клиент, совместимый с OpenAI.
Почему проект переименовали из ninfer-tp2?capy-flash — тот же проект, переименованный в версии v0.5.0. Это форк Neroued/ninfer с тензорным параллелизмом на две видеокарты, а в его файле NOTICE указаны все предыдущие работы, на которые он опирается.
Насколько он быстрый на практике?На двух RTX 5070 Ti автор измерил около 155–230 tok/s на декодировании и более 9 000 tok/s на префилле для модели 27B. Результат зависит от квантизации и редакции модели и был получен в бенчмарке агентной сессии программирования.
Какие есть похожие проекты?vllm, sglang, HyperQwen, alzeo, Strata, v100-skinny, project-maya, airllm
Автор
ValerioDolci
Размер
32.8 MB
Звёзды
157
Возраст
1 мес
Коммиты · 90 дней