capy-flash on GitHub (ValerioDolci/capy-flash)

capy-flash README

The project's README from github.com/ValerioDolci/capy-flash. As one markdown file: README.md.

Qwen3.8-Flash-Next (125B MoE) and Qwen3.8-27B on one or two 16 GB RTX 50-series GPUs, built for agents.

capy-flash is an inference server. It runs two models on consumer GPUs with 16 GB of VRAM and exposes an OpenAI- and Anthropic-compatible HTTP API, so agent tools such as Claude Code, Codex CLI and OpenCode can use it like a hosted model.

Qwen3.8-Flash-Next is a mixture-of-experts model too large for the GPUs. capy-flash keeps the experts that do not fit in VRAM in host RAM and computes them on the CPU; everything else runs on the GPU (split over two GPUs when there are two). That is why Flash-Next needs a lot of RAM and a CPU with AVX2 or AVX-512.
Qwen3.8-27B is a dense model that runs entirely on the GPUs.
For agents: a prefix cache across turns, idle conversations moved to RAM (and optionally to disk) instead of being recomputed, and the expert cache follows the conversation.

capy-flash is ninfer-tp2, renamed at v0.5.0: the two-GPU tensor-parallel fork of Neroued/ninfer (v0.1-v0.4.9 under the old name); NOTICE lists every piece of prior work it builds on. Status: v0.7.7. Built, tested and measured on one machine (below); users report other setups. Not affiliated with Qwen or NVIDIA.

Pick a model

16 GB of VRAM per GPU is the minimum. RAM matters for Flash-Next only.

what you want to run	GPUs (RTX 50 series, sm_120)	RAM	`install.sh`
Qwen3.8-27B, 3-bit	one with 16 GB of VRAM (the minimum)	nothing special	`--model 27b-16gb`
Qwen3.8-27B, NVFP4	two with 16 GB	nothing special (8 GiB of context cache)	`--model 27b`
Flash-Next, 64 GB edition	one or two with 16 GB	a 64 GB machine (~56 GiB free)	`--model flash-next-64gb`
Flash-Next, NVIDIA NVFP4 (default)	two with 16 GB	a 96 GB machine (~75 GiB free)	(default)

One GPU, little RAM: the 3-bit 27B. It costs quality: GSM8K on par with the NVFP4 27B, MMLU-Pro about 0.72 against 0.78 (benchmarks).
One GPU and 64 GB of RAM: the 64 GB edition of Flash-Next (ISTA-DASLab's GSQ-RCO experts, 48 GiB in RAM instead of 65; quality on par with NVFP4 on GSM8K and MMLU-Pro).
Two GPUs: the 64 GB edition is the fastest Flash-Next here; the NVFP4 edition uses NVIDIA's experts as published. For the 27B on two cards, the NVFP4 27B is the better choice than the 3-bit one.

What you need

Linux x86_64 (tested on Ubuntu). Users run the 27B on WSL2 and on Windows with Docker; a native Windows build exists as a community port (OmarxGx/capy-flash), not in the releases yet.
An NVIDIA driver with CUDA 13.1 support (R580 or newer). The release needs no CUDA toolkit; building does (CUDA 13.1).
sm_120 GPUs with 16 GB: RTX 5060 Ti 16 GB, 5070 Ti, 5080 and 5090 are reported to work; a mixed pair runs at the pace of the slower card. Peer-to-peer is not needed.
Flash-Next only: a CPU with AVX-512 (F/BW/VL, VNNI) or AVX2 + FMA + F16C (Zen 2/3, Intel client cores), and the free RAM of the table: the routed experts are one page-locked host allocation (64.6 GiB for NVFP4, 48 GiB for the 64 GB edition), and --profile flash-next also keeps the token embedding and up to 4 GiB of idle conversations in RAM. The server checks this at startup and refuses with the numbers when it does not fit (NINFER_RAM_GUARD=0 starts anyway). On Windows/WSL2, which limit page-locked memory to about half the RAM, the layers that do not fit stay in pageable RAM (same tokens, slower).
Disk: about 135 GB (Flash-Next NVFP4), 110 GB (64 GB edition), 20 GB (27B NVFP4), 15 GB (27B 3-bit).

Quick start

Default model (Flash-Next NVFP4, two 16 GB cards); for another row of the table add its --model:

curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash -s -- --check   # can this machine run it?
curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash               # release + model into ~/capy-flash
~/capy-flash/current/serve-flash-next.sh ~/capy-flash/models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"capy-flash","messages":[{"role":"user","content":"What is 17*23?"}]}'                # from another shell

The installer checks GPUs, driver, AVX2/AVX-512, RAM and disk and says what to fix, downloads the latest release and one model repository from Hugging Face, verifies both, and prints the command that serves it and a test request. --dir picks another place, --no-model downloads the release only, --verify-model also checks the model's digests.
The first start takes a minute or two (a cold start of Flash-Next reads its experts from disk).
Launchers. serve-flash-next.sh (Flash-Next; NINFER_DEVICES=0 for one GPU), serve-tp2.sh (27B, two GPUs) and serve-27b-16gb.sh (27B 3-bit, one GPU) start ninfer-serve with the tested profile, the model id (capy-flash or qwen3.8-27b, the model name requests must use) and the chat template. NINFER_PORT=9000 changes the port; extra ninfer-serve flags go after the model path and win over the profile.
Vision. The two 27B artifacts include the vision tower and their launchers load it (it takes VRAM from the context); NINFER_VISION=0 before the launcher, or --no-vision after the model path, turns it off. The Flash-Next downloads are text only: images on Flash-Next need the separate Vision artifact, which install.sh does not fetch, and --vision.
Keep it running. ninfer-serve exits with status 2 when the Engine fails, so run it under a supervisor that restarts it: a systemd unit and a llama-swap entry are in examples/serving.
By hand: download a release tarball (contents) and hf download Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash --local-dir <dir>.

Connect a client

The server listens on http://127.0.0.1:8080 (NINFER_HOST / NINFER_PORT change it): OpenAI Chat Completions and Responses under /v1, Anthropic Messages at /v1/messages. --api-key KEY makes it require Authorization: Bearer KEY or x-api-key: KEY; --api-key-file FILE or the NINFER_API_KEY environment variable set the key without putting it on the command line.

Claude Code: ANTHROPIC_BASE_URL=http://127.0.0.1:8080 (any model name is accepted on Messages).
Codex CLI: a custom provider with base URL http://127.0.0.1:8080/v1, the responses wire API and the server's model id. It needs the chat template the launchers pass (qwen3_8.jinja), because Codex always sends instructions.
OpenCode and other OpenAI clients: base URL http://127.0.0.1:8080/v1, model capy-flash (or qwen3.8-27b).

Details, including authentication and what each client sends: docs/serving.md.

Run it other ways

Build

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a \
  -DNINFER_BUILD_APPS=ON -DNINFER_BUILD_BENCHMARKS=OFF -DBUILD_TESTING=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc
ninja -C build apps/ninfer-serve
build/apps/ninfer-serve <dir>/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --model-id capy-flash \
  --chat-template tools/chat_templates/qwen3_8.jinja

Or take the prebuilt tarball of a release (Ubuntu, sm_120a), which has the three launchers (tools/release).

Profiles

--profile NAME applies the tested configuration of a model; flags given on the command line override it, in any order. ninfer-serve --help and docs/serving.md list the full expansions.

profile	for	in short
`flash-next`	Flash-Next on two GPUs (the launcher's default)	`--tp 2`, INT8 KV, elastic KV / expert slots, 262,144 tokens, MTP 3, 4 GiB of idle KV and 8 conversation states in RAM, embedding in RAM
`flash-next-1gpu`	the 64 GB edition on one GPU (`NINFER_DEVICES=0`)	the same on one device, 1 GiB of idle KV and 8 states in RAM
`27b`	Qwen3.8-27B NVFP4 on two GPUs	`--tp 2`, INT8 KV, 196,608 tokens, MTP 3, Vision, 8 GiB of context cache in RAM
`27b-16gb`	Qwen3.8-27B 3-bit on one GPU	INT8 KV, 65,536 tokens, MTP 3, Vision, embedding in RAM (13.9 GB of VRAM)

For the 64 GB edition on two GPUs add --host-kv-mib 1024 --host-state-slots 2 to --profile flash-next (smaller RAM caches); install.sh --model flash-next-64gb prints that command.

Docker

docker/Dockerfile.release packs a release tarball (builds in about a minute; the root Dockerfile compiles from source instead). It needs the NVIDIA Container Toolkit:

docker build -f docker/Dockerfile.release -t capy-flash .
docker run --gpus all -p 8080:8080 -v ~/capy-flash/models:/models capy-flash \
  /models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --host 0.0.0.0 \
  --model-id capy-flash --chat-template /opt/capy-flash/chat_templates/qwen3_8.jinja

Tested with rootless Podman and the two GPUs passed through (Flash-Next loads its 64.6 GiB of page-locked experts without extra limits).

Model weights

Ready to serve on Hugging Face (install.sh --model downloads one of these; each has its SHA256SUMS):

Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash (126.6 GB, the default; NVIDIA Open Model License together with the Qwen Community License of Qwen3.8-Flash-Next). Or convert nvidia/Qwen3.8-Flash-Next-NVFP4 yourself in one command, same result: python tools/capy_flash/convert.py --model <checkpoint dir> --out <dir>. It runs python -m tools.convert --recipe qwen3_8_flash_next_nvfp4_idxbf16_gdnq8 --components text,mtp (experts NVFP4 as published, indexer BF16, GDN Q8, about 5 minutes on a GPU, 118 GB) and tools/qwen4_draft_shortlist.py. The previous text artifact, with NVIDIA's MTP head, stays available as revision v0.5.4 (hf download ... --revision v0.5.4).
Feyd89/Qwen3.8-Flash-Next-NVFP4-Vision-capy-flash (the same plus the Q8 vision tower; download by hand, serve with --vision; or convert.py ... --vision).
Feyd89/Qwen3.8-Flash-Next-GSQ-RCO-capy-flash (102 GB, the 64 GB edition: ISTA-DASLab's GSQ-RCO experts, the rest as above).
Feyd89/Qwen3.8-27B-GSQ-3bit-capy-flash (12.5 GB, ISTA-DASLab's 3-bit GSQ 27B plus the official MTP layer; recipe qwen3_8_27b_gsq3).
Feyd89/Qwen3.8-27B-QUASAR-QAT-nvfp4-NInfer (17.4 GB, the 27B for two GPUs; model card).

How the engine runs these models, every server flag, the RAM tiers, the chat template and the CPU threads: docs/serving.md. Environment variables and diagnostics: docs/capy-flash/diagnostics.md. All documentation: docs/README.md.

Benchmarks

Our machine: AMD Ryzen 9 9900X (12 cores), 96 GB DDR5-5600, 2x RTX 5070 Ti 16 GB on PCIe 5.0 x8 + x8 (no P2P), Ubuntu, CUDA 13.1, full clocks. Decode is the server-side rate. The agentic replay is bench/capy_flash: 3 synthetic coding-agent sessions, 53 turns, prompts growing to ~81k tokens with ~90 % prefix reuse, 256 output tokens per turn, temperature 0. LocalMaxxing is the reasoning-v1 benchmark (short prompt, 512 tokens).

Ours (one GPU = one RTX 5070 Ti):

model, edition	GPUs	version, date	measured	result
Flash-Next NVFP4	2	v0.6.0, 2026-10-18	agentic replay: decode median / total time / time to first token, median (p90)	155 tok/s / 177 s / 1.48 s (2.62 s)
Flash-Next NVFP4	2	v0.6.0, 2026-10-18	prefill, cold, ~3.5k / ~15k / ~55k-token prompts	2,422 / 4,615 / 4,828 tok/s
Flash-Next NVFP4	2	v0.6.0, 2026-10-18	LocalMaxxing reasoning-v1	178 tok/s
Flash-Next NVFP4	2	v0.7.0, 2026-10-22	agentic replay decode median	155 tok/s (as v0.6)
Flash-Next 64 GB	2	v0.7.0, 2026-10-22	agentic replay decode median	182 tok/s
Flash-Next 64 GB	2	v0.7.0, 2026-10-22	LocalMaxxing reasoning-v1, verified run	234 tok/s
Flash-Next 64 GB	2	v0.7.2, 2026-10-23	prefill 4k / 16k / 64k	2,168 / 3,924 / 3,994 tok/s
Flash-Next 64 GB	1	v0.7.0, 2026-10-22	LocalMaxxing reasoning-v1, verified run	121 tok/s
Flash-Next 64 GB	1	v0.7.1, 2026-10-22	agentic replay: decode median / total time / time to first token, median / prompt tokens from cache	104 tok/s / 276 s / 1.9 s / 83 %
Flash-Next 64 GB	1	v0.7.2, 2026-10-23	prefill 4k / 16k / 64k	1,671 / 2,838 / 2,921 tok/s
Qwen3.8-27B 3-bit	1	v0.7.0, 2026-10-22	decode, 8k-token real-text prompt, MTP	139 tok/s
Qwen3.8-27B 3-bit	1	v0.7.0	prefill 4k / 16k	3,401 / 3,211 tok/s
Qwen3.8-27B 3-bit	1	v0.7.0, 2026-10-22	LocalMaxxing reasoning-v1, verified run	174 tok/s
Qwen3.8-27B 3-bit	2	v0.7.0, 2026-10-22	decode, 8k-token real-text prompt, MTP	230 tok/s
Qwen3.8-27B 3-bit	2	v0.7.0	prefill 4k / 16k / 64k	5,787 / 5,567 / 4,304 tok/s
Qwen3.8-27B 3-bit	2	v0.7.0, 2026-10-22	LocalMaxxing reasoning-v1, verified run	288 tok/s
Qwen3.8-27B NVFP4 (QUASAR)	2	v0.6.0	prefill 4k / 16k / 64k, `--profile 27b`	9,343 / 9,234 / 6,287 tok/s

More on the 27B NVFP4 (--profile 27b, MTP; 280 prompts in three languages, v0.6): the same tokens as v0.5.4 on 280 of 280 answers (and with DFlash2), decode +5 % (Italian 168 → 176 tok/s), prefill at 4k / 16k / 64k 5,777 / 5,585 / 4,352 → 9,343 / 9,234 / 6,287 tok/s with MTP and 5,719 / 5,530 / 4,316 → 9,205 / 9,052 / 6,198 with DFlash2; concurrent decode (NInfer's decode-saturation suite, MTP3, 8,192 tokens per request) 175 / 326 / 560 / 927 tok/s aggregate at 1 / 2 / 4 / 8 requests. On an agentic replay of the 27B (3 interleaved sessions and 11 subagents, 4 lanes), the v0.5.4 context cache took the median time to first token from 2.14 to 0.98 s and the prompt tokens served from cache from 67 to 78 %, decode unchanged. Flash-Next decode stays flat with context: 168 / 165 / 171 / 165 tok/s with 0 / 14k / 61k / 118k tokens of prompt (v0.6), and a cold 382-token prompt answers in 1.35 s. A cold start of the Flash-Next server takes 26-29 s here (v0.5.3), most of it reading the 64.6 GiB of experts. The 64 GB edition peaks at 55.2 GiB of RSS inside a 64 GiB memory cgroup without swap (57 GiB used with one GPU); its speeds were measured on the 96 GB machine.

Reported by users (not reproduced by us):

setup	model, edition	version	measured	result	source
AMD EPYC 7F52 16 cores (Zen 2, AVX2), 2x RTX 5080 (overclocked)	Flash-Next NVFP4	v0.7.2 built with CUDA 13.4	agentic replay (53 turns, greedy): decode median	141.3 tok/s with automatic threads; 144.4 with `--expert-cpu-threads 15` and `taskset -c 0-15`; release tarball (CUDA 13.1) 137.6; v0.7.1 tarball 121.8	issue #11
same	same	same	own measurement, not greedy	prefill at 131k 5,541 tok/s; time to first token 22.1 s at 131k, 1.15 s at 1k	issue #11
AMD Threadripper PRO 5975WX (AVX2), 2x RTX PRO 4000 Blackwell 24 GB (145 W), Proxmox LXC	Flash-Next 64 GB (GSQ-RCO)	v0.7.1 built from source, CUDA 13.3	decode / prefill / local aider coding benchmark	decode ~100-114 tok/s on prose, ~140-155 on reasoning; prefill at 40k ~3,300 tok/s; aider 145-146 of 148	issue #22

Against Strata. Strata 0.1.45 on our machine, same day (2026-10-18), two GPUs, capy-flash v0.6.0:

	capy-flash, NVFP4	Strata, same NVFP4 weights	Strata, 3-bit IQ3_S (tuned)
agentic replay, 53 turns up to 81k tokens: total time	177 s	268 s	197 s
agentic replay: decode, server side (median)	155 tok/s	98 tok/s	151 tok/s
agentic replay: time to first token, median (p90)	1.48 s (2.62)	2.35 s (4.17)	1.60 s (3.69)
prefill, ~3.5k-token prompt	2,422 tok/s	1,519 tok/s	2,216 tok/s
prefill, ~15k-token prompt	4,615 tok/s	3,513 tok/s	3,811 tok/s
prefill, ~55k-token prompt	4,828 tok/s	4,849 tok/s	5,233 tok/s
LocalMaxxing reasoning-v1 (short prompt, 512 tokens)	178 tok/s	119 tok/s	195 tok/s

Strata's IQ3_S experts are 26 % smaller than NVIDIA's NVFP4 ones (lower quality: KL 0.144 vs 0.070 for NVFP4, as Strata reports), which is why it leads on the short benchmark. v0.6 against the build before the v0.6 work (2026-10-17): replay total 212 → 177 s, decode 118 → 155 tok/s.

Comparing decode with other engines. capy-flash computes every routed expert the model selects; an expert that misses the VRAM cache is computed on the CPU, never skipped. Strata's "route tail skip" skips low-ranked experts that miss the cache, which changes the output slightly and saves CPU work. A decode-speed gap against Strata, especially on a slower CPU, most likely comes from that. Measured here with the same GSQ-RCO IQ3_S experts on both (one RTX 5070 Ti, LocalMaxxing decode, 2026-10-23): Strata 138 tok/s with the skip, 116.6 with STRATA_ROUTE_TAIL_SKIP=0, capy-flash 129.8. On two GPUs (capy-flash v0.7.0 against Strata 0.2.2, 2026-10-22) the agentic replay decode median was 200.9 against 171.3 tok/s; Strata finished that replay sooner overall (189 against 200 s) thanks to its faster prefill.

Quality against the GB300 reference (NVIDIA's NVFP4 checkpoint served by vLLM on a GB300, same prompts, paired):

model, version	GSM8K (reference)	MMLU-Pro (reference)
Flash-Next NVFP4, v0.6.0	0.9750 (0.9810), 1,319 questions; 9 / 1 discordant, McNemar p 0.02	0.8413 (0.8452), 1,008 questions: non-inferior (difference -0.40 pt, lower bound -1.85 pt, margin -2.5 pt)
Flash-Next NVFP4, v0.7.0	0.985 (0.985), 330 questions	0.829 (0.849), 252 questions; paired with v0.6: no difference
Flash-Next 64 GB, v0.7.0	0.982 (0.985), 330 questions	0.837 (0.849), 252 questions
Qwen3.8-27B 3-bit, v0.7.0	0.968 (0.968), 1,319 questions	0.69-0.71, about 0.72 with the empty-reply guards (0.78), 252 questions

The v0.6.0 release binary was checked on the first 330 / 252 questions (GSM8K 0.979, MMLU-Pro 0.841). Output is not bit-reproducible run to run (which experts sit in VRAM depends on timing), so two runs of the same build differ by a few questions. The 3-bit 27B loses MMLU-Pro points mostly because more answers run out of their 8,192 tokens while reasoning; the v0.7 empty-reply guards bring back the ones that stopped inside their reasoning (9 of 9, all correct: about +3.6 points, estimated on those questions). Agent tasks (2026-10-14 build): 5 multi-step coding tasks of a private task bank, 4 runs each, OpenCode with the model's sampling: 20/20 solved, 42.9k generated tokens per run (Strata 0.1.43 IQ3_S on the same tasks: 52.7k, 20/20; the GB300 reference: 26.5k), no malformed tool call in 1,124.

Known limits

Flash-Next is tested with one request at a time (--max-concurrency 1, as its profiles set) and only on the machine above. The server accepts up to 8 (--max-concurrency); the 27B NVFP4 was measured at 1 to 8.
With several requests decoding, an answer that outgrows its KV window when the GPUs are full first moves idle cached conversations to RAM (or drops them without a RAM tier); only if that cannot free a page does it end early with finish_reason: length (the request log counts it). --kv-lease-window 0 restores the full reservation of v0.5.3.
Moving a conversation between RAM and the GPUs shares PCIe with the two GPUs' exchange: decode of other requests is slower while a move is in flight (no difference otherwise).
Generated tokens per agent task are about 1.6x the GB300 reference (above): the remaining gap with the datacenter stack is length of reasoning, not correctness.
Output is not bit-reproducible run to run: which experts sit in VRAM (GPU path) or not (CPU path) depends on timing, and long streamed prefills differ in the last bits; quality is measured with that in place.
Decode of Flash-Next is bound by host-RAM bandwidth for the experts computed on the CPU (about 270-300 per MTP window), so a slower CPU or RAM decodes slower.

Tests

ctest --test-dir build runs the unit tests. The Engine integration tests run when their artifacts are set: NINFER_TEST_QWEN4_ARTIFACT (a small synthetic Flash-Next artifact with the real layer geometry, made by tools/capy_flash/make_test_artifact.sh <checkpoint dir> <out dir>, CPU only) and NINFER_TEST_ARTIFACT (a Qwen3.8-27B artifact, for the tests inherited from ninfer-tp2). For the 27B, tools/tp2/gate.sh is the reference run: it sets the artifacts, runs the two-device integration tests and excludes the seven single-GPU tests that need other artifact kinds (DFlash, MoE, vision workspace; GATE_CTEST_EXCLUDE), then checks golden ids, perplexity and greedy texts against a recorded reference (tools/tp2/README.md).

What's new

v0.7.7

Two engine-wide failures fixed. Both could follow a reasoning-close retry (--reasoning-close-retry, on by default) when a later request repeated the prompt or restored from its turn anchor. The fixes are a continuation that no longer registers the chat prompt's capture points (PR #36 by @omni326), and a state fork closed before a shared prefix is promoted at the reuse point. Recommended for everyone.

v0.7.6

/v1/messages with four cache_control breakpoints (the Anthropic maximum, which OpenCode sends) no longer returns HTTP 400: the server's own turn anchor gives way when the request already uses every marker slot (PR #34 by @omni326).
--spec off (or none) after a --profile runs the profile without speculative decoding, dropping its --draft-tokens and --lm-head-draft (issue #33).
Builds with GCC 16 (issue #31); an out-of-range write in the context cache's KV lease bookkeeping is fixed.

v0.7.5

--miss-split calibrate (Flash-Next NVFP4, opt-in; asked for in issue #11): the decode miss split with costs measured on your machine instead of the fixed ones fitted on our Ryzen 9 9900X. It times the GPU's copy of an expert over PCIe at startup and the CPU's time per expert over the first 4,096 decode calls, logs both, then splits with them. On our box it lands where the fixed costs already were, and splitting does not beat off there (agentic replay, two runs: one GPU 74.8 / 73.5 tok/s against 72.9 / 73.0 with off; two GPUs 155.5 / 152.5 against 157.6 / 154.1), so the default stays off. It is there to check whether the split pays on other CPUs and links (docs/serving.md).

v0.7.4

Disk tier: a long conversation saved on disk now resumes even when a shorter match (a system prompt another conversation left) is in memory, and resumes work on one GPU (27B: 7.63 → 0.46 s on two GPUs, 15.18 → 0.60 s on one; issue #6).
NINFER_HOST_MOE_CPUS takes ranges (0-15,32-47); --api-key-file FILE and NINFER_API_KEY keep the key off the command line; a second server on the same port is refused.

v0.7.3

CPU expert threads: when the allowed CPUs have no free SMT sibling (for example taskset -c 0-15 on a 16-core / 32-thread CPU, or SMT off) and the automatic count would take every core, one core is left to the main thread, which spin-waits on the GPU. Reported in issue #11: on an EPYC 7F52, 16 threads on taskset -c 0-15 dropped decode to 80-90 tok/s, against about 144 with 15. --expert-cpu-threads N still overrides (details).
64 GB edition: the startup line Qwen4Exp offload experts: 49 of 49 expert layers in pageable host memory was a false alarm (the check read the NVFP4 bank pointers, which ggml block banks do not use); fixed, no speed change (issue #22).
Documentation: every server flag and the four profiles in docs/serving.md, client setup, the chat template, the vision opt-out.

v0.7.2: pick up where you left off.

Disk tier of the context cache (opt-in, --disk-cache-dir DIR --disk-cache-gib N; idea by @KouShuo): conversations survive a restart and resume without a new prefill (27B, two GPUs: 140k tokens in 1.08 s instead of 48 s); see docs/serving.md.
Faster long prompts on the 64 GB edition of Flash-Next: prefill 64k +19 % on two GPUs (3,994 tok/s) and +35 % on one (2,921 tok/s), same tokens.
CPU expert threads chosen from the machine (physical cores, affinity, L3 domains) instead of fixed counts; --expert-cpu-threads N sets them by hand.

v0.7.1: --profile flash-next-1gpu keeps 8 conversation states in RAM (+0.66 GiB): on the agentic replay with one GPU 83 % of the prompt tokens come from the cache, 1.9 s to the first token (median) and 276 s for the whole replay (v0.7.0: 34 %, 13.3 s, 858 s).

v0.7.0: start small, grow later, same engine.

Flash-Next on 64 GB of RAM, with one or two GPUs (--model flash-next-64gb, GGUF i-quant experts).
Qwen3.8-27B on one 16 GB card (--model 27b-16gb, 3-bit GSQ weights, 64k context, vision).
AVX2 CPUs (Zen 2/3, Intel client) and Windows/WSL2 with its page-locked memory limit start Flash-Next too.
Empty-reply guards on by default for every model; boards with P2P overlap the prefill exchanges on the direct copy (+29-45 % measured by @tzolkincz); a context-cache accounting error fails one request instead of the server.

Older releases: GitHub releases and the development log, which records the work behind the Tnn tags in the sources.

License

Apache License 2.0 (LICENSE); attribution in NOTICE. The original ninfer-tp2 README is in docs/README-ninfer-tp2.md (historical).

Searching Database…

124 ·GitHub

Capy-flash

8-27B on one or two 16 GB RTX 50-series GPUs: experts offloaded to CPU, two-GPU tensor parallel. 64 GB of RAM is enough for Flash-Next. Formerly ninfer-tp2.

Creator: ValerioDolci
Size: 30 MB
Stars: 124
Updated: CommitOct 27 Releasev0.7.7Oct 27
Age: 1mo
Commits · 90 Days