Qwen3.8-Flash-Next on a single DGX Spark (GB10)
Run Qwen3.8-Flash-Next — a ~176B-parameter model (125B main + 51B n-gram, 6B
active) — on one NVIDIA DGX Spark / ASUS GX10 with vLLM, at full prefill
speed, with MTP speculative decoding, working prefix caching, deterministic
greedy decoding, and up to 500k tokens of context.
The catch this repo solves: the NVFP4 checkpoint is 126 GiB, which does not fit
next to a usable KV cache in the Spark's 128 GB unified pool. 48 GiB of that is
the n-gram embedding ("PLE") table — a pure lookup that a token only touches 16 rows
of. This repo patches the official vLLM image to serve that table from NVMe via
mmap instead of keeping it resident. Weights drop to ~76 GiB, the rest of the
pool goes to KV, and everything runs on stock GB10 kernels.
Along the way it also fixes two things that were broken for this model on GB10 —
prefix caching (a vLLM block-size bug silently restored an all-zero Mamba state
on every cache hit) and non-deterministic top-k in the sparse attention (a GB10
kernel that drops candidates) — and offers an optional hybrid checkpoint layout
(NVFP4 experts + fp8 side layers) that decodes ~20% faster at the same quality.
TL;DR — run it on a DGX Spark
git clone https://github.com/blazux/qwen3.8-Flash-DGX.git && cd qwen3.8-Flash-DGX
docker build -t qwen38-flash-dgx . # ~1 min: official vLLM image + the 10 patches below
scripts/download-weights.sh # RadixArk NVFP4 checkpoint, ~126 GiB, resumable (one-time)
scripts/prepare-hybrid.sh # recommended: fp8 side layers, +20% decode, same quality (~10 min, one-time)
MODE=hybrid YARN=1 CTX=500000 scripts/serve.sh # the recipe our own box runs; 500k context, ~13 min to load
docker logs -f qwen38-flash # ready at "Application startup complete"
scripts/smoke-test.sh # health, coherence, prefix-cache hit, determinism, tok/s
OpenAI-compatible API on http://localhost:18300/v1, model name qwen3.8-flash-next,
tool calling and reasoning parsers on. Every default is the setting that scored best on our
agentic tournament (see How the defaults are chosen);
what you get on a GX10: ~37 tok/s single-stream decode, ~2,500–3,000 tok/s prefill, prefix
caching, deterministic greedy output, 500k tokens of context. Want the checkpoint exactly as
published? Drop prepare-hybrid.sh and MODE=hybrid. Want speed over the last percent of
quality? MTP=3, and MODE=hybrid-mtp for more KV — both explained in the options table.
Everything below is the long version: what was broken on GB10, what was fixed, and the numbers.
Independently reproduced on a DGX Spark by
@jschmied — see
issue #1 and their
write-up.
Update 2026-09-08 — what changed
Newest first. If you cloned this before, this is the short version; details in the linked sections.
2026-09-08 — the defaults are now decided by an agentic/coding benchmark (called tournament), and two new ones came out of it:
- Quality is the gate for defaults now. Every default in
scripts/serve.sh is the setting that
scored best on the 17-scenario agentic tournament (3 repeats); anything that only buys tok/s
or TTFT is an option. MTP=3 (+7% decode, −1 point), fp8 KV, the M%4 padding and the exact
top-k fallback are documented options, not defaults.
- The MTP drafter now scores a 65,536-token vocabulary instead of 248,320 (
DRAFT_VOCAB=1,
default): the target verifies every drafted token, so outputs are unchanged; the draft step
reads 320 MiB of head instead of 1.27 GiB. Measured with the tournament, one variable at a
time on the same day: 45/51, the best score of any configuration we ran, at 38.5 tok/s
(+23%). Idea taken from MiaAI-Lab's recipe,
reimplemented here (their code is AGPL). Same source for the MADV_RANDOM advice on the
mmapped table, now on by default (no readahead: cold prefill −4–8%, cleaner page cache, a
slightly larger KV pool). → Reduced draft vocabulary
MODE=hybrid-mtp — @pfy's graft of Inferact's NVFP4 MTP draft
experts onto the hybrid checkpoint (PR #11): −3.9 GiB of weights, +22% KV pool. On our box
decode is unchanged (the cheaper draft is accepted less often) and the tournament is neutral
(44/51), so it ships as an option for people who need context or concurrency more than the
last percent of quality. → NVFP4 MTP draft experts- Full same-day comparison behind those choices (all hybrid, MTP=2, prefix caching, YaRN 500k):
exact
torch.topk 43/51 @31.4 tok/s → deterministic kernel 44/51 @31.9 → MADV_RANDOM 44.5/51 @33.5 →
reduced draft vocabulary 45/51 @38.5 (default) → same with MTP=3 44/51 @41.2 (option) → hybrid-mtp 44/51 @33.0, KV +22% (option).
2026-09-07 — kernel pin bump and multi-client guidance:
- Greedy decoding is deterministic now — at no prefill cost. The GB10 sparse-attention
top-k kernel was non-deterministic and dropped candidates, diagnosed and reported upstream by
@k3dani (issue #3, vllm#51782). First fixed with an exact
torch.topk (deterministic but −20–40% on long prefill); now replaced by
@jschmied's deterministic kernel (vllm#55122), compiled
into the image: identical outputs at temperature 0 and full prefill speed back
(32k: 1,794 → 2,996 tok/s). DET_TOPK=1 is the default; EXACT_TOPK=1 stays as a fallback.
Kernel pin bumped 2026-09-07 (PR #10): signed-zero fix, low-shared-memory path, a launcher bug
that would have crashed some long-context widths, and a faster kernel.
→ Deterministic top-k
2026-08-29 → 2026-09-07:
- Prefix caching works now —
--enable-prefix-caching was crashing, then silently
returning wrong answers on cache hits. Root cause was a vLLM block-size bug that made
every prefix hit restore an all-zero Mamba state; two-line fix in the image. Getting
there took @Saren-Arterius's pointer to
vllm#50729 and their state-copy guard, and @0xBakeer's
attempt to reproduce it, which sharpened the write-up.
PREFIX_CACHE=1 is the new default. Repeated prefixes (system prompts, multi-turn,
tool loops) skip the prefill: ~14 s → ~1.4 s TTFT on a 20k-token prefix.
→ Prefix caching now works
- Optional M%4 padding for the fp8 GEMM (
PAD_M4=1, hybrid mode) — the image's blockwise-fp8
kernel is up to 10× slower on chunks whose row count is not a multiple of 4; the padding is
@jschmied's. With prefix caching on (default) chunks are already
aligned and it changes nothing, so it is off by default; with PREFIX_CACHE=0 it is worth about
−40% TTFT at 8k. → M%4 padding
- Why decoding stalls when other clients prefill, and the slider for it — reported by
@kutovoy (issue #9), reproduced and measured: it is vLLM's
chunked prefill (one decode token per 3–7 s step while a new prompt is being prefilled), not a
bug.
EXTRA='--long-prefill-token-threshold 1024' trades single-stream TTFT for
responsiveness; numbers and guidance in the concurrency section.
- Audit fixes — @sternnick audited the repo line by line
against their own Spark (issue #8). Taken so far: the files fetched from @jschmied's repo are
pinned by sha256 as well as by commit (and that repo is Apache-2.0 now), the fp8-KV guard only
admits
e4m3 (the kernel launch never handled e5m2), GPU_MEM defaults to 0.80 as the
docs already recommended, and the undocumented VLLM_PLE_MMAP_CHUNK and the no-auth binding
are in the docs. Their three fixes are merged with their authorship: the measured sizes
(the checkpoint is 126 GiB, the table 48 GiB, plan 140 GB of disk), the PLE range guard
(the last shard is partial), and the scripts (snapshot resolved from refs/main, a start
check after docker run, the prefix-cache hit proven with vllm:prefix_cache_hits_total
instead of a stopwatch).
- Two checkpoint modes —
MODE=nvfp4 (as published) or MODE=hybrid (NVFP4 experts
- fp8 side layers, one-time
scripts/prepare-hybrid.sh): +20% decode, +8% KV,
same quality. Our box runs the hybrid. The fp8 side-layer conversion and the
original int4+fp8 dispatch it is ported from are
@Saren-Arterius's. → Two checkpoint modes
- Also in the image: vllm#50729 (Mamba state-copy race, by
@AndreasKaratzas) + a bounds guard, the GB10 FLA
fixes and the faster PLE gather from @Saren-Arterius's fork.
- We benchmarked the int4 (Intel AutoRound) variant too with the same patches:
fastest raw decode, but not deterministic and slowest cached-TTFT, so we did not adopt
it. Numbers in docs/HOW-IT-WORKS.md.
- fp8 KV cache is available (
KV_DTYPE=fp8_e4m3), contributed by
@Nanetnounou: ×1.9 KV, 1M context on one box — at a
speed and quality cost, so it is opt-in. → fp8 KV cache
scripts/smoke-test.sh now also checks the prefix-cache hit and determinism, and
measures decode on a real answer instead of ignore_eos (which produces meaningless
numbers with this model). scripts/download-weights.sh now forwards HF_TOKEN
(@wawimundo, PR #4).
Everything was measured on one ASUS GX10 with a 17-scenario agentic tournament (3 repeats
each), single-request speed benches on real prompts, and state checksums for the
prefix-caching work; nothing here is extrapolated.
|
llama.cpp IQ4_XS |
NVFP4 (this repo) |
hybrid (this repo) |
| Prefill |
~540 tok/s |
~2,400–2,900 tok/s (deterministic kernel; warm page cache — a first pass over a cold region of the table reads from NVMe and can be 2–3× slower, see PREWARM) |
same |
| Decode, single stream |
~22 tok/s (no MTP) |
~26 tok/s with MTP=2 |
~37 tok/s (reduced draft vocabulary; ~31 without) |
| Prefix-cache hit, TTFT on a 20k-token prefix |
n/a |
~1.4 s (vs ~14 s cold) |
same |
| Context |
262k |
262k native, 500k with YaRN |
same |
| KV cache @0.80 (500k YaRN, MTP) |
— |
~580k tokens |
~630k tokens |
| Deterministic at temperature 0 |
yes |
yes (DET_TOPK=1) |
yes |
Measured on an ASUS GX10 (GB10, 128 GB), single request, real prompts, greedy. Quality
(a 17-scenario agentic tournament, 3 repeats) is identical across NVFP4 and hybrid:
45/51 both, same two scenarios failed by every quantization we tried. Details and the
full comparison tables are in docs/HOW-IT-WORKS.md.
How the defaults are chosen: quality first, speed as an option
Every default in scripts/serve.sh is the setting that scored best on our 17-scenario
agentic tournament (tool loops, long-context extraction, multi-step reasoning; 3 repeats,
temperature 0.2), run on the GX10 with one variable changed at a time, on the same day. A
change that only buys tok/s or TTFT and costs even a point there ships as an option, off
by default, with its measured cost next to it. Two runs of the same configuration differ by
up to 2 points day to day, so anything inside that band is treated as equal and the faster
one wins; anything below it stays an option.
| what |
default |
measured effect (GX10, hybrid, MTP=2, prefix caching, YaRN 500k) |
Hybrid checkpoint (MODE=hybrid) |
recommended, nvfp4 as published is the default |
+20% decode, +8% KV, same tournament score |
Deterministic top-k kernel (DET_TOPK=1) |
on |
identical greedy outputs, full prefill speed, tournament neutral (44/51) |
Reduced draft vocabulary (DRAFT_VOCAB=1) |
on |
+20% decode, tournament 45/51 (the best run), outputs unchanged by construction |
MADV_RANDOM on the table (MADVISE=random) |
on |
cold prefill −4–8%, cleaner page cache, tournament neutral |
Prefix caching (PREFIX_CACHE=1) |
on |
~14 s → ~1.4 s TTFT on a repeated 20k prefix |
MTP=3 |
option (MTP=2 default) |
+7% decode, −1 point at the tournament (44 vs 45/51) |
NVFP4 MTP draft experts (MODE=hybrid-mtp) |
option |
+22% KV pool, −3.9 GiB weights, decode unchanged here, tournament neutral (44/51) |
fp8 KV cache (KV_DTYPE=fp8_e4m3) |
option |
×1.9 KV pool, 1M context; −10% decode, −30% prefill, one scenario lost |
M%4 GEMM padding (PAD_M4=1) |
option |
no-op with prefix caching on; −40% TTFT at 8k with it off |
Exact torch.topk (EXACT_TOPK=1) |
fallback |
deterministic like the kernel, −20–40% long prefill |
--long-prefill-token-threshold (via EXTRA) |
option |
keeps decoding clients responsive under concurrent prefills, at a TTFT cost |
If your priority is raw throughput rather than the agent's reliability, the fast profile is
MODE=hybrid MTP=3 (41 tok/s in the tournament against 38.5 for the default), and
MODE=hybrid-mtp on top if you need the KV pool more than the last few percent of quality.
Requirements
- An NVIDIA DGX Spark or compatible GB10 (sm_121) box, 128 GB unified memory,
aarch64, recent NVIDIA driver, Docker with the NVIDIA container runtime.
- ~140 GB free disk for the checkpoint (+13 GB for the hybrid variant, +5 GB more
for the NVFP4-MTP graft), on
reasonably fast storage (the table is read from it at runtime — NVMe strongly
recommended; the Spark's onboard NVMe is ideal).
- The base image is multi-arch, so
docker build also works on x86 Blackwell
(sm_120, e.g. RTX PRO 6000) for testing, though this is tuned for the Spark.
Quickstart
The commands are in the TL;DR at the top. Once the log says
Application startup complete, hit the OpenAI-compatible API:
curl http://localhost:18300/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen3.8-flash-next",
"messages": [{"role":"user","content":"Write a haiku about a desktop supercomputer."}],
"max_tokens": 512
}'
MODE=nvfp4 scripts/serve.sh (the default) serves the checkpoint as published at the native
262k context; YARN=1 CTX=500000 goes to 500k (validated with a needle-in-a-haystack at 414k
tokens); GPU_MEM=0.80 is the long-running-service setting, see Tuning.
Two checkpoint modes: NVFP4 or hybrid
scripts/serve.sh serves one of two layouts of the same RadixArk NVFP4 checkpoint;
pick with MODE=.
|
MODE=nvfp4 (default) |
MODE=hybrid |
| Routed experts (the bulk, ~63 GiB) |
NVFP4 |
NVFP4 (unchanged) |
| GDN in/out projections, QSA q/k/v/o, shared experts (~15 GiB) |
bf16, as published |
blockwise fp8-e4m3 (128×128 blocks, DeepSeek layout) |
| Extra preparation |
none |
scripts/prepare-hybrid.sh once (~10 min, +13 GB disk) |
| Decode (MTP=2, greedy, real answers) |
~26 tok/s |
~31 tok/s (+20%) |
| Prefill |
same |
same (±5%) |
| KV cache |
~580k tokens |
~630k tokens (+8%), weights ~7 GiB smaller |
| Tournament quality (17 agentic scenarios × 3) |
45/51 |
45/51 |
| Deterministic at T=0 |
yes |
yes |
| Behavioural difference we noticed |
— |
slightly "more careful" in tool loops: it sometimes checks state first (one extra tool call), which is the only place it scored differently before we raised the turn budget |
Why it works: those side layers are dense and read in full on every decoded token, so
they dominate decode bandwidth; the experts are sparse (10 of 512 active) and already
4-bit. Halving the dense part is where the tokens/s come from. The MoE path — where
the quality lives — is untouched. Conversion uses
@Saren-Arterius's fp8_convert.py (worst
per-tensor max relative error 3.5%), and a small dispatch shim
(src/vllm_fp8_hybrid_modelopt.py) that routes those layers to vLLM's blockwise-fp8
GEMM while the ModelOpt NVFP4 config keeps handling the experts.
scripts/prepare-hybrid.sh # builds <snapshot>-fp8hybrid/ next to the HF snapshot
MODE=hybrid YARN=1 CTX=500000 GPU_MEM=0.80 scripts/serve.sh
Our own box runs the hybrid. If you want the checkpoint exactly as published, stay on
MODE=nvfp4 — you lose ~5 tok/s and nothing else.
NVFP4 MTP draft experts (MODE=hybrid-mtp)
A graft on top of the hybrid: the MTP draft head's routed experts (BF16 fused, 4.7 GiB)
are replaced by the NVFP4 draft experts from
Inferact/Qwen3.8-Flash-Next-NVFP4
(1.4 GiB) — the combination neither parent ships: fp8 PLE table and a cheap draft.
scripts/prepare-mtp-graft.sh builds it in ~5 min (3.3 GB of real bytes: one shard
rewritten without the 2 fused BF16 MTP tensors, a symlink to the donor shard, and the
index + both quantization exclusion lists fixed — the blanket mtp.* globs are replaced
by the donor's 29 explicit non-expert MTP module names in both config.json and
hf_quant_config.json, or the draft loads unquantized and dies). Both parents are
modelopt quantizations of the same base — verified by hashing embed_tokens.weight and
all 29 shared draft tensors byte-for-byte before grafting. The graft reads through both
parents (it is a directory of symlinks); don't delete either one.
Measured on the GX10 (hybrid, MTP=2, greedy, real answers):
|
MODE=hybrid |
MODE=hybrid-mtp |
| Weights on card |
77.83 GiB |
74.75 GiB (−3.1) |
| KV cache @0.80 |
625,669 tok |
734,292 tok (+17%) |
| Max concurrency @262k |
2.39x |
2.80x |
| Decode, 5-prompt greedy probe |
26.6–33.8 tok/s |
34.7–42.5 tok/s (+20–27%) |
| Mean acceptance length |
~2.73 |
~2.58 (same band) |
| Tournament quality |
45/51 |
same target model, byte-identical greedy outputs (see below) |
Those are the author's numbers. Ours, measured the way defaults are decided here (same
box, same day as the table in Reduced draft vocabulary,
full-vocabulary draft on both arms):
|
MODE=hybrid |
MODE=hybrid-mtp |
| Tournament (17 scenarios × 3) |
44.5/51 |
44/51 (same failures, within the day's noise) |
| tok/s in the tournament |
33.5 |
33.0 |
| Decode, single stream (bench) |
33.1 tok/s |
32.6 tok/s |
| MTP acceptance |
75% (mean length 2.50) |
63.5% (2.27) |
| KV pool @0.80, YaRN 500k |
626k tok |
764k tok (+22%) |
| Weights on card |
77.8 GiB |
73.9 GiB |
| Deterministic 4/4, smoke-test (cache hit + logprobs) |
yes |
yes |
So on this box the graft is a memory win, not a speed win: the NVFP4 drafter reads ~4×
fewer bytes per draft step but its proposals are accepted less often, and the two cancel.
It is quality-neutral, which is why it ships as an option rather than the default; take it
when the KV pool or concurrency matters more than the last percent. Not yet measured in
combination
这是一份在单台 NVIDIA DGX Spark(GB10)上用 vLLM 服务约 1760 亿参数 Qwen3.8-Flash-Next 模型的方案(脚本加打过补丁的 Docker 镜像)。它要解决的问题是:126 GiB 的 NVFP4 检查点在 Spark 的 128 GB 统一内存里放不下,还要给可用的 KV 缓存留地方;做法是通过 mmap 从 NVMe 流式读取一张 48 GiB 的查找表,而不是让它常驻内存。