Qwen3.8-Flash-Next-Single-DGX-Spark README

The project's README from github.com/MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark. As one markdown file: README.md.

Qwen3.8-Flash-Next on ONE DGX Spark (TP=1)

by Mia'a AI Lab

Sponsor me on GitHub Follow Mia on X

Self-contained recipe for serving the Mia-AiLab/Qwen3.8-Flash-Next-NVFP4 checkpoint (99 GiB) from a single DGX Spark's 121 GiB unified memory, via vLLM with the PLE table offloaded and memory-mapped. This is a vision-language model: text, images and video all work out of the box (see below). Nothing here depends on the 2-node files it was derived from.

cp .env.sample .env        # edit IMAGE / HF_TOKEN if needed
./download.sh              # fetch the ~99 GiB checkpoint (resumable)
./start.sh                 # ~10-12 min to /health; serves on :8888
./stop.sh                  # container + watchdog, graceful

start.sh never downloads anything — it resolves the checkpoint from the local Hugging Face cache and fails fast if it is absent. Budget ~130 GiB of free disk: 99 GiB for the checkpoint plus the ~27 GiB packed PLE table built on first launch.

./start.sh --no-launch prints the derived memory budget and the docker command without running anything. ./stop.sh sends SIGTERM and waits up to STOP_TIMEOUT (default 30 s) so vLLM can unlink its POSIX shared memory — the container runs with --ipc host, so segments it leaves behind leak onto the host's /dev/shm until reboot. ./stop.sh --force skips the wait.

Measured profile

.env.sample ships 262,144 context (YaRN off), MTP 3, HOST_RESERVE_GIB=26, KV_TARGET_GIB=20, KV_CACHE_DTYPE=fp8, MAMBA_SSM_CACHE_DTYPE=bfloat16, MAX_NUM_SEQS=4, MAX_NUM_BATCHED_TOKENS=2048, with the V2 model runner pinned through EXTRA_DOCKER_ARGS. Everything below was measured on this host on 2026-09-04; each row names the configuration it came from, because the numbers move a lot between them. Decode numbers are not in this table: they predate the 2026-09-05 optimisation pass and are superseded by the sparkDash sweep below (48.7 tok/s single-stream prose, 162.9 aggregate at 8 streams).

Configuration KV pool Prefill @400k Needles 5/50/95%
262k, KV_TARGET_GIB=20, BF16 21.28 GiB = 736,837 tok (2.81x a 262k req) — —
512k YaRN, KV_TARGET_GIB=20, BF16 19.2 GiB = 704,558 tok (1.34x a 512k req) 1,537 tok/s (TTFT 260.3 s) 3/3 PASS
512k YaRN, KV_TARGET_GIB=22, BF16 796,196 tok (1.52x a 512k req) 1,883 tok/s @32k 12/14 (see FP8 section)
512k YaRN, KV_TARGET_GIB=22, FP8 22.2 GiB = 1,431,164 tok (2.73x a 512k req) 1,495 tok/s (TTFT 267.7 s); 1,769 tok/s @32k 15/20 (see FP8 section)

The same profile at KV_TARGET_GIB=16 and MAX_NUM_SEQS=5 (262k, HOST_RESERVE_GIB=26, FP8) was measured on 2026-09-05. The shipped wish is now 20, which this host clips to 16.67 GiB — see Configuration:
GPU budget GMU 0.780 = 94.87 GiB
Available KV cache 16.46 GiB = 992,584 tokens (3.79x a 262k request)
Time to /health 10 min 51 s (checkpoint read from NVMe)
Host MemAvailable, 2 min after /health 15.7 GiB (MemFree 5.1 GiB)
Host MemAvailable, 40 idle minutes 15.5–16.4 GiB (MemFree ≥ 4.4 GiB)
Host MemAvailable after two ~90k prompts 16.2 → 15.05 GiB after the first, 15.2 GiB 60 s after the second (min 14.9 during prefill; MemFree ≥ 3.5 GiB)
Host MemAvailable, five concurrent ~60k prompts 14.9 → 14.57 GiB at +60 s (min 14.26 during; MemFree ≥ 3.24 GiB); 5/5 completed, no watchdog event
2.5 h under the qwen-code harness (~38 requests, 19 of them 50–100k tokens, up to 3 concurrent) 14.2–14.9 GiB between turns, min 12.8 GiB at 3 concurrent; driver 96.6 → 97.5 GiB in one step
NV_ERR_NO_MEMORY in journalctl -k 0 across launch and all of the above

The idle figure used to be quoted here as ~12.9 GiB; that came from a 2026-09-04 run — before the host-side cap existed, so KV_TARGET_GIB=20 then meant 20 GiB of KV rather than today's clipped 16.67 — and before the day's co-tenants were on the box. See the safety rules for why the number matters.

One honest gap: KV_TARGET_GIB=20 is now measured end to end — ten launches, the 1/2/4/8-stream decode sweep, needles 15/15 at 32k and a 45-minute soak on 2026-09-06 — but every one of those ran at 512k YaRN, not at the shipped 262k native rope. The 262k row above is at BF16 and predates the MADV_RANDOM mmap change. Short-context (32k) prefill is measured for both dtypes; see the FP8 section.

KV_TARGET_GIB shipped as 22, then 20, until 2026-09-05. Both lost servers: three on 2026-09-04. The rows above at 22 are real measurements, but the host they were taken on had 6.9–8.8 GiB of MemAvailable left, against a 6 GiB watchdog floor and a GPU driver that refuses allocations before that. The budget is now capped from the host side (HOST_RESERVE_GIB, see Safety rules); the KV pool is whatever the cap leaves. The shipped wish is 20 and this host clips it to 16.67 GiB, so 20 and 22 of actual KV are no longer reachable through KV_TARGET_GIB alone; a pinned GPU_MEMORY_UTILIZATION still gets you there, with a warning.

Prefill and decode, measured with sparkDash

Both sweeps below were measured with sparkDash against this server. bench/sweep.py drives it one concurrency level at a time and reads vLLM's own counters around each level, which is what makes the ms-per-step and tokens-per-step columns below comparable across launches; bench/mixed.py covers the decode-under-prefill case sparkDash has no mode for. The two prefill columns are not a clean A/B — they differ in rope config and KV target as well as chunk width — so each is labelled with what it was measured at.

2026-09-06: BF16 recurrent state and every verify width on a graph

Measured on 512k YaRN, MTP 3, FP8 KV, 2,048 chunks, CUDAGRAPH_CAPTURE_SIZES=auto, MAMBA_SSM_CACHE_DTYPE=bfloat16, the V2 model runner pinned, and MAX_NUM_SEQS=8 — which is the one part of this row that .env.sample does not ship, because every sweep here was short-context. Prose, 600 tokens, three repeats per level, bench/sweep.py:

streams ms/engine step tokens/step aggregate per stream
1 61.5 3.00 48.7 tok/s 48.7 tok/s
2 74.3 2.83 74.6 tok/s 37.3 tok/s
4 96.2 2.84 113.7 tok/s 28.4 tok/s
8 131.0 2.81 162.9 tok/s 20.4 tok/s

Two changes separate this from the 2026-09-05 row. MAX_NUM_SEQS=8 with a FULL decode graph at all eight verify widths (4 through 32) is what makes the 8-stream column reachable at all. BF16 recurrent state is worth +8.5% at 8 streams on its own, in a matched pair on the same launch config: 151.6 → 164.5 tok/s, step 141.4 → 130.4 ms, with per-position draft acceptance unchanged within rounding (0.80/0.59/0.41 against 0.79/0.58/0.42) and needle retrieval unchanged at 15/15.

Speculative depth was swept properly here for the first time (K=0/1/2/3 at every stream count, FULL graphs throughout). K=3 wins at every concurrency, K=2 ties it, K=1 loses 8–14% and K=0 loses 32–46%. There is no crossover, so MTP_K_SCHEDULE has nothing to schedule. The full table is in the CHANGELOG.

Decode under a concurrent prefill is the one place the shipped chunk width hurts. With two streams decoding and one 64k prompt arriving, the gap between their streamed chunks for the 34.5 s of that prefill is p50 1,057 ms / p95 1,111 ms / p99 1,400 ms, against 78 ms on the quiet server. Those are per engine step, not per token: vLLM emits one streamed chunk per step carrying that step's accepted tokens (2.7 here, measured), so per-token latency is roughly a third of the figures above. The two streams together delivered 74 steps' worth of output inside the window — ~5.8 tok/s if acceptance holds at its quiet-server value. That is one engine step per 2,048-token chunk, by construction. Halving the chunk to 1,024 takes p95 to 666 ms and p99 to 674 ms and raises in-window delivery to 139 steps (9.3 tok/s), for 5.5% of prefill throughput at 64k. It is not the shipped default (see the CHANGELOG for why the bar was not met), but if your traffic is long prompts arriving against live streams, measure it.

Prefill on this configuration, three sparkDash ladders on 2026-09-06 (one at 256k), server warm:

context TTFT prefill 2026-09-05 change
8k 3.74 s 2,200 tok/s (2,155–2,238) 1,764 +24.7%
16k 7.13 s 2,304 tok/s (2,293–2,311) 2,265 +1.7%
32k 14.18 s 2,314 tok/s (2,305–2,324) 2,265 +2.2%
64k 29.05 s 2,257 tok/s (2,257–2,258) 2,222 +1.6%
128k 61.09 s 2,146 tok/s (2,144–2,148) 2,110 +1.7%
256k 134.89 s 1,944 tok/s 1,913 +1.6%

Prefill did not change; only the 8k row looks like it did. Fit TTFT = tokens / rate + overhead across 16k–128k and the per-token rate is 2,125 tok/s here against 2,089 on 2026-09-05 — +1.7%, with the same −0.60 s intercept in both. Every row above 8k agrees with that. The 8k point sits above the fit in both ladders, by 1.34 s in the 2026-09-05 run and 0.47 s here, because 8k runs first and pays the PLE page-cache warm-up; a server that has been up for hours has mostly already paid it. Read the 8k row as a statement about cache state, not about kernels.

BF16 recurrent state is the only serving change since 2026-09-05, and prefill is within ~2% either way, so its effect here is not separable from run-to-run variation — unlike decode, where it is worth +8.5% at 8 streams. Note the 64k and 128k rows reproduce to ±0.05% across the three ladders, so that ±2% is a between-launch figure, not measurement noise.

2026-09-05: after the decode optimisation pass

Measured on the shipped profile (512k YaRN, 2,048 chunks, MAX_NUM_SEQS=4, MTP 3, FP8 KV, KV_TARGET_GIB=20 → 16.18 GiB = 974,768 tokens) with CUDAGRAPH_CAPTURE_SIZES=auto, the PLE gather prefetch, and reduced-vocabulary drafting (MTP_DRAFT_VOCAB, 65,536 tokens) all active. The "before" columns are the same rope config and chunk width, so decode is a matched pair; prefill differs only in KV_TARGET_GIB (22 → 20), which does not affect prefill rate.

Decode on prose, by concurrent stream count:

streams TTFT aggregate per stream before change
1 270 ms 46.3 tok/s 46.3 tok/s 36.9 +25.5%
2 466 ms 73.0 tok/s 36.5 tok/s 57.4 +27.2%
3 355 ms 91.9 tok/s 31.2 tok/s — —
4 346 ms 108.1 tok/s 27.7 tok/s 85.9 +25.8%

Almost all of that is the reduced draft vocabulary. The MTP drafter reads its own 1.18 GiB BF16 lm_head once per draft step, three of the four lm_head reads in an MTP-3 engine step; slicing it to 65,536 rows saves 2.61 GiB per step, and decode here is close enough to the memory-bandwidth wall that bytes removed convert almost one-for-one into time. Accuracy is unchanged — the target model verifies every drafted token — measured at 250 MGSM problems per language, English 94.8% vs 93.6% and Chinese 86.4% vs 86.4%. See the CHANGELOG entry for the full method.

That win now ships as the default: .env.sample sets MTP_DRAFT_VOCAB to the checked-in files/draft_vocab_en_code_47k.txt (47,149 ids, code-tuned, 99.58% held-out coverage on host code+docs), and start.sh resolves relative paths against the repo and warns when MTP runs with the full head. Empty the knob to restore full-vocabulary drafting. One honest caveat: the shipped file was built from host code and docs, not from the model's own output, so non-code traffic (especially Chinese, where the measured 65k vocab covered only 50.6%) may draft worse than the numbers above; correctness is unaffected, only speed. Live A/B of the shipped file against the full head (sparkDash prose+code sweep, S=1/2/4/8, three repeats) measured +13.1% mean across eight cells with acceptance unchanged — full table in the CHANGELOG 2026-09-09 entry.

Prefill, same chunk width as the shipped column below:

context TTFT prefill before change
8k 4.67 s 1,764 tok/s 1,646 +7.2%
16k 7.25 s 2,265 tok/s 2,052 +10.4%
32k 14.49 s 2,265 tok/s 2,073 +9.3%
64k 29.52 s 2,222 tok/s 2,037 +9.1%
128k 62.15 s 2,110 tok/s 1,945 +8.5%
256k 137.03 s 1,913 tok/s 1,791 +6.8%

The prefill gain is most likely the PLE page-fault prefetch rather than the draft vocabulary, which does not touch prefill: the PLE row gather runs for every prefilled token, so a 2,048-token chunk gathers 16 rows per token — ~32,768 of them — against the ~256 a 4-stream MTP-3 decode step gathers. That gather is single-threaded and was taking each missing 4 KiB page fault on its own; batching the reads with posix_fadvise(WILLNEED) measured 13x on a cold 280-row gather in isolation and only ~3% on decode, where there are too few faults per step for it to matter. This was not isolated with an A/B, so read the attribution as inference from the mechanism, not as a measurement.

These numbers came from one run each. The decode figures are content-dependent for the reason given at the end of this section, and the x3 TTFT below x2 is run-to-run noise.

Prefill. At the shipped 2,048-token chunk width throughput peaks around 32-64k and falls away with context. Raising MAX_NUM_BATCHED_TOKENS to 8,192 flattens it from 16k out to 128k, because per-chunk overhead is amortised over 4x fewer chunks:

context shipped: 2,048 chunks (512k YaRN, KV_TARGET_GIB=22) opt-in: 8,192 chunks (262k native, KV_TARGET_GIB=20)
8k 5.00 s · 1,646 tok/s 3.69 s · 2,228 tok/s
16k 8.00 s · 2,052 tok/s 7.16 s · 2,293 tok/s
32k 15.83 s · 2,073 tok/s 13.87 s · 2,366 tok/s
64k 32.20 s · 2,037 tok/s 28.31 s · 2,316 tok/s
128k 67.41 s · 1,945 tok/s 58.88 s · 2,227 tok/s
256k 146.40 s · 1,791 tok/s not re-measured

The one matched pair — same server, same kernel, only the chunk width changed — is 32k: 2,133 → 2,366 tok/s (+10.9%), TTFT 15.38 → 13.87 s (−9.8%). The rest of the right-hand column is one run each and should be read as indicative.

Raising the prefill chunk width (opt-in)

The 8,192 column is not the default. If you want the faster prefill and TTFT above, set it yourself:

MAX_NUM_BATCHED_TOKENS=8192 ./start.sh     # one launch

or edit the line in .env to make it stick.

It is paid for out of the KV pool, not the GPU budget. 8,192 chunks raise peak activation to 1.27 GiB, and vLLM profiles that before it sizes the KV cache, so the pool absorbs it: 1,145,289 tokens measured at KV_TARGET_GIB=20, still 4.37x a full 262k request. The best matched evidence for the size of that trade is PR #2's own pair at KV_TARGET_GIB=22, 1,282,724 → 1,249,637 tokens — about −2.6% of pool for +11% prefill.

Host MemAvailable sat at 8.1-8.3 GiB idle and low-watered at 7.4 GiB across the 8,192 sweep. It is offered as a knob rather than a default because the supporting observation is minutes, not hours: if you serve long sessions near the memory floor, measure it on your own workload before committing to it.

Decode on prose (2,048 chunks, 512k YaRN), by concurrent stream count. These are the pre-2026-09-05 figures, kept because the tables above are stated as deltas against them:

streams TTFT aggregate per stream
1 418 ms 36.9 tok/s 36.9 tok/s
2 445 ms 57.4 tok/s 29.7 tok/s
4 550 ms 85.9 tok/s 23.4 tok/s

Decode speed on this model is strongly content-dependent, because MTP speculative decoding accepts more drafts on predictable text. Measured 2026-09-06 on sparkDash prose: 2.80 tokens of a possible 4 per engine step, per-position acceptance 0.80 / 0.59 / 0.41 — 60% of drafted tokens accepted. Quoting text back out of the context goes higher still; dense technical prose sits lower. Treat single-stream decode as a range rather than one number.

The figures quoted here until 2026-09-06 — acceptance length 2.1 of 4, per position 0.65 / 0.33 / 0.14, and "~41 tok/s" as the best case — were taken under PIECEWISE CUDA graphs with the full 248,320-token draft vocabulary. They are superseded in both directions: acceptance is much higher, and ordinary prose now measures 48.7 tok/s single-stream.

Multimodal (images and video)

The checkpoint is multimodal (is_multimodal: true, language_model_only: false, a 27-layer vision tower) and the launcher enables it by default — nothing extra to configure. The vision tower is already counted in the "weights on GPU" figure, so images and video cost no additional GPU budget.

Verified on this host 2026-09-04 against the running server:

Modality Test Result
Image 336x336 PNG, three colour bands named all three in order; 179 prompt tokens
Video 4 s clip, 16 frames, one colour per second named all four in temporal order; 376 prompt tokens

Use the standard OpenAI content-part shapes — image_url and video_url, either an http(s):// URL or a data: URI:

curl -s localhost:8888/v1/chat/completions -H 'Content-Type: application/json' -d '{
 "model":"qwen3.8-flash-next","max_tokens":600,"temperature":0,
 "messages":[{"role":"user","content":[
   {"type":"image_url","image_url":{"url":"https://example.com/photo.jpg"}},
   {"type":"text","text":"Describe this image."}]}]}'

Three things to know before leaning on it:

Configuration

Precedence is environment > .env > built-in default in start.sh, so any knob can be overridden per launch:

MAX_MODEL_LEN=65536 MTP_NUM_SPECULATIVE_TOKENS=0 ./start.sh

The safety-relevant knob is HOST_RESERVE_GIB (default 26): the GPU budget is capped at MemTotal − HOST_RESERVE_GIB no matter what KV_TARGET_GIB asks for, and start.sh prints "KV target X reduced to Y" when the cap binds. KV_TARGET_GIB is a wish under that cap: the shipped 20 is clipped to 16.67 GiB here, ~1.13M

Qwen3.8-Flash-Next-Single-DGX-Spark (MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark)

Searching Database…

609 ·GitHub
Qwen3.8-Flash-Next-Single-DGX-Spark
这是什么？一套自包含的方案，用于在单台 NVIDIA DGX Spark 机器的统一内存上部署 Qwen3.8-Flash-Next-NVFP4 检查点，使用 vLLM，并把 PLE 表卸载和内存映射。
它附带模型本身，还是只有部署配置？只有部署配置：用于针对本地 Hugging Face 缓存中已有检查点启动 vLLM 的脚本和配置。它不下载也不包含模型权重。
怎么启动和停止服务器？运行 ./start.sh 启动 vLLM 服务器（先用 --no-launch 可只打印推算出的内存预算和 docker 命令），运行 ./stop.sh 优雅地关闭（或用 --stop --force 跳过等待）。
运行之前需要准备什么？一台 DGX Spark，本地 Hugging Face 缓存中已放入 Qwen3.8-Flash-Next-NVFP4 检查点，另外还要约 130 GiB 的空闲磁盘空间，用于存放检查点和首次启动时构建的 PLE 表。
如果直接杀掉服务器而不是正常停止会怎样？容器以 --ipc host 方式运行，所以如果不让 stop.sh 等待 vLLM 解除其 POSIX 共享内存的链接，残留的内存段会泄漏并滞留在主机的 /dev/shm 上，直到下次重启。
它能在其他 GPU 或多节点集群上运行吗？不能——它是专门为单台 DGX Spark（TP=1）打造的。它源自一个双节点配置，但本仓库已没有任何东西依赖那些文件。
它能处理图像和视频，还是只有文本？它是一个视觉语言模型，所以文本、图像和视频都开箱即用，无需额外设置。
实际解码速度有多快？在优化配置下，单流解码实测 48.7 tok/s，8 路并发流时总吞吐可达 162.9 tok/s。
那个用来加速的草稿词表技巧是什么？投机解码的 MTP 起草器通常每一步都要读取完整的语言模型头；把该头裁剪到 65,536 行能减少每步读取的字节数，解码提速约 25%，且对经过验证的 token 没有精度损失。
What are similar projects?qwen3.8-Flash-DGX, DeepSeek-v4-Flash-One-DGX-Spark, GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, Qwen3.8-Flash-Next-Dual-DGX-Sparks, v100-skinny, HyperQwen, Qwen3.8-27B-DFlash2-EXL3-5.0bpw, dgx-spark-qwen38
Which topics is it in?llm-serving, speculative-decoding, vision-language-model, model-serving, agplv3, llm-inference, vllm, gpu-acceleration, qwen, quantization, multimodal, large-language-models
Creator
MiaAI-Lab
Size
0.8 MB
Stars
609
Updated
CommitOct 04
Age
1mo
Commits · 90 Days