Qwen3.8-Flash-Next-NVFP4-vLLM-Dual-DGX-Spark
by Mia'a AI Lab
Multi-node inference for RadixArk/Qwen3.8-Flash-Next-NVFP4 across 2 DGX Sparks using vLLM with TP2+EP+MTP3.
Based on getrefined/Qwen3.8-Flash-Next-NVFP4-vLLM-DGX-Spark.
All numbers in KV cache budget and Default runtime below were read from the running
server (docker logs vllm-fn, docker inspect vllm-fn) — they are measurements, not estimates.
That container runs GPU_MEMORY_UTILIZATION=0.835, i.e. the value .env / .env.sample ship today.
Prerequisites
- 2 DGX Spark nodes (GB10, 128 GB unified memory, sm_121) connected via ConnectX RoCE/IB
- Passwordless SSH between nodes
- Docker on both nodes
126 GiB free on each node for the checkpoint. By default both nodes keep their own copy in `/.cache/huggingface; start.sh` rsyncs the worker copy from the head once. Enable NFS weight sharing to skip the worker copy entirely.
Quick Start
# 1. Edit .env (set IPs, interface, IB HCA, etc.)
cp .env.sample .env
vim .env
# 2. Download weights onto the head
./download.sh
# ./download.sh --fp8 # official FP8 instead
# ABLIT=1 ./download.sh # gated Keys house QSA L3-47 (HF_TOKEN + accept terms)
# 3. Sync to worker, apply patches, launch (NVFP4)
./start.sh --no-download
# or ./start.sh if you want start.sh to download as well
# or ./start.sh --nfs to share the head cache over NFS instead of rsyncing
# or ABLIT=1 ./start.sh --no-download
# Optional — official FP8 instead of NVFP4 (see below):
# ./download.sh --fp8 && ./start-fp8.sh --no-download
# 4. Confirm the KV cache pool that vLLM actually allocated (~11 min after launch)
docker logs vllm-fn 2>&1 | grep -E "Available KV cache memory|GPU KV cache size"
./start.sh and ./start-fp8.sh both launch containers named vllm-fn on the same port — stop first with ./stop.sh before switching checkpoints.
Flags
| Flag |
Description |
--no-download |
Skip HF download (weights already in the head cache) |
--no-launch |
Download + distribute weights only, don't start the server |
--launch |
Skip download + sync; apply patches and launch (weights already on both nodes) |
--nfs |
Distribute weights over NFS instead of rsync (see below) |
--no-nfs |
Force rsync distribution even if NFS_SHARE=true in .env |
ABLIT=1 |
Env/.env flag: serve the gated Keys house QSA L3–47 checkpoint (see Abliterated checkpoint) |
Official FP8 checkpoint (optional)
Same two-Spark launch as ./start.sh, but serve
Qwen/Qwen3.8-Flash-Next-FP8
instead of NVFP4. Cluster IPs, TP/EP/MTP, image and ports still come from .env.
Context is native 262K with YaRN off — FP8 weights leave too little KV for 1M
on this kit. Available KV cache is around 500k tokens, not the 3.65M of
the nvidia NVFP4 checkpoint with KV_CACHE_DTYPE=fp8.
./download.sh --fp8
./stop.sh # start.sh and start-fp8.sh share vllm-fn
./start-fp8.sh --no-download
# API name: qwen3.8-flash-next-fp8
This is not FP8_DENSE=true (hybrid NVFP4 experts + FP8 dense projections)
and not fp8 KV on stock NVFP4. ABLIT=1 is ignored when ./start-fp8.sh
sets OVERRIDE_MODEL_ID.
What Happens
- Download —
./download.sh pulls MODEL_ID from .env (stock NVFP4) to the head HF cache (./download.sh --fp8 for official FP8; ABLIT=1 ./download.sh for the gated Keys house checkpoint).
- Distribute — by default
rsync copies the checkpoint into the worker's own ~/.cache/huggingface/hub over the ConnectX link, skipped when the worker already has it. With NFS_SHARE=true / --nfs this is replaced by the NFS share.
- Image sync — ensures
vllm/vllm-openai:qwen38-flash-next is on both nodes
- PLE patch — extracts
ple_layer.py from the image and patches it into files/ple_layer_patched.py (no image rebuild; bind-mounted at runtime)
- MXFP8 patch — extracts
modelopt.py and patches it into files/modelopt_patched.py, routing the MXFP8 shapes FlashInfer's mm_mxfp8 cannot run to the BF16 emulation kernel (also bind-mounted)
- Preflight + overlays — refuses to launch if another process holds a GPU on either node
(
REQUIRE_IDLE_GPU=false to override); prepares the FP8_DENSE / QSA_PROFILE bind-mounts
- Launch — worker (rank 1) starts first, then head (rank 0) serves on
:8888. Both launch
scripts render the same VLLM_ARGS array (EXTRA_VLLM_ARGS really is appended last now;
ENABLE_EXPERT_PARALLEL=false and MTP_NUM_SPECULATIVE_TOKENS=0 are honored). The head's
rendered script is kept as .last_head_launch.sh for inspection.
Not done for you: dropping page caches. start.sh needs no root and does
not drop page caches. Do it yourself on both nodes before a launch —
it matters on GB10 unified memory (see Gotchas):
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
Both containers are named vllm-fn (head and worker); ./stop.sh removes both.
NFS weight sharing (optional)
Off by default. By default each node keeps its own copy of the checkpoint in
~/.cache/huggingface, and start.sh rsyncs the worker's copy from the head once (subsequent
launches detect it and skip the transfer). That costs ~126 GiB on the worker and one long copy the
first time, but the worker is then self-sufficient.
Turn NFS sharing on to skip the worker copy entirely — the head exports its HF cache and the worker
mounts it read-only:
NFS_SHARE=true ./start.sh --launch # or set NFS_SHARE=true in .env
./start.sh --nfs # same, per-run flag
./start.sh --no-nfs # force rsync even with NFS_SHARE=true in .env
A privileged vllm-fn-nfs container on the head exports $HF_CACHE_DIR over NFSv4 on the ConnectX
address (NFS_SERVER_IP, auto-detected from IFACE — do not use the 10.0.0.1 loopback
alias). The worker Docker volume vllm-fn-hf mounts it read-only at /root/.cache/huggingface.
|
rsync (default) |
NFS_SHARE=true |
| Worker disk |
~126 GiB |
none |
| First launch |
one full copy, then free |
no copy |
| Every cold start |
reads local disk |
streams ~126 GiB over ConnectX |
| Head must stay up |
only to launch |
for the whole serving run |
| Failure mode |
worker cache goes stale silently |
share dies → worker loses its weights |
Prefer NFS when worker disk is tight or you re-pull checkpoints often; prefer the default when you
want the two nodes decoupled. Switching checkpoints is where NFS earns its keep — no re-sync.
Notes when it is on:
./stop.sh leaves the share up so the next --launch does not rebuild it; ./stop.sh --nfs
tears it down and removes the worker volume. Kernel NFS in Docker can ignore SIGKILL when
rpcbind is in D-state, so that path has a 15 s timeout and reports if the container survives.- Do not stop
vllm-fn-nfs while vLLM is loading or running — the worker reads shards from it.
./check-weights.sh follows NFS_SHARE: it verifies the worker's local copy by default, or the
NFS volume when sharing is on.
.env Reference
sample = value in .env.sample · live = value in the .env on this box, which produced
every measurement in this README.
| Variable |
sample |
live |
Description |
HEAD_IP |
10.0.0.1 |
10.0.0.1 |
Head node inter-node IP |
WORKER_IP |
10.0.0.2 |
10.0.0.2 |
Worker node inter-node IP |
WORKER_USER |
(empty) |
zurih |
SSH user on worker (blank = same user) |
IFACE |
enp1s0f0np0 |
enp1s0f1np1 |
Head-side inter-node interface |
WORKER_IFACE |
(unset → IFACE) |
enp1s0f0np0 |
Worker-side interface (nodes are cross-wired) |
IB_HCA |
=rocep1s0f0 |
=rocep1s0f1 |
NCCL IB HCA (leading = = exact match, one device) |
WORKER_IB_HCA |
(unset → IB_HCA) |
=rocep1s0f0 |
Worker-side HCA (cross-wired link) |
IB_GID_INDEX |
3 |
3 |
IB GID index (3 for ConnectX + RoCE) |
MODEL_ID |
nvidia/Qwen3.8-Flash-Next-NVFP4 |
same |
HuggingFace model when ABLIT=0 (the snapshot in the head cache) |
ABLIT |
0 |
0 |
0 = stock MODEL_ID. 1 = gated Keys house QSA o_proj L3–47 (drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47). Environment wins over .env for this flag. See Abliterated checkpoint |
FP8_DENSE |
false |
false |
true → serve the hybrid NVFP4+FP8-dense checkpoint with the files/overlay loader patches (see below) |
QSA_PROFILE |
stock |
stock |
gb10 or a JSON from files/qsa_gb10/bench_qsa_kernels.py |
REQUIRE_IDLE_GPU |
true |
true |
abort if nvidia-smi shows a compute process on either node |
SERVED_MODEL_NAME |
qwen3.8-flash-next |
same |
Name in /v1/models |
MAX_MODEL_LEN |
1000000 |
1000000 |
Context length (262144 = native, no YaRN) |
YARN_ENABLE |
true |
true |
Extend context via YaRN rope scaling — auto force-disabled when MAX_MODEL_LEN ≤ 262144 |
YARN_FACTOR |
4.0 |
4.0 |
262144 × 4.0 ≈ 1M |
GPU_MEMORY_UTILIZATION |
0.835 |
0.835 |
Fraction of the 121.69 GiB budgeted by vLLM (see KV cache budget) |
MAX_NUM_SEQS |
8 |
8 |
Max concurrent sequences |
MAX_NUM_BATCHED_TOKENS |
8192 |
8192 |
Prefill chunk / cudagraph ceiling |
PORT |
8888 |
8888 |
API server port (--network host) |
KV_CACHE_DTYPE |
fp8 |
fp8 |
Default. 3.65M cache tokens (13.93× at 262K) via files/patch_qsa_fp8_kv.py, applied automatically. auto = bf16 = 2.13M (8.13×) |
TENSOR_PARALLEL_SIZE |
2 |
2 |
1 GB10 per node × 2 nodes |
ENABLE_EXPERT_PARALLEL |
true |
true |
EP for the NVFP4 experts (required) |
MTP_NUM_SPECULATIVE_TOKENS |
3 |
3 |
MTP draft tokens (0 = disable) |
PLE_OFFLOAD |
false |
false |
true → CPU-RAM offload of the 51 GB PLE table (see gotcha) |
IMAGE |
vllm/vllm-openai:qwen38-flash-next |
same |
Day-0 image |
VLLM_ALLOW_LONG_MAX_MODEL_LEN |
1 |
1 |
Required when MAX_MODEL_LEN > 262144 |
MASTER_PORT |
50000 |
50000 |
Distributed coordination port |
NFS_SHARE |
false |
false |
true → share the head HF cache over NFS instead of rsyncing a worker copy (details) |
NFS_SERVER_IP |
(unset → IFACE IPv4) |
(unset → 10.0.22.1) |
Head ConnectX address that exports the HF cache. Only used when NFS_SHARE=true. Do not use the 10.0.0.1 loopback alias |
EXTRA_VLLM_ARGS / EXTRA_DOCKER_ARGS |
unset |
unset |
Escape hatches (EXTRA_VLLM_ARGS is appended last) |
HF_TOKEN |
unset |
unset |
Required for ABLIT=1 (gated Hugging Face repo). Environment wins over .env |
KV cache dtype: the default is now fp8, which needs files/patch_qsa_fp8_kv.py —
the stock QSA kernels declare supported_kv_cache_dtypes = ["auto", "bfloat16"] and raise
Qwen3.8-Flash-Next QSA requires a BF16 main KV cache otherwise. start.sh applies the patch
automatically whenever KV_CACHE_DTYPE starts with fp8. Set KV_CACHE_DTYPE=auto for bf16.
See KV cache budget for both measured sets.
Abliterated checkpoint (ABLIT)
ABLIT=0 serves stock MODEL_ID (nvidia NVFP4 by default). ABLIT=1 serves the gated
Keys checkpoint
drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47:
the same nvidia dual-Spark NVFP4 layout with a house residual-writer projection on 12 QSA
self_attn.o_proj tensors at layers 3, 7, 11, 15, 19, 23, 27, 31, 35, 39, 43 and 47. GDN
out_proj, routed experts, MTP, PLE, vision and the chat template stay stock. It is valid
only on this recipe — do not use it with the single-Spark MXFP8 tree, official FP8, or
FP8_DENSE=true. ./start-fp8.sh and FP8_DENSE=true still win if you set them, and
ABLIT=1 is ignored for checkpoint selection in those cases.
That Hugging Face repo is gated. ABLIT=1 ./download.sh fails immediately if
HF_TOKEN is unset and prints the steps below. Set the token, Accept the terms on that
page, then download the full snapshot (same nvidia NVFP4 size class, ~133 GiB). Stock
and ablit caches sit side by side; flipping ABLIT is the only switch. start.sh rsyncs
whichever checkpoint ABLIT selected onto the worker (or shares it over NFS). An interrupted
download is not treated as ready — rerun ABLIT=1 ./download.sh to resume.
# 1. In .env: set ABLIT=1 and uncomment HF_TOKEN (or export both for this shell)
# 2. In the browser, accept the terms on:
# https://huggingface.co/drowzeys/keys-Qwen3.8-Flash-Next-NVFP4-dual-ablit-house-qsa-L3-47
HF_TOKEN=hf_... ABLIT=1 ./download.sh
ABLIT=1 ./start.sh
HF_TOKEN is required for ABLIT=1. A 403 means access has not been granted yet —
Accept the terms on that page, then retry with HF_TOKEN set. Unlike most knobs in
this launcher, ABLIT and HF_TOKEN honour the environment over .env, so
ABLIT=1 ./start.sh works even when .env still has ABLIT=0. Switch back with
ABLIT=0 ./start.sh (stock MODEL_ID; no re-download if that cache is already complete).
The gate is a binding agreement, not a download button. The checkpoint ships its own
responsible-use terms, and requesting access means accepting them. In summary: you must be
18 or older; you are accountable for harm from generated recipes or knowledge; and the
terms prohibit sexual content involving minors, material promoting self-harm or suicide,
harassment, doxxing or fraud targeting real people, anything illegal in your jurisdiction,
and any use barred by the NVIDIA Open Model License or the Qwen Community License 1.0.
The weights are provided as-is with no warranty and inherit their licence from
nvidia/Qwen3.8-Flash-Next-NVFP4. Read the repo's own terms before requesting access —
this paragraph is a summary, and those terms are what bind.
Safety refusals are removed in this checkpoint, which moves the guardrails onto you:
filtering, human review and access control are yours to supply. That matters more here
than on stock, because start.sh binds the server to 0.0.0.0 — anything that can reach
the port can reach an unfiltered model.
The abliteration splice is by Keys (drowzeys), a house projection on NVIDIA stock
(axis recovered per QSA layer from Dealign, not a Dealign weight dump). See the
checkpoint's model card, and Credits below.
The Keys config.json mislabels MTP routed experts as FP8_PB_WO; nvidia and
this snapshot's own hf_quant_config.json record FP8_BLOCK_SCALES (same 128×128
tensors). start.sh rewrites that overlay the same way it adds the mtp.layers.48
alias, so MTP 3 still works.
KV cache budget
Measured on the running container at the shipped defaults (KV_CACHE_DTYPE=fp8,
GPU_MEMORY_UTILIZATION=0.835, MAX_MODEL_LEN=262144, MTP3, TP2, nvidia NVFP4 checkpoint).
Official FP8 (./start-fp8.sh) is a different checkpoint — see
Official FP8 checkpoint; its available KV
cache is around 500k tokens, not the 3.65M below.
[gpu_worker.py:693] Available KV cache memory: 32.02 GiB
[kv_cache_utils.py] GPU KV cache size: 3,652,200 tokens,
Maximum concurrency for 262,144 tokens per request: 13.93x
FP8 vs bf16 on this kit, same GMU, same checkpoint, KV dtype the only change:
|
auto (bf16) |
fp8 (default) |
| KV pool |
31.7 GiB |
32.02 GiB |
| Cache size |
2,131,159 tok |
3,652,200 tok |
| Concurrency at 262,144 |
8.13× |
13.93× |
| Tokens per GiB |
67,229 |
114,060 (1.70×) |
| Reasoning + needle suite |
12/12 |
12/12 (no regressions) |
1.70×, not 2×, because the QSA indexer and compressor stay BF16 — only the main K/V halve.
A 1M-token context now fits 3.5× over. Weights (68.52 GiB/GPU), not KV, are the binding
constraint on this box.
Credit. The FP8-KV kernel work in files/patch_qsa_fp8_kv.py is vendored from
MiaAI-Lab/Qwen3.8-Flash-Next-Single-DGX-Spark (AGPL-3.0-or-later),
which credits the underlying approach to
lancelind/qwen3.8-Flash-DGX (Apache-2.0).
Applied here unchanged — every anchor matched this image's QSA sources.
How to read those two lines. They are the same number in two units: 32.02 GiB is the
byte budget, 3,652,200 tokens is what fits in it (~9.2 KB/token at fp8). The token figure is
the capacity of the whole block pool, shared by every concurrent request — it is not
per-request, and it is not summed over TP (TP2 shards each token across both GPUs, so ~64 GiB
of VRAM is committed while the usable token space is 3.65M once). The third figure is the
practical one: 13.93× = 13.93 resident requests at the full 262,144 context — more than
MAX_NUM_SEQS=8 can even admit, so at these defaults the scheduler, not the cache, is the
limit. At a 1M-token context (YaRN) the same pool is 3.5×.
Per-GPU memory accounting (GB10, per node — from gpu_worker.py:919):
| Line |
GiB |
| Total memory visible to CUDA |
121.69 |
| Free at startup |
108.79 |
| Budgeted at GMU 0.835 |
101.61 |
Weights (model_runner.py:407) |
64.46 |
| Weights + non-torch |
68.52 |
| Peak activation |
1.07 |
| CUDA graphs |
0.54 |
| KV cache |
32.02 |
vLLM reports headroom for --kv-cache-memory=41360080896 (38.52 GiB) if you want to push
GMU higher.
The nvidia checkpoint is 124 GiB on disk / 11 shards (NVFP4 experts, FP8 PLE n-gram table ≈ 51 GB,
bf16 attention/dense/vision). With EP + TP the per-GPU weight footprint lands at 64.46 GiB —
roughly experts 34 GB + PLE shard ~25 GB + bf16 dense ~10 GB. Weights, not KV, are what eats
this box, and that is more true at fp8 than it was at bf16.
Why the cache is so cheap: of the 48 layers only every 4th is full_attention
(full_attention_interval: 4) → 12 KV-bearing layers + 1 MTP draft layer = 13. The other
36 are Gated-DeltaNet linear attention: constant-size recurrent state per request, not per
token. With num_key_value_heads: 2, head_dim: 256:
13 layers × 2 (K,V) × 2 kv_heads × 256 × 2 B (bf16) = 26,624 B/token (= 13,312 B/GPU at TP2)
13 layers × 2 (K,V) × 2 kv_heads × 256 × 1 B (fp8) = 13,312 B/token (= 6,656 B/GPU at TP2)
At fp8 vLLM reports 32.02 GiB / 3,652,200 tokens = 9,418 B/token/GPU, ~41% above the pure
atten
一个多节点推理启动器，使用 vLLM 配合张量并行、专家并行和投机解码，在两台 NVIDIA DGX Spark 节点上运行 Qwen3.8-Flash-Next-NVFP4 语言模型。