twinspark — DeepSeek-V4-Flash-0731 (284B MoE) on 2× NVIDIA DGX Spark
Production-grade recipe: self-healing 2-node vLLM cluster, reboot-verified, tuned DSpark speculative decoding (~75 tok/s), full benchmarks, and OpenAI Codex CLI integration. Snapshot as of 2026-08-11 — versions are pinned; expect drift.
TL;DR
- Model:
deepseek-ai/DeepSeek-V4-Flash-0731 — 284B MoE (13B active), native FP4 experts + FP8 attention, 1M context, MIT license. 167 GB of safetensors.
- Hardware: 2× DGX Spark (GB10, 121 GiB unified memory each, 4 TB NVMe), linked by a single 400G QSFP DAC cable → shows up as 2× 200GbE RoCE links.
- Engine: vLLM 0.25.2 (community image
ghcr.io/anemll/dspark-vllm-gx10:0.1.1 with GB10/sm_121 kernels: FlashInfer-B12X MoE, native FP8 DS-MLA KV cache, DSpark speculative decoding) + Ray for 2-node TP=2.
- Results: ~75 tok/s single-stream decode (DSpark speculative decoding = 2.74× over the 27 tok/s bandwidth floor), ~1.7k tok/s prefill, 1M context enabled, KV capacity 2.68M tokens. Full benchmark tables below.
- Daily driver: OpenAI Codex CLI pointed at the local endpoint (
wire_api = "responses") — agentic coding evaluated on 3 greenfield projects plus one real feature landed in a 1,831-file codebase (results below).
Quick start
Assumes both Sparks are through DGX OS first-boot, the QSFP DAC is plugged (before power-on!), and you can SSH between them.
# 0. on BOTH nodes: RDMA prerequisites (memlock etc.) — see §2 gotchas #1-2
# 1. on BOTH nodes: install the netplan file (swap IPs on node 2)
sudo cp netplan/99-qsfp.yaml /etc/netplan/ && sudo chmod 600 /etc/netplan/99-qsfp.yaml && sudo netplan apply
# 2. verify the fabric BEFORE touching any model (do not skip):
scripts/preflight.sh # links at 200000Mb/s, MTU 9000, RoCE ACTIVE
# then nccl-tests all_reduce_perf across nodes — expect ~20 GB/s bus BW (§1)
# 3. get the weights on node 1, rsync to node 2 over the QSFP link
hf download deepseek-ai/DeepSeek-V4-Flash-0731 --local-dir ~/models/DeepSeek-V4-Flash-0731
# 4. build the serving image on BOTH nodes (adds Ray to the anemll image):
docker build -t ds0731-vllm:ray - <<'DF'
FROM ghcr.io/anemll/dspark-vllm-gx10:0.1.1
RUN pip install --no-cache-dir "ray[default]"
DF
# 5. edit scripts/twinspark.env (your IPs/usernames), then from node 1:
scripts/cluster-start.sh # ~8 min to serving
scripts/status.sh
curl -s localhost:8000/v1/models | jq -r '.data[0].id' # -> ds-0731
1. Hardware & Topology
┌────────── LAN switch (management, 2.5GbE) ──────────┐
│ │
spark1 (192.168.1.10) spark2 (192.168.1.11)
│ ConnectX-7 ═══ 400G QSFP DAC (0.5 m) ═══ ConnectX-7
│ 10.0.0.1 / 10.0.1.1 10.0.0.2 / 10.0.1.2
└────── RoCE v2, MTU 9000, NCCL over 2 rails ─────┘
Each DGX Spark: GB10 (Grace 20-core ARM + Blackwell GPU, sm_121), 128 GB unified LPDDR5x (121 GiB usable), ~273 GB/s memory bandwidth, 4 TB NVMe, DGX OS 7.2.3 (Ubuntu 24.04 arm64), CUDA 13.0, driver 580.173.02.
How two Sparks serve one model (TP=2, not load balancing)
A common misconception: "requests get load-balanced between the two Sparks." No — both machines compute every single token together. The model is 167 GB of weights; one Spark has 121 GiB. It physically cannot fit on one node, so vLLM shards it with tensor parallelism (TP=2): half the attention heads and half the MoE experts live on each machine.
your clients (Codex agents, curl, ...)
│ all traffic → node 1, port 8000 — the ONLY endpoint
▼
spark1: vLLM API server + scheduler
│ continuous batching: concurrent requests merge into one batch
▼
every forward pass:
spark1 GPU ──┐ each computes its half of every layer
├── NCCL allreduce per layer (over the 2× 200G rails)
spark2 GPU ──┘
▼
each request receives its own tokens
Consequences worth internalizing:
- spark2 is invisible to clients. There is nothing to route, balance, or failover between. One endpoint, period.
- Both GPUs sit at ~95% utilization even with a single request — that's not waste, that's each node computing its half.
- The QSFP link is on the critical path of every token. Each transformer layer does an allreduce across the cable, which is why fabric quality (§1) matters more than any vLLM flag: if NCCL silently falls back to TCP, decode collapses.
- Multiple agents are "balanced" in the batch dimension, not across nodes. The scheduler on node 1 merges concurrent requests into one batch; each forward pass advances all of them by one token. Aggregate throughput goes up with concurrency (weights are read once per pass regardless of batch size — see §5 addendum).
Data parallelism (one full model replica per node + a round-robin proxy in front) is the setup where "load balancing" would apply — that only works for models that fit on a single node, which this one does not.
Discovery #1: one cable, two NICs
The single QSFP cage is internally wired to two independent ConnectX-7 controllers. Plugging one 400G DAC lights up two 200G links (enp1s0f0np0 + enP2p1s0f0np0, RDMA rocep1s0f0 + roceP2p1s0f0). Each controller sits on a PCIe Gen5 x4 link (~126 Gb/s usable), so per-rail RDMA tops out at ~109 Gb/s (ib_write_bw, 87% of the PCIe ceiling — the NIC is not the bottleneck, the PCIe lane is).
Give both rails to NCCL:
export NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
export NCCL_NET_GDR_LEVEL=5
export NCCL_SOCKET_IFNAME=enP7s7 # bootstrap over the management LAN
Measured cross-node all_reduce_perf (nccl-tests, NCCL 2.30.7): 21.7 GB/s bus bandwidth peak at 1 GB messages, 17.8 GB/s average. If you see ~2 GB/s, NCCL silently fell back to TCP — fix your RDMA before loading any model.
Discovery #2: the CX-7 is power-gated
With no cable plugged, DGX OS powers the CX-7 down and removes it from the PCI bus entirely (dmesg shows Detected insufficient power on the PCIe slot (27W) then the devices vanish). Don't panic when lspci shows nothing — plug the cable before boot (or echo 1 > /sys/bus/pci/rescan).
Netplan (/etc/netplan/99-qsfp.yaml, mirrored with .1/.2 swapped)
network:
version: 2
ethernets:
enp1s0f0np0: { addresses: [10.0.0.1/24], mtu: 9000, dhcp4: false, dhcp6: false, optional: true }
enP2p1s0f0np0: { addresses: [10.0.1.1/24], mtu: 9000, dhcp4: false, dhcp6: false, optional: true }
MTU 9000 verified end-to-end (ping -M do -s 8972), RoCE active_mtu goes 1024 → 4096.
2. Gotchas that cost us real time (read this before you start)
| # |
Symptom |
Root cause |
Fix |
| 1 |
NCCL unhandled system error, ncclIbRegMrDmaBufInternal -> 2 |
default memlock ulimit is 8 MB; RDMA can't register memory |
* soft/hard memlock unlimited in /etc/security/limits.d/99-rdma.conf + DefaultLimitMEMLOCK=infinity for systemd + --ulimit memlock=-1 on every container |
| 2 |
limits.d fix "doesn't work" |
Tailscale SSH bypasses pam_limits — sessions through tailscale up --ssh keep the old 8 MB |
test via OpenSSH (ssh localhost), not Tailscale SSH |
| 3 |
Ray worker dies during model load, killed due to memory pressure (OOM) while 117 GiB is free |
DGX Spark has unified memory — vLLM's "GPU" allocation counts as host RAM; Ray's memory monitor kills the worker |
RAY_memory_monitor_refresh_ms=0 on both nodes |
| 4 |
Model loads, /v1/completions is perfect, but chat output is nondeterministic token salad with random Chinese |
DSpark speculative decoding: rejected draft tokens leave dirty KV context; plus vLLM auto-enables breakable CUDA graphs |
VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1 and VLLM_USE_BREAKABLE_CUDAGRAPH=0 |
| 5 |
--speculative-config JSON mangled (Value {method:dspark} cannot be converted) |
quoting through nested bash -lc "..." layers |
never inline the launch command — ship a script file into the container and exec it |
| 6 |
vLLM never starts, log file absent |
pkill -f 'vllm serve' matches the wrapper shell's own command line and kills itself |
pkill -f 'bin/vllm' |
| 7 |
Official vllm/vllm-openai:cu130-nightly can't load the model |
it's pinned to vLLM 0.19.x — predates DeepseekV4ForCausalLM |
community image ghcr.io/anemll/dspark-vllm-gx10:0.1.1 (vLLM 0.25.2 + sm_121 kernels) |
| 8 |
vLLM stable arm64 images crash on GB10 |
bundled torch compiled through sm_120 only; GB10 is sm_121 |
same as #7 |
| 9 |
Codex CLI: wire_api = "chat" rejected |
removed in codex 0.147 |
wire_api = "responses" — vLLM 0.25 ships /v1/responses |
| 10 |
Codex warns Model metadata for ds-0731 not found |
unknown model → fallback metadata |
undocumented model_catalog_json key + catalog JSON (schema below) |
| 11 |
After restarting the head container, it hangs forever "waiting for 2 GPUs" |
Ray head restart = new GCS session; the worker's old raylet never re-registers, yet ray status from the worker still succeeds against the new GCS → naive watchdogs never fire |
worker watchdog must verify its own IP is in ray.nodes() alive list; param-change restarts bounce worker first, then head |
| 12 |
Agent sessions degrade at ~100–140k context: the model narrates intentions in a loop ("Let me grep… Let me view…") or repeats identical tool calls, while the server is perfectly healthy |
Model-side, not your rig: DeepSeek's own tech report shows retrieval stable only to 128K; past it, self-reinforcing in-context repetition takes over. Reproduced on official DashScope serving (qwen-code #4695 — 43 identical tool calls while the model outputs "Let me stop this loop") |
break the loop with /compact; prevent it with auto-compaction. The fence depends on KV precision (§5 Addendum 2): with native fp8_ds_mla KV → 150k (model_auto_compact_token_limit = 150000); with nvfp4_ds_mla → 100–120k. 1M context is real for reading, not for accumulated agent history |
| 13 |
After ~29h of stable serving, the whole engine crash-loops: RayWorkerProc rank=[1] died unexpectedly on the head, nothing in docker logs, container exits code 0 |
Kernel OOM on the worker node. Ray sets its own workers to oom_score_adj=1000 — first in line for the kernel OOM killer. If the worker node does double duty (agents, git checkouts, anything spiky), one memory spike kills the model worker, and TP=2 means the whole engine dies with it. Verify with `dmesg |
grep -i oom` on the worker node |
| 14 |
Long-context requests randomly collapse from ~100k — code-salad, XML repetition loops, or “this is a prompt injection test” refusals — while identical requests sometimes pass, and short requests are always perfect |
V4’s sparse attention picks top-512 compressed KV blocks per query via a small FP4 “lightning indexer”; nvfp4_ds_mla KV quantization adds noise to exactly the representations that indexer scores. Wrong blocks → the answer is literally invisible, or format-heavy blocks lock generation into loops. Stochastic and content-dependent by nature |
use the native --kv-cache-dtype fp8_ds_mla: same speed (64.1 vs 63.9 tok/s), −2% KV capacity, 4× fewer real long-context failures (measured — §5 Addendum 2) |
3. Software stack
| Layer |
Version |
| DGX OS |
7.2.3 (Ubuntu 24.04.4, kernel 6.17.0-1029-nvidia, aarch64) |
| Driver / CUDA |
580.173.02 / 13.0 |
| Container |
ghcr.io/anemll/dspark-vllm-gx10:0.1.1 + pip install ray[default] (ray 2.57.0), rebuilt as ds0731-vllm:ray |
| vLLM |
0.25.2.dev0+g752a3a504 (DeepseekV4ForCausalLM, DSpark, fp8_ds_mla KV (native — see gotcha #14), flashinfer_b12x MoE) |
| NCCL |
2.30.7+cuda13.3 |
| Model |
deepseek-ai/DeepSeek-V4-Flash-0731 (48 shards, 167 GB, native FP4/FP8 — this is the highest precision that exists; don't waste time on GGUF re-quants) |
| Codex CLI |
0.147.0 (aarch64-musl static binary) |
Weights are transferred to node 2 over the QSFP link (rsync @ ~440 MB/s, ssh-crypto-bound, 380 s) — don't download twice.
4. Architecture: self-healing containers
Both machines run one container each with --restart unless-stopped. The container entrypoints implement the full recovery logic, so a reboot of either or both machines auto-recovers to a serving state with zero manual steps (~8 min to ready):
- spark1 /
ds0731-head: scripts/boot-head.sh → start Ray head → wait until cluster has 2 GPUs → exec launch-vllm.sh (vLLM as PID 1; if it crashes, the container restarts and the whole sequence re-runs).
- spark2 /
ds0731-worker: scripts/boot-worker.sh → join Ray head (retry loop) → watchdog: if head disappears, exit → docker restarts the container → rejoin fresh.
- All launch parameters live in one file on the host (
launch/launch-vllm.sh, bind-mounted). Change params → restart-vllm.sh (= docker restart ds0731-head; worker follows automatically).
Final launch configuration (launch/launch-vllm.sh)
vllm serve /models/DeepSeek-V4-Flash-0731 \
--served-model-name ds-0731 \
--host 0.0.0.0 --port 8000 \
--tensor-parallel-size 2 \
--distributed-executor-backend ray \
--trust-remote-code \
--tokenizer-mode deepseek_v4 \
--tool-call-parser deepseek_v4 \
--reasoning-parser deepseek_v4 \
--enable-auto-tool-choice \
--kv-cache-dtype fp8_ds_mla \
--moe-backend flashinfer_b12x \
--block-size 256 \
--max-model-len 1048576 \
--gpu-memory-utilization 0.85 \
--max-num-seqs 6 \
--max-num-batched-tokens 8192 \
--max-cudagraph-capture-size 24 \
--generation-config vllm \
--async-scheduling \
--enable-chunked-prefill \
--enable-prefix-caching \
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
Container environment (both nodes)
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0 NCCL_NET_GDR_LEVEL=5
NCCL_SOCKET_IFNAME=enP7s7 GLOO_SOCKET_IFNAME=enP7s7
RAY_memory_monitor_refresh_ms=0 RAY_DISABLE_MEMORY_MONITOR=1
VLLM_USE_FLASHINFER_SAMPLER=1 VLLM_USE_B12X_MOE=1
VLLM_USE_B12X_WO_PROJECTION=1 VLLM_DSPARK_GPU_REJECTED_CONTEXT_MASK=1
VLLM_USE_BREAKABLE_CUDAGRAPH=0 DEFAULT_THINKING=low
HF_HUB_OFFLINE=1
docker: --network host --ipc=host --ulimit memlock=-1 --ulimit stack=67108864
--cap-add=IPC_LOCK --device=/dev/infiniband --gpus all
Memory budget after load: model 2× 79.2 GiB, KV cache 20.5 + 18.4 GiB → 2,682,310 tokens of KV capacity (fp8_ds_mla — the native format. We initially ran nvfp4_ds_mla: it buys only ~2% more capacity and quadruples long-context failures, see gotcha #14 / §5 Addendum 2).
5. Speculative decoding tuning
DSpark = DeepSeek's built-in MTP draft module (the extra ~20B params in the 304B checkpoint). Acceptance rate measured ~75% over 5 draft positions (pos0 93%, declining to ~60% at pos4).
| config |
single-stream tok/s (mean of 3) |
4-way concurrent agg |
7k-prompt TTFT |
prefill tok/s |
| spec decode OFF |
27.3 (27.3/27.3/27.3 — the pure bandwidth floor) |
71.3 |
4.08 s |
1745 |
| num_speculative_tokens=3 |
64.0 |
100.2 |
4.12 s |
1728 |
| num_speculative_tokens=5 (recipe default) |
70.7 |
64.1 |
4.7 s |
1517 |
| num_speculative_tokens=7 ← winner |
74.8 (74.4–75.2) |
82.6 |
4.26 s |
1674 |
Takeaways:
- DSpark = 2.74× single-stream speedup (27.3 → 74.8 tok/s). The no-spec floor of 27.3 tok/s is exactly what the memory-bandwidth math predicts for 13B active @ ~4.4 bit over 2× 273 GB/s with per-layer all-reduce over the QSFP link.
- Acceptance stays high enough (~75% over 5 positions) that deeper drafts keep paying: 7 beats 5 beats 3 for a single stream.
- Speculative decoding inverts at concurrency: at 4 concurrent streams, spec3 wins (100 tok/s agg) and spec5/7 lose to plain decoding's batching. Tune for your workload — we serve 1–2 Codex streams, so 7 it is.
Addendum: multi-agent production tuning (measured after a day of real use)
Running 2+ Codex agents against the endpoint revealed two things the clean-prompt numbers hide:
- Agentic traffic is prefill-dominated. Every tool round re-submits a large context; we observed ~8k tok/s of prompt processing sustained while decode sat at ~51 tok/s aggregate. Prefix caching absorbs most of the cost, but prefill still competes with decode under chunked prefill.
- DSpark acceptance drops from ~75% to ~40% on agent contexts — draft models predict clean prose/code well, but not tool output (file listings, test logs). Deep drafts waste compute exactly when you're busiest.
So we ship two launch profiles (launch/):
| profile |
config |
single-stream |
4-way agg |
prefill |
7k TTFT |
launch-vllm.sh (default) |
spec7, seqs 6 |
74.8 tok/s |
82.6 |
1674 |
4.26 s |
launch-vllm-multiagent.sh |
spec3, seqs 8 |
66.3 |
105.7 |
1953 |
3.65 s |
Swap = copy over launch-vllm.sh + run scripts/restart-vllm.sh (~8 min).
Winner: num_speculative_tokens=7 → baked into launch/launch-vllm.sh.
Addendum 2: the long-context minefield, measured (and why KV precision is the fuse)
We probed needle-retrieval reliability at growing context sizes: non-guessable codes planted at multiple depths in a synthetic log, 3-needle exact-match queries, temperature 0, 5 trials per size. “Real failure” = wrong/missing/refused answer; “junk” = correct answer with trailing garbage.
| context |
nvfp4_ds_mla KV |
fp8_ds_mla KV (native) |
| 74k |
0/5 |
0/5 |
| 92k |
0/5 |
— |
| 110k |
1/5 |
0/5 (2 junk) |
| 129k |
2/5 |
0/5 (3 junk) |
| 148k |
1/5 |
0/5 (1 junk) |
| 166k |
2/5 |
2/5 |
| 184k |
3/5 |
0/5 (1 junk) |
| real failures total |
8/25 |
2/25 |
What this says:
- There is no hard “128K cliff” — it’s a stochastic minefield. The same request can pass or collapse on different runs. That matches the architecture: the sparse-attention “lightning indexer” (FP4, top-512 block selection) is a small network whose scoring gets noisy as the haystack grows; near-tie scores flip block selection run to run.
- 4-bit KV quantization is an amplifier, not a bystander. Switching to the native
fp8_ds_mla cut real failures 4× at identical speed and −2% capacity. The failure pocket at ~166k persists on both — that one looks model-side.
- Real-agent burn-in (fp8 KV): a Claude Code session stayed fully coherent through 163k of accumulated tool-call history, then survived its own compaction cycle (~184k → 141k) and kept working — zero loops. On nvfp4 KV we saw agent loops from ~100k in daily use.
Practical fences: with fp8_ds_mla → auto-compact at ~150k (codex model_auto_compact_token_limit = 150000; Claude Code CLAUDE_CODE_MAX_CONTEXT_TOKENS=160000 compacts ≈148k). With nvfp4_ds_mla → stay at 100–120k. Either way, fence below the ~166k pocket.
Claude Code works against this stack directly, by the way — vLLM 0.25 serves an Anthropic-compatible /v1/messages, so ANTHROPIC_BASE_URL=http://<node1>:8000 + ANTHROPIC_MODEL=ds-0731 is all it takes.
6. Full benchmark (vllm bench serve, random dataset)
| scenario | conc | out
This is a production-grade deployment recipe for running DeepSeek-V4-Flash-0731 (a 284B-parameter MoE model) across two networked NVIDIA DGX Spark machines, using a self-healing 2-node vLLM cluster tuned for speculative decoding.