DeepSeek-v4-Flash EXL3 on one DGX Spark
by Mia'a AI Lab
Single-node launcher for DeepSeek V4 Flash 0731 (EXL3/ExLlamaV3) with DSpark speculative decoding on one NVIDIA DGX Spark (GB10, SM121, 128 GiB unified memory).
Serves the 0xSero/deepseek-v4-flash-0731-spark build (3.0 bpw EXL3) via the sparkinfer (formerly b12x) kernel stack — a complete, self-contained Docker recipe tuned for speed and KV-cache headroom on a single device.
⚙️ Defaults changed (2026-08-21): the launcher start.sh serves the deep-context NVFP4 config — KV_RECORD=stock432 (native 432-byte records), GPU_MEMORY_UTILIZATION=0.94, MAX_MODEL_LEN=384000, MAX_NUM_SEQS=1, DSpark K5 healthy → 439,622-token KV pool (a 370,104-token context stress-tested with exact needle recall — see Stress test). The NVFP4 dual-cache prefill bugs behind the old either/or are fixed — full story in the internal postmortem (kept local, not in this repo).
Highlights
- One DGX Spark, tensor-parallel 1 — no second node required (unlike the official FP4 build, which needs TP2 across two Sparks).
- DSpark K5 speculative decoding with a K64 draft model (
MODE=dspark, fixed K5, K64 draft).
nvfp4_ds_mla compressed KV cache with the B12X_MLA_SPARSE attention/MoE backend.- Tuned CUDA-graph capture (
[6,12,24]) so concurrent decode stays on captured graphs instead of falling back to eager.
- Long-prefill fairness (decode-starvation guard) enabled natively.
- Two upstream kernel backports applied as read-only bind-mounts (see Backports).
- Weights download fully locally into
./hf-hub (no remote machine involved). An optional LAN mode reuses one copy of the ~107 GB across machines via an SSHFS share (REMOTE_HOST — opt-in).
Measured results
| Metric |
Value |
Decode tok/s (structured) — start.sh, 384k context |
44–47 tok/s |
| KV cache pool |
439,622 tokens (boot-dependent — see Stress test) |
| Stress test — 320k & 370k context |
exact needle recall, 0 preemptions — see Stress test |
Stress test
Two needle-in-a-haystack runs on 2026-08-21, both on fresh boots of the deep-context config (MAX_NUM_SEQS=1, KV_RECORD=stock432, util 0.94), each at ~96% of that boot's MAX_MODEL_LEN.
Method (both runs): a single large user prompt of random-word filler, generated so prefix caching cannot deduplicate it (cached_tokens: 0 — every token fresh KV); a secret passphrase planted at token ~20; a recall question at the very end; temperature 0, thinking disabled via chat_template_kwargs, max_completion_tokens 128.
|
Run 1 |
Run 2 |
MAX_MODEL_LEN |
334,000 |
384,000 (now the default) |
| KV pool (that boot) |
402,334 tokens (1.20×) |
439,622 tokens (1.14×, cold boot) |
| Prompt size |
320,037 tokens (1.33 MB) |
370,104 tokens (1.54 MB) |
| Share of pool |
~80% |
~84% |
| Needle recall from token ~20 |
✅ exact — XQ-7741-BLUE |
✅ exact — ZK-9931-AMBER |
finish_reason |
stop (clean, not truncated) |
stop (clean, not truncated) |
| Preemptions |
0 |
0 |
| End-to-end |
517 s — prefill ~630 tok/s effective |
594 s — prefill ~625 tok/s effective |
| Server after |
healthy, /health 200, KV usage back to 0% |
healthy, /health 200, KV usage back to 0% |
Prefill throughput decays with depth — ~1,024 tok/s at the start of a request, ~350–614 tok/s past 300k accumulated — so a full-length 384k prefill takes ~10 minutes end-to-end.
These runs are the deepest exercise to date of the NVFP4 dual-cache prefill path fixed in image-patch/sparkinfer/ — the pre-fix kernel NaN'd on any prompt ≥ 7 tokens; these tests pushed 320k and 370k tokens through it with exact recall.
Why the pool size differs between boots
The pool is a leftover-derived number — (util budget) − weights − profiled activation peak − non-torch overhead — and it moves between boots. Observed on this host: 337,841 (2-seq boot) → 402,334 → 430,909 → 440,461 (334k/1-seq) → 439,622 (384k/1-seq, cold). Two effects dominate:
- The hybrid cache split. The model keeps 128-token sliding-window layers alongside full-depth global layers, and the split of KV bytes shifts with
MAX_NUM_SEQS. Single-sequence boots route more of the budget into the global cache that defines the reported pool (337,841 at 2 seqs → 402k+ at 1 seq from comparable bytes).
- Cold vs. warm JIT. A boot that compiles fresh kernels during warmup leaves less memory free at KV-sizing time than a warm boot hitting the on-disk JIT caches (observed swing: ~0.6 GiB ≈ 28k tokens).
The worst boot observed still clears MAX_MODEL_LEN with ≥ 1.14× headroom; a truly bad boot trips the boot-time KV check and stops cleanly (restart: on-failure:1).
Requirements
- Hardware: one NVIDIA DGX Spark (GB10, SM121, ≥128 GiB unified memory), GPU passthrough to Docker via the NVIDIA Container Toolkit.
- OS: Linux aarch64 (DGX OS). The runtime image is aarch64-only.
- EarlyOOM: disable it if present on the host (
sudo systemctl disable --now earlyoom). The server intentionally holds ~94% of the 128 GiB unified memory, so a user-space OOM killer can't tell a healthy server from a leak — and may kill it mid-serve.
- Software: Docker Engine + Compose v2,
curl, and ~110+ GiB free local disk. sshfs + fuse3 are only needed for the optional LAN sharing mode (auto-installed with sudo when missing; requires user_allow_other in /etc/fuse.conf).
- Network: internet access to HuggingFace for the one-time ~107 GB download. The download is fully local — no remote host required. (Optional LAN mode: set
REMOTE_HOST/REMOTE_USER/REMOTE_SHARE_DIR to reuse a single copy across machines.) No HuggingFace login required; the repo and image are public. If you do hit HF rate limits or need a private repo, set the optional HF_TOKEN in start.sh (or via env / .env).
Quick start
./start.sh # start: deep-context NVFP4 (384k, DSpark) — writes compose.yml
./start.sh --no-wait # start without waiting
First boot is intentionally long: pulls the image, downloads ~107 GB of weights locally (into ./hf-hub), coalesces TP4→TP1 losslessly, builds the K64 draft, and captures CUDA graphs. It is marked healthy only when the OpenAI-compatible endpoint responds.
ℹ️ No compose file is shipped in this repo — compose.yml is generated
by start.sh on the first run and rewritten on every launch. It is
gitignored, since the real config lives in the launcher. To produce it
without starting anything: ./start.sh compose-gen. Do not hand-edit it.
Weights bootstrap — fully local
Everything downloads on this machine; no remote host is involved in the
default path:
- Default:
./start.sh auto-downloads on first boot into the local HF
cache root ./hf-hub and coalesces into ./data/tp1. The losslessly
coalesced serving checkpoint, the K64 draft, and the runtime caches all
live on this machine.
- Optional offline prep:
./download.sh performs the same
download+coalesce+verify standalone (also fully local). Once
./data/tp1/rank-sliced-tp1-manifest.json exists, boot skips the
download/coalesce step entirely, so a network-free runtime follows — the
manifest (not an env knob) is the gate.
- Optional LAN sharing: set
REMOTE_HOST (+ REMOTE_USER /
REMOTE_SHARE_DIR / MIA_MOUNT / HF_CACHE) to reuse a single weight
copy across machines instead of downloading locally — the spark3
arrangement, preserved as an opt-in mode on main and unchanged on the
mia-shared-setup branch.
Launch notes
The only hard requirement is free host RAM ≥ 114.3 GiB at launch (0.94 ×
121.63 GiB; this UMA machine shares the 121.63 GiB between GPU and host) —
stop an old container first and check free -h before launching. If a boot
ever fails (KV check or otherwise), it stops after one failure (restart: on-failure:1 — it can never death-spiral the host); lower MAX_MODEL_LEN a
notch and retry once, don't launch repeatedly while the host is loaded.
Concurrency is a single env override away: MAX_NUM_SEQS=4 ./start.sh. Note
that the pool itself shrinks at higher sequence counts (the hybrid cache split
shifts — a 2-seq boot observed ~337k total, ≈169k per slot), so depth trades
against concurrency. The rest of the launcher is the deep-context config
described in this README.
Try it
curl -sS http://127.0.0.1:8888/v1/chat/completions \
-H 'Content-Type: application/json' -d '{
"model": "deepseek-v4-flash-0731",
"messages": [{"role":"user","content":"Write a correct Python function that returns the first n Fibonacci numbers."}],
"temperature": 0, "max_completion_tokens": 256 }'
Served model name: deepseek-v4-flash-0731. API: http://127.0.0.1:8888/v1 (OpenAI-compatible chat/responses endpoints, DSpark spec decoding active).
Repository layout
| Path |
Purpose |
start.sh |
Launcher (deep-context NVFP4, 384k/1-seq, DSpark) — all tunables live here; regenerates compose.yml (do not edit that file directly) |
compose.yml |
Generated by start.sh; pinned image + mounts + runtime env |
image-patch/ |
Read-only bind-mount overrides (coalescer + kernel backports + optional ABLATE hook) |
files/ |
Bundled refusal direction for the optional ABLATE path (MIT-licensed; see files/README.md) |
data/ |
Serving checkpoint (tp1/), K64 draft, caches (on local disk) |
cache/ |
Runtime JIT/kernel caches (CuTeDSL, TileLang, TRITON, vLLM) |
Commands
| Command |
Action |
./start.sh |
start (deep-context config) + wait for /health |
./start.sh --no-wait |
start without waiting |
./start.sh mount / unmount |
manage the optional SSHFS share (no-op in local mode) |
./start.sh logs / ps / status |
inspect runtime |
./start.sh stop / restart / down |
lifecycle (preserves data/ + caches) |
./start.sh pull |
pull the pinned image now |
./start.sh selftest-ablate |
manual ABLATE overlay probe (no GPU, no serve; not on the boot path) |
./start.sh help |
usage |
Tunables (edit in start.sh)
| Variable |
Default |
Notes |
MAX_MODEL_LEN |
384000 |
~13% under the worst-observed cold-boot pool (439,622 tokens); lower it if a boot ever fails the KV check |
MAX_NUM_SEQS |
1 |
start.sh default (single deep-context request; raise for concurrency — the pool shrinks with seq count via the hybrid cache split, e.g. 2 seqs ≈ 337k total) |
MAX_NUM_BATCHED_TOKENS |
8224 |
prefill budget and b12x MLA workspace size (locked after warmup — do not lower; see below) |
GPU_MEMORY_UTILIZATION |
0.94 |
max this host boots at (see KV section) |
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS |
0 |
removes the profiler's ~0.68 GiB graph over-reservation (real usage is 0.07 GiB) → grows KV |
LONG_PREFILL_TOKEN_THRESHOLD |
1024 |
caps prefill chunks; prevents decode starvation (0=off) |
MAX_NUM_PARTIAL_PREFILLS |
0 |
wired but currently a no-op on this fork |
MAX_CUDAGRAPH_CAPTURE_SIZE |
24 |
= seqs×(k+1); 0=image default |
CUDAGRAPH_CAPTURE_SIZES |
6,12,24 |
explicit capture list; 0=image default |
HF_TOKEN |
(empty) |
optional HF token — normally not needed (repo + image are public); set it to avoid rate limits or reach a private repo. Also honored from the environment or a local .env (the set-in-file knob is in start.sh) |
SERVED_MODEL_NAME |
deepseek-v4-flash-0731 |
id clients send as model |
MODE |
dspark |
fixed K5 DSpark draft |
DSPARK_* |
— |
draft size/experts knobs |
VERIFY_MODEL_CHECKSUMS |
1 |
0 skips SAP-256 inventory |
DEFAULT_CHAT_TEMPLATE_KWARGS_THINKING |
true |
server-side default thinking (client kwargs override) |
DEFAULT_CHAT_TEMPLATE_KWARGS_EFFORT |
max |
server-side default effort (override to high/low/false) |
ABLATE |
0 |
opt-in refusal-direction ablation (details) — 1 projects a published refusal direction out of the attention output stream at runtime. No weight edits, no measurable perf cost, fully inert while off |
REMOTE_HOST / REMOTE_USER / REMOTE_SHARE_DIR |
(empty) / mia / /home/mia/shared |
LAN-share mode, opt-in — set REMOTE_HOST to reuse one weight copy across machines; defaults target the spark3 shared folder (10.0.0.1 / mia / /home/mia/shared) |
HF_CACHE |
./hf-hub (local) or $MIA_MOUNT (remote mode) |
where weights download — always resolved absolute |
SERVING_PORT |
8888 |
OpenAI-compatible port |
SERVING_HOST |
0.0.0.0 |
bind address (passed to vLLM as HOST). 0.0.0.0 = reachable from the LAN, 127.0.0.1 = local-only, or pin one interface (192.168.1.50, ::). No auth sits in front of this port — widen it only on a trusted network |
Every tunable is an environment variable override: GPU_MEMORY_UTILIZATION=0.94 ./start.sh.
MAX_NUM_BATCHED_TOKENS is load-bearing twice. vLLM sizes the profiled activation peak (and therefore leftover KV) from it, and warmup captures the largest b12x compressed-MLA scratch, then lock_workspace() freezes that size. Lowering it to recover KV can still boot and even match prefill on a cold prompt, then crash later — typically a warm prefix-cache turn that attends a large width immediately:
AssertionError: Workspace is locked but allocation from 'b12x.py:…:_run_compressed_mla'
requires … MB, current size is … MB. Workspace growth is not allowed after locking.
Leave it at 8224. Do not treat it as a free memory knob. (Reported in #4; a 2048 trial recovered ~3.9 GiB of profiled peak but then hit the locked-workspace assertion under real traffic.)
About the KV cache
Two record layouts are switchable with KV_RECORD on start.sh:
stock432 (default, fixed 2026-08-20): native 432-byte NVFP4 records → 439,622-token pool at util 0.94 on the 384k/1-seq config (370,104-token context stress-tested, DSpark acceptance 0.65–0.92 / 0.44 / 0.29–0.46 / 0.19–0.42 / 0.12). The dual-cache prefill path had four NVFP4 bugs (fixed via image-patch/sparkinfer/ bind mounts) — see the internal postmortem (kept local, not in this repo).padded: 584-byte FP8-compat records (stock semantics, ~270k pool at 256k) — the fallback.
This host reports only 114.5 GiB of 121.63 as free (the unified-memory display/desktop holds ~7 GiB), so GPU_MEMORY_UTILIZATION above **0.940 fails to boot** — the vendor's recipe value 0.9465 does not start here. The numbers below are the historical sweep of the 584-byte FP8-compat padded layout (KV_RECORD=padded); the default 432-byte NVFP4 layout (KV_RECORD=stock432) reaches ~440k tokens (439,622 observed cold on the 384k/1-seq config; see the intro). The padded-layout ceiling on this hardware:
| Config |
KV pool |
Notes |
| 0.93 (stock) |
~142k tokens |
initial |
| 0.936 + est=0 |
~165k tokens |
graph reservation reclaimed |
| 0.940 + est=0 (sweep winner) |
~181k tokens |
validated with a 130k prefill, no OOM |
For more KV, options are structural (smaller weights / lower bpw, or a 2-node TP2 stack).
Optional: refusal-direction ablation (ABLATE, off by default)
ABLATE=1 enables a runtime abliteration path: a unit-norm 4096-dim refusal direction is projected out of the attention output (wo_b) stream of decoder layers 10–42 on every forward pass — the runtime equivalent of the weight-space edit W ← W − λ·v(vᵀW) used by abliterated releases of this model family. No weights are modified: the projection is one 4096-dim matvec per layer (no measurable throughput/KV impact), fully reversible, and completely inert while ABLATE=0 — default serving stays byte-identical to stock.
Why a runtime projection instead of editing the EXL3 weights: this family's mHC residual pathway re-normalizes low-rank weight perturbations (LoRA-style abliteration is a documented dead end), and the proven edit targets the wo_b output space — which is exactly what the hook acts on. Editing the quantized trellis tensors in place would require dequant → edit → requant; projecting the output stream is algebraically identical and leaves the checkpoint untouched.
Thanks — the bundled files/direction_r1.pt is from drowzeys/keys-DeepSeekV4-Flash-GA-0731-Dspark-Abliterated-Anchored-Tensors. Thank you for publishing the refusal direction.
Setup — a default direction ships with this repo at files/direction_r1.pt (MIT-licensed; attribution in files/README.md). The first ABLATE=1 boot auto-provisions it into ./data/files/ (container path /models/files/). To use your own direction instead, drop a unit-norm 4096-dim .pt at ./data/files/direction_r1.pt or point DSV4_ABLATE_FILE at it — a pre-existing data/files/ file is never overwritten. Then:
./start.sh # applies ABLATE as set in start.sh (0 or 1)
ABLATE=1 ./start.sh # same, env override for this boot
Flip ABLATE between 0 and 1, then run ./start.sh again (not restart). Each flag change is a slow boot: start.sh wipes the inductor / AOT compile cache and force-recreates the container so the overlay actually changes. Expect the same kind of wait as a first boot (graph recapture), not a quick restart. Unchanged ABLATE on a later ./start.sh does not wipe.
| Knob |
Default |
Notes |
ABLATE |
0 |
master switch. 1 enables (auto-provisions the bundled direction into data/files/); anything other than 0/1 aborts with an error; ABLATE=1 with no direction file anywhere also aborts (no silent stock boot) |
DSV4_ABLATE_LAMBDA |
3.5 |
projection strength. drowzeys validated 3.5; lovesenko ships 2.5 (more conservative). λ ≥ 4 risks long-CoT reasoning-loop degeneration |
DSV4_ABLATE_LAYERS |
10-42 |
inclusive target-layer range. Layers 0–9 stay stock (chat/tool protocol, drowzeys' hybrid range); the DSpark draft (layers 43+) is never touched — speculation only needs the draft to propose, verification is against the target |
DSV4_ABLATE_FILE |
(auto) |
explicit in-container path override; resolves to /models/files/direction_r1.pt when ABLATE=1 |
Implementation notes:
- Lives in
image-patch/vllm/models/deepseek_v4/nvidia/model.py — the
A single-node Docker launcher that serves the DeepSeek V4 Flash 0731 model (EXL3 3.0bpw quantization) with DSpark speculative decoding, tuned to run entirely on one NVIDIA DGX Spark machine.