Qwen3.8 27B on SGLang for DGX Spark
by Mia's AI Lab
Opinionated, ready-to-run scripts to serve Qwen3.8-27B with SGLang in Docker on an NVIDIA DGX Spark (GB10, aarch64). Three swap-in serving modes — EAGLE/MTP, DSpark, or DFlash2 — with every tuning choice measured on-device instead of guessed.
DSpark and DFlash2 are faster on code. Versus MTP, DSpark gives the essay back; DFlash2 does not. Everyday chat on the same streamed probe comes out a DFlash2 win once tokens are counted right, and the long-essay probe is on the MTP side of the table. All measured numbers, ranges, counting notes and caveats live in Measured on this box — one place, nothing repeated.
The launch flags start from the SGLang cookbook's DGX Spark cell (NVFP4 + DSpark), then pin choices measured on this box: GDN bf16 (cookbook float32 was −3%), extra_buffer_lazy, mem 0.90, chunk 8192, DSpark block 7 / 8 draft tokens, torch.compile + decode graphs, X5 cpuset.
- NVFP4 W4A4 checkpoint with a dense BF16
lm_head (default; packed-FP4 head via QUANT=nvfp4-fp4; full BF16 and FP8 via QUANT=…)
- native 262K context, YaRN off, and 10 concurrent requests by default — optionally extend to a validated 1M via YaRN (see “Long context & concurrency”)
- FP8 KV cache (
fp8_e4m3, ~2× KV memory savings; uses the NVFP4 checkpoint's calibration scales)
- GDN state pool sized correctly from
MAX_CONCURRENT_REQUESTS (concurrency × 4 state slots; the spec verify window is a separate engine-side buffer — verified in the build's kv_cache_configurator)
- Pinned to GB10's ten 3.9 GHz Cortex-X5 cores (
--cpuset-cpus 5-9,15-19) — the scheduler/tokenizer never land on the 2.8 GHz A725 efficiency cores (measured +2–7% decode)
- Thinking mode on by default (
--reasoning-parser qwen3 → reasoning_content) and tool calling (qwen3_coder parser)
Requirements
| Component |
Detail |
| Hardware |
NVIDIA DGX Spark / GB10 (aarch64, SM121; 128 GB unified memory) |
| Docker |
With NVIDIA Container Toolkit / GPU passthrough working (docker run --gpus all) |
| SGLang image |
lmsysorg/sglang:qwen38-27b for MTP/DSpark (model-specific build from the cookbook; multi-arch incl. arm64). lmsysorg/sglang:dev-qwen38-27b-dflash2 for DFlash2 (official multi-arch dev tag, digest-pinned and pulled by start-dflash.sh, ~14 GB) |
| CLI tools |
docker, curl |
| Hugging Face token |
HF_TOKEN defined in ~/.bashrc (picked up automatically; higher rate limits) |
There is no separate download step: the container pulls the checkpoint into ./.cache/huggingface on first start (~24 GB for the default NVFP4 BF16-head repo; the packed-FP4 twin is ~1.7 GB smaller on disk). The cookbook cites ~16.5 GB for the NVFP4 LM weights alone before the MTP head; the dense lm_head adds ~1.7 GB on disk / ~3.2 GB at runtime.
Quick start (ships as native 262K, 10 concurrent)
# 1. Copy the sample config once (creates ./.env if you don't have one)
cp .env.sample .env
# 2. Start the server
./start-dspark.sh # DSpark — code ~51.5; default chat ~23; long essay ~18
# ./start.sh # MTP — code ~34.5; default chat ~21; long essay ~24
# ./start-dflash.sh # DFlash2, NVFP4 target — code ~50.9; essay ~25.4; chat ~29–67 (streamed)
# # (bf16 base: DF_TARGET=bf16 — unbenched on this box)
# 3. Use it
curl http://127.0.0.1:8888/v1/models
# 4. Stop it
./stop.sh
.env.sample ships with YARN=0, CONTEXT_LENGTH=262144 (native) and MAX_CONCURRENT_REQUESTS=10 — so a fresh clone serves 262K context, YaRN off, 10 concurrent after just the cp above. .env is the live config (plain VAR=value lines read by start.sh): shell exports of the same names win, .env fills the gaps, start.sh defaults apply last. Changes only take effect on the next launch — ./stop.sh && ./start-dspark.sh (or ./start.sh for MTP). For anything above native context (e.g. 1M) or a different concurrency, see Long context & concurrency. Note: DSpark cannot use YaRN / context > 262144 on this build (ditto DFlash2 — same draft-config leak).
start-dflash.sh is self-contained but runs a different image: the official multi-arch lmsysorg/sglang:dev-qwen38-27b-dflash2 — the tag the SGLang cookbook maps to DGX Spark — pinned by its index digest (sha256:616a3e97…; upstream build 5f55db35e on branch dflash2-pin-1cf2b8c-nccl, 2026-08-22) and pulled from Docker Hub on first run (~14 GB compressed; no git, no local build). That image carries DFlash2 (sglang #35371) and the quantized-head selector (#35496), so both NVFP4 exports work, including the packed-FP4 lm_head (DF_TARGET=nvfp4-fp4). No released tag ships DFlash2 yet (v0.5.18 was tagged later but does not contain the commit), hence the dev tag: a CI build of an upstream dev branch, from the same publisher as qwen38-27b, with its content frozen by the digest pin; like every engine here it runs under start.sh's --privileged --network host container. First boot then pulls the ~2.6 GB draft if missing. The earlier self-built image and its patch/ builder were retired 2026-09-05 (last carried by commit 751e29e); an already-built local image still runs via IMAGE=lmsysorg/sglang:qwen38-27b-dflash2.
All start scripts are idempotent: if the container is already running they say so and exit; if a stopped container exists they remove it first. ./stop.sh stops whichever engine is up.
Scripts
| Script |
What it does |
start.sh |
Launches the SGLang container (docker run -d, host network, --shm-size 32g), streams logs to .sglang.log, records the container ID in .sglang.pid, and polls http://127.0.0.1:8888/v1/models until the server is ready. EAGLE/MTP speculative decoding (SPEC_STEPS/SPEC_TOPK/SPEC_DRAFT = 3/1/4). Monitoring on by default: Prometheus --enable-metrics + --enable-cache-report (per issue #3). |
start-dspark.sh |
Same service, DSpark instead of EAGLE: block-7 / unquant draft, torch.compile + decode-graph caps, --num-continuous-decode-steps 2, mem 0.90. Thin wrapper (EXTRA_ARGS → start.sh). Code 51.5 vs MTP 34.5 — see Measured on this box for all numbers. |
start-dflash.sh |
Same service, DFlash2 block-diffusion draft (default z-lab/Qwen3.8-27B-DFlash2@50307d4, pinned — same draft as the incoai/… mirror; DRAFT_MODEL/DRAFT_REVISION env-overridable). Default target: NVFP4 BF16-head RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead with --mem-fraction-static 0.90 (dense lm_head, so DFLASH's selector uses the native dense path). Packed-FP4 head: DF_TARGET=nvfp4-fp4 (the image's quantized-head selector handles it in place via lm_head.quant_method.apply; a dequant-once approach hard-rebooted this box). DF_TARGET=bf16 selects Qwen/Qwen3.8-27B. Pulls the official digest-pinned image on first run if missing. Measured numbers: Measured on this box. |
bench/ndec.py |
Two-call net-decode A/B (LRUCache + essay, thinking off). How the DSpark vs MTP numbers above were measured. Run twice; trust the second; treat code deltas <15% as noise. |
bench/bench.sh |
Essay / tool-call wall-time bench + 16K TTFT probe (includes prefill). Different clock from ndec.py. |
bench/ab-image.sh |
One-session comparison harness. Sides: A DFlash2 on main's self-built image, B DFlash2 on the official pinned image, M EAGLE/MTP on the stock image. A vs B isolates the image; B vs M is the DFlash2-vs-MTP engine question, and the summary prints that ratio. Boots each side, verifies the engine and selector from .sglang.log, throws away one warmup pass (cold start wrecks the first two-call delta), runs ndec.py n times, and reports per-side medians, per-boot values and pairwise deltas. SEQ="A B B A" or "M B B M" interleaves to cancel drift; MODEL_PATH= retargets every side at one checkpoint; LABEL= names the output; DRY_RUN=1 prints the plan. Artifacts land in bench/_ab-<label>-<stamp>/ (untracked). |
stop.sh |
Stops the serving engine (idempotent; also cleans up any experiment processes still alive). Leaves the stopped container in place for docker logs post-mortem. |
Runtime artifacts: .sglang.log (server log), .sglang.pid (container ID), .cache/ (HF + Triton caches). All are git-ignored.
Whitelisted for tracking: start.sh, start-dspark.sh, stop.sh, start-dflash.sh, bench/, docs/brainstorms/*.md, docs/plans/*.md, README.md, CHANGELOG.md, .env.sample, LICENSE, .gitignore. Experiment scripts and analysis docs stay untracked by design.
Which engine to use
EAGLE/MTP and DSpark are the same NVFP4 27B on the same lmsysorg/sglang:qwen38-27b image — only the speculative decoder changes. DFlash2 (start-dflash.sh) defaults to the same NVFP4 weights, on the official dev-qwen38-27b-dflash2 image (see Scripts / Configuration below) because no released tag ships DFlash2 yet. DFlash2 can also target the bf16 base (DF_TARGET=bf16) — not benched.
Performance numbers for all three engines live in the single table in Measured on this box — below is engine selection only (no repeated tok/s):
|
./start-dspark.sh (DSpark block-7) |
./start.sh (EAGLE/MTP 3/1/4) |
./start-dflash.sh (DFlash2, NVFP4) |
| Best for |
agents, code, tools, normal chat (default here) |
long-form writing |
code AND long-form writing (essay holds; chat is a win too) |
| Memory |
~24 GB BF16-head target + ~2.7 GB draft, mem 0.90 |
~24 GB BF16-head target (in-checkpoint MTP), mem 0.95 |
NVFP4 BF16-head: ~24 GB target + ~2.6 GB draft, mem 0.90; packed-FP4 ~22 GB; full bf16: 52 GB weights |
DSpark/MTP columns are live 2026-08-18 evening; DFlash2 on 2026-08-19 (n=5, same single boot, same probes) — numbers and caveats are in Measured on this box. Take-aways, with that caveat firmly in mind: DFlash2 on NVFP4 ties DSpark on code (inside the <15% noise band), beats MTP on the long essay, and beats both on every short-chat condition once counted from completion_tokens (the earlier ~9.2 “tok/s” reading was an SSE event-counting artifact — see the counting note in Measured on this box). Watchpoints: mem-fraction-static 0.95 + DFlash2 wedged the box once (hard reboot; see Logs & troubleshooting) — root-caused and fixed (see the DFlash2 bullet); the current 0.90 + 16-concurrent profile boot-validated 2026-08-19 with no reboot (concurrency ladder in Measured on this box). bf16 target is entirely unmeasured. Do not compare these to sparkDash fill-to-max streams.
Tuning history worth knowing: every Tier A (kernel-path) and Tier B (config/host) experiment measured zero net gain — these configs are the local optimum on this box. DSpark block-7 is the code peak; block-5 trades −16% code for +8% prose if you want it (DSPARK_EXTRA, see Configuration). Local-only write-ups: TIER_A_RESULTS.md, TIER_B_RESULTS.md, TIER_C_RESULTS.md, DS4F.md, KIMI.md, GROK.md, HANDOFF.md.
Configuration
Defaults live at the top of start.sh:
| Variable |
Default |
Notes |
YARN |
0 |
0 off / 1 on for CONTEXT_LENGTH > 262144; implicitly on at exactly 1000000. Factor = round(CONTEXT_LENGTH/262144) |
CONTEXT_LENGTH |
262144 |
Range 262144..1000000 (native..1M). Combined with YARN=1 for values above native; 1M auto-enables YaRN even with YARN=0. Invalid values abort at startup |
MAX_CONCURRENT_REQUESTS |
10 |
Sizes --max-mamba-cache-size = concurrency × 4 slots and passes --max-running-requests |
SPEC_STEPS / SPEC_TOPK / SPEC_DRAFT |
3 / 1 / 4 |
MTP chain drafting; topk=1 requires SPEC_DRAFT = SPEC_STEPS + 1 (validated at launch). Sweep the pair on your box and pin the winner — 3/1/4 is the measured peak here |
CHUNKED_PREFILL |
8192 |
Prefill chunk tokens. Cookbook DGX Spark cell uses 2048; we keep 8192 (prefill/TTFT, not decode tok/s). |
CPUSET |
5-9,15-19 |
Docker --cpuset-cpus pin to GB10's Cortex-X5 cores (A725s are 0-4, 10-14). Empty = no pinning |
MAMBA_SKIP_DECODE_LOCK |
0 |
1 sets SGLANG_OPT_MAMBA_SKIP_DECODE_LOCK in the container — frees one GDN state slot per request (S 4→3) |
PREFILL_CUDA_GRAPH |
0 |
1 drops --disable-prefill-cuda-graph. Info: this build auto-disables prefill graphs on this model anyway (GDN layers ≠ standard GQA) |
EXTRA_ARGS |
— |
Free-form extra SGLang flags, appended last (argparse last-wins, so they can override built-ins). The experiment hatch: EXTRA_ARGS="--fp4-gemm-runner-backend triton" ./start.sh |
QUANT |
nvfp4 |
nvfp4 → RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead (dense BF16 lm_head; cookbook-measured export), nvfp4-fp4 → RadixArk/Qwen3.8-27B-NVFP4 (packed FP4 lm_head), fp8 → Qwen/Qwen3.8-27B-FP8, bf16 → Qwen/Qwen3.8-27B. All fit in the Spark's 128 GB. |
| (shell overrides) |
— |
Any variable above can also be set as a shell env var, or put in .env |
SERVED_MODEL_NAME |
qwen3.8-27b-sglang |
Name clients use in API requests |
IMAGE |
lmsysorg/sglang:qwen38-27b |
Cookbook-pinned image for this model |
CONTAINER_NAME |
qwen3.8-27b-sglang |
Also used by stop.sh |
PORT |
8888 |
Listens on 0.0.0.0 via host networking |
The shipped .env/.env.sample match the start.sh defaults above, so a fresh clone serves native 262K context, YaRN off, 10 concurrent out of the box. Raise context above 262K with YARN=1 + CONTEXT_LENGTH (see below).
start-dspark.sh adds a couple of knobs (shell-env or .env, optional):
| Variable |
Default |
Notes |
DSPARK_EXTRA |
— |
Extra SGLang flags appended AFTER the base DSpark stack, for per-boot experiments without editing the script. E.g. DSPARK_EXTRA="--speculative-dspark-block-size 5 --speculative-num-draft-tokens 6" ./start-dspark.sh (prose-tuned block; see below) |
IMAGE |
lmsysorg/sglang:qwen38-27b |
env override (IMAGE=tag ./start-dspark.sh) to run a patched derivative image; roll back by not setting it. |
Note: --cuda-graph-max-bs is a deprecated alias in this build; the DSpark stack uses --cuda-graph-max-bs-decode 4.
start-dflash.sh knobs (shell-env; it doesn't otherwise change start.sh/.env behavior except the model-path override):
| Variable |
Default |
Notes |
DF_TARGET |
nvfp4 |
nvfp4 (default) → RadixArk/Qwen3.8-27B-NVFP4-BF16-LMHead (+--mem-fraction-static 0.90; dense head, native DFLASH selector); nvfp4-fp4 → packed-FP4 RadixArk/Qwen3.8-27B-NVFP4 (uses the image's quantized-head selector); bf16 → Qwen/Qwen3.8-27B. |
DF_EXTRA |
— |
Extra SGLang flags appended AFTER the base DFlash2 stack (last-wins). E.g. DF_EXTRA="--mem-fraction-static 0.90" ./start-dflash.sh |
IMAGE |
lmsysorg/sglang@sha256:616a3e97… (= dev-qwen38-27b-dflash2) |
The official image, pinned by its multi-arch index digest; pulled on first run and aliased locally as lmsysorg/sglang:dev-qwen38-27b-dflash2 (an existing local tag of that name is never re-pointed; the container runs by digest regardless). Override with IMAGE=<ref> — a locally present image is used as-is (e.g. IMAGE=lmsysorg/sglang:qwen38-27b-dflash2 for the retired self-built image). To bump the pin: docker buildx imagetools inspect lmsysorg/sglang:dev-qwen38-27b-dflash2 prints the index digest → update IMAGE_DIGEST in the script → re-validate on the box. If the pull ever 404s, upstream moved the tag and Docker Hub dropped the old index — re-pin the same way. |
DRAFT_MODEL / DRAFT_REVISION |
z-lab/Qwen3.8-27B-DFlash2 / 50307d4… |
Pinned DFlash2 draft (same weights as the incoai/… mirror). DRAFT_MODEL=incoai/Qwen3.8-27B-DFlash2 DRAFT_REVISION=dedf8df… reproduces the original n=5 baseline; DRAFT_REVISION='' follows a branch head. |
DFlash2-specific stack facts (so nobody re-learns them on a crash): extra_buffer_lazy was rejected by DFLASH (AssertionError) on the self-built image → the script forces --mamba-radix-cache-strategy extra_buffer (the official image includes upstream #34763, which adds lazy support for DFLASH — untested here, so the override stays); DFLASH only supports speculative_num_steps == 1 (engine auto-overrides start.sh's MTP 3); --enable-dp-attention and the overlap scheduler are off in this path; the draft (incoai/z-lab/…-DFlash2) is a block-diffusion drafter, not a token LLM, so EAGLE knobs (topk, num_steps) don't apply.
Long context & concurrency (up to 1M, 10 concurrent)
All long-context and concurrency handling is driven by three variables in .env (or as shell exports):
| Variable |
Meaning |
YARN |
1 = enable YaRN rope scaling — required for any CONTEXT_LENGTH > 262144; 0 = off (sensible only at/below 262144) |
CONTEXT_LENGTH |
desired context in tokens, range 262144..1000000 |
MAX_CONCURRENT_REQUESTS |
parallel requests; also sets --max-running-requests, and sizes the GDN pool = value × 4 slots |
Step by step — 1M context with 10 concurrent (goes above the shipped default):
cp .env.sample .env # once, if you have no .env yet
nano .env # make sure these are set:
# YARN=1
# CONTEXT_LENGTH=1000000
# MAX_CONCURRENT_REQUESTS=10
./stop.sh && ./start.sh # relaunch so new values apply
# verify after boot:
grep -E "context_len|max_running_requests" .sglang.log
expect: context_len=1000000, max_running_requests=10, mamba pool 40 slots
| You want |
YARN |
CONTEXT_LENGTH |
MAX_CONCURRENT_REQUESTS |
YaRN factor |
| 1M + 10 concurrent |
1 |
1000000 |
10 |
4.0 (also auto-on) |
| 512K + 10 concurrent |
1 |
524288 |
10 |
2.0 |
| 768K + 10 concurrent |
1 |
786432 |
10 |
3.0 |
| native 262K + 10 concurrent |
0 |
262144 |
10 |
— |
| 1M + 2 concurrent |
1 |
1000000 |
2 |
4.0 |
Rules of thumb:
- Above 262144 you must set
YARN=1 (1M is the one exception — it auto-enable
A set of opinionated, ready-to-run Docker scripts that serve the Qwen3.8-27B model through SGLang on an NVIDIA DGX Spark (GB10, aarch64), offering three swap-in speculative-decoding modes — EAGLE/MTP, DSpark, and DFlash2 — each tuned with on-device measurements rather than guesses.