Qwen3.8 on DGX Spark (GB10): 27B at 50 tok/s, Flash-Next 176B on one box
One command installs a boot-persistent, hardened serving stack for the Qwen3.8 family on a single DGX Spark, with seven switchable targets and zero quality loss on each (NVFP4 is the quantization floor, Qwen's own FP8 is available above it; every speculative path is lossless by construction). Since v1.8 the flash lane serves an official SGLang image for this hardware with nothing added, serves 4 concurrent requests where it used to serve one, and got 14 to 25% of its decode back from one flag (--speculative-token-map, see below):
| target |
model |
engine |
headline (measured here) |
stock (default) |
Qwen3.8-27B NVFP4 |
SGLang + DFlash2 |
50 tok/s greedy median, 148+ aggregate at 8 streams, optional 1M context |
uncensored |
Qwen3.8-27B abliterated NVFP4 |
SGLang + DFlash2 |
same speed and serving path as stock |
fp8 |
Qwen3.8-27B FP8, Qwen's own release |
SGLang + DFlash2 |
the quantization reference: 108 tok/s aggregate at 8 streams, ~92K less KV pool |
uncensored-fp8 |
Qwen3.8-27B abliterated FP8 |
SGLang + DFlash2 |
same serving path and same cost as fp8 |
flash (default of its lane) |
Qwen3.8-Flash-Next 176B hybrid MoE NVFP4 |
SGLang + NEXTN |
47.9 tok/s on code, 47.1 on math, 29-31 on prose, 27.0 ms/tok on an agent loop, prefix caching, vision, 262K on ONE box |
flash-uncensored |
the abliterated build of that same tree |
SGLang + NEXTN |
205 of 206 shards identical in size to stock, so the same flags: 45-46 on code, 0 refusals of 5 |
flash-nvda |
the same 176B from NVIDIA's mixed-precision export |
SGLang + NEXTN |
ties the stock export inside the spread, measured here |
The 27B path is the fastest configuration measured so far on GB10 (SGLang + NVFP4 + DFlash2 speculative decoding with deterministic kernels): 50 tok/s greedy median on ./bench.sh (code 41-47, reasoning 52-57, math peak 60), free prose 17-23 in any language, 135-148 tok/s aggregate at 8 concurrent streams, 258 at 32. Reproducible to the decimal across boots: see BENCHMARKS.md, "The boot lottery".
The flash path serves a model that does not otherwise fit: the 176B checkpoint's 47.7 GiB FP8 N-gram table is served from a sparse file on NVMe, read row by row by the gather kernel through GB10's host page tables, leaving the unified pool to the compute weights and a real KV cache. Until v1.7 that was a vendored patch of this repo's own; since v1.8 it is upstream (--ple-offload-backend file, sglang#37068) and the overlay is retired, along with the vendored sm_121 QSA kernel and the workaround for the GB10 MTP collapse. Prefix caching works (27k tokens re-served in 2.5 s against 12.0 s cold), decode is 47.9 tok/s on code and 47.1 on math single stream (29-31 on prose), prefill ~2,250 tok/s cold, and image input stays available. Since v1.8 it also takes --speculative-token-map, which hands the speculative draft the target's lm_head sliced to 65,536 rows instead of all 248,320: that removes 2.6 GiB from every engine step on a lane that is memory-bandwidth bound, and it is worth 14 to 25% of decode without changing what the model can say, because the target still verifies every drafted token over the whole vocabulary.
Whatever the target, you get the same surface: an OpenAI-compatible API on port 30000 (both lanes also speak the Anthropic protocol), a keepalive proxy for agent CLIs on 30001, and opencode works out of the box (the installer writes a ready-to-use provider config; the chat template ships pre-patched for agentic clients). The stack is built to grow: more targets, engines and drafters will slot into the same switch surface.
Quickstart
Requirements: DGX Spark or other GB10 machine (128 GB unified), stock DGX OS (Docker + NVIDIA container toolkit). Free disk: ~84 GB for a 27B target (39 under $HOME for checkpoints and caches, ~45 on the Docker partition for the 39 GB image; caching the other 27B targets adds ~21 GB per NVFP4 target and ~31 GB per FP8 one), **225 GB** for a flash target (~175 under $HOME: the ~126 GB checkpoint, ~124 for NVIDIA's export, plus the 47.7 GiB sparse file the N-gram table is served from and rewritten into on every boot; ~35 on the Docker partition for the 30 GB image).
One command, first install and updates alike (clones or updates ~/dgx-spark-qwen38, then runs the pinned installer):
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash
Options ride on the bash side of the pipe (an env prefix on curl would not reach the installer):
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=uncensored CONTEXT_MODE=1m bash
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=flash bash
Or the explicit way:
git clone https://github.com/hasso5703/dgx-spark-qwen38.git
cd dgx-spark-qwen38
./install.sh # image + checkpoints + systemd service, starts at every boot
./bench.sh # verify your tok/s
First boot takes ~7-9 minutes for a 27B target (CUDA graph capture + kernel compilation, cached afterwards; later boots 5-7 min) and **12-15 minutes** for a flash target, every boot: the server writes the whole 47.7 GiB N-gram table into its file each time (measured here: 12 min 21 s to /health on a fresh table). Then:
- opencode: ready config at
~/.config/qwen38/opencode.json, see "opencode integration" below
- Any OpenAI client:
http://<host>:30000/v1/chat/completions, model qwen3.8-27b (flash: qwen3.8-flash-next), Bearer key from ~/.config/qwen38/api-key
- Anthropic protocol:
http://<host>:30000/v1/messages (Authorization: Bearer only, not x-api-key)
- Don't want a systemd service?
./install.sh --no-service && ./run.sh: same config, foreground, no sudo, Ctrl+C and it's gone (27B targets; flash is service-only in this release).
- Everything is pinned twice (base image digest + checkpoint revisions at download, and the same
--revision passed to the server itself, so an upstream push to a checkpoint repo can never change what you serve; plus sha256-verified overlay files: five for DFlash2, dflash2/ATTRIBUTION.md, two for flash, flash-sglang/ATTRIBUTION.md). It still works months from now; the installer is idempotent and every failure path says how to fix itself. MODEL_REV=main ./install.sh overrides the pins; git checkout v1.1 && ./install.sh returns to the DSpark config.
- Since 2026-08-21, this same combination (DFLASH2, draft block 8) is the official recipe in the SGLang cookbook, and since 2026-08-22 there is an official multi-arch image that ships it (
lmsysorg/sglang:dev-qwen38-27b-dflash2, built from 1cf2b8c). It is not what this repo serves yet, for one reason: it was built on 2026-08-22 and the mrope fix this repo's 27B overlay carries (sglang#34446, the fused Qwen3.5 RoPE kernel discarding mrope height and width) merged on 2026-08-30, so serving it as it stands rotates every image token as if it sat at its temporal position on all three axes. The flash lane, whose overlay is entirely upstream, does serve its official image since v1.8.
Where each lane stands against upstream, checked on 2026-09-10 against the GitHub API rather than against dates: the flash lane serves 4ccff141d, which is the head of the qwen4-main-squashed preview branch its features live on, so it is not behind anything; that branch is diverged from main (995 commits behind, 11 ahead), which is why no main-based image can serve this checkpoint at all: the file-backed PLE table 176B needs to fit one GB10 (sglang#37068) was merged into the preview branch, never into main. The 27B lane is the one with something to gain: the arm64 nightly lmsysorg/sglang:nightly-cu134-20260909-708f51e (main at 708f51e44) contains all four of DFlash2 (#35371), the quantized-lm_head selector (#35496), the mrope fix that blocked the official image above (#34446) and the engine-side zombie fix (#35255, the one the v1.8.4 proxy works around), each verified as an ancestor with 0 commits behind. The blocker is therefore gone upstream; the migration is not done here, because moving a 27B pin costs a full bench and quality cycle on this box and that has not been run yet. Until it is, the 27B lane keeps its pinned image and the proxy keeps holding the line on both lanes.
Your choices, and how they combine
Everything below is optional and combinable. Variables ride on the bash side of the one-liner, flags go after bash -s --; with a clone, they go straight on ./install.sh.
| Choice |
How |
Default |
Notes |
| Model |
MODEL_CHOICE=stock, uncensored, fp8, uncensored-fp8, flash, flash-uncensored, flash-nvda |
stock |
27B NVFP4 stock or abliterated, the same pair in Qwen's FP8, or Flash-Next 176B in one of three NVFP4 exports (see the seven targets) |
| Reduced draft vocabulary |
SPEC_TOKEN_MAP_SIZE=65536, or 0 to serve without it |
65536 |
flash only: hands the speculative draft the target's lm_head sliced to that many rows, which is 14 to 25% of decode and cannot change what the model may say |
| Flash serving tier |
FLASH_TIER=context, concurrency, throughput |
context |
flash only: 4 concurrent requests and a pool that takes a full 262K prompt, 8 requests at a third of the pool, or 24 without speculation |
| Context mode (27B) |
CONTEXT_MODE=native or 1m |
native |
1m = 1,010,000 window, mem-fraction 0.70, proxy required (see the 1M section) |
| systemd service |
default, or --no-service |
service |
--no-service: foreground with ./run.sh, no sudo, 27B native only |
| Start now |
default, or --no-start |
starts |
install everything, start later with sudo systemctl start |
| opencode integration |
default, or --no-opencode |
on |
on = ready config + oc launcher + default model following every switch; off = none of that, your own opencode config is never touched. --with-opencode turns it back on |
| Ports |
PORT=, PROXY_PORT= |
30000, 30001 |
agent clients use the proxy port |
| Storage |
HF_CACHE=, PLE_DIR= |
~/.cache/huggingface, ~/flashnext-ple |
checkpoints, and the 48 GB flash PLE backing file |
| Clone location |
DIR= (one-liner only) |
~/dgx-spark-qwen38 |
must be a clone of this repo on main |
| Cockpit dashboard |
dashboard/install-dashboard.sh, DASH_PORT=, DASH_BIND= |
not installed, loopback when installed |
opt-in, never run by install.sh; installs a sudoers allowlist, see "The cockpit" |
| Agent tab (opencode in the cockpit) |
dashboard/install-agent.sh, AGENT_PORT=, AGENT_BIND=, OPENCODE_PORT= |
not installed; relay on the tailnet address when installed |
opt-in, needs the cockpit and opencode on your PATH; opencode itself stays on loopback, see "The Agent tab" |
Combinations that make sense:
# one-liner forms (variables on the bash side, flags after "bash -s --")
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash # 27B stock, native, service, opencode on
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | CONTEXT_MODE=1m bash # 27B stock, 1M context
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=uncensored CONTEXT_MODE=1m bash
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | MODEL_CHOICE=flash bash # Flash-Next lane (service only)
curl -fsSL https://raw.githubusercontent.com/hasso5703/dgx-spark-qwen38/main/get.sh | bash -s -- --no-opencode # API only, no opencode files
# clone forms
./install.sh --no-service && ./run.sh # no systemd, foreground, Ctrl+C stops it
CONTEXT_MODE=1m ./install.sh --no-start # prepare the 1M unit, start it yourself later
./install.sh --with-opencode # turn the opencode integration back on
./switch-model.sh stock | uncensored | flash # change model later, no reinstall (then stop/start the units it prints)
Re-running the installer (upgrades included) remembers what you chose: the installed model, the context mode, the port, the HF cache, and the opencode on/off choice. Pass the variable or flag again only to change something. ./uninstall.sh --list shows everything the repo put on the box before removing anything.
What speed to expect
Speculative decoding accepts predictable tokens, so speed depends on what the model generates, not on one magic number:
Two instruments, both in the box, both reproducible. The headline 50 tok/s greedy median is ./bench.sh (streaming decode rate net of TTFT, the repo's historical headline instrument: v1.0-v1.1 measured ~36-40 on it, v1.2 measures 41-57 per probe). The table below is the harsher one: the frozen battery ./bench-matrix.sh (two-call wall-clock delta, comparable across engines and boxes):
| What you generate (thinking on, battery v1) |
v1.2 (DFlash2, this repo) |
v1.1 (DSpark) |
Stable-MTP engines |
| Agentic coding (code, diffs, tool calls) |
32-40 tok/s |
28-36 |
24-28 |
| Math & structured reasoning |
41-44 |
38-42 |
24-33 |
| Technical explanations (FR) |
26 |
23-25 |
~22 |
| Free-form prose EN / FR / DE |
22 / 20 / 17 |
17 / 14 / 13 |
17-20 |
| 8 concurrent streams, aggregate |
135-148 |
100-104 |
~92 |
| 32 concurrent streams, aggregate |
258 |
not measured |
not measured |
v1.2 wins every row of the frozen battery except eval-style math (parity with stock), including free prose, historically the weak spot of block drafters. Every number above is deterministic across boots (--disable-flashinfer-autotune, see BENCHMARKS.md "The boot lottery") and was re-verified after a full machine reboot, with output-quality canaries passing. This machine serves its own opencode sessions daily on this config (stretched to the 1M preset from the field report below): if something breaks, it breaks here first.
Quality, measured (not claimed). Same box, v1.2.1, thinking on:
| Quality check |
Result |
| GSM8K, 200 problems |
94.0% (188/200), exact parity with the DSpark profile |
| IFEval, 200 prompts |
81.4% prompt-level / 87.4% instruction-level |
| tool-eval-bench, 69 scenarios |
91/100 (Excellent), reproducible to the point across seeds |
| Independent users on this config |
92-94/100 tool-calling (forum thread) |
| Losslessness |
token-identity study vs the pure model in BENCHMARKS.md ("The losslessness study") |
Full study (methodology, engine-vs-engine matrix, an independent reproduction, the physics of the GB10 ceiling, and a frozen benchmark battery you can run against any engine, ./bench-matrix.sh): in BENCHMARKS.md.
⚠️ The GB10 unified-memory trap (read this before changing anything)
SGLang's memory accounting does not see 25-40 GB of transient allocations on GB10 unified memory (the flashinfer fp8 autotuner and CUDA graph capture allocate outside the tracked pool). Running --mem-fraction-static above 0.50, or running SGLang natively (outside Docker), can drive host available memory to zero: on a machine where SSH often rides on the same memory, that means a hard freeze only a power cycle fixes. We learned this the hard way.
This repo's service is safe by construction:
- Docker hard caps:
--memory 100g --memory-swap 100g (a runaway kills the container, never the host; note the cgroup does not see CUDA unified allocations, so the real guard is the fraction)
--mem-fraction-static 0.50 (plenty for 262K context at batch ≤ 4)Restart=always + a clean ExecStartPre docker rm -f so even a power cut leaves nothing stale (always and not on-failure: a Triton compile crash measured on 2026-08-22 ended in SystemExit: 0, which on-failure never relaunches)
The 1m mode deliberately runs 0.70 inside the same docker caps, with the autotuner
disabled: field-tested continuously on the reference box (~17 GiB host headroom). 0.80 was
measured crashing under 3 concurrent requests (2 GiB free, Triton CUDA operation not permitted), and the 25-40 GB invisible-allocation bursts above all belong to native runs and
the autotuner. Treat anything past 0.70 as livelock territory.
The SGLang cookbook pins 0.80 on DGX Spark, and that is not a contradiction.
Its GB10 cells ran 48 configurations at ISL 8192 / OSL 1024, concurrency 1,
boot-and-serve only, and 0.80 served every cell on every attempt; it rejects 0.85
because 0.85 of 128 GB leaves about 8 GB for the OS, exactly DGX OS earlyoom's
SIGTERM threshold, and 15 of 48 cells were killed there (exit -15, no traceback,
visible in journalctl -u earlyoom). This repo's 0.80 failure was measured under
three concurrent requests on a box that also runs the operator's tools. One
number is a single-stream boot-and-serve bound, the other is a multi-client
operating point, and this repo optimises for the second. If you serve one stream
on a dedicated box, the cookbook's 0.80 is the better-evidenced pin.
opencode integration
The installer writes a complete, ready-to-use opencode config at ~/.config/qwen38/opencode.json (the API key is referenced via {file:...}, no secret inside). It contains one provider per installed engine (qwen38 for the 27B pair, flashnext for flash), each with low / medium / xhigh reasoning-effort variants (no variant = the template's own default, xhigh), and its default model follows the installed target (./switch-model.sh re-points it on every switch):
# no opencode config yet? use it as-is:
mkdir -p ~/.config/opencode && cp ~/.config/qwen38/opencode.json ~/.config/opencode/opencode.json
# already have one? merge the "qwen38" (and/or "flashnext") provider block into it
opencode
Do not want any of it? ./install.sh --no-opencode (one-liner: | bash -s -- --no-opencode) installs the API only: no generated config, no oc launcher, and switch-model.sh never touches your opencode default model. The choice is remembered by later runs (marker ~/.config/qwen38/opencode.off); ./install.sh --with-opencode turns it back on. Your own ~/.config/opencode/opencode.json is never rewritten in either mode: the installer only merges the served lane's limits into it when the integration is on.
What the shipped config gets right for you:
- The hidden 32K output cap: opencode sends
max_tokens = min(limit.output, OPENCODE_EXPERIMENTAL_OUTPUT_TOKEN_MAX or 32000). Without that env var, a long thinking phase hits 32000 tokens, the turn ends silently (finish_reason: length, no text, no tool call) and you have to re-prompt. The installer ships an oc launcher (~/.local/bin/oc, skipped if an unrelated oc binary exists) that exports the right value and execs opencode --yolo: launch with oc instead of opencode and the cap sits above every output limit this repo declares, on every target and in either context mode. It is deliberately a ceiling rather than the installed target's own number, because limit.output is what follows a model switch: a ceiling copied fro
Установщик одной командой, разворачивающий укреплённый, сохраняющийся между перезагрузками стек обслуживания SGLang для семейства моделей Qwen3.8 на одном NVIDIA DGX Spark (GB10): медиана 50 tok/s в greedy-режиме и контекст до 1M на конфигурации 27B NVFP4, при этом модель 176B Flash-Next MoE тоже помещается на одну машину.