Qwen3.8-27B on one RTX 3090
Serving setup for Qwen3.8-27B on a
single 24 GB consumer GPU with vLLM — 150k token context and an OpenAI-compatible
API with key auth, in two ready-made modes.
Quick start
The image is prebuilt and pushed to
ghcr.io
on every commit — the build applies all patches/ and runs verify.sh as its
gate, so latest is always the current stack. The first start pulls it (9.5 GB),
downloads and requantizes the model (~20 GB, once, into ./models), and serves
on port 18020. Pick a mode — one GPU serves one at a time:
git clone https://github.com/syv-ai/qwen38-27b-rtx3090 && cd qwen38-27b-rtx3090
cp .env.example .env # Linux / WSL
# PowerShell: Copy-Item .env.example .env
docker compose --profile single up -d # one or a few people chatting
docker compose --profile batch up -d # API backend, many concurrent requests
The example uses the recommended single-user SPEC=dflash2 profile. If Docker
Desktop is using WSL2, keep VLLM_WSL2_ENABLE_PIN_MEMORY=1 enabled in .env or
the V2 runner will abort with RuntimeError: UVA is not available. The example
leaves API-key authentication disabled for local-only use; set VLLM_API_KEY
before exposing the server beyond this machine.
|
--profile batch → batch/ |
--profile single → single-user/ |
| for |
API backends, pipelines, many concurrent requests |
one or a few people chatting |
| aggregate, 64 concurrent (128 in / 512 out) |
~1,035 tok/s steady-state decode, 948 end-to-end (~1,222 / 1,042 with all layers int8) |
n/a (8 slots) |
| single-stream (C1) decode rate, realistic prompts |
46 tok/s |
MTP: 121 tok/s at default sampling, 120 greedy (CTX=fast, 64k; 96 / 102 with CTX=long, 150k). DFlash2 (SPEC=dflash2): 127 default, 130 greedy |
| reproducing its own context (quoting a document, applying an edit) |
46 tok/s |
381 tok/s at 25k context — 15.0 tokens per verify step, drafted straight from the prompt (SPEC=dflash2 + DFLASH_TOKENS=15) |
| trick |
16-bit recurrent state + int8 tensor-core GEMMs |
MTP speculation with 4 cheap drafts, a draft vocabulary that covers what the model says, calibrated int4 lm_head/drafter, split-KV verify attention; optionally native vLLM 0.28.0 DFlash2 (7 drafts in one pass, int4-requantized) with a verify block the context fills |
| Single-stream numbers re-measured 2026-08-22 on current main with |
|
|
bash bench/run_benchmarks.sh single — vllm bench serve, the 8 prompts in |
|
|
bench/prompts_real.jsonl, 1024 output tokens, C1, decode rate taken as |
|
|
C / mean TPOT. Quote them against that harness: a client with a different output |
|
|
| length is not measuring the same thing, and mixing the two is how |
|
|
| #3 got confusing. |
|
|
Version note: this branch pins vLLM 0.28.0; the throughput and quality tables are
retained as reference baselines while the v0.28.0 GPU matrix is being re-measured.
Both modes share one install — the mode is just which launch script you run.
Speculation wins below ~8 concurrent users on short prompts, plain batching above;
on long independent sessions the crossover is much earlier, because a speculating
request reserves recurrent-state pages the pool has few of — the concurrency
paragraph under "DFlash2 at 240k" has the measurement. Numbers are vllm bench serve on an
RTX 3090 at a 250 W power limit. If the card is yours alone, the fastest
configuration is three environment variables away:
If you are the only user.
Prefill is a separate budget from either: 1,810 tok/s at 1k inputs in batch
mode, and in single-user mode ~1,440 tok/s stock or **1,850-1,940 with
INT8_ACT=int8** (1,423 at 51k in), measured on the seeded benchmark protocol
(full matrix; older published prefill rows came
from an unseeded harness that let the prefix cache contaminate the numbers,
and are not comparable). How each number was won:
docs/optimizations.md.
The server listens on 0.0.0.0 and is unauthenticated unless you give it a key.
For anything past your own machine, add one first — everything reads it from
.env or api_key.txt, and nothing needs it otherwise:
echo "VLLM_API_KEY=$(openssl rand -hex 24)" > .env
No compose, no clone — plain Docker runs the same image with one command and
prepares the model itself on the first boot (into a named volume, so it
survives container replacement):
docker run -d --name qwen --gpus all --ipc=host -p 18020:18020 \
-v qwen-models:/app/models -v qwen-cache:/cache \
--restart unless-stopped ghcr.io/syv-ai/qwen38-27b-rtx3090:latest
batch after the image name is the other mode, and the knobs compose reads
from .env become -e flags (-e VLLM_API_KEY=..., -e SPEC=dflash2, ...) —
docs/docker.md has the mapping.
Or by hand in a venv (same steps: model download, requantization, vLLM
patches, verify.sh) — see Setup.
If you are the only user, do this
The command above starts the conservative default — MTP speculation, 8 request
slots, 64k context, 120 tok/s greedy at C1. Two settings are worth more than
every other knob in this repo put together, and a third is worth a great deal on
one particular workload:
printf 'SPEC=dflash2\nPREFIX_CACHE=1\n' >> .env
# add DFLASH_TOKENS=15 if your answers quote your prompts — see below
docker compose --profile single up -d
or, in the venv install:
venv/bin/python prepare/fetch_dflash2.py # once, 1.2 GB (Docker's prepare step does it for you)
SPEC=dflash2 PREFIX_CACHE=1 bash single-user/start_qwen.sh
SPEC=dflash2 swaps Qwen's MTP head for the DFlash2 block drafter: 7 tokens
proposed in one pass instead of 4 chained ones. DFLASH_TOKENS=15 then lets the
target verify 16 tokens per step — the drafter still proposes the 7 it was
trained for, and the remaining positions are filled from the request's own
context, which costs nothing to draft and is exactly right whenever the answer
quotes the prompt. PREFIX_CACHE=1 keeps the document you already sent, both
its attention KV and its recurrent state. One request at a time, greedy, RTX
3090 at 250 W:
| decode |
MTP (default) |
SPEC=dflash2 |
+ DFLASH_TOKENS=15 |
| 8 real chat prompts |
118 tok/s |
132 |
133 |
| reproducing a 25k-token document |
n/a* |
260 |
382 |
| request slots / context |
8 / 64k |
8 / 64k |
4 / 56k |
VLLM_DFLASH2_CHAIN=1 adds drafter-free n-gram chains on top
(#38, ported from
@Dmtrii-tesla's fork with permission): while a request keeps reproducing its
context, whole verify blocks come from history alone and the drafter's forward
and graph replay are skipped until the first rejected token — +7% on the copy
cell here (256.9 → 276 tok/s at DFLASH_TOKENS=7), flat on prose, greedy
requests only by default (patches/dflash2-ngram-chains.patch explains why
sampling keeps the drafter). Off by default.
* drafting from the context only exists in SPEC=dflash2. The two right
columns are one server session, where run-to-run greedy divergence is ±3-5%;
reproduce them with venv/bin/python bench/labd_bench.py <tag> --ctx 20000.
PREFIX_CACHE=1 is orthogonal to the other two and worth as much again in a
chat client: a second turn against that same 25k-token document takes 0.56 s to
first token instead of 22.4 s, with the answers unchanged token for token.
Read that table by column, not by its last cell. SPEC=dflash2 is the upgrade
for everyone; DFLASH_TOKENS=15 is for one workload. On chat it is worth 1%,
because the eight positions past the drafter's own block are filled from the
prompt and a chat answer does not quote the prompt — measured over
bench/prompts_real.jsonl, positions 7-14 take 72 of 11,069 accepted tokens
(0.65%), and @changtimwu measured exactly zero for them on a TP=2 box in
#22. What you pay for
that 1% is half the request slots and 8k of context, because a 16-token verify
block doubles the recurrent-state page every resident request holds (1.66 GiB
against 0.88 by the gotcha-33 fit). So: set it if you are quoting documents or
applying edits, where it is worth 47%, and leave it at the default 7 for a chat
or agentic client. DRAFT_TOKENS/DFLASH_TOKENS is one variable you can flip
per service.
All of it is lossless: speculative decoding samples the same distribution as no
speculation at all, the prefix cache resumes recurrent state rather than
approximating it, and GSM8K reads 96.0-96.5% across the three columns.
SPEC=dflash2 is a one-user mode either way
(see concurrency).
Every other knob: single-user/.
DFlash2 at 240k: CTX=huge (KVarN) also combines with SPEC=dflash2
bash kvarn/install.sh # applies the v0.28.0 KVarN + V2-runner ports
SPEC=dflash2 CTX=huge PREFIX_CACHE=1 bash single-user/start_qwen.sh
Where CTX=long doubles the DFlash2 pool with int8 KV (138k), the KVarN cache
takes the same idea further: 268k tokens of pool at 245760 max-model-len, on the
same pinned budget. No kernel work — the KVarN Triton kernels run unmodified on
the V2 runner; the seven fixes in kvarn/kvarn-v2-runner-0.28.0.patch are allocator and
geometry logic (the patch header walks through them, including an upstream vLLM
bug in the mamba align resume path, and a NaN path in the DFlash2 candidate
selector that KVarN noise exposes on verbatim-reproduction content). Two
machines, both RTX 3090 at 250 W, bench/labd_bench.py --ctx 20000 — the
contributor's WSL2 box and this repo's bare-metal one, which do not agree on
decode rate and do agree on everything else:
SPEC=dflash2 CTX=huge PREFIX_CACHE=1 |
WSL2 |
bare metal |
| copy (reproduction) |
130 tok/s, 7.8 tok/step |
164 tok/s, 7.83 tok/step |
| code / edit / quote / summary / qa |
89 / 65 / 44 / 38 / 36 |
109 / 83 / 58 / 51 / 43 |
| all six tasks together |
53 tok/s, 3.0 tok/step |
67 tok/s, 3.15 tok/step |
| verbatim reproduction, 25k document |
correct |
1,150 / 1,150 chars |
| KV capacity at 245760 max-model-len |
268,169 tokens |
268,169 tokens |
| GSM8K exact-match (thinking off) |
97.0% (n=200) |
95.2% (n=600), 95.0% (n=200) |
| 100k-deep needle, both turns |
correct |
— |
| turn 2 over a 100k cached prefix |
4.7 s (vs 169 s cold) |
— |
Context for the GSM8K column: every configuration this repo already ships
reads 95.0-96.5% on the same 200-question harness (docs/quality.md),
and 95.0% is the batch-mode default. 95.2% at n=600 (±0.9 points) therefore sits
inside the band rather than below it — which is the useful comparison, since this
mode inherits KVarN's lossy 4/2-bit cache and should be judged against the other
lossy configurations rather than against bf16. Repeat runs of the reproduction
check on bare metal are bit-identical (same step count, same 1,150 characters),
which is the property that was missing before PIECEWISE — see below.
One caveat to the "all of it is lossless" paragraph above: the speculation here
is still exact, but this mode inherits KVarN's 4/2-bit KV cache, which is lossy —
the same trade CTX=huge already makes (deep-needle retrieval passes at 200k).
The WSL2 column's ~20% deficit is a WSL2 tax, not a Windows tax, and leaving
WSL for native Windows does not recover it: the same contributor ran
aivrar/vllm-windows-build (0.27.1, 18 of 19 patches apply after a CRLF→LF
pass) on the same box and measured native Windows slower than WSL2 — 66.0
vs 76.2 tok/s across the task mix, a 4.4× longer warm boot, and the same
WDDM paging behavior underneath (#25).
The bare-metal column is reachable from a Windows box only by putting Linux
on the metal.
On WSL2, every SPEC=dflash2 profile needs VLLM_WSL2_ENABLE_PIN_MEMORY=1 —
not just CTX=huge. The drafter's architecture forces vLLM's V2 model runner
(_is_dflash2_draft() in config/vllm.py), the V2 runner allocates UVA buffers
before the weights load, and vLLM leaves pinned memory off by default under WSL2,
so a clean venv aborts with RuntimeError: UVA is not available before it prints
anything model-shaped. Those buffers work fine on the paravirt driver. Note the
name: VLLM_WSL_PIN_MEMORY is not a vLLM variable and setting it does
nothing — this README named it for 22 minutes on 2026-08-21 (589daae, fixed in
27f51fa), so a tree cloned in that window will have it.
On WSL2 the usable dedicated memory is about half a gigabyte less than the
same card on bare metal, and the shipped SPEC=dflash2 boot sits about 50 MiB
under it. Anything larger (a wider verify block, a bigger drafter, a raised
pin, a boot that recompiles a graph) runs two to six times slower instead of
failing, and the log does not say so; nvidia-smi looks the same either way.
Gotcha 58 has the counters to read, the four costs that turned out to be this,
and the profile that gives it room (KV_MEM=3000000000 DFLASH_MAX_LEN=8192,
free for the shipped head at width 7).
Running a DSpark drafter. vLLM 0.28.0 can serve RadixArk/Qwen3.8-27B-DSpark
(bf16, seven drafts per step like the shipped head) once two things are in
place: patches/dspark-draft-quant-config.patch (the loader refuses a bf16
drafter beside the quantized target without it), and a copy of the checkpoint
whose config.json names the architecture Qwen3DSparkModel instead of
DSparkDraftModel (the registry maps the published name to the DeepSeek V4
class; the weights are unchanged, so hard-link the safetensors). Then
DRAFT=/path/to/that/copy DRAFT_METHOD=dspark KV_MEM=3000000000 DFLASH_MAX_LEN=8192 SPEC=dflash2 CTX=fast bash single-user/start_qwen.sh. It
serves, and loses to the shipped head on the same requests: 3.19 against 3.77
tokens per step (accepted drafts plus the bonus token) and 115 against 160 tok/s on a 4090, 3.37 against 3.83
and 114 against 150 on a 3090 (issue #25, items 15 and 16). Documented so nobody
re-derives the two errors, not as a recommendation.
One knob this mode used to set for you, and now sets only for MTP:
cudagraph_mode=PIECEWISE. Prefix caching and a captured (FULL) verify step
did not mix on this path. On WSL2 that showed up as acceptance collapsing to
about one token per step; on bare metal it also corrupted the output —
special-token ids leaking into the stream, 1 of 1,176 characters matching the
source instead of all of them. It is the capture rather than the drafter: eager
is clean, LOOKUP=0 is not, forcing a fixed verify-block length is not, and
PIECEWISE — which keeps the compiled graphs and leaves only the multi-query
verify uncaptured — restored both the speed and the correctness on both machines.
dflash2 has its FULL graphs back (a75ee4b fixed the residue, b356e31 then
swept all 128 residues under FULL with 0 broken), so at HEAD
SPEC=dflash2 CTX=huge PREFIX_CACHE=1 runs captured. SPEC=mtp CTX=huge
still forces PIECEWISE, and that one is a correctness constraint rather than a
preference: under FULL it breaks at one prompt length in 128 and nobody has
fixed it. Either way prefix caching stays on, which is what the mode is for —
turn 2 over a cached 100k document costs 4.7 s against 169 s cold.
CUDAGRAPH_MODE=FULL_AND_PIECEWISE overrides the MTP line for anyone hunting
the root cause. Treat that as unsafe rather than merely slower.
What that trade costs, re-measured at HEAD. The numbers this README used to carry here
had FULL_AND_PIECEWISE at 38 tok/s (1.97 per step) on the 25k copy task against
PIECEWISE's 132, and called it 3.5x. That was not the capture mode — it was the residue
bug, which a75ee4b fixed. With the same server and only the capture toggled
(bench/labd_bench.py --ctx 20000, SPEC=dflash2 CTX=huge PREFIX_CACHE=1, decode tok/s):
|
copy |
code |
edit |
quote |
summary |
qa |
all six |
| FULL (the default now) |
167.1 |
111.1 |
84.7 |
55.0 |
47.8 |
43.4 |
65.7 (3.03/step) |
| PIECEWISE |
166.3 |
111.3 |
83.0 |
62.4 |
48.6 |
43.1 |
67.6 (3.18/step) |
They are the same. Five of the six are within 2%; quote differs by 13% in PIECEWISE's
favour, which is greedy divergence on the task that diverges most, and it is what puts
PIECEWISE 3% ahead overall. So at this context length the capture mode is not a
performance decision at all, in either direction.
Short prompts are where a difference was measured, and that measurement is older:
78/125/202 tok/s captured against 74/102/176 piecewise on de/en/code, i.e. 13-18%.
Treat that as an upper bound — @mjungnickel18 measures 0.2-2.3% for the same comparison
when only runs with identical step counts are compared, and he is right that greedy runs
which take a different number of steps are not comparable. Past 8k the two are within
noise on bare metal (111.8 vs 109.3 tok/s at 8k, 78.2 vs 86.1 at 16k, 68.9 vs 73.3 at
32k, 58.4 vs 56.0 at 50k, unique prompts, one server per mode). Under GPU passthrough on
a VM the same comparison costs 2-3x, reported in
#13 and consistent with the
uncaptured verify being launch-bound: launches that are nearly free here are not free
there.
Two limits worth knowing before you point this at anything: what it does with
more than one user, and what the long verify block costs.
It is a one-stream mode, and the limit is the pool rather than MAX_SEQS. An
earlier version of this paragraph said the knob was the seat count and that
MAX_SEQS=8 lifts it. It does not, and @mjungnickel18 was right to push back in
#25. A resident request
reserves 1+k = 8 recurrent-state slots — 15.8% of the 69,758-token CTX=fast pool,
~0.82 GiB of its pinned 5.20, which is the 0.88 GiB gotcha 33 fitted
from the memory model — before it holds one token of context. Seven fit with 128-token
prompts, five with 4k-token ones and two with 16k ones. MTP's k=4 costs 0.44 GiB, so
eight fit, four of them at 16k. Past that the extras queue, and once the pool is full
something has to be preempted and recomputed to make room — a ramp of tiny requests
hits that at the seventh. MAX_SEQS decides how many requests are admitted, not how
many can run. The pool itself is unchanged by the setting (268,169 tokens at
CTX=huge either way, ~8 MiB total between 1 slot and 8 —
gotcha 33), which is the part of the old paragraph that was right.
What concurrency actually costs, measured with bench/conc_ladder.py on distinct
4k-token prompts — each salted so nothing is served out of the prefix cache — 256-token
answers, MAX_SEQS=8, CTX=fast, 250 W:
| streams |
1 |
2 |
4 |
8 |
| dflash2 per-stream decode tok/s |
137 |
97 |
46 |
33 |
| dflash2 aggregate decode tok/s |
137 |
225 |
309 |
5 resident, no steady state |
| dflash2 ms per forward pass |
25.9 |
32.8 |
49.1 |
— |
| mtp per-stream decode tok/s |
126 |
103 |
46 |
23 |
| mtp aggregate decode tok/s |
124 |
212 |
280 |
383 |
| mtp ms per forward pass |
24.8 |
29.8 |
43.1 |
62.3 |
Two things to read off it. The verify step does batch — aggregate throughput keeps
climbing for both speculators, and no preemption happens at any of these points — so
"the block verify does not batch" is not wha
HyperQwen — сетап сервинга для собственного хостинга, который запускает модель Qwen3.8-27B на одной потребительской GPU с 24 ГБ памяти (вроде RTX 3090) средствами vLLM, выставляя её через OpenAI-совместимый API.