Qwen3.8-27B MTP: the flag was free the whole time
One llama.cpp flag unlocks +33% to +145% decode speed for Qwen3.8-27B on consumer GPUs, depending on the card, and the knobs the community mapped push further still. No new files, no conversion, no custom build. The MTP head already ships inside the GGUF you downloaded on launch night.
Opened hours after the Aug 14 2026 release. Within four days the community grew it into a living record: 53 configurations, 40 contributors, a decade of silicon from 2016 Pascal to Blackwell workstation cards plus Apple Silicon and an unlocked mining card, three quant makers, and seven tuning rules nobody knew at launch, see Community numbers.
The flag
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1
Full launch command
llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
-c 131072 -ngl 999 -fa 1 \
--cache-type-k q4_0 --cache-type-v q4_0 \
--spec-type draft-mtp --spec-draft-n-max 2 --parallel 1
The KV cache flags matter on their own: without them, context creation fails past roughly 90K next to 17GB of weights. With them, the full 262K window fits a 24GB card at 22.2GB (drop -c to 262144 and remove the spec flags if you want maximum window instead of maximum speed).
Weights: unsloth/Qwen3.8-27B-GGUF. Official model: Qwen/Qwen3.8-27B.
Tuning quickstart
Start with the launch command above, --spec-draft-n-max 2. Then sweep n-max 2-4 for your card, and on bandwidth-poor rigs try --spec-draft-p-min 0.60-0.75, per rules 1 and 2 below. Every contributor's full sweep lives in sweeps/.
Discovered by contributors in the table across the first three days, detailed in the sections and footnotes below:
- The n-max sweet spot is card- and topology-dependent. 24GB cards peak at n-max 2, bigger or faster cards at 3-4, and switching split mode moves it too, so re-sweep after any config change (@lingster, @Jackwwg83).
--spec-draft-p-min helps starved cards and hurts fast ones. The ~0.60 confidence gate makes deep drafting nearly free on bandwidth-poor rigs (@tomertec), and inverts on desktop Blackwell, where three independent RTX 5090s ran fastest ungated. Acceptance is a vanity metric there: gating raised it and lowered throughput (@taco-devs, @paulomcg, @jcr211). Sweep it, don't adopt it.- The gain scales with generation length where overhead dominates. Short generations can pay more than they win (@Spadav); on rigs whose baseline is already bandwidth-bound the full gain shows at 400 tokens and length adds nothing (@Jackwwg83).
- On multi-GPU boxes, fix the split mode before touching spec flags. The default
--split-mode layer serializes decode; tensor was +68% on its own on a 5060 Ti pair, and the two levers stack to 3.14x (@Jackwwg83).
- Speculative decode is a single-stream optimization. The advantage is gone by
--parallel 4, and a --parallel 2 BASELINE reads ~20% low, which inflates your gain claim, so measure both arms at --parallel 1 (@Jackwwg83, @paulomcg).
- Rebuild llama.cpp before you tune anything, and re-derive whole flag stacks per card class. A config that pays on a bandwidth-starved APU inverted below baseline on a 960 GB/s card (@Splizard). Upstream is optimizing this arch weekly: a current build was +10-15% on every quant before any flag (@taco-devs), and a fresh 3090 baseline now equals the day-one with-flag number (@hauntedhost).
- A shared desktop halves everything, silently. A live compositor and browser spilled 3.5 GB of weights to host RAM over PCIe while
/health stayed green; decode halved with no error anywhere. Bench headless, or verify the weights are actually resident (mem_info_gtt_used or your vendor's equivalent) before trusting a number (@Splizard).
How it works
Qwen trained multi-token-prediction (nextn) layers into Qwen3.8. The quantizers kept them: unsloth's GGUFs carry the blk.*.nextn.* tensors, which llama.cpp loads and, without the flag, ignores. llama.cpp added draft-mtp speculative decoding in PR #22673 (July 2026): the server drafts tokens with the built-in head and verifies them with the main model, so accepted drafts cost a fraction of a full forward pass. Everything was in place on release night. The flag connects it.
Caveats
--parallel 1 for measurement, always: some hosts do serve with higher parallel, but the spec advantage is gone by 4 concurrent streams, and a parallel-2 baseline corrupts the A/B (see rule 5)- prompt processing takes a small hit from device-to-host embedding transfers
- these are day-one llama.cpp speeds through the qwen3_5 code path, the hybrid attention kernels are young and the floor should rise with upstream work
- your absolute numbers will differ with hardware, drivers, and thermals, the deltas are the durable part
Measure it yourself
probe.py is the streaming client behind every number here. It clocks every generated token (reasoning and content deltas both) against a live server and prints per-prompt medians.
python3 probe.py # defaults to http://127.0.0.1:8080
python3 probe.py http://127.0.0.1:8090
Run it once against a baseline serve and once with the flag, same everything otherwise. That pairing is the whole method.
Ran the A/B on your card? Open a PR and add a row.
| Card |
Baseline |
With flag |
n-max |
Acceptance |
Contributor |
| RTX 3090 24GB |
31.0 |
41.3 |
2 |
0.78 |
@sudoingX |
| RTX 5090 mobile 24GB |
36.7 |
50.9 |
2 |
0.79 |
@sudoingX |
| RTX 4090 24GB |
47.7 |
76.3 |
2 |
0.56 |
@Spadav_ |
| RTX A6000 48GB (Ada) |
26.7 |
52.5 |
2 |
0.54-0.98 |
@lingster |
| RX 7900 XTX 24GB |
30.7 |
43.9 |
2 |
0.60-0.95 |
@Jqianggu |
| 2× RTX 3090 + 3090 Ti 24GB (TP) |
49.1 |
81.1 |
2 |
0.52-0.96 |
@guilhermedemelocabral |
| RTX 4090 24GB (UD-Q4_K_XL) |
36.1 |
74.8 |
2 |
0.56-0.94 |
@rkvhtd |
| 2x RX 9070 16GB (Vulkan) |
22.1 |
41.6 |
2 |
0.73 |
@tomertec |
| AMD Radeon AI PRO R9700 32GB |
27.0 |
43.3 |
2 |
0.60-0.94 |
@ajnytebot |
| Ryzen AI Max+ 395 / Radeon 8060S |
11.5 |
23.7 |
2 |
0.52-0.94 |
@shiwuxiu |
| AMD Radeon 890M iGPU (Strix Point) 48GB UMA |
2.7 |
5.7 |
2 |
0.59-0.91 |
@davidglogan |
| RTX PRO 6000 Blackwell 96GB |
63.8 |
91.5 |
2 |
0.72-0.81 |
@commdata2338 |
| RTX 5090 32GB (desktop) |
61.4 |
135.0 |
4 |
0.38-0.87 |
@taco-devs |
| RTX 5090 32GB (Q6_K, 128K) |
61.9 |
130.0 |
2 |
0.52-0.95 |
@hypertectonic |
| RTX 5090 32GB (Q6_K, 256K) |
62.0 |
121.7 |
2 |
0.50-0.95 |
@hypertectonic |
| RTX 3090 Ti 24GB (Q4_K_M, 128K) |
42.0 |
60.9 |
2 |
0.47-0.93 |
@hypertectonic |
| RTX 3090 Ti 24GB (Q4_K_M, 256K) |
41.2 |
61.4 |
2 |
0.55-0.94 |
@hypertectonic |
| RTX 5090 32GB desktop (UD-Q5_K_XL) |
66.3 |
144.2 |
4 |
0.32-0.89 |
@TrickRiggin |
| 3× RTX 3060 12GB (layer split) |
17.3 |
24.5 |
8 |
0.87 |
@EamonMcKiernan05 |
| AMD Radeon 9060 XT 16 GB |
15.2 |
28.7 |
2 |
0.62-0.93 |
@kdrapel |
| RTX 3090 24GB (turboquant, n-max 6) |
39.8 |
61.5 |
6 |
0.61-0.90 |
@NicholaiVogel |
| RTX 3090 24GB (b10450) |
41.3 |
63.5 |
2 |
0.69-0.87 |
@hauntedhost |
| GMK EVO-X2, Ryzen AI Max+ 395 (64GB unified, ROCm/HIP) |
10.5-11.1 |
21.4-22.2 |
12 |
0.95-1.0 |
@KyaniteLabs |
2× RTX 5060 Ti 16GB (PP, default -sm layer) |
22.1 |
42.8 |
2 |
0.53-0.94 |
@Jackwwg83 |
2× RTX 5060 Ti 16GB (TP, -sm tensor) |
37.1 |
65.9 |
2 |
0.51-0.88 |
@Jackwwg83 |
| RTX 5090 32GB (desktop) |
62.7 |
108.7 |
3 |
0.72 |
@jcr211 |
| RTX 5090 32GB (desktop) |
69.3 |
129.1 |
4 |
0.55 |
@paulomcg |
| RX 7900 GRE 16GB (Vulkan, packed) |
28.7 * |
47.8 avg (36.6–53.8) |
3 |
0.87–0.96 (avg ~0.93) |
@lsunay (Hermes agent on PC-12) |
| RTX 5090 32GB (Q4_K_M, 131K) |
76.9 |
155.5 |
2 |
0.50-0.95 |
@anstaendig |
| RTX 5090 32GB (UD_Q4_K_XL, 262K) |
74.3 |
179.7 |
4 |
0.35–0.91 |
@anstaendig |
| RTX PRO 6000 Blackwell Max-Q 96GB |
45.7 |
97.1 |
2 |
0.52-0.95 |
@awilliamson |
| 2× Tesla P40 24GB (tensor split) |
11.7 |
22.6 |
4 |
0.68-0.87 |
@lyesrock |
| RTX 5090 32GB (UD-Q5_K_XL, 262K) |
69.6 |
147.5 |
6 |
0.69-0.87 |
@lyesrock |
| 2× RTX 5060 Ti 16GB (TP, Q4_K_M) |
38.3 |
76.0 |
3 |
0.43-0.93 |
@mgoswick |
| RTX 5090 32GB (UD-Q4_K_XL, 262K, q4_0 KV) |
76.3 |
171.7 |
4 |
0.69-0.87 |
@lyesrock |
| 2× Tesla P40 24GB (UD-Q4_K_XL, 262K, q4_0 KV) |
13.2 |
23.4 |
4 |
0.82 |
@lyesrock |
| RX 7900 XTX 24GB (Vulkan/RADV) |
28.8 |
70.7 |
3 |
0.43-0.95 |
@Splizard |
| RTX 3090 24GB (UD-Q2_K_XL, OC) |
52.4 |
85.6 |
2 |
0.76 |
@dcrey7 |
| RTX 3090 24GB (UD-Q4_K_XL, OC) |
43.9 |
79.6 |
2 |
0.78 |
@dcrey7 |
RTX 3060 Ti 8GB + RTX 3080 Ti 12GB (TP, -sm tensor) |
39.4 |
49.2 |
2 |
0.895 |
@randyjohnson11 |
| Apple M4 24GB (Metal) |
5.8 |
5.8 |
2 |
0.48-0.95 |
@sternryan |
| Mac Studio (M3 Ultra, 96GB UMA, Q6_K, Metal) |
22.8 |
24.2 |
2 |
0.53-0.95 |
@adityavsingh |
| RTX 3090 24GB (Q4_K_M, 131K, 250W) |
31.9 |
52.8 |
4 |
0.65 |
@ctaylor83 |
| RTX 5090 32GB (UD-Q4_K_XL, 131K, q4_0 KV) |
74.4 |
182.0 |
4 |
0.38-0.90 |
@cmoro-deusto |
| RTX 5080 16GB + RTX 3090 24GB (TP) |
55.6 |
92.9 |
2 |
0.51-0.95 |
@plyra |
| RTX 5060 Laptop 8GB |
4.2 |
7.3 |
2 |
0.75-0.97 (0.90 aggregate) |
@ENY66n |
| RTX PRO 3000 Blackwell Laptop 12GB |
5.3 |
10.8 |
2 |
0.67-0.93 |
@ENY66n |
| RTX 3080 10GB (UD-IQ2_XXS, 16K, headless) |
45.1 |
64.4 |
2 |
0.44-0.93 |
@ntindle |
| RTX 3080 Ti Laptop 16GB (UD-Q3_K_XL) |
14.8 |
28.6 |
2 |
0.76 |
@salibam99 |
| NVIDIA CMP 170HX 64GB (unlocked HBM2e) |
33.0 |
46.7 |
2 |
0.53-0.94 |
@shiwuxiu |
| RTX 4090 24GB |
47.0 |
68.8 |
2 |
— |
@snakeyjay63-png |
| RTX 4090 24GB (Purusha, LoRA merge) |
47.0 |
83.4 |
2 |
— |
@snakeyjay63-png |
| Tesla V100 32GB (Volta) |
33.3 |
48.0 |
2 |
0.75 |
@cameron |
| 2× Tesla V100-SXM2-16GB (Volta, NVLink, layer split) |
35.7 |
59.1 |
4 |
0.33-0.92 |
@ATIVX928 |
| RTX 5060 Ti 16GB (Q4-XYZ-v2, 32K) |
26.3 |
59.5 |
4 |
0.34-0.69 |
@jaisusx |
| RX 7900 XTX 24GB (Windows/Vulkan) |
41.0 |
85.4 |
3 |
0.47-0.90 (0.73) |
@pparuzel |
| 2× RTX A5000 24GB (NVLink, TP) |
51.9 |
81.1 |
2 |
0.56-0.96 |
@TheRiotCoder |
| Ryzen AI Max+ 395 / Radeon 8060S 64GB (Linux, Vulkan/RADV) |
11.9 |
28.7 |
4 |
0.30-0.92 |
@Nathanw1014 |
| RTX 5090 32GB (UD-Q4_K_XL Dynamic 3.0, 192K) |
74.7 |
160.8 |
4 |
0.86-0.93 |
@paulomcg |
| RTX 5080 16GB |
53.4 |
101.3 |
2 |
0.53-0.95 |
@ChumBoxBaron |
| RTX 4060 Ti 16GB (Q4-XYZ-v2, 32K) |
17.6 |
40.0 |
3 |
0.41-0.94 |
@CeIest2 |
| RX 7900 XTX 24GB (Linux/ROCm 10, UD-Q4_K_M, 131K) |
36.3 |
62.6 |
2 |
0.56-0.94 (0.80) |
@vijay-14 |
| RTX 5090 32GB (UD-Q4_K_XL Dynamic 3.0, 131K, b10680) |
71.9 |
170.3 |
7 |
0.19-0.85 (0.48 aggregate) |
@hagope |
| RTX 2070 8GB + RTX 5060 Ti 16GB (mismatched Turing+Blackwell, PCIe x1 riser, TP)* |
25.4 |
41.5 |
2 |
0.81 (0.50-0.96) |
@Lucas12807 |
| AMD Radeon 780M iGPU (Phoenix, 32GB UMA) |
4.1 |
8.4 |
2 |
0.50-0.92 (0.77 aggregate) |
@ob7282 |
| AMD Radeon AI PRO R9700 32GB (Windows/Vulkan) |
29.7 |
56.3 |
3 |
0.39-0.97 (0.72) |
@misterkerns |
| 2× RTX 5090 32GB (TP, UD-Q4_K_M) |
107.5 |
191.9 |
3 |
0.46-0.93 (0.72 aggregate) |
@Enigmatic331 |
| AMD Instinct MI210 64GB (CDNA2/gfx90a, Q8_0, 32K, f16 KV, custom build) |
30.4 |
46.8 |
8 |
0.375 aggregate |
@pestopoppa |
| RTX 3060 12GB + Tesla P100 16GB (tensor 1,1, PCIe) |
19.3 |
31.0 |
2 |
0.79 |
@eightman999 |
* A6000 row: unsloth Q8_K_XL, 256K context, q8_0 KV cache — 40.0 GB VRAM baseline, 41.4 GB with spec (rows above: Q4_K_M, 131K, q4_0 KV).
* RX 7900 XTX row: unsloth Q4_K_M, 131K context, q4_0 KV cache — 18.9 GB VRAM baseline, 19.7 GB with spec.
* 3×24GB TP row: two RTX 3090 + one 3090 Ti, Unsloth UD-Q6_K_XL, tensor-parallel --split-mode tensor, --parallel 4, 500K unified KV pool, q8_0 KV, f16 draft KV, mmproj Q8, temp 1.0. VRAM ~16.6 GB/GPU baseline, ~18.7 GB/GPU with spec (tightest card). Method: stock probe.py. --parallel 1 was not required on this host.
* RTX 4090 row: unsloth UD-Q4_K_XL, 160K context, q4_0 KV cache, llama.cpp b10360, Windows/CUDA, 275W power limit — 20.6 GB VRAM baseline, 21.9 GB with spec.
* RX 9070 row: two 16GB cards on one 31.84 GiB pool, Vulkan build b10426, unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache — 28.0 GiB across the pool with spec. Method differs from the rows above and is spelled out under the sweep below.
* R9700 row: unsloth UD-Q4_K_XL, 262K context, q4_0 KV cache, llama.cpp b10433, Vulkan/RADV — 22.53 GB VRAM baseline, 24.55 GB with n-max 2. Method: unchanged probe.py at commit 67c20536, three runs x three prompts, thinking off.
* Ryzen AI Max+ 395 row: 64GB unified memory, unsloth UD-Q4_K_XL, 32K context, q8_0 KV cache, llama.cpp b10437, Windows build 26200, Vulkan with AMD driver 32.0.31035.1003. Method: unchanged probe.py at commit 67c2053, three runs x three prompts, thinking off.
* RTX 4090 Spadav_ row: unsloth Q4_K_M, 200K context, q4_0 KV cache, q8_0 draft KV, mmproj loaded (888MB on GPU) — method: stock probe.py + 4096-token curl (MTP crossover: overhead dominates at ≤400 tokens, +60% at 4096 tokens).
* Radeon 890M row: Ryzen AI 9 HX 370 (Strix Point, gfx1150), 48 GB UMA carve of 96 GB DDR5, unsloth UD-Q4_K_XL, 131K context, q4_0 KV cache, llama.cpp Vulkan/RADV, Ubuntu 26.04 — 19.15 GB VRAM baseline, 20.13 GB with n-max 2. Method: unchanged probe.py, three runs x three prompts, thinking off. Note this is a different APU class from the Ryzen AI Max+ 395 row above: Strix Point 890M is 16 CUs on a 128-bit bus, Strix Halo 8060S is 40 CUs on 256-bit, and this row holds 131K context resident against that row's 32K — both differences push this baseline down.
* RTX PRO 6000 row: unsloth Q4_K_M, 131K context, q4_0 KV cache, llama.cpp b10335, CUDA — method: stock probe.py, thinking off, --parallel 1 both sides. n-max 4 on this card: 85.7 overall (code 105.7 up, prose 58.8 down, acceptance 0.65-0.77) — overall peaks at n-max 2 here, same code-up/prose-down shape as the A6000 and 3×3090 sweeps. Cross-engine bonus on the same card: vLLM 0.27.1 with unsloth's NVFP4 build and MTP (--speculative-config '{"method":"mtp","num_speculative_tokens":N}') gives 63.3 -> 96.1 at n=2 and 116.3 (+84%) at n=4 — vLLM keeps climbing where llama.cpp has peaked. (An earlier revision carried a streamed-word-drop caveat for vLLM MTP; it was traced to the benchmark harness's own SSE parsing — a grep regex truncating multi-token deltas at escaped quotes — not to vLLM. Retracted with verification at vllm-project/vllm#52469, closed; streamed output is byte-identical to non-streamed with MTP on.)
* RTX 5090 desktop row: unsloth UD-Q4_K_XL, 192K context, q8_0 KV cache, llama-server self-built from the PR #26704 branch (master-equivalent for this path, CUDA arch 120), Windows 11 native — ~26.3 GB VRAM baseline, ~28.2 GB at n-max 4. Method differs from probe.py: server timings.predicted_per_second over 900-token generations, 2 runs x the same 3 prompts, thinking off, warmup discarded. Acceptance range is per-request (prose low end, Python high end), 0.60 aggregate.
* RTX 5090 32GB rows: unsloth Q6_K, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Windows/CUDA. Loaded VRAM was 24,441/27,385 MiB baseline and 25,797/29,381 MiB with spec. MTP gained 110.0% at 128K and 96.3% at 256K.
* RTX 3090 Ti rows: Q4_K_M, 128K/256K context, q4_0 KV cache, llama.cpp 62bf73d, Linux/CUDA. Loaded VRAM was 18,842/21,786 MiB baseline and 20,134/23,718 MiB with spec. MTP gained 45.0% at 128K and 49.0% at 256K. The 256K MTP arm had 846 MiB free at load.
* RTX 5090 32GB and RTX 3090 Ti method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. Baseline and MTP used the same model and serving config at each context. Only the MTP arm added --spec-type draft-mtp --spec-draft-n-max 2.
* RTX 5090 desktop UD-Q5_K_XL row: 131K context, q4_0 KV cache, llama.cpp b10448 (ad1de39e0), Windows/CUDA 13.3. Loaded VRAM was 22,449 MiB baseline and 24,083 MiB at n-max 4. Method: unchanged probe.py at commit b299c0f, three runs x three prompts, thinking off, warmup discarded. N-max 2 reached 129.4 tok/s with 80.9% aggregate acceptance; n-max 4 reached 144.2 tok/s with 63.8% aggregate acceptance, a 117.5% gain over baseline. Only the MTP flags changed within each pair.
* 3×3060 row: unsloth UD-Q4_K_XL, 262K context, q8_0 KV cache, layer split --split-mode layer --tensor-split 35,37,28 (no NVLink), llama.cpp b10068, --parallel 1, --spec-draft-n-max 8 --spec-draft-p-min 0.85. Method differs from the rows above: serve-timing averages from llama-server logs, not probe.py — baseline = weighted mean of 126K decoded tokens with MTP off (17.3 t/s); with flag = weighted mean over 23.4M decoded tokens of live traffic (24.5 t/s, short bursts 28-39 t/s, long-context drops to 14-16 t/s). Acceptance = aggregate 0.87 (176,446/202,620 drafted tokens, per-task median 0.884, range 0.70-1.00) from draft acceptance log lines.
* Radeon 9060 XT 16 row: AtomicChat AD-IQ3_S_IQ3_XXS, 128k context, q4_0 KV cache — method: stock probe.py. Windows, ROCM
* RTX 3090 b10450 row: unsloth Q4_K_M, 131K context, q8_0
Этот репозиторий описывает один флаг llama.cpp server, который включает спекулятивное декодирование с многотокенным предсказанием (MTP) для Qwen3.8-27B, ускоряя декодирование на 33–145% на потребительских GPU. В нём собраны точная команда запуска, краудсорсинговая таблица бенчмарков по десяткам GPU и скрипт probe.py, чтобы измерить прирост самостоятельно.