Qwen3.8-27B · EXL3 · DFlash2
by Mia'a AI Lab
Deployment kit for serving Qwen3.8-27B quantized to EXL3 3.5bpw
with speculative decoding — MTP by default (draft head inside the
checkpoint, best context per GB) or DFlash2 (EXL3 5.0bpw draft model,
~15% faster) — plus a ~4.5-bit KV cache lane (NVFP4 on Ada/Hopper/Blackwell,
Hadamard-4 on Ampere) for maximum context per GB.
This repo contains the launcher, server and configuration. The model weights
live on Hugging Face (see Model cards) and the inference
engine is our exllamav3 fork
(DFlash2/MTP drafting, NVFP4/FP8 KV, aarch64 GB10 + x86 CUDA).
What's in the box
| file |
what |
start.sh |
env-driven launcher for the OpenAI-compatible server |
stop.sh |
stop the server (graceful shutdown, safe to re-run) |
tools/serve_openai.py |
the server itself (chat completions, streaming, tool calling) |
.env.example |
documented configuration knobs |
model-cards/ |
model cards for the two HF weight sets |
Quick start
# 1. configure and launch:
cp .env.example .env # edit: context, GPU memory
./start.sh # first run: builds .venv + installs the engine,
# downloads weights, serves http://localhost:8888/v1
start.sh is self-bootstrapping: on first run it creates .venv, installs
GPU torch and pip-installs the exllamav3 engine from this fork (all of
DFlash2/MTP drafting, NVFP4/FP8 KV and the aarch64 GB10 port are fork
features). EXL3_REPO in .env overrides where the engine comes from — any
git URL or a local path.
Running on a 24 GB GPU (RTX 3090/4090)? Jump to
24 GB config.
start.sh auto-creates .env from the example on first run, prefers
.venv/bin/python when present, and downloads the target and draft weight
sets from Hugging Face automatically on first launch (and resumes partial
downloads) — no manual fetching. Set HF_TARGET_REPO / HF_DRAFT_REPO in
.env to pull from a mirror instead.
Configuration highlights (.env)
DRAFT — speculative decoding method: mtp (default; the draft head
lives inside the target checkpoint — no download, ~50% more context on
small GPUs), dflash2 (dedicated draft model, fastest tokens/s), or
none. MODEL_DIR / DRAFT_DIR set the target and (for dflash2) draft
locations; missing dirs are fetched from the Hub automatically
(HF_TARGET_REPO / HF_DRAFT_REPO override the repo ids)CONTEXT_SIZE — KV cache size in tokens (native limit 262,144; 1M works and
the launcher swaps in the YaRN config variant automatically)CACHE_QUANT — KV format: none (fp16) / 8 / 8,4 / 4 / fp8 / nvfp4.
nvfp4 and fp8 need compute capability ≥ 8.9 (Ada 4090, Hopper, Blackwell /
GB10). Ampere (3090, sm_86) cannot compile those Triton kernels — use
Hadamard 4 (~4.5 bits/elem, same density as NVFP4) or 8 / 8,4GPU_MEM_GB — memory budget for weights + caches; auto-detected when unset
(discrete GPU: VRAM − 2 GB, unified/GB10: available RAM − 16 GB)CPU_CACHE_GB — CPU second-tier cache knob (accepted but not yet active;
see the 24 GB notes)
Concurrency note: the server generates one request at a time (batch-1
speculative decoding); concurrent requests queue automatically. Measured on
DGX Spark with DRAFT=dflash2: 8 concurrent requests ran fully sequentially
(no batching benefit), aggregate throughput ~16.7 tok/s at that concurrency
— plan capacity accordingly if your workload is concurrent rather than
single-request.
Reasoning note: the server always reasons — there is currently no way to
disable it. chat_template_kwargs.enable_thinking (the vLLM/SGLang
convention) is silently ignored. The reasoning trace comes back in a
separate reasoning_content field (both in the full response and in each
streamed delta), but it is not a separate token budget: reasoning and
visible content both draw from the same max_tokens. With a tight budget
(e.g. max_tokens: 16 on a short-answer prompt) I got a response with no
content at all, entirely consumed by the reasoning trace — clients
expecting a quick, cheap warmup/probe call should budget generously rather
than assuming a short max_tokens implies a short wait.
24 GB GPUs (RTX 3090 / 4090)
The kit targets DGX Spark (121 GB) by default. On a 24 GB card the recipe is
the same weights, a tighter KV budget, and a ~4.5-bit KV format so the native
262k context still fits. The default drafter (MTP) is the right choice here.
KV format is GPU-dependent. This fork's nvfp4 / fp8 lanes are software
packed caches (E2M1 + E4M3 scales, or raw E4M3) with Triton online dequant.
Those kernels use fp8e4nv, which Triton only compiles on compute capability
≥ 8.9. They are not Blackwell tensor-core NVFP4 matmul.
| GPU |
Arch |
CACHE_QUANT for 262k |
| RTX 4090, GB10, Hopper+ |
sm ≥ 8.9 |
nvfp4 (measured lossless at generation level) |
| RTX 3090 |
Ampere sm_86 |
4 (Hadamard int4; same ~4.5 bits/elem). nvfp4 and fp8 will fail to compile |
# .env — 24 GB recipe (MTP, the default)
GPU_MEM_GB=22 # auto-detected when unset; explicit for clarity
CONTEXT_SIZE=262144 # full native context fits (see math below)
CACHE_QUANT=nvfp4 # 4090 / GB10 / Hopper+. On a 3090 use: CACHE_QUANT=4
# DRAFT=mtp is the default — no extra config, no draft download
DRAFT=mtp (default): no draft weights, ~20% less KV per token → the
full native 262k context fits fully resident, at ~30 tok/s (GB10; an RTX
card should be faster — decode is memory-bandwidth-bound).DRAFT=dflash2: ~34.5 tok/s but 1.4 GB of draft weights + draft KV →
~220k–262k tokens fully resident.
# .env — alternative: DFlash2 at 24 GB
GPU_MEM_GB=22
CONTEXT_SIZE=200000
CACHE_QUANT=nvfp4 # 3090: CACHE_QUANT=4
DRAFT=dflash2
DRAFT_DIR=models/Qwen3.8-27B-DFlash2-EXL3-5.0bpw # auto-downloaded
# CPU_CACHE_GB=16 # CPU spill tier: NOT YET ACTIVE (planned; see notes)
Memory math (GiB): target weights 14.2 + MTP head 0.05 → ~6.7 GB left for
KV at a 22 GB budget; NVFP4 or Hadamard-4 KV costs ~18 KB/token for the
target plus ~1.2 KB/token for the MTP head (only the 16 full-attention layers
hold KV; fp16 is ~64 KB/token) → the full native 262k fits with ~1 GB to
spare. With the DFlash2 draft instead: 15.6 GiB of weights + ~24 KB/token →
~220k–262k. CACHE_QUANT=8 (8.5 bits) needs 9.1 GB at 262k and will not
leave that much headroom; fp16 (17 GB at 262k) does not fit. Past the
resident limit the cache does not hold today; the CPU spill tier is planned
but not yet implemented.
RTX-class notes (vs the DGX Spark the numbers above were measured on):
- Perf is memory-bandwidth-bound at batch 1, and should be higher on an
RTX card: GB10 reads weights from ~273 GB/s-class LPDDR5x unified memory;
a 3090/4090 reads the same weights from ~1 TB/s-class VRAM. Acceptance and
quality are unchanged — only tok/s moves. Not yet benchmarked on RTX, treat
the table in the model cards as the lower bound.
- CPU spill (planned,
CPU_CACHE_GB): once implemented (T6 follow-up),
a desktop's 32–64 GB of system RAM backs cold KV pages over PCIe — slow to
touch, but it converts the hard ~220k resident limit into a graceful
decline. Today the knob is accepted but inert.
- The 1M YaRN config does not fit: 1M tokens of ~4.5-bit KV ≈ 19 GB on
top of 15.6 GB of weights. Only reachable with CPU spill, and 262k is the
quality-faithful limit anyway — don't expect the 1M headroom to be useful
here.
- Use the fork on x86 too: stock exllamav3 serves the model on CUDA, but
NVFP4/FP8 KV and DFlash2 drafting are fork features; install the fork's x86
CUDA build. Hadamard
4 / 8 / 8,4 KV is upstream.
- Need more context than ~262k? That requires the YaRN 1M config and more
memory than a 24 GB card has.
DRAFT=none frees the draft memory too, but
gives up speculative decoding — DRAFT=mtp already dominates it on both
memory and speed.
OpenAI-style tools / tool_choice work (referenced in the file table
above but not otherwise documented here). Verified request/response shape:
curl http://localhost:8888/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3.8-27b-exl3-3.5bpw-wm",
"messages": [{"role": "user", "content": "What is the weather in Lyon?"}],
"tools": [{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
}
}],
"tool_choice": "auto",
"max_tokens": 300
}'
Returns a normal OpenAI tool_calls array (finish_reason: "tool_calls",
message.content: null) alongside the usual reasoning_content. The
model id in responses doesn't necessarily match HF_TARGET_REPO or
MODEL_DIR verbatim (I saw qwen3.8-27b-exl3-3.5bpw-wm for
MODEL_DIR=models/Qwen3.8-27B-EXL3-3.5bpw) — check /v1/models rather
than assuming the id a client should send.
Model cards
Measured on DGX Spark (GB10): HumanEval-class decode 47.5 tok/s at
T=0.6 with acceptance 4.43 tokens/step; tool-eval-bench hardmode @ T=1.0:
87–88 / 100.
License
Code in this repo: MIT. Model weights are Apache-2.0 derivatives
(see model cards); exllamav3 is MIT.
一套部署工具包，用于通过 OpenAI 兼容服务器提供 Qwen3.8-27B（量化为 EXL3 3.5bpw）服务，并利用投机解码加速生成。它捆绑了启动器、服务器和配置；模型权重托管在 Hugging Face 上，推理则在团队的 exllamav3 分支上运行。