Qwen3.8-27B-DFlash2-EXL3-5.0bpw README

The project's README from github.com/MiaAI-Lab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw. As one markdown file: README.md.

Qwen3.8-27B · EXL3 · DFlash2

by Mia'a AI Lab

Sponsor me on GitHub Follow Mia on X

Deployment kit for serving Qwen3.8-27B quantized to EXL3 3.5bpw with speculative decoding — MTP by default (draft head inside the checkpoint, best context per GB) or DFlash2 (EXL3 5.0bpw draft model, ~15% faster) — plus a ~4.5-bit KV cache lane (NVFP4 on Ada/Hopper/Blackwell, Hadamard-4 on Ampere) for maximum context per GB.

This repo contains the launcher, server and configuration. The model weights live on Hugging Face (see Model cards) and the inference engine is our exllamav3 fork (DFlash2/MTP drafting, NVFP4/FP8 KV, aarch64 GB10 + x86 CUDA).

What's in the box

file what
start.sh env-driven launcher for the OpenAI-compatible server
stop.sh stop the server (graceful shutdown, safe to re-run)
tools/serve_openai.py the server itself (chat completions, streaming, tool calling)
.env.example documented configuration knobs
model-cards/ model cards for the two HF weight sets

Quick start

# 1. configure and launch:
cp .env.example .env      # edit: context, GPU memory
./start.sh                # first run: builds .venv + installs the engine,
                          # downloads weights, serves http://localhost:8888/v1

start.sh is self-bootstrapping: on first run it creates .venv, installs GPU torch and pip-installs the exllamav3 engine from this fork (all of DFlash2/MTP drafting, NVFP4/FP8 KV and the aarch64 GB10 port are fork features). EXL3_REPO in .env overrides where the engine comes from — any git URL or a local path.

Running on a 24 GB GPU (RTX 3090/4090)? Jump to 24 GB config.

start.sh auto-creates .env from the example on first run, prefers .venv/bin/python when present, and downloads the target and draft weight sets from Hugging Face automatically on first launch (and resumes partial downloads) — no manual fetching. Set HF_TARGET_REPO / HF_DRAFT_REPO in .env to pull from a mirror instead.

Configuration highlights (.env)

Concurrency note: the server generates one request at a time (batch-1 speculative decoding); concurrent requests queue automatically. Measured on DGX Spark with DRAFT=dflash2: 8 concurrent requests ran fully sequentially (no batching benefit), aggregate throughput ~16.7 tok/s at that concurrency — plan capacity accordingly if your workload is concurrent rather than single-request.

Reasoning note: the server always reasons — there is currently no way to disable it. chat_template_kwargs.enable_thinking (the vLLM/SGLang convention) is silently ignored. The reasoning trace comes back in a separate reasoning_content field (both in the full response and in each streamed delta), but it is not a separate token budget: reasoning and visible content both draw from the same max_tokens. With a tight budget (e.g. max_tokens: 16 on a short-answer prompt) I got a response with no content at all, entirely consumed by the reasoning trace — clients expecting a quick, cheap warmup/probe call should budget generously rather than assuming a short max_tokens implies a short wait.

24 GB GPUs (RTX 3090 / 4090)

The kit targets DGX Spark (121 GB) by default. On a 24 GB card the recipe is the same weights, a tighter KV budget, and a ~4.5-bit KV format so the native 262k context still fits. The default drafter (MTP) is the right choice here.

KV format is GPU-dependent. This fork's nvfp4 / fp8 lanes are software packed caches (E2M1 + E4M3 scales, or raw E4M3) with Triton online dequant. Those kernels use fp8e4nv, which Triton only compiles on compute capability ≥ 8.9. They are not Blackwell tensor-core NVFP4 matmul.

GPU Arch CACHE_QUANT for 262k
RTX 4090, GB10, Hopper+ sm ≥ 8.9 nvfp4 (measured lossless at generation level)
RTX 3090 Ampere sm_86 4 (Hadamard int4; same ~4.5 bits/elem). nvfp4 and fp8 will fail to compile 
# .env — 24 GB recipe (MTP, the default)
GPU_MEM_GB=22            # auto-detected when unset; explicit for clarity
CONTEXT_SIZE=262144      # full native context fits (see math below)
CACHE_QUANT=nvfp4        # 4090 / GB10 / Hopper+. On a 3090 use: CACHE_QUANT=4
# DRAFT=mtp is the default — no extra config, no draft download

# .env — alternative: DFlash2 at 24 GB
GPU_MEM_GB=22
CONTEXT_SIZE=200000
CACHE_QUANT=nvfp4        # 3090: CACHE_QUANT=4
DRAFT=dflash2
DRAFT_DIR=models/Qwen3.8-27B-DFlash2-EXL3-5.0bpw   # auto-downloaded
# CPU_CACHE_GB=16        # CPU spill tier: NOT YET ACTIVE (planned; see notes)

Memory math (GiB): target weights 14.2 + MTP head 0.05 → ~6.7 GB left for KV at a 22 GB budget; NVFP4 or Hadamard-4 KV costs ~18 KB/token for the target plus ~1.2 KB/token for the MTP head (only the 16 full-attention layers hold KV; fp16 is ~64 KB/token) → the full native 262k fits with ~1 GB to spare. With the DFlash2 draft instead: 15.6 GiB of weights + ~24 KB/token → ~220k–262k. CACHE_QUANT=8 (8.5 bits) needs 9.1 GB at 262k and will not leave that much headroom; fp16 (17 GB at 262k) does not fit. Past the resident limit the cache does not hold today; the CPU spill tier is planned but not yet implemented.

RTX-class notes (vs the DGX Spark the numbers above were measured on):

Tool calling

OpenAI-style tools / tool_choice work (referenced in the file table above but not otherwise documented here). Verified request/response shape:

curl http://localhost:8888/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.8-27b-exl3-3.5bpw-wm",
    "messages": [{"role": "user", "content": "What is the weather in Lyon?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
      }
    }],
    "tool_choice": "auto",
    "max_tokens": 300
  }'

Returns a normal OpenAI tool_calls array (finish_reason: "tool_calls", message.content: null) alongside the usual reasoning_content. The model id in responses doesn't necessarily match HF_TARGET_REPO or MODEL_DIR verbatim (I saw qwen3.8-27b-exl3-3.5bpw-wm for MODEL_DIR=models/Qwen3.8-27B-EXL3-3.5bpw) — check /v1/models rather than assuming the id a client should send.

Model cards

Measured on DGX Spark (GB10): HumanEval-class decode 47.5 tok/s at T=0.6 with acceptance 4.43 tokens/step; tool-eval-bench hardmode @ T=1.0: 87–88 / 100.

License

Code in this repo: MIT. Model weights are Apache-2.0 derivatives (see model cards); exllamav3 is MIT.

Qwen3.8-27B-DFlash2-EXL3-5.0bpw (MiaAI-Lab/Qwen3.8-27B-DFlash2-EXL3-5.0bpw)

nvfp4
Searching Database…

232 ·GitHub
Qwen3.8-27B-DFlash2-EXL3-5.0bpw
What is this?A deployment kit for serving Qwen3.8-27B (quantized to EXL3 3.5bpw) through an OpenAI-compatible server, using speculative decoding to speed up generation. It bundles the launcher, server, and config; the model weights live on Hugging Face and inference runs on the team's exllamav3 fork.
Is this the model itself or just the serving code?Just the serving code. This repo holds start.sh/stop.sh launchers, the tools/serve_openai.py server, and .env config; the actual weights live on Hugging Face (two model cards), and the GPU inference engine is a separate exllamav3 fork.
How do I get this running?Run start.sh — it is self-bootstrapping: on first run it creates a .venv, installs GPU torch, pip-installs the exllamav3 fork, writes .env from the example, and downloads the target and draft weights from Hugging Face automatically (resuming partial downloads).
What GPU do I need to run this?An Nvidia GPU with CUDA; the kit targets DGX Spark (121 GB, GB10/aarch64) by default but also ships a 24 GB config for RTX 3090/4090. The NVFP4/FP8 KV cache lane needs compute capability ≥8.9 (Ada/Hopper/Blackwell); Ampere (3090, sm86) must fall back to the Hadamard-4 KV format.
Can it serve multiple requests at once?No — the server generates one request at a time (batch-1 speculative decoding); concurrent requests simply queue, so throughput doesn't scale with concurrency (measured ~16.7 tok/s aggregate at 8 concurrent requests on DGX Spark).
Does this run on any GPU brand, or only Nvidia?Nvidia only — it's built on CUDA via the exllamav3 fork, with specific Triton-kernel paths for Ada/Hopper/Blackwell GPUs versus Ampere, plus an aarch64 build for DGX Spark's GB10.
What's the difference between MTP and DFlash2 draft modes?MTP uses a draft head built into the target checkpoint — no extra download, best context per GB, ~30 tok/s on DGX Spark. DFlash2 uses a separate 1.4 GB EXL3 5.0bpw draft model for ~34.5 tok/s but costs extra KV memory and reduces fully-resident context headroom.
How large a context window does it support?The native limit is 262,144 tokens. Setting CONTEXTSIZE up to 1M works and the launcher auto-swaps in a YaRN config variant, but 262k is described as the quality-faithful limit.
Can I turn off the model's reasoning output?No — the server always reasons and there's currently no way to disable it; the vLLM/SGLang enable_thinking flag is silently ignored, and the reasoning trace shares the same max_tokens budget as the visible answer.
Does it support tool calling?Yes — it returns a standard OpenAI tool_calls array with finish_reason: 'tool_calls', verified to work alongside the separate reasoning_content field in both full and streamed responses.
What are similar projects?qwen3.8-Flash-DGX, dgx-spark-qwen38, v100-skinny, DeepSeek-v4-Flash-One-DGX-Spark, GLM-5.3-Flash-EXL3-2x-DGX-Sparks, Qwen3.8-Flash-Next-Single-DGX-Spark, GLM-5.3-Flash-EXL3-2x-DGX-Sparks-TensorFold, TensorFold
Which topics is it in?nvfp4, triton, aarch64, speculative-decoding, model-serving, deployment, llm-inference, gpu-acceleration, quantization, large-language-models, openai-compatible, local-llm
Creator
MiaAI-Lab
Size
0.1 MB
Stars
232
Updated
CommitSep 01
Age
1mo
Commits · 90 Days