GLM-5.3 Flash EXL3 for 2x DGX Sparks
by Mia'a AI Lab
OpenAI-compatible vLLM serve of
zai-org/GLM-5.3-Flash as
Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw
— a byte-identical public mirror of
brandonmusic/GLM-5.3-Flash-tr3-4bpw
snapshot 5ab363a8… (uniform-K4 EXL3/TR3 routed-experts, 4 bpw, ~164 GiB, 120 shards)
so this recipe stays fetchable if the upstream Hub id moves. On a 2× NVIDIA GB10
kit: tensor-parallel size 2 over CX7, native sm_121a cubins, API on :8888.
Served model id: GLM-5.3-Flash-EXL3. EXL3/TR3 quant by
brandonmusic.
This is EXL3 weights + fp8 KV on GB10. Do not pass --moe-backend marlin.
The Hub card on brandonmusic (TP2/EP2/DCP2 + calibrated NVFP4 MLA KV) is the SM120 B12X
image (verdictai/glm53-flash-exl3-k4:…-v84-dflash2), not this overlay. Target KV
stays packed fp8_ds_mla. Speculator is DFlash2 k=7
(incoai/GLM-5.3-Flash-DFlash2);
draft attention is FLASH_ATTN (do not pin TRITON_ATTN — that mask is causal
inside the draft block on this image and collapses later-position accept).
Cold prefill (E3 grouped MoE, this kit, 2026-09-07)
EXL3_FAT_GROUPED=1 (launcher default since 2026-09-07) replaces the E2 per-expert host loop for "fat" experts with three
GPU-driven launches per MoE layer (gather, gate/up + SwiGLU, down + scatter) built from
device-side segment tables — no per-expert launches, no weight repacking, no host sync
(overlay/exl3_fat_moe.cu, Fable's design, E2 rounding boundaries restored).
Latest run — 2026-09-07, measured with sparkDash.
Shipped E3 image glm53-flash-sm121:e3-20260907 at MAX_MODEL_LEN=900000, GPU_MEM_UTIL=0.86,
GLM53_INDEXER_WORKSPACE=rightsize, EXL3_TEMP_ROWS_FUSED=32, MNBT 7168, MAX_NUM_SEQS=4,
DFlash2 k=7 draft TP=2, C4, thinking off.
| Prompt |
Prompt tok |
TTFT |
Prefill tok/s |
| ~8k |
8,221 |
5.51 s |
1,492.1 |
| ~16k |
16,411 |
10.56 s |
1,553.7 |
| ~32k |
32,797 |
22.96 s |
1,428.2 |
| ~64k |
65,566 |
41.31 s |
1,587.0 |
| ~128k |
131,101 |
83.95 s |
1,561.7 |
| ~256k |
262,173 |
172.84 s |
1,516.8 |
A/B receipt vs E2 (2026-09-06 overnight run). Measured
on this kit: MAX_MODEL_LEN=500000, GPU_MEM_UTIL=0.84, GLM53_INDEXER_WORKSPACE=rightsize,
EXL3_TEMP_ROWS_FUSED=32, MNBT 7168, DFlash2 k=7 draft TP=2, C4, thinking off, unique cold
prompts (0 prefix hits). Prefill tok/s = prompt tokens / TTFT (client side).
| Prompt |
Prompt tok |
TTFT |
Prefill tok/s |
E2 before (tok/s) |
Gain |
| ~8k (first request after boot) |
8,221 |
8.23 s |
999 |
1,112 warm / 418 first |
one-time allocation; warm ~1,500 |
| ~16k |
16,417 |
10.41 s |
1,578 |
1,155 |
+37% |
| ~32k |
32,798 |
20.20 s |
1,624 |
~1,160 |
+40% |
| ~64k |
65,565 |
40.06 s |
1,637 |
~1,170 |
+40% |
| ~128k |
131,101 |
80.46 s |
1,629 |
1,148–1,183 (100k) |
+38–42% |
| ~256k |
262,173 |
166.4 s |
1,576 |
1,087 |
+45% |
E2 column: the 2026-09-01 E2 receipts (logs/pr77-build-kernel-20260901) and the matched
E2 baselines of the 2026-09-06 overnight run (logs/overnight-20260906T164059Z/exp500k,
A/B/A at 500k: 8k 1,112 / 16k 1,154 / 100k 1,183, repeats within 0.5%); 32k/64k interpolated.
Decode is unchanged (E3 never runs on decode-sized steps; structured / prose / code within
run-to-run spread). Isolated layer bench: 7168-token MoE layer 77–91 ms (E2) → 31 ms (E3),
numerically indistinguishable from E2 vs the LinearEXL3 reference. Full receipts, gates and
the memory caveat: logs/overnight-20260906T164059Z/report.md.
Context and headroom: the shipped default is now 850k / util 0.85 / rightsize (boots with ~0.5 GiB of KV margin on the head; a 256k prefill ran at 900k / 0.87 with driver retries but no failure). E3 keeps a 560 MiB fat-row scratch that vLLM's profile run charges to
the KV budget, so at 1M / util 0.87 the pool no longer fits one 1M request on this kit
(needs 14.52 GiB). 500k needs 10.98 GiB and boots reliably at 0.84 (1.2× at 500k). Prompts
≥ ~100k are near the head's host-memory limit at any setting (a 256k prefill at util 0.87
with zero MemAvailable crashed the head on 2026-09-06); the 256k row above ran at 0.84.
Decode (this kit, 2026-08-28)
Official numbers: sparkDash Decode bench, DFlash2 k=7, Structured (count 1→200) and Code (clamp_00…clamp_49) — same high-accept regime. Temp 0, thinking off, 400 tokens, CUDA graphs, fused EXL3 MoE. Prompt types, not grammar / schema. Stream tok/s is per request; aggregate is all streams.
| Concurrency |
TTFT |
Stream tok/s |
Aggregate tok/s |
| ×1 |
719 ms |
62.9 |
62.9 |
| ×2 |
6.62 s |
51.7 |
103.3 |
| ×4 |
6.30 s |
37.1 |
146.5 |
That 2026-08-28 decode serve used --max-model-len 1000000 with a 1,754,237-token KV pool. These runs are warm / empty KV — they do not need a filled 1M cache.
Prose (sparkDash Decode bench, prose prompt type, 2026-09-08) on the adaptive-verification + FP8-dense serve
(GLM53_ADAPTIVE_K=ema, GLM53_DENSE_FP8=dense,kda, 850k context, KV pool capped at 14 GiB — turned on
as below; the stock k=7 / BF16 serve measured ~18–27 tok/s per stream on the lab prose prompts):
| Concurrency |
TTFT |
Stream tok/s |
Aggregate tok/s |
| ×1 |
268 ms |
32.1 |
32.1 |
| ×2 |
399 ms |
22.1 |
41.2 |
Faster prose decode (opt-in, 2026-09-08)
Two decode speed-ups ship in the overlay, both off by default (matched A/B/A at 131k and 850k, 8 runs per prompt, bootstrap 95 % CI; receipts in logs/overnight-decode-20260907T224521Z/):
- Adaptive verification length (
GLM53_ADAPTIVE_K=ema): the DFlash2 drafter still proposes 7 tokens, but the scheduler verifies only a per-step prefix (2, 4 or 7) chosen from a running average of how many drafts have been surviving, batch-uniform so every decode step keeps its FULL CUDA graph. Lossless at temperature 0. Measured vs stock k=7 (8 runs/prompt, 131k and 850k): Silk Road essay +21 %, sky/sunset +13 %, hash-map +10 %, code +5–15 %, counting unchanged.
- FP8 weight-only dense projections (
GLM53_DENSE_FP8=dense,kda): KDA and dense-MLP projections quantised per output channel to FP8 at load and run through the Marlin kernel, ~11 ms less per step on everything (+10 % on counting, prose +12–19 % alone, +37 % on hard prose stacked with adaptive-k). PROVISIONAL: it changes target numerics by FP8 rounding (KL proxy vs stock 0.002–0.013 nats/position, argmax agreement 94–100 %; no full KLD panel yet).
Turn on (no rebuild; the patches apply at container start on both nodes):
# .env
GLM53_ADAPTIVE_K=ema
GLM53_ADAPTIVE_K_SET=2,4,7
GLM53_DENSE_FP8=dense,kda # drop this line to keep BF16 dense weights (lossless config)
EXTRA_ARGS="--cudagraph-capture-sizes 1 2 3 4 5 6 8 9 10 12 15 16 20 24 32 --kv-cache-memory-bytes 15032385536"
then ./start.sh restart. The capture-size list is required for adaptive-k (multiples of 3, 5 and 8 up to 4 requests; the stock 1 2 4 8 16 24 32 misses the 3- and 5-token shapes). The KV cap turns FP8's freed GPU memory into host headroom instead of a bigger pool: uncapped, the head dropped to 1.5 GiB MemAvailable at 850k. 14 GiB leaves an 883,552-token pool (1.04x of 850k) and ~5 GiB free; 15 GiB buys 1.11x but measured only 0.8-2.2 GiB free under load, which is not enough margin on this UMA. Do not go much lower at 850k either — the boot refuses a pool that cannot hold one max-length request (13 GiB is ~820k tokens). Verify after boot: docker logs glm53-exl3-head | grep -a "adaptive-k\|dense fp8" should show uniform decode graph query lens: [3, 5, 8] and dense fp8 groups: dense,kda, and with ABLIT=1 the line ABLIT_METHOD=auto -> transplant (a missing ablit/transplant/ silently falls back to the projection edit, which garbles sampled output). A running server can be retuned without a reboot through `/.cache/vllm-glm53-flash/glm53_adaptive_k.json ({"mode":"ema","set":"2,4,7","margin":1.0}; {"mode":"off"}` restores k=7). Live sparkDash prose numbers with both on are in the table above.
Lab tests/bench_decode.py on the same protocol (median of 5 × 400, 2026-08-30 C4, DFLASH_DRAFT_TP=2): Structured 65.1 tok/s (0.959 accept / 6.71 per step); Prose (hash-map) 27.1 (0.341 / 2.39). Prior TP=1 lab: 61.7 / 26.9. Long context / mixed (~60–100k KV) 24–27. MTP k=2 baseline ~24.6.
Structured per-pos (lab median): 0.98 / 0.98 / 0.94 / 0.94 / 0.91 / 0.83 / 0.83.
Prose per-pos: 0.75 / 0.58 / 0.41 / 0.28 / 0.16 / 0.09 / 0.06.
Pinning attention_backend=TRITON_ATTN dropped structured to ~29 tok/s / 0.31 accept
(pos0 healthy, later positions collapsed).
Re-measure:
# structured (count 1→200)
python3 tests/bench_decode.py --phase structured --structured --runs 5 --max-tokens 400 --skip-coherence --out /tmp/glm53-structured.json
# prose (hash-map explanation)
python3 tests/bench_decode.py --phase prose --runs 5 --max-tokens 400 --skip-coherence --out /tmp/glm53-prose.json
E2 fat-expert prefill — PR77 (2026-09-01)
PR77 adds purpose-built direct/scatter CUDA kernels for the routed “fat”
experts and lifts fully uncached long-context prefill by about 20–21%.
The controlled promotion ran E2 on → legacy off → E2 on at MNBT 2048, with
five unique-salt cold samples per rung per boot:
| Fully uncached rung |
Legacy mean tok/s |
PR77 pooled mean tok/s |
Gain |
| ~8K |
941.04 |
1132.32 |
+20.33% |
| ~100K |
1023.20 |
1241.71 |
+21.36% |
| ~300K |
995.05 |
1201.02 |
+20.70% |
All 45 observations passed the cold gate, with complete separation at every
rung. A second 2× DGX Spark deployment reproduced +21.0% at 100K and
+20.4% at 300K. These repeated results establish E2 as the production
prefill path; the decode path is unchanged. They replace earlier preliminary
figures that included APC hits.
On the independent MAX_NUM_SEQS=16 geometry, MNBT 2048 delivered the best
measured balance of prefill throughput and KV capacity. MNBT 7168 remains the
current maintainer default for MAX_NUM_SEQS=4, pending a repeated same-kit
comparison.
Quality (KLD)
Independent teacher-logit panel from
malaiwah on the 4bpw discussion:
KLD(teacher ‖ model), five cold runs, 25 sealed windows (51,175 positions). This
scores the weights, not this GB10 overlay. We serve the 4bpw row.
| Model |
Mean KLD (nats) |
Size |
| TR3 K6 (6bpw) |
0.013723 |
254 GB |
| Official FP8 (cross-stack) |
0.020615 |
328 GB |
| This checkpoint — EXL3 4bpw |
0.024555 |
176 GB |
| Official FP8 (brandonmusic stack, v44) |
0.024629 |
328 GB |
| NVFP4 (brandonmusic stack, v44) |
0.060535 |
~180 GB |
On the same stack, 4bpw matches official FP8 (~1.00× KLD) at 54% of the bytes.
K6 (malaiwah/GLM-5.3-Flash-TR3-6bpw) is a different checkpoint. Padded DFlash
slot-share is an allocator change only — target KV stays packed fp8_ds_mla,
same path as the compact-64 fp8 serve (not NVFP4 KV).
What runs
| Layer |
Runtime |
| API |
vLLM OpenAI (/v1/chat/completions) on the head, port 8888. Open by default; set VLLM_API_KEY for optional Bearer auth |
| Weights |
Mia-AiLab/GLM-5.3-Flash-EXL3-TR3-4bpw (mirror of brandonmusic/… snapshot 5ab363a8…) |
| Model id |
GLM-5.3-Flash-EXL3 (--served-model-name) |
| Image |
ghcr.io/miaai-lab/glm-5.3-flash-2x-dgx-sparks:exl3 FROM vllm/vllm-openai:glm53-flash-arm64-cu130@sha256:905c0293… (arm64, CUDA 13.0) |
| Executor |
mp, --nnodes 2, --tensor-parallel-size 2 |
| Head |
this machine, HEAD_IP=10.0.0.1, container glm53-exl3-head |
| Worker |
WORKER_USER@WORKER_IP (this kit: [email protected]), --headless, glm53-exl3-worker |
| Fabric |
CX7 QSFP: enp1s0f1np1/rocep1s0f1 ↔ enp1s0f0np0/rocep1s0f0. Image NCCL (USE_HOST_NCCL=0) |
| Attention |
FLASHINFER_MLA_SPARSE_SM120 (NoPE MLA padded into GLM_NSA 576-wide) |
| KV |
--kv-cache-dtype fp8 → packed fp8_ds_mla (target). Draft DFlash2 KV is auto/bf16. Latest validated 7168/rightsize pool: 1,243,902 tokens / 1.24× at 1M. --enable-prefix-caching (block-aligned hits; see Prefix caching) |
| Context |
850k default since 2026-09-07 (E3; EXL3_FAT_GROUPED=0 fits 1M again). KV pool is ~1M tokens (measured 989,010 at 900k / util 0.85, and 1,023,626-1,084,615 at 900k / util 0.86; pool size varies ±0.4 GiB between identical boots, and the 850k default is not yet measured). Pre-E3 1M receipts: live pool 1,754,237 tokens (1.75×) / 690 GPU blocks / 18.67 GiB at MNBT 2048; latest validated 7168/rightsize E2 pool at 1M 1,243,902 tokens / 1.24×. Pool size varies with MNBT, activation/graph reservations and hybrid block geometry. Padded slot-share is why 1M allocates; the old 900k cap was 1.95× on this same pool. Do not drop to 256k to “free” slots (hybrid mamba + DFlash window block-id demand is mostly length-independent) |
| Experts |
packed trellis + suh + svh + mcg, codebook MCG, one fused exllamav3_ext.exl3_moe launch per layer |
| Dense / shared / attn / embed / lm_head |
native (unquantized) |
| Tools / reasoning |
--tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 |
| Graphs |
on (ENFORCE_EAGER=0) — MTP capture 1 2 3 4 6 8 12; DFlash2 capture 1 2 4 8 16 24 32 |
| Spec |
DFlash2 k=7 (incoai/GLM-5.3-Flash-DFlash2); draft KV auto/bf16, draft TP=2, FLASH_ATTN. Rollback SPEC_METHOD=mtp |
| Vision |
on (LANGUAGE_MODEL_ONLY=0) — image + video, --limit-mm-per-prompt {image:100,video:1}, --skip-mm-profiling |
| Ablit |
off (ABLIT=0). Stock o_proj. Set ABLIT=1 to enable; see Abliteration |
Kernels: TORCH_CUDA_ARCH_LIST=12.1a. ExLlamaV3 pin c5d9c657 (0.0.43) exposes
exl3_moe / exl3_moe_max_concurrency; aarch64 CPU allreduce stubs in
overlay/patch_exl3_ext_aarch64.py.
Abliteration (ABLIT=1)
Off by default. Unset or ABLIT=0 serves stock o_proj. ABLIT=1 is
opt-in refusal-direction ablation at weight-load on top of the EXL3 checkpoint
— nothing on disk is rewritten.
Artifacts live in ablit/ (from
drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock,
method dealign-oproj-transplant: layers 15–45 edited, 0–14 stock
safety anchors, MTP block included).
Method — ABLIT_METHOD=transplant (default via auto). The published
checkpoint's o_proj L15–45 are byte-copied from that recipe
(same 120-shard layout; o_proj is native BF16 in both). Since this EXL3
checkpoint's o_proj is byte-identical to the NVFP4 body's, replacing those
31 tensors at load reproduces the published edit exactly. Fetched once (~2.7 GiB,
range-requests only the o_proj tensors — no full checkpoint download):
python3 ablit/fetch_transplant.py # donor is public; HF token optional
ABLIT=1 ./start.sh restart
Fidelity checks (logged per layer): mean rel_l2 vs stock ≈ the published
fingerprint (Mean Δrel 0.126; their L44 outlier 0.74 matches our measured
0.737). Their own gate: refusal32 → 32/32 bypass on the NVFP4 stack.
Why not direction orthogonalization (ABLIT_METHOD=proj)? Measured: the
shipped direction vectors (refusal_direction_glm53_*.pt) are statistically
random against the stock o_proj (max |rᵀW| 0.084 vs 0.084 for random unit
vectors; mean identical). The publisher's own table shows projection variants
(Blackfrost V / Dealign V SVD) topping out at 9/32 bypass — only the byte-copy
hits 32/32. proj stays available for custom directions you extract yourself:
W' = (I - alpha * r rᵀ) W # ABLIT_ALPHA, default 3.0 (alpha_ref)
Mechanics (both methods): o_proj is native BF16 and RowParallelLinear shards
only the input dim, so edits are per-rank row-space ops with no collectives —
identical result on both TP ranks. Applied by overlay/ablit_runtime.py at
the end of Glm5NextModel.load_weights / Glm5NextMTP.load_weights, before
CUDA-graph capture. The DFlash2 drafter is never touched (for the checkpoint's
own MTP block, ABLIT_INCLUDE_MTP=1 transplants its o_proj too — the
publisher found a stock MTP draft head keeps proposing refusals).
Disable with ABLIT=0 (or unset) — hook is a no-op, stock weights. No
rebuild: artifacts + hook are bind-mounted into both containers every start.
| Knob |
Default |
What |
ABLIT |
0 (off) |
1 = apply the o_proj edit at load (both ranks). Default and unset = stock weights |
ABLIT_METHOD |
auto |
auto = transplant when ablit/transplant/ is populated, else proj | transplant | proj |
ABLIT_LAYERS |
15-45 |
inclusive range; 45 is the checkpoint MTP block |
ABLIT_ALPHA |
3.0 |
proj-only: projection scale (1.0 = plain projection) |
ABLIT_DIRECTION |
dealign |
proj-only: dealign | bf_oproj | path to a custom direction .pt |
ABLIT_INCLUDE_MTP |
1 |
also edit the MTP block's o_proj when it loads (SPEC_METHOD=mtp) |
Caveats: the KLD quality panel above was measured without ablit; expect
behavioral drift and re-run tests/bench_decode.py after enabling (DFlash2
acceptance can shift). Donor licensing: Dealign weights carry their own
terms — see the donor card before redistributing anything derived.
Why the overlay exists
Stock vllm/vllm-openai:glm53-flash-arm64-cu130 loads this checkpoint and dies on
the first forward: pe_dim must be 64 for fp8_ds_mla. GLM-5.3-Flash is NoPE MLA
(qk_rope_head_dim=0, kv_lora_rank=512). On SM12x the only sparse-MLA backend is
FLASHINFER_MLA_SPARSE_SM120, whose packed record is 512 NoPE + 16 B scales + 128 B
RoPE (656 B). The overlay zero-pads the 512-d latent into that GLM_NSA geometry
(RoPE pad is zeros; the QK dot is unchanged) and registers a real EXL3 method so
routed experts stay packed instead of expanding to BF16.
Registering the name "exl3" is not enough. Experts must stay trellis + suh +
svh + mcg and run Trellis/MCG. Shared experts, attention, embeddings, and
lm_head stay native. TP=2 shards gate/up column-wise and down row-wise;
the MoE runner all-reduces once per layer.
DFlash2 on this fork also needs three GLM-specific hooks the stock image lacks:
EAGLE3 aux capture at mHC (hc_post then hc_contract → 4096-wide, taps log as
(6, 15, 25, 34, 43)), drafter SWA padded slot-share onto the MLA tensors
(block_size=64, `page_siz
A deployment recipe and Docker image that serves the GLM-5.3-Flash model as an OpenAI-compatible vLLM endpoint across 2-4 NVIDIA DGX Spark (GB10) kits, using a custom EXL3/TR3 4-bit quantization of the weights.