README проекта с github.com/sybil-solutions/inference-optimisation-skill. Одним файлом markdown: README.md.
An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:
It works under a memory cap, and every result has to survive a full-context stability check.
It encodes how we actually got these speedups. It is not a list of flags. The workflow:
It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.
Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.
| Hardware | Model / quant / engine | Before → after |
|---|---|---|
| Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) | Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP | Decode, 1 user: 43.6 → 53 tok/s (median) Decode, 4 users: 84.7 → 103 tok/s Prefill 32K: 1,037 → 1,660 tok/s Prefill 256K: 297 → 628 tok/s (2.1×) KV cache: 264K tokens → 4 × 256K Startup: 113 → 43 s Registry Lab gates: decode 28.6 → 41.0, prefill 255 → 667 |
| Intel Arc Pro B70 | Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels | Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on) Same card with llama.cpp Q4_K_M: 25.0 Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911 Agent-session replay: 193 → 147 s |
| Intel Arc Pro B70 | Same model, SGLang + the same kernels | Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day |
| RTX 3090 | Qwen3.8-27B EXL3 3 bpw + MTP | 96.2 tok/s prose / 141.1 code, 1 user |
| RTX 3090 | Qwen3.6-35B-A3B EXL3 3 bpw + MTP | 251.7 / 357.9 tok/s, 1 user |
| RTX 3090 | Flash-Next EXL3, 256K context | MTP: 45.5 → 65.6 tok/s |
| 4× H200 NVL | DeepSeek-V4.1-Flash, vLLM + speculative decoding | Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass |
Sources:
xpu/README.md, xpu/docs/PROGRESS.md and xpu/docs/SGLANG.md for B70cuda/README.md for the RTX 3090That run is the one this skill grew out of. Its ceiling calculation said:
The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.
┌────────────────────────────────────────────────────────────────┐
measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
└────────────────────────────────────────────────────────────────┘
│
baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
▲ │
│ queue.sh ctl NEW ctl (serialized, AC-only,
│ memory watchdog, ≥6 samples, medians)
│ │
PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
│ and regresses nothing by >3%
▼
long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
─▶ acceptance gates ─▶ publish
The skill has three parts:
SKILL.md: the procedure.
references/: loaded only when needed.
ceilings.md: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.methodology.md: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.levers.md: every lever with measured evidence. It is tagged general vs stack-specific.gotchas.md: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.publishing.md: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.
bench.py benchmarks any OpenAI-compatible server. It measures:
It flags looping output, failed requests and runs on battery. Token counts come from server usage stats, never from streamed chunks, and everything lands in ledger.jsonl.
ceiling.py computes the theoretical ceilings from a JSON of measured bandwidth and FLOPs plus component sizes.
queue.sh is a serialized, detachable run queue, for interleaving control/candidate/control runs.
mem.sh reports a container's real footprint. That includes driver residency on unified-memory GPUs, which cgroups don't see.
Claude Code (user-level):
git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/
Then ask, for example:
Other agents: point them at skills/inference-optimisation/SKILL.md, or paste PROMPT.md as the task prompt.
S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json # ceilings
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json # edit base_url/model
python3 $S/scripts/bench.py run configs/base.json --long 131072 # one run
bash $S/scripts/queue.sh base cand base # interleaved A/B
python3 $S/scripts/bench.py show # ledger table
PLAN.md is the list of things that did not work and why.PRs with new levers, gotchas or results are welcome. Each needs the hardware, model, quantization, engine version, before → after numbers, how they were measured, and a link to raw data.
MIT