inference-optimisation-skill README

The project's README from github.com/sybil-solutions/inference-optimisation-skill. As one markdown file: README.md.

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:

It works under a memory cap, and every result has to survive a full-context stability check.

It encodes how we actually got these speedups. It is not a list of flags. The workflow:

  1. Compute the theoretical ceiling.
  2. Change one thing at a time.
  3. Measure against interleaved controls.
  4. Log every result, including the failures.
  5. Gate on long-context retrieval.
  6. Qualify a clean, pinned build before publishing.

It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.

What we've been able to accomplish

Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.

Hardware Model / quant / engine Before → after
Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP Decode, 1 user: 43.6 → 53 tok/s (median)
Decode, 4 users: 84.7 → 103 tok/s
Prefill 32K: 1,037 → 1,660 tok/s
Prefill 256K: 297 → 628 tok/s (2.1×)
KV cache: 264K tokens → 4 × 256K
Startup: 113 → 43 s
Registry Lab gates: decode 28.6 → 41.0, prefill 255 → 667
Intel Arc Pro B70 Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)
Same card with llama.cpp Q4_K_M: 25.0
Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911
Agent-session replay: 193 → 147 s
Intel Arc Pro B70 Same model, SGLang + the same kernels Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day
RTX 3090 Qwen3.8-27B EXL3 3 bpw + MTP 96.2 tok/s prose / 141.1 code, 1 user
RTX 3090 Qwen3.6-35B-A3B EXL3 3 bpw + MTP 251.7 / 357.9 tok/s, 1 user
RTX 3090 Flash-Next EXL3, 256K context MTP: 45.5 → 65.6 tok/s
4× H200 NVL DeepSeek-V4.1-Flash, vLLM + speculative decoding Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass

Sources:

Reading the B390 result

That run is the one this skill grew out of. Its ceiling calculation said:

The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.

How it works

                ┌────────────────────────────────────────────────────────────────┐
  measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
                └────────────────────────────────────────────────────────────────┘
                                         │
    baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
                    ▲                                                     │
                    │                                    queue.sh ctl NEW ctl (serialized, AC-only,
                    │                                    memory watchdog, ≥6 samples, medians)
                    │                                                     │
   PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
          │                    and regresses nothing by >3%
          ▼
  long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
                                                                      ─▶ acceptance gates ─▶ publish

The skill has three parts:

SKILL.md: the procedure.

references/: loaded only when needed.

scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.

Install

Claude Code (user-level):

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then ask, for example:

Other agents: point them at skills/inference-optimisation/SKILL.md, or paste PROMPT.md as the task prompt.

Quick start without an agent

S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json        # ceilings
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json   # edit base_url/model
python3 $S/scripts/bench.py run configs/base.json --long 131072            # one run
bash $S/scripts/queue.sh base cand base                                     # interleaved A/B
python3 $S/scripts/bench.py show                                            # ledger table

Principles that carried the most weight

Contributing

PRs with new levers, gotchas or results are welcome. Each needs the hardware, model, quantization, engine version, before → after numbers, how they were measured, and a link to raw data.

License

MIT

inference-optimisation-skill (sybil-solutions/inference-optimisation-skill)

Searching Database…

What is this?It is an agent skill that hill-climbs the speed of a local LLM server: decode, speculative decoding, prefill, KV cache, startup and time to first token, on any hardware and quantization. It works with Claude Code and any agent that loads SKILL.md-style skills.
Is it an inference engine, or something else?Neither an engine nor a flag reference. It is a procedure (SKILL.md) plus reference docs and a small benchmark harness that the agent follows to tune your existing server.
How do I start using it?Install it as a user-level skill in Claude Code, then ask for something like 'hill-climb decode and prefill for the model on :8000 under a 40 GiB cap'. Other agents can read SKILL.md or use PROMPT.md as the task prompt.
What do I need on my machine first?A local OpenAI-compatible LLM server already running, plus Claude Code or another agent that loads SKILL.md-style skills. The bundled scripts need nothing beyond Python and bash.
Any traps I should know about before trying it?Single-run wins are often noise of ±5–10%, so the skill insists on interleaved controls and medians. Its published results come from specific Intel Arc and NVIDIA setups, so your own gains will differ.
Which hardware has it been tested on?Its results were measured on Intel Arc B390 and B70 GPUs, RTX 3090 and 4x H200 NVL. mem.sh also counts driver residency on unified-memory GPUs, which cgroups miss.
Why was it built instead of a list of flags?It encodes how the speedups were actually found, including the failures, so the agent computes theoretical ceilings first and avoids chasing targets above what the hardware allows.
How big were the speedups it was distilled from?On an Intel Arc B390 laptop running a 35B model, single-user decode went from 43.6 to 53 tok/s and 256K prefill from 297 to 628 tok/s, against a ceiling of roughly 670 tok/s.
Are the speed numbers trustworthy?Token counts come from server usage stats rather than streamed chunks, every run is logged to ledger.jsonl, and results must pass a full-context retrieval gate before they count as best.
Size
<0.1 MB
Stars
916
Updated
CommitNov 16
Age
2d
Commits · 90 Days