inference-optimisation-skill README

项目的 README,来自 github.com/sybil-solutions/inference-optimisation-skill。合并为一个 markdown 文件:README.md。

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:

It works under a memory cap, and every result has to survive a full-context stability check.

It encodes how we actually got these speedups. It is not a list of flags. The workflow:

  1. Compute the theoretical ceiling.
  2. Change one thing at a time.
  3. Measure against interleaved controls.
  4. Log every result, including the failures.
  5. Gate on long-context retrieval.
  6. Qualify a clean, pinned build before publishing.

It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.

What we've been able to accomplish

Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.

Hardware Model / quant / engine Before → after
Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP Decode, 1 user: 43.6 → 53 tok/s (median)
Decode, 4 users: 84.7 → 103 tok/s
Prefill 32K: 1,037 → 1,660 tok/s
Prefill 256K: 297 → 628 tok/s (2.1×)
KV cache: 264K tokens → 4 × 256K
Startup: 113 → 43 s
Registry Lab gates: decode 28.6 → 41.0, prefill 255 → 667
Intel Arc Pro B70 Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)
Same card with llama.cpp Q4_K_M: 25.0
Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911
Agent-session replay: 193 → 147 s
Intel Arc Pro B70 Same model, SGLang + the same kernels Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day
RTX 3090 Qwen3.8-27B EXL3 3 bpw + MTP 96.2 tok/s prose / 141.1 code, 1 user
RTX 3090 Qwen3.6-35B-A3B EXL3 3 bpw + MTP 251.7 / 357.9 tok/s, 1 user
RTX 3090 Flash-Next EXL3, 256K context MTP: 45.5 → 65.6 tok/s
4× H200 NVL DeepSeek-V4.1-Flash, vLLM + speculative decoding Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass

Sources:

Reading the B390 result

That run is the one this skill grew out of. Its ceiling calculation said:

The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.

How it works

                ┌────────────────────────────────────────────────────────────────┐
  measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
                └────────────────────────────────────────────────────────────────┘
                                         │
    baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
                    ▲                                                     │
                    │                                    queue.sh ctl NEW ctl (serialized, AC-only,
                    │                                    memory watchdog, ≥6 samples, medians)
                    │                                                     │
   PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
          │                    and regresses nothing by >3%
          ▼
  long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
                                                                      ─▶ acceptance gates ─▶ publish

The skill has three parts:

SKILL.md: the procedure.

references/: loaded only when needed.

scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.

Install

Claude Code (user-level):

git clone https://github.com/sybil-solutions/inference-optimisation-skill
mkdir -p ~/.claude/skills
cp -r inference-optimisation-skill/skills/inference-optimisation ~/.claude/skills/

Then ask, for example:

Other agents: point them at skills/inference-optimisation/SKILL.md, or paste PROMPT.md as the task prompt.

Quick start without an agent

S=skills/inference-optimisation
python3 $S/scripts/ceiling.py $S/examples/ceiling-ornith-b390.json        # ceilings
mkdir -p configs && cp $S/examples/config-existing-server.json configs/base.json   # edit base_url/model
python3 $S/scripts/bench.py run configs/base.json --long 131072            # one run
bash $S/scripts/queue.sh base cand base                                     # interleaved A/B
python3 $S/scripts/bench.py show                                            # ledger table

Principles that carried the most weight

Contributing

PRs with new levers, gotchas or results are welcome. Each needs the hardware, model, quantization, engine version, before → after numbers, how they were measured, and a link to raw data.

License

MIT

inference-optimisation-skill (sybil-solutions/inference-optimisation-skill)

正在搜索数据库…

这是什么?这是一个智能体技能,用于逐步迭代优化本地 LLM 服务器的速度,涵盖解码、投机解码、预填充、KV 缓存、启动速度和首 token 延迟,适用于任何硬件和量化方式。它可以与 Claude Code 以及任何能加载 SKILL.md 风格技能的智能体配合使用。
它是推理引擎,还是别的东西?两者都不是,也不是参数手册。它是一套流程(SKILL.md),加上参考文档和一个小型基准测试工具,智能体按照这套流程来调优你现有的服务器。
我该怎么开始用?在 Claude Code 中以用户级别安装该技能,然后提出类似这样的需求:'在 40 GiB 显存上限下,为 :8000 端口上的模型逐步优化解码和预填充'。其他智能体可以读取 SKILL.md,或将 PROMPT.md 用作任务提示词。
我的机器上得先准备什么?需要一台已经在运行、兼容 OpenAI 接口的本地 LLM 服务器,以及 Claude Code 或其他能加载 SKILL.md 风格技能的智能体。随附脚本只需要 Python 和 bash 即可运行。
试用之前,有没有需要注意的坑?单次运行的结果往往有 ±5–10% 的波动,因此该技能要求使用交错对照组并取中位数。它公布的结果来自特定的 Intel Arc 和 NVIDIA 环境,你自己的提速幅度会有所不同。
它在哪些硬件上测试过?其结果是在 Intel Arc B390 和 B70 GPU、RTX 3090 以及 4x H200 NVL 上测得的。mem.sh 还会统计统一内存 GPU 上的驱动常驻内存,这是 cgroups 无法统计到的。
它为什么不做成参数列表,而是做成这样?它记录了这些提速效果是如何真正找到的,包括失败的尝试,因此智能体会先计算理论上限,避免去追逐硬件本身无法达到的目标。
它所总结的那些提速效果有多大?在一台运行 35B 模型的 Intel Arc B390 笔记本上,单用户解码速度从 43.6 提升到 53 tok/s,256K 预填充速度从 297 提升到 628 tok/s,理论上限约为 670 tok/s。
这些速度数据靠谱吗?token 数来自服务器的用量统计,而不是流式输出的分块;每次运行都会记录到 ledger.jsonl 中;结果必须通过完整上下文的检索检验,才能被计入最佳成绩。
大小
<0.1 MB
星标
916
更新
提交Nov 16
时长
2天
提交 · 90 天