resin README

项目的 README,来自 github.com/ekzhang/resin。合并为一个 markdown 文件:README.md。

Resin

Resin is not an inference engine. It's a library of building blocks for LLM inference.

At its core is the headless Engine: a fast Rust scheduler, KV cache, and metrics server. You then write the forward pass as model-specific kernels, optimized for your hardware.

token IDs ──► Rust Engine ──► frozen BatchPlan ──► Python Runner ───► token IDs
                  ▲                                    │
                  └──────── ownership & commit ────────┘

Resin also has optional HTTP serving and Prometheus metrics.

[!WARNING] This is not production-quality software. If you're looking for an actual inference engine, please use SGLang or vLLM. Read on if you want to hack on LLM inference systems, new models or hardware.

With the right kernels, this makes it possible to autonomously "vibe code" even the most complex models. In 12 hours, we used Resin to build standalone engines for DeepSeek-V4.1-Flash and GLM-5.3-Flash on Blackwell GPUs that rival vLLM and SGLang.

Prefill Decode
DeepSeek V4.1 Flash prefill DeepSeek V4.1 Flash interactivity-throughput curve
Prefill and decode for DeepSeek V4.1 Flash, within 5-10% of DeepInfer.

Features

Start with the two-page architecture overview.

Getting started

Either fork the repository or import resin and resin_models_cuda as standalone packages. Then use the following skills:

Importing resin does not import PyTorch or initialize CUDA.

Installation

Install the core library directly from GitHub with Python 3.11+ and Rust installed:

pip install "git+https://github.com/ekzhang/resin.git"

Pip builds the Rust extension locally. The import remains resin. Use @<commit> after .git to pin a revision.

Model runners are separate packages: resin-models-cuda for optimized CUDA implementations and resin-models-torch for reference implementations. The repository's development workspace installs both; the examples below assume that workspace or a separately installed runner package.

Development

Resin requires Python 3.11+ and Rust.

uv run scripts/rust_build_and_types.py

uv run syncs the project dependencies automatically. The build command then syncs type-library sources and builds the Rust extension with generated stubs.

The Modal harnesses are launched from the repository root. Install the development dependencies first, then authenticate with Modal once if this is a new machine:

uv run modal setup

modal setup is interactive and only needs to be repeated when the local Modal credentials change.

All models share one benchmark CLI and one parameterized serving app. The model registry in deploy/models.py supplies checkpoint revisions, hardware, parallelism, and model-specific runner construction.

# Populate the pinned checkpoint without allocating a GPU.
uv run scripts/bench.py --backend modal --model gpt-oss-20b --download-only

# Sweep prefill and decode, save traces, and plot the measured results.
uv run scripts/bench.py --backend modal --model gpt-oss-20b \
  --context-lengths 4096,8192 --batch-sizes 1,2,4,8 \
  --output bench/gpt-oss-20b.jsonl --trace-dir bench/traces --plot

For local GPUs, use --backend local --checkpoint /path/to/checkpoint; models with a separate draft checkpoint also accept --assistant-checkpoint. The local backend is the default. Modal submits the repository's source into the shared CUDA image, so the client machine does not need CUDA installed.

Prefill charts compare complete-context latency with input tokens/s/GPU; decode charts compare per-user output tokens/s with output tokens/s/GPU. Draft acceptance uses normal verification; the plotter's --accept-rate projects other acceptance rates. Model routing is the default; --balanced-experts enables synthetic balanced routing. See benchmarking for the timing protocol and trace scope.

For an already running endpoint, scripts/check_server.py URL --output REPORT checks serving correctness, prefix-cache reuse, context-size TTFT, and optional image/reasoning/stress cases. It can also evaluate a small local MMLU CSV or JSONL subset with --eval-file (defaults: --eval-limit 100, --eval-concurrency 32); see server checks.

Minimal use

from resin import Engine, SamplingParams
from resin_models_torch.lfm2 import Lfm2TorchRunner

runner = Lfm2TorchRunner(device="cpu")

with Engine(runner, context_length=32768) as engine:
    request = engine.submit(
        [1, 6, 24131],
        sampling_params=SamplingParams(
            max_output_tokens=8,
            stop_token_ids=[runner.eos_token_id],
            temperature=1.0,
            top_p=0.95,
        ),
    )
    while engine.has_work():
        for event in engine.step():
            print(event)
    generated_ids = request.take_result()

SamplingParams is the frozen request policy for generation limits, stop token IDs, sampling, and logprob options. Use the recipe's stop_token_ids for chat requests, or supply the model's stop tokens explicitly for raw token requests. Engine has no default EOS token.

A custom synchronous runner only needs one method:

import numpy as np

from resin import BatchPlan, ForwardOutput


class Runner:
    def forward(self, plan: BatchPlan) -> ForwardOutput:
        return ForwardOutput(
            token_ids=np.zeros(plan.num_sequences, dtype=np.uint32),
            token_indptr=np.arange(plan.num_sequences + 1, dtype=np.uint32),
        )

ForwardOutput carries flat token IDs and row offsets, with optional logprob columns. The same result type covers ordinary and speculative decoding.

Managed history and recurrent state are cache groups declared through a frozen RunnerConfig. The Engine binds their one physical arena during construction. Pass prefix_cache=True to reuse aligned prefixes across requests.

Local HTTP server

The built-in native server exposes one Engine through OpenAI Chat Completions and Responses, Anthropic Messages, and a small demo chat page. Model-specific prompt rendering and incremental response parsing live in the independent resin-recipe Rust crate.

# Run one LFM2 prompt in the terminal.
uv run python scripts/lfm2_chat.py

# Serve LFM2 locally over HTTP.
uv run python scripts/lfm2_serve.py

Then open http://127.0.0.1:8000. The page reports prompt and generated token counts, elapsed generation time, and output tokens per second. The same process serves /v1/chat/completions, /v1/responses, /v1/messages, and /v1/models. Both examples explicitly use the CPU Torch runner.

Modal HTTP endpoints

deploy/modal_serve.py serves every registered model. Select RESIN_MODEL at launch; each selection has a separate app and endpoint, with shared download, warmup, process management, and HTTP startup code. Populate its pinned checkpoint before deploying:

RESIN_MODEL=gpt-oss-20b uv run modal run deploy/modal_serve.py
RESIN_MODEL=gpt-oss-20b uv run modal deploy deploy/modal_serve.py

RESIN_MODEL=deepseek-v41 uv run modal run deploy/modal_serve.py
RESIN_MODEL=deepseek-v41 uv run modal deploy deploy/modal_serve.py

Use uv run scripts/bench.py --help to see the current model keys accepted by RESIN_MODEL and the benchmark CLI. The shared MODEL_SPECS registry defines each selection's checkpoint, hardware, and serving limits.

MLX runner (Apple Silicon)

Model runners are not tied to CUDA. resin_models_mlx runs dense and MoE Qwen3.5 models on Apple Silicon with MLX, on the same core engine. For a small local model:

QWEN_CHECKPOINT="$(uv run hf download mlx-community/Qwen3.5-0.8B-8bit \
  --revision 87e768fbfa03994095f3d14527c80c5ae70c5758 --quiet)"
uv run scripts/mlx_serve.py --model qwen3.5-0.8b-mlx --checkpoint "$QWEN_CHECKPOINT"

The download uses Hugging Face's normal cache and returns its snapshot path.

Checks

cargo test --all-targets
uv run pytest -q
uv run ruff format --check .
uv run ruff check .
uv run ty check
uv run python scripts/cloc.py

GPU validation is model- and machine-specific; local tests use fakes or skip unavailable accelerator libraries.

The code-size report uses Tokei to count production Rust/Python by component, excluding comments, blank lines, tests, deployment tools, and generated stubs. Use the current report instead of a checked-in line-count snapshot.

For resident prefill/decode sweeps, optional fixed speculative acceptance and balanced routing, Perfetto traces, and throughput/latency charts, see benchmarking.

Acknowledgements

Resin was based on and inspired by systems from SGLang, vLLM, TokenSpeed, and DeepInfer.

resin (ekzhang/resin)

正在搜索数据库…

437 ·GitHub
这是什么?Resin 是一个实验性质的库,提供用于构建自定义大模型推理引擎的基础组件,它本身并不是开箱即用的推理引擎。
这是现成的推理引擎,还是需要我自己动手搭建的东西?它是一个不带现成界面的库:核心 Engine 提供 Rust 调度器、KV 缓存和指标服务器,而针对具体模型的前向计算内核需要你自己编写。
只是导入它就需要 PyTorch 或 CUDA 环境吗?不需要。仅导入 resin 本身不会加载 PyTorch,也不会初始化 CUDA;只有在安装并使用可选的模型运行器包时才需要 CUDA。
这个能用于生产环境吗?不能。作者明确提示它不是生产级质量,并建议如果需要真正的推理引擎,可以使用 SGLang 或 vLLM。
目前已经用 Resin 构建了哪些东西?团队仅用 12 小时,基于 Resin 在 Blackwell GPU 上为 DeepSeek-V4.1-Flash 和 GLM-5.3-Flash 分别构建了独立引擎,性能可与 vLLM 和 SGLang 相媲美,并且与 DeepInfer 相差在 5-10% 以内。
核心引擎开箱即用地提供了哪些功能?它包括:Rust 批处理调度器、基于 slab 分配的统一缓存池(KV/SWA/Recurrent)、推测解码(MTP、DSpark)、预填充/解码 CUDA 图,以及 TP/EP/CP/DP 并行。
我可以通过 HTTP 提供模型服务吗?可以,它内置了一个可选的原生服务器,提供 OpenAI Chat Completions/Responses、Anthropic Messages 接口,以及 Prometheus 的 /metrics 端点。
add-resin-model 和 optimize-resin-performance 这两个技能是什么?这两个都是引导式工作流:一个用于从零为新模型创建推理引擎,另一个用于反复进行性能分析并提升其性能。
作者
ekzhang
大小
7.9 MB
星标
437
更新
提交Nov 17
时长
3个月
提交 · 90 天