Edge0 README

The project's README from github.com/Edge0-AI/Edge0. As one markdown file: README.md.

edge0

edge0

An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction.

Hugging Face Hugging Face GitHub License

English | 中文

edge0 is an open-source streaming MoE inference framework. It generalizes the production-proven recipe — SSD expert offload + Recover-LoRA + prerouter routing prediction — into an extensible framework. The backend is isolated by design: the current MLX backend runs on Apple Silicon, and additional platforms (CUDA, …) plug into the same core abstractions.

Two model tiers ship with the framework. Each tier is an end-to-end release: the released checkpoint, the trained LoRA adapters, and the trained prerouter heads work together as one unit.

Tier Released checkpoint Inference profile
edge0-35b Edge0/Edge0-35B-A3B-preview 4-bit, 40 layers, 256 experts, prerouter K=4
edge0-8b Edge0/Edge0-8B-A1B-preview 4-bit, 24 layers, 128 experts, prerouter K=8

Both checkpoints are built on open sparse-MoE base models (Qwen3.5-MoE 35B-A3B and the Ling 3.0 bailing hybrid respectively) and ship with the LoRA and prerouter training done for this framework — the adapter files are co-located with each checkpoint and load automatically, so edge0 serve <tier> runs the trained pipeline out of the box.

Requirements

Design

Core mechanisms

Quick start

1) Install

# Python >= 3.10; the MLX backend requires macOS with Apple Silicon
python3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'

2) Download a model

The two tiers are published on Hugging Face — each repo bundles the base checkpoint and the trained LoRA + prerouter adapters in one directory, so a single download is a ready-to-run model:

# with the repo's helper (defaults to the two repos above):
.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models
.venv/bin/python scripts/fetch_models.py --tier edge0-8b --target-dir models

# or directly with the CLI:
.venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview     --local-dir models/edge0-35b
.venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview     --local-dir models/edge0-8b

Either way you end up with a directory like:

models/edge0-35b/
├── config.json, model-*.safetensors, tokenizer files   # base checkpoint
├── lora_edge0_35b.safetensors          # trained LoRA adapters
└── prerouter_edge0_35b.safetensors     # trained prerouter heads

3) Point edge0 at it

Tier names resolve to local directories via environment variables (where you put the download is up to you):

export EDGE0_35B_MODEL=$PWD/models/edge0-35b
export EDGE0_8B_MODEL=$PWD/models/edge0-8b

Or skip the env vars entirely and pass the directory directly — the tier is auto-detected from the checkpoint's config.json:

edge0 demo models/edge0-35b
edge0 serve models/edge0-8b

4) Run

# quick demo
edge0 demo edge0-35b

# serve (OpenAI-compatible /v1/chat/completions)
edge0 serve edge0-35b

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":32}'

# 5) One-shot chat (pass --max-new to cap length; add --show-thinking to
#    print the model's reasoning block too)
edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence."

python -m edge0 ... is equivalent to edge0 ....

Python API

from edge0 import AutoEngine
from edge0.server.chat import ChatMessage, ChatRequest, ChatSession

engine = AutoEngine.from_pretrained("/path/to/model")  # tier auto-detected
req = ChatRequest(
    model=engine.name,
    messages=[ChatMessage(role="user", content="Hello!")],
    max_tokens=64,
)
tokens, meta = ChatSession(engine, req).run()
print(engine._tok.decode(tokens))
engine.close()   # release mmaps / expert cache

examples/demo.py is the same minimal walkthrough (edge0 demo runs this exact path).

Models and adapters

Quality

All benchmarks were run by us with OpenCompass under identical settings and parameters for both the edge0 models (int4 + trained adapters + prerouter routing) and the original fp16 base models. The loss of the edge0 pipeline is small: 3.9 points on average for edge0-35b, 2.8 for edge0-8b (MMLU-Pro is even above the base). Max 100:

Benchmark edge0-35b (int4) Qwen3.5-MoE 35B-A3B (fp16) edge0-8b (int4) Ling 3.0 tiny (fp16)
AIME 2026 86.6 92.7 63.3 73.3
HumanEval 90.9 95.1 91.5 92.7
GPQA-Diamond 79.8 81.8 70.7 71.2
MMLU-Pro 81.0 84.6 70.1 65.8
IFBench 57.9 61.7 53.9 60.6
Average 79.2 83.2 69.9 72.7

Benchmark

Measured with examples/bench.py (3.3k-token prompt prefill → 10 sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier):

Tier Decode speed Prefill throughput (cold / warm)* Peak active memory** Test machine
edge0-35b 14.9–17.7 tok/s 113 / 140 tok/s 2.9 GiB Mac mini M4 Pro, 24 GB
edge0-8b 23.9–25.3 tok/s 500 / 1428 tok/s 1.0 GiB Mac mini M4 Pro, 24 GB

Cold = first request after process start (expert weights fault in from SSD); warm = subsequent requests (page cache resident). Prefill numbers are throughput over a ~3.3k-token prompt (BENCH_LONG=1).

*Peak active memory at short contexts (MLX allocator peak; expert weights stream from SSD via mmap and are not resident). Long contexts add KV cache: ~3.3 GiB on edge0-8b at 3.3k tokens.

Reproduce:

python examples/bench.py edge0-35b    # via $EDGE0_35B_MODEL
python examples/bench.py edge0-8b    # via $EDGE0_8B_MODEL

Tests

pytest                 # unit tests (no real weights)
EDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q
                        # real-weight generation; missing tiers are skipped
.venv/bin/python scripts/e2e_smoke.py \
  --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b
                        # staged vs exact consistency + generation smoke
scripts/generate_example.py   # full-pipeline API example
examples/demo.py       # minimal API walkthrough

Documentation

License

Apache-2.0, including vendored third-party code (see NOTICE).

Edge0 (Edge0-AI/Edge0)

Searching Database…

5.3K ·GitHub
Edge0
What is this?Edge0 is an open-source streaming MoE (Mixture-of-Experts) inference framework that lets you run large MoE models, like a 35B-parameter model, using only about 2.5-2.9 GB of active RAM by streaming expert weights from SSD and predicting routing one step ahead.
Do I use this as a Python library or run it as a server?Both: it offers a transformers-style Python API (AutoModel / AutoConfig / AutoEngine) and a CLI/server (edge0 serve, or python -m edge0) that runs the trained pipeline out of the box.
How do I get it running?Install edge0, download one of the two published tiers from Hugging Face (Edge0/Edge0-35B-A3B-preview or Edge0/Edge0-8B-A1B-preview), point edge0 at the downloaded directory via env vars or a direct path (the tier auto-detects from config.json), then run edge0 serve.
What hardware and Python version do I need?Python 3.10+ (3.12 recommended), and currently a Mac with Apple Silicon (M1/M2/M3/M4) — that is the only supported platform via the MLX backend right now.
Does it really run on iPhone, Android, and Windows like the description says?Not yet according to the README — only the MLX backend on Apple Silicon Macs is actually supported today; CUDA support is on the roadmap but other platforms are not implemented.
How much accuracy do I lose from the 4-bit quantization?Very little — OpenCompass benchmarks show an average drop of 3.9 points for edge0-35b and 2.8 points for edge0-8b versus the fp16 base models, with MMLU-Pro even scoring above the base model.
How much disk space do the models actually need?The 4-bit checkpoints are about 23 GB for edge0-35b and 4.2 GB for edge0-8b; expert weights are memory-mapped and read on demand rather than fully loaded into RAM.
What is Recover-LoRA?It's a LoRA adapter trained by distillation from the full-precision teacher model on top of a frozen int4 base, recovering most of the accuracy lost to 4-bit quantization while keeping the base model unmerged and read-only.
What does the prerouter actually do?A trained head predicts which experts will be needed one step ahead, so the next experts' weights can stream in from SSD while the current forward pass runs, giving up to +59% decode throughput.
How fast is decoding in practice?On a Mac mini M4 Pro, edge0-35b decodes at 14.9-17.7 tok/s and edge0-8b at 23.9-25.3 tok/s, measured with the included benchmark script.
Can I plug in my own base model?The framework separates backends and core logic behind a shared facade so new backends can be added, but only two published tiers (based on Qwen3.5-MoE 35B-A3B and Ling 3.0) currently ship with trained adapters.
What are similar projects?airllm, decider, needle, laya-coreml, MiniCPM, warp, backburner, openTPU
Which topics is it in?on-device-ai, edge-ai, language-model, moe, lora, llm-inference, quantization, local-llm, apple-silicon, artificial-intelligence, inference, huggingface
Creator
Edge0-AI
Size
4.2 MB
Stars
5.3K
Updated
CommitOct 16
Age
1mo
Commits · 90 Days