BigMoeOnEdge README

项目的 README,来自 github.com/Helldez/BigMoeOnEdge。合并为一个 markdown 文件:README.md。

BigMoeOnEdge

Latest release CI License

English · 中文 · 日本語 · Español · Français · Italiano

Run Mixture-of-Experts models larger than your device's memory. The experts are read from flash as the model needs them. Android, iOS, macOS, Windows and Linux, built on llama.cpp. Unified Streaming Framework - from phones, laptops, consumer and data-center GPUs - Supports any hardware, any OS, any model, any quant. Automated resource scheduling tailor made to optimize tok/s.

Qwen3.6-35B-A3B generating an answer on a 12 GB Android phone, in real time. The iPhone app on a 12 GB iPhone with Qwen3.6-35B-A3B loaded: a chat about a picture, the plan, the models and the settings, in real time.
12 GB Android phone
Qwen3.6-35B-A3B · 5-6 tok/s
12 GB iPhone
Qwen3.6-35B-A3B · 6 tok/s
Qwen3.6-35B-A3B generating an answer in the desktop app on a 16 GB Apple-silicon laptop, in real time, at 21 tok/s.
16 GB Apple-silicon laptop
Qwen3.6-35B-A3B · 18-24 tok/s

Qwen3.6-35B-A3B, larger than each device's memory; any MoE model in GGUF runs, at any quantization. Decode speeds are the Performance table's.

The agent on a 2021 laptop, no GPU: the viral voxel-pagoda prompt to the built-in agent on a ThinkPad with 16 GB of RAM, CPU only, and Qwen3.6-35B-A3B Q4_K_M. 988 lines at 2.3 tok/s, then the page in the app's preview; the speed-ups are marked on screen.

The built-in agent writing a voxel pagoda garden as one HTML file on a 2021 ThinkPad with 15 GB of RAM and no dedicated GPU, then the page in the app's preview, turning.

What it does

Performance

Hardware Memory Model On disk Experts from Decode tok/s
Android phone 12 GB DeepSeek V4 Flash 284B 91 GB flash 1
Qwen3.8-Flash-Next 125B 80 GB flash 2-4
Qwen3.6-35B-A3B 21 GB flash 5-6
iPhone 12 GB Qwen3.6-35B-A3B 21 GB flash 6
Apple-silicon laptop 16 GB Qwen3.8-Flash-Next 125B 76 GB flash 7-9
Qwen3.6-35B-A3B 21 GB flash 18-24
Windows laptop 16 GB Qwen3.6-35B-A3B 21 GB flash 5
Desktop GPU (NVIDIA) 12 GB + 128 GB RAM Qwen3.8-Flash-Next 125B 84 GB RAM, GPU cache 16-17
Desktop GPU (NVIDIA) 16 GB + 32 GB RAM Qwen3.8-Flash-Next 125B 66 GB flash, GPU cache 19-21
Desktop GPU (NVIDIA) 24 GB + 96 GB RAM Qwen3.8-Flash-Next 125B 66 GB RAM, GPU cache 70
Qwen3.6-35B-A3B 21 GB GPU 217
Desktop GPU (NVIDIA) 24 GB + 62 GB RAM Qwen3.6-35B-A3B 21 GB GPU 154
12 GB of it free Qwen3.6-35B-A3B 21 GB RAM, GPU cache 59-67
Gemma 4 26B-A4B 17 GB GPU 114
Laguna XS 2.1 33B 20 GB GPU 93-94
Desktop GPU (NVIDIA) 8 GB + 31 GB RAM Qwen3.8-Flash-Next 125B 66 GB flash, GPU cache 16
Qwen3.6-35B-A3B 21 GB RAM, GPU cache 37-44
Two desktop GPUs (NVIDIA) 2 × 16 GB + 60 GB RAM Qwen3.8-Flash-Next 125B 66 GB RAM, GPU cache 41-48
DeepSeek V4 Flash 284B 91 GB flash, GPU cache 5
Datacenter GPU (AMD, ROCm) 192 GB Qwen3.8-Flash-Next 125B 66 GB GPU 53

A range is the spread across runs. Method and every run: docs/benchmarks.md. To add your machine, run scripts/bench-report.sh model.gguf and open a benchmark report.

Install

From the latest release:

Open the app and pick a model under Models. An NVIDIA or AMD card gets its backend from Settings (docs/desktop-app.md).

Command-line tools only (bmoe-cli, bmoe-server):

curl -fsSL https://github.com/Helldez/BigMoeOnEdge/releases/latest/download/install.sh | sh
irm https://github.com/Helldez/BigMoeOnEdge/releases/latest/download/install.ps1 | iex

Use

bmoe-cli -m model.gguf --auto -p "Explain MoE routing."   # plan and run
bmoe-cli -m model.gguf --plan                              # print the plan
bmoe-server -m model.gguf -c 32768                         # the server and its web UI, port 8765

The server speaks the OpenAI API (/v1/chat/completions, /v1/responses) and the Anthropic one (/v1/messages), with streaming, tool calls and reasoning. Add --host 0.0.0.0 --api-key KEY to serve other machines. Reference: docs/server-api.md.

On a machine that stays on, register it as a service of your login instead: it starts with the machine, comes back with its model, and reloads it when a request finds none loaded (running as a service).

bmoe-server service install --host 0.0.0.0 -m model.gguf -c 65536   # systemd, launchd or Task Scheduler
bmoe-server service status
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="local")
reply = client.chat.completions.create(model="local", messages=[{"role": "user", "content": "Hello"}])
print(reply.choices[0].message.content)

Coding agents need a large context: a few times their first request. 32768 is enough for opencode, 65536 for Claude Code (sizing the context).

// opencode, ~/.config/opencode/opencode.json
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "bmoe": {
      "npm": "@ai-sdk/openai-compatible",
      "options": { "baseURL": "http://127.0.0.1:8765/v1", "apiKey": "local" },
      "models": { "<model id from /v1/models>": { "limit": { "context": 32768, "output": 8192 } } }
    }
  },
  "model": "bmoe/<model id from /v1/models>"
}
# Claude Code
export ANTHROPIC_BASE_URL=http://127.0.0.1:8765
export ANTHROPIC_API_KEY=local
export ANTHROPIC_MODEL=<model id from /v1/models>
export ANTHROPIC_DEFAULT_HAIKU_MODEL=$ANTHROPIC_MODEL
export CLAUDE_CODE_MAX_CONTEXT_TOKENS=65536
claude
# Codex CLI, ~/.codex/config.toml
model = "<model id from /v1/models>"
model_provider = "bmoe"

[model_providers.bmoe]
name = "bmoe"
base_url = "http://127.0.0.1:8765/v1"
wire_api = "responses"
env_key = "BMOE_API_KEY"   # export BMOE_API_KEY=local

Under Connect apps, the app writes the settings for opencode and Codex CLI and shows the ones for Claude Code.

The agent. The desktop app has its own agent: it works in a folder you choose, reads and edits files and runs commands, and asks before it changes anything unless you allow it. A conversation can also be run by OpenCode, Claude Code, Codex or Pi. See docs/agent.md.

How it works

For each token, the router of each layer picks a few experts. The engine looks for them in up to three tiers, each a cache for the one below:

An expert missing from VRAM is copied from RAM; one missing from RAM is read from flash. On a phone or an Apple-silicon Mac the GPU uses the same RAM as the CPU, so there are two tiers. A model that fits the card runs on it whole. Streaming only reads, so it does not wear the drive.

The engine uses llama.cpp's public API, so its quantizations, tokenizers and chat templates work unchanged, and an upgrade is a submodule bump. Which tensors are experts is read from the file (docs/which-models-stream.md). Design: docs/architecture.md.

The planner

The Plan screen on a 16 GB Apple-silicon laptop with Qwen3.6-35B-A3B Q4_K_M: experts stream from flash with a 6.6 GiB cache, the prompt runs on the GPU and the tokens on the CPU, 23.9 tok/s and 94% cache hits measured, and where the memory goes.

The planner reads the model's header and measures the machine: memory, storage speed, each CPU, GPU and NPU. Its rules name no device or platform, so a phone and a desktop get different plans from the same code. Values you set yourself are kept. See docs/hardware-planning.md.

The planner: the machine and the model go into the plan; the plan decides where each weight goes (VRAM, RAM or flash), which device reads the prompt and which one generates, and the cache sizes and threads; the first answer is measured and what is faster is kept for the next load.

Platforms

Android macOS Windows Linux iOS
Engine and CLI arm64 Apple silicon x86_64, arm64 x86_64, aarch64 arm64 (in the app)
App native app desktop app desktop app desktop app native app, built from source
Accelerators Hexagon NPU, Vulkan, OpenCL Metal Vulkan, CUDA, OpenCL (arm64), Hexagon NPU (arm64) Vulkan, CUDA, HIP Metal
Tested on 12 GB phone 16 GB laptop 16 GB laptop rented 12-24 GB NVIDIA and 192 GB AMD cards 12 GB iPhone

Details: docs/platforms.md.

How it compares

BigMoeOnEdge Strata ds4 colibri KTransformers
Android phone (NPU too) ✅ ❌ ❌ ❌ ❌
iPhone ✅ ❌ ❌ ❌ ❌
Mac ✅ ✅ ✅ ✅ ❌
Windows / Linux, CPU only ✅ ❌ Linux, for debugging ✅ ❌
NVIDIA ✅ ✅ ✅ ✅ ✅
AMD (ROCm) ✅ (MI300X too) Radeon RX and Strix Halo Strix Halo and RDNA Vulkan ✅
Any MoE model in GGUF ✅ few models few models its own format few models
Configures itself ✅ --auto picks the model by memory expert-cache budget coli tune ❌

Speeds and sources: docs/comparison.md.

Build from source

git clone --recursive https://github.com/Helldez/BigMoeOnEdge.git
cd BigMoeOnEdge
cd ui && npm ci && npm run build && cd ..
scripts/build-host.sh
build/bin/bmoe-server --open

A llama.cpp backend is a CMake flag passed through: scripts/build-host.sh -DGGML_VULKAN=ON (or -DGGML_CUDA=ON). Tests: cd build && ctest --output-on-failure (needs python3 with gguf). Android: scripts/build-android.ps1, then the APK in examples/android. More: CONTRIBUTING.md.

Documentation

docs/. Start with hardware-planning.md, settings.md, server-api.md and limitations.md.

Prior art

AirLLM, Apple's LLM in a flash, FlexGen, PowerInfer, EdgeMoE and flash-moe, a Metal engine that streams a 397B MoE from SSD on Apple silicon.

License

Apache-2.0 (LICENSE). The binaries and apps also carry llama.cpp, ggml and other libraries under MIT, BSD, Apache-2.0 and SIL OFL 1.1: THIRD_PARTY_LICENSES.

BigMoeOnEdge (Helldez/BigMoeOnEdge)

正在搜索数据库…

这是什么?BigMoeOnEdge 是一个自适应流式推理引擎,它能运行比设备内存大好几倍的 MoE(Mixture-of-Experts,混合专家)大语言模型。它只在内存中保留每个 token 所需的部分,其余数据从存储中流式读取。输出结果与把整个模型完全加载进内存运行时完全一致。它还基于 llama.cpp 提供一个兼容 OpenAI 接口的本地 API。
这是独立应用,还是需要集成到我自己的代码里?这是一个可以直接安装运行的真正应用,而不是需要你写代码调用的库。它包括适用于手机的原生 Android 应用,以及适用于 macOS、Windows 和 Linux 的 bmoe-desktop 程序包(内含命令行工具和服务器)。任何兼容 OpenAI 接口的客户端(包括编程智能体)都可以连接它的本地服务器。
我该怎么让它跑起来?在 Android 上,从最新版本安装 APK,然后在“Models”下选择一个模型。在 macOS、Windows 或 Linux 上,下载 bmoe-desktop 压缩包并运行 `bmoe-server --open`,这会打开一个网页应用,包含 Chat、Models、Plan、Settings 和 Metrics 几个页面。
现在任何 MoE 模型和任何硬件都能用吗?只要 llama.cpp 能加载的 MoE GGUF 模型,理论上都能正确流式运行,因为路由专家张量是从文件头中自动识别的。但目前 iOS 版本只能在模拟器中运行。此外,CUDA、HIP、SYCL 等额外的加速后端是以独立模块形式加载的,需要你自己编译。
它会使用我的 GPU 或 NPU,还是只用 CPU?它会自动选择设备上可用的加速器:在 Android 上可用 NPU(Hexagon)、Vulkan 或 OpenCL;在 macOS 上使用 Metal;在 Windows 和 Linux 上使用 Vulkan。第三方的 CUDA、HIP、SYCL 模块可以用与内置后端相同的方式添加进来。
自适应流式优化器到底做什么?内置的规划器会测量设备的具体情况(内存、存储速度,以及每个 CPU、GPU 和 NPU),决定哪些数据常驻内存、哪些从磁盘流式读取,并在单独的 Plan 页面上显示每个选择背后的理由。
在真实手机上速度怎么样?在一台 12 GB 内存的 Android 手机上,一个 125B 参数的 MoE 模型(磁盘占用 80 GB,是其内存的 6.7 倍)解码速度为 2.2–3.5 tok/s;较小的 30B-A3B 模型约为 5.2 tok/s。
为什么说它是无损的,而不是近似结果?项目把正确性作为测试来对待:ctest 会验证流式输出与把所有专家完全放进内存运行的结果一致,测试覆盖缓存、淘汰、重叠执行、会话以及拆分模型等场景。
我能把它和 opencode 之类的编程智能体一起用吗?可以。任何兼容 OpenAI 接口的客户端都能连接本地服务器,README 中还专门演示了如何通过 baseURL 将它配置为 opencode 的 provider。
作者
Helldez
大小
31.9 MB
星标
1K
时长
4个月
提交 · 90 天