mesh-llm README

项目的 README,来自 github.com/Mesh-LLM/mesh-llm。合并为一个 markdown 文件:README.md。

Mesh LLM

Mesh LLM web console

Mesh LLM pools GPUs and memory across machines and exposes the result as one OpenAI-compatible API at http://localhost:9337/v1. Start one node, add more nodes later, and let the mesh decide whether a model runs locally, routes to a peer, or uses Skippy stage splits for models that are too large for one box.

Mesh and standalone Skippy

Skippy owns model management, native inference, split execution and OpenAI serving. Mesh adds peer discovery, transport, routing, plugins and the console around the same Skippy serving implementation.

The source lives in those two product trees. They share one root Cargo workspace and lockfile. Product crates, authored docs, scripts and evaluations live with their owner; Mesh also owns the website and SDKs. Root docs/ is generated website output.

Bare just (or just build) builds the standalone Skippy product (patched llama.cpp runtime and skippy CLI) first, then the MeshLLM host and console. Use just skippy to build Skippy alone, just mesh to rebuild only MeshLLM, or just skippy-cli-build for only the Skippy CLI. Skippy's separately packaged native runtime can be supplied with --runtime-bundle. See Skippy configuration and split serving.

Quick start

Install the latest release executable:

curl -fsSL https://meshllm.cloud/install.sh | bash

On Windows, use PowerShell:

irm https://meshllm.cloud/install.ps1 | iex

Install the Apple Silicon Homebrew formula with brew install Mesh-LLM/tap/mesh-llm. Versioned formulas, Ubuntu and Arch packages, checksums, SBOMs, and OCI images are produced by the public Mesh-LLM/mesh-packaging repository. See the platform install guides for the supported package matrix and install commands.

Finish setup:

mesh-llm setup

On Windows PowerShell, use mesh-llm.exe setup. (For native Windows notes, and for the optional WSL2 setup and multi-node LAN clustering, see the Windows & WSL2 Troubleshooting Guide.)

To remove an executable install later, preview the cleanup first:

mesh-llm uninstall --dry-run
mesh-llm uninstall --yes

Uninstall preserves ~/.mesh-llm configuration and identity data unless you explicitly pass --purge-config.

Join the public mesh and start serving:

mesh-llm serve --auto

That command chooses a backend flavor, downloads a suitable model if needed, joins the best discovered public mesh, starts the local API on port 9337, and starts the web console on port 3131.

Check available models:

curl -s http://localhost:9337/v1/models | jq '.data[].id'

Send an OpenAI-compatible request:

curl http://localhost:9337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"GLM-4.7-Flash-Q4_K_M","messages":[{"role":"user","content":"hello"}]}'

For server deployments, add --headless to hide the web UI while keeping the management API on the --console port:

mesh-llm serve --auto --headless

Pick the workflow you need

Goal Command Full guide
Try the public mesh mesh-llm serve --auto docs/MESHES.md
Start a private mesh mesh-llm serve --model Qwen3-8B-Q4_K_M docs/MESHES.md
Serve one model without mesh networking (debugging/dev) mesh-llm serve --local-model-only --gguf /models/model.gguf OpenAI API defaults to 127.0.0.1:9337 (--port and --listen-all change it)
Publish your own mesh mesh-llm serve --model Qwen3-8B-Q4_K_M --publish docs/MESHES.md
Join by invite token mesh-llm serve --join <token> docs/MESHES.md
Run an API-only client mesh-llm client --auto docs/MESHES.md
Run a big model with splits mesh-llm serve --model hf://meshllm/<repo>@<rev> --split docs/SKIPPY_SPLITS.md
Run DeepSeek V4 and other ds4 models with DwarfStar mesh-llm plugins install Mesh-LLM/ds4-plugin Setup and configuration
Attach a Flash-MoE SSD backend mesh-llm serve with [[plugin]] name = "flash-moe" docs/plugins/flash-moe.md
Automatically select a compatible model curl ... -d '{"model":"mesh", ...}' docs/plugins/virtual-models.md
Use Goose, OpenCode, Claude Code, or Pi mesh-llm goose, mesh-llm opencode, mesh-llm claude, mesh-llm pi docs/AGENTS.md
Build or contribute just build CONTRIBUTING.md

How the mesh works

For a deeper operator guide, see docs/USAGE.md. For every CLI command and switch, see docs/CLI.md.

Local model-only serving

Debugging and development endpoint. --local-model-only wires the OpenAI frontend directly to one local Skippy instance and bypasses mesh networking entirely. The supported path for OpenAI clients in production — including tool calls — is the regular mesh inference port, which includes the host-runtime normalizer. Use --local-model-only when you want a lightweight single-process setup without joining a mesh, not as a drop-in for the full serving path.

Use the direct topology when a process should expose one complete local model through the OpenAI API without becoming a mesh node:

mesh-llm serve \
  --local-model-only \
  --gguf /models/model.gguf \
  --port 9337

Use --gguf <absolute-path> to serve a specific GGUF file directly. --model <absolute-path> also works. Note that --model <catalog-name> does not work here — catalog and HF-ref resolution is a mesh function and is not available in the direct topology; the resulting error is not obvious. See the path requirements below.

This mode starts the OpenAI frontend and one local Skippy model runtime. It does not start QUIC, discovery, peer maintenance, split planning, plugins, release lookup, the web console, or the management API. Add --listen-all only when the OpenAI endpoint must bind beyond loopback. Startup fails if the complete model does not fit within detected local capacity (or --max-vram); it never falls back to distributed serving.

Values passed to --model, --gguf, and --mmproj must be absolute paths and must not be symlinks. If you downloaded a model through Hugging Face, resolve the cache symlink before passing the path: realpath <path-from-hf-cache>.

Automatic model selection (model: "mesh")

Send model: "mesh" (or the compatibility alias auto) to select one capability-compatible model. Streaming and non-streaming requests both use core single-model routing by default. The Mixture-of-Agents committee (crates/mesh-llm-moa-plugin) ships in the binary but is off by default; enable it with a [[plugin]] entry named mesh-moa in ~/.mesh-llm/config.toml, and non-streaming text chat to mesh is then answered by the committee instead (see docs/design/MOA_GATEWAY.md). Any installed virtual-model plugin can override mesh the same way.

Supported model families

Mesh LLM's Skippy runtime tracks llama.cpp family parity with reviewed GGUF representatives. The current reviewed support set covers 72 P0/P1 family rows, with 89 certified rows in the full parity inventory, including Qwen, Llama, Gemma, Mistral, DeepSeek, GLM, MiniMax, Phi, Granite, Hunyuan, EXAONE, Cohere, Falcon, RWKV, and many others.

Split multimodal serving is certified for Qwen2-VL, Qwen3-VL, Qwen3-VL-MoE, HunyuanOCR/Hunyuan-VL, and DeepSeek-OCR using real GGUF plus projector fixtures. DeepSeek3 and EXAONE-MoE use package-backed stages because the full GGUFs are too large for the cheap local baseline.

See docs/skippy/FAMILY_STATUS.md for the full artifact, split, wire dtype, cache policy, and exception matrix. See docs/skippy/LLAMA_PARITY.md for the remaining llama.cpp parity queue.

Install and build notes

Tagged releases publish macOS bundles plus Linux CPU, Linux ARM64 CPU, Linux ARM64 CUDA, Linux CUDA, Linux CUDA Blackwell, Linux ROCm, Linux Vulkan, Windows CPU, Windows CUDA, Windows ROCm, and Windows Vulkan bundles. Metal is macOS-only. Every flavor is composed from the same backend-neutral host for its OS/architecture plus one versioned native runtime. The Linux ARM64 CPU artifact is mesh-llm-aarch64-unknown-linux-gnu.tar.gz; the Linux ARM64 CUDA artifact is mesh-llm-aarch64-unknown-linux-gnu-cuda.tar.gz. In install and release contexts, arm64 and aarch64 mean the same 64-bit ARM target. Portable archives work offline: the host discovers the adjacent native-runtimes/<runtime-id> tree before consulting the user cache.

Build from source with just:

git clone https://github.com/Mesh-LLM/mesh-llm
cd mesh-llm
just build

Source builds require just, cmake, Rust, sccache, the platform fast linker (mold on Linux and lld on macOS/Windows), and Node.js 24 + npm. Cargo uses these defaults from the checked-in configuration, including plain cargo build. See CONTRIBUTING.md for installation commands and linker fallback behavior. To exercise the release boundary locally, build the neutral host and one runtime:

just release-host-build
just release-runtime-build metal # or cpu, cuda, rocm, vulkan
MESH_LLM_NATIVE_RUNTIME_BUNDLE_DIR="$PWD/dist/native-runtimes" \
MESH_LLM_NATIVE_RUNTIME_CACHE_DIR="$(mktemp -d)" \
  ./target/release/mesh-llm runtime list

CUDA runtimes need nvcc, ROCm runtimes need ROCm/HIP, and Vulkan runtimes need Vulkan development files plus glslc. See docs/design/NATIVE_RUNTIMES.md for the manifest, discovery, and compatibility contract.

The shipped mesh-llm executable uses embedded release attestation for provenance and admission hardening only. It does not apply to SDK, XCFramework, or other native artifacts, and it is not a runtime integrity proof. Verify a stamped packaged executable with cargo run -p xtask -- release-attestation inspect --binary <path-to-packaged-mesh-llm> --public-key-file <release-signing-public-key.json>. A packaged release binary reports valid, an unstamped local or dev build reports missing, and a binary that changed after packaging reports invalid. Bare inspect --binary ... is only enough to classify an unstamped binary as missing; stamped binaries require --public-key-file and otherwise report invalid with an explicit error. Post-download mutation can flip a stamped binary to invalid, but default startup still allows it.

🪟 Windows & WSL2 Troubleshooting

Running natively on Windows (NVIDIA)

Native Windows CUDA works, including on CUDA 13.x drivers: GPU detection (mesh-llm gpus) and full-speed CUDA inference have been verified on driver 610.74 (CUDA UMD 13.3) with an RTX 4070 Ti. The earlier advice to switch to WSL2 when mesh-llm reported 0 GPUs on CUDA 13 drivers predates the cuda12-runtime compatibility fix (#1127) and no longer applies to current releases.

The stable release works natively: the installer puts the product bundle and its CPU runtime in place, then add the CUDA runtime for your release and serve. No prerelease is needed, and no manual PATH edit for the runtime's DLLs:

irm https://meshllm.cloud/install.ps1 | iex
mesh-llm runtime install cuda
mesh-llm serve --local-model-only --model "C:\path\to\model.gguf"

Good to know on native Windows:

Running under WSL2 (alternative)

WSL2 remains a solid alternative if you prefer the Linux CUDA runtimes or want the multi-node LAN setups described below.

1. Install CUDA 13.0 Toolkit inside WSL2

Inside your Ubuntu WSL2 terminal, install cuda-toolkit-13-0 to supply libcudart.so.13:

wget https://developer.download.nvidia.com/compute/cuda/repos/wsl-ubuntu/x86_64/cuda-wsl-ubuntu.pin
sudo mv cuda-wsl-ubuntu.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/13.0.0/local_installers/cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo dpkg -i cuda-repo-wsl-ubuntu-13-0-local_13.0.0-1_amd64.deb
sudo cp /var/cuda-repo-wsl-ubuntu-13-0-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update && sudo apt-get -y install cuda-toolkit-13-0

echo 'export LD_LIBRARY_PATH=/usr/local/cuda-13.0/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

2. Enable Hyper-V & Windows Firewall for WSL2 Mirrored Mode

If you use WSL2 networkingMode=mirrored in %UserProfile%\.wslconfig, Windows 11 manages a separate Hyper-V VM Firewall that defaults to Block for inbound network traffic when third-party security software (e.g. Norton, McAfee) is present.

2.1 Configure Mirrored Networking (.wslconfig) Part 1

Create or edit C:\Users\<username>\.wslconfig on the Windows host:

notepad $env:USERPROFILE\.wslconfig
2.2 Configure Mirrored Networking (.wslconfig) Part 2
[wsl2]
networkingMode=mirrored
autoProxy=true
2.3 Restart WSL in Powershell

Restart WSL in PowerShell:

wsl --shutdown
2.4 Enable Windows Firewall Needs

To allow incoming LAN connections to the Web UI (3131) and P2P QUIC mesh transport (9337), run PowerShell as Administrator on the Windows host:

# 1. Allow Inbound Traffic through the WSL Hyper-V VM Firewall Container
$vmCreatorId = '{40E0AC32-46A5-438A-A0B2-2B479E8F2E90}'
Set-NetFirewallHyperVVMSetting -Name $vmCreatorId -DefaultInboundAction Allow

# 2. Allow MeshLLM Ports in Windows Defender Firewall
New-NetFirewallRule -DisplayName "MeshLLM TCP In" -Direction Inbound -Action Allow -Protocol TCP -LocalPort 3131,9337 -Profile Any
New-NetFirewallRule -DisplayName "MeshLLM UDP In" -Direction Inbound -Action Allow -Protocol UDP -LocalPort 9337,5353 -Profile Any

3. Match Model Paths for Direct LAN Reading

To ensure worker nodes load GGUF model shards directly off local NVMe/SSD storage without streaming tens of gigabytes over the network, ensure the --gguf file path string is identical across all nodes (or use symlinks/bind mounts):

# Example: Mount or symlink model path on worker nodes
sudo mkdir -p /mnt/d/models/
sudo mount --bind /path/to/local/fast/nvme/ /mnt/d/models/

# Launch Master and Worker with matching path strings
mesh-llm --llama-flavor cuda serve \
  --console 3131 \
  --gguf "/mnt/d/models/model.gguf" \
  --mesh-name "MainMesh" \
  --listen-all \
  --auto

Documentation hub

Doc Use it for
docs/MESHES.md Private meshes, public discovery, publishing, invite tokens, API-only clients
docs/SKIPPY_SPLITS.md Running big models with package-backed Skippy stage splits
docs/LAYER_PACKAGE_REPOS.md Contributing and publishing layer package repositories
docs/AGENTS.md Goose, Claude Code, OpenCode, Pi, curl, and blackboard
docs/EXO_COMPARISON.md Balanced comparison with Exo
docs/CLI.md Command reference and JSON automation
docs/USAGE.md Longer operational usage guide, runtime control, owner-control operator flows
docs/design/TESTING.md Testing playbook, mixed-version QA, remote deploy checks
docs/plugins/flash-moe.md Optional Flash-MoE SSD expert streaming backend setup
docs/plugins/dwarfstar.md DwarfStar (ds4) alternative engine setup on Apple Silicon
docs/skippy/FAMILY_STATUS.md Certified Skippy model-family status
docs/specs/layer-package-repos.md Manifest and artifact format spec
docs/specs/mesh-setup-installer.md Installer/bootstrap and setup command behavior spec

CI infrastructure

Depot

Trusted main Linux jobs may use Depot's managed GitHub Actions runners through the checked-in runner policy. Pull requests remain on GitHub-hosted runners until the cache isolation and protected-workflow gates in ci/DEPOT_MIGRATION.md are proven. Hardware-qualified GPU tests remain on dedicated runners.

Community

See MAINTAINERS.md for the Mesh maintainers and CONTRIBUTING.md to get involved.

Mesh LLM is experimental distributed-systems software. When you report bugs, include the command you ran, platform/backend flavor, /api/status output if available, and whether the node was private, published, or joined with --auto.

Support

The Mesh LLM project thanks spiral.xyz for their support.

mesh-llm (Mesh-LLM/mesh-llm)

正在搜索数据库…

这是什么?Mesh LLM 可以把多台机器上的 GPU 和内存资源汇集起来,并将整个集群以一个兼容 OpenAI 接口的 API 对外提供。这样,模型可以在本地运行,也可以转发给其他节点处理;如果模型太大,一台机器装不下,还能拆分到多台机器上运行。
这是一个单独的应用,还是需要我加入的节点网络?它是一个命令行程序(mesh-llm),你把它作为一个节点运行。各个节点通过 QUIC 加入 mesh 网络,共同对外提供一个兼容 OpenAI 接口的 API,并把请求转发给能够处理所请求模型的节点。
怎样才能让它运行起来?下载最新版本的可执行文件(或者在 Apple Silicon 芯片的 Mac 上运行 brew install Mesh-LLM/tap/mesh-llm),然后运行 mesh-llm serve --auto。该命令会自动选择后端,需要时下载模型,加入发现的最佳公共 mesh 网络,并在 9337 端口启动 API,在 3131 端口启动网页控制台。
有哪些需要注意的粗糙之处吗?在 v0.76.0-rc8 版本中,原生 Windows CUDA 仍存在三个分发和加载方面的 bug,导致开箱即用的原生运行方式无法使用,需要手动变通处理。另外,混合智能体(Mixture-of-Agents)网关(model: 'mesh')明确标注为实验性功能,其路由和错误处理行为可能还会变化。
它支持哪些操作系统和 GPU 后端?正式发布的版本提供 macOS(Metal)安装包,以及 Linux 的 CPU、ARM64、CUDA、ROCm、Vulkan 版本,还有 Windows 的 CPU、CUDA、ROCm、Vulkan 版本。
mesh 节点之间的通信是加密的吗?是的。QUIC 对所有节点之间的通信进行端到端加密,包括推理请求、响应以及模型拆分时的中间激活数据。中继服务器只转发加密数据包,无法读取其内容。
Skippy 是什么?Skippy 是 Mesh LLM 内部的原生推理运行时,负责模型管理、基于 llama.cpp 的推理、跨机器的拆分执行,以及兼容 OpenAI 接口的服务。
如果模型太大,一台机器装不下,会怎样?Mesh LLM 会把模型拆分成 Skippy 阶段:协调器在各个节点之间规划连续的层范围,先启动下游阶段,等待其就绪,然后发布阶段 0 的路由。这样,模型的服务方式就和在单台机器上运行一样。
model: 'mesh' 选项是什么?这是一个实验性的混合智能体网关。它会把一个提示词分发给 mesh 网络中当前的每一个模型,用确定性代码对各个回复进行仲裁(只有出现真正冲突时才会交给归约 LLM 处理),最后返回一个兼容 OpenAI 接口的响应。
支持哪些模型系列?Skippy 与 llama.cpp 的模型系列保持对齐,并审查了 72 个 P0/P1 级别的模型系列,包括 Qwen、Llama、Gemma、Mistral、DeepSeek、GLM、MiniMax、Phi、Granite、Hunyuan、EXAONE、Cohere、Falcon 和 RWKV 等。
作者
Mesh-LLM
大小
172.9 MB
星标
4.2K
时长
9个月
提交 · 90 天