inferssd README

The project's README from github.com/wici-ai/inferssd. As one markdown file: README.md.

Large MoE models, powered by your SSD

InferSSD

Website  ·  Blog  ·  Slack  ·  Apache-2.0

macOS / Metal  ·  Linux / CUDA  ·  Windows (in progress)

Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It currently supports DeepSeek-V4.1-Flash-1017, Muse-Spark-Open and Qwen3.6-35B-A3B, with more models coming.

Models

These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.

Model Parameter count GGUF reference
DeepSeek-V4.1-Flash-1017 763B total / 8–16B active mrfakename · Q8_0
Muse-Spark-Open 816B total / 42B active Unsloth · UD-Q4_K_XL
Qwen3.6-35B-A3B 35B total / 3B active Unsloth · UD-Q4_K_XL

Performance benchmarks

4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See measurement details.

Test hardware

Hardware CPU (cores) GPU Memory SSD
MacBook Pro Apple M5 Max (18) 40-core 48 GB unified 2 TB internal
RTX 5060 Ti Intel Core 3 100U (6) RTX 5060 Ti 8 GB + 16 GB VRAM 2 TB PCIe 4 ×4
RTX 5090 Intel Core Ultra 5 230F (10) RTX 5090 32 GB + 32 GB VRAM 2 TB PCIe 5 ×4
DGX Spark NVIDIA GB10 (20) Blackwell 128 GB unified 4 TB PCIe 5 ×4

Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.

DeepSeek-V4.1-Flash-1017

HardwarePrefill (token/s)Decode (token/s)
llama.cppColibriInferSSDllama.cppColibriInferSSD
MacBook Pro5.9290.70828.766 (4.85×)0.8730.4372.874 (3.29×)
RTX 5060 Ti6.9507.02075.100 (10.81×)0.2420.9002.550 (10.54×)
RTX 50908.06039.210164.500 (20.41×)0.8671.8787.270 (8.39×)
DGX Spark2.7300.826127.500 (46.70×)1.6080.3567.260 (4.51×)

Muse-Spark-Open

HardwarePrefill (token/s)Decode (token/s)
llama.cppColibriInferSSDllama.cppColibriInferSSD
MacBook Pro3.140N/S14.503 (4.62×)0.153N/S1.097 (7.17×)
RTX 5060 Ti3.410N/S52.100 (15.28×)0.059N/S0.219 (3.71×)
RTX 50904.000N/S106.600 (26.65×)0.155N/S1.520 (9.81×)
DGX Spark1.740N/S83.900 (48.22×)0.120N/S2.140 (17.83×)

N/S: Muse is not currently supported by Colibri.

Get started

Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

InferSSD currently installs from source, so the engine is compiled for your machine's CPU and GPU; prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.

The installer works in a private environment and uses no sudo. It needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs systemd with cgroup v2, and the CUDA toolkit on a machine with an NVIDIA GPU. Follow the printed PATH instruction if it shows one.

Then start a model:

# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

For Muse, use unsloth/Muse-Spark-Open-816B-A42B-GGUF; for Qwen3.6, unsloth/Qwen3.6-35B-A3B-GGUF. With serve, open http://127.0.0.1:8080 in your browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

The first run downloads the model files (about 470 GiB for DeepSeek or Muse, 21 GiB for Qwen3.6) to ~/.local/share/inferssd/models and measures your machine. Use --model-dir /path/on/ssd to choose the drive, or -m /path/to/first-shard.gguf for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with Ctrl+C, or /exit in chat.

See the quick start for upgrades and uninstalling.

Versions

InferSSD has a release number, 0.1.3 in framework/pyproject.toml, and every build has a label of its own, v<build>.<mmdd>.<commit>: the branch's commit count up to the commit it was built from, that commit's day (UTC) and the commit. The installer records the three in engine-resources.json beside the build, from the clone's Git metadata, and so does a local build that runs scripts/engine-resources.py; a source archive has no Git metadata and records none. inferssd --version prints the release and the label, inferssd --version --json every field:

$ inferssd --version
inferssd 0.1.3 (v202.1106.03af979)
$ inferssd --version --json
{"version": "0.1.3", "build": 202, "revision": "03af97…", "date": "2026-11-06", "branch": "main", "label": "v202.1106.03af979"}

The engine only says what it is. Whether a build is one it can work with is for whatever runs the engine to decide; the WiCi AI desktop app, for one, installs the head of main, offers its newer commits as updates, and keeps a floor of its own on build. Nothing here refers to it.

Further reading

Contributing

Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the contributing guide.

Acknowledgments and license

Built on llama.cpp and ggml, with thanks to their contributors and the model-adaptation community.

InferSSD is licensed under Apache-2.0; bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see NOTICE. Model weights remain subject to their publishers' licenses.

inferssd (wici-ai/inferssd)

Searching Database…

346 ·GitHub
Inferssd
What is this?InferSSD runs large mixture-of-experts language models that are bigger than your RAM or VRAM by streaming the experts from SSD. It is built on llama.cpp and uses standard GGUF files.
How do I get it running?You install from source with the installer, which compiles the engine for your own CPU and GPU in about 2 to 12 minutes, then start a model with serve. Stop it with Ctrl+C or /exit in chat.
What do I need installed first?Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools, and Linux needs systemd with cgroup v2 plus the CUDA toolkit if you have an NVIDIA GPU.
What is the catch?There are no prebuilt packages yet, so every install compiles from source. Only three models are supported, and the model file downloads are huge.
Does it work on Windows?Not yet: Windows support is still in progress. For now it targets Apple Silicon macOS and Linux with an NVIDIA GPU.
Which hardware has it been tested on?The README tests a MacBook Pro with an Apple M5 Max and 48 GB unified memory, a PC with an RTX 5060 Ti, a PC with an RTX 5090, and an NVIDIA DGX Spark with 128 GB unified memory. Each has an SSD of 2 TB or more.
Is there a desktop app?The README mentions a WiCi AI desktop app that installs the head of main and offers its newer commits as updates. The setup described here is the command-line engine itself.
What are similar projects?Strata, BigMoeOnEdge, decider, kapitsa, qwen38-mtp, project-maya, rizzo-flow, deltafin
Which topics is it in?llama-cpp, mixture-of-experts, gguf, metal, llm-inference, gpu-acceleration, quantization, large-language-models, openai-compatible, local-llm, apple-silicon, inference
Creator
wici-ai
Size
40.2 MB
Stars
346
Updated
CommitNov 06Releasev0.1.3Nov 05
Age
16d
Commits · 90 Days