# inferssd (wici-ai/inferssd)
> InferSSD runs large mixture-of-experts language models that are bigger than your RAM or VRAM by streaming the experts from SSD. It is built on llama.cpp and uses standard GGUF files.
- GitHub: https://github.com/wici-ai/inferssd
- GitHub.Locker page: https://github.locker/wici-ai/inferssd
- Stars: 346
- Language: C++
- License: Apache-2.0
- Last push: 2026-11-06
## Questions and answers
Written by GitHub.Locker from the README and the repository; check the README below for details.
**How do I get it running?**
You install from source with the installer, which compiles the engine for your own CPU and GPU in about 2 to 12 minutes, then start a model with serve. Stop it with Ctrl+C or /exit in chat.
**What do I need installed first?**
Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools, and Linux needs systemd with cgroup v2 plus the CUDA toolkit if you have an NVIDIA GPU.
**What is the catch?**
There are no prebuilt packages yet, so every install compiles from source. Only three models are supported, and the model file downloads are huge.
**Does it work on Windows?**
Not yet: Windows support is still in progress. For now it targets Apple Silicon macOS and Linux with an NVIDIA GPU.
**Which hardware has it been tested on?**
The README tests a MacBook Pro with an Apple M5 Max and 48 GB unified memory, a PC with an RTX 5060 Ti, a PC with an RTX 5090, and an NVIDIA DGX Spark with 128 GB unified memory. Each has an SSD of 2 TB or more.
**Is there a desktop app?**
The README mentions a WiCi AI desktop app that installs the head of main and offers its newer commits as updates. The setup described here is the command-line engine itself.
---
## README (checked against GitHub 2026-11-07)
Source: https://github.com/wici-ai/inferssd/blob/main/README.md
Large MoE models, powered by your SSD
Website ·
Blog ·
Slack ·
Apache-2.0
macOS / Metal · Linux / CUDA · Windows (in progress)
**Run models larger than your RAM or VRAM.** InferSSD builds on [llama.cpp](https://github.com/ggml-org/llama.cpp) to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It currently supports **DeepSeek-V4.1-Flash-1017**, **Muse-Spark-Open** and **Qwen3.6-35B-A3B**, with more models coming.
## Models
These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See [weight-format details](docs/benchmark.md#checkpoint-formats).
| Model | Parameter count | GGUF reference |
| --- | --- | --- |
| DeepSeek-V4.1-Flash-1017 | 763B total / 8–16B active | [mrfakename · Q8_0](https://huggingface.co/mrfakename/DeepSeek-V4.1-Flash-1017-GGUF) |
| Muse-Spark-Open | 816B total / 42B active | [Unsloth · UD-Q4_K_XL](https://huggingface.co/unsloth/Muse-Spark-Open-816B-A42B-GGUF) |
| Qwen3.6-35B-A3B | 35B total / 3B active | [Unsloth · UD-Q4_K_XL](https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF) |
## Performance benchmarks
**4K prompt tier · 256 output tokens · decode measured over tokens 129–256.** All rates are in **token/s**; higher is faster. The multiple next to each **InferSSD** rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See [measurement details](docs/benchmark.md).
### Test hardware
| Hardware | CPU (cores) | GPU | Memory | SSD |
| --- | --- | --- | --- | --- |
| MacBook Pro | Apple M5 Max (18) | 40-core | 48 GB unified | 2 TB internal |
| RTX 5060 Ti | Intel Core 3 100U (6) | RTX 5060 Ti | 8 GB + 16 GB VRAM | 2 TB PCIe 4 ×4 |
| RTX 5090 | Intel Core Ultra 5 230F (10) | RTX 5090 | 32 GB + 32 GB VRAM | 2 TB PCIe 5 ×4 |
| DGX Spark | NVIDIA GB10 (20) | Blackwell | 128 GB unified | 4 TB PCIe 5 ×4 |
Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See [hardware details](docs/benchmark.md#hardware-details) for SSD models, CPU core layouts, and configuration notes.
### DeepSeek-V4.1-Flash-1017
| Hardware | Prefill (token/s) | Decode (token/s) |
|---|
| llama.cpp | Colibri | InferSSD | llama.cpp | Colibri | InferSSD |
|---|
| MacBook Pro | 5.929 | 0.708 | 28.766 (4.85×) | 0.873 | 0.437 | 2.874 (3.29×) |
| RTX 5060 Ti | 6.950 | 7.020 | 75.100 (10.81×) | 0.242 | 0.900 | 2.550 (10.54×) |
| RTX 5090 | 8.060 | 39.210 | 164.500 (20.41×) | 0.867 | 1.878 | 7.270 (8.39×) |
| DGX Spark | 2.730 | 0.826 | 127.500 (46.70×) | 1.608 | 0.356 | 7.260 (4.51×) |
### Muse-Spark-Open
| Hardware | Prefill (token/s) | Decode (token/s) |
|---|
| llama.cpp | Colibri | InferSSD | llama.cpp | Colibri | InferSSD |
|---|
| MacBook Pro | 3.140 | N/S | 14.503 (4.62×) | 0.153 | N/S | 1.097 (7.17×) |
| RTX 5060 Ti | 3.410 | N/S | 52.100 (15.28×) | 0.059 | N/S | 0.219 (3.71×) |
| RTX 5090 | 4.000 | N/S | 106.600 (26.65×) | 0.155 | N/S | 1.520 (9.81×) |
| DGX Spark | 1.740 | N/S | 83.900 (48.22×) | 0.120 | N/S | 2.140 (17.83×) |
**N/S:** Muse is not currently supported by Colibri.
## Get started
Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:
```sh
curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh
```
InferSSD currently installs from source, so the engine is compiled for your machine's CPU and GPU; prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.
The installer works in a private environment and uses no `sudo`. It needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs systemd with cgroup v2, and the CUDA toolkit on a machine with an NVIDIA GPU. Follow the printed PATH instruction if it shows one.
Then start a model:
```sh
# Chat in your terminal
inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
# Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080
inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF
```
For Muse, use `unsloth/Muse-Spark-Open-816B-A42B-GGUF`; for Qwen3.6, `unsloth/Qwen3.6-35B-A3B-GGUF`. With `serve`, open `http://127.0.0.1:8080` in your browser, or point any OpenAI-compatible client at `http://127.0.0.1:8080/v1`.
The first run downloads the model files (about 470 GiB for DeepSeek or Muse, 21 GiB for Qwen3.6) to `~/.local/share/inferssd/models` and measures your machine. Use `--model-dir /path/on/ssd` to choose the drive, or `-m /path/to/first-shard.gguf` for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with **Ctrl+C**, or `/exit` in chat.
See the [quick start](docs/quickstart.md#one-command-installation) for upgrades and uninstalling.
## Versions
InferSSD has a release number, `0.1.3` in `framework/pyproject.toml`, and every build has a
label of its own, `v..`: the branch's commit count up to the commit it was
built from, that commit's day (UTC) and the commit. The installer records the three in
`engine-resources.json` beside the build, from the clone's Git metadata, and so does a local
build that runs `scripts/engine-resources.py`; a source archive has no Git metadata and records
none. `inferssd --version` prints the release and the label, `inferssd --version --json` every
field:
```sh
$ inferssd --version
inferssd 0.1.3 (v202.1106.03af979)
$ inferssd --version --json
{"version": "0.1.3", "build": 202, "revision": "03af97…", "date": "2026-11-06", "branch": "main", "label": "v202.1106.03af979"}
```
The engine only says what it is. Whether a build is one it can work with is for whatever runs
the engine to decide; the WiCi AI desktop app, for one, installs the head of `main`, offers its
newer commits as updates, and keeps a floor of its own on `build`. Nothing here refers to it.
## Further reading
- [How it works](docs/architecture.md)
- [Benchmark methodology](docs/benchmark.md)
## Contributing
Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the [contributing guide](CONTRIBUTING.md).
## Acknowledgments and license
Built on [llama.cpp and ggml](https://github.com/ggml-org/llama.cpp), with thanks to their contributors and the model-adaptation community.
InferSSD is licensed under [Apache-2.0](LICENSE); bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see [NOTICE](NOTICE). Model weights remain subject to their publishers' licenses.