slotstream README

The project's README from github.com/carloslfu/slotstream. As one markdown file: README.md.

slotstream

Latest release GitHub stars

Run a 105 GB AI model on a 48 GB Mac.

Slotstream runs Qwen3.8-Flash-Next on your Mac by keeping most of the model on SSD and loading the parts it needs into memory. After a one-time download, the model works offline. No Python or cloud API account is required.

Use it to chat, ask about pictures, or work with files through an agent such as Hermes. Developers can connect their own apps through its APIs or Swift library.

Get started · Performance · Guides · Get help

I'm building Sevra on Slotstream: private, personal AI optimized for your computer. Sevra will choose a tested model for your hardware and keep that choice current as models improve, with inference, memory and tools tuned together. You'll control what it remembers and can access. The app is in development, starting with Mac. See Sevra and join the waitlist. Slotstream's command-line tool, APIs, and Swift library remain independently usable.

Will it run on my Mac?

You need an Apple Silicon Mac, macOS 14 or later, and about 110 GB of free SSD space. Choose Apple menu → About This Mac to check your chip and memory.

Smaller Macs can run the model too, but speed depends on the chip, SSD, and available memory. An 8 GB Mac needs swap and can slow down the whole computer. The installer has been tested on macOS 14 and 15; model runs have been tested on macOS 26. Windows, Linux, and Intel Macs are not currently supported.

Speed

These reply speeds were measured on real Macs after the model warmed up. tok/s means tokens per second; a token is a small piece of text, often part of a word.

Mac Memory Reply speed
MacBook Pro, M5 Pro 48 GB ~12 tok/s
Mac mini, M2 (base storage) 16 GB 1.41 tok/s
MacBook Air, M5 32 GB 6.22 tok/s
MacBook Pro, M5 Max 128 GB ~21–22 tok/s

The M5 Pro is the development Mac; the other rows are community reports. They use different releases and settings, including speculative decoding on the M5 Max. See hardware results and test conditions for credits, methods, and separate estimates for other Macs.

Starting a reply takes additional time. Slotstream must first process your question and conversation history. Long prompts can take minutes; follow-up turns can reuse unchanged history. Terminal shows progress while you wait.

Install

Open Terminal and paste this command:

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

Run the same command to update. If slotstream isn't found afterward, open a new terminal window.

Use it

Check your Mac, then ask for a first reply:

slotstream doctor
slotstream run --prompt "Why is the sky blue?"

doctor checks memory and disk space without loading the model. The first run asks to download it, then prints a reply. This download can take hours, but you only need to do it once. Interrupted downloads resume when you try again. Follow the step-by-step setup for more help.

Guides

What would you like to do? Guide
Chat in Open WebUI or another app Connect a chat app
Work with files and tools through an agent Use Hermes
Ask about a picture Use an image
Use a coding agent Connect fx
Fix a problem, move the model, or uninstall Troubleshooting

Install chat apps and agents separately. They provide the interface and tools; Slotstream runs the model. Keep its server running while a connected app uses it.

For developers, the engineering guide links to the OpenAI- and Ollama-compatible API references, Swift library, command options, build instructions, and tests. Release notes show what changed.

How it works

Qwen3.8-Flash-Next is a mixture-of-experts model: generating each piece of text uses only a subset of its expert networks. Slotstream keeps shared weights in memory and reads the needed experts from SSD into a cache. Frequently used experts stay in RAM, reducing repeated disk reads.

The whole model stays available even though it doesn't all fit in memory. Slotstream chooses a memory target for your Mac and adjusts its cache as other apps need room. Cache size changes speed without removing experts from the model. The engineering explanation covers the implementation.

Status and limits

FAQ

Does it work offline?

Yes, after downloading the model. Inference runs on your Mac. Connected agents may still use internet services for web searches or other tools; their settings determine what those tools send.

Why doesn't Slotstream use all of my RAM?

Auto intentionally caps the total memory target at 33 GB, or 34.6 GB with speculative decoding enabled, even on larger Macs. This is our current best-supported balance of speed and memory use for this model: tests on the development Mac showed diminishing speed gains from a larger expert cache. We'll adjust the default as real measurements across hardware and workloads show a better tradeoff. More allocated RAM does not guarantee more speed.

If your Mac has spare memory, you can try a larger target. Stop any running server, then preview this example without loading the model:

slotstream doctor --memory-gb 40

If the plan fits the available memory and Metal working-set limit with headroom, start the server with the same target:

slotstream serve --memory-gb 40

Replace 40 with your chosen total-process budget in decimal GB. An explicit target keeps the cache fixed and disables automatic resizing, so leave room for macOS and other apps and watch memory pressure. See the memory options for details.

Will this wear out my SSD?

Generation reads the model files without rewriting them. macOS swap adds writes when memory runs short. Automatic memory sizing helps, but a small Mac or an oversized manual setting can still swap heavily.

Can I run it on Linux or Windows?

Support for AMD and NVIDIA on Windows and Linux is planned for Sevra. It isn't available in the current Slotstream engine.

Can I use a different model?

Not with Slotstream today. Its loader and memory planner are built for this model. See related projects for runtimes with different model and hardware support.

Why this exists

I wanted to run this model on my own Mac, but the standard loader exhausted memory before producing a reply. Slotstream grew out of that experiment. The published measurements include that failed load and the experiments that followed.

The project was also discussed on Hacker News. The questions and hardware reports from that discussion help guide the work.

Support

Report a bug if something doesn't work, or share your Mac's results to help others know what to expect. Reports are credited to their authors. Code and documentation contributions are welcome; see Contributing for the workflow.

Who made this

I'm Carlos Galarza. I work on efficient AI and Executable Rationality, making machine cognition explicit and runnable. I also help teams run open models on their own hardware and debug agent workflows. For help or consulting, email me.

Star history

Slotstream GitHub star history, updated weekly

The badge at the top shows the latest star count; this chart is updated weekly.

License

Slotstream is MIT-licensed. The model weights have their own Qwen community license. See credits for the model and code this project builds on.

slotstream (carloslfu/slotstream)

speculative-decoding
Searching Database…

477 ·GitHub
Slotstream
What is this?Slotstream lets you run a 105 GB AI model (Qwen3.8-Flash-Next, a 125B-parameter mixture-of-experts model) on Macs with only 16 to 64 GB of RAM, by streaming most of the model from SSD and caching the busiest experts in memory.
Is this a standalone server or a library I import?It is a native Swift binary that runs as a local server exposing OpenAI- and Ollama-compatible APIs, and it also ships a Swift library for developers who want to embed it in their own apps.
How do I install and run it?Open Terminal and paste the install command from the README (running it again later updates Slotstream), then use the doctor check and ask for a first reply, which triggers a one-time model download.
Do I need Python or a special GPU setup first?No Python is required - it is a single native Swift binary built on MLX and Metal, Apple's own GPU framework, so you just need an Apple Silicon Mac.
Does it run on any Mac?You need an Apple Silicon Mac, macOS 14 or later, and about 110 GB of free SSD space; Windows, Linux, and Intel Macs are not currently supported, and an 8 GB Mac will need swap and can slow the whole machine down.
Does it need an internet connection after setup?No, after the one-time model download inference runs entirely on your Mac; connected agents may still use the internet for their own tools like web search.
Can I use it with a different model instead of Qwen3.8-Flash-Next?Not yet - the loader and memory planner are built specifically for this one model, so swapping in a different model isn't supported today.
What coding agents and chat apps work with it?Because it exposes OpenAI- and Ollama-compatible APIs, it works with Claude Code, Codex, Open WebUI, and agent tools like Hermes.
How fast is it in practice?Speed depends heavily on the Mac: about 1.4 tok/s on a 16 GB Mac mini M2, roughly 12 tok/s on a 48 GB MacBook Pro M5 Pro, and up to ~21-22 tok/s on a 128 GB MacBook Pro M5 Max.
Why doesn't it use all my Mac's RAM even if I have plenty free?Auto mode caps the total memory target at 33 GB (34.6 GB with speculative decoding) because tests showed diminishing speed gains from a bigger expert cache; you can manually set a higher target if you want to try it.
Can multiple apps use it at the same time?No - Slotstream generates one reply at a time, so connected apps share the same running model instance.
What are similar projects?deltafin, Strata, project-maya, mlx-serve, Qwen3.8-Flash-Next, turbo-fieldfare, backburner, uzu
Which topics is it in?qwen3, on-device-ai, mlx, speculative-decoding, inference-engine, mixture-of-experts, openai-api, moe, local-ai, metal, llm-inference, qwen
Creator
carloslfu
Size
565.3 MB
Stars
477
Updated
CommitOct 16Releasev0.2.29Oct 13
Age
1mo
Commits · 90 Days