slotstream README

The project's README from github.com/carloslfu/slotstream. As one markdown file: README.md.

slotstream

Latest release GitHub stars

Run a 105 GB AI model on a 48 GB Mac.

Slotstream runs Qwen3.8-Flash-Next on your Mac by keeping most of the model on SSD and loading the parts it needs into memory. After a one-time download, the model works offline. No Python or cloud API account is required.

Use it to chat, ask about pictures, or work with files through an agent such as Hermes. Developers can connect their own apps through its APIs or Swift library.

Get started · Performance · Guides · Get help

I'm building Sevra on Slotstream: private, personal AI optimized for your computer. Sevra will choose a tested model for your hardware and keep that choice current as models improve, with inference, memory and tools tuned together. You'll control what it remembers and can access. The app is in development, starting with Mac. See Sevra and join the waitlist. Slotstream's command-line tool, APIs, and Swift library remain independently usable.

Will it run on my Mac?

You need an Apple Silicon Mac, macOS 14 or later, and about 110 GB of free SSD space. Choose Apple menu → About This Mac to check your chip and memory.

Smaller Macs can run the model too, but speed depends on the chip, SSD, and available memory. An 8 GB Mac needs swap and can slow down the whole computer. The installer has been tested on macOS 14 and 15; model runs have been tested on macOS 26. Windows, Linux, and Intel Macs are not currently supported.

Speed

These reply speeds were measured on real Macs after the model warmed up. tok/s means tokens per second; a token is a small piece of text, often part of a word.

Mac Memory Reply speed
MacBook Pro, M5 Pro 48 GB ~12 tok/s
Mac mini, M2 (base storage) 16 GB 1.41 tok/s
MacBook Air, M5 32 GB 6.22 tok/s
MacBook Pro, M5 Max 128 GB ~21–22 tok/s

The M5 Pro is the development Mac; the other rows are community reports. They use different releases and settings, including speculative decoding on the M5 Max. See hardware results and test conditions for credits, methods, and separate estimates for other Macs.

Starting a reply takes additional time. Slotstream must first process your question and conversation history. Long prompts can take minutes; follow-up turns can reuse unchanged history. Terminal shows progress while you wait.

Install

Open Terminal and paste this command:

curl -fsSL https://raw.githubusercontent.com/carloslfu/slotstream/main/install.sh | sh

Run the same command to update. If slotstream isn't found afterward, open a new terminal window.

Use it

Check your Mac, then ask for a first reply:

slotstream doctor
slotstream run --prompt "Why is the sky blue?"

doctor checks memory and disk space without loading the model. The first run asks to download it, then prints a reply. This download can take hours, but you only need to do it once. Interrupted downloads resume when you try again. Follow the step-by-step setup for more help.

Guides

What would you like to do? Guide
Chat in Open WebUI or another app Connect a chat app
Work with files and tools through an agent Use Hermes
Ask about a picture Use an image
Use a coding agent Connect fx
Fix a problem, move the model, or uninstall Troubleshooting

Install chat apps and agents separately. They provide the interface and tools; Slotstream runs the model. Keep its server running while a connected app uses it.

For developers, the engineering guide links to the OpenAI- and Ollama-compatible API references, Swift library, command options, build instructions, and tests. Release notes show what changed.

How it works

Qwen3.8-Flash-Next is a mixture-of-experts model: generating each piece of text uses only a subset of its expert networks. Slotstream keeps shared weights in memory and reads the needed experts from SSD into a cache. Frequently used experts stay in RAM, reducing repeated disk reads.

The whole model stays available even though it doesn't all fit in memory. Slotstream chooses a memory target for your Mac and adjusts its cache as other apps need room. Cache size changes speed without removing experts from the model. The engineering explanation covers the implementation.

Status and limits

FAQ

Does it work offline?

Yes, after downloading the model. Inference runs on your Mac. Connected agents may still use internet services for web searches or other tools; their settings determine what those tools send.

Why doesn't Slotstream use all of my RAM?

Auto intentionally caps the total memory target at 33 GB, or 34.6 GB with speculative decoding enabled, even on larger Macs. This is our current best-supported balance of speed and memory use for this model: tests on the development Mac showed diminishing speed gains from a larger expert cache. We'll adjust the default as real measurements across hardware and workloads show a better tradeoff. More allocated RAM does not guarantee more speed.

If your Mac has spare memory, you can try a larger target. Stop any running server, then preview this example without loading the model:

slotstream doctor --memory-gb 40

If the plan fits the available memory and Metal working-set limit with headroom, start the server with the same target:

slotstream serve --memory-gb 40

Replace 40 with your chosen total-process budget in decimal GB. An explicit target keeps the cache fixed and disables automatic resizing, so leave room for macOS and other apps and watch memory pressure. See the memory options for details.

Will this wear out my SSD?

Generation reads the model files without rewriting them. macOS swap adds writes when memory runs short. Automatic memory sizing helps, but a small Mac or an oversized manual setting can still swap heavily.

Can I run it on Linux or Windows?

Support for AMD and NVIDIA on Windows and Linux is planned for Sevra. It isn't available in the current Slotstream engine.

Can I use a different model?

Not with Slotstream today. Its loader and memory planner are built for this model. See related projects for runtimes with different model and hardware support.

Why this exists

I wanted to run this model on my own Mac, but the standard loader exhausted memory before producing a reply. Slotstream grew out of that experiment. The published measurements include that failed load and the experiments that followed.

The project was also discussed on Hacker News. The questions and hardware reports from that discussion help guide the work.

Support

Report a bug if something doesn't work, or share your Mac's results to help others know what to expect. Reports are credited to their authors. Code and documentation contributions are welcome; see Contributing for the workflow.

Who made this

I'm Carlos Galarza. I work on efficient AI and Executable Rationality, making machine cognition explicit and runnable. I also help teams run open models on their own hardware and debug agent workflows. For help or consulting, email me.

Star history

Slotstream GitHub star history, updated weekly

The badge at the top shows the latest star count; this chart is updated weekly.

License

Slotstream is MIT-licensed. The model weights have their own Qwen community license. See credits for the model and code this project builds on.

slotstream (carloslfu/slotstream)

Searching Database…

476 ·GitHub
Slotstream
Что это такое?Slotstream позволяет запускать ИИ-модель размером 105 GB (Qwen3.8-Flash-Next — mixture-of-experts модель со 125B параметров) на Mac всего с 16–64 GB RAM, подгружая большую часть модели с SSD и кэшируя самые востребованные эксперты в памяти.
Это отдельный сервер или библиотека для импорта?Это нативный бинарник Swift, работающий как локальный сервер с API, совместимыми с OpenAI и Ollama, а также он поставляется с библиотекой Swift для разработчиков, желающих встроить его в свои приложения.
Как его установить и запустить?Откройте Терминал и вставьте команду установки из README (повторный её запуск позже обновит Slotstream), затем выполните проверку doctor и запросите первый ответ — это запустит одноразовую загрузку модели.
Нужны ли сначала Python или особая настройка GPU?Python не требуется — это единый нативный бинарник Swift, построенный на MLX и Metal, собственном GPU-фреймворке Apple, так что нужен просто Mac на Apple Silicon.
Работает ли он на любом Mac?Нужен Mac на Apple Silicon, macOS 14 или новее и около 110 GB свободного места на SSD; Windows, Linux и Intel Mac сейчас не поддерживаются, а Mac с 8 GB потребует swap и может замедлить всю машину.
Нужно ли интернет-соединение после настройки?Нет, после одноразовой загрузки модели инференс полностью идёт на вашем Mac; подключённые агенты всё ещё могут использовать интернет для собственных инструментов вроде веб-поиска.
Можно ли использовать его с другой моделью вместо Qwen3.8-Flash-Next?Пока нет — загрузчик и планировщик памяти построены специально под эту одну модель, так что подмена другой моделью сегодня не поддерживается.
С какими кодинг-агентами и чат-приложениями он работает?Поскольку он открывает API, совместимые с OpenAI и Ollama, он работает с Claude Code, Codex, Open WebUI и агентными инструментами вроде Hermes.
Насколько он быстрый на практике?Скорость сильно зависит от Mac: около 1.4 tok/s на 16 GB Mac mini M2, примерно 12 tok/s на 48 GB MacBook Pro M5 Pro и до ~21-22 tok/s на 128 GB MacBook Pro M5 Max.
Почему он не использует всю RAM моего Mac, даже если свободно много?Режим Auto ограничивает общую целевую память 33 GB (34.6 GB со speculative decoding), потому что тесты показали убывающий прирост скорости от большего кэша экспертов; вручную можно задать более высокую цель, если хотите попробовать.
Могут ли несколько приложений использовать его одновременно?Нет — Slotstream генерирует один ответ за раз, так что подключённые приложения делят один и тот же запущенный экземпляр модели.
What are similar projects?deltafin, Strata, project-maya, mlx-serve, Qwen3.8-Flash-Next, turbo-fieldfare, Rapid-MLX, backburner
Which topics is it in?qwen3, mlx, on-device-ai, speculative-decoding, inference-engine, mixture-of-experts, openai-api, moe, local-ai, metal, llm-inference, qwen
Creator
carloslfu
Size
567.1 MB
Stars
476
Updated
CommitOct 19Releasev0.2.30Oct 19
Age
1mo
Commits · 90 Days