kapitsa on GitHub (nace-ai/kapitsa)

kapitsa README

The project's README from github.com/nace-ai/kapitsa. As one markdown file: README.md.

Kapitsa: a blazing fast Apple Silicon optimized inference engine built for Qwen, Nemotron and Flint. Writing speed of Qwen3.6 35B-A3B on a MacBook Pro (2024), M4 Pro, 48 GB, in tokens per second, at 4-bit: Kapitsa with KFlash 161, Splash with DFlash2 135, oMLX with DFlash2 127, Ollama with MTP 89, LM Studio 86, MTPLX with MTP 80, llama.cpp with MTP 69.

Kapitsa is an inference engine for Apple silicon that runs Qwen, Nemotron and Recursive's Flint models locally, built for coding agents. It starts opencode, Claude Code, Codex or pi on a local model, and serves OpenAI and Anthropic APIs.

Quickstart · CLI · Benchmarks · Models · How KFlash works · Docs

🔥 What's new

[2026/10] 🪶 Flint on a 16 GB Mac: decodes at 61 tok/s vs llama.cpp's 43 on the same 2-bit GGUF. --model flint takes unsloth's 2-bit GGUF on 16-24 GB Macs: its routed experts run as stored, the rest requantized to 3-6 bits. 61 tok/s decode on an M1 Pro, in 11.6 GB (numbers).
[2026/10] 🪨 Recursive's Flint 40B-A2.5B: Kapitsa reads prompts faster than Recursive's own engine and decodes faster than llama.cpp. On the M4 Pro every Kapitsa run read 4K and 16K prompts faster than every run of rec 0.1.0. On the M5 Pro, since Flint's attention moved to the tensor units (#345), both runs read 4K and 16K prompts faster than rec's and llama.cpp's; earlier builds read slower, and both runs carry harness flags. On both Macs every Kapitsa run decodes faster than llama.cpp (M4 Pro, M5 Pro).
[2026/10] 🚀 KFlash: tree speculative decoding, up to 3.5× llama.cpp and 2.2× oMLX. One verify pass checks a whole tree of the drafter's guesses (how it works).
[2026/10] 🏆 Kapitsa: the fastest decode in 32 of 36 model and Mac pairs against nine engines, up to 2.9× MLX (poster).
[2026/10] 🤝 KFlash budget: subagents decode together. Three requests at once share one verify pass, up to +23% total throughput on the 27B.
[2026/10] ⚡ QMT prefill: read the prompt up to 1.5× faster on M5. Metal 4 matrix units for the prompt (4B, M5 Pro).
[2026/10] 💾 SSD prefix cache: restart the server, keep the conversation. A 16K-token follow-up on the 35B starts in 0.16 s instead of 17 s, about 100× (--disk-cache).
[2026/10] 🧠 Prefix reuse: subagents start warm. A new question over the shared system prompt answers in 0.06 s instead of 1.9 s (4K prompt, 4B, M5 Pro).
[2026/10] 🧪 Output check: speed has to prove it kept the answers. A default changes only after its logits stay within twice the model's noise floor (how).
[2026/10] 📊 kapitsa bench compare: see it on your own Mac. The same test against every engine installed here, in one command.

Quickstart

Requirements: a Mac with Apple silicon (M1 or newer) and macOS 14 Sonoma or later.

brew install nace-ai/tap/kapitsa
# or: curl -fsSL https://raw.githubusercontent.com/nace-ai/kapitsa/main/install.sh | bash

kapitsa opencode --model 35b    # also: claude, codex, pi

_{On a 16-24 GB Mac use --model 9b, or 4b for the quickest replies.}

CLI

kapitsa serve                   # chat + OpenAI/Anthropic APIs at http://127.0.0.1:8000
kapitsa claude                  # Claude Code on Kapitsa
kapitsa doctor                  # this Mac's GPU and memory, and which models fit it
kapitsa cluster                 # the Macs on this network that run Kapitsa, and their models
kapitsa bench                   # benchmark this Mac, share it with the community
kapitsa bench compare           # the same test against the other engines installed here
kapitsa pull 27b                # download a model without starting it
kapitsa list                    # downloaded models and their size
kapitsa rm 27b                  # delete a downloaded model
kapitsa import model.gguf       # use a Qwen or Flint GGUF from LM Studio, Ollama or llama.cpp
kapitsa update                  # update now (it also updates itself)

_{--model picks the model and its precision (35b:6bit). kapitsa <command> --help has the options.
A Mac that cannot hold a model whole uses another Mac on the network that runs it, and two Macs that would each stream
Flash-Next pair to hold it in memory between them (Several Macs); --private keeps a
server to its own Mac. Usage guide}

Benchmarks

Kapitsa has the fastest decode in 32 of the 36 model and Mac pairs measured against nine engines (how each number was measured).

Poster of prefill / decode speed for Kapitsa and nine other engines on a MacBook Pro M5 Max, M5 Pro, M4 Pro, M3 Pro and M1 Pro; the numbers are in the poster and in the Models section's tables

_{Each cell reads prefill / decode in tokens per second. Prefill is a 4K-token prompt, decode a 512-token answer at T 0.6 (Flint at its card's T 1.0). × cannot run that model · — not measured. How these were measured, run dates and flags: docs/benchmarks.md.}

The poster's numbers, and the models it leaves out, are in each Mac's table under Models. Every recorded run, with its Mac, model, prefill and decode speed, is in the benchmark table. Measure your Mac: kapitsa bench --submit adds it to the community results. Another engine faster on your Mac? kapitsa bench compare ranks every engine you have installed, and --report writes a Markdown report.

Models

The catalog models measured on each Mac, against the engines that run them: one collapsed section per Mac, with a table for each model family. Decode is a 512-token coding answer at T 0.6 (Flint at its card's T 1.0), prefill a 4K-token prompt. Under each model's name are the weights Kapitsa ran; the other engines' weights and how each number was measured are in docs/benchmarks.md. Flint's speed from 1K to 128K tokens of prompt on the M5 Pro, against llama.cpp: bench/results/2026-10-24-flint-m5pro-context.

_{Each cell: prefill / decode, tokens/s (a lone number is decode: no prefill run) · 🟩 fastest decode · ≈ a decode tie with Kapitsa · small type: the decoding method (none: no speculative decoding) · × can't run on that Mac · — not measured}

MacBook Pro (16-inch, 2026) · M5 Max · 128 GB · 7 models

Model	Kapitsa	_{mlx‑serve 26.10.2}	_{oMLX 0.7.1rc1}	_Splash 1.4.0	_{Ollama 0.40.5 / 0.40.6}	_MTPLX 2.12.2	_{MLX mlx-lm 0.32.2, mlx-vlm 0.7.8}	_{llama.cpp b11669}	_{LM Studio MLX runtime 1.13.1, llama.cpp runtime 2.60.0}
Qwen3.8-27B · `27b` _{dense · 4-bit, 20.2 GB}	🟩 459 / 140 _KFlash	690 / 130 _DFlash2	761 / 93 _DFlash2	898 / 116 _DFlash2	739 / 68	733 / 69 _MTP	784 / 54 _MTP	516 / 51 _MTP	217 / 32
Qwen3.6-35B-A3B · `35b` _{MoE, 3B active · 4-bit, 22.1 GB}	🟩 4702 / 325 _KFlash	3818 / 264 _MTP	2968 / 232 _DFlash2	6640 / 290 _DFlash2	3690 / 166	4328 / 123 _MTP	4271 / 132 _DFlash2	2881 / 171 _MTP	1873 / 127
Qwen3.8-Flash-Next 180B-A7B · `flash-next` _{MoE, 7B active · 4-bit, 105.9 GB}	1165 / 106 _KFlash	🟩 1720 / 108	1260 / 107 _MTP	×	60	1263 / 88 _MTP	1030 / 77 _MTP	842 / 96	×
Nemotron Dory1.0 4B · `dory` _{looped dense · NVFP4, 3.2 GB}	🟩 119 _KFlash	×	×	×	×	×	76	9	×
Nemotron 3.5 Lightning 30B-A3B · `lightning` _{hybrid MoE, 3B active · NVFP4, 22.9 GB}	🟩 267 _KFlash	190	190	×	153	131	139	159	110
Nemotron 3.5 Bolt 74B-A9B · `bolt` _{hybrid MoE, 9B active · 2/4/8-bit mix, 25.1 GB}	🟩 1016 / 153 _KFlash	×	913 / 69	×	×	×	941 / 69	×	×
Nemotron 3.5 Super-VL 120B-A12B · `super` _{hybrid MoE, 12B active · NVFP4, 76.4 GB}	🟩 94 _KFlash	×	52	×	64	44	46	66	61

MacBook Pro (16-inch, 2026) · M5 Pro · 48 GB · 6 models

Qwen and Nemotron

Model	Kapitsa	_{mlx‑serve 26.10.2}	_{oMLX 0.7.1rc1}	_Splash 1.4.0	_{Ollama 0.40.5 / 0.40.6}	_MTPLX 2.12.2	_{MLX mlx-lm 0.32.2, mlx-vlm 0.7.8}	_{llama.cpp b11669}	_{LM Studio MLX runtime 1.13.1, llama.cpp runtime 2.60.0}
Qwen3.8-27B · `27b` _{dense · 4-bit, 20.2 GB}	606 / 77 _KFlash	🟩 464 / 78 _DFlash2	446 / 51 _DFlash2	674 / 72 _DFlash2	470 / 40 _MTP	462 / 41 _MTP	421 / 32 _MTP	322 / 30 _MTP	118 / 17
Qwen3.6-35B-A3B · `35b` _{MoE, 3B active · 4-bit, 22.1 GB}	🟩 2802 / 201 _KFlash	2494 / 156 _MTP	1619 / 144 _DFlash2	3514 / 181 _DFlash2	1518 / 102 _MTP	2569 / 89 _MTP	2474 / 98 _DFlash2	1755 / 107 _MTP	997 / 96
Nemotron Dory1.0 4B · `dory` _{looped dense · NVFP4, 3.2 GB}	🟩 70 _KFlash	×	×	×	×	×	53	13	×
Nemotron 3.5 Lightning 30B-A3B · `lightning` _{hybrid MoE, 3B active · NVFP4, 22.9 GB}	🟩 3141 / 159 _KFlash	1728 / 115 _MTP	1938 / 124 _MTP	×	2369 / 92	1743 / 101	1881 / 94	1915 / 101 _MTP	1479 / 72
Nemotron 3.5 Bolt 74B-A9B · `bolt` _{hybrid MoE, 9B active · 2/4/8-bit mix, 25.1 GB}	🟩 79 _KFlash	×	39	×	×	×	41	×	×

Recursive Flint

Model	Kapitsa	_{llama.cpp b11804}	_rec 0.1.0
Recursive Flint 40B-A2.5B · `flint` _{MoE, 2.5B active · 4-bit, 22.6 GB}	🟩 2691 / 149 _KFlash	2072 / 92	2138 / 145

_{Qwen3.8-Flash-Next 180B-A7B does not fit in 48 GB, so both engines stream its experts from the SSD. With Q2_0 weights on both and Kapitsa's expert store pinned at 24 GB (the auto size here is about 29.9 GB, but four runs at that size were stopped at memory pressure), Kapitsa decodes faster than llama.cpp up to 4K tokens of context and slower at 16K and 32K, and its prefill is behind on short prompts, level at 4K and 16K and 2.7% ahead at 32K: not the fastest everywhere (numbers and caveats).}

MacBook Pro (16-inch, 2024) · M4 Pro · 48 GB · 7 models

Qwen and Nemotron

Model	Kapitsa	_{mlx‑serve 26.10.2}	_{oMLX 0.7.1rc1}	_Splash 1.4.0	_{Ollama 0.40.5 / 0.40.6}	_MTPLX 2.12.2	_{MLX mlx-lm 0.32.2, mlx-vlm 0.7.8}	_{llama.cpp b11669}	_{LM Studio MLX runtime 1.13.1, llama.cpp runtime 2.60.0}
Qwen3.5-4B · `4b` _{dense · 4-bit, 4.6 GB}	🟩 1930 / 201 _KFlash	874 / 137 _MTP	803 / 102 _MTP	×	758 / 109 _MTP	780 / 104 _MTP	843 / 88 _MTP	650 / 70 _MTP	801 / 81
Qwen3.8-27B · `27b` _{dense · 4-bit, 20.2 GB}	🟩 396 / 63 _KFlash	129 / 54 _DFlash2	136 / 45 _DFlash2	53 _DFlash2	124 / 32 _MTP	126 / 36 _MTP	131 / 28 _MTP	84 / 18 _MTP	127 / 15
Qwen3.6-35B-A3B · `35b` _{MoE, 3B active · 4-bit, 22.1 GB}	🟩 1168 / 161 _KFlash	986 / 124 _MTP	1035 / 127 _DFlash2	135 _DFlash2	892 / 89 _MTP	918 / 80 _MTP	985 / 86 _DFlash2	663 / 69 _MTP	957 / 86
Nemotron Dory1.0 4B · `dory` _{looped dense · NVFP4, 3.2 GB}	🟩 64 _KFlash	×	×	×	×	×	48	6	×
Nemotron 3.5 Lightning 30B-A3B · `lightning` _{hybrid MoE, 3B active · NVFP4, 22.9 GB}	🟩 945 / 112 _KFlash	—	—	×	—	—	721 / 65 _DSpark	—	—
Nemotron 3.5 Bolt 74B-A9B · `bolt` _{hybrid MoE, 9B active · 2/4/8-bit mix, 25.1 GB}	🟩 519 / 69 _KFlash	—	298 / 35	×	—	—	300 / 32	×	—

Recursive Flint

Model	Kapitsa	_{llama.cpp b11804}	_rec 0.1.0
Recursive Flint 40B-A2.5B · `flint` _{MoE, 2.5B active · 4-bit, 22.6 GB}	🟩 968 / 128 _KFlash	977 / 80	796 / 127

MacBook Pro (16-inch, 2023) · M3 Pro · 36 GB · 10 models

Qwen and Nemotron

Model	Kapitsa	_{Ollama 0.40.8}	_MTPLX 2.12.2	_{MLX mlx-lm 0.32.2, mlx-vlm 0.7.8}	_{llama.cpp b11669}
Qwen3.5-4B · `4b` _{dense · 4-bit, 4.6 GB}	🟩 703 / 120 _KFlash	600 / 67 _MTP	612 / 64 _MTP	642 / 56 _MTP	512 / 50 _{MTP · b11804}
Qwen3.5-9B · `9b` _{dense · 4-bit, 9.0 GB}	🟩 381 / 77 _KFlash	—	—	353 / 27	307 / 23 _b11804
Qwen3.8-27B · `27b` _{dense · 4-bit, 20.2 GB}	🟩 110 / 40 _KFlash	99 / 23 _MTP	99 / 22 _MTP	102 / 18 _MTP	86 / 16 _{MTP · b11804}
Qwen3.6-35B-A3B · `35b` _{MoE, 3B active · 4-bit, 22.1 GB}	🟩 855 / 106 _KFlash	681 / 58 _MTP	671 / 53 _MTP	695 / 53 _DFlash2	578 / 53 _{MTP · b11804}
Nemotron Dory1.0 4B · `dory` _{looped dense · NVFP4, 3.2 GB}	🟩 450 / 36 _KFlash	×	×	344 / 30	—
Nemotron 3 Nano 4B · `nano-4b` _{hybrid Mamba-2 · FP8, 5.3 GB}	765 / 34 _KFlash	—	—	—	🟩 413 / 35 _{Q4_K_M, b11804}
Nemotron Nano 9B v2 · `nano-9b` _{hybrid Mamba-2 · NVFP4, 7.9 GB}	330 / 21 _KFlash	—	—	🟩 25 _4-bit	—
Nemotron 3.5 Lightning 30B-A3B · `lightning` _{hybrid MoE, 3B active · NVFP4, 22.9 GB}	🟩 788 / 75 _KFlash	—	—	514 / 39 _DSpark	—
Nemotron 3.5 Bolt 74B-A9B · `bolt` _{hybrid MoE, 9B active · 2/4/8-bit mix, 25.1 GB}	🟩 256 / 50 _KFlash	—	—	—	—

Recursive Flint

Model	Kapitsa	_{llama.cpp b11804}
Recursive Flint 40B-A2.5B · `flint` _{MoE, 2.5B active · 4-bit, 22.6 GB}	🟩 801 / 74 _KFlash	772 / 51

_{macOS 14.1, on AC power: Kapitsa's macOS 14 kernels (macOS 15 and later run fused ones). mlx-serve, oMLX and Splash need a newer macOS (×); LM Studio was not installed; rec 0.1.0 refuses this Mac: it requires 48 GiB. Each cell but Flint's is one run of the M4 Pro's protocol (Flint's is the mean of two runs, #341), the other engines in the M4 Pro's setup (not re-tuned here). Nano 4B and 9B v2: Kapitsa runs NVIDIA's published checkpoints, which keep some layers in BF16/FP8, so it reads more per token than the 4-bit files: the 4B's FP8 is ~1 byte per weight against Q4_K_M's ~0.6, the 9B's NVFP4 ~6.11 GB per token against 4.67 GB for mlx-lm's 4-bit. Nano 9B v2 on mlx-lm: no prefill shown, because its template ignores enable_thinking and the timed first token is a thinking token.}

MacBook Pro (14-inch, 2021) · M1 Pro · 16 GB · 7 models

Qwen and Nemotron

Model	Kapitsa	_{mlx‑serve 26.10.2}	_{oMLX 0.7.1rc1}	_{Ollama 0.40.5 / 0.40.6}	_MTPLX 2.12.2	_{MLX mlx-lm 0.32.2, mlx-vlm 0.7.8}	_{llama.cpp b11669}	_{LM Studio MLX runtime 1.13.1, llama.cpp runtime 2.60.0}
Qwen3.5-4B · `4b` _{dense · 4-bit, 4.6 GB}	🟩 401 / 97 _KFlash	286 / 62 _MTP	281 / 55 _MTP	298 / 55 _MTP	305 / 42 _MTP	308 / 46	315 / 38 _MTP	308 / 47
Qwen3.5-9B · `9b` _{dense · 4-bit, 9.0 GB}	🟩 217 / 57 _KFlash	159 / 37 _MTP	—	—	—	170 / 30	—	—
Qwen3.6-35B-A3B · `35b:16gb` _{MoE, 3B active · 2-bit experts (unsloth's GGUF), 10.8 GB}	🟩 462 / 52 _KFlash	×	×	—	×	×	38	—
Nemotron Dory1.0 4B · `dory` _{looped dense · NVFP4, 3.2 GB}	🟩 39 _KFlash	×	×	×	×	30	4	×
Nemotron 3 Nano 4B · `nano-4b` _{hybrid Mamba-2 · FP8, 5.3 GB}	🟩 37 _KFlash	—	—	—	—	21 _BF16	28 _{Q4_K_M}	—
Nemotron Nano 9B v2 · `nano-9b` _{hybrid Mamba-2 · NVFP4, 7.9 GB}	🟩 25 _KFlash	—	—	—	—	18	14 _{Q4_K_M}	—

Recursive Flint

Model	Kapitsa	_{llama.cpp d772569}
Recursive Flint 40B-A2.5B · `flint:16gb` _{MoE, 2.5B active · 2-bit experts (unsloth's GGUF), 11.9 GB}	🟩 414 / 61 _KFlash	43

_{Other precisions: --model 35b:6bit is five times closer to bf16 than the 4-bit default (KL). All but Lightning, Dory, the two Nemotron Nanos, Flint and the 35B's 16gb read images. Memory: 16 GB for the 4B and 9B models, 32 GB
for the 27B, 35B, Lightning, Bolt and Flint, 96 GB for Super-VL, 128 GB for Flash-Next. On a 16 or 24 GB Mac, 35b and flint take their 16gb variants: unsloth's 2-bit GGUFs, downloaded and imported in one step (how). Every model's variants and draft models}

How it works

How Kapitsa works: DFlash2 vs KFlash. DFlash2, the drafter's chain, as Splash, oMLX and mlx-serve run it: the drafter guesses the next 7 tokens, the big model checks them in one pass and keeps them up to the first wrong guess (return a, 2 tokens). KFlash, Kapitsa's: in the same one pass the big model checks a best-first tree of the drafter's top candidates in 8 rows; where the drafter is unsure its runner-ups ride along, and the longest branch the big model agrees with is kept (return a + b, 4 tokens). Same output as plain decoding, tree size measured on each Mac, 8-17% more tokens per pass on short prompts. Below: GPU, hand-written Metal code; Neural Engine, helps read long prompts on the 4B, 27B, Bolt and Super-VL; Memory, keeps conversations ready, up to 4 at once.

Kapitsa decodes with KFlash. A draft model such as DFlash2 guesses the next seven tokens, with a few candidates for each one. Engines that run it as published (Splash, oMLX, mlx-serve) check one chain of guesses and stop at the first wrong one. KFlash checks a best-first tree of the drafter's candidates in the same pass, so where the drafter hesitates, its runner-ups are checked too and more tokens survive each pass. The tree's size comes from what a verify row costs on your Mac, measured there, and it changes no output: the big model checks every token. It runs on the published drafters as they are: z-lab's DFlash, DFlash2, and NVIDIA's DSpark and LiLiCorr. How KFlash works

Get release notices

On the Kapitsa repository page: Watch, then Custom, tick Releases, Apply

_{Watch → Custom → Releases on nace-ai/kapitsa puts every new version in your notifications. A drawn illustration of the steps, not a screen recording; the page source is in tools/readme_images.}

Star history

Searching Database…

427 ·GitHub

Kapitsa

Blazing fast Apple Silicon optimized inference engine built for Qwen & Nemotron. OpenAI + Anthropic API compatible. ai

Creator: nace-ai
Size: 56.8 MB
Stars: 427
Updated: CommitOct 25 Releasev0.2.17Oct 24
Age: 9d
Commits · 90 Days

kapitsa README

🔥 What's new

Quickstart

CLI

Benchmarks

Models

How it works

Get release notices

Star history

kapitsa (nace-ai/kapitsa)