What is this?
MoE4All is a portable Windows app that runs large Mixture-of-Experts (MoE) LLMs beyond what fits in VRAM by streaming expert weights between GPU VRAM, system RAM and SSD, built on a Vulkan-first Rust inference engine so consumer AMD (and other Vulkan) GPUs can run huge models locally.
MoE4All is a portable Windows app that runs large Mixture-of-Experts (MoE) LLMs beyond what fits in VRAM by streaming expert weights between GPU VRAM, system RAM and SSD, built on a Vulkan-first Rust inference engine so consumer AMD (and other Vulkan) GPUs can run huge models locally.Is this a GUI app or a command-line tool?The Windows release is driven by a command-line setup wizard offering terminal chat, an OpenAI-compatible API, and benchmarking; a browser GUI only exists in the source tree, not in the prebuilt release.
The Windows release is driven by a command-line setup wizard offering terminal chat, an OpenAI-compatible API, and benchmarking; a browser GUI only exists in the source tree, not in the prebuilt release.How do I get it running?Download MoE4All-Windows-x8664-v.zip from the latest Release and extract it, then double-click the launcher and pick terminal chat, OpenAI-compatible API, or performance test, pointing it at your GGUF model path.
Download MoE4All-Windows-x8664-v.zip from the latest Release and extract it, then double-click the launcher and pick terminal chat, OpenAI-compatible API, or performance test, pointing it at your GGUF model path.Do I need to install Rust, Visual Studio or the Vulkan SDK first?No - the release package is self-contained; you only need a normal 64-bit GPU driver with its Vulkan runtime, plus your own GGUF model files, which are not bundled.
No - the release package is self-contained; you only need a normal 64-bit GPU driver with its Vulkan runtime, plus your own GGUF model files, which are not bundled.What are the current rough edges?Cold starts can be noticeably slow while large models stream in from SSD, SSD-to-RAM prefetching during prefill still needs tuning, and the auto-budget aims for a reliable start rather than guaranteed peak throughput on every machine.
Cold starts can be noticeably slow while large models stream in from SSD, SSD-to-RAM prefetching during prefill still needs tuning, and the auto-budget aims for a reliable start rather than guaranteed peak throughput on every machine.Does it run just as well on NVIDIA and Intel GPUs, or is this really built for AMD?AMD Radeon on native Windows 11 with Vulkan (tested on an RX 7900 XTX) is the primary, verified setup; other Vulkan-capable GPUs including NVIDIA and Intel may work but aren't the project's main validation target yet.
AMD Radeon on native Windows 11 with Vulkan (tested on an RX 7900 XTX) is the primary, verified setup; other Vulkan-capable GPUs including NVIDIA and Intel may work but aren't the project's main validation target yet.What models does it support out of the box?It supports GGUF models including Llama/Llama4, the Qwen2 through Qwen3.8-Flash-Next family, Gemma 3/4, Ling 3.0 Flash, DeepSeek V4 Flash, DiffusionGemma, and embedding-only GGUFs, each mapped to a specific supported GGUF architecture.
It supports GGUF models including Llama/Llama4, the Qwen2 through Qwen3.8-Flash-Next family, Gemma 3/4, Ling 3.0 Flash, DeepSeek V4 Flash, DiffusionGemma, and embedding-only GGUFs, each mapped to a specific supported GGUF architecture.Why can it run models bigger than my VRAM at all?Large MoE models activate only a small fraction of their experts per token, so MoE4All keeps hot experts in VRAM, a larger set in RAM, and the rest on SSD, reshuffling that budget between prefill and decode.
Large MoE models activate only a small fraction of their experts per token, so MoE4All keeps hot experts in VRAM, a larger set in RAM, and the rest on SSD, reshuffling that budget between prefill and decode.What is 'infr' and how does it relate to MoE4All?infr is the Pure-Rust, Vulkan-first inference engine by kryptic.sh that MoE4All is built on top of; MoE4All adds the Windows packaging, setup wizard, and AMD-focused expert paging around it.
infr is the Pure-Rust, Vulkan-first inference engine by kryptic.sh that MoE4All is built on top of; MoE4All adds the Windows packaging, setup wizard, and AMD-focused expert paging around it.Can I point my existing OpenAI-style client at it?Yes, it exposes an OpenAI-compatible chat and embedding API on a local base URL, but the README warns not to expose that unauthenticated endpoint directly to the public internet.
Yes, it exposes an OpenAI-compatible chat and embedding API on a local base URL, but the README warns not to expose that unauthenticated endpoint directly to the public internet.What are similar projects?unsloth, ktransformers, vllm, Strata, Offensive-Security-AI-Models, project-maya, sglang, dgx-spark-qwen38
unsloth, ktransformers, vllm, Strata, Offensive-Security-AI-Models, project-maya, sglang, dgx-spark-qwen38Which topics is it in?intel, gemma, vulkan-api, portable, amd-gpu, amd, moe, gguf, llama, windows-11, llm-inference, embeddings
intel, gemma, vulkan-api, portable, amd-gpu, amd, moe, gguf, llama, windows-11, llm-inference, embeddings