The project's README from github.com/omnimind-ai/OmniInfer. As one markdown file: README.md.
English | 简体中文
Easy, fast, and private LLM & VLM inference for every device.
Quick Start · Documentation · Releases
| Linux x64 | macOS arm64 | Windows x64 PowerShell |
|---|---|---|
curl -fsSL https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.sh | bash |
curl -fsSL https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.sh | bash |
irm https://raw.githubusercontent.com/omnimind-ai/OmniInfer/main/scripts/install.ps1 | iex |
The installers download the latest CLI-only GitHub Release, verify its SHA-256 checksum, and install it for the current user. For fixed versions, custom paths, manual installation, source setup, and removal, see Installation.
omniinfer in a terminal.To run the local OpenAI- and Anthropic-compatible service, start omniinfer serve and follow the CLI Guide or API Reference. Source builds and mobile embedding are documented separately.
b10665 and can load split GGUF model directories directly. Use a source-built llama.cpp backend until the matching prebuilt runtime is published in OmniInfer.Short, focused recordings of the main entry points. Each demo covers one capability.
Running omniinfer opens a terminal UI that recommends a compatible backend, loads a local model, and starts a fully local chat session.
Static preview: terminal UI screenshot
The optional vla-libero example runs a SmolVLA policy through a managed vla.cpp runtime in the LIBERO simulator, showing live camera views, the predicted action, and latency in a browser dashboard.
Static preview: SmolVLA LIBERO dashboard screenshot
OmniInfer is a high-performance, cross-platform inference engine for running Large Language Models (LLM) and Vision-Language Models (VLM) locally. It abstracts away model compilation, hardware adaptation, and deployment complexity, enabling efficient local inference with minimal configuration.
OmniInfer powers the inference layer of Omni Studio, a unified model orchestration platform.
OmniInfer is fast with:
OmniInfer is flexible and easy to use with:
OmniInfer runs everywhere:
| Platform | Distribution | Representative runtimes |
|---|---|---|
| Linux x64 | Release CLI and source checkout | llama.cpp, stable-diffusion.cpp, ik_llama.cpp, vLLM, vla.cpp |
| macOS arm64 | Release CLI and source checkout | llama.cpp, MLX, TurboQuant |
| Windows x64 | Release CLI and source checkout | llama.cpp, stable-diffusion.cpp, vLLM through WSL2 |
| Android | Gradle module | llama.cpp, MNN, LiteRT-LM, ExecuTorch QNN |
| iOS | Swift package | Embedded native inference service |
Runtime availability depends on the device and accelerator. Use omniinfer backend list for the current machine; backend names explains selectors and legacy compatibility and see the Build Guide for the full platform matrix.
We welcome contributions and collaborations. See Contributing to OmniInfer to get involved.
If you use OmniInfer in research, cite this repository. GitHub can generate additional formats from CITATION.cff.
@software{omniinfer,
author = {{Omnimind AI}},
title = {OmniInfer},
url = {https://github.com/omnimind-ai/OmniInfer}
}
OmniInfer is licensed under the Apache License 2.0. See LICENSE.
The installers download the latest CLI-only GitHub Release, verify its SHA-256 checksum, and install it for the current user. For fixed versions, custom paths, manual installation, source setup, and removal, see Installation.