Global

8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. 8-Flash-Next (125B) on 6GB, and DeepSeek-V3 (671B) on 12GB. 8-Flash-Next (125B) under 6GB.

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev stats / lib llama API / llama-server REST API

8B to 27B: a calibrated probability for every option in one pass, and a short thought only when unsure. cpp and MLX.

A high-throughput and memory-efficient inference and serving engine for LLMs

SGLang is a high-performance serving framework for large language models and multimodal models.

预览地址为 ，简体中文位于根目录，英文与繁體中文分别位于 en/ 和 zh-tw/。生成文件位于 build/，详细说明见网站构建与发布。

This repo is a structured, machine-readable directory of every free LLM API — rate limits, context windows, one-click config snippets, and direct API key links. Updated daily.

To download the model weights manually, use the following command:

sh as its gate, so latest is always the current stack. /models), and serves on port 18020. Pick a mode — one GPU serves one at a time:

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

3-Flash (321B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu bar

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

WARNING] > Gitleaks is feature complete. I'm not merging new features into Gitleaks. Future releases will be security patches only.

llamafile lets you distribute and run LLMs with a single file.

Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.

An open-source AI accelerator, developed by AI: RTL, ISA, simulator, compiler and profiler in one repo. 5 on a Kintex-7 PCIe card.

Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.

cpp — a real-time multilingual ASR inference engine on CPU using BitNet I2S quantization, achieving RTF Code ] [ Models ] [ Report ]

A framework for efficient model inference with omni-modality models

AI System Design - Learn how to design AI systems built on LLMs, RAG, and AI Agents step by step.

24 GB of RAM. Portable C99: no BLAS, no framework, no GPU.

This release contains training, two inference backends, benchmark scoring, temperature calibration, and a browser demo.

8-27B bf16 (130 tok/s), 262k context, prefix caching. Works with Claude Code, Codex, opencode and pi.

A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc.

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python.