Global

A unified distributed systems framework bridging centralized model orchestration with decentralized edge intelligence.

A unified distributed systems framework bridging centralized model orchestration with decentralized edge intelligence.

FractalCore is the central server, control plane, and model compilation engine for the Fractal decentralized compute ecosystem.

8-Flash-Next on any consumer hardware: one-click install for Windows / Linux. Strata inference engine, OpenAI/Anthropic API on localhost, optional image input.

NOTE] > Based on this course, I co-wrote the LLM Engineer's Handbook, a hands-on book that covers an end-to-end LLM application from design to deployment. The LLM course will always stay free, but you can support my work by purchasing this book.

5 GB of RAM. Weights stream from SSD; a trained router predicts loads a step ahead. Mac, iPhone, Android, Windows supported

AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card — without quantization, distillation, or pruning. 8-Flash-Next (125B) on 6GB, and DeepSeek-V3 (671B) on 12GB. 8-Flash-Next (125B) under 6GB.

Fully automatic censorship removal for language models

ggml / ops / maintainer PRs%20sort%3Aupdated-desc) / dev stats / lib llama API / llama-server REST API

WanGP by DeepBeepMeep : The best Open Source Generative Models Accessible to the GPU Poor

Most people learn ML frameworks by importing them. We built ours by hand, then made it hit harder.

Fine-tune an 8B model on a 4 GB laptop GPU. Layer streaming keeps the frozen base out of VRAM and feeds it to the GPU one decoder layer at a time. 32 GB.

A high-throughput and memory-efficient inference and serving engine for LLMs

> Breakthrough inference speed for MiniMax H3 Reference-to-Video (Ref2VA / R2VA): Generate a full 5-second, 24 fps video conditioned on reference character images in 72s (warm) / 95s (first run) on a single consumer NVIDIA GeForce RTX 4090 (24GB).

Find out which open-source Large Language Models (LLMs) your hardware can comfortably run. llmfit inspects your CPU, system RAM, GPU(s), VRAM, and accelerator configuration to recommend models across popular quantizations.

SGLang is a high-performance serving framework for large language models and multimodal models.

Comprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. Maintained by Orchestra Research.

sh as its gate, so latest is always the current stack. /models), and serves on port 18020. Pick a mode — one GPU serves one at a time:

llama.cpp on a jailbroken PS5: run any GGUF model natively on the console, starting with Qwen 3.8

Local UI to run and train LLMs and diffusion models. 8, DeepSeek-V4, MiniMax-H3, Gemma 4, FLUX and more.

faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is a fast inference engine for Transformer models.

1-Flash-1017 (748B MoE) privately on your own GPU(s), from one card to 16, NVIDIA or AMD, Linux or Windows. Engine, server, dashboard, OpenAI- and Anthropic-compatible API, quant tools.

🐧 PenguinHarness: Local-first multi-agent app development platform - create, fine-tune, deploy AI apps automatically!

DeepSeek 4 Flash and PRO local inference engine for Metal, CUDA and ROCm

cpp. The rest of the code is part of the ggml machine learning library.

MiniCPM-V and MiniCPM-o are multimodal LLM series designed for strong performance and efficient deployment on devices. MiniCPM-V focuses on efficient vision-language understanding across image, video and text inputs.

FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.

Use as the client base URL and the model ID from /v1/models. 2 or newer on a Mac (pip installs it). See the runbook for installation and a first request.

An open-source AI accelerator, developed by AI: RTL, ISA, simulator, compiler and profiler in one repo. 5 on a Kintex-7 PCIe card.

Note: This series will continue to grow as I write more blogs and create more videos on new topics. Keep learning.