This is EXL3 weights + fp8 KV on GB10. Do not pass --moe-backend marlin. The Hub card on brandonmusic (TP2/EP2/DCP2 + calibrated NVFP4 MLA KV) is the SM120 B12X image (verdictai/glm53-flash-exl3-k4:…-v84-dflash2), not this overlay.
One command turns a DGX Spark or ASUS Ascent GX10 into a private AI server: seven measured, pinned AI lanes (text up to 1M tokens, images, video with sound), run from a local cockpit on a laptop or a phone. Nothing leaves the box.
8-27B's published mixed NVFP4/FP8 weight representation unchanged on four Tesla V100-SXM2-16GB. m8n8k4. No FP8→FP4 requantization.
0-9B is a multimodal foundation model for general visual understanding, spatial reasoning, agentic tool use, and embodied-AI research. 5-9B language backbone with a C-RADIOv4-H vision encoder, supports text, images and videos with any-resolution visual input.
TypeLLM brings type-safe generation to existing autoregressive LLMs without changing their architecture or weights. Inspired by TypeSafe AI's Jev, it lets models retain their native thinking and free-form generation while producing schema-guaranteed outputs through JSON Schema.
This repo contains the launcher, server and configuration. The model weights live on Hugging Face (see Model cards) and the inference engine is our exllamav3 fork (DFlash2/MTP drafting, NVFP4/FP8 KV, aarch64 GB10 + x86 CUDA).