Janus — Local LLM Server & OpenAI-Compatible API
Janus is a single Go binary that runs .gguf models on your machine (GPU or CPU) and exposes an OpenAI-compatible API. No Python, no Docker, no Ollama required.
Use it your way: call it from the command line (curl, PowerShell, scripts), wire it into Cursor / Cline / any OpenAI client — same local models, whatever workflow fits you.
What you get
- Local inference — llama.cpp via Vulkan (AMD / Intel / NVIDIA) or CPU fallback
- OpenAI-compatible API —
/v1/chat/completions, /v1/models
- Hot-swap models — change
.gguf without restarting
- Thinking model support —
<think> reasoning split into reasoning_content
- Chat template auto-detection — uses the template from GGUF metadata
- Zero dependencies — one
.exe on Windows, no Python, no Docker
Requirements
| Platform |
What you need |
| Windows (primary) |
Windows 10/11, Go 1.25+, Vulkan-capable GPU recommended |
| Linux |
Go 1.25+, curl, tar, Vulkan (Mesa/NVIDIA driver; check with vulkaninfo) or CPU |
| macOS |
Go 1.25+, CPU backend (Vulkan varies by hardware) |
Disk: plan for the model size (often 2–8 GB per model) plus ~50 MB for Janus + llama.dll.
Quick start (Windows)
1. Clone and build
git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
.\build.ps1
build.ps1 downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe.
2. Download a model
Put a .gguf file in the models\ folder. Use the included downloader:
go build -o dist\modelget.exe .\cmd\modelget
.\dist\modelget.exe -repo meta-llama/Llama-3.2-3B-Instruct -file Llama-3.2-3B-Instruct-Q8_0.gguf -out .\models\
Or download any GGUF from Hugging Face.
copy .env.example .env
Edit .env:
INFERENCE_BACKEND=vulkan
JANUS_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q8_0.gguf
JANUS_MAX_TOKENS=4096
| Variable |
Default |
Meaning |
INFERENCE_BACKEND |
vulkan |
vulkan, cpu, or openrouter |
JANUS_MODEL_PATH |
(required) |
Path to your .gguf file |
JANUS_GPU_LAYERS |
-1 |
-1 = all layers on GPU, 0 = CPU only |
JANUS_VRAM_CEILING_MB |
9216 |
VRAM budget hint (MiB) |
JANUS_MAX_TOKENS |
4096 |
Max tokens per reply |
JANUS_LISTEN_ADDR |
127.0.0.1:8990 |
Bind address |
4. Run
.\dist\janus.exe
Opens http://127.0.0.1:8990 in your browser.
5. Verify
curl http://127.0.0.1:8990/health
Quick start (Linux)
git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
./build.sh # downloads llama.cpp b11146 (Vulkan) into lib/linux, builds dist/janus
cp .env.example .env
# edit .env — set JANUS_MODEL_PATH (and INFERENCE_BACKEND=cpu if no Vulkan)
./dist/janus
build.sh pins llama.cpp b11146 (the struct layouts in internal/bridge/llama_dl.go match that tag); ./build.sh --version bNNNN tries another one, --skip-download reuses lib/linux/.- The
.so files are copied next to the binary (dist/); they find each other via RUNPATH=$ORIGIN, no LD_LIBRARY_PATH needed.
- Go 1.25+ is required (purego v0.11 is needed to pass llama.cpp's structs by value on Linux). With
GOTOOLCHAIN=auto (default) an older go downloads the right toolchain itself.
JANUS_GPU_LAYERS=-1 (default) offloads every layer; check the log for offloaded N/N layers to GPU.
Supported: linux/amd64; arm64/macOS not tested/not supported.
macOS: build with go build -o dist/janus ./cmd/janus and provide libllama.dylib next to the binary (untested).
OpenAI-compatible API
curl http://127.0.0.1:8990/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "local",
"messages": [{"role": "user", "content": "Hello!"}]
}'
Base URL: http://127.0.0.1:8990/v1
Endpoints
| Method |
Path |
Description |
| GET |
/health |
Liveness check (?deep=true for details) |
| GET |
/v1/models |
Model list |
| POST |
/v1/chat/completions |
Chat (streaming supported) |
| POST |
/models/load |
Hot-swap model |
| GET |
/models/list |
Available .gguf files |
| GET |
/engine/status |
VRAM and backend info |
Streaming
curl http://127.0.0.1:8990/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"local","stream":true,"messages":[{"role":"user","content":"Tell me a joke"}]}'
Connect to Cursor / Cline / other clients
Base URL: http://127.0.0.1:8990/v1
API Key: (leave blank)
Pitfalls (learned the hard way)
| Problem |
What's going on |
Fix |
| "It built but my changes aren't there" |
On Windows, Go can't overwrite a running .exe. |
Stop all janus.exe in Task Manager, then rebuild. |
| "Address already in use" |
A leftover process holds port 8990. |
Task Manager → end all janus.exe. |
| Server starts, chat fails |
JANUS_MODEL_PATH wrong or no .gguf in models/. |
Set the path in .env, put the file in models/. |
| "local engine failed to start" |
Missing llama.dll or GPU driver issue. |
Run .\build.ps1. Update GPU drivers, or set INFERENCE_BACKEND=cpu. |
| Wrong URL |
Default is http://127.0.0.1:8990, not 8080. |
Bookmark 8990. |
| First reply takes forever |
Model loading into VRAM — normal. |
Wait 10–60s; smaller quants (Q4) load faster. |
Project layout
cmd/janus/ Main server (OpenAI-compatible API)
cmd/modelget/ Hugging Face model downloader
internal/engine/ llama.cpp Vulkan/CPU backend
internal/bridge/ DLL loader and FFI bindings
internal/singleton/ Single-instance guard
models/ Put .gguf files here (not committed)
dist/ janus.exe + llama.dll after build
Development
Get-Process -Name "janus" -ErrorAction SilentlyContinue | Stop-Process -Force
go test ./...
go build -o dist\janus.exe .\cmd\janus
.\dist\janus.exe
Or just .\run.ps1. For a full build including llama DLLs, use .\build.ps1.
Contributions welcome — see CONTRIBUTING.md.
License
MIT — see LICENSE.
Janus 是一个单一的 Go 二进制文件，可以在你的机器（GPU 或 CPU）上运行本地 .gguf 语言模型，并暴露兼容 OpenAI 的 API，让你无需 Python、Docker 或 Ollama 就能使用本地模型。