Janus README

The project's README from github.com/Vibra-Ingenn/Janus. As one markdown file: README.md.

Janus logo

Janus — Local LLM Server & OpenAI-Compatible API

Janus is a single Go binary that runs .gguf models on your machine (GPU or CPU) and exposes an OpenAI-compatible API. No Python, no Docker, no Ollama required.

Use it your way: call it from the command line (curl, PowerShell, scripts), wire it into Cursor / Cline / any OpenAI client — same local models, whatever workflow fits you.

What you get

Requirements

Platform What you need
Windows (primary) Windows 10/11, Go 1.25+, Vulkan-capable GPU recommended
Linux Go 1.25+, curl, tar, Vulkan (Mesa/NVIDIA driver; check with vulkaninfo) or CPU
macOS Go 1.25+, CPU backend (Vulkan varies by hardware)

Disk: plan for the model size (often 2–8 GB per model) plus ~50 MB for Janus + llama.dll.

Quick start (Windows)

1. Clone and build

git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
.\build.ps1

build.ps1 downloads pre-built llama.cpp Vulkan DLLs and compiles dist\janus.exe.

2. Download a model

Put a .gguf file in the models\ folder. Use the included downloader:

go build -o dist\modelget.exe .\cmd\modelget
.\dist\modelget.exe -repo meta-llama/Llama-3.2-3B-Instruct -file Llama-3.2-3B-Instruct-Q8_0.gguf -out .\models\

Or download any GGUF from Hugging Face.

3. Configure

copy .env.example .env

Edit .env:

INFERENCE_BACKEND=vulkan
JANUS_MODEL_PATH=./models/Llama-3.2-3B-Instruct-Q8_0.gguf
JANUS_MAX_TOKENS=4096
Variable Default Meaning
INFERENCE_BACKEND vulkan vulkan, cpu, or openrouter
JANUS_MODEL_PATH (required) Path to your .gguf file
JANUS_GPU_LAYERS -1 -1 = all layers on GPU, 0 = CPU only
JANUS_VRAM_CEILING_MB 9216 VRAM budget hint (MiB)
JANUS_MAX_TOKENS 4096 Max tokens per reply
JANUS_LISTEN_ADDR 127.0.0.1:8990 Bind address

4. Run

.\dist\janus.exe

Opens http://127.0.0.1:8990 in your browser.

5. Verify

curl http://127.0.0.1:8990/health

Quick start (Linux)

git clone https://github.com/Vibra-Ingenn/Janus.git
cd Janus
./build.sh                      # downloads llama.cpp b11146 (Vulkan) into lib/linux, builds dist/janus
cp .env.example .env
# edit .env — set JANUS_MODEL_PATH (and INFERENCE_BACKEND=cpu if no Vulkan)
./dist/janus

Supported: linux/amd64; arm64/macOS not tested/not supported.

macOS: build with go build -o dist/janus ./cmd/janus and provide libllama.dylib next to the binary (untested).

OpenAI-compatible API

curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Base URL: http://127.0.0.1:8990/v1

Endpoints

Method Path Description
GET /health Liveness check (?deep=true for details)
GET /v1/models Model list
POST /v1/chat/completions Chat (streaming supported)
POST /models/load Hot-swap model
GET /models/list Available .gguf files
GET /engine/status VRAM and backend info

Streaming

curl http://127.0.0.1:8990/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"local","stream":true,"messages":[{"role":"user","content":"Tell me a joke"}]}'

Connect to Cursor / Cline / other clients

Base URL:  http://127.0.0.1:8990/v1
API Key:   (leave blank)

Pitfalls (learned the hard way)

Problem What's going on Fix
"It built but my changes aren't there" On Windows, Go can't overwrite a running .exe. Stop all janus.exe in Task Manager, then rebuild.
"Address already in use" A leftover process holds port 8990. Task Manager → end all janus.exe.
Server starts, chat fails JANUS_MODEL_PATH wrong or no .gguf in models/. Set the path in .env, put the file in models/.
"local engine failed to start" Missing llama.dll or GPU driver issue. Run .\build.ps1. Update GPU drivers, or set INFERENCE_BACKEND=cpu.
Wrong URL Default is http://127.0.0.1:8990, not 8080. Bookmark 8990.
First reply takes forever Model loading into VRAM — normal. Wait 10–60s; smaller quants (Q4) load faster.

Project layout

cmd/janus/          Main server (OpenAI-compatible API)
cmd/modelget/       Hugging Face model downloader
internal/engine/    llama.cpp Vulkan/CPU backend
internal/bridge/    DLL loader and FFI bindings
internal/singleton/ Single-instance guard
models/             Put .gguf files here (not committed)
dist/               janus.exe + llama.dll after build

Development

Get-Process -Name "janus" -ErrorAction SilentlyContinue | Stop-Process -Force
go test ./...
go build -o dist\janus.exe .\cmd\janus
.\dist\janus.exe

Or just .\run.ps1. For a full build including llama DLLs, use .\build.ps1.

Contributions welcome — see CONTRIBUTING.md.

License

MIT — see LICENSE.

Janus (Vibra-Ingenn/Janus)

Searching Database…

135 ·GitHub
Janus
What is this?Janus is a single Go binary that runs local .gguf language models on your machine (GPU or CPU) and exposes an OpenAI-compatible API, so you can use local models without Python, Docker, or Ollama.
Is this a chat app or a local API server?It is a local API server you run as one executable (janus.exe on Windows) — there is no bundled chat UI, you talk to it via curl, scripts, or any OpenAI-compatible client.
How do I get it running?Clone the repo and run build.ps1 on Windows (or build.sh on Linux) to compile dist/janus.exe, drop a .gguf file into the models/ folder, edit .env to point at it, then run the binary.
What do I need installed before building it?You need Go 1.25+ to build it, and a Vulkan-capable GPU (AMD/Intel/NVIDIA) is recommended for acceleration, otherwise it falls back to CPU.
Any platform or build pitfalls I should know about?Linux support is amd64-only with macOS/arm64 explicitly untested and unsupported, and on Windows you must kill any running janus.exe in Task Manager before rebuilding or it will silently keep running the old binary.
Does Janus work with coding tools like Cursor or Cline?Yes, since it exposes an OpenAI-compatible API at endpoints like /v1/chat/completions, you can point Cursor, Cline, or any OpenAI-compatible client at it to use your local models.
Can I switch models without restarting the server?Yes, the /models/load endpoint hot-swaps the loaded .gguf model without needing to restart Janus.
Does it support reasoning (thinking) models?Yes, Janus has thinking model support and splits the model's reasoning into a separate reasoning_content field in its responses.
Why pick this over Ollama?Janus ships as a single zero-dependency binary with no Python, Docker, or Ollama needed, uses Vulkan for GPU acceleration across AMD, Intel, and NVIDIA cards, and auto-detects the chat template from GGUF metadata.
What are similar projects?project-maya, Strata, llama-cpp-python, rizzo-flow, llamafile, Qwen3.8-27B-16gb-NVIDIA-GPUs-one-click-install, llama-gpt, deltafin
Which topics is it in?llamacpp, gguf, local-ai, llm-inference, openai-compatible, local-llm, streaming, offline, inference, nvidia, local-first, rest-api
Creator
Vibra-Ingenn
Size
0.7 MB
Stars
135
Updated
CommitOct 08
Age
23d
Commits · 90 Days