localjev README

The project's README from github.com/githubnext/localjev. As one markdown file: README.md.

LocalJev

A local, Jev-compatible POST /v1/systemone API written in TypeScript for Bun, backed by DiffusionGemma through an OpenAI-compatible Chat Completions endpoint.

The defaults target:

Why a bridge is needed

Jev uses a typed decision API rather than an OpenAI chat API. OpenJev implements the Jev wire protocol and obtains probabilities with a special one-step DiffusionGemma structured read. Its backend depends on unmerged vLLM request extensions such as diffusion_seed_canvas, diffusion_read_only, and requested token logprobs.

The normal oMLX API does not expose those primitives. LocalJev therefore takes the portable approach:

  1. translate state and typed Jev questions into a classification prompt;
  2. ask DiffusionGemma for a JSON probability scalar/vector;
  3. validate the complete result and retry malformed output;
  4. normalize vectors and calculate Jev-compatible choices, expected scores, and entropy-based confidence;
  5. return the normal Jev response shape.

This is wire-compatible, but not mathematically equivalent to OpenJev's logit read. The probabilities are generated/self-reported by the model rather than read directly from its logits. Evaluate their calibration on your own workload before relying on them for consequential decisions.

Run with oMLX

Requires Bun 1.2+ and a running oMLX server.

bun install
cp .env.example .env
$EDITOR .env # replace the upstream API-key placeholder
bun run start

Bun loads .env automatically. Alternatively, set the key in your shell before starting the server:

# fish
set -gx LOCALJEV_UPSTREAM_API_KEY 'your-local-omlx-key'

# bash/zsh
export LOCALJEV_UPSTREAM_API_KEY='your-local-omlx-key'

LocalJev listens on http://127.0.0.1:8080. Check that the configured model is available:

curl http://127.0.0.1:8080/ready

Make a decision:

curl http://127.0.0.1:8080/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "jev-latest",
    "state": "Hi, I have been trying to connect Stripe but keep getting a 403 error.",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "Payment or subscription issues",
          "technical": "Bugs or integration problems",
          "sales": "Pricing or account questions"
        }
      },
      "frustration": {
        "type": "score",
        "instructions": "How frustrated does the customer appear?",
        "criteria": ["Calm", "Frustrated but civil", "Very angry"]
      },
      "urgent": {
        "type": "noul",
        "instructions": "Does this require an immediate response?"
      }
    }
  }'

Use the TypeSafe SDK

The SDK requires an API-key value. LocalJev accepts any value unless LOCALJEV_API_KEY is configured. Set the SDK environment for your shell:

# fish
set -gx TYPESAFE_BASE_URL http://127.0.0.1:8080
set -gx TYPESAFE_API_KEY local

# bash/zsh
export TYPESAFE_BASE_URL=http://127.0.0.1:8080
export TYPESAFE_API_KEY=local

from typesafe_sdk import TypeSafeClient

client = TypeSafeClient()
response = client.system_one(
    "I was charged twice this month.",
    {
        "billing": {
            "type": "noul",
            "instructions": "Is this a billing issue?",
        }
    },
)
print(response.nouls["billing"].noul)

jev-latest and jev-preview are accepted aliases so SDK defaults work unchanged.

Configuration

Variable Default Purpose
LOCALJEV_UPSTREAM http://127.0.0.1:8000 OpenAI-compatible base URL, with or without /v1
LOCALJEV_UPSTREAM_API_KEY empty Bearer key sent to the inference server
LOCALJEV_UPSTREAM_MODEL diffusiongemma-26B-A4B-it-4bit Upstream model identifier
LOCALJEV_API_KEY empty Optional Bearer key required from LocalJev clients
LOCALJEV_HOST 127.0.0.1 Listen address
LOCALJEV_PORT 8080 Listen port
LOCALJEV_TIMEOUT 180 Upstream timeout in seconds
LOCALJEV_MAX_INFLIGHT 2 Concurrent calls admitted upstream
LOCALJEV_MAX_QUEUE 64 Waiting decisions before HTTP 529
LOCALJEV_MALFORMED_RETRIES 2 Corrective retries for invalid model JSON
LOCALJEV_MAX_OUTPUT_TOKENS 2048 Per-completion output ceiling
LOCALJEV_QUESTIONS_PER_CALL 16 Chunking limit per model call
LOCALJEV_OUTCOMES_PER_CALL 128 Choice/score outcomes per model call

Bun automatically loads .env, so you can also copy .env.example, replace its placeholder, and run the server.

Development

bun install
bun test
bun run typecheck
bun run smoke       # live call to the configured inference server

Evaluate different models

The repeatable bake-off uses public gold labels for news categorization (AG News), yes/no reading comprehension (BoolQ), and five-level sentiment (SST-5). It runs the same LocalJev engine against five installed models, comparing quality, calibration, retries, and full-decision latency at two actual input lengths.

# Quick integration check (30 requests, not a meaningful quality sample)
bun run eval --out eval/runs/pilot --limit 3

# 5 models × 120 labeled examples × 2 input lengths = 1,200 requests
bun run eval --out eval/runs/my-bakeoff

# Regenerate a completed or partial report without running inference
bun run eval:report eval/runs/my-bakeoff

Requires oMLX and the upstream key in .env; no running LocalJev HTTP server or Python is needed. See the evaluation guide for pinned data sources, methodology, configuration, resuming runs, and limitations.

The first completed bake-off includes 1,200 requests on an M5 Max. Gemma 4 26B-A4B and Qwen3.6 were the strongest overall candidates in this small screening sample; the report includes per-task results, latency, context effects, and caveats rather than claiming a definitive winner.

Should you use LM Studio instead?

Not currently for this model. As of September 18, 2026, DiffusionGemma support is still tracked as open in both lmstudio-ai/mlx-engine#336 and lmstudio-ai/lmstudio-bug-tracker#2037. The reported MLX backend fails to load diffusion_gemma, while the normal llama.cpp backend reports an unknown architecture. oMLX already loads and serves your exact checkpoint successfully, so it is the better runner for this Mac today.

Even after LM Studio adds ordinary generation support, changing runners alone will not make the result OpenJev-equivalent. The runner must expose seeded diffusion canvases, read-only denoising, and selected-token logits/logprobs. If LM Studio only provides standard Chat Completions, LocalJev can use it by changing LOCALJEV_UPSTREAM, but the probability path remains prompted/self-reported.

For direct model probabilities, the best paths are:

  1. add the structured-read primitives to oMLX's DiffusionGemma lane and consume them here; or
  2. run OpenJev's patched vLLM backend on a supported NVIDIA machine.

localjev (githubnext/localjev)

Searching Database…

828 ·GitHub
Localjev
What is this?LocalJev is a local TypeScript/Bun server that implements a Jev-compatible POST /v1/systemone decision API, backed by the DiffusionGemma model through an OpenAI-compatible Chat Completions endpoint. It lets Jev clients run against a local open model instead of OpenJev's hosted service.
Is this a standalone server or a library I import into code?It is a standalone HTTP server you run with Bun, plus a companion TypeSafe SDK for calling it from client code; you don't write server-side code yourself.
How do I get it running?Install Bun 1.2+, start an oMLX server with the diffusiongemma-26B-A4B-it-4bit model loaded, then run the LocalJev server; it auto-loads a .env file and listens on 127.0.0.1:8080 by default.
Does it work with any OpenAI-compatible inference server?It needs an OpenAI-compatible upstream, but right now oMLX is the only backend confirmed to load DiffusionGemma; LM Studio's MLX and llama.cpp backends currently fail to load this model.
Are the decision probabilities as accurate as OpenJev's real logit read?No — DiffusionGemma's probabilities are self-reported by the model via a JSON prompt rather than read from logits, so results are wire-compatible but not mathematically equivalent to OpenJev; the README advises checking calibration before using them for consequential decisions.
Does this need Apple Silicon?Effectively yes today, since the only working backend (oMLX) targets Apple Silicon Macs; the published benchmark ran on an M5 Max. For true logit-based probabilities the alternative is OpenJev's patched vLLM backend on an NVIDIA machine.
Why not just use OpenJev directly?Jev uses a typed decision API rather than a plain chat API, and OpenJev's real backend depends on unmerged vLLM extensions (seeded diffusion canvases, read-only denoising, token logprobs) that normal servers don't expose. LocalJev approximates the same wire protocol by prompting the model for a JSON probability and normalizing it.
What is the model bake-off for?A repeatable benchmark comparing five installed models on AG News, BoolQ, and SST-5 for quality, calibration, retries, and latency, to help pick the best model for this bridge; Gemma 4 26B-A4B and Qwen3.6 scored strongest in the first run.
Can I switch to LM Studio instead of oMLX?Not for DiffusionGemma yet, since LM Studio's backends can't load it. You can repoint LOCALJEVUPSTREAM once support lands, but probabilities will remain self-reported rather than logit-based.
What SDK is LocalJev meant to work with?A TypeSafe SDK that accepts jev-latest and jev-preview as model aliases, so existing SDK code works unchanged; any API key value is accepted unless LOCALJEVAPIKEY is explicitly configured.
What are similar projects?uzu, slotstream, Rapid-MLX, jeff, dflash, unsloth, DeepAnalyze, NeoHorse
Which topics is it in?gemma, qwen3, openai-api, bun, vllm, openai-compatible, apple-silicon, inference, proxy, sdk, rest-api, local-first
Creator
githubnext
Size
0.3 MB
Stars
828
Updated
CommitSep 18
Age
1mo
Commits · 90 Days