SemIf-OpenJev README

The project's README from github.com/TheoLeeCJ/SemIf-OpenJev. As one markdown file: README.md.

SemIf (formerly OpenJev)

Semantic ifs from open models, on a 3090 at home.

Independent project; not affiliated with Jev or TypeSafe.

Wow! No waitlist. Run it in your browser today.

Measured replay: typed decisions appear together while JSON streams token by token

Same frozen 4B model · same state · same 21 questions · measured separately, aligned at t=0 in the replay

Independent research project. SemIf was formerly called OpenJev. It is not affiliated with or endorsed by TypeSafe. Jev, TypeSafe, and other names and marks are the property of their respective owners. No infringement is intended.

Some AI company asks you to join a waitlist; SemIf runs in your browser today

Most agent decisions are small: route this, retry that, does the evidence support X? A chat model can answer them, but it spends time generating text that software immediately parses back into an if statement.

Jev is TypeSafe's closed service for runtime-defined semantic decisions. This project reproduces that interface pattern with open models; it does not reproduce Jev's undisclosed model or training.

This baseline reads typed option probabilities directly from a model. No answer sentence, JSON repair, or decoding loop.

Latest changes

2026-09-22

2026-09-18

Quick start

Apple Silicon: use the native MLX backend for direct scoring, serial prefix reuse, and parallel shared-state decisions on macOS arm64. Install pip install -e '.[test,mlx]' and add --backend mlx to the scorer command. PyTorch/MPS (--device mps) is also supported for direct, serial, and shared modes — see Apple Silicon.

Python 3.10+, CUDA, and a GPU that can hold a 4B BF16 model:

python -m venv .venv
. .venv/bin/activate
export HF_HOME=/path/to/large-drive/huggingface
pip install -e '.[test]'

CPU only: the llama.cpp backend scores the same prompts from a local GGUF checkpoint with no CUDA device. Install pip install -e '.[test,llamacpp]', fetch a GGUF (for example Qwen_Qwen3.5-4B-Q4_K_M.gguf from bartowski/Qwen_Qwen3.5-4B-GGUF), and add --backend llamacpp --gguf /path/to/model.gguf; --llama-threads caps the CPU threads. Prompt construction stays on the pinned reference tokenizer, so prompt_sha256 matches the Torch backend row for row; scores carry the GGUF checksum and are conditional on the quantized weights. Direct and prefix-cached execution can have small numerical differences from different llama.cpp evaluation paths; compare decisions or probabilities with a tolerance rather than raw logits bit for bit. One loaded backend owns one stateful scoring context.

Run the owned examples:

CUDA_VISIBLE_DEVICES=0 semif-score \
  --mode direct \
  --model Qwen/Qwen3.5-4B \
  --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
  --input examples/decisions.jsonl \
  --output results.jsonl

Each result contains typed option scores, timing, the exact model revision, and a prompt hash.

If every row has the same exact state, switch to --mode shared to prefill it once and evaluate the criteria in parallel.

How it works

flowchart LR
    S[Unstructured state] --> M[4B model]
    C[Runtime criteria] --> M
    O[Typed options] --> M
    M -- native option logits --> P[Probabilities]

Speed

Decisions versus a compact generated array

Same frozen Qwen3.5-4B, same owned state, same 21 binary criteria, one RTX 3090:

Output path Time Output tokens Result
Direct typed logits, median of 3 1.023 s 0 21 probability pairs
Autoregressive JSON array, median of 3 5.332 s 111 Valid ordered 21-value array

The compact generative baseline emits only ordered "yes"/"no" values—no keys, confidence objects, or explanations. Its median first-token time was 0.489 s, but completing the array took 5.21× as long as direct readout. All three arrays were valid and identical. Their choices agreed with direct argmax on 18/21 criteria, so this is a systems comparison rather than a claim that the two readouts are semantically equivalent. Exact prompt, outputs, token timeline, and runs are committed.

Reusing a state across 21 decisions

On an owned 37-state × 21-criterion workload:

Execution path Decisions/s 777 decisions
Fresh direct scoring 2.33 333.1 s
Serial prefix reuse 10.75 72.3 s
Parallel suffixes 20.03 38.8 s
Native reranker 1.86 417.3 s

The owned 37×21 fixture, direct/reuse runner, reranker runner, raw timings, and row-level predictions are included. The fast reuse paths are experimental: BF16 execution changed 5–6 of 777 argmaxes relative to fresh scoring.

Quality

Browser model ladder

System Browser artifact Download Authored balanced accuracy Perturbation balanced accuracy TypeSafe subset agreement
Qwen3-0.6B Q8_0 639 MB 0.440 0.528 0.407
MiniCPM5-2B Q4_K_M 1.56 GB 0.686 0.693 0.637
Qwen3.5-4B Q4_K_M 3.01 GB 0.813 0.766 0.845
Published Jev Closed hosted service — — — 0.883

Native BF16 scores. Browser builds use quantized GGUF. Jev is TypeSafe's published result on the same 102-row subset.

General decision baseline

Frozen workload Rows Direct logits (4B BF16) EXL3 direct (27B, 5 bpw) Native reranker (4B) Published Jev
Authored decisions, balanced accuracy 144 0.813 0.958 0.625 —
WANLI, balanced accuracy 256 0.637 — 0.522 —
TypeSafe selected subset, modal agreement 102 across 20 cases 0.845 — 0.560 0.883
Every judgment grid, accuracy 36 0.806 — 0.694 —
Every action firewall, composed accuracy 10 actions 0.700 — 0.700 —
Every code retrieval, Recall@1 6 queries 1.000 — 1.000 —
Every company knowledge, Recall@1 7 queries 0.929 — 0.929 —

The reranker remained strong at retrieval ranking, but direct logits were the better general-decision baseline.

The Jev number is read from TypeSafe's published records; we did not run a live Jev endpoint. The comparison covers the 102 rows that could be aligned from public artifacts, not TypeSafe's reported 711-row aggregate.

The Qwen3.8-27B EXL3 bridge uses the same 144 authored rows, matching prompt hashes, options, direct-logit readout, and metric as the 4B baseline. It is a system-level quality comparison rather than a controlled model-size or quantization ablation: model family, size, quantization, and runtime all differ. It has not yet been run on the other quality workloads. Across the 777-decision shared-state fixture, its choices agree with the pinned 4B model on 84.43% of rows.

Calibration

Option probabilities are useful only when their confidence matches observed accuracy. SemIf includes per-workload temperature scaling fitted on labeled decisions:

Workload Raw ECE Calibrated ECE, out of fold Temperature
Authored decisions 0.068 0.038 1.23
WANLI 0.208 0.069 2.50
Every judgments 0.050 0.047 1.71

Calibration does not change the selected option. The clear improvement is on WANLI; the intervals overlap on the authored and Every workloads. See the method, caveats, and reproduction commands.

Input

{
  "id": "route-1",
  "state": "Customer cannot access an account after a password reset.",
  "question": "Which queue should handle this request?",
  "options": [
    {"id": "access", "description": "Account access support."},
    {"id": "billing", "description": "Billing support."}
  ]
}

Returned probabilities are conditional on the supplied options. Calibrate and validate them on the workload where they will make decisions. state may also be a nonempty JSON object or array. Direct modes preserve it as structured JSON; reranker mode renders it as document text.

Documentation

Star history

SemIf star history

Evaluation sources

Model weights and third-party source records are not included. Upstream models retain their licenses. Project code is released under the MIT License.

SemIf-OpenJev (TheoLeeCJ/SemIf-OpenJev)

Searching Database…

4.8K ·GitHub
SemIf-OpenJev
What is this?SemIf (formerly OpenJev) is an independent research project that runs runtime-defined semantic decisions with open models on one consumer GPU, mirroring the interface of TypeSafe's closed Jev service but not its undisclosed model.
Is it a hosted service I sign up for, or a package I install?Neither a service nor a waitlist: it is an installable Python scorer with committed benchmark fixtures, plus a browser-only WebGPU demo you can open right now.
How do I get it running?On Apple Silicon run pip install -e '.[test,mlx]' and add --backend mlx to the scorer command; for CPU-only machines install '.[test,llamacpp]', fetch a GGUF checkpoint, and pass --backend llamacpp --gguf /path/to/model.gguf. There is also a zero-install browser demo.
Do I need Python and an NVIDIA GPU to run it?Python 3.10+ yes; the default path also wants CUDA and a GPU that can hold a 4B BF16 model. Alternatives: Apple Silicon via MLX or PyTorch/MPS, or CPU-only scoring through llama.cpp.
How much should I trust the probabilities it returns?They are conditional on the options you supply, so calibrate them on your own workload (per-workload temperature scaling is built in), and the fast shared-state reuse paths are experimental — BF16 execution flipped 5-6 of 777 argmaxes versus fresh scoring.
Does everything stay on my machine, or does it call a cloud API?It is fully local: CUDA, MLX, MPS and CPU backends score local weights, and the browser demo runs 0.6B-4B models in-browser over WebGPU from 639 MB-3.01 GB downloads.
Is the code open source?MIT. Model weights are not bundled — upstream models keep their own licenses.
Does it have to be an RTX 3090, or will other hardware work?Any GPU that fits a 4B BF16 model works, Apple Silicon runs natively via MLX/MPS, and a CPU-only llama.cpp backend scores the same prompts from a local GGUF checkpoint with no CUDA device at all.
How much faster is direct logit readout than generating a JSON array?On one RTX 3090 with the same frozen Qwen3.5-4B, 21 binary criteria took a median 1.023 s with zero output tokens versus 5.332 s and 111 tokens for an autoregressive JSON array — 5.21x slower.
How close does it get to the closed Jev service on quality?Qwen3.5-4B agreed with TypeSafe's published results on 0.845 of the aligned 102-row subset versus Jev's 0.883; the Qwen3.8-27B EXL3 bridge reached 0.958 balanced accuracy on the authored-decisions workload.
Which models are supported out of the box?Frozen baselines and browser builds for Qwen3-0.6B, MiniCPM5-2B and Qwen3.5-4B, plus a Qwen3.8-27B EXL3 bridge and Qwen3-Reranker-4B for comparison; weights are fetched separately, never bundled.
What are similar projects?decider, StartLux-Decision, rizzo-flow, NeoHorse, nimble, strands-decider, Strata, qwen38-mtp
Which topics is it in?decision-making, webgpu, llama-cpp, mlx, gguf, qwen, evaluation, quantization, local-llm, apple-silicon, inference, cuda
Creator
TheoLeeCJ
Size
9.3 MB
Stars
4.8K
Updated
CommitSep 23
Age
1mo
Commits · 90 Days