Semantic ifs from open models, on a 3090 at home.
Independent project; not affiliated with Jev or TypeSafe.
Wow! No waitlist. Run it in your browser today.
Same frozen 4B model · same state · same 21 questions · measured separately, aligned at t=0 in the replay
Independent research project. SemIf was formerly called OpenJev. It is not affiliated with or endorsed by TypeSafe. Jev, TypeSafe, and other names and marks are the property of their respective owners. No infringement is intended.
Most agent decisions are small: route this, retry that, does the evidence support X? A chat model can answer them, but it spends time generating text that software immediately parses back into an if statement.
Jev is TypeSafe's closed service for runtime-defined semantic decisions. This project reproduces that interface pattern with open models; it does not reproduce Jev's undisclosed model or training.
This baseline reads typed option probabilities directly from a model. No answer sentence, JSON repair, or decoding loop.
Latest changes
2026-09-22
- Added PyTorch/MPS scoring for Apple Silicon — @dp-IED in #15.
- Added a Qwen3.8-27B EXL3 bridge with corrected, committed evidence — @jkyamog in #9.
- Added per-workload temperature calibration and calibrated prediction outputs — @samarthpatel24 in #19.
2026-09-18
- Added MiniCPM5 2B and Qwen3.5 4B to the browser demo.
- Added Unsloppify site, a switch to a conventional interface.
Quick start
Apple Silicon: use the native MLX backend for direct scoring,
serial prefix reuse, and parallel shared-state decisions on macOS arm64.
Install pip install -e '.[test,mlx]' and add --backend mlx to the scorer command.
PyTorch/MPS (--device mps) is also supported for direct, serial, and shared modes — see
Apple Silicon.
Python 3.10+, CUDA, and a GPU that can hold a 4B BF16 model:
python -m venv .venv
. .venv/bin/activate
export HF_HOME=/path/to/large-drive/huggingface
pip install -e '.[test]'
CPU only: the llama.cpp backend scores the same prompts from a local GGUF
checkpoint with no CUDA device. Install pip install -e '.[test,llamacpp]',
fetch a GGUF (for example Qwen_Qwen3.5-4B-Q4_K_M.gguf from
bartowski/Qwen_Qwen3.5-4B-GGUF), and add --backend llamacpp --gguf /path/to/model.gguf; --llama-threads caps the CPU threads. Prompt
construction stays on the pinned reference tokenizer, so prompt_sha256
matches the Torch backend row for row; scores carry the GGUF checksum and are
conditional on the quantized weights. Direct and prefix-cached execution can
have small numerical differences from different llama.cpp evaluation paths;
compare decisions or probabilities with a tolerance rather than raw logits
bit for bit. One loaded backend owns one stateful scoring context.
Run the owned examples:
CUDA_VISIBLE_DEVICES=0 semif-score \
--mode direct \
--model Qwen/Qwen3.5-4B \
--revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a \
--input examples/decisions.jsonl \
--output results.jsonl
Each result contains typed option scores, timing, the exact model revision, and a prompt hash.
If every row has the same exact state, switch to --mode shared to prefill it once and evaluate the criteria in parallel.
How it works
flowchart LR
S[Unstructured state] --> M[4B model]
C[Runtime criteria] --> M
O[Typed options] --> M
M -- native option logits --> P[Probabilities]
- Runtime-defined: criteria and option descriptions arrive with the request.
- Decision-native: one forward pass reads declared option logits; no answer token is sampled.
- Shared-state aware: one long state can be prefetched once, then branched across many criteria.
- Auditable: the owned fixture, exact runners, row-level outputs, revisions, prompts, and known failures are committed.
Speed
Decisions versus a compact generated array
Same frozen Qwen3.5-4B, same owned state, same 21 binary criteria, one RTX 3090:
| Output path |
Time |
Output tokens |
Result |
| Direct typed logits, median of 3 |
1.023 s |
0 |
21 probability pairs |
| Autoregressive JSON array, median of 3 |
5.332 s |
111 |
Valid ordered 21-value array |
The compact generative baseline emits only ordered "yes"/"no" values—no keys, confidence objects, or explanations. Its median first-token time was 0.489 s, but completing the array took 5.21× as long as direct readout. All three arrays were valid and identical. Their choices agreed with direct argmax on 18/21 criteria, so this is a systems comparison rather than a claim that the two readouts are semantically equivalent. Exact prompt, outputs, token timeline, and runs are committed.
Reusing a state across 21 decisions
On an owned 37-state × 21-criterion workload:
| Execution path |
Decisions/s |
777 decisions |
| Fresh direct scoring |
2.33 |
333.1 s |
| Serial prefix reuse |
10.75 |
72.3 s |
| Parallel suffixes |
20.03 |
38.8 s |
| Native reranker |
1.86 |
417.3 s |
The owned 37×21 fixture, direct/reuse runner, reranker runner, raw timings, and row-level predictions are included. The fast reuse paths are experimental: BF16 execution changed 5–6 of 777 argmaxes relative to fresh scoring.
Quality
Browser model ladder
| System |
Browser artifact |
Download |
Authored balanced accuracy |
Perturbation balanced accuracy |
TypeSafe subset agreement |
| Qwen3-0.6B |
Q8_0 |
639 MB |
0.440 |
0.528 |
0.407 |
| MiniCPM5-2B |
Q4_K_M |
1.56 GB |
0.686 |
0.693 |
0.637 |
| Qwen3.5-4B |
Q4_K_M |
3.01 GB |
0.813 |
0.766 |
0.845 |
| Published Jev |
Closed hosted service |
— |
— |
— |
0.883 |
Native BF16 scores. Browser builds use quantized GGUF. Jev is TypeSafe's published result on the same 102-row subset.
General decision baseline
| Frozen workload |
Rows |
Direct logits (4B BF16) |
EXL3 direct (27B, 5 bpw) |
Native reranker (4B) |
Published Jev |
| Authored decisions, balanced accuracy |
144 |
0.813 |
0.958 |
0.625 |
— |
| WANLI, balanced accuracy |
256 |
0.637 |
— |
0.522 |
— |
| TypeSafe selected subset, modal agreement |
102 across 20 cases |
0.845 |
— |
0.560 |
0.883 |
| Every judgment grid, accuracy |
36 |
0.806 |
— |
0.694 |
— |
| Every action firewall, composed accuracy |
10 actions |
0.700 |
— |
0.700 |
— |
| Every code retrieval, Recall@1 |
6 queries |
1.000 |
— |
1.000 |
— |
| Every company knowledge, Recall@1 |
7 queries |
0.929 |
— |
0.929 |
— |
The reranker remained strong at retrieval ranking, but direct logits were the better general-decision baseline.
The Jev number is read from TypeSafe's published records; we did not run a live Jev endpoint. The comparison covers the 102 rows that could be aligned from public artifacts, not TypeSafe's reported 711-row aggregate.
The Qwen3.8-27B EXL3 bridge uses the same 144 authored rows, matching prompt hashes, options, direct-logit readout, and metric as the 4B baseline. It is a system-level quality comparison rather than a controlled model-size or quantization ablation: model family, size, quantization, and runtime all differ. It has not yet been run on the other quality workloads. Across the 777-decision shared-state fixture, its choices agree with the pinned 4B model on 84.43% of rows.
Calibration
Option probabilities are useful only when their confidence matches observed accuracy. SemIf includes per-workload temperature scaling fitted on labeled decisions:
| Workload |
Raw ECE |
Calibrated ECE, out of fold |
Temperature |
| Authored decisions |
0.068 |
0.038 |
1.23 |
| WANLI |
0.208 |
0.069 |
2.50 |
| Every judgments |
0.050 |
0.047 |
1.71 |
Calibration does not change the selected option. The clear improvement is on WANLI; the intervals overlap on the authored and Every workloads. See the method, caveats, and reproduction commands.
{
"id": "route-1",
"state": "Customer cannot access an account after a password reset.",
"question": "Which queue should handle this request?",
"options": [
{"id": "access", "description": "Account access support."},
{"id": "billing", "description": "Billing support."}
]
}
Returned probabilities are conditional on the supplied options. Calibrate and validate them on the workload where they will make decisions.
state may also be a nonempty JSON object or array. Direct modes preserve it as structured JSON; reranker mode renders it as document text.
Documentation
- Results — quality, speed, perturbations, and claim boundaries
- Method — frozen prompts, metrics, and timing scope
- Reproduce — exact environment, pinned commands, perturbations, and verification
- Apple Silicon — MPS and optional MLX backends
- Calibration — fitted temperatures, out-of-fold evidence, and application
- EXL3 bridge — quantized 27B runner and committed evidence
- Interactive replay
- Browser-only WebGPU demo — no waitlist; use it today
- Machine-readable summary
- Benchmark bundle — fixtures, runners, selection IDs, and reproduction commands
- Raw results and checksums
- Third-party sources
Star history
Evaluation sources
Model weights and third-party source records are not included. Upstream models retain their licenses. Project code is released under the MIT License.
SemIf(原名 OpenJev)是一个独立研究项目,在一块消费级 GPU 上用开放模型运行运行时定义的语义判断,接口与 TypeSafe 的闭源 Jev 服务一致,但不涉及后者未公开的模型。