JevAny README

项目的 README,来自 github.com/SimpleJev/JevAny。合并为一个 markdown 文件:README.md。

JevAny: Your Jev from Any Model to Any Application

Homepage Checkpoints API docs Examples Python 3.12+ Tests License

🇺🇸 English | 🇨🇳 简体中文
🎮 Results & Demos | ⚡ Quickstart | 💻 Run locally | 🤗 Models | 📊 Benchmarks | 📚 Docs

JevAny is open infra for System 1 decision model training and deployment, covering data preparation, model adaptation and evaluation. Use a released model or train on your own data to route support tickets, select tools, or choose a robot's next action. One API takes the state, question and candidate options, then directly returns a choice and its probabilities.

JevAny infra for System 1 decision model training, deployment and application integration

🎮 Results and Demos

JevAny checkpoints and baselines compared on Transfer and JevBench in side-by-side bar charts

Interactive benchmark results.

The following 30 examples are archived replays from an earlier compatible JevAny checkpoint. The current default release is JevAny-Qwen3.8-27B. Explore the cases, or run a model locally to try your own inputs and see its choices and probabilities.

JevAny choosing actions across robotics, browser, software, laboratory and mobility tasks

⚡ Jev inside LLM agent loops

The LLM plans, recovers and finishes the task; Jev picks the next action wherever the loop offers a bounded set of candidates. Each animation replays one recorded pair, LLM only on the left and LLM + Jev on the right, with the measured totals on the clocks and schematic step artwork.

1. WebShop. Jev picks the requested color and size from menus the LLM generated, and the LLM buys the product.

WebShop: LLM only needs nine actions and 18.54 seconds; LLM + Jev buys the same blazer in five actions and 7.83 seconds

2. FrozenLake. One LLM plan, then Jev compares the four moves at each new state.

FrozenLake: both runs reach the goal in four moves; LLM only spends four LLM calls and 19.7 seconds, LLM + Jev one call and 16.7 seconds

3. Terminal-Bench. On sqlite-db-truncate, Jev picks raw-page inspection out of three probes, and the LLM recovers and verifies ten rows.

Terminal-Bench sqlite-db-truncate: both runs recover ten rows; LLM only needs thirteen commands and 187.9 seconds, LLM + Jev seven commands and 144.7 seconds

Overview · All demos

Full results, delegation protocol and technical report.

📑 Table of Contents

⚡ 1. Quickstart

Use Python 3.12 or newer. Clone the repository and install the lightweight package:

git clone https://github.com/SimpleJev/JevAny.git
cd JevAny
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -e .

Keep this environment active and work from the repository root. Start with a local demo, then train on your own data or use the API.

💻 1.1 Run locally

Choose a model that fits your computer:

Model Hardware Start here
Qwen 0.8B starter CPU · 16 GB RAM recommended Train the small adapter on the bundled tickets
JevAny-Qwen 4B CUDA · ~8 GB for BF16 base weights, plus runtime memory Load the released model
JevAny-Qwen 27B CUDA · ~54 GB for BF16 base weights, plus runtime memory Choose the larger checkpoint

The local model guide covers preparation and loading. Released models download on first use and reuse the local cache. With the model server running, open a second terminal in the same checkout:

source .venv/bin/activate
jevany demo --base-url http://127.0.0.1:8008 --text-only

Open http://127.0.0.1:8090, choose Test and connect, then edit Try your own decision and press Ask the model. Change the state or options to see how its decision changes. Games, robotics and replays are available in the same playground.

🛠️ 1.2 JevAny Training

Train your own System 1 model on the same state and questions you send at inference, with a label for each question. Start with the bundled synthetic support tickets, then train on your own labelled data. The starter recipe uses Qwen3.5-0.8B on CUDA with BF16 and writes runs/my-jev:

python -m pip install -e '.[train]'
jevany data init --out data/starter
jevany data validate data/starter/train.jsonl
jevany train --config recipes/sft.toml --dry-run
jevany train --config recipes/sft.toml

After training, try the checkpoint on the included ticket request:

jevany decide examples/request.json --checkpoint runs/my-jev

Pass --data to train on your own JSONL data, or use recipes/finetune.toml to adapt the released 27B model. See the training guide for CPU settings, multimodal data and standard torchrun launches. For image/video training or fine-tuning the released 27B model, install .[train,multimodal].

After SFT, you can continue with experimental RLCR, which rewards correctness and probability calibration:

jevany train --config recipes/rlcr.toml

🚀 1.3 JevAny Deployment

Install the serving dependencies and start the released Qwen 4B model on a CUDA GPU. See the hardware and loading guide for memory requirements.

python -m pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Qwen3.5-4B-LoRA \
  --device cuda --dtype bf16 --port 8008

The default path favors reproducibility. CUDA deployments can opt into BF16 LoRA merging, SDPA and torch.compile; the useful settings differ between 4B and 27B. See the inference acceleration guide for commands, H200 measurements and accuracy caveats.

To serve your training output, replace the checkpoint ID with runs/my-jev. Keep the server running. In a Python session using the same environment, send a ticket and the departments that can handle it:

from jevany import Choice, JevClient

jev = JevClient("http://127.0.0.1:8008")
result = jev.system_one(
    state={"ticket": "I was charged twice. Please help."},
    questions={
        "department": Choice(
            instructions="Which team should handle this?",
            criteria={"billing": "Payment problems", "shipping": "Delivery problems"},
        ),
    },
)
answer = result["answers"]["department"]
print("Selected team:", answer["choice"])
print("Probabilities:", answer["probabilities"])

choice is one of the department names; probabilities maps each name to its probability. Your application can use these fields to route the ticket or ask for review when the decision is uncertain. Use Noul for yes/no questions, such as whether a ticket needs urgent review, and Score for ordered levels, such as low, normal and high priority. See the API reference for all three question types.

For in-process inference, load a model in Python and use the same interface. For image and video inputs, follow the media setup.

🤗 2. Pretrained Models

For a first local run, choose a model and hardware in Run locally.

Model Readout Intended use
 JevAny-Gemma-4B Pointer Compact Gemma release
 JevAny-Qwen3.5-4B Pointer Compact, flexible choice count
 JevAny-Qwen3.5-4B-Direct-Token Direct-token Best released 4B JevBench accuracy
 JevAny-Qwen3.8-27B Pointer Default; highest released accuracy
 JevAny-Muse-Glimmer-30B Pointer Muse Glimmer alternative

These LoRA adapters were trained with SFT on 1,772,725 text records containing 2,180,242 labelled decisions; see training compute and experiments for the setup. Full-parameter SFT and further post-training improvements are planned.

The corresponding base model is loaded separately and its license and access terms apply. Allow roughly twice the base parameter count in bytes for BF16 weights, plus runtime memory. See the hardware and loading guide.

Pointer and direct-token models share the same API. Pointer supports up to 4,096 options within the context limit; direct-token supports up to 255. See readout choices for training and accuracy tradeoffs.

Training-free choice-token readout

Choice-token is a training-free readout for up to 52 options: label them with the one-token IDs A–Z, a–z, score the answer-position logits, and renormalize. It works on a frozen base or with a Pointer/Direct-Token checkpoint; unlike temperature calibration, it can change the selected answer.

jevany eval --run SimpleJev/JevAny-Qwen3.5-4B-Direct-Token-LoRA \
  --suite /path/to/suite --out runs/choice --device cuda --readout choice

Method, commands and full results · Machine-readable results

Training-free choice-token, checkpoint-native, Transfer-dev-tuned blend, and external baseline accuracy on Typed Decisions and JevJudge text

Tune one blend weight on development data, freeze it before evaluation, and blend only when native and choice errors are complementary. Temperature changes confidence, not argmax.

Metric note: Cygnet's 73.70 is a v1.5.4 composite over 1,624 open and sealed items—not accuracy. Its comparable public-development accuracy is 203/231 (87.9%).

Key takeaway: Choice-token helps when extracting the decision—not reasoning ability—is the bottleneck; keep a blend only when it improves target-like held-out data.

📊 3. Benchmark Results

The release table below uses each checkpoint's native readout. JevAny-Qwen3.8-27B leads both benchmarks and has the lowest NLL and Brier. Among 4B releases, direct-token leads on JevBench; pointer leads on Transfer.

JevAny checkpoints and baselines ranked by mean accuracy on Transfer and JevBench

Model Transfer ↑ JevBench ↑ NLL ↓ Brier ↓ ECE ↓
 Kev-4B 74.19% 75.32% 0.858 0.380 0.125
 Kev-27B 82.31% 85.28% 0.533 0.265 0.050
 Jev 1.13.0 85.37% 86.58% 0.644 0.212 0.033
 Laya 52.29% 58.01% 1.264 0.615 0.127
JevAny releases
 JevAny-Gemma-4B 70.84% 77.49% 0.706 0.369 0.056
 JevAny-Qwen3.5-4B 78.68% 80.09% 0.587 0.297 0.035
 JevAny-Qwen3.5-4B-Direct-Token 78.20% 80.95% 0.564 0.291 0.029
 JevAny-Muse-Glimmer-30B 83.46% 87.45% 0.464 0.229 0.032
 JevAny-Qwen3.8-27B 86.04% 90.04% 0.388 0.195 0.026

NLL, Brier and ECE are measured on Transfer.

Full results and protocols · Machine-readable results · Method and ablation report

The same 13-model cohort is compared by accuracy on Typed Decisions, JevJudge full (3,220 multimodal records), and its 724-record text subset. — means unsupported input or no matching result.

Accuracy on Typed Decisions, full JevJudge, and the JevJudge text-only subset for the same thirteen-model cohort

Full external tables and reproducibility notes · Machine-readable chart results

⏱️ 3.1 Inference efficiency

On H200, CUDA acceleration cuts Qwen3.8-27B median latency from 113.54 to 30.53 ms (3.72×) and Muse-Glimmer-30B from 100.71 to 43.25 ms (2.33×), with identical decisions (207/231 and 38/44). Each speedup is a within-row comparison; H200 and A100 rows use different fixed panels, so absolute latency is not compared across hardware.

Accuracy vs median latency before and after acceleration for JevAny and other decision models

Model Hardware Before After Speed-up Accuracy check Fixed panel
JevAny-Qwen3.5-4B A100 104.6 ms 25.3 ms 4.1× 78.68% → 78.87% Transfer, 1,046
JevAny-Qwen3.5-4B-Direct-Token A100 106.4 ms 25.9 ms 4.1× 78.11% → 78.39% Transfer, 1,046
JevAny-Gemma-4B A100 106.3 ms 31.9 ms 3.3× 70.84% → 70.84% Transfer, 1,046
JevAny-Muse-Glimmer-30B H200 100.71 ms 43.25 ms 2.33× 86.36% → 86.36% Transfer sample, 44
JevAny-Qwen3.8-27B H200 113.54 ms 30.53 ms 3.72× 89.61% → 89.61% JevBench public, 231

Median model-call latency, serial batch size 1. See the full report for the apples-to-apples A100 comparison and panel limitations.

Animation: normalized latency ladders for the 4B releases on A100 and the 27B and 30B releases on H200, with exact median latency and speed-up labels

4B on A100 and 27–30B on H200. Each row is normalized to its own baseline; compare stages only within that row.

Full tables, setup and other models · How to enable · H200 results · A100 results

🕹️ 4. Examples & Test Environments

The playground includes the three environments below. These GIFs preserve historical model actions and option probabilities; run the current JevAny-Qwen3.8-27B checkpoint with the commands in the playground guide.

🤖 4.1 Robot peg insertion

Use a Franka gripper to grasp, align and insert a peg, checked by PyBullet contact physics.

Robot browser replay showing the Franka arm inserting a peg, recorded model probabilities and physical success checks

🔫 4.2 Doom corridor · 3D

Clear the final room by defeating the enemies on the left and right, then move forward. The environment uses ViZDoom and the included Freedoom assets.

Doom checkpoint replay: kill both enemies, then advance

⛏️ 4.3 Crafter survival · 2D

Gather wood, craft tools and mine stone while managing health and supplies.

Crafter browser replay showing resource gathering, crafting actions and progress through four goal milestones

🎮 4.4 Try the playground

With a model running from Run locally, open the playground:

jevany demo --base-url http://127.0.0.1:8008 --text-only

Open http://127.0.0.1:8090, choose Test and connect, and try your own decision. To let the model control a game, install the optional engines and restart the playground:

python -m pip install -e '.[demo]'
jevany demo --base-url http://127.0.0.1:8008 --text-only

Choose Run model, then One decision or Run automatically. Play yourself lets you control the game. Live control sends text state to the model; robot control uses the .[robotics] extra.

For the bundled recordings, run jevany demo and choose Replay. Playback works on CPU without model weights. See the playground guide for platform requirements and environment APIs, or integrations to combine JevAny decisions with an LLM planner.

🧩 5. Supported Model Families

Model IDs, supported inputs and setup requirements.

26 supported models across Qwen, Gemma, Muse, Mistral, GLM, Nemotron and Llama

📚 6. Documentation and Contributing

Documentation hub · Training · Deployment · API · Data · Evaluation · Agent harness protocol · Contributing

The technical report describes model design, multimodal support, the agent-harness study and appendix, negative results, and open questions.

Code and starter data are Apache-2.0. Some components are adapted from Kev; see NOTICE and acknowledgements. Base models and upstream datasets retain their own terms.

🤝 Contributing

We're building open infrastructure for System 1 decision models, and we'd love your help. Small improvements are welcome. Here are a few ways to get involved:

CI checks and automated AI code review provide feedback on PRs, with maintainers handling review and merging. Small fixes can go straight to a PR; for larger changes, open an issue to discuss the approach first.

Read the contribution guide for setup and development steps, or browse the issues to find a place to start.

JevAny (SimpleJev/JevAny)

正在搜索数据库…

138 ·GitHub
这是什么?JevAny 是一套开放的基础设施,用于训练、评估和部署“System 1”决策模型。这类模型根据给定的状态和问题,从有限的候选集合中选出一个操作,并返回选择结果及其概率。
这是我要导入的库,还是我要运行的服务器?两者都是:你可以将它安装为轻量级 Python 包,运行一个通过 HTTP 提供决策 API 的本地模型服务器,也可以直接在 Python 中加载检查点,进行进程内推理。
我该如何让它运行起来?使用 Python 3.12+,克隆仓库并安装轻量级包,然后根据你的硬件选择合适大小的模型(Qwen 0.8B 入门版对 CPU 友好)。之后,你既可以基于附带的工单训练适配器,也可以加载已发布的检查点并启动模型服务器。
较大的已发布模型有什么局限或代价?仅 27B 模型就需要约 54 GB 显存来存放 BF16 权重,此外还需要额外的运行时内存。Pointer 与 Direct-Token 两种读出格式的选项数量上限也不同,分别最多支持 4,096 个和 255 个候选项。
这是为谁打造的?面向开发 Agent 循环、机器人控制器或支持/路由系统的开发者。他们需要一个快速且校准良好的“从 N 个选项中选一个”决策步骤,而不是依赖 LLM 直接做选择。
我可以向 API 发送哪些类型的问题?支持三种类型:普通的 Choice(从候选列表中选择,并返回概率)、用于是非题的 Noul,以及用于有序等级(如低、普通、高优先级)的 Score。
开箱即用的预训练模型有哪些?JevAny 附带多个 LoRA 适配器:JevAny-Gemma-4B、JevAny-Qwen3.5-4B(普通版和 Direct-Token 版)、JevAny-Muse-Glimmer-30B,以及默认的 JevAny-Qwen3.8-27B。这些适配器均通过 SFT,在约 177 万条带标签的记录上训练而成。
默认模型与其他决策模型相比如何?JevAny-Qwen3.8-27B 在发布基准测试表中名列榜首,在 Transfer 上得分 86.04%,在 JevBench 上得分 90.04%。在所有受测模型中,它的 NLL、Brier 以及校准误差(ECE)也都是最低的。
我能不写代码就测试它吗?可以。附带的 Playground 界面允许你连接到正在运行的模型服务器,输入自己的状态和选项,并实时查看决策及概率的变化。你还可以体验机器人、Doom 和 Crafter 演示环境。
作者
SimpleJev
大小
347.9 MB
星标
138
时长
1个月
提交 · 90 天