JIT README

The project's README from github.com/bingreeky/JIT. As one markdown file: README.md.

JIT-Agent

Scaling Harness Intelligence via Just-in-Time Harness Evolution

GitHub arXiv Hugging Face License

What is JIT-Agent?

JIT-Agent is a compact meta-agent that writes your agent harness on the fly. Instead of precompiling one general-purpose scaffold and hoping it transfers, JIT-Agent takes a task spec, a protocol, a tool/skill registry, and a few retrieved prior harnesses, and emits an executable, task-specific harness that wraps any off-the-shelf agentic LLM — Model-as-a-Harness.

Overview of JIT-Agent
Overview of JIT-Agent. Given a task, JIT-Agent composes a problem-specific agent harness by selecting and instantiating four modules: memory, planning, action, and capability. Different task structures therefore induce distinct executable protocols and state organizations.

Every harness is factored into four modules — memory, planning, action, capability orchestration — implemented against the shared interfaces in HarnessFactory, so generation means emitting structured code rather than free-form agent programs. As traces and feedback come back, JIT-Agent revises the harness and updates the archive: harnesses keep improving at test time while the generator itself stays frozen.

JIT-Agent leaderboard across four representative agent benchmarks

Results. The resulting JIT-Agent-27B lifts a wide range of backbone agents across deep research, daily work, planning, and workspace tasks.

Building the scaffold turns out to be a trainable, transferable axis of agent intelligence — orthogonal to scaling the base model.

Repository layout

Directory What it holds
jit/ The meta agent: generation / repair prompts, best-of-N selection
scripts/ The agent kernel, tools, models, evaluation engine, and the two runners
harness_factory/ Hand-written harness implementations and their design write-ups
benchmark/ One adapter, config and evaluator per benchmark
dataset/ The benchmark data itself

Each directory has its own README with the details.

Setup

1. Clone the repository

git clone https://github.com/bingreeky/JIT.git
cd JIT

2. Environment (Python 3.11)

conda env create -f environment.yml && conda activate jit

or, in an existing environment: pip install -r requirements.txt. Serving a local meta model (vLLM/SGLang + torch) is deliberately not included — the pipeline only ever talks HTTP to it.

3. Credentials

cp .env.example .env   # then fill it in

Anything already exported in the shell wins over .env, and every model role can also be overridden per run on the command line.

Group Keys Used for
Execution model OPENAI_API_BASE, OPENAI_API_KEY, EXEC_MODEL runs the generated harness's agent loop
Judge model JUDGE_MODEL, optional JUDGE_API_* grades produced artifacts (falls back to the execution endpoint)
Meta model META_MODEL, META_API_BASE, META_API_KEY, META_TOKENIZER writes the harness (JIT pipeline only)
Tools SERPER_API_KEY, JINA_API_KEY web_search / crawl_page

4. Data

Small datasets ship in the repo; anything large is a documented download.

python scripts/check_datasets.py        # present / partial / missing, per benchmark
bash scripts/fetch_datasets.sh travel   # one benchmark ("all" ≈ 1 GB)

Usage

All modes share the same benchmark adapters, execution model, judge, and scoring path. The meta model writes a harness, the execution model runs it, and the judge model grades the result. Configure credentials in .env; CLI flags override them.

Goal Entry point Meta model Selection
Test a fixed HarnessFactory design scripts.run_seed_harness None None
Use a hosted API as the meta-agent scripts.run_jit OpenAI-compatible API judge
Evaluate the JIT checkpoint serve_meta_model.sh + scripts.run_jit Local JIT-27B logprob

1. Test a fixed HarnessFactory design. No meta model is called; the selected harness is executed and scored directly.

python -m scripts.run_seed_harness --bench xbench --list-harnesses
python -m scripts.run_seed_harness --bench xbench \
    --harness plan_and_execute --max-samples 5

See the HarnessFactory guide for the eleven included designs.

2. Use a hosted API model as the meta-agent. Hosted APIs usually do not expose prompt_logprobs, so use judge selection explicitly. META_API_KEY is read from .env.

python -m scripts.run_jit --bench xbench \
    --meta-model provider-model --meta-base https://api.provider.com/v1 \
    --selector judge --rollouts 3 --max-samples 5

3. Evaluate the JIT checkpoint. Serve the checkpoint, then use its tokenizer for the published log-probability selector.

MODEL=JIT-Agent/jit-27b SERVED_NAME=jit TP=4 \
    bash scripts/serve_meta_model.sh

python -m scripts.run_jit --bench xbench \
    --meta-model jit --meta-base http://127.0.0.1:8000/v1 \
    --selector logprob --tokenizer JIT-Agent/jit-27b \
    --rollouts 3 --meta-temperature 1.0 --max-samples 5

Drop --max-samples for a full run. MODEL may also be a local checkpoint path; SERVED_NAME must match --meta-model. The shell wrapper bash scripts/run_jit.sh xbench reads the same settings from the environment.

Key arguments

Arguments Purpose
--bench, --dataset-path Select the benchmark and optionally override its data path.
--meta-model/base/key Configure the harness-generating model; JIT runs only.
--exec-model/base/key Configure the model that runs the harness.
--judge-model/base/key Configure the benchmark evaluator.
--rollouts, --meta-temperature Control candidate count and generation diversity.
--selector, --tokenizer Use judge for hosted APIs or logprob with a local tokenizer.
--harness-refs {desc,code} Choose design descriptions or sampled source harnesses as references.
--max-samples, --cases, --output Control smoke tests, case selection, and output location.
--workers-gen, --workers-exec Tune generation and execution concurrency independently.

Supported benchmarks are xbench, deepsearchqa, agentif, officebench, odyssey, shopping, and travel.

Output and resume. JIT runs separate generation, selection, and execution artifacts:

summary.json    headline metrics + how the run was configured
generate/       the N candidate harnesses per case, with prompts and responses
select/         the pick per case, the rule that produced it, per-candidate scores
execute/        the harness that actually ran, its trajectory, and the numbers you report

Fixed-harness runs write summary.json, scores.jsonl, and per-case reports directly. Re-running an identical command resumes completed work and retries only infrastructure failures; --skip-generate and --skip-select reuse earlier JIT phases.

Detailed documentation: JIT pipeline · HarnessFactory · CLI and runtime. Run either entry point with --help for the full flag list.

Citation

If you find JIT-Agent useful, please cite:

@misc{zhang2026jitagentscalingharnessintelligence,
      title={JIT-Agent: Scaling Harness Intelligence via Just-in-Time Harness Evolution},
      author={Guibin Zhang and Leo Lu and Fangzhou Xie and Kang Zhu and Junhao Wang and Zhifei Xie and Zhaochen Yu and Zihang Liu and Zhongxiang Sun and Qiankun Li and Yue Liao and Heng Chang and Xiaobin Hu and Qibing Ren and Wangchunshu Zhou and Shuicheng Yan},
      year={2026},
      eprint={2608.25593},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.25593},
}

License

See LICENSE.

JIT (bingreeky/JIT)

Searching Database…

556 ·GitHub
JIT
What is this?JIT-Agent is a compact meta-agent that writes a task-specific agent harness on the fly instead of relying on one precompiled general-purpose scaffold, wrapping any off-the-shelf agentic LLM in what the authors call 'Model-as-a-Harness'.
Is this a ready-made agent or something that builds agents for me?It is a meta-agent framework: given a task spec, protocol, and tool/skill registry, it composes an executable harness out of memory, planning, action, and capability modules, then that harness runs a separate execution LLM.
How do I get it running?Clone the repo, set up a Python 3.11 environment (or run pip install -r requirements.txt in an existing one), then configure execution/judge/meta model credentials in .env before launching scripts.runseedharness or scripts.runjit.
Do I need to host my own meta model, or can I use a hosted API?Either works: point METAMODEL at a hosted OpenAI-compatible API, or serve the JIT-27B checkpoint yourself with vLLM/SGLang since the pipeline only talks HTTP to it; you also need separate keys for the execution and judge models, plus optional SERPERAPIKEY/JINAAPIKEY for web tools.
Can I run it without ever calling a meta model?Yes — scripts.runseedharness executes one of the eleven hand-written HarnessFactory designs directly, with no meta model call or candidate selection involved, making it the simplest entry point.
What is HarnessFactory?It is the shared interface plus a library of eleven hand-written harness designs that JIT-Agent instantiates, samples from as references, or runs directly in fixed-harness mode.
What are the four modules every generated harness is built from?Memory, planning, action, and capability orchestration — every harness is factored into these shared interfaces so generation emits structured code rather than free-form agent programs.
Why generate a custom harness instead of using one fixed scaffold?A single general-purpose scaffold does not transfer well across different task structures, so JIT-Agent treats harness design itself as a trainable, transferable axis of agent intelligence, separate from scaling the base model.
Which benchmarks can I evaluate against?xbench, deepsearchqa, agentif, officebench, odyssey, shopping, and travel, each with its own adapter, config, and evaluator under the benchmark/ directory.
What are similar projects?Agentlas-OS, Proteus, Agent-S, SystemOneHarness, Raven, hive, Recuris, ai-agents-the-definitive-guide
Which topics is it in?planning, evaluation-framework, harness-engineering, llm-agents, agent-framework, autonomous-agents, evaluation, agent-memory, memory, agent-orchestration, agentic-ai, benchmark
Creator
bingreeky
Size
18.3 MB
Stars
556
Updated
CommitAug 27
Age
2mo
Commits · 90 Days