switch README

项目的 README,来自 github.com/osolmaz/switch。合并为一个 markdown 文件:README.md。

switch: a framework for building decision-model harnesses for any kind of task

switch

switch is a harness framework for decision models like Jev. It gives a decision model an agent loop for finite choices. Your code turns the state of a task into typed questions, such as which next step to take from a short list, and the model picks the answers.

https://github.com/user-attachments/assets/a5d834cb-f55b-4f2c-96cf-61ab1ba1e432

A simple explainer for switch, also on YouTube

A decision model answers typed questions with probabilities and writes no text, so it cannot generate the tool calls that a coding agent harness such as Pi relies on. switch generates the tool calls for it in code, and the model only picks one from that finite set. TypeSafe calls these models System One models, and its Jev was the first; llama.cpp serves open ones through the same API.

switch puts what many people already write by hand, a decision-model API called over several steps, behind one abstract interface. You implement a domain, which defines the state of a task and the functions that turn a state into the model's questions and its answers into the next state. Chess is a domain, and so is a rubric judge for evals.

Uses

A main use is a judge in an eval. At the end of a task run that involves many documents, the usual judge extracts the text of the PDF, Word and Markdown files and puts all of it into the context. When the visual formatting of the output is a criterion, every page has to go in as an image too. With 200 pages and a question that needs only a few of them, most of that context is wasted. With switch, the decision model navigates the documents and takes in only what it needs. The rubric judge works this way. It reads parts of files, looks at page images and searches for terms, one step at a time, until it can grade each rubric item.

switch works with any decision model that pi-ai lists as a classifier, Jev among them, as docs/DECISION_MODEL_API.md explains. Jev itself is weak at chess, and as a judge it is not yet good enough to replace the strongest chat models. Once a better model has generated enough data, an open decision model can be fine-tuned on one domain. Trained in a harness like switch's, it should do that job both better and more cheaply. A narrow domain leaves a lot to gain, since an agent built for one specific job could cost a hundredth of what a general model costs.

Concepts

How a run works

  1. The domain makes the case's initial state.
  2. The question function gives the state's questions. If there are none, the run ends.
  3. The model reads the view of the state and answers every question.
  4. The transition function makes the next state from the answers, and the run goes back to step 2.

A run also ends when a request fails, or at a step limit that a domain may set as a safety net. docs/MATH.md writes the loop down in symbols.

Install

switch needs Node.js 22.18 or newer. Add it to a project from npm:

npm install @osolmaz/switch
Import What it gives
@osolmaz/switch The loop, the domain types, the pick helper, traces and replay
@osolmaz/switch/models piAiDecider, which asks a classifier model that pi-ai lists
@osolmaz/switch/cli commandLine, which gives a harness the run and replay commands
@osolmaz/switch/testing A fake decider for tests that make no model calls

The examples and the viewer live in this repository and are not on npm. To run them, clone it and install its workspace with pnpm:

git clone https://github.com/osolmaz/switch.git
cd switch
pnpm install

Examples

Write a domain

A domain gives initial, questions and transition, and optionally view. This one routes a support ticket. The model reads logs one at a time until it picks stop, and then a last step asks which team gets the ticket.

import {
  type Choice,
  type ChoiceQuestion,
  type DomainShared,
  type JsonObject,
  picking,
  type StateMachine,
} from "@osolmaz/switch";

export interface Ticket {
  readonly text: string;
  readonly logs: Readonly<Record<string, string>>;
}

/** Where a ticket is: the logs read so far, with what each one said, whether reading is over, and the team. */
interface TriageState extends JsonObject {
  readonly read: readonly { readonly log: string; readonly text: string }[];
  readonly done: boolean;
  readonly team: string | null;
}

/** A log to read, or null to stop reading; the transition gets the choice back. */
interface ReadLog extends Choice {
  readonly log: string | null;
}

export const TRIAGE: DomainShared = { name: "triage", version: "0.1.0", limits: { steps: 20, windowShare: 0.9 } };

const TEAM: ChoiceQuestion = {
  type: "choice",
  instructions: "Which team should handle the ticket?",
  criteria: { billing: "Billing", infra: "Infrastructure", app: "Application" },
};

export function triageMachine(ticket: Ticket): StateMachine<TriageState> {
  const pick = picking<TriageState, ReadLog>({
    instructions: "Which log should be read next to route this ticket?",
    limit: 20,
    choices: (state) => [
      ...Object.keys(ticket.logs)
        .filter((log) => !state.read.some((entry) => entry.log === log))
        .map((log) => ({ type: "read-log", key: log, text: `Read ${log}`, log })),
      { type: "stop", key: "stop", text: "Stop reading: the evidence is enough", log: null, priority: -1 },
    ],
    transition: (state, { log }) =>
      Promise.resolve(
        log === null
          ? { ...state, done: true }
          : { ...state, read: [...state.read, { log, text: ticket.logs[log] ?? "" }] },
      ),
  });
  return {
    initial: () => Promise.resolve({ read: [], done: false, team: null }),
    questions: (state) => {
      if (state.team !== null) return {};
      return state.done ? { team: TEAM } : pick.questions(state);
    },
    transition: (state, answers) => {
      if (!state.done) return pick.transition(state, answers);
      const team = answers["team"];
      return Promise.resolve({ ...state, team: team?.type === "choice" ? team.choice : "unknown" });
    },
    view: (state) => Promise.resolve({ decisionState: { ticket: ticket.text, read: state.read } }),
  };
}

Run it with any classifier model that pi-ai lists. pi-ai finds the key in the environment, such as TYPESAFE_API_KEY:

import { builtinModels } from "@earendil-works/pi-ai/providers/all";
import { run } from "@osolmaz/switch";
import { piAiDecider } from "@osolmaz/switch/models";

const decider = piAiDecider("typesafe/jev-latest", { registry: builtinModels() });
const result = await run(TRIAGE, triageMachine(ticket), decider);
// result.state is the last state, result.trace the trace, and result.ended "done", "step limit" or "failed".

docs/DOMAINS.md has the rules of picks, stop choices, views, limits and choice types.

Write a harness

A harness turns a domain into a program that runs case folders. Its setup returns a runner:

import { basename } from "node:path";
import { type Runner, runnerOf } from "@osolmaz/switch";
import { TRIAGE, triageMachine } from "./index.ts";
import { loadTicket } from "./load.ts";

export function triageSetup(): Promise<Runner> {
  return Promise.resolve(
    runnerOf({
      ...TRIAGE,
      caseId: (folder) => Promise.resolve(basename(folder)),
      load: async (folder) => triageMachine(await loadTicket(folder)),
      outcome: ({ state }) => ({ team: state.team }),
    }),
  );
}

commandLine makes it a program with run and replay. It is a Commander program, so the harness can add commands of its own:

import { commandLine } from "@osolmaz/switch/cli";
import { triageSetup } from "./setup.ts";

await commandLine({ triage: { setup: triageSetup } })
  .name("triage")
  .parseAsync();

docs/DOMAINS.md covers definitions, a domain's own flags and API keys.

Run, watch and replay

examples/chess/src/main.ts run examples/chess/cases/white-seed-1 --out runs/chess
examples/chess/src/main.ts view runs/chess
examples/chess/src/main.ts replay examples/chess/cases/white-seed-1 runs/chess/cases/white-seed-1/trace.jsonl --fork 5 --out runs/fork

docs/DOMAINS.md says when a replay fails and what a run folder holds.

License

MIT

switch (osolmaz/switch)

正在搜索数据库…

446 ·GitHub
这是什么?switch 是一个 TypeScript 框架,用于围绕决策模型(如 Jev)构建 harness。它为模型提供一个智能体循环,在一组简短、有限的带类型下一步选项中做选择,直到模型能给出最终答案。
switch 是框架、普通库还是 CLI 工具?三者兼而有之。核心 npm 包(@osolmaz/switch)以库的形式提供智能体循环、领域类型和 pick 辅助函数;@osolmaz/switch/cli 中的 commandLine 则能把任意领域变成可运行的程序,提供 run 和 replay 命令。
怎样把它加入我的项目?从 npm 安装 @osolmaz/switch(按需安装 @osolmaz/switch/models、/cli 或 /testing),然后编写一个 'domain',为你的任务定义 initial、questions 和 transition 函数。
使用前需要先安装什么?Node.js 22.18 或更高版本。内置示例和查看器未发布到 npm,所以要运行它们,需要克隆仓库并安装 pnpm 工作区。
决策模型在 switch 里做不了什么?决策模型只会以概率回答带类型的问题,不输出自由文本,因此与基于文本的编程智能体 harness 不同,它本身无法生成任意工具调用——必须由 switch 的代码为它生成有限的选项集合。Jev 是第一个这样的模型,它下棋能力仍然较弱,作为评判者也还不足以取代最强的聊天模型。
为什么要用决策模型做评判,而不是把所有文档都塞进 LLM 的上下文?普通评判方式会把每个 PDF、Word 或 Markdown 文件的每一页(视觉标准还要加上图片)都塞进上下文,即使只有少数几页有用;而用 switch 时,决策模型会逐步浏览文档,只读取为每条评分标准打分所需的内容。
什么是 Jev?Jev 是 TypeSafe 推出的决策模型,也是其所称“ System One 模型”中的第一个。这类模型以概率回答带类型的 bool、choice 或 score 问题,而不是输出文本。llama.cpp 可以通过 switch 使用的同一套 API 提供开源决策模型服务。
仓库里附带了哪些示例领域?包括一个评分标准评判器,可根据评分标准为已保存的任务答案打分;一个国际象棋领域,可与随机走法、Stockfish 或自己对弈;以及一个 WikiRace 领域,通过跟随链接在维基百科条目之间竞速。
运行结束后,我能检查或回放吗?每次运行都会写入一份追踪记录,包含每个状态、问题和答案;查看器可在浏览器中逐步回放;replay 命令则可以根据追踪记录重新运行某个用例而无需调用模型,还可以通过 --fork 在中途分叉。
有哪些类似的项目?jevals, eval, promptfoo, tardigrade, nimble, Valen, decider, livenerf
作者
osolmaz
大小
1.4 MB
星标
446
时长
24天
提交 · 90 天