spec-ptc README

The project's README from github.com/alexzhang13/spec-ptc. As one markdown file: README.md.

Speculative Programmatic Tool Calling

Speculative programmatic tool-calling (sPTC) is a technique for harnesses that use tools like sub-agents / sub-calls in code. While the LLM is streaming tokens to generate a REPL call, sPTC speculates and queues up tool calls in the partially-generated code that act as Futures when the actual code is executed.

Learn more in the blogpost here.

sPTC vs serial comparison

Many harness designs like Recursive Language Models (RLMs) and CodeAct rely on programmatic tool-calling (PTC), where all tools are embedded as functions inside a single code REPL tool that is generated per turn. For RLMs in particular, sub-LLM and sub-RLM calls are expensive, often blocking tools in code that take up a majority of the runtime. sPTC is the general technique of speculating tool and sub-LLM calls that will happen as the root LLM is generating the codeblock, allowing the RLM to batch and asynchronously compute these expensive calls while the full codeblock is still being generated to overlap these calls with the logic of the code REPL.

baseline   tokens──────────────────▶ exec: call₁──▶call₂──▶…──▶callₙ──▶ answer
spec-ptc   tokens──────────────────▶ exec: claim·claim·claim ──▶ answer
                 ╲ call₁ ▶▶▶ done ╱
                  ╲ call₂ ▶▶▶ done╱     (calls run inside generation time)

This repository is a simple library and demo for this technique.

Getting Started

You can either clone this repository (uses uv), or install with:

pip install spec-ptc

The Speculator object is used to track and store tools to be speculated, as well as the shadow REPL that is used to speculate. You can add tools with the spec.tool decorator and control whether you want them to be speculated or not.

The simplest example is to install tool hooks into the REPL you already have, feed tokens as they stream and feed them to the speculator, then exec as usual when finished:

from spec_ptc import Speculator

spec = Speculator()


# tools can also be async
@spec.tool(speculatable=True, pure=True)  # add as tool to be speculated
def llm_query(prompt: str) -> str:
    return sub_lm.complete(prompt)


@spec.tool()  # side effects: never speculated
def send_report(text: str) -> str:
    return mail.send(text)


ns.update(spec.hooks())  # same names, claim-or-run

code = ""
with spec.turn(repl_locals=ns) as t:  # snapshot → discarded shadow fork
    for delta in model_stream:
        code += delta
        t.feed(delta)  # closed stmts launch calls now
exec(code, ns)  # hits return immediately

For the RLM this is one line: from demo.rlm import patch_rlm; patch_rlm().

example.py is an example you can start with for looking how this is done for the RLM.

For arbitrary harness, we provide a simple daemon spec-ptc-daemon that runs the same shadow + store out of process (default socket /tmp/spec-ptc.sock) with four JSON-lines messages:

turn_begin {vars}      snapshot REPL variables into the shadow
feed {delta}           stream tokens; the daemon launches calls
resolve {tool, args}   → hit{result} | miss   (miss: run the tool yourself)
turn_end               evict leftovers, return hit/miss counts

from plugins.client import SpecClient  # ~60 lines, stdlib only — copy it

c = SpecClient()
c.turn_begin({"context": doc})
c.feed(delta)  # per streamed token
hit = c.resolve("llm_query", [prompt])  # result, or None → call it yourself
c.turn_end()

Wrappers in plugins/: Claude Code (PreToolUse), OpenCode, Pi-mono.

spec-ptc (alexzhang13/spec-ptc)

Searching Database…

223 ·GitHub
Spec-ptc
这是什么？sPTC（投机式程序化工具调用）是一种技术和一个小型库，它会推测并排队隐藏在部分生成的 REPL 代码块中的工具调用，这样 RLM 或 CodeAct 之类的框架就能在模型还没写完代码时就启动昂贵的子 LLM/工具调用。
这是一个完整的 agent 框架，还是需要加到现有框架上的东西？它是一个轻量级库加一个演示，目的是接在你已有的程序化工具调用框架上（RLM、CodeAct 等），本身并不是一个替代性的框架。
怎么把 sPTC 接入我自己的框架？克隆仓库（它使用 uv）或安装该软件包，创建一个 Speculator，用 spec.tool 装饰器标记你想被投机的函数，在 REPL 代码生成过程中把流式 token 喂给它，然后照常执行完成的代码。
使用它需要预先准备好什么？你需要 Python 和一个可以挂接的现成程序化工具调用框架——对 RLM 来说只需调用一次 demo.rlm.patchrlm，此外还有针对 Claude Code、OpenCode 和 Pi-mono 的现成封装。
它适用于任何 agent 设置，还是仅限于进程内的设置？它围绕工具调用位于每轮单个代码 REPL 内的框架（PTC 风格）而构建；对于非进程内的设置，还有一个单独的进程外守护进程 spec-ptc-daemon，通过 Unix socket 镜像同一个影子 REPL。
谁会真正用到这个？正在构建或调优每轮生成代码 REPL 工具调用的 agent 框架的开发者，比如 Recursive Language Models 或 CodeAct 风格的 agent，在这些场景中子 LLM 调用占据了大部分延迟。
在这个语境下 RLM 是什么？RLM 是 Recursive Language Model（递归语言模型）的缩写，这种框架把子 LLM 和子 RLM 调用作为函数嵌入到每轮生成的一个代码 REPL 工具中，而这些调用往往占据大部分运行时间。
哪些框架已经有现成的封装？plugins/ 目录为 Claude Code（通过 PreToolUse 挂钩）、OpenCode 和 Pi-mono 提供了封装，还有一行代码的 RLM 补丁。
投机工具调用究竟是怎么省时间的？当根 LLM 为一次 REPL 调用流式生成 token 时，sPTC 会解析部分生成的代码，把识别出的工具调用和子 LLM 调用提前作为 Future 启动，让这些工作与其余代码生成及最终执行重叠进行，而不是先等整个代码块写完。
What are similar projects?pstack-claude, moyai, fable-orchestrator, Gnos, pstack, orca, agentbox, harmonist
Which topics is it in?asynchronous, daemon, concurrency, async, harness, opencode, streaming, agentic-ai, multi-agent, coding-agent, performance, agent
Creator
alexzhang13
Size
114.4 MB
Stars
223
Updated
CommitAug 24Releasev0.1.1Aug 24
Age
1mo
Commits · 90 Days