spec-ptc README

The project's README from github.com/alexzhang13/spec-ptc. As one markdown file: README.md.

Speculative Programmatic Tool Calling

Speculative programmatic tool-calling (sPTC) is a technique for harnesses that use tools like sub-agents / sub-calls in code. While the LLM is streaming tokens to generate a REPL call, sPTC speculates and queues up tool calls in the partially-generated code that act as Futures when the actual code is executed.

Learn more in the blogpost here.

sPTC vs serial comparison

Many harness designs like Recursive Language Models (RLMs) and CodeAct rely on programmatic tool-calling (PTC), where all tools are embedded as functions inside a single code REPL tool that is generated per turn. For RLMs in particular, sub-LLM and sub-RLM calls are expensive, often blocking tools in code that take up a majority of the runtime. sPTC is the general technique of speculating tool and sub-LLM calls that will happen as the root LLM is generating the codeblock, allowing the RLM to batch and asynchronously compute these expensive calls while the full codeblock is still being generated to overlap these calls with the logic of the code REPL.

baseline   tokens──────────────────▶ exec: call₁──▶call₂──▶…──▶callₙ──▶ answer
spec-ptc   tokens──────────────────▶ exec: claim·claim·claim ──▶ answer
                 ╲ call₁ ▶▶▶ done ╱
                  ╲ call₂ ▶▶▶ done╱     (calls run inside generation time)

This repository is a simple library and demo for this technique.

Getting Started

You can either clone this repository (uses uv), or install with:

pip install spec-ptc

The Speculator object is used to track and store tools to be speculated, as well as the shadow REPL that is used to speculate. You can add tools with the spec.tool decorator and control whether you want them to be speculated or not.

The simplest example is to install tool hooks into the REPL you already have, feed tokens as they stream and feed them to the speculator, then exec as usual when finished:

from spec_ptc import Speculator

spec = Speculator()


# tools can also be async
@spec.tool(speculatable=True, pure=True)  # add as tool to be speculated
def llm_query(prompt: str) -> str:
    return sub_lm.complete(prompt)


@spec.tool()  # side effects: never speculated
def send_report(text: str) -> str:
    return mail.send(text)


ns.update(spec.hooks())  # same names, claim-or-run

code = ""
with spec.turn(repl_locals=ns) as t:  # snapshot → discarded shadow fork
    for delta in model_stream:
        code += delta
        t.feed(delta)  # closed stmts launch calls now
exec(code, ns)  # hits return immediately

For the RLM this is one line: from demo.rlm import patch_rlm; patch_rlm().

example.py is an example you can start with for looking how this is done for the RLM.

For arbitrary harness, we provide a simple daemon spec-ptc-daemon that runs the same shadow + store out of process (default socket /tmp/spec-ptc.sock) with four JSON-lines messages:

turn_begin {vars}      snapshot REPL variables into the shadow
feed {delta}           stream tokens; the daemon launches calls
resolve {tool, args}   → hit{result} | miss   (miss: run the tool yourself)
turn_end               evict leftovers, return hit/miss counts

from plugins.client import SpecClient  # ~60 lines, stdlib only — copy it

c = SpecClient()
c.turn_begin({"context": doc})
c.feed(delta)  # per streamed token
hit = c.resolve("llm_query", [prompt])  # result, or None → call it yourself
c.turn_end()

Wrappers in plugins/: Claude Code (PreToolUse), OpenCode, Pi-mono.

spec-ptc (alexzhang13/spec-ptc)

Searching Database…

224 ·GitHub
Spec-ptc
What is this?sPTC (speculative programmatic tool-calling) is a technique and small library that speculates and queues tool calls hidden inside a partially-generated REPL code block, so harnesses like RLM or CodeAct can start expensive sub-LLM/tool calls before the model even finishes writing the code.
Is this a full agent harness or something I add to an existing one?It is a lightweight library plus a demo, meant to be bolted onto programmatic tool-calling harnesses you already have (RLM, CodeAct, etc.), not a replacement harness itself.
How do I wire sPTC into my own harness?Clone the repo (it uses uv) or install the package, create a Speculator, mark functions you want speculated with the spec.tool decorator, feed streamed tokens into it as the REPL code is generated, then exec the finished code as usual.
What do I need already set up to use it?You need Python and an existing programmatic tool-calling harness to hook into - for RLMs this is a single call to demo.rlm.patchrlm, and there are ready-made wrappers for Claude Code, OpenCode, and Pi-mono.
Does this work for any agent setup, or just in-process ones?It is built around harnesses whose tool calls live inside a single per-turn code REPL (PTC-style); for setups that are not in-process there is a separate out-of-process daemon, spec-ptc-daemon, that mirrors the same shadow REPL over a Unix socket.
Who would actually use this?Developers building or tuning agent harnesses that generate code-REPL tool calls per turn, like Recursive Language Models or CodeAct-style agents, where sub-LLM calls dominate latency.
What is an RLM in this context?RLM stands for Recursive Language Model, a harness where sub-LLM and sub-RLM calls are embedded as functions inside one code REPL tool generated per turn, and those calls often take up most of the runtime.
Which harnesses already have ready-made wrappers?The plugins/ directory ships wrappers for Claude Code (hooked via PreToolUse), OpenCode, and Pi-mono, on top of the one-line RLM patch.
How does speculating tool calls actually save time?As the root LLM streams tokens for a REPL call, sPTC parses the partially generated code, launches recognized tool and sub-LLM calls early as Futures, and overlaps that work with the rest of the code generation and eventual execution instead of waiting for the full block to finish first.
What are similar projects?pstack-claude, moyai, fable-orchestrator, Gnos, pstack, orca, agentbox, harmonist
Which topics is it in?asynchronous, daemon, concurrency, async, harness, opencode, streaming, agentic-ai, multi-agent, coding-agent, performance, agent
Creator
alexzhang13
Size
114.4 MB
Stars
224
Updated
CommitAug 24Releasev0.1.1Aug 24
Age
1mo
Commits · 90 Days