Sentry README

项目的 README,来自 github.com/nuglifeleoji/Sentry。合并为一个 markdown 文件:README.md。

Sentry Logo

An external runtime failure management layer for LLM agents.

Detect execution failures, guide recovery, and learn reusable lessons from verified recoveries.
Sentry runs alongside your agent loop without replacing the agent, environment, or evaluator.

Quick Start  |  Paper (arXiv)  |  License

Python 3.10+ MIT License arXiv

What is Sentry?

Sentry is an external runtime failure management layer for LLM agents. It helps agents fix invalid actions and recover when they get stuck in loops, drift from the task, or make unsupported assumptions. Unlike typical runtime monitors with fixed advice, Sentry learns and evolves from recoveries that work. By detecting failures, guiding recovery, and checking that the repair worked, Sentry helps agents become more reliable across tasks without cluttering their context.

Sentry's demo

Sentry demo

Key Results

Quick Start

Installation

Sentry requires Python 3.10 or later.

git clone https://github.com/nuglifeleoji/Sentry.git
cd Sentry
python -m venv .venv
source .venv/bin/activate
python -m pip install -e .
sentry-validate configs/paper.yaml

Integrate Sentry into Your Agent Loop

Configure a model provider, then submit each completed agent-environment cycle to Sentry:

from Sentry import SentryRunner, agent_step_from_parts

runner = SentryRunner.from_yaml("configs/providers/openai_compatible.yaml")
runner.start_task(
    objective=task_objective,
    action_schema=environment_action_schema,
)

try:
    step = agent_step_from_parts(
        step_id=0,
        reasoning=reasoning,
        raw_action=raw_action,
        tool_name=tool_name,
        tool_args=tool_args,
        observation=observation,
        parsed_ok=True,
        schema_valid=True,
        accepted_by_environment=True,
    )

    repair = runner.step(step, terminal=False)
    if repair is not None:
        agent_messages.append(repair.prompt_text)
finally:
    runner.finalize()

Your application remains responsible for generating, validating, and executing actions. See the integration guide for the full lifecycle and validity rules.

Run the AppWorld Benchmark

The reference benchmark in benchmarks/appworld/ uses the official AppWorld environment and evaluator. Run it from the repository checkout:

python3.11 -m venv .venv-appworld
source .venv-appworld/bin/activate
python -m pip install -e '.[appworld]'
appworld install
export APPWORLD_ROOT="$PWD/outputs/appworld-root"
appworld download data --root "$APPWORLD_ROOT"
export OPENROUTER_API_KEY="your-api-key"
python -m benchmarks.appworld.run --config configs/providers/openrouter.yaml

See the AppWorld guide for environment setup, credentials, and run limits.

How It Works

Sentry framework: failure detection, repair, verification, and playbook learning

  1. Failure Detection: Sentry monitors recent agent steps for invalid actions and behavioral failures.
  2. Hard Repair: Sentry asks the agent to retry invalid actions in the required format.
  3. Soft Repair: Progress and reasoning failures receive targeted guidance from relevant playbook lessons.
  4. Recovery Verification: Sentry checks whether the agent recovers over the following steps.
  5. Online Playbook Learning: Verified soft recoveries become reusable lessons for similar failures across tasks.

Further Findings

Release

This repository currently contains the project overview and paper. We are finalizing our code and documentation and plan to release the full version in 2-3 weeks.

Sentry (nuglifeleoji/Sentry)

正在搜索数据库…

1.6K ·GitHub
这是什么?Sentry 是面向 LLM 智能体的外部运行时故障管理层。它在智能体循环旁运行,检测执行故障,引导恢复,并从经过验证的恢复过程中总结可复用的经验教训。
这是独立的智能体,还是可以接入我自己智能体的一层?它是一层,你需要把它集成到现有的智能体循环中。Sentry 不会替代智能体、环境或评估器,它只负责监控并在出现故障时进行干预。
我该如何将 Sentry 集成到我的智能体中?先配置模型提供方,然后把每一轮已完成的智能体与环境交互提交给 Sentry,以便它监控故障。你的应用仍然负责生成、校验和执行动作。
我需要什么版本的 Python?Sentry 需要 Python 3.10 或更高版本。
完整代码现在已经可以获取了吗?还没有。该仓库目前只有项目概览和论文;作者表示完整代码和文档仍在整理定稿中。
谁会用 Sentry?在网页导航、应用自动化或软件工程基准等任务上构建或评估 LLM 智能体的开发者,他们希望直接使用即插即用的故障恢复层,而不必从零编写。
Sentry 在哪些基准上做过测试?它在 WebShop、AppWorld、SWE-bench Lite 和 Mind2Web Replay 上进行了评估,相比表现最好的运行时干预基线方法,平均提升 37%。
Sentry 会给智能体运行增加多少开销?Sentry 的 Token 用量约为基础智能体的 1.54 倍,而其他运行时方法最高可达 2.44 倍;同时,在 SWE-bench Lite 上每个任务节省 31.7 秒,在 AppWorld 上节省 45.6 秒。
Sentry 实际修复故障的效果如何?在已报告的实验中,它在检测到的 939 个故障中实现了 81.7% 的恢复率。
Sentry 如何决定帮助卡住的智能体?它分五步进行:检测故障;针对无效动作请求强制重试;针对推理类故障,依据经验手册中的经验教训给出软性指导;验证恢复是否成功;并将经过验证的修复方法保存为可复用的经验教训,供未来任务使用。
大小
16.2 MB
星标
1.6K
更新
提交Oct 11
时长
1个月
提交 · 90 天