Global

Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.

Cayu is an open-source Python framework for building and running domain-specific long-horizon agents, with a durable runtime built in.

Mastra is the modern TypeScript framework for AI-powered applications and agents.

Arize Phoenix is Arize's open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. For managed production workflows, Arize also offers Arize AX. Phoenix provides:

An LLM agent's capability is largely set by its harness: the prompts, control flow, tools, memory and context management around a frozen model.

Build an agent harness and control it end-to-end. Open-source SDK for production AI agents in Python & TypeScript - any model, any cloud.

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready?

Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks.

Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA…

Point it at your agent's codebase and it turns production traces (or any dataset) into a fine-tuned model, benchmarked against the eval metrics you define and served via 1 unified API, with no ML infrastructure to build.

This repository ships a reproduction workflow for evaluating another harness on the published task set. It records environment differences from the original baseline run; a matched control run is needed to establish comparability.

Help build ASI-Bench: we are actively seeking high-quality scientific task contributions.

This repo contains the harness and six complete session records. The full post-broker record will be released shortly. The score by itself is not the interesting part: several systems built specifically for ARC score this high or higher.

Evals and guardrails for agents, using Jev-style decision models instead of an LLM judge.