autoagent README

The project's README from github.com/thirdlayerinc/autoagent. As one markdown file: README.md.

thirdlayer

We're launching a product around self-configuring agents soon. Sign up here.
We're hiring engineers. If this work interests you, reach out to [email protected] with your Github link.

AutoAgent

Like autoresearch but for agent engineering. Give an AI agent a task, let it build and iterate on an agent harness autonomously overnight. It modifies the system prompt, tools, agent configuration, and orchestration, runs the benchmark, checks the score, keeps or discards the change, and repeats.

teaser

The core idea is the same: you're not touching the harness Python files like you normally would as an engineer. Instead, you program program.md, the Markdown file that provides context to the meta-agent and defines the agent-engineering loop.

How it works

The repo has a few files and directories that matter:

The metric is total score produced by the benchmark's task test suites. The meta-agent hill-climbs on this score.

Quick start

Requirements: Docker, Python 3.10+, uv, and whatever model-provider credentials your current agent.py harness requires.

# 1. Install uv (if you don't have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install dependencies
uv sync

# 3. Set up the environment variables required by your current agent/runtime
# Example:
cat > .env << 'EOF'
OPENAI_API_KEY=...
EOF

# 4. Build base image
docker build -f Dockerfile.base -t autoagent-base .

# 5. Add tasks to tasks/ (see Task format section below)

# 6. Run a single benchmark task
rm -rf jobs; mkdir -p jobs && uv run harbor run -p tasks/ --task-name "<task-name>" -l 1 -n 1 --agent-import-path agent:AutoAgent -o jobs --job-name latest > run.log 2>&1

# 7. Run all tasks in parallel (-n = concurrency, default 4)
rm -rf jobs; mkdir -p jobs && uv run harbor run -p tasks/ -n 100 --agent-import-path agent:AutoAgent -o jobs --job-name latest > run.log 2>&1

Running the meta-agent

Point your coding agent at the repo and prompt:

Read program.md and let's kick off a new experiment!

The meta-agent will read the directive, inspect the current harness, run the benchmark, diagnose failures, modify agent.py, and iterate.

Project structure

agent.py                       -- single-file harness under test
  editable harness section     -- prompt, registries, tools, routing
  fixed adapter section        -- Harbor integration + trajectory serialization
program.md                     -- meta-agent instructions + directive
Dockerfile.base                -- base image
.agent/                        -- optional agent workspace artifacts
tasks/                         -- benchmark tasks, typically added in benchmark-specific branches
jobs/                          -- Harbor job outputs
results.tsv                    -- experiment log (created by meta-agent, gitignored)
run.log                        -- latest run output

Task format

The repo ships without tasks. Add your own to tasks/ following Harbor's task format:

tasks/my-task/
  task.toml           -- config (timeouts, metadata)
  instruction.md      -- prompt sent to the agent
  tests/
    test.sh           -- entry point, writes /logs/reward.txt
    test.py           -- verification (deterministic or LLM-as-judge)
  environment/
    Dockerfile        -- task container (FROM autoagent-base)
  files/              -- reference files mounted into container

Tests write a score (0.0-1.0) to the verifier logs. The meta-agent hill-climbs on this. See the Harbor docs for full details on writing and porting tasks.

Design choices

Cleanup

Docker images and containers accumulate across runs. Clean up regularly:

# Harbor's cached task images + task cache
uv run harbor cache clean -f

# Full Docker nuke (all unused images, build cache, etc.)
docker system prune -a -f

# Lighter: just dead containers
docker container prune -f

If Docker becomes unresponsive (for example after many concurrent runs), restart Docker Desktop:

killall Docker && open -a Docker

Improving performance with skills

You can equip the agent with Agent Skills for Context Engineering and context7 skills to improve performance.

License

MIT

autoagent (thirdlayerinc/autoagent)

Searching Database…

4.6K ·GitHub
Autoagent
What is this?AutoAgent lets an AI agent build and iterate on an agent harness (agent.py) autonomously overnight: it tweaks the system prompt, tools, config, and orchestration, scores each change against a benchmark, and keeps only the improvements.
Am I editing the harness directly, or something else?You do not touch agent.py yourself. Instead you write program.md, a Markdown file that gives the meta-agent context and a directive, and the meta-agent edits the single-file harness (agent.py) on its own.
How do I actually run the meta-agent loop?Point your coding agent at the repo and prompt it; the meta-agent then reads the directive in program.md, inspects the current harness, runs the benchmark, diagnoses failures, modifies agent.py, and repeats.
What do I need installed before trying this?You need Docker, Python 3.10+, uv, and whatever model-provider API credentials your current agent.py harness requires.
Does it come with benchmarks ready to run out of the box?No, the repo ships without any evaluation tasks. You need to add your own tasks to tasks/ following Harbor's task format before the meta-agent has a score to optimize.
What is the point of Docker isolation here?The meta-agent's harness experiments run inside a container so a bad change or runaway process can't damage the host machine while it iterates overnight.
How does the meta-agent decide what to keep?Every experiment produces a numeric score (0.0-1.0) written by the benchmark's test suites to verifier logs; the meta-agent keeps the change if the score improves and discards it otherwise.
Why is agent.py a single file?The authors chose a single-file, registry-driven design for simplicity, while still keeping agent and tool registration structured enough for the harness to evolve cleanly.
Can I boost the meta-agent's own performance?Yes, you can equip it with Agent Skills for Context Engineering and context7 skills to improve how well it iterates.
What are similar projects?rrsi, Proteus, Raven, PocketFlow, awesome-harness-engineering, JIT, hive, imp
Which topics is it in?harness-framework, self-evolving, harness-engineering, llm-agents, agentic, context-engineering, harness, agent-framework, autonomous-agents, evaluation, orchestration, agents
Creator
thirdlayerinc
Size
1.2 MB
Stars
4.6K
Updated
CommitApr 03
Age
6mo
Commits · 90 Days