Global

is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

DeepEval is a simple-to-use, open-source LLM evaluation framework, for evaluating large-language model systems. It is similar to Pytest but specialized for unit testing LLM apps. , which use LLM-as-a-judge and other NLP models that run locally on your machine.

0 release of lm-evaluation-harness is available !

The series is running. 5 days after launch. It runs once a day for 30 days: days 1–10 are the baseline, then there are two 10-day windows, so the first possible call is around 2026-10-24.

Every notebook can be opened and run on Google Colab directly from the links below. Just click the Open In Colab badge next to the notebook you want to run.

This repository ships a reproduction workflow for evaluating another harness on the published task set. It records environment differences from the original baseline run; a matched control run is needed to establish comparability.

JIT-Agent is a compact meta-agent that writes your agent harness on the fly.

ParseBench is a benchmark for evaluating how well document parsing tools convert PDFs into structured output that AI agents can reliably act on.

Evals and guardrails for agents, using Jev-style decision models instead of an LLM judge.