1.5K ·GitHub
What is this?
A collection of agent skills that guide AI coding agents to build product-specific AI evals — not foundation-model benchmarks. It encodes mistakes the authors saw across 50+ companies and thousands of students in their AI Evals course.
A collection of agent skills that guide AI coding agents to build product-specific AI evals — not foundation-model benchmarks. It encodes mistakes the authors saw across 50+ companies and thousands of students in their AI Evals course.So is this a tool I run myself, or instructions my coding agent follows?Not an app — it's a set of skills your AI coding agent loads and follows. 'evals-start' is the entry point: it looks at your situation and routes you to the right skill.
Not an app — it's a set of skills your AI coding agent loads and follows. 'evals-start' is the entry point: it looks at your situation and routes you to the right skill.How do I install the skills?Run 'npx skills' to install them into your coding agent. You can also install just one skill, and check for updates with separate commands.
Run 'npx skills' to install them into your coding agent. You can also install just one skill, and check for updates with separate commands.Do I need to install anything besides a coding agent?No — just your existing AI coding agent. The only runtime touchpoint is error-discovery's generated review app: a single-file HTML page served by a Python stdlib server with zero dependencies.
No — just your existing AI coding agent. The only runtime touchpoint is error-discovery's generated review app: a single-file HTML page served by a Python stdlib server with zero dependencies.What parts of eval work do these skills not cover?Only the parts that generalize across projects. Production monitoring, CI/CD regression suites, and cost optimization are explicitly out of scope — the team's AI Evals course covers those.
Only the parts that generalize across projects. Production monitoring, CI/CD regression suites, and cost optimization are explicitly out of scope — the team's AI Evals course covers those.I'm building an AI product but new to evals — where do I start?These skills are aimed at developers building evals for their own product, and they encode mistakes from helping 50+ companies and teaching thousands of students. evals-start will send you to eval-audit if you already have a pipeline, or error-discovery if you have traces you haven't analyzed yet.
These skills are aimed at developers building evals for their own product, and they encode mistakes from helping 50+ companies and teaching thousands of students. evals-start will send you to eval-audit if you already have a pipeline, or error-discovery if you have traces you haven't analyzed yet.What skills are in the collection?Eight: evals-start (entry point that routes you), eval-audit (audits an existing eval pipeline and flags problems by severity), error-discovery (builds a review app and sorts traces into failure modes), generate-synthetic-data (dimension-based tuple generation), write-judge-prompt (designs LLM-as-Judge evaluators), validate-evaluator (calibrates judges against human labels with TPR/TNR and bias correction), evaluate-rag (retrieval and generation quality), and build-review-interface (custom annotation UIs).
Eight: evals-start (entry point that routes you), eval-audit (audits an existing eval pipeline and flags problems by severity), error-discovery (builds a review app and sorts traces into failure modes), generate-synthetic-data (dimension-based tuple generation), write-judge-prompt (designs LLM-as-Judge evaluators), validate-evaluator (calibrates judges against human labels with TPR/TNR and bias correction), evaluate-rag (retrieval and generation quality), and build-review-interface (custom annotation UIs).Why is error-discovery called the most important skill?Because you should only write evals after doing error analysis. You point it at a JSONL/CSV/JSON file of LLM outputs or traces: it builds a review app, clusters the data and picks a diverse sample, then runs an interactive loop — your free-text notes get sorted into failure modes while it picks new samples to fill coverage gaps.
Because you should only write evals after doing error analysis. You point it at a JSONL/CSV/JSON file of LLM outputs or traces: it builds a review app, clusters the data and picks a diverse sample, then runs an interactive loop — your free-text notes get sorted into failure modes while it picks new samples to fill coverage gaps.Why skills instead of just reading an evals guide?There are many easily avoidable footguns in evals. The skills encode the common mistakes the AI Evals course team saw helping 50+ companies, so your agent sidesteps them while building evals for your specific product.
There are many easily avoidable footguns in evals. The skills encode the common mistakes the AI Evals course team saw helping 50+ companies, so your agent sidesteps them while building evals for your specific product.Can I write my own eval skills?Yes, and the authors encourage it: these only cover what generalizes across projects, while skills grounded in your own data and domain will outperform them. Matt Pocock's writing-for-ai-agents is a good starting resource.
Yes, and the authors encourage it: these only cover what generalizes across projects, while skills grounded in your own data and domain will outperform them. Matt Pocock's writing-for-ai-agents is a good starting resource.What are similar projects?skills, SwiftUI-Agent-Skill, ruflo, jev-align, claude-plugins-official, superpowers-zh, Coding-Agent-Skills, compound-engineering-plugin
skills, SwiftUI-Agent-Skill, ruflo, jev-align, claude-plugins-official, superpowers-zh, Coding-Agent-Skills, compound-engineering-pluginWhich topics is it in?annotation-tool, ai-skills, claude-plugin, annotation, llm-evaluation, human-in-the-loop, best-practices, evaluation, prompt-engineering, rag, skills, testing
annotation-tool, ai-skills, claude-plugin, annotation, llm-evaluation, human-in-the-loop, best-practices, evaluation, prompt-engineering, rag, skills, testing