Evaluation projects on GitHub

1. 1
   #1 · datawhalechina/hello-agents

   hello-agents

   Hello-Agents is Datawhale's free, open-source tutorial on building AI agent systems from scratch — theory and hands-on practice in equal measure, taking you from LLM user to agent builder.

   83.6K
2. 2
   #2 · jingyaogong/minimind

   minimind

   MiniMind is an open-source project that lets you train a tiny large language model (around 64M parameters) completely from scratch, in about 2 hours on a single RTX 3090 for roughly the cost of 3 RMB in GPU rental.

   63.9K
3. 3
   #3 · karpathy/nanochat

   nanochat

   A minimal, hackable harness for training your own LLM end to end (tokenization, pretraining, finetuning, evaluation, inference) on a single GPU node, ending in a chatbot you talk to over a CLI.

   58.7K

4. 4

   ai-agent-book

   54.8Kbojieli/ai-agent-book · Python

   An open-source book by Li Bojie taking AI agents from design principles to engineering practice, built on the formula Agent = LLM + context + tools; text, figures and labs all live in this repo.

   • experiments
   • epub
   • computer-use
   • llm-agents
5. 5

   LightRAG

   40.2KHKUDS/LightRAG · Python

   LightRAG is an open-source, graph-based RAG framework (EMNLP 2025) by HKUDS. It indexes your documents into a knowledge graph plus vector embeddings so you can query them — a lighter, faster alternative to Microsoft GraphRAG.

   • question-answering
   • graphrag
   • genai
   • graph-database
6. 6

   langfuse

   36Klangfuse/langfuse · TypeScript

   Langfuse is an open source LLM engineering platform for tracing, evaluating, and debugging LLM and agent applications, with prompt management and a playground built in.

   • autogen
   • llama-index
   • playground
   • llm-observability
7. 7

   heretic

   35.7Kp-e-w/heretic · Python

   Heretic is a command-line tool that automatically removes censorship (safety alignment) from transformer-based language models using directional ablation combined with an Optuna-powered parameter optimizer, without any training.

   • uncensored
   • optimization
   • quantization
   • transformer
8. 8

   WeKnora

   34.6KTencent/WeKnora · Go

   WeKnora is an open-source, self-hosted knowledge platform that turns raw documents into a queryable RAG knowledge base, an autonomous ReAct agent, and a self-maintaining Wiki with an interactive knowledge graph.

   • reranking
   • notion
   • question-answering
   • multi-tenant
9. 9

   agents-course

   33.7Khuggingface/agents-course · MDX

   The official Hugging Face Agents Course: a free, hands-on curriculum that teaches how to build AI agents, progressing from the basics to agent frameworks and a certified final project.

   • llamaindex
   • certification
   • mdx
   • langchain
10. 10

    RAG_Techniques

    29.9KNirDiamant/RAG_Techniques · Jupyter Notebook

    A community-driven hub of advanced techniques for Retrieval-Augmented Generation (RAG) — grounding LLM answers in retrieved documents, distilled into runnable notebook tutorials.

    • llama-index
    • agentic-rag
    • tutorials
    • retrieval-augmented-generation
11. 11

    awesome-generative-ai-guide

    29.9Kaishwaryanr/awesome-generative-ai-guide · HTML

    A one-stop GitHub hub for generative AI learning: research updates, 90+ free courses, code notebooks, and interview prep materials, all organized around one question — what do you want to do with AI?

    • jupyter-notebook
    • llms
    • papers
    • interview-questions
12. 12

    iFixAi

    29.2Kifixai-ai/iFixAi · Python

    iFixAi is an independent auditing tool for AI agents: it runs a diagnostic suite against your agent and answers the core question — is the agent doing the job it is supposed to do — with an A-F grade in under 120 seconds.

    • hallucination-detection
    • risk-management
    • agent-evaluation
    • ai-governance
13. 13

    mlflow

    28.4Kmlflow/mlflow · Python

    MLflow is an open source AI engineering platform for agents, LLM apps, and classic ML models: one place to trace, evaluate, monitor, and optimize production AI while controlling costs. It is widely adopted, with over 60 million monthly downloads.

    • ai-governance
    • guardrails
    • cost-tracking
    • ml
14. 14

    open-r1

    26.5Khuggingface/open-r1 · Python

    Open R1 is a fully open effort to reproduce the training pipeline behind DeepSeek-R1, providing scripts to run SFT and GRPO training, generate synthetic reasoning data, and evaluate the resulting reasoning models.

    • sft
    • rlhf
    • model-training
    • distributed-training
15. 15

    promptfoo

    26.1Kpromptfoo/promptfoo · TypeScript

    promptfoo is a CLI and library for evaluating and red-teaming LLM apps — you test prompts, agents, and RAGs and run security scans on them, instead of relying on trial-and-error.

    • vulnerability-scanners
    • evaluation-framework
    • cicd
    • red-teaming
16. 16

    Mask_RCNN

    25.6Kmatterport/Mask_RCNN · Python

    An implementation of the Mask R-CNN model on Python 3, Keras and TensorFlow that generates bounding boxes and a segmentation mask for each object instance in an image, built on a Feature Pyramid Network and a ResNet101 backbone.

    • tensorboard
    • mask-rcnn
    • instance-segmentation
    • transfer-learning
17. 17

    llm-cookbook

    24.8Kdatawhalechina/llm-cookbook · Jupyter Notebook

    A developer-oriented LLM starter handbook: a Chinese adaptation of Andrew Ng's LLM course series, translated and re-run for Chinese developers, covering prompt engineering through RAG development and model fine-tuning.

    • chatgpt-api
    • cookbook
    • wandb
    • chromadb
18. 18

    courses

    22.9Kanthropics/courses · Jupyter Notebook

    Anthropic's official set of educational courses on working with Claude: five hands-on courses covering API essentials, prompt engineering, real world prompting, prompt evaluations, and tool use.

    • claude-api
    • anthropic-api
    • function-calling
    • tool-calling
19. 19

    opik

    22.7Kcomet-ml/opik · Python

    Opik is an open-source platform for debugging, evaluating, and monitoring LLM applications, RAG systems, and agentic workflows, with tracing, automated evaluation, and production dashboards.

    • llama-index
    • playground
    • llm-observability
    • guardrails
20. 20

    recommenders

    22Krecommenders-team/recommenders · Python

    An open-source Python library and notebook collection with best practices for building, evaluating and deploying recommendation systems — a Linux Foundation of AI & Data project that started at Microsoft.

    • ranking
    • recommendation-system
    • spark
    • mlops
21. 21

    adk-python

    21.9Kgoogle/adk-python · Python

    Agent Development Kit (ADK) is Google's open-source, code-first Python framework for building, evaluating, and deploying sophisticated AI agents — from simple tasks to complex multi-agent systems.

    • vertex-ai
    • google-cloud
    • genai
    • tool-calling
22. 22

    agents-towards-production

    21.7KNirDiamant/agents-towards-production · Jupyter Notebook

    It's an open-source collection of 28+ code-first, hands-on tutorials for taking GenAI agents from a prototype to a production-ready product, covering orchestration, memory, observability, deployment, and security.

    • production
    • genai
    • langgraph
    • tutorials
23. 23

    pydantic-ai

    20.7Kpydantic/pydantic-ai · Python

    Pydantic AI is the Python AI SDK for building agents: a typed, extensible agent loop where every model is a string swap away. The same agent runs behind a web frontend, in the terminal, on a voice call, or as a plain object you call run() on.

    • pydantic
    • genai
    • harness-engineering
    • validation
24. 24

    DeepResearch

    20.1KAlibaba-NLP/DeepResearch · Python

    Tongyi DeepResearch — an open-source agentic LLM from Alibaba's Tongyi Lab, built for long-horizon, deep information-seeking on the web. This repo ships the model weights plus Python inference and benchmark evaluation scripts.

    • long-context
    • alibaba
    • deep-research
    • web-search
25. 25

    evals

    19.6Kopenai/evals · Python

    Evals is OpenAI's framework for evaluating LLMs and LLM-based systems, bundled with an open-source registry of benchmarks you can run or extend.

    • evals
    • llm-evaluation
    • registry
    • yaml
26. 26

    awesome-nlp

    19.1Kkeon/awesome-nlp

    A curated list of resources dedicated to Natural Language Processing — research hubs, tutorials, courses, books, libraries, datasets, and task methods, collected as links with one-line descriptions.

    • named-entity-recognition
    • question-answering
    • sentiment-analysis
    • datasets
27. 27

    SkillOpt

    18.5Kmicrosoft/SkillOpt · Python

    A text-space optimizer that trains reusable natural-language skills for frozen LLM agents without touching model weights. The trained skill ships as a compact markdown file (typically 300–2,000 tokens) that adds zero inference-time model calls at deployment.

    • prompt-optimization
    • optimizer
    • self-evolving-agents
    • minimax
28. 28

    Agent-Skills-for-Context-Engineering

    18.2Kmuratcankoylan/Agent-Skills-for-Context-Engineering · Python

    A curated collection of Agent Skills that teach context engineering, multi-agent architecture, and production agent system design, so an AI agent can load the right skill when it needs to manage context, memory, evaluation, or harness design.

    • long-horizon-agents
    • kv-cache
    • agentic-workflow
    • multi-agent-systems
29. 29

    aie-book

    17.9Kchiphuyen/aie-book · Jupyter Notebook

    The companion repo for Chip Huyen's book 'AI Engineering' (2025) and a collection of resources for AI engineers: chapter summaries, study notes, prompt examples, case studies. It's a work in progress and keeps growing.

    • hallucination-detection
    • study-guide
    • jupyter notebook
    • fine-tuning
30. 30

    RagaAI-Catalyst

    16.2Kraga-ai-hub/RagaAI-Catalyst · Python

    RagaAI Catalyst is a platform for managing and optimizing LLM projects — tracing, evaluating and safeguarding LLM and agent applications, shipped as a Python SDK.

    • guardrails
    • red-teaming
    • cost-tracking
    • llmops
31. 31

    ragas

    16Kvibrantlabsai/ragas · Python

    Ragas is an open-source Python toolkit for evaluating and optimizing LLM applications: objective metrics, automatic test-set generation and data-driven insights replace slow, subjective manual assessment.

    • evals
    • llmops
    • llm-evaluation
    • test-automation
32. 32

    ms-swift

    15.9Kmodelscope/ms-swift · Python

    ms-swift is the ModelScope community's fine-tuning and deployment framework for LLMs and multimodal models, covering training, inference, evaluation, quantization and deployment in one pipeline.

    • peft
    • embedding
    • sft
    • grpo
33. 33

    StableLM

    15.7KStability-AI/StableLM · Jupyter Notebook

    StableLM is Stability AI's family of open-source language models, including the StableLM-3B-4E1T model, StableLM-Alpha v1/v2 base and tuned models, and StableVicuna, an RLHF-tuned chatbot built on Vicuna-13B.

    • rlhf
    • model-training
    • language-model
    • llama
34. 34

    openface

    15.4Kcmusatyalab/openface · Lua

    OpenFace is a free, open-source face recognition toolkit built on deep neural networks, developed by Carnegie Mellon University.

    • face-recognition
    • deep-neural-networks
    • classification
    • image-classification
35. 35

    math

    15.2Kopenai/math · Lean

    A public catalogue of 719 mathematical manuscripts grouped into 372 families, written by an internal OpenAI model during evaluations on open research problems. About 42% of top-line results already have Lean formalizations.

    • lean
    • theorem-proving
    • research-tools
    • science
36. 36

    easy-dataset

    15KConardLi/easy-dataset · JavaScript

    Easy Dataset is an app that turns domain documents (PDF, Markdown, DOCX, TXT, EPUB) into structured datasets for LLM fine-tuning, RAG, and model evaluation. It extracts questions from your text, generates answers with an LLM, and exports ready-to-use training data.

    • document-parsing
    • agplv3
    • dataset
    • fine-tuning
37. 37

    litgpt

    13.7KLightning-AI/litgpt · Python

    LitGPT is a collection of 20+ high-performance large language models (Llama, Gemma, Phi, Qwen, Mistral, Falcon, TinyLlama and more) implemented from scratch, with ready-made recipes to pretrain, finetune, evaluate and deploy them at scale.

    • tpu
    • qlora
    • llama
    • llms
38. 38

    AI-Research-SKILLs

    13.6KOrchestra-Research/AI-Research-SKILLs · TeX

    It's an open-source library of 98 AI research and engineering skills that turn a Claude Code, Codex, or Gemini CLI agent into a full AI research agent, covering everything from idea generation to paper writing.

    • gpt-5
    • paper-writing
    • grpo
    • ai-research
39. 39

    gorilla

    13.1KShishirPatil/gorilla · Python

    Gorilla is a Berkeley research project that trains and evaluates LLMs for calling APIs and tools correctly from natural-language requests. It has grown into a suite covering finetuned models (OpenFunctions), the Berkeley Function Calling Leaderboard (BFCL), an execution runtime (GoEx), a RAG fine-tuning recipe (RAFT) and a CLI tool.

    • claude-api
    • api-documentation
    • function-calling
    • tool-calling
40. 40

    open_deep_research

    12.7Klangchain-ai/open_deep_research · Python

    Open Deep Research is a simple, configurable, fully open-source deep research agent that can research a topic across the web and MCP servers and write a report, working with many different LLM providers and search tools.

    • deep-research
    • web-search
    • llm-agents
    • langchain
41. 41

    Chinese-Word-Vectors

    12.2KEmbedding/Chinese-Word-Vectors · Python

    A collection of 100+ pre-trained Chinese word vectors (embeddings) trained with different representations, context features and corpora, plus the CA8 analogy dataset and an evaluation toolkit.

    • word-embeddings
    • word2vec
    • chinese-nlp
    • embedding
42. 42

    opensre

    12.1KTracer-Cloud/opensre · Python

    An open-source framework for building AI SRE agents that resolve production incidents on your own infrastructure — and a training and evaluation environment for those agents.

    • sre
    • alerting
    • grafana
    • gcp
43. 43

    bisheng

    12.1Kdataelement/bisheng · Python

    BISHENG is an open-source LLM application devops platform built for enterprise AI, covering GenAI workflow building, RAG, agents, model management, evaluation, SFT, dataset management and observability.

    • llm-ops
    • sft
    • ldap
    • genai
44. 44

    seamless_communication

    11.9Kfacebookresearch/seamless_communication · Jupyter Notebook

    Seamless Communication is Meta's family of foundational AI models for speech and text translation across roughly 100 languages, covering speech-to-speech, speech-to-text, text-to-speech and text-to-text translation plus automatic speech recognition.

    • foundation-models
    • whisper
    • audio-processing
    • speech-to-text
45. 45

    tensorzero

    11.7Ktensorzero/tensorzero · Rust

    TensorZero is an open-source LLMOps platform that combines an LLM gateway, observability, evaluation, optimization, and experimentation tooling into one self-hosted stack.

    • ai-engineering
    • genai
    • gateway
    • ml
46. 46

    OpenJarvis

    11.5Kopen-jarvis/OpenJarvis · Python

    OpenJarvis is an open-source framework for personal AI that runs on your own devices: agents use local models by default and call the cloud only when truly necessary.

    • personal-ai
    • on-device
    • text-to-speech
    • ollama
47. 47

    arc-task-gen

    11Kpathwaycom/arc-task-gen · Python

    A generator for fresh, original ARC-AGI-1-style puzzle tasks whose statistical distribution matches the public ARC-AGI-1 evaluation set, so models can be tested on problems they haven't seen before.

    • llm-evaluation
    • reasoning
    • paper
    • dataset
48. 48

    AgentGuide

    11Kadongwanai/AgentGuide · MDX

    AgentGuide is a free, open-source Chinese-language knowledge base for AI Agent engineering and job hunting, modeled on JavaGuide. It stitches scattered courses, tutorials, papers and projects into one systematic path from beginner to a job offer.

    • job-hunting
    • sft
    • grpo
    • graphrag
49. 49

    llm-engineer-toolkit

    10.9KKalyanKS-NLP/llm-engineer-toolkit

    A curated catalog of 120+ LLM libraries organized by category, covering training, application development, RAG, inference, serving, data extraction, agents, evaluation, monitoring, prompts, safety and embeddings.

    • mlops
    • llms
    • embeddings
    • fine-tuning
50. 50

    awesome-semantic-segmentation

    10.8Kmrgloom/awesome-semantic-segmentation

    A curated list of resources for semantic segmentation: network architectures, papers, code implementations, datasets, benchmarks, annotation tools, and losses used in the field.

    • autonomous-driving
    • medical-imaging
    • semantic-segmentation
    • image-segmentation
51. 51

    voltagent

    10.8KVoltAgent/voltagent · TypeScript

    VoltAgent is an end-to-end AI agent engineering platform: an open-source TypeScript framework for building agents with memory, tools, RAG, workflows and voice, plus the VoltOps Console for observability and operations.

    • multiagent
    • chatbots
    • llm-observability
    • agent-observability
52. 52

    gpt-instruct

    10.3KMDX-Tom/gpt-instruct · Python

    gpt-instruct is a collection of jailbreak-style custom instructions for OpenAI's Codex/GPT models, shipped together with a reproducible Python test harness that scores how well each prompt survives regression testing.

    • ai-safety
    • jailbreak
    • red-team
    • gpt
53. 53

    kev

    9.3Kjaredpalmer/kev · Python

    Kev is an open family of small decision models (0.8B to 27B parameters) built on Qwen3.5/3.8 that answer yes/no, multiple-choice and rating questions about text with calibrated probabilities, acting as a self-hosted drop-in replacement for TypeSafe's hosted Jev model.

    • qwen3
    • decision-model
    • jev
    • mlx
54. 54

    Agent-Learning-Hub

    8.8Kdatawhalechina/Agent-Learning-Hub · HTML

    Agent Learning Hub is a curated, stage-by-stage roadmap and resource library for learning to build AI agents, covering everything from a minimal tool-call loop to shipping a production-ready multi-agent system.

    • roadmap
    • multi-agent-systems
    • autonomous-agents
    • agent-framework
55. 55

    ClawWork

    8.6KHKUDS/ClawWork · Python

    ClawWork turns AI agents into 'AI coworkers' that must earn real income by completing real professional tasks from the GDPVal dataset, starting with only $10 and paying for their own token usage out of their earnings.

    • openclaw
    • gpt
    • finance
    • qwen
56. 56

    reef

    7.6KHuman-Agent-Society/reef · Python

    Reef is open-source infrastructure for continually self-improving agents. It wires together agent inference, feedback collection, learning, and versioned delivery so an agent keeps getting better from how you interact with it.

    • agent-infrastructure
    • self-improving-agents
    • llm-training
    • sglang
57. 57

    cheat-on-content

    7.3KXBuilderLAB/cheat-on-content · Python

    A skill for AI coding agents aimed at content creators: every post becomes a calibrated experiment — score it, blind-predict its performance, publish, retro the numbers at T+3 days, and evolve your scoring rubric. The author credits it with going from 0 to 1M followers in a month.

    • forecasting
    • decision-making
    • social-media
    • content-creation
58. 58

    DeepSpec

    7.2Kdeepseek-ai/DeepSpec · Python

    DeepSpec is DeepSeek's full-stack Python codebase for training and evaluating draft models for speculative decoding — the technique where a small draft model accelerates a larger target LLM.

    • model-training
    • speculative-decoding
    • training
    • llm-inference
59. 59

    open-multi-agent

    7Kopen-multi-agent/open-multi-agent · TypeScript

    open-multi-agent (OMA) is a self-hosted TypeScript framework for orchestrating teams of AI agents: a coordinator turns a single goal into a task DAG at runtime, a deterministic scheduler executes it, and every run stays inspectable, approvable, and replayable.

    • ai-governance
    • durable-execution
    • human-in-the-loop
    • multi-agent-systems
60. 60

    darwin-skill

    6.4Kalchaincyf/darwin-skill · HTML

    Darwin.skill is a system that evolves Claude Code Agent Skills through an automated evaluate-improve-test-keep-or-revert loop, inspired by Andrej Karpathy's autoresearch project.

    • self-evolving-agents
    • self-evolving
    • skill-md
    • optimization
61. 61

    agents-cli

    6.1Kgoogle/agents-cli · Python

    agents-cli gives your coding assistant (Claude Code, Codex, Antigravity CLI, etc.) the CLI commands and skill files it needs to build, evaluate, and deploy AI agents on Google Cloud using the Agent Development Kit (ADK).

    • agent-evaluation
    • agent-observability
    • google-cloud
    • gcp
62. 62

    CUAWright

    6.1Kmicrosoft/CUAWright · Python

    Webwright is a lightweight coding-agent framework that gives an LLM a terminal to launch and inspect browser sessions, completing web tasks end-to-end as a single re-runnable Python script instead of a step-by-step click/type loop.

    • playwright
    • autonomous-agents
    • browser-automation
    • skill
63. 63

    SWE-bench

    6KSWE-bench/SWE-bench · Python

    SWE-bench is a benchmark that tests large language models on real GitHub issues: the model gets a codebase plus an issue and must produce a patch that resolves it. An automated harness scores the patches.

    • software-engineering
    • language-model
    • code-generation
    • aws
64. 64

    hermes-agent-self-evolution

    5.5KNousResearch/hermes-agent-self-evolution · Python

    It is an evolutionary self-improvement system for Hermes Agent that uses DSPy and GEPA (Genetic-Pareto Prompt Evolution) to automatically optimize the agent's skills, tool descriptions, system prompts, and code, producing measurably better versions through reflective evolutionary search.

    • nous-research
    • self-improvement
    • recursive-self-improvement
    • self-evolving-agents
65. 65

    Raven

    5.5KEverMind-AI/Raven · Python

    Raven is an open-source, self-improving AI agent harness that runs long-running agent work from the terminal, with built-in long-term memory, skills, tracing, and evaluation. Its stated long-term direction ('Harness of Harnesses') is a multi-agent ecosystem that composes different harnesses together, but that part is not yet implemented in the current release.

    • rsi
    • self-improving
    • recursive-self-improvement
    • llm-agent
66. 66

    PRAXIST

    5.4Ksapientinc/PRAXIST · Python

    Praxist is an autonomous research system that turns an already-runnable, measurable project into a continuous, evidence-driven research loop, running parallel research agents across generations until the search converges or the budget runs out.

    • fair-source
    • self-evolving
    • autonomous-agents
    • workflow-automation
67. 67

    FrontierAgent

    5.3KApodexAI/FrontierAgent · Python

    FrontierAgent is an open-source agent runtime, terminal TUI product, and evaluation suite for long-horizon research and file-based work, built around two workflows called ReAct and Agent Team.

    • agentic-framework
    • autonomous-agents
    • harness
    • agent-framework
68. 68

    awesome-harness-engineering

    4.9Kai-boost/awesome-harness-engineering · Python

    A curated list of resources on 'harness engineering' — the discipline of designing the scaffolding (context delivery, tool interfaces, planning artifacts, verification loops, memory, sandboxes) around an AI agent that determines whether it succeeds on real tasks.

    • harness-engineering
    • agent-harness
    • context-engineering
    • agent-framework
69. 69

    SemIf-OpenJev

    4.8KTheoLeeCJ/SemIf-OpenJev · Python

    SemIf (formerly OpenJev) is an independent research project that runs runtime-defined semantic decisions with open models on one consumer GPU, mirroring the interface of TypeSafe's closed Jev service but not its undisclosed model.

    • decision-making
    • webgpu
    • llama-cpp
    • mlx
70. 70

    autoagent

    4.6Kthirdlayerinc/autoagent · Python

    AutoAgent lets an AI agent build and iterate on an agent harness (agent.py) autonomously overnight: it tweaks the system prompt, tools, config, and orchestration, scores each change against a benchmark, and keeps only the improvements.

    • harness-framework
    • self-evolving
    • harness-engineering
    • llm-agents
71. 71

    awesome-harness-engineering

    4.4Kwalkinglabs/awesome-harness-engineering

    A curated list of articles, playbooks, benchmarks, specifications, and open-source projects about harness engineering — shaping the environment around AI agents so they can work reliably.

    • harness-framework
    • benchmarks
    • harness-engineering
    • agent-harness
72. 72

    MSA

    3.5KEverMind-AI/MSA · Python

    MSA (Memory Sparse Attention) is an end-to-end trainable, scalable latent-memory framework that lets LLMs handle context windows up to 100M tokens, replacing RAG or fixed-size RNN-style memory with a differentiable sparse-attention memory bank.

    • context-window
    • kv-cache
    • pretraining
    • long-context
73. 73

    ZDTaichu5.0-9B

    3.1KTaichu-AI/ZDTaichu5.0-9B · Python

    ZDTaichu5.0-9B is a multimodal foundation model that combines a Qwen3.5-9B language backbone with a C-RADIOv4-H vision encoder for general visual understanding, spatial reasoning, embodied-AI research and agentic tool use.

    • nvfp4
    • vision-language-action-model
    • large-language-model
    • embodied-ai
74. 74

    CLM

    3.1KContrastive-LM/CLM · Python

    CLM (Contrastive Language Models) is a 'System One' model trained with a contrastive learning objective that connects states and actions, served behind a TypeSafe-compatible API as CLM-8B. It is meant to be plugged into agents as a fast scorer/verifier for decisions, tool calls, and best-of-N candidates.

    • model-serving
    • agentic-coding
    • llm-inference
    • embeddings
75. 75

    ai-agents-the-definitive-guide

    2.8KNicolepcx/ai-agents-the-definitive-guide · Jupyter Notebook

    It is the companion code repository for the book 'AI Agents - The Definitive Guide': a set of runnable Jupyter notebooks that walk through every chapter, from basic agent patterns to production deployment and threat modeling.

    • benchmarks
    • production
    • evaluation-framework
    • pydantic
76. 76

    NanoJev

    2.6KTianyuCodings/NanoJev · Python

    A nano replica of Jev: a 0.6B-parameter parallel decision model that takes states and questions in and returns complete probability distributions — no output tokens are ever decoded.

    • game-ai
    • imitation-learning
    • sft
    • qwen3
77. 77

    nimble

    2.2Kbespokelabsai/nimble · Python

    Bespoke Nimble turns text plus a schema of questions into typed decisions: it picks one allowed answer per question and returns the probability of every option, in a single fast step. The repo shares the recipe and the trained Bespoke-Nimble-9B model.

    • jev-model
    • calibration
    • typed-decisions
    • decision-making
78. 78

    tapnet

    2.2Kgoogle-deepmind/tapnet · Jupyter Notebook

    This is Google DeepMind's official repository for Tracking Any Point (TAP) research: it bundles the TAP-Vid and TAPVid-3D benchmark datasets, the TAPIR/BootsTAPIR/TAPNext/TAPNext++ point-tracking models, the RoboTAP robotics extension, and the TRAJAN trajectory autoencoder.

    • colab
    • segmentation
    • robotics
    • dataset
79. 79

    deepopen

    2Kdeepopen-com/deepopen · Python

    DeepOpen is an open-source, non-autoregressive 'System 1' decision engine: instead of generating text token-by-token like an LLM, it makes typed decisions (choice, score, or boolean) over text, email, tickets or JSON in a single forward pass, across 100+ languages, in as little as 33ms.

    • bert
    • typed-decisions
    • decision-model
    • model
80. 80

    ai-agent-handbook

    2Kaliyun/ai-agent-handbook · JavaScript

    An open-source Alibaba Cloud whitepaper (handbook) distilling hands-on experience building enterprise-grade AI agents across their full lifecycle: architecture, building, running, governance, and tuning. It is maintained as a living open project rather than a one-off release.

    • agent-governance
    • governance
    • ai-security
    • context-engineering
81. 81

    ai-engineering-interview-questions-company-wise

    1.8Kpallavi-shekhar/ai-engineering-interview-questions-company-wise · Markdown

    A free, continuously updated cheat sheet of real interview questions asked in AI Engineering interviews at 35 companies (Anthropic, OpenAI, Google DeepMind, Meta, NVIDIA, Databricks and more), organized company by company with answers linked wherever available.

    • ai-engineering
    • speculative-decoding
    • job-search
    • llms
82. 82

    NeoHorse

    1.7KTokenRhythm/NeoHorse · Python

    NeoHorse is TokenRhythm's family of open-weight language models built for agent workflows, with the 4B and 9B NeoHorse-1 checkpoints handling tool use, coding, and instruction following, plus a companion decision model called NeoHorse-Jev.

    • rsi
    • recursive-self-improvement
    • sglang
    • function-calling
83. 83

    delivery-harness

    1.7Kdiudiu-tech/delivery-harness · Java

    An AI harness reference implementation for on-demand delivery operations that combines deterministic workflows, mock business tools, lightweight knowledge retrieval, an OpenAI-compatible model endpoint, evaluation scaffolding, and human-review guardrails.

    • guardrails
    • spring
    • spring-boot
    • retrieval-augmented-generation
84. 84

    rrsi

    1.5Kgoogle-research/rrsi · Python

    RRSI (Regularized Recursive Self-Improvement) is a Google research framework that automatically improves an LLM agent's harness (prompts, control flow, tools, memory, context management around a frozen model) by searching over harness edits and keeping only those with measured, auditable gains.

    • vertex-ai
    • benchmarks
    • self-improvement
    • self-improving
85. 85

    livenerf

    1.5Kninjahawk/livenerf · Python

    livenerf is a long-running, append-only benchmark that checks whether a frontier Claude model quietly gets worse after it ships, by comparing daily performance against a locked day-0 baseline.

    • evaluation-framework
    • reproducible-research
    • ai-research
    • ai-safety
86. 86

    evals-skills

    1.5Kai-evals-course/evals-skills

    A collection of agent skills that guide AI coding agents to build product-specific AI evals — not foundation-model benchmarks. It encodes mistakes the authors saw across 50+ companies and thousands of students in their AI Evals course.

    • annotation-tool
    • ai-skills
    • claude-plugin
    • annotation
87. 87

    AnyJev

    1.4Knokia-applied-research/AnyJev · Python

    A Python library that turns any open LLM into a Jev-style decision model: ask a typed question, get back a choice with a probability you can threshold — no generation, no parsing, no fine-tuning.

    • jev-model
    • system-one
    • calibration
    • decision-model
88. 88

    jevlike

    1.4Kvinnylarouge/jevlike · Python

    A small research model that reads a piece of text plus a list of options and returns a probability for each option in one pass, instead of writing an answer token by token.

    • text-classification
    • attention
    • decision-model
    • classification
89. 89

    nautilus-compass

    1.3Kchunxiaoxx/nautilus-compass · Python

    nautilus-compass is an open-source memory and reliability layer for AI coding agents: it gives them long-term memory, catches cases where the agent ignores recalled rules (drift detection), and ships its own benchmark suite to verify the claims independently.

    • reproducible-research
    • verification
    • a2a
    • rpc
90. 90

    lift

    1.2Kdatalab-to/lift · Python

    lift is an open-source 9B vision-language model and toolkit that extracts structured JSON data from PDFs and images by matching a schema you provide, using schema-constrained decoding to guarantee valid output.

    • document-intelligence
    • vision-language
    • json-schema
    • schema
91. 91

    RelateAnything

    1.1KMaelic/RelateAnything · Python

    RelateAnything is a real-time, open-vocabulary relation prediction model: give it an image, object regions and the relations to look for, and it returns scored (subject, relation, object) triplets like person → riding → horse.

    • tensorrt
    • pretrained-models
    • onnx
    • training
92. 92

    Tracely-ai

    1Kjwuthri-cloud/Tracely-ai · Python

    Tracely is a trace-native CI/CD platform for AI agents: it grades every production agent trace, clusters failures into issues, freezes bad runs into hermetic regression tests, blocks the PR that would ship them again, and alerts you when it happens.

    • agent-evaluation
    • llm-observability
    • sse
    • tracing
93. 93

    GLM-5.3-Flash-J-Space-Capability-Realization-Report

    1KTiger3807861189/GLM-5.3-Flash-J-Space-Capability-Realization-Report

    J-Space Cognition Suite is a model-agnostic inference-time control system for deep reasoning, long-horizon work, tool use, verification, and recovery, packaged as a Skill that loads into AI agent hosts.

    • glm
    • harness-engineering
    • long-term-memory
    • reasoning
94. 94

    marigold-v2

    914huawei-bayerlab/marigold-v2 · Python

    Marigold V2 is a family of models and a cost-effective fine-tuning protocol that repurposes a pretrained diffusion transformer into single-step dense predictors: depth, see-through depth, surface normals, albedo, and other dense modalities.

    • diffusion
    • diffusion-models
    • paper
    • lora
95. 95

    APSD

    897hcliucs/APSD

    APSD is a large-scale dataset for studying how visually stealthy adversarial perturbations look to humans, containing 10,586 clean-adversarial image pairs with human opinion scores.

    • adversarial-attacks
    • vision
    • security-research
    • ai-security
96. 96

    Valen

    861Liuziyu77/Valen · Jupyter Notebook

    Valen (万澜) is a multimodal decision model that adds visual perception to System One-style decision-making: it scores candidate actions against text, images and video using a task instruction, instead of generating free-form answers.

    • vision-language
    • multimodal-ai
    • jev-model
    • system-one
97. 97

    Intern-Decision

    829InternLM/Intern-Decision · Python

    Intern-Decision turns a state, optional images, and multiple typed questions into decisions with probabilities, by fine-tuning the language backbone of Qwen3.5 while freezing its vision tower and projector.

    • calibration
    • qwen3
    • decision-model
    • model-training
98. 98

    XRAG

    811DocAILab/XRAG · Python

    XRAG is a benchmarking framework that dissects and evaluates the foundational components of Retrieval-Augmented Generation (RAG) systems, such as retrievers, embeddings, and LLMs, to show how different configurations affect overall RAG performance.

    • llamaindex
    • arxiv
    • retrieval-augmented-generation
    • llama
99. 99

    awesome-rsi

    801lobehub/awesome-rsi

    Awesome RSI is a curated research map of Recursive Self-Improvement (RSI): a taxonomy of papers, frameworks, benchmarks, and safety work on AI systems that improve their own capabilities and the mechanisms that generate further improvements.

    • agi
    • rsi
    • self-improving
    • self-improving-agents
100. 100

     RSI-Anything

     785Gen-Verse/RSI-Anything · HTML

     RSI-Anything is a domain-general framework that turns world-model predictions and interaction feedback into recursive self-improvement, built on JEPA-Anything.

     • recursive-self-improvement
     • world-models
     • llm-agents
     • paper

Showing the 100 most-starred of 155. See every one in the catalog.

• python2,418
• llm1,489
• ai1,957
• benchmark430
• research764
• open-source5,667
• machine-learning918
• ai-agents1,298
• linux4,182
• apache-2.01,434
• macos3,545
• deep-learning737
• gpu788
• cli2,229
• windows3,403
• mit3,311
• huggingface361
• agent820
• agents422
• fine-tuning185
• agentic-ai312
• rag294
• developer-tools2,930
• multi-agent406