ragas README

The project's README from github.com/vibrantlabsai/ragas. As one markdown file: README.md.

Supercharge Your LLM Application Evaluations 🚀

Latest release Made with Python License Apache-2.0 Ragas Downloads per month Join Ragas community on Discord Ask DeepWiki.com

Documentation | Quick start | Join Discord | Blog | NewsLetter | Careers

Objective metrics, intelligent test generation, and data-driven insights for LLM apps

Ragas is your ultimate toolkit for evaluating and optimizing Large Language Model (LLM) applications. Say goodbye to time-consuming, subjective assessments and hello to data-driven, efficient evaluation workflows. Don't have a test dataset ready? We also do production-aligned test set generation.

Key Features

:shield: Installation

Pypi:

pip install ragas

Alternatively, from source:

pip install git+https://github.com/vibrantlabsai/ragas

:fire: Quickstart

Clone a Complete Example Project

The fastest way to get started is to use the ragas quickstart command:

# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project

Available templates:

Coming Soon:

Evaluate your LLM App

ragas comes with pre-built metrics for common evaluation tasks. For example, Aspect Critique evaluates any aspect of your output using DiscreteMetric:

import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.

Response: {response}

Answer with only 'accurate' or 'inaccurate'."""
)

# Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")  # 'accurate' or 'inaccurate'
    print(f"Reason: {score.reason}")


if __name__ == "__main__":
    asyncio.run(main())

Note: Make sure your OPENAI_API_KEY environment variable is set.

Find the complete Quickstart Guide

Want help in improving your AI application using evals?

In the past 2 years, we have seen and helped improve many AI applications using evals. If you want help with improving and scaling up your AI application using evals.

🔗 Book a slot or drop us a line: [email protected].

🫂 Community

If you want to get more involved with Ragas, check out our discord server. It's a fun community where we geek out about LLM, Retrieval, Production issues, and more.

Contributors

+----------------------------------------------------------------------------+
|     +----------------------------------------------------------------+     |
|     | Developers: Those who built with `ragas`.                      |     |
|     | (You have `import ragas` somewhere in your project)            |     |
|     |     +----------------------------------------------------+     |     |
|     |     | Contributors: Those who make `ragas` better.       |     |     |
|     |     | (You make PR to this repo)                         |     |     |
|     |     +----------------------------------------------------+     |     |
|     +----------------------------------------------------------------+     |
+----------------------------------------------------------------------------+

We welcome contributions from the community! Whether it's bug fixes, feature additions, or documentation improvements, your input is valuable.

  1. Fork the repository
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

🔍 Open Analytics

At Ragas, we believe in transparency. We collect minimal, anonymized usage data to improve our product and guide our development efforts.

✅ No personal or company-identifying information

✅ Open-source data collection code

✅ Publicly available aggregated data

To opt-out, set the RAGAS_DO_NOT_TRACK environment variable to true.

Cite Us

@misc{ragas2024,
  author       = {VibrantLabs},
  title        = {Ragas: Supercharge Your LLM Application Evaluations},
  year         = {2024},
  howpublished = {\url{https://github.com/vibrantlabsai/ragas}},
}

ragas (vibrantlabsai/ragas)

test-automation
Searching Database…

16K ·GitHub
Ragas
What is this?Ragas is an open-source Python toolkit for evaluating and optimizing LLM applications: objective metrics, automatic test-set generation and data-driven insights replace slow, subjective manual assessment.
Is it a library I call from my own code, or a hosted platform?A Python library: you install it from PyPI (or build from source) into your own project, and it also ships a small CLI whose 'ragas quickstart' command scaffolds a complete example project.
How do I get started with it?Install it with pip, export your OPENAI_API_KEY, then run 'ragas quickstart' — it clones a ready-to-run evaluation project; the template available today is rageval for evaluating RAG systems.
Do I need an OpenAI API key to run the evaluations?Yes for the quickstart: the README reminds you to set the OPENAI_API_KEY environment variable, since the metrics include LLM-based ones that call a model under the hood.
Can I evaluate AI agents with it, or is it RAG-only for now?Ready-made coverage is RAG-first: rageval evaluates RAG systems today, while templates for agent evals, prompt variations, workflow evals and LLM benchmarking are still marked Coming Soon.
Does it collect any data about my usage?Yes, minimal anonymized usage analytics are collected by default to improve the product; the collection code is open source and the aggregates are public. Set RAGAS_DO_NOT_TRACK=true to opt out.
Which frameworks and tools does it plug into?It advertises seamless integrations with popular LLM frameworks such as LangChain and with major observability tools, so it can drop into the stack your app already uses.
What if I don't have a test dataset yet?Ragas generates one for you: it automatically builds comprehensive, production-aligned test datasets covering a wide range of scenarios, so you can start evaluating before collecting real data.
What kinds of metrics are built in?Pre-built metrics for common evaluation tasks, both LLM-based and traditional — for example, Aspect Critique scores any aspect of your output using DiscreteMetric.
Why use it instead of eyeballing model outputs myself?Manual assessment is time-consuming and subjective; Ragas replaces it with objective metrics, generated test data and production feedback loops that continually improve your LLM application.
What are similar projects?mlflow, jevals, opik, tensorzero, langfuse, phoenix, ai-engineer-notebooks, bisheng
Which topics is it in?evals, llmops, llm-evaluation, test-automation, metrics, mlops, toolkit, langchain, dataset, evaluation, observability, rag
Creator
vibrantlabsai
Size
46.4 MB
Stars
16K
Updated
CommitFeb 24Releasev0.4.3Jan 13
Age
3y
Commits · 90 Days