XRAG README

The project's README from github.com/DocAILab/XRAG. As one markdown file: README.md.

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

XRAG

PyPI version Python License Downloads GitHub stars GitHub issues arXiv

📑 Table of Contents

:mega: Updates

XRAG Roadmap

Welcome developers, researchers, and enthusiasts to join the XRAG open-source project!

📊 Novel Novel Evaluation Metrics

🤖 LLM Experiments (Various Parameter Sizes)

🧠 Ever-Evolving Agent Workflows

📊 Other Novel RAG modules

🙌 Your contributions—code, data, ideas, or feedback—are the heartbeat of XRAG!
Repository: https://github.com/DocAILab/XRAG

📖 Introduction

XRAG_Framework XRAG is a benchmarking framework designed to evaluate the foundational components of advanced Retrieval-Augmented Generation (RAG) systems. By dissecting and analyzing each core module, XRAG provides insights into how different configurations and components impact the overall performance of RAG systems.

✨ Features

🧠 Agentic RAG

Orchestrator Orchestrators are used to organize and manage the execution logic and workflow of RAG components, thereby achieving agentic RAG methods in XRAG. As illustrated in figure, the XRAG framework includes five types of orchestrators: sequential, conditional, iterative, parallel, and hybrid.

🌐 WebUI Demo

XRAG provides an intuitive web interface for interactive evaluation and visualization. Launch it with:

xrag-cli webui

The WebUI guides you through the following workflow:

1. Dataset Upload and Configuration

Dataset Selection

Upload and configure your datasets:

2. Index Building and Configuration

Index Building

Configure system parameters and build indices:

3. RAG Strategy Configuration

RAG Strategies

Define your RAG pipeline components:

4. Interactive Testing

Testing Interface

Test your RAG system interactively:

5. Comprehensive Evaluation

Evaluation Metrics

🛠️ Installation

Before installing XRAG, ensure that you have Python 3.11 or later installed.

Create a Virtual Environment via conda(Recommended)

  
# Create a new conda environment
conda create -n xrag python=3.11

# Activate the environment
conda activate xrag

Install via pip

You can install XRAG directly using pip:

# Install XRAG
pip install examinationrag

# Install 'jury' without dependencies to avoid conflicts
pip install jury --no-deps

# Adjust some package versions
pip install requests==2.27.1
pip install urllib3==1.25.11
pip install jiwer<4.0.0

🚀 Quick Start

Here's how you can get started with XRAG:

1. Prepare Configuration:

Modify the config.toml file to set up your desired configurations.

2. Using xrag-cli

After installing XRAG, the xrag-cli command becomes available in your environment. This command provides a convenient way to interact with XRAG without needing to call Python scripts directly.

Command Structure

xrag-cli [command] [options]

Commands and Options

Using the API Service

Once the API server is running, you can interact with it using HTTP requests. Here are the available endpoints:

1. Query Endpoint

Send a POST request to /query to get answers based on your documents:

curl -X POST "http://localhost:8000/query" \
     -H "Content-Type: application/json" \
     -d '{
           "query": "your question here",
           "top_k": 3
         }'

Response format:

{
    "answer": "Generated answer to your question",
    "sources": [
        {
            "content": "Source document content",
            "id": "document_id",
            "score": 0.85
        }
    ]
}

2. Health Check

Check the API server status with a GET request to /health:

curl "http://localhost:8000/health"

Response format:

{
    "status": "healthy",
    "engine_status": "initialized"
}

The API service supports both custom JSON datasets and folder-based documents:

Overriding Configuration Parameters

Use the --override flag followed by key-value pairs to override configuration settings:

xrag-cli run --override embeddings="new-embedding-model"

Generate QA pairs from a folder

xrag-cli generate -i <input_file> -o <output_file> -n <num_questions> -s <sentence_length>

Automatically generate QA pairs from a folder.

⚙️ Configuration

XRAG uses a config.toml file for configuration management. Here's a detailed explanation of the configuration options:

[api_keys]
api_key = "sk-xxxx"          # Your API key for LLM service
api_base = "https://xxx"     # API base URL
api_name = "gpt-4o"     # Model name
auth_token = "hf_xxx"        # Hugging Face auth token

[settings]
llm = "openai" # openai, huggingface, ollama
ollama_model = "llama2:7b" # ollama model name
huggingface_model = "llama" # huggingface model name
embeddings = "BAAI/bge-large-en-v1.5"
split_type = "sentence"
chunk_size = 128
dataset = "hotpot_qa"
persist_dir = "storage"
# ... additional settings ...

❗ Troubleshooting

📝 Changelog

Version 0.1.3

Version 0.1.2

Version 0.1.1

Version 0.1.0

💬 Feedback and Support

We value feedback from our users. If you have suggestions, feature requests, or encounter issues:

:round_pushpin: Acknowledgement

📚 Citation

If you find this work helpful, please cite our paper:

@INPROCEEDINGS{11629265,
  author={Zhang, Qili and Mao, Qianren and Luo, Yangyifei and Luo, Yashuo and Hao, Hanwen and Cao, Zhilong and Jiang, Weifeng and Chen, Zhijun and Liu, Junnan and Yan, Feng and Wang, Xiaolong and Zhang, Jinlong and Huang, Zhenting and Tan, Zhixing and Sun, Jie and Li, Bo and Li, Jianxin and Yu, Philip S.},
  booktitle={2026 IEEE 42nd International Conference on Data Engineering (ICDE)}, 
  title={XRAG: Examining the Core - Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation}, 
  year={2026},
  pages={2697-2710},
  keywords={Modeling;Retrieval augmented generation;Context;Measurement;Large language models;Printing;Indexes;Indexing;Generative Pre-trained transformer;Modules (abstract algebra);Database;Retrieval-Augmented Generation;Evaluation;Benchmark;Data Management},
  doi={10.1109/ICDE65706.2026.00201}}

Or access via arXiv.

🙏 Thank You

Thank you for using XRAG! We hope it proves valuable in your research and development efforts in the field of Retrieval-Augmented Generation.

XRAG (DocAILab/XRAG)

vector-search
Searching Database…

811 ·GitHub
XRAG
What is this?XRAG is a benchmarking framework that dissects and evaluates the foundational components of Retrieval-Augmented Generation (RAG) systems, such as retrievers, embeddings, and LLMs, to show how different configurations affect overall RAG performance.
Is this a CLI tool, a web app, or both?Both, plus an API server: XRAG ships an xrag-cli for running benchmarks and generating QA pairs, a browser-based WebUI for interactive dataset upload, index building and evaluation, and an 'api' command to run it as a backend service.
How do I install and run it?Create a Python 3.11+ virtual environment (conda recommended), install XRAG via pip, edit config.toml for your settings, then use xrag-cli run/webui/api/generate to launch the corresponding mode.
What LLMs do I need to plug in?You configure either an OpenAI-compatible API or a local model like Qwen or LLaMA served through Ollama; API keys and model parameters are set in config.toml.
Any installation or config pitfalls to watch for?The README warns about dependency conflicts from requirements.txt, config override keys that must match config.toml exactly, and override values needing the correct data type.
Who is this aimed at?It is aimed at RAG researchers and developers who want to systematically compare retrieval, embedding, and generation modules rather than ship a production chatbot.
What evaluation metrics can XRAG compute?It supports traditional metrics (F1, EM, MRR, Hit@K, MAP, NDCG) plus LLM-based metrics (Faithfulness, Relevancy, Correctness) and deep-eval metrics like contextual precision/recall, hallucination, and bias.
What retrieval methods can I benchmark?BM25, vector-based semantic search, tree-structured retrieval, keyword-based retrieval, document summary retrieval, and custom retrieval strategies are all built in.
What are the 'agentic RAG orchestrators'?Orchestrators (sequential, conditional, iterative, parallel, hybrid) manage the execution logic of RAG components so you can implement agentic methods like Self-RAG and Adaptive-RAG inside XRAG.
Can XRAG generate its own test data?Yes, the generate command automatically creates QA pairs from a folder of your own documents for use in benchmarking.
What are similar projects?production-agentic-rag-course, ParseBench, LightRAG, unsloth, MSA, txtai, xerj, Hyper-Extract
Which topics is it in?llamaindex, arxiv, retrieval-augmented-generation, llama, vector-search, paper, webui, transformers, semantic-search, qwen, evaluation, academic-research
Creator
DocAILab
Size
16.2 MB
Stars
811
Updated
CommitOct 11Releasev0.1.4Feb 07
Age
2y
Commits · 90 Days