XRAG README

The project's README from github.com/DocAILab/XRAG. As one markdown file: README.md.

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced Retrieval-Augmented Generation

XRAG

PyPI version Python License Downloads GitHub stars GitHub issues arXiv

📑 Table of Contents

:mega: Updates

XRAG Roadmap

Welcome developers, researchers, and enthusiasts to join the XRAG open-source project!

📊 Novel Novel Evaluation Metrics

🤖 LLM Experiments (Various Parameter Sizes)

🧠 Ever-Evolving Agent Workflows

📊 Other Novel RAG modules

🙌 Your contributions—code, data, ideas, or feedback—are the heartbeat of XRAG!
Repository: https://github.com/DocAILab/XRAG

📖 Introduction

XRAG_Framework XRAG is a benchmarking framework designed to evaluate the foundational components of advanced Retrieval-Augmented Generation (RAG) systems. By dissecting and analyzing each core module, XRAG provides insights into how different configurations and components impact the overall performance of RAG systems.

✨ Features

🧠 Agentic RAG

Orchestrator Orchestrators are used to organize and manage the execution logic and workflow of RAG components, thereby achieving agentic RAG methods in XRAG. As illustrated in figure, the XRAG framework includes five types of orchestrators: sequential, conditional, iterative, parallel, and hybrid.

🌐 WebUI Demo

XRAG provides an intuitive web interface for interactive evaluation and visualization. Launch it with:

xrag-cli webui

The WebUI guides you through the following workflow:

1. Dataset Upload and Configuration

Dataset Selection

Upload and configure your datasets:

2. Index Building and Configuration

Index Building

Configure system parameters and build indices:

3. RAG Strategy Configuration

RAG Strategies

Define your RAG pipeline components:

4. Interactive Testing

Testing Interface

Test your RAG system interactively:

5. Comprehensive Evaluation

Evaluation Metrics

🛠️ Installation

Before installing XRAG, ensure that you have Python 3.11 or later installed.

Create a Virtual Environment via conda(Recommended)

  
# Create a new conda environment
conda create -n xrag python=3.11

# Activate the environment
conda activate xrag

Install via pip

You can install XRAG directly using pip:

# Install XRAG
pip install examinationrag

# Install 'jury' without dependencies to avoid conflicts
pip install jury --no-deps

# Adjust some package versions
pip install requests==2.27.1
pip install urllib3==1.25.11
pip install jiwer<4.0.0

🚀 Quick Start

Here's how you can get started with XRAG:

1. Prepare Configuration:

Modify the config.toml file to set up your desired configurations.

2. Using xrag-cli

After installing XRAG, the xrag-cli command becomes available in your environment. This command provides a convenient way to interact with XRAG without needing to call Python scripts directly.

Command Structure

xrag-cli [command] [options]

Commands and Options

Using the API Service

Once the API server is running, you can interact with it using HTTP requests. Here are the available endpoints:

1. Query Endpoint

Send a POST request to /query to get answers based on your documents:

curl -X POST "http://localhost:8000/query" \
     -H "Content-Type: application/json" \
     -d '{
           "query": "your question here",
           "top_k": 3
         }'

Response format:

{
    "answer": "Generated answer to your question",
    "sources": [
        {
            "content": "Source document content",
            "id": "document_id",
            "score": 0.85
        }
    ]
}

2. Health Check

Check the API server status with a GET request to /health:

curl "http://localhost:8000/health"

Response format:

{
    "status": "healthy",
    "engine_status": "initialized"
}

The API service supports both custom JSON datasets and folder-based documents:

Overriding Configuration Parameters

Use the --override flag followed by key-value pairs to override configuration settings:

xrag-cli run --override embeddings="new-embedding-model"

Generate QA pairs from a folder

xrag-cli generate -i <input_file> -o <output_file> -n <num_questions> -s <sentence_length>

Automatically generate QA pairs from a folder.

⚙️ Configuration

XRAG uses a config.toml file for configuration management. Here's a detailed explanation of the configuration options:

[api_keys]
api_key = "sk-xxxx"          # Your API key for LLM service
api_base = "https://xxx"     # API base URL
api_name = "gpt-4o"     # Model name
auth_token = "hf_xxx"        # Hugging Face auth token

[settings]
llm = "openai" # openai, huggingface, ollama
ollama_model = "llama2:7b" # ollama model name
huggingface_model = "llama" # huggingface model name
embeddings = "BAAI/bge-large-en-v1.5"
split_type = "sentence"
chunk_size = 128
dataset = "hotpot_qa"
persist_dir = "storage"
# ... additional settings ...

❗ Troubleshooting

📝 Changelog

Version 0.1.3

Version 0.1.2

Version 0.1.1

Version 0.1.0

💬 Feedback and Support

We value feedback from our users. If you have suggestions, feature requests, or encounter issues:

:round_pushpin: Acknowledgement

📚 Citation

If you find this work helpful, please cite our paper:

@INPROCEEDINGS{11629265,
  author={Zhang, Qili and Mao, Qianren and Luo, Yangyifei and Luo, Yashuo and Hao, Hanwen and Cao, Zhilong and Jiang, Weifeng and Chen, Zhijun and Liu, Junnan and Yan, Feng and Wang, Xiaolong and Zhang, Jinlong and Huang, Zhenting and Tan, Zhixing and Sun, Jie and Li, Bo and Li, Jianxin and Yu, Philip S.},
  booktitle={2026 IEEE 42nd International Conference on Data Engineering (ICDE)}, 
  title={XRAG: Examining the Core - Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation}, 
  year={2026},
  pages={2697-2710},
  keywords={Modeling;Retrieval augmented generation;Context;Measurement;Large language models;Printing;Indexes;Indexing;Generative Pre-trained transformer;Modules (abstract algebra);Database;Retrieval-Augmented Generation;Evaluation;Benchmark;Data Management},
  doi={10.1109/ICDE65706.2026.00201}}

Or access via arXiv.

🙏 Thank You

Thank you for using XRAG! We hope it proves valuable in your research and development efforts in the field of Retrieval-Augmented Generation.

XRAG (DocAILab/XRAG)

Searching Database…

811 ·GitHub
XRAG
这是什么？XRAG 是一个基准测试框架，用于剖析和评估检索增强生成（RAG）系统的基础组件（如检索器、嵌入和 LLM），展示不同配置如何影响 RAG 的整体性能。
这是 CLI 工具、网页应用，还是两者都是？两者都是，还有一个 API 服务器：XRAG 自带用于运行基准测试和生成问答对的 xrag-cli、用于交互式上传数据集、构建索引和评估的浏览器版 WebUI，以及把它作为后端服务运行的 'api' 命令。
怎么安装和运行它？创建 Python 3.11+ 虚拟环境（推荐 conda），通过 pip 安装 XRAG，编辑 config.toml 做好你的设置，然后用 xrag-cli run/webui/api/generate 启动对应模式。
需要接入哪些 LLM？你可以配置兼容 OpenAI 的 API，或通过 Ollama 提供的 Qwen、LLaMA 等本地模型；API 密钥和模型参数都在 config.toml 中设置。
安装或配置上有什么坑需要留意？README 警告了 requirements.txt 带来的依赖冲突、配置覆盖键必须与 config.toml 完全一致，以及覆盖值需要正确的数据类型。
它面向哪些人？它面向想要系统性地比较检索、嵌入和生成模块的 RAG 研究者和开发者，而不是要上线生产级聊天机器人的人。
XRAG 能计算哪些评估指标？它支持传统指标（F1、EM、MRR、Hit@K、MAP、NDCG），以及基于 LLM 的指标（Faithfulness、Relevancy、Correctness）和上下文精确率/召回率、幻觉、偏见等 deep-eval 指标。
可以对哪些检索方法做基准测试？内置了 BM25、基于向量的语义搜索、树状检索、基于关键词的检索、文档摘要检索，以及自定义检索策略。
「智能体式 RAG 编排器」是什么？编排器（顺序、条件、迭代、并行、混合）管理 RAG 组件的执行逻辑，让你可以在 XRAG 内实现 Self-RAG 和 Adaptive-RAG 等智能体方法。
XRAG 能自己生成测试数据吗？可以，generate 命令会自动从你自己的文档文件夹创建问答对，用于基准测试。
What are similar projects?production-agentic-rag-course, ParseBench, LightRAG, unsloth, MSA, txtai, xerj, Hyper-Extract
Which topics is it in?llamaindex, arxiv, retrieval-augmented-generation, llama, vector-search, paper, webui, transformers, semantic-search, qwen, evaluation, academic-research
Creator
DocAILab
Size
16.2 MB
Stars
811
Updated
CommitOct 11Releasev0.1.4Feb 07
Age
2y
Commits · 90 Days