Gaussian_Clef_RAG README

项目的 README,来自 github.com/myneace/Gaussian_Clef_RAG。合并为一个 markdown 文件:README.md。

Manifold — GaussianRAG

Manifold — Gaussian-Clef RAG

Probabilistic retrieval-augmented generation on statistical manifolds with sub-100ms Cloudflare Clef decision routing, Woodbury-accelerated geometric kernels, and universal corpus generation.

Every chunk of knowledge is a Gaussian distribution, not a flat vector. Retrieval is information geometry, accelerated to production speed.

Python 3.10+ License: AGPL-3.0 Type Checked Tests


⚡ Quick Start (3 Lines)

Install as a standalone library:

pip install -e .

Instantiate and search in Python — runs 100% locally with zero API keys required:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()
rag.add_texts([
    "Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact.",
    "Trust the long cold rise. A dough that spends a night in the fridge bakes better than one you rush out in the afternoon.",
    "Handle the dough gently. Every bubble you crush in the final shape was made by hours of careful fermentation.",
])

results = rag.search("Why is my loaf coming out dense?", top_k=3)
for chunk in results:
    print(f"[{chunk.rank}] {chunk.knowledge.text} (confidence: {chunk.confidence:.3f})")

Ask questions with grounded answer synthesis (optional OpenAI, Gemini, or Mistral provider):

answer = rag.query("How long should I let the dough rise in the fridge?")
print(answer)

🚀 Key Innovations & Architecture

Gaussian-Clef RAG transforms probabilistic retrieval from a multi-second research prototype into a production-grade sub-100ms retrieval engine by combining Information Geometry with non-autoregressive decision intelligence:

Pillar Capability Technical Mechanism Latency Impact
Woodbury ELK Acceleration $O(d \cdot r^2)$ kernel evaluation Uses low-rank matrix inversion lemma & Sylvester determinant theorem for Expected Likelihood Kernel 150ms → <3ms
Cloudflare Clef Decision Routing Non-autoregressive ambiguity gating Replaces slow autoregressive query dissolution with typed noul and choice questions via @cf/cloudflare/clef-flash 1,500ms → <40ms
Native Dual-Store Ingestion Lockstep hybrid lexical-vector storage Automatically indexes into dense covariance KnowledgeStore and embedded SQLite FTS5 LexicalStore simultaneously Sub-2ms candidate search
Taxonomy Branch Pruning Search space reduction Hierarchical document clustering prunes 60–80%+ irrelevant candidate branches prior to reranking Eliminates 80% matrix ops
Clef 4-Question Passage Gate Noise, hallucination, and contradiction filter Evaluates candidate chunks in parallel batches for relevance, evidence, and contradictions Prunes 30 → top 5–8 survivors
Universal Ingestion Engine Format-agnostic text processing Natively ingests raw text, Markdown heading hierarchies, JSON/JSONL records, CSV tables, and directories Zero pre-processing boilerplate
Flexible Corpus Generator Unstructured text normalization & caching Cleans dirty HTML, OCR line breaks, timestamps, and adaptively structures raw text with content-addressed cache Instant clean corpus creation

📐 End-to-End Pipeline Architecture

Manifold Pipeline Architecture


🧹 Flexible Corpus Generator & Raw Text Normalization

The gaussian_rag.creator package features an intelligent, multi-stage text normalization and adaptive corpus generation engine. It processes messy, unstructured pasted text (HTML/web clippings, OCR/PDF line breaks, chat transcripts, unformatted text blobs) into structured corpora with content-addressed caching.

1. Command-Line Interface (data/generate_corpus.py)

Generate benchmark corpora or process arbitrary raw text from files or stdin:

# 1. Structure the bundled 12 Laws of Good Bread benchmark corpus
python data/generate_corpus.py -i benchmarks/bakers_handbook.md -o benchmarks/bakers_handbook.jsonl

# 2. Pipe messy, unformatted text from stdin directly into structured JSONL
cat raw_notes.txt | python data/generate_corpus.py --stdin -o corpus.jsonl

# 3. Clean messy OCR scan text into structured Markdown with hierarchy
python data/generate_corpus.py -i dirty_scan.txt -o clean_corpus.md -f markdown

# 4. Pass raw text directly via argument
python data/generate_corpus.py -t "Law 1: Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact." -o bread.jsonl

# 5. Optional AI-assisted normalization using OpenAI-compatible endpoints
python data/generate_corpus.py -i ocr_scan.txt -o corpus.jsonl --ai

2. Programmatic Python Ingestion of Dirty Text

Ingest unformatted, messy text directly into your RAG instance with automatic HTML stripping, OCR hyphenation fixing, and structural detection:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()

# Raw messy HTML/OCR text
dirty_text = """
<p>Law 1: Weigh, Don't Scoop<br/>
A cup of flour is a guess; 125 grams is a fact. Scooping from the bag
can pack in fifty extra grams with-
out you noticing.</p>

<p>Law 4: Trust The Long Cold Rise<br/>
A dough that spends a night in the fridge bakes better than one you rush.</p>
"""

# Normalizes, structures, embeds as Gaussians, and updates both stores in 1 call:
rag.add_raw_text(dirty_text, title="Baker's Handbook Excerpt")

results = rag.search("how much flour should I use?")
print(results[0].knowledge.text)

📦 Universal Format-Agnostic Ingestion

The UniversalIngestor natively parses, chunks, embeds, and dual-indexes arbitrary text formats into KnowledgeStore and SQLite FTS5:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()

# 1. Ingest Raw Strings
rag.add_texts([
    "Passage 1 content...",
    "Passage 2 content..."
], metadata=[{"source": "memo1"}, {"source": "memo2"}])

# 2. Ingest Markdown with Heading Hierarchy (Breadcrumbs preserved in metadata)
markdown_content = """# The 12 Laws of Good Bread
## Law 1: Weigh, Don't Scoop
A cup of flour is a guess; 125 grams is a fact.
## Law 2: Hydrate Before You Knead
Mix flour and water alone, then rest for thirty minutes.
"""
rag.add_markdown(markdown_content, document_id="bakers-handbook")

# 3. Ingest Structured JSON / JSONL Records
records = [
    {"id": "doc_101", "text": "Customer agreement policy...", "department": "Legal"},
    {"id": "doc_102", "text": "Refund processing workflow...", "department": "Billing"}
]
rag.add_records(records, text_field="text", id_field="id")

# 4. Ingest Files (.txt, .md, .json, .jsonl, .csv)
rag.add_file("path/to/handbook.md")
rag.add_file("path/to/dataset.csv")

# 5. Ingest Entire Directory Trees Recursively
rag.add_directory("path/to/knowledge_base/")

💾 Persistence (Save & Load)

Save both dense covariance representations and SQLite FTS5 lexical indexes to disk with full cross-platform handle cleanup:

# Save knowledge store + lexical index
rag.save("my_knowledge_store")

# Load existing store from disk in 1 line
from gaussian_rag import GaussianRAG

loaded_rag = GaussianRAG.load("my_knowledge_store")
results = loaded_rag.search("cold fermentation and oven steam", top_k=5)

🔬 Mathematical Foundations

1. Multivariate Gaussian Representation

Every chunk $c$ is mapped to a probability distribution $\mathcal{N}(\mu_c, \Sigma_c)$ living on the statistical manifold $\mathbb{R}^d \times \text{SPD}(d)$:

$$\mathcal{N}(\mu_c, \Sigma_c), \quad \mu_c \in \mathbb{R}^d, \quad \Sigma_c \in \text{SPD}(d)$$

$$\Sigma = \text{diag}(v) + L L^T, \quad v \in \mathbb{R}^d_+, \quad L \in \mathbb{R}^{d \times r}, \quad r = 8$$

2. Woodbury-Accelerated Expected Likelihood Kernel (ELK)

The canonical overlap score evaluates how much probability mass the query distribution places on the document distribution:

$$\text{Score}(Q, D) = \int \mathcal{N}(x; \mu_q, \Sigma_q) \mathcal{N}(x; \mu_d, \Sigma_d) , dx = \mathcal{N}(\mu_q; \mu_d, \Sigma_q + \Sigma_d)$$

Using the Woodbury Matrix Identity and Sylvester Determinant Theorem, inversion of $(\Sigma_q + \Sigma_d) \in \mathbb{R}^{d \times d}$ runs in $O(d \cdot r^2)$ instead of $O(d^3)$:

$$(\text{diag}(v) + U V^T)^{-1} = \text{diag}(v)^{-1} - \text{diag}(v)^{-1} U (I_k + V^T \text{diag}(v)^{-1} U)^{-1} V^T \text{diag}(v)^{-1}$$

$$\det(\text{diag}(v) + U V^T) = \det(\text{diag}(v)) \det(I_k + V^T \text{diag}(v)^{-1} U)$$

This achieves sub-3ms reranking across dozens of candidate passages.


🛠️ CLI Usage & Evaluation

Run End-to-End Evaluation on The 12 Laws of Good Bread

Validates live Clef disambiguation, taxonomy pruning, FTS5 pre-filtering, passage gating, and exact-match retrieval:

python test_bakers_handbook_e2e.py

Ingest and Query via CLI

# Ingest documents into dual-store
python -m gaussian_rag.main ingest --input data/my_docs --output .my_store

# Query store with ELK metric
python -m gaussian_rag.main query --text "Why did my crust set before the loaf finished rising?" --store .my_store --metric elk

Run Latency & Benchmark Verification

Proves sub-100ms retrieval latency across pipeline stages:

pytest tests/test_benchmark_latency.py -v

Run Test Suites

# Run corpus generator tests
pytest tests/test_corpus_generator.py -v

# Run facade & universal ingestion tests
pytest tests/test_library_facade.py -v

# Run dual-store synchronization tests
pytest tests/test_dual_store_sync.py -v

# Run full test suite
pytest

🌐 3D Manifold Dashboard & Telemetry Server

Gaussian RAG includes a real-time 3D visualization dashboard with live stage latency telemetry:

# Start FastAPI backend (port 8001)
python dashboard_server.py

# In another terminal, start the Vite frontend (port 5173)
cd dashboard
npm install
npm run dev

Visit http://localhost:5173 to explore:


⚙️ Configuration & Environment Variables

Copy .env.example to .env:

# Optional Cloudflare Workers AI (Enables Clef decision routing & passage gating)
CLOUDFLARE_ACCOUNT_ID="your_account_id"
CLOUDFLARE_API_TOKEN="your_api_token"

# Optional Cloud LLM Generation (OpenAI / Gemini / Mistral)
GEMINI_API_KEY="your_gemini_key"
OPENAI_API_KEY="your_openai_key"
MISTRAL_API_KEY="your_mistral_key"

Configure core parameters in config/config.yaml:

encoder:
  embedding_backend: "local"          # "local" (all-MiniLM-L6-v2) | "gemini_cloud"
  dimension: 384
  sigma_mode: "heuristic"             # "heuristic" | "learned" | "dissolver" | "entity_aware"
  low_rank: 8

retrieval:
  metric: "elk"                       # "elk" | "wasserstein2" | "fisher_rao"
  top_k: 5
  ann_candidates: 15

chunking:
  max_tokens: 128
  overlap_tokens: 16

📚 Public Python API Reference

import gaussian_rag as gr

# 1. High-Level Facade
rag = gr.GaussianRAG.create(
    dimension=384,
    use_clef=False,           # set True to enable Cloudflare Clef passage gating
    use_generator=False       # set True to enable LLM response generation
)

# 2. Universal Ingestion & Normalization
normalizer = gr.TextNormalizer()
clean_text = normalizer.clean("<p>Some raw text...</p>")

generator = gr.CorpusGenerator()
documents = generator.generate(clean_text, title="My Corpus")

cache = gr.CorpusCache()
assistant = gr.OpenAICorpusAssistant()

# 3. Core Components
embedder = gr.GaussianEmbedder(sigma_mode="heuristic", rank=8)
store = gr.KnowledgeStore()
lexical = gr.LexicalStore()
retriever = gr.GaussianRetriever(store=store)
ingestor = gr.UniversalIngestor(embedder, store)
router = gr.TaxonomyRouter()

# 4. Data Models
from gaussian_rag import (
    GaussianKnowledge,        # 𝒩(μ, Σ) node with metadata
    QueryRepresentation,      # Query mean + covariance
    RetrievedChunk,           # Ranked chunk with confidence score & rank
    CorpusDocument,           # Cleaned document with structure metadata
)

📖 Complete Documentation Index

For in-depth guides and architectural blueprints, explore the docs/ directory:


📜 License

GNU Affero General Public License v3.0 (AGPL-3.0). See LICENSE for details.

Gaussian_Clef_RAG (myneace/Gaussian_Clef_RAG)

正在搜索数据库…

这是什么?Gaussian-Clef RAG 是一个检索增强生成(RAG)库。它不使用平面向量,而是把每个知识片段表示为统计流形上的高斯分布,并借助信息几何进行检索,而不是简单的余弦相似度。
这是我可以导入的库,还是独立应用?它是一个可以安装并在代码中实例化的独立 Python 库(gaussianrag 包)。此外还提供用于评估和摄入的 CLI 命令行工具,以及一个可选的 3D 仪表板服务器。
我该怎么让它跑起来?将它作为独立库安装,然后直接在 Python 中实例化检索器并进行搜索。它完全在本地运行,无需任何 API 密钥。
我需要 OpenAI 或 Gemini 的 API 密钥吗?不需要。核心检索完全在本地运行,无需任何 API 密钥。只有在需要基于依据的答案合成时,才可以选择 OpenAI、Gemini 或 Mistral 提供商。Cloudflare 的 Clef 模型则用于快速决策路由。
检索是否依赖 Cloudflare 的 Clef 服务?是的。快速消歧步骤(将查询拆解为有类型的是非题和选择题)通过 Cloudflare 的 @cf/cloudflare/clef-flash 模型路由,而不是使用较慢的本地自回归步骤。
这是 AGPL 许可证吗?这会影响我的使用方式吗?是的,它采用 AGPL-3.0 许可证,这意味着任何基于它构建的网络服务,都必须向用户公开其源代码。
Woodbury 加速在这里为何重要?它借助 Woodbury 矩阵恒等式和 Sylvester 行列式定理,把高斯重叠核的计算从 O(d3) 的矩阵求逆降低为 O(d·r2),将重排序延迟从约 150 毫秒缩短到 3 毫秒以内。
我可以摄入哪些文件格式?通用摄入引擎原生支持解析纯文本、Markdown 标题层级、JSON/JSONL、CSV 表格以及整个目录,无需任何预处理。
3D 仪表板显示什么?一个实时仪表板,包含经 PCA 投影的高斯椭球可视化、通过服务器发送事件(SSE)推送的各阶段实时延迟遥测数据,以及分类体系聚类和 Clef 决策追踪记录。
它如何存储数据以便检索?它同时写入稠密协方差 KnowledgeStore 和嵌入式 SQLite FTS5 LexicalStore 两个索引,结合几何搜索与词法搜索,实现低于 2 毫秒的候选检索。
有哪些类似的项目?FlagEmbedding, m_flow, weaviate, memvid, xerj, txtai, carrel, serviette
作者
myneace
大小
2.5 MB
星标
169
更新
提交Nov 01
时长
5个月
提交 · 90 天