Gaussian_Clef_RAG README

README проекта с github.com/myneace/Gaussian_Clef_RAG. Одним файлом markdown: README.md.

Manifold — GaussianRAG

Manifold — Gaussian-Clef RAG

Probabilistic retrieval-augmented generation on statistical manifolds with sub-100ms Cloudflare Clef decision routing, Woodbury-accelerated geometric kernels, and universal corpus generation.

Every chunk of knowledge is a Gaussian distribution, not a flat vector. Retrieval is information geometry, accelerated to production speed.

Python 3.10+ License: AGPL-3.0 Type Checked Tests


⚡ Quick Start (3 Lines)

Install as a standalone library:

pip install -e .

Instantiate and search in Python — runs 100% locally with zero API keys required:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()
rag.add_texts([
    "Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact.",
    "Trust the long cold rise. A dough that spends a night in the fridge bakes better than one you rush out in the afternoon.",
    "Handle the dough gently. Every bubble you crush in the final shape was made by hours of careful fermentation.",
])

results = rag.search("Why is my loaf coming out dense?", top_k=3)
for chunk in results:
    print(f"[{chunk.rank}] {chunk.knowledge.text} (confidence: {chunk.confidence:.3f})")

Ask questions with grounded answer synthesis (optional OpenAI, Gemini, or Mistral provider):

answer = rag.query("How long should I let the dough rise in the fridge?")
print(answer)

🚀 Key Innovations & Architecture

Gaussian-Clef RAG transforms probabilistic retrieval from a multi-second research prototype into a production-grade sub-100ms retrieval engine by combining Information Geometry with non-autoregressive decision intelligence:

Pillar Capability Technical Mechanism Latency Impact
Woodbury ELK Acceleration $O(d \cdot r^2)$ kernel evaluation Uses low-rank matrix inversion lemma & Sylvester determinant theorem for Expected Likelihood Kernel 150ms → <3ms
Cloudflare Clef Decision Routing Non-autoregressive ambiguity gating Replaces slow autoregressive query dissolution with typed noul and choice questions via @cf/cloudflare/clef-flash 1,500ms → <40ms
Native Dual-Store Ingestion Lockstep hybrid lexical-vector storage Automatically indexes into dense covariance KnowledgeStore and embedded SQLite FTS5 LexicalStore simultaneously Sub-2ms candidate search
Taxonomy Branch Pruning Search space reduction Hierarchical document clustering prunes 60–80%+ irrelevant candidate branches prior to reranking Eliminates 80% matrix ops
Clef 4-Question Passage Gate Noise, hallucination, and contradiction filter Evaluates candidate chunks in parallel batches for relevance, evidence, and contradictions Prunes 30 → top 5–8 survivors
Universal Ingestion Engine Format-agnostic text processing Natively ingests raw text, Markdown heading hierarchies, JSON/JSONL records, CSV tables, and directories Zero pre-processing boilerplate
Flexible Corpus Generator Unstructured text normalization & caching Cleans dirty HTML, OCR line breaks, timestamps, and adaptively structures raw text with content-addressed cache Instant clean corpus creation

📐 End-to-End Pipeline Architecture

Manifold Pipeline Architecture


🧹 Flexible Corpus Generator & Raw Text Normalization

The gaussian_rag.creator package features an intelligent, multi-stage text normalization and adaptive corpus generation engine. It processes messy, unstructured pasted text (HTML/web clippings, OCR/PDF line breaks, chat transcripts, unformatted text blobs) into structured corpora with content-addressed caching.

1. Command-Line Interface (data/generate_corpus.py)

Generate benchmark corpora or process arbitrary raw text from files or stdin:

# 1. Structure the bundled 12 Laws of Good Bread benchmark corpus
python data/generate_corpus.py -i benchmarks/bakers_handbook.md -o benchmarks/bakers_handbook.jsonl

# 2. Pipe messy, unformatted text from stdin directly into structured JSONL
cat raw_notes.txt | python data/generate_corpus.py --stdin -o corpus.jsonl

# 3. Clean messy OCR scan text into structured Markdown with hierarchy
python data/generate_corpus.py -i dirty_scan.txt -o clean_corpus.md -f markdown

# 4. Pass raw text directly via argument
python data/generate_corpus.py -t "Law 1: Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact." -o bread.jsonl

# 5. Optional AI-assisted normalization using OpenAI-compatible endpoints
python data/generate_corpus.py -i ocr_scan.txt -o corpus.jsonl --ai

2. Programmatic Python Ingestion of Dirty Text

Ingest unformatted, messy text directly into your RAG instance with automatic HTML stripping, OCR hyphenation fixing, and structural detection:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()

# Raw messy HTML/OCR text
dirty_text = """
<p>Law 1: Weigh, Don't Scoop<br/>
A cup of flour is a guess; 125 grams is a fact. Scooping from the bag
can pack in fifty extra grams with-
out you noticing.</p>

<p>Law 4: Trust The Long Cold Rise<br/>
A dough that spends a night in the fridge bakes better than one you rush.</p>
"""

# Normalizes, structures, embeds as Gaussians, and updates both stores in 1 call:
rag.add_raw_text(dirty_text, title="Baker's Handbook Excerpt")

results = rag.search("how much flour should I use?")
print(results[0].knowledge.text)

📦 Universal Format-Agnostic Ingestion

The UniversalIngestor natively parses, chunks, embeds, and dual-indexes arbitrary text formats into KnowledgeStore and SQLite FTS5:

from gaussian_rag import GaussianRAG

rag = GaussianRAG.create()

# 1. Ingest Raw Strings
rag.add_texts([
    "Passage 1 content...",
    "Passage 2 content..."
], metadata=[{"source": "memo1"}, {"source": "memo2"}])

# 2. Ingest Markdown with Heading Hierarchy (Breadcrumbs preserved in metadata)
markdown_content = """# The 12 Laws of Good Bread
## Law 1: Weigh, Don't Scoop
A cup of flour is a guess; 125 grams is a fact.
## Law 2: Hydrate Before You Knead
Mix flour and water alone, then rest for thirty minutes.
"""
rag.add_markdown(markdown_content, document_id="bakers-handbook")

# 3. Ingest Structured JSON / JSONL Records
records = [
    {"id": "doc_101", "text": "Customer agreement policy...", "department": "Legal"},
    {"id": "doc_102", "text": "Refund processing workflow...", "department": "Billing"}
]
rag.add_records(records, text_field="text", id_field="id")

# 4. Ingest Files (.txt, .md, .json, .jsonl, .csv)
rag.add_file("path/to/handbook.md")
rag.add_file("path/to/dataset.csv")

# 5. Ingest Entire Directory Trees Recursively
rag.add_directory("path/to/knowledge_base/")

💾 Persistence (Save & Load)

Save both dense covariance representations and SQLite FTS5 lexical indexes to disk with full cross-platform handle cleanup:

# Save knowledge store + lexical index
rag.save("my_knowledge_store")

# Load existing store from disk in 1 line
from gaussian_rag import GaussianRAG

loaded_rag = GaussianRAG.load("my_knowledge_store")
results = loaded_rag.search("cold fermentation and oven steam", top_k=5)

🔬 Mathematical Foundations

1. Multivariate Gaussian Representation

Every chunk $c$ is mapped to a probability distribution $\mathcal{N}(\mu_c, \Sigma_c)$ living on the statistical manifold $\mathbb{R}^d \times \text{SPD}(d)$:

$$\mathcal{N}(\mu_c, \Sigma_c), \quad \mu_c \in \mathbb{R}^d, \quad \Sigma_c \in \text{SPD}(d)$$

$$\Sigma = \text{diag}(v) + L L^T, \quad v \in \mathbb{R}^d_+, \quad L \in \mathbb{R}^{d \times r}, \quad r = 8$$

2. Woodbury-Accelerated Expected Likelihood Kernel (ELK)

The canonical overlap score evaluates how much probability mass the query distribution places on the document distribution:

$$\text{Score}(Q, D) = \int \mathcal{N}(x; \mu_q, \Sigma_q) \mathcal{N}(x; \mu_d, \Sigma_d) , dx = \mathcal{N}(\mu_q; \mu_d, \Sigma_q + \Sigma_d)$$

Using the Woodbury Matrix Identity and Sylvester Determinant Theorem, inversion of $(\Sigma_q + \Sigma_d) \in \mathbb{R}^{d \times d}$ runs in $O(d \cdot r^2)$ instead of $O(d^3)$:

$$(\text{diag}(v) + U V^T)^{-1} = \text{diag}(v)^{-1} - \text{diag}(v)^{-1} U (I_k + V^T \text{diag}(v)^{-1} U)^{-1} V^T \text{diag}(v)^{-1}$$

$$\det(\text{diag}(v) + U V^T) = \det(\text{diag}(v)) \det(I_k + V^T \text{diag}(v)^{-1} U)$$

This achieves sub-3ms reranking across dozens of candidate passages.


🛠️ CLI Usage & Evaluation

Run End-to-End Evaluation on The 12 Laws of Good Bread

Validates live Clef disambiguation, taxonomy pruning, FTS5 pre-filtering, passage gating, and exact-match retrieval:

python test_bakers_handbook_e2e.py

Ingest and Query via CLI

# Ingest documents into dual-store
python -m gaussian_rag.main ingest --input data/my_docs --output .my_store

# Query store with ELK metric
python -m gaussian_rag.main query --text "Why did my crust set before the loaf finished rising?" --store .my_store --metric elk

Run Latency & Benchmark Verification

Proves sub-100ms retrieval latency across pipeline stages:

pytest tests/test_benchmark_latency.py -v

Run Test Suites

# Run corpus generator tests
pytest tests/test_corpus_generator.py -v

# Run facade & universal ingestion tests
pytest tests/test_library_facade.py -v

# Run dual-store synchronization tests
pytest tests/test_dual_store_sync.py -v

# Run full test suite
pytest

🌐 3D Manifold Dashboard & Telemetry Server

Gaussian RAG includes a real-time 3D visualization dashboard with live stage latency telemetry:

# Start FastAPI backend (port 8001)
python dashboard_server.py

# In another terminal, start the Vite frontend (port 5173)
cd dashboard
npm install
npm run dev

Visit http://localhost:5173 to explore:


⚙️ Configuration & Environment Variables

Copy .env.example to .env:

# Optional Cloudflare Workers AI (Enables Clef decision routing & passage gating)
CLOUDFLARE_ACCOUNT_ID="your_account_id"
CLOUDFLARE_API_TOKEN="your_api_token"

# Optional Cloud LLM Generation (OpenAI / Gemini / Mistral)
GEMINI_API_KEY="your_gemini_key"
OPENAI_API_KEY="your_openai_key"
MISTRAL_API_KEY="your_mistral_key"

Configure core parameters in config/config.yaml:

encoder:
  embedding_backend: "local"          # "local" (all-MiniLM-L6-v2) | "gemini_cloud"
  dimension: 384
  sigma_mode: "heuristic"             # "heuristic" | "learned" | "dissolver" | "entity_aware"
  low_rank: 8

retrieval:
  metric: "elk"                       # "elk" | "wasserstein2" | "fisher_rao"
  top_k: 5
  ann_candidates: 15

chunking:
  max_tokens: 128
  overlap_tokens: 16

📚 Public Python API Reference

import gaussian_rag as gr

# 1. High-Level Facade
rag = gr.GaussianRAG.create(
    dimension=384,
    use_clef=False,           # set True to enable Cloudflare Clef passage gating
    use_generator=False       # set True to enable LLM response generation
)

# 2. Universal Ingestion & Normalization
normalizer = gr.TextNormalizer()
clean_text = normalizer.clean("<p>Some raw text...</p>")

generator = gr.CorpusGenerator()
documents = generator.generate(clean_text, title="My Corpus")

cache = gr.CorpusCache()
assistant = gr.OpenAICorpusAssistant()

# 3. Core Components
embedder = gr.GaussianEmbedder(sigma_mode="heuristic", rank=8)
store = gr.KnowledgeStore()
lexical = gr.LexicalStore()
retriever = gr.GaussianRetriever(store=store)
ingestor = gr.UniversalIngestor(embedder, store)
router = gr.TaxonomyRouter()

# 4. Data Models
from gaussian_rag import (
    GaussianKnowledge,        # 𝒩(μ, Σ) node with metadata
    QueryRepresentation,      # Query mean + covariance
    RetrievedChunk,           # Ranked chunk with confidence score & rank
    CorpusDocument,           # Cleaned document with structure metadata
)

📖 Complete Documentation Index

For in-depth guides and architectural blueprints, explore the docs/ directory:


📜 License

GNU Affero General Public License v3.0 (AGPL-3.0). See LICENSE for details.

Gaussian_Clef_RAG (myneace/Gaussian_Clef_RAG)

Ищем в базе…

Что это такое?Gaussian-Clef RAG — это библиотека для генерации с дополнением через поиск (RAG), которая представляет каждый фрагмент знаний не плоским вектором, а гауссовским распределением на статистическом многообразии. Для поиска она использует информационную геометрию вместо простого косинусного сходства.
Это библиотека, которую я подключаю в коде, или отдельное приложение?Это самостоятельная библиотека на Python, которую вы устанавливаете и создаёте в коде (пакет gaussianrag). Дополнительно есть CLI для оценки и загрузки данных и необязательный сервер панели 3D.
Как его запустить?Установите его как отдельную библиотеку, затем создайте поисковый модуль (retriever) и выполните поиск прямо в Python. Всё работает полностью локально, без API-ключей.
Нужен ли мне ключ API OpenAI или Gemini?Нет. Базовый поиск работает на 100% локально, без каких-либо API-ключей. Провайдеры OpenAI, Gemini или Mistral нужны только по желанию, если требуется синтез ответов с опорой на источники. Модель Clef от Cloudflare используется для быстрой маршрутизации решений.
Поиск зависит от сервиса Clef от Cloudflare?Да. Этап быстрого уточнения (разбиение запроса на типизированные вопросы «да/нет» и вопросы с выбором) направляется через модель @cf/cloudflare/clef-flash от Cloudflare, а не через более медленный локальный авторегрессионный шаг.
Это лицензия AGPL, и влияет ли она на то, как я могу это использовать?Да, библиотека распространяется по лицензии AGPL-3.0: любой сетевой сервис, построенный на её основе, также должен предоставлять пользователям свой исходный код.
Почему здесь важно ускорение по Вудбери?Оно превращает вычисление ядра перекрытия гауссиан из обращения матрицы за O(d3) в вычисление за O(d·r2) с помощью тождества Вудбери и теоремы Сильвестра об определителе, сокращая задержку переранжирования примерно с 150 мс до менее 3 мс.
Какие форматы файлов я могу загружать?Универсальный движок загрузки данных напрямую разбирает простой текст, иерархии заголовков Markdown, JSON/JSONL, таблицы CSV и целые папки без какой-либо предварительной обработки.
Что показывает 3D-панель?Панель в реальном времени с визуализацией гауссовских эллипсоидов, спроецированных методом PCA, телеметрией задержек по этапам в потоковом режиме через Server-Sent Events, а также кластеризацией таксономии и трассировкой решений Clef.
Как он хранит данные для поиска?Он одновременно индексирует данные в плотном ковариационном хранилище KnowledgeStore и во встроенном хранилище SQLite FTS5 LexicalStore, объединяя геометрический и лексический поиск для выборки кандидатов менее чем за 2 мс.
Какие есть похожие проекты?FlagEmbedding, m_flow, weaviate, memvid, xerj, txtai, carrel, serviette
Автор
myneace
Размер
2.5 MB
Звёзды
169
Обновлён
КоммитNov 01
Возраст
5 мес
Коммиты · 90 дней