README проекта с github.com/myneace/Gaussian_Clef_RAG. Одним файлом markdown: README.md.

Probabilistic retrieval-augmented generation on statistical manifolds with sub-100ms Cloudflare Clef decision routing, Woodbury-accelerated geometric kernels, and universal corpus generation.
Every chunk of knowledge is a Gaussian distribution, not a flat vector. Retrieval is information geometry, accelerated to production speed.
Install as a standalone library:
pip install -e .
Instantiate and search in Python — runs 100% locally with zero API keys required:
from gaussian_rag import GaussianRAG
rag = GaussianRAG.create()
rag.add_texts([
"Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact.",
"Trust the long cold rise. A dough that spends a night in the fridge bakes better than one you rush out in the afternoon.",
"Handle the dough gently. Every bubble you crush in the final shape was made by hours of careful fermentation.",
])
results = rag.search("Why is my loaf coming out dense?", top_k=3)
for chunk in results:
print(f"[{chunk.rank}] {chunk.knowledge.text} (confidence: {chunk.confidence:.3f})")
Ask questions with grounded answer synthesis (optional OpenAI, Gemini, or Mistral provider):
answer = rag.query("How long should I let the dough rise in the fridge?")
print(answer)
Gaussian-Clef RAG transforms probabilistic retrieval from a multi-second research prototype into a production-grade sub-100ms retrieval engine by combining Information Geometry with non-autoregressive decision intelligence:
| Pillar | Capability | Technical Mechanism | Latency Impact |
|---|---|---|---|
| Woodbury ELK Acceleration | $O(d \cdot r^2)$ kernel evaluation | Uses low-rank matrix inversion lemma & Sylvester determinant theorem for Expected Likelihood Kernel | 150ms → <3ms |
| Cloudflare Clef Decision Routing | Non-autoregressive ambiguity gating | Replaces slow autoregressive query dissolution with typed noul and choice questions via @cf/cloudflare/clef-flash |
1,500ms → <40ms |
| Native Dual-Store Ingestion | Lockstep hybrid lexical-vector storage | Automatically indexes into dense covariance KnowledgeStore and embedded SQLite FTS5 LexicalStore simultaneously |
Sub-2ms candidate search |
| Taxonomy Branch Pruning | Search space reduction | Hierarchical document clustering prunes 60–80%+ irrelevant candidate branches prior to reranking | Eliminates 80% matrix ops |
| Clef 4-Question Passage Gate | Noise, hallucination, and contradiction filter | Evaluates candidate chunks in parallel batches for relevance, evidence, and contradictions | Prunes 30 → top 5–8 survivors |
| Universal Ingestion Engine | Format-agnostic text processing | Natively ingests raw text, Markdown heading hierarchies, JSON/JSONL records, CSV tables, and directories | Zero pre-processing boilerplate |
| Flexible Corpus Generator | Unstructured text normalization & caching | Cleans dirty HTML, OCR line breaks, timestamps, and adaptively structures raw text with content-addressed cache | Instant clean corpus creation |

The gaussian_rag.creator package features an intelligent, multi-stage text normalization and adaptive corpus generation engine. It processes messy, unstructured pasted text (HTML/web clippings, OCR/PDF line breaks, chat transcripts, unformatted text blobs) into structured corpora with content-addressed caching.
data/generate_corpus.py)Generate benchmark corpora or process arbitrary raw text from files or stdin:
# 1. Structure the bundled 12 Laws of Good Bread benchmark corpus
python data/generate_corpus.py -i benchmarks/bakers_handbook.md -o benchmarks/bakers_handbook.jsonl
# 2. Pipe messy, unformatted text from stdin directly into structured JSONL
cat raw_notes.txt | python data/generate_corpus.py --stdin -o corpus.jsonl
# 3. Clean messy OCR scan text into structured Markdown with hierarchy
python data/generate_corpus.py -i dirty_scan.txt -o clean_corpus.md -f markdown
# 4. Pass raw text directly via argument
python data/generate_corpus.py -t "Law 1: Weigh, don't scoop. A cup of flour is a guess; 125 grams is a fact." -o bread.jsonl
# 5. Optional AI-assisted normalization using OpenAI-compatible endpoints
python data/generate_corpus.py -i ocr_scan.txt -o corpus.jsonl --ai
Ingest unformatted, messy text directly into your RAG instance with automatic HTML stripping, OCR hyphenation fixing, and structural detection:
from gaussian_rag import GaussianRAG
rag = GaussianRAG.create()
# Raw messy HTML/OCR text
dirty_text = """
<p>Law 1: Weigh, Don't Scoop<br/>
A cup of flour is a guess; 125 grams is a fact. Scooping from the bag
can pack in fifty extra grams with-
out you noticing.</p>
<p>Law 4: Trust The Long Cold Rise<br/>
A dough that spends a night in the fridge bakes better than one you rush.</p>
"""
# Normalizes, structures, embeds as Gaussians, and updates both stores in 1 call:
rag.add_raw_text(dirty_text, title="Baker's Handbook Excerpt")
results = rag.search("how much flour should I use?")
print(results[0].knowledge.text)
The UniversalIngestor natively parses, chunks, embeds, and dual-indexes arbitrary text formats into KnowledgeStore and SQLite FTS5:
from gaussian_rag import GaussianRAG
rag = GaussianRAG.create()
# 1. Ingest Raw Strings
rag.add_texts([
"Passage 1 content...",
"Passage 2 content..."
], metadata=[{"source": "memo1"}, {"source": "memo2"}])
# 2. Ingest Markdown with Heading Hierarchy (Breadcrumbs preserved in metadata)
markdown_content = """# The 12 Laws of Good Bread
## Law 1: Weigh, Don't Scoop
A cup of flour is a guess; 125 grams is a fact.
## Law 2: Hydrate Before You Knead
Mix flour and water alone, then rest for thirty minutes.
"""
rag.add_markdown(markdown_content, document_id="bakers-handbook")
# 3. Ingest Structured JSON / JSONL Records
records = [
{"id": "doc_101", "text": "Customer agreement policy...", "department": "Legal"},
{"id": "doc_102", "text": "Refund processing workflow...", "department": "Billing"}
]
rag.add_records(records, text_field="text", id_field="id")
# 4. Ingest Files (.txt, .md, .json, .jsonl, .csv)
rag.add_file("path/to/handbook.md")
rag.add_file("path/to/dataset.csv")
# 5. Ingest Entire Directory Trees Recursively
rag.add_directory("path/to/knowledge_base/")
Save both dense covariance representations and SQLite FTS5 lexical indexes to disk with full cross-platform handle cleanup:
# Save knowledge store + lexical index
rag.save("my_knowledge_store")
# Load existing store from disk in 1 line
from gaussian_rag import GaussianRAG
loaded_rag = GaussianRAG.load("my_knowledge_store")
results = loaded_rag.search("cold fermentation and oven steam", top_k=5)
Every chunk $c$ is mapped to a probability distribution $\mathcal{N}(\mu_c, \Sigma_c)$ living on the statistical manifold $\mathbb{R}^d \times \text{SPD}(d)$:
$$\mathcal{N}(\mu_c, \Sigma_c), \quad \mu_c \in \mathbb{R}^d, \quad \Sigma_c \in \text{SPD}(d)$$
SentenceTransformer or Gemini Cloud embeddings.$$\Sigma = \text{diag}(v) + L L^T, \quad v \in \mathbb{R}^d_+, \quad L \in \mathbb{R}^{d \times r}, \quad r = 8$$
The canonical overlap score evaluates how much probability mass the query distribution places on the document distribution:
$$\text{Score}(Q, D) = \int \mathcal{N}(x; \mu_q, \Sigma_q) \mathcal{N}(x; \mu_d, \Sigma_d) , dx = \mathcal{N}(\mu_q; \mu_d, \Sigma_q + \Sigma_d)$$
Using the Woodbury Matrix Identity and Sylvester Determinant Theorem, inversion of $(\Sigma_q + \Sigma_d) \in \mathbb{R}^{d \times d}$ runs in $O(d \cdot r^2)$ instead of $O(d^3)$:
$$(\text{diag}(v) + U V^T)^{-1} = \text{diag}(v)^{-1} - \text{diag}(v)^{-1} U (I_k + V^T \text{diag}(v)^{-1} U)^{-1} V^T \text{diag}(v)^{-1}$$
$$\det(\text{diag}(v) + U V^T) = \det(\text{diag}(v)) \det(I_k + V^T \text{diag}(v)^{-1} U)$$
This achieves sub-3ms reranking across dozens of candidate passages.
Validates live Clef disambiguation, taxonomy pruning, FTS5 pre-filtering, passage gating, and exact-match retrieval:
python test_bakers_handbook_e2e.py
# Ingest documents into dual-store
python -m gaussian_rag.main ingest --input data/my_docs --output .my_store
# Query store with ELK metric
python -m gaussian_rag.main query --text "Why did my crust set before the loaf finished rising?" --store .my_store --metric elk
Proves sub-100ms retrieval latency across pipeline stages:
pytest tests/test_benchmark_latency.py -v
# Run corpus generator tests
pytest tests/test_corpus_generator.py -v
# Run facade & universal ingestion tests
pytest tests/test_library_facade.py -v
# Run dual-store synchronization tests
pytest tests/test_dual_store_sync.py -v
# Run full test suite
pytest
Gaussian RAG includes a real-time 3D visualization dashboard with live stage latency telemetry:
# Start FastAPI backend (port 8001)
python dashboard_server.py
# In another terminal, start the Vite frontend (port 5173)
cd dashboard
npm install
npm run dev
Visit http://localhost:5173 to explore:
Copy .env.example to .env:
# Optional Cloudflare Workers AI (Enables Clef decision routing & passage gating)
CLOUDFLARE_ACCOUNT_ID="your_account_id"
CLOUDFLARE_API_TOKEN="your_api_token"
# Optional Cloud LLM Generation (OpenAI / Gemini / Mistral)
GEMINI_API_KEY="your_gemini_key"
OPENAI_API_KEY="your_openai_key"
MISTRAL_API_KEY="your_mistral_key"
Configure core parameters in config/config.yaml:
encoder:
embedding_backend: "local" # "local" (all-MiniLM-L6-v2) | "gemini_cloud"
dimension: 384
sigma_mode: "heuristic" # "heuristic" | "learned" | "dissolver" | "entity_aware"
low_rank: 8
retrieval:
metric: "elk" # "elk" | "wasserstein2" | "fisher_rao"
top_k: 5
ann_candidates: 15
chunking:
max_tokens: 128
overlap_tokens: 16
import gaussian_rag as gr
# 1. High-Level Facade
rag = gr.GaussianRAG.create(
dimension=384,
use_clef=False, # set True to enable Cloudflare Clef passage gating
use_generator=False # set True to enable LLM response generation
)
# 2. Universal Ingestion & Normalization
normalizer = gr.TextNormalizer()
clean_text = normalizer.clean("<p>Some raw text...</p>")
generator = gr.CorpusGenerator()
documents = generator.generate(clean_text, title="My Corpus")
cache = gr.CorpusCache()
assistant = gr.OpenAICorpusAssistant()
# 3. Core Components
embedder = gr.GaussianEmbedder(sigma_mode="heuristic", rank=8)
store = gr.KnowledgeStore()
lexical = gr.LexicalStore()
retriever = gr.GaussianRetriever(store=store)
ingestor = gr.UniversalIngestor(embedder, store)
router = gr.TaxonomyRouter()
# 4. Data Models
from gaussian_rag import (
GaussianKnowledge, # 𝒩(μ, Σ) node with metadata
QueryRepresentation, # Query mean + covariance
RetrievedChunk, # Ranked chunk with confidence score & rank
CorpusDocument, # Cleaned document with structure metadata
)
For in-depth guides and architectural blueprints, explore the docs/ directory:
CODEBASE.md) — Module hierarchy and dependency mapUSAGE.md) — Comprehensive developer workflows, CLI, and API examplesPIPELINE.md) — Step-by-step description of the 6-stage retrieval and dual-store ingestion pipelinesDECISIONS.md) — Architectural decision records (DD-01 through DD-14)THEORY.md) — Mathematical formulation of statistical manifolds and ELKABOUT.md) — Learned covariance estimation and continual learningconcepts/) — Multi-agent disambiguation and polysemy specificationsGNU Affero General Public License v3.0 (AGPL-3.0). See LICENSE for details.