leviathan README

The project's README from github.com/elstongun/leviathan. As one markdown file: README.md.

Leviathan

Deep memory for agents over large datasets.
Index any table, export or log once; your agent gets the few records that answer the question.

CI License: Apache-2.0 Rust 1.88+ MCP optional

Leviathan is a single binary that turns records (JSONL, JSON, CSV/TSV, SQLite, or any database CLI's output) into a ranked full-text index. Agents ask in plain words and get short, cited result cards: ~450 tokens per answer at any dataset size, instead of grepping and reading raw history.

Median tokens per question at 1M records: Leviathan 436, grep entity + question words 107K, grep entity history 209K, read all 203M

Median tokens per question vs dataset size Answer rate vs dataset size

Tokens per question vs entity history size Latency vs dataset size

At 1M records (678 MB) Leviathan best grep strategy
Median tokens per question 436 107,122 (245×)
Relevant record returned 99.0% top 5 · 98.5% rank 1 96.0% within a 30K-char output
Worst case (1,200 questions) 602 tokens 9.7M tokens
Median latency 33 ms 92 ms

Measured on a synthetic maintenance log (one example dataset; nothing in Leviathan is specific to it). Methodology, all six scales and caveats: docs/BENCHMARKS.md.

Quickstart

cargo install leviathan-index       # or: cargo install --git https://github.com/elstongun/leviathan
cd examples/tickets && leviathan index
leviathan search -g acme "sso login loop after password reset"

leviathan search · customer C-ACME "Acme Corp" (7 tickets) · query "sso login loop after password reset" · shown 3 of 3 · 24 tickets indexed
[1] T-1001 · 2024-01-08 09:12 · rel 16.9
  Login loops back to sign-in page after password reset
  status: closed · priority: high
  resolution: Cleared stale session cookies on password reset; shipped in 4.2.1. Workaround: clear site data.
  match: Users who reset their password get redirected to the sign-in page again in an endless loop.
...

Prebuilt binaries are on Releases. No runtime dependencies; the index is one SQLite file.

Map your data

Fields are paths (a.b, items[].name); only id is required.

Field Enables
id get, upsert, delete, citations
title / text Card headline (weighted 2×) / searched text (default: all strings)
group / group_name -g scoped search, name resolution, labeled fallback
date --since, --until, recent
filters / display --where facets / fields shown on cards
empty_values / rank.boost Placeholders treated as missing / favor complete records 
leviathan init ./export                       # infer a commented leviathan.toml
leviathan index tickets.csv --id "Ticket ID" --group customer_id --date created_at   # or flags
leviathan index -c tickets.toml               # or a config your agent wrote from the schema
leviathan describe                            # fields, groups, filter values, example calls

Any database works through its own CLI; Leviathan never holds credentials:

psql "$DATABASE_URL" -At -c "SELECT row_to_json(t) FROM tickets t" | leviathan index - -c tickets.toml
leviathan index app.db --sql "SELECT * FROM tickets" -c tickets.toml
duckdb -json -c "SELECT * FROM 'events/*.parquet'" | leviathan index - -c events.toml

Builds are atomic and skipped when nothing changed; upsert and delete keep an index fresh. Full reference: docs/CONFIG.md.

Plug it into your agent

CLI + skill (recommended): any agent with a shell can call it. Copy skills/leviathan into ~/.claude/skills/ or paste it into AGENTS.md / .cursor/rules. Costs 0 tokens until used.

MCP (optional): leviathan mcp serves four read-only stdio tools (search, resolve_group, get, describe) whose descriptions include a summary of your dataset (~640 tokens per session). leviathan wrap <claude|cursor|codex|vscode|gemini|windsurf|generic> prints the config.

Commands

Command Does
init / index / upsert / delete Propose a mapping / build / update / remove records
search [-g G] [words] Ranked records (--scope, --where, --since, --until, -n, --offset)
recent [-g G] · resolve <g> · get <id…> · describe Newest · group candidates · full records · index summary
mcp · wrap <agent> MCP server · agent config

Global: --index PATH, --json, --max-chars N. Exit codes: 0 ok (zero hits included), 1 error, 2 bad request, 3 group unknown/ambiguous.

How it works

Records stream into one SQLite file with an FTS5 index. A search resolves the group (exact → name → contains → fuzzy), runs one FTS5 match where group and filter values are indexed tokens (no post-filtering), ranks by BM25 × boosts, and decodes only the top N into capped cards. A record's own group name is excluded from scoped matching, placeholders count as missing, and every answer reports shown N of M so agents can tell "no match" from "no data".

Reproduce the benchmark

python3 -m venv bench/.venv && bench/.venv/bin/pip install -r bench/requirements.txt
bench/.venv/bin/python bench/run_bench.py && bench/.venv/bin/python bench/report.py   # ~30 min, ~4 GB

Index build time and size

Contributing, security, license

See CONTRIBUTING.md (ranking changes need before/after benchmark numbers) and SECURITY.md (read-only, offline, unsafe-free; report privately). Licensed under Apache-2.0; dependencies in THIRD_PARTY.md.

leviathan (elstongun/leviathan)

full-text-search
Searching Database…

831 ·GitHub
Leviathan
What is this?Leviathan turns your records (JSONL, JSON, CSV/TSV, SQLite, or anything a database CLI can export) into a single-file, ranked full-text search index so an AI agent gets the few records that answer a question instead of grepping through raw history.
Is this a database, or does it sit on top of data I already have?It is not a database itself — it builds one SQLite file with an FTS5 index from data you export, and any source database works as long as its own CLI can dump records; Leviathan never touches or stores the original database's credentials.
How do I get it running?Grab a prebuilt binary from GitHub Releases, map your fields (only id is required) with init, build the index, then query it with search.
Do I need to run a server or any other service alongside it?No — there are no runtime dependencies, the entire index lives in a single SQLite file, and Leviathan just reads whatever your own database CLI exports.
What happens if I search for an unknown or ambiguous group?Leviathan never guesses: unknown or ambiguous -g values list the matching candidates and exit with code 3 instead of silently picking one.
Does my data or source-database credentials ever leave my machine?Leviathan is built to be read-only and offline, operating on a local SQLite index it builds itself, and it never holds the credentials of the database it exported from.
How fast and token-efficient is it compared to grep?On a synthetic 1M-record benchmark, Leviathan answers questions with a median of 436 tokens and 33ms latency, versus about 107,000 tokens and 92ms for the best grep strategy, while returning the relevant record in the top 5 results 99% of the time.
What does a search result look like?Searches return short, cited result cards (about 450 tokens per answer no matter the dataset size) ranked by BM25 with field boosts, and each one reports 'shown N of M' so an agent can tell no-match from no-data.
Can I update the index without rebuilding it from scratch?Yes, upsert and delete update records in place, and a full build is atomic and skipped entirely if nothing changed since the last run.
What is the MCP mode for?Running leviathan mcp exposes four read-only stdio tools (search, resolvegroup, get, describe) for MCP-compatible agents, at a one-time cost of about 640 tokens per session to describe the dataset.
What are similar projects?xerj, okf-agent-memory, codify, graphicle, coral, zvec, turnstone, sgrep
Which topics is it in?bm25, agent-tools, full-text-search, search-engine, no-dependencies, knowledge-management, agent-memory, model-context-protocol, cli-tool, offline, sqlite, agents
Creator
elstongun
Size
1.8 MB
Stars
831
Updated
CommitOct 09Releasev0.2.0Oct 09
Age
12d
Commits · 90 Days