llm-wiki-compiler README

README проекта с github.com/atomicstrata/llm-wiki-compiler. Одним файлом markdown: README.md.

llmwiki

CI npm docs license

On main — Leaner startup (unreleased)

These changes are merged for the next release. The published npm latest release remains 1.5.0.

New in 1.5 — Wiki behavior separated from storage.

1.5.0 is the stable release on npm latest: npm install --global llm-wiki-compiler.

What's in 1.5.0 · Changelog

New in 1.4 — Review answers before publishing.

Review generated answers and their citations before they become pages, approve a batch of pages in one operation, and trace an answer back to the pages that informed it. Release notes · Upgrade guide

New in 1.3 — A fresh look for your wiki.

Meet Scientific Clay, with soft surfaces and rounded typography, and Minimal, which follows your system’s light or dark setting. Switch instantly between four themes, including Nebula Light and Dark.

Scientific ClayMinimal
Knowledge compilation page in Scientific Clay, with soft lavender surfaces and rounded typography The same populated Knowledge compilation page in Minimal, following the system light setting

The same page in a demonstration wiki. Click either screenshot for a closer look.

Recursive source folders, path exclusions, project-specific compile instructions, and storage for larger embedding indexes.

New in 1.2 — Explore your records and their evidence.

Browse profile-defined categories, declared fields, connected records, provenance, and supporting source passages in the local viewer.

New in 1.1 — Publish and share domain templates.

Create signed template distributions, discover them through explicitly trusted catalogs, and install or update them with compatibility checks.

New in 1.0 — Configurable Lifecycle Profiles.

Build a knowledge system around the way you work. Define your records, relationships, review gates, and workflows in one validated profile. Start with AutoSci for research or Newsroom for editorial work, or create your own.

Explore Configurable Lifecycle Profiles →


What llmwiki does

Compile raw sources into an interlinked, citation-traceable markdown wiki that agents and humans can browse, query, lint, export, and reuse. The default profile preserves the classic concepts-and-queries layout; optional profiles add domain-specific types and workflows without adding domain branches to the compiler.

llmwiki implements the LLM Wiki pattern: instead of re-discovering knowledge from raw files at query time, compile it once into durable pages that accumulate structure, provenance, review state, and retrieval metadata over time.

When to use this repo

Use llmwiki when you need a persistent knowledge base from raw material:

Do not use llmwiki as a general static-site generator, a heavy ontology database, or a replacement for ad-hoc search over fast-changing raw logs. It is strongest when source knowledge is worth compiling, reviewing, and reusing.

What you get

Configurable Lifecycle Profiles (CLP)

CLP turns llmwiki's knowledge compiler into a reusable substrate for domain-specific knowledge systems. A validated .llmwiki/profile.json is the single contract for:

These rules are enforced by the runtime, not left as prompt conventions. The CLI, SDK, MCP server, viewer, context builder, lint, status, export, and OKF exchange surfaces all operate from the same profile contract. Invalid profiles and writes that bypass a declared gate fail closed.

CLP is backward-compatible by construction: a project without .llmwiki/profile.json uses the built-in default concepts-and-queries profile and preserves the pre-1.0 behavior. You can start three ways — scaffold your own profile, install a built-in or local template, or install a signed template from a trusted tap:

# author your own profile, one entity type at a time
llmwiki profile init research --entity paper

# or install a built-in or local declarative template
llmwiki template list
llmwiki template inspect autosci
llmwiki template init autosci

llmwiki profile validate
llmwiki workflow list

autosci is a practical research system with papers, ideas, experiments, manuscripts, evidence artifacts, workflows, and Crossref ingestion. newsroom applies the same generic machinery to articles, desks, bylines, and editorial workflows. Templates contain configuration and examples, never executable plugin code.

Templates can also be distributed securely. Publishers build signed, offline distributions with llmwiki template publish — Ed25519 signing, key rotation, and package revocation — and verify them with template publish verify. Consumers add explicitly trusted taps, discover and inspect signed catalogs, and install or update templates with continuity, revocation, and compatibility checks enforced under lock.

Read the CLP concept guide, follow the AutoSci research workflow, or explore the Newsroom editorial workflow.

Karpathy's LLM Wiki pattern

Andrej Karpathy described the LLM Wiki pattern as a way to turn raw material into compiled knowledge that future agents can reuse. llmwiki is a concrete compiler for that pattern.

The key shift is moving work from query time to compile time. Traditional RAG repeatedly retrieves raw chunks and asks the model to reconstruct relationships for each question. llmwiki first turns sources into typed, interlinked pages with citations, metadata, and review state. Queries, context packs, exports, and MCP tools then operate over that compiled artifact.

That makes llmwiki useful when knowledge should compound: concepts shared across sources become one page, saved answers become future context, stale pages can be detected and repaired, and agents can consume a stable evidence pack instead of re-reading the same raw files from scratch.

See docs/concepts/karpathy-pattern.mdx for the deeper explanation.

Agent decision guide

Use llmwiki for a reusable corpus of documents, research, or project knowledge. Start with wiki_status and get_context_pack when you need evidence for your own reasoning. A one-off file read or live web search does not need a wiki.

The npm package includes an Agent Skill. See the MCP setup guide for installation. The MCP Registry identifier is io.github.atomicstrata/llmwiki; the npm package is llm-wiki-compiler and the executable is llmwiki.

Choose the entry point that matches the task:

Goal Use
Create a wiki from one source and inspect it llmwiki quickstart <source>
Start a typed research or editorial project llmwiki template list, then llmwiki template init autosci|newsroom
Inspect or validate the active domain contract llmwiki profile show and llmwiki profile validate
Run a declared lifecycle workflow llmwiki workflow list, then llmwiki workflow start <id>
Write or verify a profile-declared artifact llmwiki artifact write ... and llmwiki artifact verify <ref>
Import external records through a connector llmwiki connector list, then llmwiki connector run <id> --input key=value
Add more files or URLs llmwiki ingest <url-or-file>
Compile or recompile changed sources llmwiki compile
Add project-specific writing guidance llmwiki compile --instructions ./SOUL.md; pass the option on each instructed compile
Remove a bad source and its derived pages llmwiki rm <source>
Hold generated pages for human approval llmwiki compile --review or review policy config
Ask grounded questions llmwiki query "question"
Save an answer back into the wiki llmwiki query "question" --save
Build an evidence pack for another agent llmwiki context "<task>" --json or MCP get_context_pack
Inspect the compiled knowledge base llmwiki view --open
Check broken links, citations, confidence, freshness, and quality llmwiki lint and llmwiki eval
Repair stale compiled pages llmwiki refresh --stale --dry-run, then llmwiki refresh --stale
Drive llmwiki from an agent llmwiki serve --root <project>
Drive llmwiki from TypeScript createWiki({ root })
Export for another system llmwiki export --target <format>
Export an Open Knowledge Format bundle llmwiki export --target okf --out <dir>
Import an Open Knowledge Format bundle llmwiki import --okf <dir> --dry-run, then review/approve

Quick start

npm install -g llm-wiki-compiler

export ANTHROPIC_API_KEY=sk-...
# or choose another provider:
# export LLMWIKI_PROVIDER=openai
# export OPENAI_API_KEY=sk-...

llmwiki quickstart ./notes.md
llmwiki query "what are the key ideas?"
llmwiki view --open

quickstart ingests one source, compiles pages, and opens the viewer. Inside an existing project, run llmwiki next when you want the safest next action.

To start with a domain model instead of the default concepts-and-queries layout:

mkdir research-wiki && cd research-wiki
llmwiki template inspect autosci
llmwiki template init autosci
llmwiki profile validate
llmwiki workflow list

Template installation is for a new or empty typed project. It materializes the chosen profile into .llmwiki/profile.json; normal project loading never depends on a template registry or lockfile.

Demo

Try it on any article or document:

mkdir my-wiki && cd my-wiki
llmwiki quickstart https://en.wikipedia.org/wiki/Andrej_Karpathy
llmwiki query "What terms did Andrej coin?"

The examples/basic/ directory includes a small pre-generated wiki you can inspect without an API key.

Core commands

Command What it does
llmwiki ingest <url-or-file> Fetch a URL or copy a local file into sources/.
llmwiki ingest-session <path> Import exported Claude, Codex, or Cursor sessions into sources/.
llmwiki quickstart <source> Ingest, compile, and optionally open the viewer in one step.
llmwiki compile Incrementally extract concepts and generate wiki pages.
llmwiki rm <source> [--dry-run] Delete a source and the concept pages derived exclusively from it.
llmwiki refresh --stale [--dry-run] Recompile changed owners of stale pages and clean selected orphaned ownership.
llmwiki template list|inspect|init Discover and install validated declarative profile templates.
llmwiki profile init|show|validate|diff Create a minimal profile, inspect it, validate it, or assess profile changes.
llmwiki workflow ... Discover and drive profile-declared workflows, stages, gates, and outputs locally, retaining run state between invocations.
llmwiki artifact write|verify Write trusted profile-declared artifacts and verify hash-pinned references.
llmwiki connector list|run Discover first-party connectors and stage external records for review.
llmwiki review list/show/approve/reject Inspect and manage held candidates.
llmwiki query "question" [--save] Ask questions against the compiled wiki, optionally saving the answer.
llmwiki context "<prompt>" --json Build a citation-aware evidence pack for agents.
llmwiki view [--open] Start the read-only local browser viewer.
llmwiki status [--json] Report page/source counts, stale and orphaned pages, pending work, and state health.
llmwiki lint Validate wiki structure, citations, links, metadata, and freshness.
llmwiki eval [--suite fast|full] Measure wiki quality and optional citation support.
llmwiki export --target <format> Export the wiki to portable formats, including Open Knowledge Format (okf).
llmwiki import --okf <dir> [--dry-run] [--trusted] Import an Open Knowledge Format bundle, staged for review by default.
llmwiki serve --root <dir> Start the MCP server.

Full command docs live in docs/cli/.

Open Knowledge Format

llmwiki is an Open Knowledge Format (OKF) producer and consumer. OKF is a Google Cloud initiative for sharing compiled knowledge as portable markdown files with structured frontmatter.

llmwiki export --target okf --out ./dist/okf
llmwiki import --okf ./dist/okf --dry-run
llmwiki import --okf ./dist/okf

OKF import is intentionally review-first: untrusted bundles become review candidates, not live wiki pages. The importer preserves foreign OKF metadata, stores llmwiki provenance under x-llmwiki, and re-exports imported pages honestly after local edits, including safe original nested paths.

See docs/guides/open-knowledge-format.mdx, docs/cli/export.mdx, and docs/cli/import.mdx.

What llmwiki creates

A project has raw inputs in sources/, compiled markdown in wiki/, and compiler state under .llmwiki/:

sources/
  raw source files
wiki/
  concepts/      compiled pages
  queries/       saved answers
  <entity>/      profile-declared typed pages
  graph/         typed relation and audit-event stores
  outputs/       derived workflow projections
  index.md       generated TOC
.llmwiki/
  profile.json   active domain contract
  template-lock.json  advisory install provenance
  config.json    review policy and source selection
  schema.json    page-kind/cross-link policy
  state.json     source hashes and ownership
  candidates/    held review candidates
  workflows/     signed workflow run state
  eval/          quality history and thresholds
artifacts/       hash-pinned profile-declared files and manifests
log.md           activity journal

Compiled pages are plain markdown with YAML frontmatter, plus enough metadata for agents to reason about citations, freshness, confidence, contradictions, and review state. See docs/concepts/wiki-model.mdx.

Sources are top-level Markdown files by default. Opt into nested source folders and exclusions in project config. Excluding a compiled source retires its contribution on the next ordinary compile without deleting the source file.

Agent integration

MCP

Run:

llmwiki serve --root /path/to/wiki-project

MCP clients can ingest sources, compile, query, search pages, read pages, lint, run eval, inspect status, request context packs, and exchange OKF bundles. Read-only tools work without provider credentials; LLM-backed tools validate provider credentials at call time. The run_eval tool runs its fast suite without a provider; its full suite (which LLM-judges citation support) requires one.

See docs/guides/mcp-agent-integration.mdx.

SDK

import { createWiki } from "llm-wiki-compiler";

const wiki = createWiki({ root: "/path/to/wiki-project" });
await wiki.ingest({ source: "./notes.md" });
await wiki.compile();
const answer = await wiki.query({ question: "What changed?" });

See docs/guides/sdk.mdx. llm-wiki-compiler is the supported entry point; its scoped supporting packages are implementation dependencies. The local workflow engine and an application's own coordinator are two tiers, not a migration path: llmwiki workflow runs a profile-declared workflow through local invocations with persisted state between sessions. An external coordinator calls the same SDK without using local workflow execution methods; the supporting engine dependency remains installed. See SDK package selection, SDK upgrade notes, and When to use the local engine.

Configuration

Minimum requirement: Node.js 24 or newer.

The default provider is Anthropic:

export ANTHROPIC_API_KEY=sk-...

Provider selection is environment-driven:

Provider Typical setup
Anthropic ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN
Claude Agent SDK Local Claude Code login, LLMWIKI_PROVIDER=claude-agent
OpenAI Codex CLI Local Codex login / ChatGPT subscription, LLMWIKI_PROVIDER=codex-agent; explicit embedding provider required
OpenAI-compatible LLMWIKI_PROVIDER=openai, OPENAI_API_KEY, optional OPENAI_BASE_URL
Converse-compatible gateway LLMWIKI_PROVIDER=converse, explicit LLMWIKI_MODEL, LLMWIKI_CONVERSE_BASE_URL, and LLMWIKI_CONVERSE_API_KEY; separate embedding provider required when enabled
Ollama LLMWIKI_PROVIDER=ollama, OLLAMA_HOST
GitHub Copilot LLMWIKI_PROVIDER=copilot, GITHUB_TOKEN=$(gh auth token)
Atlas Cloud LLMWIKI_PROVIDER=atlascloud, ATLASCLOUD_API_KEY
OrcaRouter LLMWIKI_PROVIDER=orcarouter, ORCAROUTER_API_KEY
Runcrate (on main) LLMWIKI_PROVIDER=runcrate, RUNCRATE_API_KEY; separate embedding provider required when enabled, or set LLMWIKI_EMBEDDINGS=off

See docs/configuration/providers.mdx and docs/configuration/environment-variables.mdx.

Quality and safety model

llmwiki is designed for auditable generated knowledge:

See docs/configuration/review-policy.mdx, docs/troubleshooting/stale-pages.mdx, and docs/guides/ci-quality-gates.mdx.

Scale and what works

llmwiki is still early software, but it is no longer a toy pipeline for a handful of notes.

The current sweet spot is a durable project or domain wiki: research folders, codebase docs, team handbooks, standards, design notes, decision logs, or curated source packs. The less ideal fit is a high-churn firehose where raw search is enough and compiled structure would go stale faster than it can be reviewed.

Documentation

The full docs site source is in docs/:

Preview the docs locally with Node 24:

cd docs
volta run --node 24 npx mint dev --port 3001

Current release

Stable release 1.5.0: The current release on the npm latest dist-tag. It ships the same code as the 1.5.0-rc.1 candidate.

npm install --global llm-wiki-compiler

The supporting packages are pinned to the same version and install with it. The MCP Registry entry io.github.atomicstrata/llmwiki is published for 1.5.0 after the npm release. Existing projects need no migration; read the upgrade notes for the prompt-version change and the opt-in Converse dependencies.

Released 1.4.2: Fixes llmwiki compile with the codex-agent provider and carries the 1.4.0 features: answer review before publishing, batch approval, extraction reuse, page-level answer citations, and the split-package SDK with matching supporting dependencies.

Experimental artifact types and workflow statuses can require source updates; read the SDK upgrade notes before upgrading from 1.3. See the 1.4.0 release summary and changelog for migration details, contributor credits, and limits.

Released 1.3.0:

See the 1.3.0 release summary and changelog for migration details, contributor credits, and limits.

Released 1.2.0:

Windows path fixes are included, but native Windows validation and CI remain outstanding. Full filesystem-confinement guarantees on Windows remain unverified. Use a supported Linux environment for untrusted projects; see #217.

Released 1.1.0:

Released 1.0.0:

See CHANGELOG.md for release history.

Companion: Atomic Memory

llmwiki and Atomic Memory are complementary open context infrastructure:

Use them independently or together. The @atomicmemory/llmwiki bridge imports llmwiki export --target json --project-id <id> as durable memory records.

Contributing

Contributions are welcome. If llmwiki is missing something you need, open an issue or PR and describe the workflow you are trying to support - need-driven improvements are often the best ones. If you want to contribute more generally, roadmap items are a good place to start. For larger changes to core compile, review, import/export, or retrieval semantics, please start with an issue or design discussion so we can align on the contract first.

For repository development, use the Bun version pinned in package.json (currently 1.4.2). Dependencies are locked in bun.lock, and tests use bun:test. Node.js 24 or newer is still required for installed-package checks.

Install dependencies and run the code-change checks from the repository root:

bun install --frozen-lockfile
bunx tsc --noEmit
bun run typecheck:tests
bun run build
bun run test
bun run fallow:ci

For packaging or export changes, also run bun run test:pack, which checks the packed packages in fresh installed consumers.

See CONTRIBUTING.md.

License

MIT

Disclaimer

No LLMs were harmed in the making of this repo.

llm-wiki-compiler (atomicstrata/llm-wiki-compiler)

Ищем в базе…

Что это такое?Инструмент командной строки, который компилирует исходные материалы («статьи», заметки, README, PDF и веб-страницы) во взаимосвязанную вики в формате markdown с указанием источников. Затем вики могут просматривать, запрашивать и повторно использовать как агенты, так и люди.
Это библиотека или CLI?В основном это CLI (команда llmwiki). Также он работает как MCP-сервер для агентов через llmwiki serve и предоставляет TypeScript SDK (createWiki), если нужно встроить его в свой код.
Как его запустить?Установите командой npm install --global llm-wiki-compiler, затем выполните llmwiki quickstart: он загрузит один источник и скомпилирует его. Добавлять файлы позже можно командой llmwiki ingest, а пересобирать вики командой llmwiki compile.
Что нужно установить заранее?Node.js 24 или новее для запуска установленного CLI и SDK (Bun нужен только тем, кто участвует в разработке и собирает репозиторий). Также нужен провайдер LLM: Anthropic, сервер, совместимый с OpenAI, Ollama или локальный вход для Claude Agent SDK или OpenAI Codex CLI.
Когда это не подойдёт?Это не универсальный генератор статических сайтов, не тяжёлая база данных онтологий и не замена разовому поиску по быстро меняющимся логам. Для разового чтения файла или живого поиска в интернете вики не нужна.
Чем это отличается от обычного RAG?RAG извлекает сырые фрагменты для каждого вопроса. llmwiki компилирует источники один раз в типизированные взаимосвязанные страницы с цитатами, поэтому запросы и агенты работают со стабильным артефактом, который со временем накапливает структуру.
Из каких источников он умеет компилировать вики?Статьи, заметки, README, расшифровки, PDF, изображения и веб-страницы превращаются в страницы понятий, сущностей, сравнений и обзоров. Результат можно экспортировать в JSON, GraphML, слайды Marp или llms.txt.
Можно ли проверить страницы перед сохранением?Да. Запустите llmwiki compile --review, чтобы задерживать страницы до одобрения, или задайте политику проверки, которая автоматически задерживает страницы, если срабатывают правила уверенности, противоречий, схемы или происхождения данных.
Что такое настраиваемые профили жизненного цикла?Это проверенный файл .llmwiki/profile.json, который определяет типы сущностей, связи, состояния жизненного цикла, шлюзы проверки и рабочие процессы. Без него используется стандартная структура «понятия и запросы».
Какие есть похожие проекты?Agentero, openwiki, obsidian-mind, OpenKB, operator-memory, okf-agent-memory, llm_wiki, chubbyskills
Автор
atomicstrata
Размер
12.4 MB
Звёзды
2.3K
Возраст
7 мес
Коммиты · 90 дней