What is this?
It is an engine that schedules, leases, and tracks large-scale repeated checks of sources (web pages, feeds, listings) by a fleet of workers — plain fetchers, scrapers, or LLM agents — detecting changes by content hash and keeping an auditable ledger of every write.
It is an engine that schedules, leases, and tracks large-scale repeated checks of sources (web pages, feeds, listings) by a fleet of workers — plain fetchers, scrapers, or LLM agents — detecting changes by content hash and keeping an auditable ledger of every write.Is this a framework I build on, or a ready-made scraper?It is a framework/library built around five SQLite tables (source, action, batch, observation, changelog); it never interprets what a source actually is and leaves fetching logic entirely to the workers you plug in.
It is a framework/library built around five SQLite tables (source, action, batch, observation, changelog); it never interprets what a source actually is and leaves fetching logic entirely to the workers you plug in.How do I try it out?Run `python examples/demo.py` to see the whole sweep-lease-reconcile loop on local files with no network; the `sweep` command queues due sources and leases batches, and `stats` prints per-source change/error rates and latency.
Run `python examples/demo.py` to see the whole sweep-lease-reconcile loop on local files with no network; the `sweep` command queues due sources and leases batches, and `stats` prints per-source change/error rates and latency.Do I need to install any extra libraries or set up a database server?No — it uses only the Python standard library, and the entire system is one SQLite file.
No — it uses only the Python standard library, and the entire system is one SQLite file.Does the engine know what my sources actually are?No, it treats the 'route' field as meaningless bookkeeping data — all interpretation of what to fetch and how is left entirely to your worker code.
No, it treats the 'route' field as meaningless bookkeeping data — all interpretation of what to fetch and how is left entirely to your worker code.What is SKILL.md for?It teaches a coding agent how to operate the full sweep-lease-reconcile loop, so an LLM can run the fleet on its own.
It teaches a coding agent how to operate the full sweep-lease-reconcile loop, so an LLM can run the fleet on its own.What happens if a worker crashes mid-batch?Workers only write result files into their own write-scope directory, and the parent reconciles them afterward; a missing result file is simply treated as a failure, so a crashed or misbehaving worker can't corrupt canonical state.
Workers only write result files into their own write-scope directory, and the parent reconciles them afterward; a missing result file is simply treated as a failure, so a crashed or misbehaving worker can't corrupt canonical state.Can I watch this running in Grafana?Yes, since the tables map directly onto spans (run to trace, batch to parent span, action to child span), the engine can post OTLP/HTTP JSON telemetry to any collector, including a Grafana Cloud OTLP gateway, or just print it to stdout.
Yes, since the tables map directly onto spans (run to trace, batch to parent span, action to child span), the engine can post OTLP/HTTP JSON telemetry to any collector, including a Grafana Cloud OTLP gateway, or just print it to stdout.What kinds of things do people actually track with this?Examples include sales prospecting on careers pages, competitor price and feature watching, regulatory/compliance page monitoring, keeping research corpora current, and marketplace or listing monitoring.
Examples include sales prospecting on careers pages, competitor price and feature watching, regulatory/compliance page monitoring, keeping research corpora current, and marketplace or listing monitoring.What are similar projects?agento11y, 3d-asset-server, opensre, bareloop, evenfire, youtube-automation-agent, prefect, pyspider
agento11y, 3d-asset-server, opensre, bareloop, evenfire, youtube-automation-agent, prefect, pyspiderWhich topics is it in?grafana, opentelemetry, scheduler, crawler, web-scraping, observability, sqlite, monitoring, agent, automation, ai-agents, tool
grafana, opentelemetry, scheduler, crawler, web-scraping, observability, sqlite, monitoring, agent, automation, ai-agents, tool