What is this?
walshadow replicates PostgreSQL rows into ClickHouse by reading the physical WAL — initial load, continuous changes, schema evolution, restarts, and planned source switchover are all part of the flow.
walshadow replicates PostgreSQL rows into ClickHouse by reading the physical WAL — initial load, continuous changes, schema evolution, restarts, and planned source switchover are all part of the flow.What am I actually installing — a service, a CLI tool, a library?Mainly the walshadow-stream replication daemon, plus helper tools for filtering and classifying WAL. On the Postgres side it installs a module loaded via shared_preload_libraries — not a regular SQL extension.
Mainly the walshadow-stream replication daemon, plus helper tools for filtering and classifying WAL. On the Postgres side it installs a module loaded via shared_preload_libraries — not a regular SQL extension.How do I get a first setup running?Follow the Quickstart to bring up PostgreSQL, ClickHouse, and walshadow together with Docker Compose; to plug into databases you already run, use the Getting started guide instead.
Follow the Quickstart to bring up PostgreSQL, ClickHouse, and walshadow together with Docker Compose; to plug into databases you already run, use the Getting started guide instead.Do I need a specific PostgreSQL version for it to work?Yes — PostgreSQL 16 or newer, and walshadow must run the same PostgreSQL major version as the source database.
Yes — PostgreSQL 16 or newer, and walshadow must run the same PostgreSQL major version as the source database.Is it ready for production use?Not yet — it is an experimental Development Preview under active development and hardening, and interfaces and behavior may change, so validate it thoroughly with your own workloads.
Not yet — it is an experimental Development Preview under active development and hardening, and interfaces and behavior may change, so validate it thoroughly with your own workloads.What do the walshadow-filter and walshadow-classify helpers do?walshadow-filter filters offline WAL files at the segment level, and walshadow-classify classifies individual records — both are diagnostics helpers that ship alongside the main stream daemon.
walshadow-filter filters offline WAL files at the segment level, and walshadow-classify classifies individual records — both are diagnostics helpers that ship alongside the main stream daemon.Does it compress the data it replicates?Yes — default builds use LZ4 compression, you can enable zstd for Zstandard, or build with --no-default-features for uncompressed mode.
Yes — default builds use LZ4 compression, you can enable zstd for Zstandard, or build with --no-default-features for uncompressed mode.How does it handle schema changes and restarts?Schema evolution, restarts, and planned source switchover are explicit parts of the replication flow, each covered by its own documentation page rather than left to the user to improvise.
Schema evolution, restarts, and planned source switchover are explicit parts of the replication flow, each covered by its own documentation page rather than left to the user to improvise.Can I replicate only some tables or several databases?Yes — the docs cover table selection, several source databases, and destination table configuration, so selective and multi-database replication are supported scenarios.
Yes — the docs cover table selection, several source databases, and destination table configuration, so selective and multi-database replication are supported scenarios.What are similar projects?airbyte, kafgres, ClickHouse, debezium, electric, pgrust, druid, rankme.fast
airbyte, kafgres, ClickHouse, debezium, electric, pgrust, druid, rankme.fastWhich topics is it in?data-integration, change-data-capture, olap, replication, data-pipeline, etl, clickhouse, postgres, data-engineering, infrastructure, analytics, sql
data-integration, change-data-capture, olap, replication, data-pipeline, etl, clickhouse, postgres, data-engineering, infrastructure, analytics, sql