ducklake README

DuckDB DuckLake Extension

DuckLake is an open Lakehouse format that is built on SQL and Parquet. DuckLake stores metadata in a catalog database, and stores data in Parquet files. The DuckLake extension allows DuckDB to directly read and write data from DuckLake.

See the DuckLake website for more information.

Installation

DuckLake can be installed using the INSTALL command:

INSTALL ducklake;

The latest development version can be installed from core_nightly:

FORCE INSTALL ducklake FROM core_nightly;

Usage

DuckLake databases can be attached using the ATTACH syntax, after which tables can be created, modified and queried using standard SQL.

Below is a short usage example that stores the metadata in a DuckDB database file called metadata.ducklake, and the data in Parquet files in the file_path directory:

ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/');
USE my_ducklake;
CREATE TABLE my_ducklake.my_table(id INTEGER, val VARCHAR);
INSERT INTO my_ducklake.my_table VALUES (1, 'Hello'), (2, 'World');
FROM my_ducklake.my_table;
┌───────┬─────────┐
│  id   │   val   │
│ int32 │ varchar │
├───────┼─────────┤
│     1 │ Hello   │
│     2 │ World   │
└───────┴─────────┘

Updates

UPDATE my_ducklake.my_table SET val='DuckLake' WHERE id=2;
FROM my_ducklake.my_table;
┌───────┬──────────┐
│  id   │   val    │
│ int32 │ varchar  │
├───────┼──────────┤
│     1 │ Hello    │
│     2 │ DuckLake │
└───────┴──────────┘

Time Travel

FROM my_ducklake.my_table AT (VERSION => 2);
┌───────┬─────────┐
│  id   │   val   │
│ int32 │ varchar │
├───────┼─────────┤
│     1 │ Hello   │
│     2 │ World   │
└───────┴─────────┘

Schema Evolution

ALTER TABLE my_ducklake.my_table ADD COLUMN new_column VARCHAR;
FROM my_ducklake.my_table;
┌───────┬──────────┬────────────┐
│  id   │   val    │ new_column │
│ int32 │ varchar  │  varchar   │
├───────┼──────────┼────────────┤
│     1 │ Hello    │ NULL       │
│     2 │ DuckLake │ NULL       │
└───────┴──────────┴────────────┘

Change Data Feed

FROM my_ducklake.table_changes('my_table', 2, 2);
┌─────────────┬───────┬─────────────┬───────┬─────────┐
│ snapshot_id │ rowid │ change_type │  id   │   val   │
│    int64    │ int64 │   varchar   │ int32 │ varchar │
├─────────────┼───────┼─────────────┼───────┼─────────┤
│           2 │     0 │ insert      │     1 │ Hello   │
│           2 │     1 │ insert      │     2 │ World   │
└─────────────┴───────┴─────────────┴───────┴─────────┘

See the Usage guide for more information.

Building & Loading the Extension

To build, type

git submodule init
git submodule update
# to build with multiple cores, use `make GEN=ninja release`
make pull
make

To run, run the bundled duckdb shell:

 ./build/release/duckdb

Contributing

We are happy to welcome external contributions to our DuckLake extension. Our active development branch is currently main, and all contributions should target that branch.

Testing

# run all DuckLake extension tests
./build/release/test/unittest
# run a single test file
./build/release/test/unittest test/sql/transaction/create_conflict.test
# run tests matching a pattern
./build/release/test/unittest "test/sql/partitioning/*"

# run DuckDB core tests using DuckLake as storage backend
./build/release/test/unittest --test-config test/configs/attach_ducklake.json --test-dir duckdb

# run DuckLake tests using PostgreSQL as catalog database (requires running PostgreSQL)
./build/release/test/unittest --test-config test/configs/postgres.json
# run DuckLake tests using SQLite as catalog database
./build/release/test/unittest --test-config test/configs/sqlite.json

# run tests with deletion vectors enabled
./build/release/test/unittest --test-config test/configs/deletion_vectors.json

3.3K ·GitHub

Ducklake

What is this?

DuckLake is an open lakehouse table format that stores data in Parquet files and stores metadata/catalog info in a SQL database, implemented as a DuckDB extension.

Is this a file format, or a database engine itself?

It's a table format plus a DuckDB extension: it defines how data (Parquet) and metadata (catalog database) are organized, and DuckDB reads/writes it directly.

How do I start using DuckLake?

Install it with the DuckDB INSTALL command, then ATTACH a DuckLake database pointing to a metadata file (e.g. metadata.ducklake) and a data directory, after which you create/query tables with standard SQL.

Do I need anything besides DuckDB to use it?

Just DuckDB itself (with the extension installed); the catalog metadata is stored in a DuckDB database file and data goes to Parquet files in a filepath directory.

Is there a nightly/dev build if I want the latest features?

Yes, a corenightly development build can be installed in addition to the stable release channel.

Where does my data actually live?

Data lives in Parquet files you control on your own filesystem/storage, while the catalog metadata lives in a separate DuckDB database file — both local and under your control.

Does DuckLake support time travel and schema evolution?

Yes, the README lists Time Travel, Schema Evolution, Updates, and Change Data Feed as supported features of DuckLake.

Is DuckLake tied only to DuckDB or is it an open standard?

It's described as an open Lakehouse format built on SQL and Parquet, with DuckDB as the first implementation reading and writing it directly.

What are similar projects?

ClickHouse, tidb, trino, oceanbase, dagster, duckdb, polyledger, libsql

Which topics is it in?

parquet, etl, duckdb, embedded-database, distributed-database, data, data-engineering, catalog, storage, cloud-native, analytics, sql

Creator: duckdb
Size: 8.3 MB
Stars: 3.3K
Updated: CommitOct 17
Age: 1y
Commits · 90 Days