duckpad README

项目的 README,来自 github.com/hugues31/duckpad。合并为一个 markdown 文件:README.md。

Duckpad

See the flow. Trust the result.

Duckpad is a free, open-source visual tool for building data pipelines.
Connect steps on a canvas, write SQL or Python where you need it, and see the data at every step. Powered by DuckDB.

Tests PyPI version Python versions License: MIT

Website · Quickstart · Documentation · Releases

A Duckpad pipeline: two sources are joined, filtered and aggregated into two exports, with the result table of the selected step under the canvas

▶ Watch Duckpad in action

What is Duckpad?

Duckpad is a workspace for preparing and transforming data. You build a pipeline by connecting steps on a canvas: load files or database tables, join, filter, aggregate, export. Each step is a SQL query or a Python class that you can read and edit, and each step keeps its result. You can open any table along the way instead of guessing what the pipeline did.

It runs on your laptop or on a server your team shares. There is no cloud service, no sign-up and no paid edition.

Use it to:

Try it

You need Python 3.11+ and Git.

pip install duckpad
duckpad start

Your browser opens Duckpad: choose the administrator password, and you are in.

On macOS or Windows, run duckpad start --trusted-local instead, for your own projects on your own machine. The default mode confines pipelines to the files they are allowed to read, which requires Linux with Landlock; see the platform requirements.

Then choose New project → Start from an example and Run to explore a finished pipeline, or follow Duckpad in 15 minutes to build your first one from scratch.

A closer look

Trace a number back to its source

Click a cell in a result to see how it was calculated: the rows behind a total, through joins and aggregates, down to the original source cells. Coverage is explicit: when a step cannot be explained, Duckpad tells you.

The lineage of a regional revenue total: the first five of the 27 source amounts behind the sum

Write SQL and preview it on real rows

Guided forms cover the common steps: select and rename columns, clean values, filter, join, aggregate. When you need more, open the code. It is plain DuckDB SQL, with the input columns one click away, and you can preview the query you are typing on real rows.

The default palette also includes union, sorting, deduplication, calculated columns, pivot/unpivot, reproducible sampling, ranking, running totals and Parquet export. Every default has its own icon, and its SQL and form parameter types are available to user-created nodes too.

A SQL node edited beside the canvas, with its input columns and a preview of the result

Review changes before they run

Work on a branch, compare two versions node by node, check the impact on the data and merge after review. Projects are ordinary Git repositories, so you can also push and pull them.

Two branches compared: the modified node stands out on the flow, with the changed parameter before and after

And also

Build your own nodes

The built-in nodes are a starting point. A node is a file: a SQL query with a few comment directives, or a Python class. Duckpad reads the declarations and turns them into ports, a parameter form and a palette entry. There is nothing to compile or register.

A SQL node declares its inputs, typed parameters, outputs and tests in comments, and optionally its icon and color:

-- @label Revenue breakdown
-- @description Total revenue for each value of the column you choose.
-- @icon chart-bar
-- @color green
-- @input orders
-- @param dimension: column(orders) = "region"
-- @param minimum: number = 100
-- @output result
SELECT {{ dimension }}, SUM(amount) AS revenue
FROM {{ orders }}
WHERE amount >= {{ minimum }}
GROUP BY ALL
ORDER BY revenue DESC;
-- @test revenue_is_positive
SELECT * FROM {{ result }} WHERE revenue <= 0;

A Python node is a class with typed attributes. It receives and returns DuckDB relations, and can use the packages your project declares:

import duckpad as dp
from duckdb import col


# @label Split orders
# @icon split
# @color purple
class Split(dp.Node):
    minimum: int = dp.param(default=300, ge=0)
    column: dp.Column = dp.param(default="amount", input="orders")

    class Outputs(dp.Outputs):
        accepted: dp.Relation
        rejected: dp.Relation

    def run(self, orders: dp.Relation) -> Outputs:
        return self.Outputs(
            accepted=orders.filter(col(self.column) >= self.minimum),
            rejected=orders.filter(col(self.column) < self.minimum),
        )

Click SQL or Python at the bottom of the palette, paste the code, and the node is ready to drop into a pipeline. Python nodes are available once an administrator enables Python for the project.

Both nodes in a pipeline: Duckpad generated their ports, their palette entries, their icons and the parameter form from the code

More ways to extend Duckpad:

See the SQL syntax and Python nodes guides for the full contracts.

How Duckpad compares

Duckpad sits between visual data preparation tools and SQL frameworks.

If you come from You will recognize What Duckpad does differently
Visual data preparation tools such as Alteryx Designer, Dataiku DSS or KNIME A canvas of connected steps, with a result to inspect at each one Every step, including the guided ones, is SQL or Python you can read, edit and diff. See the vocabulary map for Dataiku and Alteryx users.
SQL frameworks such as dbt or SQLMesh SQL files in Git, tests and environments A visual canvas, with a result to inspect at each step and cell-level lineage through SQL. Duckpad computes in DuckDB, not inside your warehouse.
Notebooks and scripts SQL and Python, quick iterations on real data A dependency graph instead of cell order, a form for each step's parameters, and runs you can review and repeat.

Duckpad is probably not what you need for:

Duckpad is young and moves fast. Releases are frequent, and a breaking one can require a fresh instance: read the release notes before upgrading.

Free and open source

Duckpad is released under the MIT license and has nothing to sell.

If Duckpad is useful to you, star the repository and tell a colleague: that is how other people find it.

Find the right guide

I want to… Guide
Install Duckpad or join a team instance Installation and accounts
Learn the basics with a small example Duckpad in 15 minutes
Upload my data and download a result Use your own data
Navigate, save and run a pipeline Workspace
Write SQL or prepare data with forms SQL syntax · Data preparation
Create my own nodes and reuse them SQL syntax · Python nodes · Components
Read from S3 or a database Connections
Check or explain a result Results · Data tests · Lineage
Review changes and automate a version Git · Jobs · CLI and automation · External agents and MCP
Fix a startup, source or execution problem Troubleshooting

The documentation index also covers variables, the assistant and instance administration. Reference material includes project files, architecture, validation, releases and security.

Contributing

Bug reports, ideas and pull requests are welcome. Open an issue or start with CONTRIBUTING.md.

duckpad (hugues31/duckpad)

正在搜索数据库…

这是什么?Duckpad 是一个基于 DuckDB 构建的开源 SQL 工作区,可以让你搭建可视化的数据管道,把任意结果追溯到其源数据值,并用 Git 对所有内容进行版本管理。
这是命令行工具还是图形界面应用?它是一个自托管的网页应用:你在浏览器中打开它,以可视化方式组合管道;同时它也内置了 SQL 编辑器,可以编写原始查询。
怎样才能让它运行起来?安装 Python 3.11+ 和 Git,运行该包启动本地服务器,然后打开终端中显示的本地地址,并使用终端里显示的初始管理员密码登录。
试用之前需要准备什么?运行 Duckpad 只需要 Python 3.11+ 和 Git;只有在你打算开发 Duckpad 本身时,才需要 Node.js。
安装时有哪些需要注意的坑吗?SQL 执行仅支持启用了 Landlock 的 Linux(Landlock ABI 2+,内核 5.19+),因此无法直接在 macOS 或 Windows 上运行。
如何创建我的第一个管道?选择 'New project',为其命名,然后选择 'Start from an example',再点击 'Create and Run pipeline',即可立即看到一个可以运行的管道。
团队能否在同一个项目上协作?可以。Duckpad 支持将工作保存到 Git,提供独立分支、更改的可视化对比以及合并审核,并支持为团队按项目管理访问权限。
点击结果单元格时会发生什么?点击一个有效的单元格后,你可以借助明确的血缘覆盖范围,探查它所依赖的源数据值,把这个数字追溯到它的数据来源。
不同环境的连接可以不一样吗?可以。Duckpad 允许你为每个环境单独配置数据源连接,同时项目访问权限仍由你掌控。
作者
hugues31
大小
116 MB
星标
321
时长
24天
提交 · 90 天