dagster README

The project's README from github.com/dagster-io/dagster. As one markdown file: README.md.

dagster logo

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Here is an example of a graph of three assets defined in Python:

import dagster as dg
import pandas as pd

from sklearn.linear_model import LinearRegression


@dg.asset
def country_populations() -> pd.DataFrame:
    df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
    df["change"] = df["change"].str.rstrip("%").astype("float")
    return df


@dg.asset
def continent_change_model(country_populations: pd.DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(pd.get_dummies(data[["continent"]]), data["change"])


@dg.asset
def continent_stats(
    country_populations: pd.DataFrame, continent_change_model: LinearRegression
) -> pd.DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

The graph loaded into Dagster's web UI:

An example asset graph as rendered in the Dagster UI

Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.

Quick Start:

If you're new to Dagster, we recommend checking out the docs or following the hands-on tutorial.

Dagster is available on PyPI and officially supports Python 3.9 through Python 3.14.

uv add dagster dagster-webserver dagster-dg-cli

Documentation

You can find the full Dagster documentation here, including the Quickstart guide.

Key Features:

image

Dagster as a productivity platform

Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.

Dagster as a robust orchestration engine

Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.

Dagster as a unified control plane

Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.

Master the Modern Data Stack with integrations

Dagster provides a growing library of integrations for today’s most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.


image

Community

Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.

Join our community here:

Contributing

For details on contributing or running the project for development, check out our contributing guide.

License

Dagster is Apache 2.0 licensed.

dagster (dagster-io/dagster)

Searching Database…

16.3K ·GitHub
Dagster
这是什么?Dagster 是一个云原生的数据流水线编排工具,覆盖从开发到上线的完整流程:它可以构建、运行并监控各类数据资产,比如数据表、数据集、机器学习模型和报表,并自带血缘追踪和可观测性功能。
它是一个代码框架,还是我要自己运行的服务?两者都是:你把数据资产写成 Python 函数,Dagster 会在合适的时机运行它们、保持数据是最新的,并在网页界面里展示资产之间的关系图。
怎么把它跑起来?用 pip install dagster 从 PyPI 安装即可;官方支持 Python 3.9 到 3.14,文档里还有快速上手指南和动手教程。
它能接上我已经在用的数据工具吗?可以——Dagster 为常见的数据工具提供了越来越多的集成组件,而且你可以把它部署在自己的基础设施上。
它是能用于生产环境,还是主要用来在本地做实验?它是为完整生命周期设计的——本地开发、单元测试、集成测试、预发布环境,一直到生产环境。
它适合做哪类数据工作?它面向数据资产的开发和维护——数据表、数据集、机器学习模型和报表,所以主要服务数据工程、数据分析和 MLOps 这类工作。
用它要花钱吗?不用——Dagster 是开源软件,采用 Apache 2.0 许可证。
代码大概长什么样?你只需把数据资产定义成普通的 Python 函数;README 里展示了用 Python 定义的三个资产组成的图,这个图可以直接加载到 Dagster 的网页界面里进行可视化。
我怎么随时掌握流水线的运行状况?Dagster 相当于一个统一的控制中心:所有元数据都集中在一个工具里,自带可观测性、问题诊断、数据目录和血缘追踪,随着复杂度上升,你也能及时发现问题和性能优化空间。
它能超出个人脚本规模、继续扩展吗?可以——它的编排引擎支持多租户、多种工具,从技术上和组织上都能扩展,还提供可复用的组件,并从一开始就鼓励接入 CI/CD 实践。
新手应该从哪里入手?README 指向了官方文档和动手教程;想最快跑起第一条流水线,看快速上手指南就行。
What are similar projects?airflow, kestra, taipy, prefect, dolphinscheduler, full-stack-data-engineering, kedro, OpenMetadata
Which topics is it in?data-integration, data-pipelines, etl, metadata, mlops, scheduler, data-engineering, cloud-native, analytics, workflow, workflow-automation, orchestration
Creator
dagster-io
Size
1.4 GB
Stars
16.3K
Updated
CommitOct 18Release1.13.26Oct 09
Age
8y
Commits · 90 Days