dagster README

The project's README from github.com/dagster-io/dagster. As one markdown file: README.md.

dagster logo

Dagster is a cloud-native data pipeline orchestrator for the whole development lifecycle, with integrated lineage and observability, a declarative programming model, and best-in-class testability.

It is designed for developing and maintaining data assets, such as tables, data sets, machine learning models, and reports.

With Dagster, you declare—as Python functions—the data assets that you want to build. Dagster then helps you run your functions at the right time and keep your assets up-to-date.

Here is an example of a graph of three assets defined in Python:

import dagster as dg
import pandas as pd

from sklearn.linear_model import LinearRegression


@dg.asset
def country_populations() -> pd.DataFrame:
    df = pd.read_html("https://tinyurl.com/mry64ebh")[0]
    df.columns = ["country", "pop2022", "pop2023", "change", "continent", "region"]
    df["change"] = df["change"].str.rstrip("%").astype("float")
    return df


@dg.asset
def continent_change_model(country_populations: pd.DataFrame) -> LinearRegression:
    data = country_populations.dropna(subset=["change"])
    return LinearRegression().fit(pd.get_dummies(data[["continent"]]), data["change"])


@dg.asset
def continent_stats(
    country_populations: pd.DataFrame, continent_change_model: LinearRegression
) -> pd.DataFrame:
    result = country_populations.groupby("continent").sum()
    result["pop_change_factor"] = continent_change_model.coef_
    return result

The graph loaded into Dagster's web UI:

An example asset graph as rendered in the Dagster UI

Dagster is built to be used at every stage of the data development lifecycle - local development, unit tests, integration tests, staging environments, all the way up to production.

Quick Start:

If you're new to Dagster, we recommend checking out the docs or following the hands-on tutorial.

Dagster is available on PyPI and officially supports Python 3.9 through Python 3.14.

uv add dagster dagster-webserver dagster-dg-cli

Documentation

You can find the full Dagster documentation here, including the Quickstart guide.

Key Features:

image

Dagster as a productivity platform

Identify the key assets you need to create using a declarative approach, or you can focus on running basic tasks. Embrace CI/CD best practices from the get-go: build reusable components, spot data quality issues, and flag bugs early.

Dagster as a robust orchestration engine

Put your pipelines into production with a robust multi-tenant, multi-tool engine that scales technically and organizationally.

Dagster as a unified control plane

Maintain control over your data as the complexity scales. Centralize your metadata in one tool with built-in observability, diagnostics, cataloging, and lineage. Spot any issues and identify performance improvement opportunities.

Master the Modern Data Stack with integrations

Dagster provides a growing library of integrations for today’s most popular data tools. Integrate with the tools you already use, and deploy to your infrastructure.


image

Community

Connect with thousands of other data practitioners building with Dagster. Share knowledge, get help, and contribute to the open-source project. To see featured material and upcoming events, check out our Dagster Community page.

Join our community here:

Contributing

For details on contributing or running the project for development, check out our contributing guide.

License

Dagster is Apache 2.0 licensed.

dagster (dagster-io/dagster)

cloud-native
Searching Database…

16.3K ·GitHub
Dagster
What is this?Dagster is a cloud-native data pipeline orchestrator covering the whole development lifecycle: it builds, runs, and observes data assets like tables, datasets, ML models, and reports, with built-in lineage and observability.
Is it a code framework or a service I run?Both: you declare data assets as Python functions, Dagster runs them at the right time, keeps them up to date, and shows the asset graph in a web UI.
How do I get it running?Install it from PyPI with pip install dagster; it officially supports Python 3.9 through 3.14, and the docs include a quickstart guide plus a hands-on tutorial.
Does it plug into the data tools I already use?Yes — Dagster ships a growing library of integrations for popular data tools, and you deploy it to your own infrastructure.
Is it production-ready or mainly for local experimentation?It's built for the entire lifecycle — local development, unit tests, integration tests, staging environments, all the way up to production.
What kind of data work is it built for?Developing and maintaining data assets — tables, datasets, machine learning models, and reports — so it targets data engineering, analytics, and MLOps work.
Do I have to pay for it?No — Dagster is open source under the Apache 2.0 license.
What does the code look like?You define data assets as ordinary Python functions; the README shows a graph of three assets defined in Python, and that same graph loads straight into Dagster's web UI for visualization.
How do I keep track of pipeline health?Dagster acts as a unified control plane: metadata is centralized in one tool with built-in observability, diagnostics, cataloging, and lineage, so you can spot issues and find performance improvements as complexity scales.
Can it scale beyond one person's scripts?Yes — the orchestration engine is multi-tenant and multi-tool, built to scale both technically and organizationally, with reusable components and CI/CD practices encouraged from the start.
Where should someone new begin?The README points to the documentation and a hands-on tutorial; the quickstart guide is the fastest route to a first running pipeline.
What are similar projects?airflow, kestra, taipy, prefect, dolphinscheduler, full-stack-data-engineering, kedro, OpenMetadata
Which topics is it in?data-integration, data-pipelines, etl, metadata, mlops, scheduler, data-engineering, cloud-native, analytics, workflow, workflow-automation, orchestration
Creator
dagster-io
Size
1.4 GB
Stars
16.3K
Updated
CommitOct 16Release1.13.26Oct 09
Age
8y
Commits · 90 Days