Topics / #data-engineering

Data-engineering projects on GitHub

Building pipelines and infrastructure that move and transform data at scale.

As of , the most-starred Data-engineering projects on GitHub are superset (75.2K stars), ClickHouse (50.5K stars) and Made-With-ML (49.8K stars). GitHub.Locker tracks 56 Data-engineering repositories with 1M stars combined. The fastest-growing on October 16 was kestra, with +29 stars in a day.

56Projects
1MStars in total
+280Stars in a day
Browse in the catalog →github.com/topics/data-engineering ↗

Hot right now

Most stars gained in a day · Oct 16

+29

kestra

kestra-io/kestra · ★ 29.7K

Kestra is an open-source, event-driven orchestration platform for building data, AI, and infrastructure workflows. You describe them in declarative YAML and manage everything from a built-in web UI.

+29

helix-foundry

HelixDB/helix-foundry · ★ 1.5K

Helix Foundry is a local-first data workspace that connects your company's tools (databases, Stripe, WorkOS, PostHog, files, APIs) into one suggested ontology stored in HelixDB, letting you explore and ask questions about your data.

+17

data-engineering-zoomcamp

DataTalksClub/data-engineering-zoomcamp · ★ 46.2K

This is 'Data Engineering Zoomcamp', a free 9-week course that teaches the fundamentals of data engineering by building an end-to-end data pipeline from scratch.

+16

superset

apache/superset · ★ 75.2K

Apache Superset is a modern, enterprise-ready business intelligence web application for data exploration and visualization: build dashboards, charts, and query data with a built-in SQL editor.

The all-time top

56 projects · by stars

  1. #1 · apache/superset

    superset

    Apache Superset is a modern, enterprise-ready business intelligence web application for data exploration and visualization: build dashboards, charts, and query data with a built-in SQL editor.

    75.2K
  2. #2 · ClickHouse/ClickHouse

    ClickHouse

    ClickHouse is an open-source column-oriented database management system for generating analytical data reports in real time.

    50.5K
  3. #3 · GokuMohandas/Made-With-ML

    Made-With-ML

    Made-With-ML is a free hands-on course that teaches how to design, develop, deploy, and iterate on production-grade machine learning applications, combining ML with software engineering best practices.

    49.8K
  1. 4

    airflow

    47.2Kapache/airflow · Python

    Apache Airflow is a platform to programmatically author, schedule, and monitor workflows, defined as code using Python.

  2. 5

    data-engineering-zoomcamp

    46.2KDataTalksClub/data-engineering-zoomcamp · Jupyter Notebook

    This is 'Data Engineering Zoomcamp', a free 9-week course that teaches the fundamentals of data engineering by building an end-to-end data pipeline from scratch.

  3. 6

    data-engineer-handbook

    44.5KDataExpert-io/data-engineer-handbook · Jupyter Notebook

    A curated handbook of links and resources for learning data engineering: roadmaps, bootcamp materials, books, communities, company tooling lists, blogs, whitepapers, and a directory of data engineering content creators.

  4. 7

    spark

    44.2Kapache/spark · Scala

    Apache Spark is a unified analytics engine for large-scale data processing, offering high-level APIs in Scala, Java, Python, and R on top of an optimized execution engine.

  5. 8

    AI-Expert-Roadmap

    31.3KAMAI-GmbH/AI-Expert-Roadmap · JavaScript

    A set of charts that map the paths and technologies for becoming a data scientist, machine learning engineer, or AI expert — the i.am.ai AI Expert Roadmap, drawn up by AMAI and published as the 2022 edition.

  6. 9

    applied-ml

    30.6Keugeneyan/applied-ml

    applied-ml is a curated collection of papers, articles, and tech blogs where companies like Airbnb, Uber, Netflix, Google, and DoorDash share how they run data science and machine learning in production.

  7. 10

    kestra

    29.7Kkestra-io/kestra · Java

    Kestra is an open-source, event-driven orchestration platform for building data, AI, and infrastructure workflows. You describe them in declarative YAML and manage everything from a built-in web UI.

  8. 11

    flink

    26.4Kapache/flink · Java

    Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities, running your programs on its own distributed runtime.

  9. 12

    llm-action

    25.2Kliguodongiot/llm-action · HTML

    It is a Chinese-language hub of in-depth tutorials on large language model engineering: training, fine-tuning, distributed training, inference, compression, evaluation, data engineering, LLMOps and AI infrastructure, organized as a big index linking the author's article series, many with companion code.

  10. 13

    prefect

    24KPrefectHQ/prefect · Python

    Prefect is a Python workflow orchestration framework for turning ordinary scripts into resilient, production-grade data pipelines with scheduling, retries, caching, and event-based automations.

  11. 14

    ddia

    23.8KVonng/ddia · Python

    A free Chinese translation of Martin Kleppmann's 'Designing Data-Intensive Applications' (DDIA), covering both the 1st and the complete 2nd edition; 210 contributors have polished it since 2018.

  12. 15

    airbyte

    22.2Kairbytehq/airbyte · Python

    Airbyte is an open-source data movement platform for ELT pipelines and AI agents: it moves data from APIs, databases, and files into warehouses, data lakes, and AI applications, backed by a catalog of 600+ connectors. You can run it self-hosted or as Airbyte Cloud.

  13. 16

    taipy

    19.4KAvaiga/taipy · Python

    Taipy is a Python framework that turns data and AI code into production-ready web applications, taking you from a simple pilot script to a deployable app without extra web development work.

  14. 17

    luigi

    18.8Kspotify/luigi · Python

    Luigi is a Python package for building complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization, failure handling and command line integration.

  15. 18

    DataX

    17.4Kalibaba/DataX · Java

    DataX is an offline data synchronization tool/platform from Alibaba Cloud, open-sourced from the DataWorks Data Integration product, used widely inside Alibaba to move data between heterogeneous data sources.

  16. 19

    arrow

    17.2Kapache/arrow · C++

    Apache Arrow is a universal columnar format plus a multi-language toolbox for fast data interchange and in-memory analytics — technologies that let data systems store, process, and move tabular data efficiently.

  17. 20

    argo-workflows

    17Kargoproj/argo-workflows · Go

    An open source, container-native workflow engine for Kubernetes: it orchestrates parallel jobs where every step of your workflow runs as a container. It is a CNCF graduated project.

  18. 21

    BigData-Notes

    17Kheibaiying/BigData-Notes · Java

    This is a comprehensive Chinese-language study guide covering the big data ecosystem: Hadoop, Hive, Spark, Storm, Flink, HBase, Kafka, Zookeeper, Flume, Sqoop, Azkaban, and Scala.

  19. 22

    presto

    16.8Kprestodb/presto · Java

    Presto is a distributed SQL query engine for running interactive analytic queries against big data sources.

  20. 23

    dagster

    16.3Kdagster-io/dagster · Python

    Dagster is a cloud-native data pipeline orchestrator covering the whole development lifecycle: it builds, runs, and observes data assets like tables, datasets, ML models, and reports, with built-in lineage and observability.

  21. 24

    doris

    16.1Kapache/doris · Java

    Apache Doris is an open-source, real-time analytics and search database built on MPP architecture. It serves fast SQL analytics, lakehouse query acceleration, and hybrid search across structured, text, and vector data.

  22. 25

    dvc

    15.9Ktreeverse/dvc · Python

    DVC (Data Version Control) is an open-source tool for versioning data and tracking ML experiments; its own citation tagline calls it 'Git for Data & Models'.

  23. 26

    hadoop

    15.7Kapache/hadoop · Java

    Apache Hadoop is the open-source big data platform for reliable, distributed storage and processing of very large data sets across clusters of machines.

  24. 27

    OpenMetadata

    15.5Kopen-metadata/OpenMetadata · TypeScript

    OpenMetadata is an open platform that unifies technical metadata, lineage, data quality, ownership and business glossaries into one knowledge graph so both humans and AI agents can discover, trust and reuse context about enterprise data.

  25. 28

    Cookbook

    15.5Kandkret/Cookbook · Python

    A free online book (the Data Engineering Cookbook) that covers everything a data engineer needs to know, from basic engineering skills to advanced platform design, case studies and interview prep.

  26. 29

    flink-learning

    15.1Kzhisheng17/flink-learning · Java

    A companion repo to a Chinese-language Apache Flink learning blog, packed with runnable example code, source-code analysis articles, and real-world project case studies (PV/UV stats, log storage, large-scale dedup, monitoring/alerting).

  27. 30

    logstash

    15Kelastic/logstash · Java

    Logstash is a server-side data processing pipeline, part of the Elastic Stack, that ingests data from many sources at once, transforms it, and sends it to a destination like Elasticsearch.

  28. 31

    dolphinscheduler

    14.5Kapache/dolphinscheduler · Java

    Apache DolphinScheduler is a modern data orchestration platform for building and running complex data pipeline workflows with low-code, drag-and-drop tools.

  29. 32

    awesome-mlops

    14.2Kvisenger/awesome-mlops

    A curated list of references for MLOps (Machine Learning Operations) — links to articles, books, courses, papers, tools and communities about running machine learning systems in production.

  30. 33

    druid

    14.1Kapache/druid · Java

    Apache Druid is a high-performance real-time analytics database built to reduce the time between data arriving and being queryable, so dashboards and ad-hoc queries stay fast even under high concurrency.

  31. 34

    semantica

    14.1Ksemantica-agi/semantica · Python

    Semantica is an open-source, graph-native infrastructure layer that ingests enterprise data and builds a structured Context Graph and knowledge graph with deterministic reasoning, ontology governance, and full decision provenance, so AI systems can be explained and audited instead of treated as black boxes.

  32. 35

    dbt

    14Kdbt-labs/dbt · Rust

    dbt lets data analysts and engineers transform data in their warehouse by writing SQL select statements ('models'), applying the same practices — version control, testing, modularity — that software engineers use to build applications.

  33. 36

    dask

    13.9Kdask/dask · Python

    Dask is a flexible parallel computing library for analytics — it runs Python data workloads as scheduled tasks so they can use many cores or machines instead of a single process.

  34. 37

    fg-data-profiling

    13.7KData-Centric-AI-Community/fg-data-profiling · Python

    fg-data-profiling (formerly ydata-profiling / pandas-profiling) is a Python package that generates a one-line exploratory data analysis (EDA) report for pandas or Spark DataFrames, similar to df.describe() but far more detailed.

  35. 38

    trino

    13.3Ktrinodb/trino · Java

    Trino is a fast distributed SQL query engine for big data analytics — you run it as a server (or cluster) and query large datasets with standard SQL. It was formerly known as PrestoSQL.

  36. 39

    debezium

    13.2Kdebezium/debezium · Java

    Debezium is an open source change data capture (CDC) platform that monitors databases and streams row-level change events to applications in real time.

  37. 40

    datahub

    12.9Kdatahub-project/datahub · Python

    DataHub is an open-source metadata platform that lets you discover, understand, and govern data across your entire data stack, connecting warehouses, lakes, BI tools, and AI agents into one unified metadata graph.

  38. 41

    data-engineer-roadmap

    12.8Kdatastacktv/data-engineer-roadmap

    A roadmap that maps out the modern data engineering landscape, meant to serve as a study guide for people who want to become data engineers.

  39. 42

    great_expectations

    11.9Kfivetran/great_expectations · Python

    GX Core (Great Expectations) is a Python framework for testing, validating, and documenting the quality of your data. Its core concept is the 'Expectation' — an expressive unit test for your data, built from the collective experience of thousands of data quality deployments.

  40. 43

    cocoindex

    11.7Kcocoindex-io/cocoindex · Rust

    CocoIndex is an incremental engine that turns codebases, meeting notes, inboxes, Slack, PDFs, and videos into live, continuously fresh context for AI agents and LLM apps, using minimal incremental processing instead of stale batch jobs.

  41. 44

    software-architecture-books

    11.4Kmhadidg/software-architecture-books

    A comprehensive list of books on Software Architecture — explicitly not a recommendation list — where each entry is a table row with title, ISBN, Goodreads rating, number of raters, and publication year.

  42. 45

    kedro

    11Kkedro-org/kedro · Python

    Kedro is an open-source Python framework for building production-ready data engineering and data science pipelines, using software engineering best practices to make them reproducible, maintainable, and modular.

  43. 46

    machine-learning-systems-design

    10.7Kchiphuyen/machine-learning-systems-design · HTML

    A free booklet on designing machine learning systems, covering four main steps: project setup, data pipeline, modeling (selecting, training, debugging), and serving (testing, deploying, maintaining), with links to practical resources and case studies.

  44. 47

    knowledge-catalog

    9.5KGoogleCloudPlatform/knowledge-catalog · TypeScript

    This is the samples and tools repository for Google Cloud's Knowledge Catalog, an AI-powered data catalog and metadata management platform (formerly called Dataplex) that builds a dynamic knowledge graph over structured and unstructured data to give AI agents semantic and business context.

  45. 48

    DeepAnalyze

    4.7Kruc-datalab/DeepAnalyze · Python

    DeepAnalyze is an open-source agentic large language model built for fully autonomous data science: it can handle data preparation, analysis, modeling, visualization, and report generation on its own, across structured, semi-structured, and unstructured data sources.

  46. 49

    ducklake

    3.3Kduckdb/ducklake · C++

    DuckLake is an open lakehouse table format that stores data in Parquet files and stores metadata/catalog info in a SQL database, implemented as a DuckDB extension.

  47. 50

    hotdata-cli

    1.8Khotdata-dev/hotdata-cli · Rust

    Hotdata CLI is a command-line client that lets you query, search, and join data from one place — external databases, APIs, cloud storage, Iceberg catalogs, and uploaded files — using plain SQL and a handful of commands.

  48. 51

    helix-foundry

    1.5KHelixDB/helix-foundry · TypeScript

    Helix Foundry is a local-first data workspace that connects your company's tools (databases, Stripe, WorkOS, PostHog, files, APIs) into one suggested ontology stored in HelixDB, letting you explore and ask questions about your data.

  49. 52

    hflow

    298Hebbian-Robotics/hflow · Python

    HFlow is an open source SDK from Hebbian Robotics (YC S26) for building scalable multimodal data pipelines for robotics and physical AI, letting teams verify the quality of the data they use to train AI models.

  50. 53

    projeto-dados-ia-databricks

    279lvgalvao/projeto-dados-ia-databricks · Python

    This is a 4-night hands-on course that builds the data and sales analytics stack of a fictional B2B perfume distributor from scratch, using Databricks Free Edition, SQL, Python and Claude Code.

  51. 54

    walshadow

    242ClickHouse/walshadow · Rust

    walshadow replicates PostgreSQL rows into ClickHouse by reading the physical WAL — initial load, continuous changes, schema evolution, restarts, and planned source switchover are all part of the flow.

  52. 55

    crab

    155crabbuild/crab · Rust

    Crab is a Git extension that keeps large files like models, datasets, media, and build artifacts out of ordinary Git blobs. Git stores small pointer files while Crab stores the real content as deduplicated chunks in an S3, GCS, Azure, or S3-compatible bucket you own.

  53. 56

    full-stack-data-engineering

    145schumannx/full-stack-data-engineering · Python

    A full reference implementation of a streaming data warehouse on AWS: playback events flow from Kafka into a five-zone S3 Iceberg lake, get modeled with Kimball dimensional design, and are served through Athena SQL and HTML reports, all orchestrated by self-hosted Apache Airflow 3.

Often tagged together