Global

Event Driven Orchestration & Scheduling Platform for Mission Critical Applications

Your company's data, connected into one ontology, on your own computer. A local-first data workspace built on HelixDB and DuckDB.

Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼

Apache Superset is a Data Visualization and Data Exploration Platform

> Ingest your enterprise data, extract what matters, build a Context Graph and knowledge graph (KG), and run graph analytics and causal reasoning over all of it, with full decision provenance baked in. Explainable, traceable, and trustworthy by design.

ClickHouse® is a real-time analytics database management system

📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Apache Spark - A unified analytics engine for large-scale data processing

In this course, we'll go from experimentation (design + development) to production (deployment + iteration). We'll do this iteratively by motivating the components that will enable us to build a reliable production system.

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

This is a repo with links to everything you'd ever want to learn about data engineering

Below you find a set of charts demonstrating the paths that you can take and the technologies that you would want to adopt in order to become a data scientist, machine learning or an AI expert.

An orchestration platform for the development, production, and observation of data assets.

The Open Context Layer for Data and AI , OpenMetadata is the open platform for building trusted data context and business semantics for humans, AI assistants, and agents.

Knowledge Catalog (formerly Dataplex), is an AI-powered data catalog and metadata management platform. It provides a dynamic knowledge graph of all your data, structured and unstructured, to provide semantics and business context to AI agents

Apache Doris is a real-time analytics and hybrid search database for AI agents.

Enterprise-grade metadata platform enabling discovery, governance, and observability across your entire data ecosystem

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

Taipy is designed for data scientists and machine learning engineers to create data & AI driven web applications.

Data Version Control or DVC is a command line tool and VS Code Extension to help you develop reproducible machine learning projects:

dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

Incremental engine for long horizon agents 🌟 Star if you like it!

Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.

本项目旨在分享大模型相关技术原理以及实战经验（大模型工程化、大模型应用落地）

《Designing Data-Intensive Application》DDIA 第一版 / 第二版 中文翻译

What is this Book? How to Contribute YouTube Twitter Amazon Shop

Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (

Debezium is an open source project that provides a low latency data streaming platform for change data capture (CDC). You set up and configure Debezium to monitor your databases, and then your applications consume events for each row-level change made to the database.