Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
Global
#1447 · +21 · 1d ·GitHub
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here 👇🏼
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
#3507 · +5 · 1d ·GitHub
Convert documents to structured data effortlessly. Unstructured is open-source ETL solution for transforming complex documents into clean, structured formats for language models.
An orchestration platform for the development, production, and observation of data assets.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
#5877 · +1 · 1d ·GitHub
Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
#6581 · +1 · 1d ·GitHub
End-to-end streaming data platform on AWS: Kafka → Iceberg lake → Kimball model → Athena, orchestrated with Airflow 3. Terraform, Glue PySpark, dbt.