A pipeline visualisation generated using Kedro-Viz
What is this?
Kedro is an open-source Python framework for building production-ready data engineering and data science pipelines, using software engineering best practices to make them reproducible, maintainable, and modular.
Kedro is an open-source Python framework for building production-ready data engineering and data science pipelines, using software engineering best practices to make them reproducible, maintainable, and modular.Is this a framework I build my project on, or a standalone app?It is a framework: you scaffold a project from a standard Cookiecutter-based template, then write Python functions that Kedro wires into pipelines via automatic dependency resolution.
It is a framework: you scaffold a project from a standard Cookiecutter-based template, then write Python functions that Kedro wires into pipelines via automatic dependency resolution.Do I need anything else before starting, like Jupyter or a specific Python version?Just Python — Kedro supports all Python versions still actively maintained by CPython core, dropping support once a version reaches end of life. Jupyter notebook integration is supported but optional.
Just Python — Kedro supports all Python versions still actively maintained by CPython core, dropping support once a version reaches end of life. Jupyter notebook integration is supported but optional.Does the separate Kedro Datasets package follow the same Python support as core Kedro?No — kedro-datasets generally drops Python version support earlier than the core framework because its many dependencies follow a more conservative NEP 29 policy.
No — kedro-datasets generally drops Python version support earlier than the core framework because its many dependencies follow a more conservative NEP 29 policy.What is the Data Catalog feature?It is a set of lightweight data connectors for saving and loading data across many file formats and systems (local, network, cloud object stores, HDFS), with built-in versioning for file-based data and models.
It is a set of lightweight data connectors for saving and loading data across many file formats and systems (local, network, cloud object stores, HDFS), with built-in versioning for file-based data and models.How do I visualize my pipeline?Kedro-Viz automatically generates a visualisation of your pipeline's dependency graph, and is recommended for new and intermediate users to understand their Kedro projects.
Kedro-Viz automatically generates a visualisation of your pipeline's dependency graph, and is recommended for new and intermediate users to understand their Kedro projects.Where can I deploy a Kedro pipeline?Kedro supports single-machine or distributed deployment, with additional support for Argo, Prefect, Kubeflow, AWS Batch, and Databricks.
Kedro supports single-machine or distributed deployment, with additional support for Argo, Prefect, Kubeflow, AWS Batch, and Databricks.Where should I start learning Kedro?The docs walk through installation and key concepts, then the spaceflights tutorial lets you build a real Kedro project hands-on.
The docs walk through installation and key concepts, then the spaceflights tutorial lets you build a real Kedro project hands-on.What are similar projects?metaflow, airflow, taipy, kubeflow, dagster, mlops-zoomcamp, great_expectations, kestra
metaflow, airflow, taipy, kubeflow, dagster, mlops-zoomcamp, great_expectations, kestraWhich topics is it in?data-pipelines, reproducible-research, pipeline, agentic-workflow, jupyter, mlops, data-engineering, aws, orchestration, data-science, agentic-ai, kubernetes
data-pipelines, reproducible-research, pipeline, agentic-workflow, jupyter, mlops, data-engineering, aws, orchestration, data-science, agentic-ai, kubernetes