14.5K ·GitHub
What is this?
Apache DolphinScheduler is a modern data orchestration platform for building and running complex data pipeline workflows with low-code, drag-and-drop tools.
Apache DolphinScheduler is a modern data orchestration platform for building and running complex data pipeline workflows with low-code, drag-and-drop tools.Is this a library I embed, or a standalone service I run?It's a standalone orchestration platform/server with its own Web UI, master/worker processes, and a database backend, not a library you import into code.
It's a standalone orchestration platform/server with its own Web UI, master/worker processes, and a database backend, not a library you import into code.How do I get it running?You can start it in Standalone mode for a quick try, or deploy via Docker Compose, Kubernetes, or Terraform for cluster setups.
You can start it in Standalone mode for a quick try, or deploy via Docker Compose, Kubernetes, or Terraform for cluster setups.Do I need a separate database and cluster setup, or can I just try it standalone?For a quick trial you only need Standalone mode; production deployments need a database (MySQL/PostgreSQL etc.) plus cluster/Kubernetes infrastructure for the master-worker architecture.
For a quick trial you only need Standalone mode; production deployments need a database (MySQL/PostgreSQL etc.) plus cluster/Kubernetes infrastructure for the master-worker architecture.Is it only for simple workflows, or can it handle real production scale?It's built for production scale, claiming performance several times faster than other orchestration platforms and capable of handling tens of millions of tasks per day, though running the full cluster setup properly requires more effort than a single script.
It's built for production scale, claiming performance several times faster than other orchestration platforms and capable of handling tens of millions of tasks per day, though running the full cluster setup properly requires more effort than a single script.How can I build workflows without writing a lot of code?Workflows can be created and managed visually via the Web UI with drag-and-drop, or programmatically through the Python SDK or Open API.
Workflows can be created and managed visually via the Web UI with drag-and-drop, or programmatically through the Python SDK or Open API.Can I pause or recover a running workflow?Yes, DolphinScheduler offers flexible state control, supporting pausing, stopping, and recovering workflows and individual tasks at any time.
Yes, DolphinScheduler offers flexible state control, supporting pausing, stopping, and recovering workflows and individual tasks at any time.Does it support multiple teams sharing one deployment?Yes, it has multi-tenancy support along with permission control at the project and data source level.
Yes, it has multi-tenancy support along with permission control at the project and data source level.How is it different from Airflow or Azkaban?DolphinScheduler emphasizes a decentralized multi-master/multi-worker high-availability architecture, native workflow versioning, and claims several times higher throughput than comparable orchestration platforms.
DolphinScheduler emphasizes a decentralized multi-master/multi-worker high-availability architecture, native workflow versioning, and claims several times higher throughput than comparable orchestration platforms.What are similar projects?airflow, kestra, argo-workflows, dagster, astron-agent, JeecgBoot, airbyte, taipy
airflow, kestra, argo-workflows, dagster, astron-agent, JeecgBoot, airbyte, taipyWhich topics is it in?workflow-orchestration, job-scheduler, data-pipelines, data-engineering, low-code, cloud-native, enterprise, workflow, orchestration, kubernetes, dashboard, monitoring
workflow-orchestration, job-scheduler, data-pipelines, data-engineering, low-code, cloud-native, enterprise, workflow, orchestration, kubernetes, dashboard, monitoring