In simple English
This tool helps you run different tasks in a specific order on a schedule. You connect steps together using a visual screen instead of writing complex code. It keeps track of when jobs fail and lets you restart them easily. Running it needs a server with enough memory to handle large data pipelines. The project is free to use under an open-source licence.
What it is
Apache DolphinScheduler is a workflow management system built for data pipelines. It lets you create task dependencies and run them across servers using a visual web interface or a Python script.
What it is good for
Data engineers and solo builders who need to run scheduled scripts, database queries, and file transfers in a specific order without writing custom scheduling code.
- Provides a drag-and-drop web interface to connect tasks without writing code.
- Supports many built-in task types including database queries and shell scripts out of the box.
- Includes version control for workflows so you can roll back broken changes.
- Needs 4 to 8 GB of memory and a stable database server to run smoothly, which costs money.
- Setup is complex for non-programmers because it requires managing Java and database dependencies.
- Maintenance takes ongoing work to monitor failed tasks and server health.
cron jobs and manual scripts
PostgreSQL
- apache/airflow47k stars
Airflow is another popular workflow orchestrator, but it defines tasks entirely in Python code instead of using a drag-and-drop visual interface.
- kestra-io/kestra30k stars
Kestra is a workflow orchestrator that uses declarative configuration files, while DolphinScheduler focuses on a visual drag-and-drop web interface.
- PrefectHQ/prefect24k stars
Prefect is a workflow management tool built for Python developers, whereas DolphinScheduler provides multi-language support and a visual platform.
- vectordotdev/vector23k stars
One collector that gathers logs and numbers from machines and sends them to one or more monitoring services.
- bluewave-labs/Checkmate11k stars
A self-hosted monitor that checks whether sites and services answer, records response times and sends alerts.
- statping-ng/statping-ng2.0k stars
A single-file status page and uptime checker that works with SQLite, PostgreSQL or MySQL.
docker run -d --name dolphinscheduler-standalone -p 12345:12345 apache/dolphinscheduler:latest