When you were building these pipelines, how did you handle orchestration and scheduling?

I primarily used Apache Airflow for orchestration. I defined Directed Acyclic Graphs (DAGs) that represented each pipeline stage, set the schedule interval using cron expressions, and leveraged Airflow’s built‑in scheduler to trigger tasks. For dynamic workloads, I used Airflow’s SubDagOperator to split complex jobs into smaller, reusable sub‑DAGs. I also configured task retries, SLA alerts, and email notifications for failures. In addition to Airflow, I experimented with Prefect for pipelines that required dynamic branching based on runtime data; Prefect’s Flow API allowed me to programmatically add tasks and set dependencies. Both tools integrate with Kubernetes, enabling scalable execution. I monitored DAG runs via the Airflow UI and set up Grafana dashboards to track task durations and failure rates.