In Databricks Workflows, how do task dependencies affect retry behavior when an upstream task fails?
💡 Model Answer
In Databricks Workflows, tasks are linked by explicit dependencies. When an upstream task fails, the downstream tasks that depend on it are marked as 'skipped' by default, and the workflow stops unless you configure a retry policy. Each task can have a retry count and a retry interval. If a downstream task has a dependency on a failed upstream task, the downstream task will not retry unless you set the 'continue_on_failure' flag to true. By default, the workflow will not retry a failed task more than the configured number of times, and any dependent tasks will be skipped. You can also use the 'depends_on' field to specify that a task should run only if all upstream tasks succeed, or use 'depends_on' with 'on_failure' to trigger alternate paths. Understanding these settings allows you to design robust pipelines that either halt on failure or continue with fallback logic.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500