How would you handle a failure in Airflow, and how would you prevent duplicate data when the DAG retries?

Handling failures in Airflow and preventing duplicate data on DAG retries requires a combination of Airflow configuration, idempotent task design, and data layer safeguards. First, set retry and retry_delay on the task to control how many times it will be retried and the back‑off strategy. Use depends_on_past=True if you want a task to wait for the previous run to finish, which can help avoid overlapping data loads. For duplicate prevention, design the task to be idempotent: write to a staging table, then use an UPSERT (INSERT … ON CONFLICT) or MERGE statement in the target database so that re‑running the same data does not create duplicates. Store a checksum or hash of the processed batch in XCom or a metadata table; before processing, check if that hash already exists and skip if it does. Use Airflow’s TriggerRule to ensure downstream tasks only run after all upstream tasks succeed. If you need to manually clear a failed run, use airflow dags clear or the UI to reset the state. Finally, consider using a stateful operator like S3KeySensor that only triggers when a new file appears, or a TriggerDagRunOperator that passes a unique run_id to avoid duplicate processing. These practices together make retries safe and idempotent.