When does Spark Adaptive Query Execution (AQE) happen?

Spark Adaptive Query Execution (AQE) is a runtime optimization feature that rewrites the physical execution plan after the first stage of execution. It activates after the initial logical plan is converted to a physical plan and the first set of statistics (e.g., row counts, data sizes) are collected from shuffle or broadcast operations. Once the job has executed the first stage, Spark gathers runtime metrics and can apply several optimizations: dynamic partition pruning, shuffle partition coalescing, broadcast join re‑evaluation, and skew join handling. AQE runs before the final physical plan is executed, so the second stage benefits from updated statistics. It is enabled by default in Spark 3.x but can be toggled with spark.sql.adaptive.enabled. AQE is particularly useful when data statistics are inaccurate or highly skewed, as it allows Spark to adapt the plan to actual runtime conditions, reducing shuffle size and improving performance.