Spark has a feature called AQE. What does AQE stand for and what does it do?

AQE stands for Adaptive Query Execution. It is a Spark 3.0+ feature that optimizes query plans at runtime based on actual data statistics. Instead of relying solely on the static plan generated by the optimizer, AQE can adjust shuffle partition counts, switch between broadcast and shuffle joins, merge stages, and eliminate unnecessary shuffles after the job starts. By collecting metrics from the first stage, Spark can make data‑driven decisions that improve performance, reduce shuffle size, and avoid skew. AQE is enabled with spark.sql.adaptive.enabled=true and can be fine‑tuned with options like spark.sql.adaptive.skewJoin.enabled and spark.sql.adaptive.shuffle.targetPostShuffleInputSize.