I noticed you didn't share your experience in developing ETL pipelines with Databricks and PySpark. Could you walk me through a recent example where you optimized pipeline performance or handled a specific challenge?

In a recent project, we had to ingest 10 TB of log data daily and transform it into a fact table for analytics. The initial pipeline ran for 12 hours due to a skewed join on a user ID column. I first identified the skew by inspecting the spark.sql.shuffle.partitions and the distribution of keys.

To resolve it, I applied a combination of techniques: I broadcasted the smaller dimension table, used a custom partitioner that added a random salt to the skewed key, and enabled spark.sql.adaptive.enabled to let Spark dynamically adjust shuffle partitions. I also switched from a standard DataFrame API to Delta Lake, which allowed me to use OPTIMIZE and ZORDER to improve read performance.

After these changes, the pipeline time dropped from 12 hours to 2 hours, and the cost per run decreased by 70%. I validated the results with a checksum comparison and added a unit test that asserts the row count and key distribution to guard against future regressions.