When choosing columns for partitioning in Spark, what trade‑offs are you considering regarding the files it produces?

The main trade‑offs involve cardinality, file size, and read/write patterns. High‑cardinality columns create many small files, which can overwhelm the filesystem and slow down reads due to metadata overhead. Low‑cardinality columns produce fewer, larger files that are easier to manage but may not provide enough pruning for selective queries. You also need to consider the typical query filters: partition on columns that are often used in WHERE clauses to enable predicate pushdown. Additionally, think about the write workload; frequent writes to many partitions can cause a high number of open files and increase the cost of compaction. Balancing these factors typically means selecting a few moderate‑cardinality columns that align with the most common query patterns.