What is the trade‑off between read performance (filtering) and write performance (file count/size) when partitioning data in Spark?

Partitioning in Spark allows the engine to skip entire directories when filtering on the partition column, improving read performance. However, each partition becomes a separate file or set of files. If you partition on a high‑cardinality column, you create many small files, which increases the number of open files, slows down the driver, and can lead to a write‑heavy workload. Conversely, too few partitions can result in large files that are expensive to read and may not be parallelized efficiently. The sweet spot is to partition on columns that are frequently filtered and have moderate cardinality, ensuring that each partition is large enough to be read efficiently but small enough to be skipped when not needed. Balancing these factors often requires profiling read/write times and adjusting the partition strategy accordingly.