When writing CSV files in Spark, how does the partitioning strategy work? How many partitions are created? Do we need to apply any logic behind it, or is it random or default? Should we let Spark decide the number of partitions, or can we set a custom partitioning strategy? Provide an example.

When writing CSV files in Spark, the number of output partitions is determined by the DataFrame’s current partitioning and the repartition or coalesce transformations applied before the write. By default, Spark writes one file per partition, so the output size and file count depend on the number of partitions. The default number of partitions is usually the value of spark.sql.shuffle.partitions (often 200) or the number of cores in the cluster. If you do not explicitly repartition, Spark may use the existing partitioning, which could be many small partitions or a single large one. You can let Spark decide by simply calling df.write.csv(...); Spark will use the current partition count. However, for large datasets (100–200 GB), you often want to control the number of output files to balance parallelism and file system overhead. A common strategy is to repartition(n) where n is roughly the number of executors times the number of cores per executor, or to use coalesce to reduce the number of partitions after a shuffle. For example: df.repartition(200).write.option("header","true").csv("s3://bucket/path"). This ensures 200 output files, each roughly 1 GB, which is efficient for downstream processing.