Is groupBy a wide partitioning operation in Spark?

groupBy in Spark is a wide partitioning operation. It triggers a shuffle because all records with the same key need to be co-located on the same partition. The shuffle writes data to local disk on each executor and then reads it on the target executor. Thus, groupBy is a wide transformation. For example, when you call df.groupBy('col').agg(...) on a DataFrame, Spark will perform a shuffle to bring all rows with the same value of 'col' together before applying the aggregation. This shuffle is expensive, so in many cases you might prefer using reduceByKey or aggregateByKey on an RDD, which can perform partial aggregation locally before shuffling. In DataFrames, groupBy triggers a shuffle by default because the aggregation must be performed on a per-key basis. Spark optimizes this by using the Tungsten execution engine and Catalyst optimizer to minimize data movement, but the underlying shuffle still occurs. You can control the number of shuffle partitions with spark.sql.shuffle.partitions, which defaults to 200. If you have a very large number of distinct keys, you may need to increase this value to avoid excessive task spill. Additionally, you can use the hint 'repartition' before groupBy to explicitly control the partitioning scheme, or use the 'bucketBy' feature to pre-partition data at write time. Understanding when a groupBy will trigger a shuffle helps you design efficient pipelines, especially when working with large-scale data or when the cluster has limited resources.