Is groupBy a narrow or wide transformation in Spark?

groupByKey is a wide transformation. It requires a shuffle because all values for a given key must be brought together on a single partition. The shuffle writes intermediate data to disk and then reads it on the reducer executor. Because groupByKey brings all values of a key together, it can lead to data skew if some keys have many more values than others. This skew can cause some reducers to become bottlenecks. To mitigate this, you can use reduceByKey, which performs a local aggregation on each mapper before shuffling, reducing the amount of data that needs to be transferred. Alternatively, you can use aggregateByKey or combineByKey to perform partial aggregation with custom logic. Spark also provides the option to use map-side combine by setting the map-side combine flag, which can further reduce shuffle traffic. Understanding the difference between groupByKey and reduceByKey is crucial for writing efficient Spark jobs, especially when dealing with large datasets or skewed key distributions. For example, if you have an RDD of (key, value) pairs and you call groupByKey, Spark will hash each key to a partition, shuffle the data across the cluster, and then aggregate the values for each key on the target partition. This incurs network and disk I/O, making groupByKey less efficient than reduceByKey, which performs a partial aggregation before shuffling.