How would you process 100GB of data in a distributed computing environment? How would you split it into chunks and aggregate the results?

Processing 100 GB of data in a distributed environment typically involves a cluster framework such as Apache Spark or Dask. First, partition the data into logical chunks—e.g., 10 partitions of 10 GB each—using a partitioning key that balances load and preserves locality. Each executor processes its chunk in parallel, performing map operations to compute partial aggregates (sum, count, mean). To reduce network traffic, a combiner stage merges partial results locally before shuffling. After the map phase, a reduce phase aggregates the partial results across executors to produce the final output. Spark’s shuffle mechanism handles data skew by redistributing hot partitions. Fault tolerance is achieved through lineage; if a node fails, Spark recomputes only the affected partitions. The overall time complexity is O(n) with a constant factor determined by the number of partitions and executor memory. Choosing an optimal chunk size (e.g., 128 MB per partition) balances I/O overhead and parallelism, ensuring efficient use of cluster resources.