How would you design the cluster configuration for a 100 GB Spark job to ensure efficient resource usage and auto‑tuning?

Designing a Spark cluster for a 100 GB job starts with sizing. Estimate the data size per executor: a good rule of thumb is 1–2 GB of data per executor for shuffle‑heavy workloads. If you have 8 GB of RAM per node, you might allocate 4 GB to the driver and 4 GB to each executor, leaving some headroom for overhead. Enable dynamic allocation (spark.dynamicAllocation.enabled=true) so Spark can scale the number of executors up or down based on workload. Tune spark.executor.cores to 4–5 to balance parallelism and CPU contention. Set spark.sql.shuffle.partitions to a value close to the number of cores in the cluster (e.g., 200–400 for a 100 GB job) to avoid too many small tasks. Use adaptive query execution (spark.sql.adaptive.enabled=true) to let Spark merge or split shuffle partitions at runtime. For memory‑intensive jobs, increase spark.memory.fraction and enable off‑heap memory (spark.memory.offHeap.enabled=true). Cache intermediate RDDs or DataFrames that are reused. Finally, monitor the cluster with Ganglia or Spark UI, adjust the number of executors, and consider using YARN or Kubernetes autoscaling to add nodes when the job spikes. These settings help keep CPU, memory, and network I/O balanced, reducing job runtime and avoiding OOM errors.