Is there a service feature that can automatically scale up? If not, how would you design the initial low cluster? How would you handle daily data or entire data? You must have some basic calculation before doing experiments. For example, if you want to process this much data, what calculations would you do before starting?

Auto‑scaling can be achieved with services like AWS Auto Scaling, Kubernetes HPA, or managed Spark clusters that adjust node count based on metrics such as CPU, memory, or job queue length. If a built‑in feature is unavailable, you design an initial low‑capacity cluster that meets the minimum daily load. First, estimate the data volume per day and the peak hourly volume. Use historical metrics or a pilot run to calculate average and peak processing time per record. Multiply by the number of records to get total processing time, then divide by the desired job window to determine required cluster size. Add a safety margin (10‑20 %) for spikes. For experiments, run a small subset of data to validate the scaling logic and cost model. Monitor metrics and adjust thresholds. This approach balances cost, performance, and reliability.