You're tasked with building a real‑time data pipeline using Apache Kafka and Apache Spark. The pipeline needs to process 1 million events per second and write the results to a data warehouse. How would you design the pipeline and what technologies would you use?

To handle 1 M events/s, start with a highly partitioned Kafka topic (e.g., 200–400 partitions) so that consumer groups can scale horizontally. Use a schema registry (Avro/Protobuf) to enforce data contracts and enable compression. Consume the stream with Spark Structured Streaming in a cluster managed by YARN or Kubernetes, allocating enough executors to match the Kafka throughput (roughly 1 executor per 10 k events/s). Enable checkpointing to HDFS/S3 for fault tolerance and use the foreachBatch sink to write aggregated results to the warehouse. For low latency, use Spark’s continuous processing mode and a stateful aggregation with a tumbling window (e.g., 1‑second windows). Persist intermediate state in a distributed store like RocksDB or Delta Lake to avoid recomputation. To write to the warehouse, use a connector such as Snowflake’s JDBC or Redshift’s COPY command; batch the writes in micro‑batches (e.g., 5 s) to reduce overhead. Monitor metrics (Kafka lag, Spark UI, warehouse load) and auto‑scale executors based on CPU/memory usage. This architecture balances throughput, fault tolerance, and low‑latency writes to the warehouse.