Suppose the source system contains duplicate records. How would you design a scalable pipeline to handle them, and can you describe your end‑to‑end experience from design to deployment?

I would start by defining a unique key for the entity and capturing changes via CDC or incremental snapshots. The ingestion layer uses Kafka or a cloud queue to buffer records, ensuring at‑least‑once delivery. In the processing layer, a Spark Structured Streaming job reads the stream, applies a dedupe window using a stateful aggregation keyed on the unique key, and writes the cleaned data to a Delta Lake or Snowflake. Partitioning by date or key range keeps the job scalable. For the target, I use a MERGE statement to upsert into the final table, guaranteeing idempotence. I prototype the job in a dev cluster, run unit tests with sample data, and use CI/CD pipelines (GitHub Actions + Airflow) to deploy to production. Monitoring is set up with Prometheus and Grafana to track duplicate rates and job latency. My experience includes designing such pipelines for a financial services client, where we reduced duplicate processing by 95% and achieved 99.9% data freshness.