What approach would you take to avoid duplicate records in a data pipeline, both before and after transformation?

To avoid duplicates you first enforce uniqueness at the source: use primary keys or unique constraints, or apply CDC to capture only changes. In the pipeline you can dedupe by hashing a composite key (e.g., user_id + timestamp) and using Spark’s dropDuplicates() or groupBy(). For streaming, use Structured Streaming’s watermarking and stateful aggregation to keep a set of seen keys. After transformation, you can perform a final dedupe in the target system: in Snowflake use a MERGE statement with a unique key, or in a relational DB use a unique index. Additionally, maintain a dedupe table that records processed record IDs to prevent re‑processing. Logging and monitoring of duplicate counts help catch regressions. This two‑stage approach ensures that duplicates are eliminated early and any that slip through are caught before they reach downstream consumers.