How did you manage late arriving data and deduplication strategies, and how did you monitor and validate AI feature pipelines to quickly detect and resolve data inconsistencies?

To handle late arriving data, I use event‑time timestamps and watermarks so that the pipeline can wait for a configurable delay before finalizing a window. Deduplication is performed by maintaining a hash of the key attributes in a distributed cache (e.g., Redis or Flink state) and discarding duplicates that arrive after the first occurrence. For monitoring, I expose metrics such as "late event ratio", "deduplication hit rate", and "feature drift" to Prometheus. I also set up alerting in Grafana for thresholds that exceed acceptable limits. Validation is done by running a reference model on a small sample of the data and comparing the output against a golden dataset stored in S3. If discrepancies are detected, an automated rollback or re‑ingestion is triggered. This combination of event‑time handling, stateful deduplication, and real‑time monitoring allows me to detect and resolve inconsistencies within minutes.