How do you handle late arriving data in a data pipeline?

Late arriving data is common in streaming pipelines. The typical approach is to use event‑time processing with watermarks. In frameworks like Flink or Spark Structured Streaming you define a watermark that represents the maximum event time you expect to see. When a record arrives later than the watermark plus an allowed lateness window, you can decide to drop it, send it to a side output, or re‑process it. For example, in Flink you set a watermark strategy and an allowed lateness of 5 minutes. Late events that arrive within that window are still aggregated; those that arrive after are routed to a side output. In Spark you can use the watermark function and specify a grace period. If you need to keep late data for audit, you can write it to a separate topic or table. If you want to ignore it, simply drop it. The key is to balance latency, correctness, and storage cost. In batch pipelines you can use a time‑based partitioning scheme and periodically re‑run the job on new partitions to capture late data.