How do you handle late arrival data in a streaming pipeline?

Late arrival data is common in event‑time streaming. The typical approach is to use watermarks to define how late data can be accepted. In Spark Structured Streaming, you set a watermark with a delay (e.g., 10 minutes) and then define a window. Data arriving after the watermark is considered late and can be dropped or sent to a side output. In Flink, you configure a watermark strategy and allowed lateness. You can also use a stateful operator to keep late events for a short period before discarding. If you need to preserve late data, you can write it to a separate sink and later re‑process it. It’s important to monitor the lateness metrics and adjust the watermark delay based on the observed latency. Additionally, ensure that your source (Kafka, Pulsar) has sufficient retention so that late events are still available. By tuning watermarks and allowed lateness, you can balance correctness and latency.