Consider you are building a streaming job that consumes CDC records from Amazon Kinesis. How do you enforce exactly‑once processing and handle out‑of‑order records?

Question

Assisting AI · Accepted Answer

To achieve exactly‑once semantics in a Glue Streaming job that reads CDC records from Kinesis, you need to combine several AWS services and design patterns:

1. **Source** – Use Kinesis Data Streams with a retention period that covers the maximum expected delay. Enable *Enhanced Fan‑Out* to reduce read latency.
2. **Checkpointing** – Glue Streaming uses the Kinesis Client Library (KCL) under the hood. Store the last processed sequence number in a durable store (e.g., DynamoDB or S3). Glue automatically writes checkpoints to the checkpoint location in S3, which is used to resume processing after a failure.
3. **Event‑time ordering** – Each CDC record contains a transaction timestamp. In the Glue job, parse this timestamp and assign it as the event time. Use a watermark (e.g., 10 minutes) to allow late events while still closing windows.
4. **Stateful processing** – For out‑of‑order handling, maintain state keyed by the primary key of the CDC record. Use Glue’s stateful transformations (e.g., `groupByKey` + `reduce`) to merge late events with the latest state. If a record arrives with a timestamp older than the current state, you can either ignore it or merge based on business rules.
5. **Idempotent writes** – Write the final state to a target that supports idempotent operations, such as DynamoDB with conditional writes or S3 with a deterministic key (e.g., hash of the primary key). Alternatively, use a transactional write service like Amazon RDS with `INSERT … ON DUPLICATE KEY UPDATE`.
6. **Exactly‑once delivery** – Combine the checkpointing and idempotent writes: the job only commits a checkpoint after the write succeeds. If the job crashes before checkpointing, the next run will re‑process the same records but the idempotent write will prevent duplicates.
7. **Monitoring & retries** – Use CloudWatch metrics for checkpoint lag and Kinesis consumer lag. Configure Glue’s retry policy and dead‑letter queues for transient failures.

**Complexity**: The per‑record processing is O(1) for timestamp extraction and state lookup. The overall throughput is bounded by the Kinesis shard count and the write latency to the target. The design ensures that even with out‑of‑order arrivals, the final state reflects the latest CDC snapshot without duplicates.

Consider you are building a streaming job that consumes CDC records from Amazon Kinesis. How do you enforce exactly‑once processing and handle out‑of‑order records?

💡 Model Answer

🎤 Get questions like this answered in real-time