You've explained how schema versioning and watermarking help maintain feature consistency. Can you describe a specific challenge you faced during this implementation and how you resolved it to ensure the Smoke AI feature?

During the rollout of the Smoke AI feature, we discovered that the incoming event stream had a new optional field that was added after the initial schema. Because the feature store was using a strict schema, the new field caused deserialization errors and the pipeline stalled. I resolved this by integrating Confluent Schema Registry with Avro schemas and enabling backward compatibility. I added a versioned schema for the new field and updated the ingestion job to read the latest schema from the registry. For watermarking, I extended the watermark to account for the additional field’s timestamp, ensuring that late events were still processed correctly. I also added a feature‑store test harness that automatically runs unit tests against each schema version, catching drift before it reaches production. This approach preserved feature consistency while allowing the pipeline to evolve.