Have you heard of AWS Glue crawlers? How would you handle schema drift in a crawler?

Schema drift occurs when the structure of data changes over time, such as new columns or changed data types. In Glue, you can handle it by enabling the "Update schema" option in the crawler configuration, which allows the crawler to add new columns to the catalog table. For more control, you can use a custom classifier that matches the new schema and set the "Crawler output" to "Update existing table". Glue also supports schema versioning via the Glue Data Catalog; you can enable versioning and keep previous schemas for audit. If you need to preserve data integrity, you can set the crawler to "Add new columns only" and use a Glue job to transform the data into a stable schema before writing to the target. Additionally, you can use the Glue Schema Registry to enforce a schema contract and detect drift at the application level. By combining these options, you can automatically adapt to schema changes while maintaining downstream compatibility.