What did you do when a batch load introduced bad rows?

When a batch load introduced bad rows, we treated it as a data quality failure and followed a three‑step process. First, we validated incoming data against a schema using a library like Apache Avro or Spark’s DataFrame schema enforcement; any row that didn’t match was flagged. Second, we routed bad rows to an error table that stored the raw record, the error type, and a timestamp. This allowed downstream teams to investigate without blocking the main pipeline. Third, we set up automated alerts in Airflow that sent an email to the data steward when the error rate exceeded a threshold. For large volumes, we used Spark Structured Streaming to process data in micro‑batches, which made it easier to isolate and retry bad records. We also maintained a data quality dashboard that tracked metrics such as row count, error count, and data freshness, giving us visibility into the health of our ingestion process.