Did you decide whether to fail the entire job or quarantine just the bad records and continue processing?

When deciding between failing a job or quarantining bad records, I evaluate the impact on downstream consumers and the severity of the error. For critical data that feeds real‑time dashboards, I prefer to fail the job so that no corrupted data propagates. For batch analytics where a few bad rows are acceptable, I quarantine them by writing to a separate error table, logging the issue, and continuing. In Spark, I use the DataFrameWriter with mode('append') and a custom error handler that captures exceptions, writes the offending rows to an error sink, and records metadata. I also implement a retry policy: if the error is transient (e.g., a temporary network glitch), I retry the job a limited number of times before quarantining. This approach balances data integrity with system availability.