In Spark, you receive data from a webhook that streams every second. After some records, blank records appear, then values again. How would you handle this in Spark to keep reading the data and store it in a table?

I would use Spark Structured Streaming with a source that reads the webhook payloads as a continuous stream (e.g., via a custom source or Kafka). The stream would be parsed into a DataFrame and a UDF would filter out or flag blank records. To avoid losing data, I’d use a watermark on a timestamp column and a stateful aggregation that keeps the last non‑blank value per key. The logic would look like:

val df = spark.readStream.format("kafka").option("subscribe", "webhook").load()
  .selectExpr("CAST(value AS STRING)")
  .withColumn("parsed", from_json(col("value"), schema))
  .withWatermark("timestamp", "10 minutes")
  .groupByKey(row => row.getAs[String]("id"))
  .flatMapGroupsWithState(OutputMode.Append(), GroupStateTimeout.NoTimeout()) { (id, rows, state) =>
    rows.map(r => if (r.getAs[String]("data") == "") state.getOrElse(r) else { state.update(r); r })
  }

The resulting stream is written to a Delta Lake table, which supports ACID transactions and schema evolution. This approach ensures continuous ingestion, handles intermittent blanks by carrying forward the last valid value, and stores clean data for downstream analytics.