What scheduling tool did you use for your pipelines, how did it handle failures, and what was your experience with DBT and building trusted dashboards? Additionally, have you worked on streaming data pipelines, and how did a change you made on your laptop get to production?

I have used Apache Airflow as the primary scheduler for batch pipelines. Airflow’s DAGs define tasks, and the scheduler triggers them based on cron or event triggers. Failure handling is managed via retries, SLA alerts, and alerting through Slack or email. For DBT, I used it to transform raw data in Snowflake; DBT models are version‑controlled, and the dbt run command is invoked in a CI job. I also built trusted dashboards in Looker, where I defined data models and used LookML to enforce data quality checks. For streaming, I worked with Kafka and Spark Structured Streaming; the pipeline ingests events, processes them in near‑real‑time, and writes to a Delta Lake. To promote a change from my laptop to production, I followed a CI/CD flow: commit code to Git, run unit tests, build Docker images, push to ECR, and deploy via ArgoCD to the Kubernetes cluster. The pipeline automatically picks up the new image, runs integration tests, and, upon success, promotes the change to the production namespace. This end‑to‑end process ensures that both batch and streaming workloads are reliably updated.