What tools do you use in your end-to-end production pipeline, and how do they fit together?

In my most recent role I built an end‑to‑end pipeline that moved data from raw ingestion to a data warehouse and downstream analytics. The core stack is Airflow for orchestration, Spark for large‑scale batch and streaming transformations, and Kafka for real‑time event ingestion. Raw files land in S3, where they are catalogued by Glue and processed by Spark jobs that write to a Delta Lake layer for ACID semantics. The processed data is then loaded into Snowflake for analytics and BI. I also use Datadog for monitoring, Prometheus for metrics, and PagerDuty for alerting. The pipeline is containerized with Docker and deployed on Kubernetes, which gives us horizontal scaling and self‑healing. I write unit tests for each transformation and use CI/CD pipelines (GitHub Actions + ArgoCD) to promote changes through dev, staging, and prod. This combination gives me a robust, observable, and scalable production pipeline.