In your current project, how are you running your Python job?

In my current project, I run Python jobs as part of a data‑processing pipeline orchestrated by Apache Airflow. Each DAG contains a PythonOperator that calls a PySpark script via spark‑submit. The script is packaged in a Docker image that contains the required libraries and a virtual environment. The Airflow worker nodes run on a Kubernetes cluster, so the job is submitted to a Spark cluster on the same cluster. For smaller ETL tasks that do not require a full Spark cluster, I use a local PySpark session inside a virtualenv and schedule it with cron. When the job needs to process large volumes of data, I submit it to an EMR cluster, specifying the number of executors, memory, and cores through the spark‑submit arguments. I also use the AWS Glue job wrapper for serverless execution when the workload is sporadic. This approach gives me flexibility to run lightweight scripts locally and scale up to a distributed cluster when needed.