Have you worked with AWS Glue? What types of Glue jobs can be executed, and what types of jobs can use Glue?

Question

Assisting AI · Accepted Answer

AWS Glue offers several job types to accommodate different ETL workloads. The primary categories are:
1. **Spark ETL jobs** – written in Python (PySpark) or Scala, these jobs run on a managed Spark cluster and are ideal for large‑scale data transformations.
2. **Python shell jobs** – execute pure Python scripts without Spark; useful for lightweight tasks such as data validation or invoking external APIs.
3. **Scala shell jobs** – similar to Python shell but for Scala code.
4. **Glue Studio visual jobs** – a drag‑and‑drop interface that generates Spark code behind the scenes, suitable for users who prefer a GUI.
5. **Glue Crawlers** – not jobs per se, but they scan data stores, infer schemas, and populate the Glue Data Catalog.
Glue jobs can be triggered on a schedule, by an event (e.g., S3 file arrival), or manually. They integrate with other AWS services like S3, Redshift, Athena, and EMR. Choosing the right job type depends on data size, transformation complexity, and developer familiarity with Spark or Python.

Have you worked with AWS Glue? What types of Glue jobs can be executed, and what types of jobs can use Glue?

💡 Model Answer

🎤 Get questions like this answered in real-time