Write a PySpark job that joins a 300 GB transaction dataset with a 2 GB customer dataset and calculates the total successful transaction amount per customer.

To process a 300 GB transaction file and a 2 GB customer file efficiently, I would use an AWS Glue ETL job that runs on Spark. First, read both datasets into DataFrames, ensuring the customer file is read with a small number of partitions to keep it in memory. Because the customer dataset is only 2 GB, I would broadcast it to all executors to avoid a shuffle. The code would look like this:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum as _sum

spark = SparkSession.builder.appName("TransactionJoin").getOrCreate()

# Load datasets
trans_df = spark.read.parquet("s3://bucket/transactions/")
cust_df = spark.read.parquet("s3://bucket/customers/")

# Broadcast the small customer table
cust_broadcast = spark.sparkContext.broadcast(cust_df.collect())

# Join on customer_id
joined = trans_df.join(cust_broadcast.value, "customer_id", "inner")

# Filter successful transactions and aggregate
result = (
    joined.filter(col("status") == "success")
    .groupBy("customer_id")
    .agg(_sum("amount").alias("total_success_amount"))
)

# Write the result back to S3 or Redshift
result.write.mode("overwrite").parquet("s3://bucket/aggregated/")

Key decisions: broadcasting the 2 GB customer table eliminates the expensive shuffle that would otherwise occur with a standard join. If the customer table grew larger, I would partition it by customer_id and use a sort‑merge join. The overall time complexity is O(n + m), where n is the number of transactions and m is the number of customers. Spark handles the parallelism, so the job scales with the cluster size. This approach ensures the job completes in a reasonable time while keeping memory usage bounded.