Write PySpark code to filter rows where status is 'success' and aggregate the total amount per customer using groupBy and sum.
💡 Model Answer
The provided code demonstrates how to use PySpark’s DataFrame API to filter rows and aggregate data. First, filter(col('status') == 'success') keeps only rows where the status column equals 'success'. Then groupBy('customer_id') groups the remaining rows by customer ID. Finally, agg(sum('amount').alias('total_amount')) calculates the sum of the amount column for each group and names the resulting column total_amount. The resulting DataFrame contains one row per customer with the total amount of successful transactions. The operation is performed in a distributed manner; each partition computes partial sums, which are then combined. The time complexity is O(n) where n is the number of rows, and the space complexity is O(k) for k distinct customers. This pattern is common for summarizing transactional data in Spark.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500