How would you retrieve the top two orders by amount for each store for a given month using PySpark?

In PySpark you can achieve the same result using the Window API. First import the necessary functions:

from pyspark.sql import Window
from pyspark.sql.functions import row_number, col

Define a window that partitions by store_id and orders by amount descending:

w = Window.partitionBy('store_id').orderBy(col('amount').desc())

Add a row number column and filter:

df_top2 = (orders_df
           .filter((col('order_date') >= '2023-08-01') & (col('order_date') <= '2023-08-31'))
           .withColumn('rn', row_number().over(w))
           .filter(col('rn') <= 2)
           .select('store_id', 'order_id', 'amount', 'order_date'))

orders_df is a DataFrame containing the order data. The row_number() function assigns a unique rank within each store, and the subsequent filter keeps only the top two amounts. This approach is efficient because Spark applies the window operation in a distributed manner, and the filter reduces the data early. If you need the top N per group for a dynamic N, replace row_number() with rank() or dense_rank() accordingly.