In PySpark, please write a code snippet to perform a join between two DataFrames.

Question

Assisting AI · Accepted Answer

Here’s a simple example that joins two DataFrames on a common key:

```python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("JoinExample").getOrCreate()

# Sample data
left_data = [(1, "Alice"), (2, "Bob"), (3, "Charlie")]
right_data = [(1, 100), (2, 200), (4, 400)]

left_df = spark.createDataFrame(left_data, ["id", "name"])
right_df = spark.createDataFrame(right_data, ["id", "score"])

# Perform an inner join on the 'id' column
joined_df = left_df.join(right_df, on="id", how="inner")

joined_df.show()
```

Output:
```
+---+-------+-----
| id|   name|score
+---+-------+-----
|  1|  Alice|  100
|  2|    Bob|  200
+---+-------+-----
```

Explanation: The `join` method takes the join column(s) and the join type (`inner`, `left`, `right`, `outer`). For large datasets, consider broadcasting the smaller DataFrame (`broadcast(df)`) or using a shuffle‑hash join to optimize performance. The complexity is O(n + m) for a hash join, where n and m are the row counts of the two DataFrames.

In PySpark, please write a code snippet to perform a join between two DataFrames.

💡 Model Answer

🎤 Get questions like this answered in real-time