Can you explain how joins work in Spark?

Spark supports several join types: inner, left, right, full outer, cross, semi, and anti. The Catalyst optimizer chooses the most efficient strategy based on data size, partitioning, and hints. For small dimension tables, a broadcast join is ideal: Spark sends the small table to all executors, avoiding a shuffle. For larger tables, Spark uses a shuffle hash join or a sort‑merge join. Shuffle hash joins hash the join key and redistribute data, while sort‑merge joins sort both sides on the key and merge them. Spark also supports a broadcast‑hash join hint (broadcast()) and a shuffle‑hash join hint (shuffle_hash()). The optimizer can also use a partition‑pruning strategy if the join key is partitioned. When designing a join, consider the size of each dataset, the available memory, and whether the join key is already partitioned. Using broadcast joins for small tables and ensuring both sides are partitioned on the join key can dramatically reduce shuffle cost and improve performance.