Spark Interview Questions

62 real Spark questions asked in live technical interviews — each with a model answer. Updated weekly.

🎤 Auto-captured by Assisting AI during live interviews

These Spark interview questions were captured from real interviews by candidates using Assisting AI. Each links to a full model answer. For real-time help during your own interview, get Assisting AI from ₹500/day.

When you execute this notebook, how many stages will be created in the backend?🟡 Medium · Conceptual · Asked 3×What are some join strategies in Spark, and when is each best used?🟡 Medium · Conceptual · Asked 2×You have a driver and executors. What is the role of each in Spark?🟢 Easy · Conceptual · Asked 1×How do you avoid data skew in Spark? What is sorting? What is broadcast join, and can you give an example scenario where you would use it?🟡 Medium · Conceptual · Asked 1×What is broadcast join in Spark, and why would you broadcast a small table across all executors?🟡 Medium · Conceptual · Asked 1×Can you explain how joins work in Spark?🟡 Medium · Conceptual · Asked 1×What is the difference between an anti join and a left anti join?🟡 Medium · Conceptual · Asked 1×You have hands‑on experience with PySpark, right? How does a PySpark job execute, and how do you run a PySpark job?🟡 Medium · Conceptual · Asked 1×How does PySpark distribute data across a cluster, and what Spark configurations would you use to process and transform large data files in a distributed manner?🟡 Medium · Conceptual · Asked 1×Suppose you have a join where one customer ID accounts for about 40% of the entire dataset. What problems can arise from this skew?🟡 Medium · Conceptual · Asked 1×Spark has a feature called AQE. What does AQE stand for and what does it do?🟢 Easy · Conceptual · Asked 1×You have a Spark job that normally runs in 15 minutes but today it ran for two hours. How would you debug this performance regression?🟡 Medium · Debugging · Asked 1×When would you use cache versus persist in Spark?🟡 Medium · Conceptual · Asked 1×What is the difference between cache and persist in Spark?🟢 Easy · Conceptual · Asked 1×In Spark, how do you configure components?🟢 Easy · Conceptual · Asked 1×What does excessive memory overhead mean?🟢 Easy · Conceptual · Asked 1×What is the difference between wide and narrow transformations in Spark?🟡 Medium · Conceptual · Asked 1×Is partitioning a narrow or wide transformation in Spark?🟡 Medium · Conceptual · Asked 1×Is groupBy a narrow or wide transformation in Spark?🟡 Medium · Conceptual · Asked 1×Is groupBy a wide partitioning operation in Spark?🟡 Medium · Conceptual · Asked 1×In Spark, where is shuffle data written and who reads it? Can you give an example of a transformation that triggers a shuffle?🟡 Medium · Conceptual · Asked 1×Can you explain what happens during a shuffle partition in Spark, especially during a wide transformation, and how data moves between executors?🟡 Medium · Conceptual · Asked 1×What is the default size threshold for broadcasting a table in Spark?🟢 Easy · Conceptual · Asked 1×What happens during a broadcast join in Spark?🟢 Easy · Conceptual · Asked 1×You have two tables: a customer table (~20–30 MB) and a transaction table (~50 MB). How would you join them efficiently, possibly using a broadcast join?🟡 Medium · Conceptual · Asked 1×Given a dataframe, how would you keep only the latest record for each customer?🟡 Medium · Coding · Asked 1×Given a denormalized dataset, how would you extract the desired columns or transform it into a normalized form using Python, Spark, or SQL?🟡 Medium · Coding · Asked 1×How does Spark’s Catalyst optimizer improve query planning for DataFrames?🟡 Medium · Conceptual · Asked 1×Why are Spark DataFrames and Datasets optimized in the backend compared to using RDDs?🟡 Medium · Conceptual · Asked 1×Why did Spark evolve from RDDs to DataFrames and Datasets?🟡 Medium · Conceptual · Asked 1×What is shuffling in Spark?🟢 Easy · Conceptual · Asked 1×Which repartition strategy is the best option for Spark code?🟡 Medium · Conceptual · Asked 1×How would you solve the small file problem?🟡 Medium · Conceptual · Asked 1×Have you ever come across the small file problem? What are the issues?🟡 Medium · Conceptual · Asked 1×When does Spark Adaptive Query Execution (AQE) happen?🟡 Medium · Conceptual · Asked 1×When writing CSV files in Spark, how does the partitioning strategy work? How many partitions are created? Do we need to apply any logic behind it, or is it random or default? Should we let Spark decide the number of partitions, or can we set a custom partitioning strategy? Provide an example.🟡 Medium · Conceptual · Asked 1×How does Parquet partitioning work for 100 to 200 GB of data? What is the logic behind the job?🟡 Medium · Conceptual · Asked 1×What are the advantages of converting a CSV file to Parquet in Spark, and how do worker types, partitions, and Adaptive Query Execution (AQE) affect this process?🟡 Medium · Conceptual · Asked 1×How can you optimize the conversion of a 100‑200 GB CSV file to Parquet using Spark, including partitioning and Adaptive Query Execution (AQE)?🔴 Hard · Conceptual · Asked 1×Suppose you have a 100‑200 GB CSV file stored on S3. How would you use AWS Glue and Spark to transform the data and load the final dataset into a target data warehouse (e.g., Redshift or Snowflake)?🟡 Medium · Conceptual · Asked 1×How do you optimize Spark jobs? What happens internally when you run a Spark job?🟡 Medium · Conceptual · Asked 1×How would you read and process this raw transaction log file in a Spark or Pandas ETL job?🟡 Medium · Conceptual · Asked 1×How do you perform basic operations on a dataframe?🟢 Easy · Conceptual · Asked 1×How would you design the cluster configuration for a 100 GB Spark job to ensure efficient resource usage and auto‑tuning?🔴 Hard · Conceptual · Asked 1×What is the default partition size in Spark, and how can it be configured?🟢 Easy · Conceptual · Asked 1×What types of clusters are available in Databricks?🟢 Easy · Conceptual · Asked 1×Explain how Amazon Kinesis works for real‑time data streaming and how you can run Spark code on Kinesis streams.🟡 Medium · Conceptual · Asked 1×Imagine you need to process a large daily dataset on E‑D. You are using PySpark on EMR, but the input volume suddenly doubles and the job starts missing its SLE because of skew and shuffle. How would you redesign the job and the EMR setup to keep it both fast and cost‑efficient?🟡 Medium · Conceptual · Asked 1×In a real project, tell me about a time you had to troubleshoot an EMR issue that affected performance. What specific steps did you take to isolate the root cause?🟡 Medium · Behavioral · Asked 1×How would you isolate and fix a skewed join in a PySpark job using Spark UI and AWS CloudWatch?🟡 Medium · Conceptual · Asked 1×How would you use a Spark window function with row_number() partitioned by customer and ordered by timestamp descending to get the latest order per customer?🟡 Medium · Coding · Asked 1×What is the difference between repartition and coalesce?🟡 Medium · Conceptual · Asked 1×What is repartitioning in Spark, and when would you use it?🟡 Medium · Conceptual · Asked 1×What is a PySpark job?🟡 Medium · Conceptual · Asked 1×What is a PySpark job?🟡 Medium · Conceptual · Asked 1×Can you explain how you assign clusters and lead a team?🟡 Medium · System Design · Asked 1×What are the different types of join strategies in Spark, and which one should you use at a time?🔴 Hard · System Design · Asked 1×What is the difference between repartition and coalesce in Spark, and how could choosing the wrong one lead to production issues?🔴 Hard · System Design · Asked 1×Can you explain the difference between repartition and partition in Spark, and what are the potential production issues if either one is chosen incorrectly?🔴 Hard · System Design · Asked 1×Can you explain the difference between repartition and partition in Spark, and what are the potential production issues if either one is chosen incorrectly?🔴 Hard · System Design · Asked 1×Can you explain the difference between repartition and partition in Spark, and what are the potential production issues if either one is chosen incorrectly?🔴 Hard · System Design · Asked 1×Can you explain the difference between repartition and partition in Spark?🟡 Medium · Conceptual · Asked 1×

🎤 Get Spark questions answered in real-time

Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.

Get Assisting AI — Starts at ₹500

Browse Other Topics