Write PySpark code to read a JSON file containing records with fields first_name and last_name. Drop records where first_name is null. Add a new column FULL_NAME by concatenating first_name and last_name. Then display all three columns only when FULL_NAME length is greater than 30 characters.
💡 Model Answer
You can accomplish this with the DataFrame API. First, read the JSON: df = spark.read.json("path/to/file.json"). Then drop rows where first_name is null: df = df.filter(col("first_name").isNotNull()). Next, create FULL_NAME: df = df.withColumn("FULL_NAME", concat_ws(" ", col("first_name"), col("last_name"))). Finally, filter by length: df.filter(length(col("FULL_NAME")) > 30).select("first_name", "last_name", "FULL_NAME").show() The concat_ws function adds a space between names; length() returns the number of characters. This pipeline is O(n) in the number of rows, with constant‑time operations per row. If the JSON file is large, Spark will automatically partition the data and process it in parallel. Remember to import the necessary functions: from pyspark.sql.functions import col, concat_ws, length. This code will drop null first_name rows, build the full name, and only display rows where the full name exceeds 30 characters.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500