HomeInterview QuestionsHow do you read a CSV file in PySpark?

How do you read a CSV file in PySpark?

🟢 Easy Conceptual Fresher level
1Times asked
Sep 2026Last seen
Sep 2026First seen

💡 Model Answer

In PySpark you read a CSV file using the DataFrameReader API. The most common method is spark.read.csv("path/to/file.csv"). You can pass options to control the parsing: header="true" to treat the first row as column names, inferSchema="true" to automatically cast columns to appropriate types, sep="," for the delimiter, quote and escape for quoted fields, and nullValue to specify how to treat missing values. For example:

python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadCSV").getOrCreate()

df = (spark.read
      .option("header", "true")
      .option("inferSchema", "true")
      .csv("s3://bucket/data/*.csv"))

After loading, you can inspect the schema with df.printSchema() and preview data with df.show(). If you need to read a large dataset from S3 or HDFS, you can also set the path to a directory and Spark will automatically read all files in that directory. For performance, you can repartition the DataFrame or use the spark.read.option("maxFilesPerTrigger", "10") for streaming scenarios. The key points are using the correct options and ensuring the file path is accessible to the cluster.

This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.

🎤 Get questions like this answered in real-time

Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.

Get Assisting AI — Starts at ₹500