How do you read a CSV file in PySpark?
💡 Model Answer
In PySpark you read a CSV file using the DataFrameReader API. The most common method is spark.read.csv("path/to/file.csv"). You can pass options to control the parsing: header="true" to treat the first row as column names, inferSchema="true" to automatically cast columns to appropriate types, sep="," for the delimiter, quote and escape for quoted fields, and nullValue to specify how to treat missing values. For example:
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ReadCSV").getOrCreate()
df = (spark.read
.option("header", "true")
.option("inferSchema", "true")
.csv("s3://bucket/data/*.csv"))After loading, you can inspect the schema with df.printSchema() and preview data with df.show(). If you need to read a large dataset from S3 or HDFS, you can also set the path to a directory and Spark will automatically read all files in that directory. For performance, you can repartition the DataFrame or use the spark.read.option("maxFilesPerTrigger", "10") for streaming scenarios. The key points are using the correct options and ensuring the file path is accessible to the cluster.
This answer was generated by AI for study purposes. Use it as a starting point — personalize it with your own experience.
🎤 Get questions like this answered in real-time
Assisting AI listens to your interview, captures questions live, and gives you instant AI-powered answers on a discreet on-screen overlay.
Get Assisting AI — Starts at ₹500