How does Parquet partitioning work for 100 to 200 GB of data? What is the logic behind the job?

Parquet partitioning in Spark works by adding a directory hierarchy based on one or more partition columns. When you write a DataFrame to Parquet with df.write.partitionBy("country","year"), Spark creates folders like country=US/year=2023/ and stores all rows with those values in the same folder. The logic behind the job is to enable partition pruning: when a query filters on a partition column, Spark can skip entire directories, reducing I/O. For 100–200 GB of data, you typically choose partition columns that have a moderate cardinality (e.g., daily or monthly dates, region codes) to avoid creating too many small files. Spark automatically splits the data into partitions based on the number of executors and the spark.sql.files.maxPartitionBytes setting (default 128 MB). Each partition will contain roughly that many bytes, resulting in many Parquet files. You can tune spark.sql.files.maxPartitionBytes and spark.sql.files.openCostInBytes to control file size and number. After writing, you can run spark.read.parquet("s3://bucket/path") and Spark will prune partitions based on query predicates, dramatically speeding up reads.