How does PySpark distribute data across a cluster, and what Spark configurations would you use to process and transform large data files in a distributed manner?

PySpark distributes data by partitioning RDDs or DataFrames across the executors in the cluster. When a DataFrame is created from a file, Spark uses the Hadoop InputFormat to split the file into blocks (default 128 MB). Each block becomes a partition that can be processed in parallel. The number of partitions is controlled by the configuration spark.default.parallelism (for RDDs) or spark.sql.shuffle.partitions (for shuffle operations). For large files you can increase the number of partitions to match the number of cores, use compression (e.g., Parquet with snappy), and enable column pruning. You can also coalesce or repartition to balance skew. In a distributed transform, you write transformations as lazy operations; Spark builds a DAG and schedules stages across executors. Key configurations include spark.executor.memory, spark.executor.cores, spark.driver.memory, and spark.sql.autoBroadcastJoinThreshold to control broadcast joins. By tuning these settings you ensure that data is evenly distributed, memory is sufficient, and shuffle traffic is minimized, leading to efficient distributed processing.