You have experience with PySpark. Could you explain the two main components of PySpark?

PySpark exposes two primary APIs that developers use to build distributed data pipelines: the low‑level RDD API and the higher‑level DataFrame/Dataset API. RDDs (Resilient Distributed Datasets) are immutable, partitioned collections of objects that provide fine‑grained control over transformations and actions. They are fault‑tolerant and allow custom logic but require manual optimization and are less efficient for many common operations. DataFrames, on the other hand, are schema‑aware tabular data structures that map closely to relational tables. They support a declarative API, allow Spark’s Catalyst optimizer to automatically rewrite queries, and use Tungsten for efficient execution. DataFrames are the recommended API for most use cases because they provide better performance, easier integration with SQL, and richer built‑in functions. In practice, developers often start with DataFrames for data ingestion, cleaning, and aggregation, and fall back to RDDs when they need custom transformations that cannot be expressed in DataFrame operations.