If a CSV file has a large number of rows, how would you process it?

Processing a large CSV requires streaming or chunked processing to avoid memory exhaustion. I would use a library like pandas with the chunksize parameter, iterating over each chunk, performing the necessary transformations, and writing the results incrementally to disk or a database. For even larger datasets, I’d switch to a distributed engine such as Spark, which can read the CSV in parallel across a cluster, apply transformations in a lazy, optimized manner, and write the output in a columnar format. If the data needs to be queried frequently, I’d convert it to Parquet or ORC, partition it by a meaningful key, and load it into a data warehouse or a NoSQL store. This pipeline ensures scalability, fault tolerance, and efficient resource usage.