I am working with large datasets using PySpark and need to process my data in chunks of 500 records each. I am contemplating between converting my Spark DataFrames to Pandas DataFrames using toPandas() for easy chunking or sticking with Spark RDDs and using foreachPartition() to manually handle the chunking.