Delete a delta table partition based on the creation/modification date of the partition folder
As I understand, there is a vaccum command in delta to remove the old versions from delta table. However, I would like to remove the partion based on the modification date of the partition.
eg: date=2024-11-22 is my partition name, this modification date is 2024-11-30, I want to remove the partions whose modification date is below my cutoff date which is 2024-12-01
Which is the correct approach to operate on Delta Files
New to Py Spark, parquet and delta ecosystem and little confused on seeing multiple ways to play with delta files.
Which is correct approach to operate on Delta Files
New to Py Spark, parquet and delta ecosystem and little confused on seeing multiple ways to play with delta files. Can someone help me understand on which one is correct or preferred?
Pyspark – read csv and saving it to delta format folder errors out ( no pandas)
New to Pyspark and trying to play with parquet/delta ecosystem.