I have blob data with in different folder by year, month and date(nested folder) refreshing daily.
I need to design a pipeline which will efficiently load the historical data from blob to azure databricks. Could you please guide the proper way of storing daily and historical data in databricks?