How to import only the new data and new records that have changed using the ETL process in a data warehouse?
I have an ETL process that allows me to load data from one database to another, applying transformations along the way. The process currently starts by deleting all records from all tables, and then reloading everything to avoid duplicates. Over time, this ETL process is taking a lot of time, as the data is being updated daily.