Relative Content

Tag Archive for pythonpysparkdatabricksdelta-live-tablesdlt

DELTA LIVE TABLES – apply_changes from source dataset not working

I am using delta live tables in medallion architecture.I have columns id, timestamp, operation, deleted_flag. I have operations “Insert” and “delete”. When the operation is “insert” the deleted_flag will be null and for “delete” it will be yes. This is correctly populating in bronze table. but for silver table i am taking bronze table as source and performing apply_changes. But after that for series of insert and delete operations for same id , if last operation is “insert” then the deleted flag should be null. But i am getting as yes. Can someone please help me. I don’t want to delete old records in silver table

DELTA LIVE TABLES – apply_changes from source dataset not working

I am using delta live tables in medallion architecture.I have columns id, timestamp, operation, deleted_flag. I have operations “Insert” and “delete”. When the operation is “insert” the deleted_flag will be null and for “delete” it will be yes. This is correctly populating in bronze table. but for silver table i am taking bronze table as source and performing apply_changes. But after that for series of insert and delete operations for same id , if last operation is “insert” then the deleted flag should be null. But i am getting as yes. Can someone please help me. I don’t want to delete old records in silver table