Only transfer data that has changed

Let’s be honest. How many times do you transfer data from A to B because there might be something changed, but actually there is nothing changed.

Because simple copy tasks don’t take up a lot of time it’s not considered important to write code that checks whether source data has changed. However when working with a generic datawarehouse it’s relatively easy to implement check ‘is the source changed?’ no, then do nothing.

Implementation:

indexed last_update_dt colunmn in source.
compare this with meta-data last_update_dt for this specific transfer, src, target combination.  It can also be done with 100 % meta-data.

 

 

2 thoughts on “Only transfer data that has changed”

Leave a Reply

Your email address will not be published. Required fields are marked *