I have a process in Azure Data Factory that downloads about 55k images and stores them to a storage container for use on an external eCommerce website.
Sometimes I have to do a full sync/download, the issue is that it takes forever to download them all due to API limits etc. Are there any tips on how I can overcome it? My download process can take 4-5 days to complete if I have to reseed the data...I am just looking for something that isn’t as long.
Thanks!
Eric