Load recurring CSV exports automatically
Recurring file exports are the most common manual data job there is: someone downloads a CSV, tidies it, and pastes it somewhere. Reading the file from its URL on a schedule removes the person from the loop, and the cleaning happens the same way every time.
How to set it up
- 1
Get a stable link
It must return the current file. A URL that keeps returning last week’s export will sync perfectly and tell you nothing.
- 2
Pin the format
Detection reads the extension, then the Content-Type. Override it when a URL lies about what it serves, which is more often than it should be.
- 3
Clean on the way through
Select columns, filter rows, cast types. Doing it in the pipeline means it happens identically every run, rather than depending on who did the tidying.
- 4
Upsert on the file’s own ID
Otherwise every run appends the whole file again. This is the single most common mistake with recurring file loads.
Questions
What if the file is not public?
Upload it to your account storage and read it from there. The same job configuration works either way — only the location changes.
How big can the file be?
256 MB by default. It is streamed rather than loaded whole, so the cap is about keeping the pipeline predictable rather than about memory.