variables flag not respected for agency ncro
Closed this issue · 4 comments
populate_repo --agencies ncro --variables flow --dest raw
results in all ~6000 data streams to be downloaded.
I'll see what I can do. You have a use case for this?
At some point the cumulative issues with NCRO may motivate us to try to create a downloader based on web services.
Besides the fact that I don't have a use case for this, my concern here is that there are not a total of 6000 files in /raw for all the agencies, let alone NCRO. Should be 2000 across all the agencies in large time shards. Moreover, in my downloads NOAA is so dominant in its slowness (slightly USGS) that it would be the focus of any optimization.
NOAA is fixed using the executor ( i followed your pattern).
For NCRO it is not possible as it does all variables no matter what is specified.
Ready for testing