Support for web-hosted data
Closed this issue · 3 comments
It would be a nice feature. urllib/urllib2 can be used to implement it. Looking at AstroPy's download_file might provide hints about things to watch out for, and how to go about implementation.
There's some question regarding RAM, and whether it would be desirable overall to write the downloaded file to disk rather than reading it into memory. It would complicate the code, but allow larger files to be used this way. Currently an on-disk cache exists only for the Viewer; however TempFile might be of use here. I am not sure it is worth it though, as the chance of someone trying to pull a file large enough to cause memory issues, and not want to download it for future use, seems rather low - and I do not think a real caching environment for this use is worth it.
I was thinking of the simple approach and using tempfile, leaving clean up for the system. I know that astropy uses an internal cache that prevents the same URL from being downloaded every time the script is run. That works fine for small files.