Replace astropy.utils.data.download_file with httpx
Closed this issue · 8 comments
download_file() from astropy.utils.data can lead to various Exceptions with the univie webserver, e.g.
urllib.error.ContentTooShortError: <urlopen error File was supposed to be 10713600 bytes but we only got 10713359 bytes. Download failed.
This bothers end users, and makes our tests fail unreliably.
Replacing download_file() with equivalent functionality from httpx with proper retry functions seems to 'fix' this problem. So we should use that everywhere.
Places where we use download_file():
-
ScopeSim/scopesim/server/example_data_utils.py
13:from astropy.utils.data import download_file -
ScopeSim/scopesim/server/OLD_database.py
12:from astropy.utils.data import download_file -
ScopeSim/scopesim/effects/ter_curves_utils.py
9:from astropy.utils.data import download_file -
Pyckles/pyckles/utils.py
3:from astropy.utils.data import download_file -
ScopeSim_Templates/scopesim_templates/extragalactic/galaxies.py
8:from astropy.utils.data import download_file -
irdb/MICADO/docs/example_notebooks/3_scopesim_SCAO_4mas_fv-psf.ipynb
275:"from astropy.utils.data import download_file\n",
So this issue transcends just ScopeSim.
Maybe some of those might also be better off using pooch, especially that one notebook. See also AstarVienna/speXtra#10
Thanks, pooch seems a nice way to fetch things. Assuming it doesn't have the same problem..
Thanks, pooch seems a nice way to fetch things. Assuming it doesn't have the same problem..
Done with #446 I guess. I forgot we already had the pooch conversation half a year ago 😂
The one in ScopeSim_Templates was removed in AstarVienna/ScopeSim_Templates#88.
The one in pyckles seems to have caused this: https://github.com/AstarVienna/ScopeSim/actions/runs/10233695289/job/28316097461
The one in the MICADO notebook was removed in AstarVienna/irdb#185
4 down, 2 to go 🚀
The Pyckles one was replaced in AstarVienna/Pyckles#9