feat: Investigate switch to `curl` for downloading things in modules
Closed this issue · 4 comments
Intro
We currently use wget for this functionality in all of our modules where downloading things from the internet is required.
While wget works alright, I think that we should investigate using curl instead, as it's reported as a more reliable tool for this purpose.
curl's log output doesn't contain URL it downloads, as it has less verbose output in general compared to wget, so that needs to be manually added for troubleshooting purposes.
Notes
@jonerrr faced & reported the issue in Discord with gnome-extensions module when curl was used.
It partially downloaded the ZIP file without reporting about any errors & caused the module to fail in installing the extension. This only happened once, while every other usage was successful, including mine.
Failed curl:
https://github.com/jonerrr/os-image/actions/runs/8639540324/job/23686029150#step:2:2607
Successful curl:
https://github.com/fiftydinar/gidro-os/actions/runs/8650263976/job/23718377093#step:2:1682
Notice that the file-size is different, yet the failed one reports "100%" download percent.
It is not clear if it was the server issue with https://extensions.gnome.org/, with my usage of curl with curl -L command or with curl as a tool itself.
Advantages
curl supports much more formats for downloading things than wget does.
Notably, ftp is not supported in wget.
curl doesn't support renaming the downloaded file if the same-named file already exists, like wget.
wget adds the .1, .2, etc. suffix in this case, which is not desirable if full extension for the file is required (like for rpm-ostree module when downloading repos).
https://unix.stackexchange.com/questions/460676/making-wget-rename-existing-filenames-sanely
Generally, I find the .1 .2 behaviour less desirable than overwriting or giving an error, because that creates a situation where the file path is different than expected.
wget follows redirects, curl I think not. Otherwise curl -Lo should be a fine replacement. Bazzite uses it everywhere.