Downloader broken after wallhaven upgrade
Maveee opened this issue · 3 comments
You may (or may not) know that wallhaven is out of alpha since of today!
I have an extensive wallpaper collection myself and scraping the latest 1920x1080 and 4k wallpapers using your tool is a daily habit for me.
However as of today the scraper doesn't work. The wallhaven devs obviously changed some of the wallpaper structure, which broke this magnificent script.
https://i.imgur.com/UHqeY3I.png
Could you please update your script to work with the non-alpha version of wallhaven?
Thanks in advance! Love this tool.
I was aware of the end of the alpha ;)
The site went live again during nighttime here. I had a first look at the new API and I will probably push an update in the next couple of hours.
The update will probably mean that a new dependency ( jq ) needs to be installed. And I might have to disable the option to download user specific uploads and favorites at least temporarily as I'm not sure if there is an actual demand for these options and the API does not have endpoints for it, so I would have to hack it in again.
Edit: oh also old downloaded.txt files will probably be useless now as all wallpaper names changed, this will probably mean that everyone with a large local collection will download wallpaper again that they already have under a different name, there is probably nothing I can do to prevent that.
Edit2: just pushed the update, besides favorites everything should work again.
Thank you for updating the code @macearl !
I have updated my scripts. I do notice that it's a bit slower than before, I'm guessing because of the API's limitations?
If anything it should be faster than before, the main reason for the API is probably the reduced load on the server by serving smaller files
The json the API returns for a search is roughly 1/3 the size of the html which was downloaded before.
But i also haven't used the script in quite some time and don't have an exact number to compare it to.
maybe the new server plays a role as well
If you want to speed things up considerably try the parallel option (be sure to pull the latest version, i just fixed a mistake i made yesterday)
edit: i doubt that this is the cause for your slowdowns but keep in mind that the API is limited to 45 requests per minute. The script makes 1 API call per page it downloads so if you want to reduce the amount of API calls consider changing the Thumbs per page option in your wallhaven account and the script