grrttedwards/wow-addon-updater

curse download fails again - possible cloudscraper fix

Opened this issue · 8 comments

ilu33 commented

Describe the bug
All curse downloads fail with "403 None"

If you run pipenv update on the command line and try again, does the issue go away?
No

Log output
All curse downloads fail with "403 None" - no need to post the same log entry over and over.

I see that requests are made with HTTP/1.1. According to
https://stackoverflow.com/questions/65604551/cant-bypass-cloudflare-with-python-cloudscraper
it might help to

  • simulate a browser as explained by dcts and
  • use http2 like explained by Zorome

Sadly I'm not good enough to know where exactly to place these things in your code.

Deleted user commented

I just found a github user who mirrors curseforge addons. They also have a fork of cloudscraper with 34 new commits since the last update on upstream. This may be a solution, but I have not tried it. https://github.com/curseforge-mirror/cloudscraper

DUTNO1 commented

It's the hardcoded UserAgent that's mucking things up, in updater/site/__init__.py change CURSE_UA from:
CURSE_UA = {'custom': 'Chrome/80'}
to:
CURSE_UA = {'browser': 'chrome', 'platform': 'windows', 'mobile': False} - as mentioned in the SO thread and you should be good to go.

  • 👍1
grrttedwards commented

@DUTNO1
Thanks. Yeah the user agent trick was a random discovery to fix Curse blocking years ago.

I may use Dragonflight as a reason to come and update this code. But it's tough for me to find the time. I appreciate everyone's dedication to this tool!

ilu33 commented

@DUTNO1
Changing that did not help. Does it work for you?

@omicron-o
Do you have any idea how one could include the modified library you mentioned instead of the on provided by pip? Is there a way to create a temporary local override to test it?

DUTNO1 commented

@ilu33 welp, it seems like they "fixed" it - doesn't work for me anymore either.

grrttedwards commented

Looks like it's spontaneously working fine again:

Installing/updating addon: bagnon to version: 10.0.3...
...
Installing/updating addon: advancedinterfaceoptions to version: 1.8.1...

Name                        Prev. Version               New Version
────                        ─────────────               ───────────
hear-kitty                  -----                       1.10.0
claw                        -----                       9.0.2-6
...
deadly-boss-mods            -----                       DBM-10.0.0
Press any key to continue . . .
Deleted user commented

I suspect that sometimes it breaks because curse/cloudflare sometimes puts the site into a higher level of protection, causing some "v2" captcha to pop up that the library does not support. I'm not super sure though, just guessing based on log/exception output.

arsirantala commented

Would love that this issue is solved somehow - I checked where the addon files are actually downloaded (in curseforge's case), and they are hosted in mediafilez.forgecdn.net.

The url's suffix follows the addon name and its version.. I think it should be pretty straight forward to form a valid url in the script for the addon file from forgecdn to be downloaded (without headers/cookies been set).