crash downloading large library

Question

crash downloading large library

Oryhara opened this issue 2 years ago · 5 comments

I bought this bundle: https://itch.io/bundle/download/fCCJhYMFZALEAiZP7yTQ6UTHjm7WHfNdoGrsnxez
and now i'm trying to download them all locally.
Installed this app per instructions. running command as indicated in readme.

It repeatedly gives this error:

Downloaded Clerics of Creation: Apocalypse and Genesis Domains (25 of 505)
Traceback (most recent call last):
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests-2.28.1-py3.11.egg\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\json\decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\downloader_main.py", line 66, in
main()
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\downloader_main.py", line 62, in main
lib.download_library(args.platform)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\library.py", line 108, in download_library
for _ in r:
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\library.py", line 99, in dl
g.download(self.login, platform)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\game.py", line 75, in download
self.do_download(d, token)
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\itchiodl\game.py", line 136, in do_download
j = r.json()
^^^^^^^^
File "C:\Users\oryhara\AppData\Local\Programs\Python\Python311\Lib\site-packages\requests-2.28.1-py3.11.egg\requests\models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I'm having limited success running it repeatedly with thousands of jobs. But of the 254 creators in this bundle i've only gotten 68 downloaded so far.

Answer 1 · 2023-03-17T15:39:47.000Z

Could you be hitting some kind of rate limit? I think there have been issues people have had if connecting from a cloud box

Answer 2 · 2023-03-20T15:54:54.000Z

Idk what a cloud box is. I'm running this locally on a laptop using powershell.
I'm not sure if rate limiting makes sense because I had more success when I added a really high number of jobs.
and initially i ran with just 1 job.

Answer 3 · 2023-03-22T09:53:00.000Z

I'm having the same issue on win 10 LTSC, with the added issue of "I have no idea where the downloads are going, and so cannot check if they're actually working". ^^; A fix would be good, or at least instructions for someone who doesn't work with python very much. (I uh. Also don't know how to change the amount of jobs. ^^; )

Answer 4 · 2023-03-23T20:38:33.000Z

I'm having the same issue on win 10 LTSC, with the added issue of "I have no idea where the downloads are going, and so cannot check if they're actually working"

Whatever directory the shell is currently in

Answer 5 · 2024-02-19T07:37:36.000Z

i am having the very same issue with the same bundle, but on a different game. mine stopped working after seyvoth manor. i know everything up till that point has downloaded, but after that it just freezes up and stops. here's the message it gives me:

Traceback (most recent call last):
File "", line 198, in run_module_as_main
File "", line 88, in run_code
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\downloader_main.py", line 66, in
main()
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\downloader_main.py", line 62, in main
lib.download_library(args.platform)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\library.py", line 108, in download_library
for _ in r:
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 619, in result_iterator
yield _result_or_cancel(fs.pop())
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 317, in _result_or_cancel
return fut.result(timeout)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\library.py", line 99, in dl
g.download(self.login, platform)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\game.py", line 75, in download
self.do_download(d, token)
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\game.py", line 187, in do_download
if utils.md5sum(out_file) != d["md5_hash"]:
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\site-packages\itchiodl\utils.py", line 51, in md5sum
with path.open("rb") as f:
^^^^^^^^^^^^^^^
File "C:\Users\pickl\AppData\Local\Programs\Python\Python312\Lib\pathlib.py", line 1013, in open
return io.open(self, mode, buffering, encoding, errors, newline)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'ladyorthetiger\birb-heist\Birb Heist — For Everyone.pdf'

that last filenotfound is actually one of the files that successfully downloaded. also another thing that may or may not be relevant is this bundle is like all pdfs and images, as it's a ttrpg bundle.
I'm not excellent at interpreting the errors it's throwing, any advice?
im on a laptop with windows 11 running this in the normal command prompt
thanks