osteele/matrix-archive

Download image script aborts as soon as an image cannot be downloaded

yayayayaka opened this issue · 0 comments

When an image cannot be downloaded (for example when the hosting server has gone extinct) ,the script aborts with an AssertionError without downloading the rest of the images.

Stack Trace:

$ python download_images.py --no-thumbnails
Skipping 872 already-downloaded images
Downloading 73 new images...
Traceback (most recent call last):
  File "download_images.py", line 63, in <module>
    download_images()
  File "/home/naka/.local/share/virtualenvs/matrix-archive-j2607j-8/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/naka/.local/share/virtualenvs/matrix-archive-j2607j-8/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/naka/.local/share/virtualenvs/matrix-archive-j2607j-8/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/naka/.local/share/virtualenvs/matrix-archive-j2607j-8/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "download_images.py", line 59, in download_images
    run_downloads(new_messages, download_dir, prefer_thumbnails=thumbnails)
  File "download_images.py", line 23, in run_downloads
    assert res.status_code == 200
AssertionError

Rather than asserting a 200 response, I'd propose catching the requests.exceptions.RequestException and continue downloading the next images.