faldez/tanoshi

DownloadChapters does not throttle api requests for chapter metadata

joeframbach opened this issue · 0 comments

joeframbach commented

Steps to reproduce:

  1. Visit a manga in the library with more than 30 chapters
  2. Click "edit"
  3. Click "select all"
  4. Click "download"

Result:

  • Only the first 30 or so chapters have been added to the download queue.
  • Logs show ERROR tanoshi::application::worker::downloads] failed to insert queue, reason (URL): status code 429

Explanation:

I'm being throttled by mangadex in the chapter metadata api request.
which is probably originating from https://github.com/faldez/tanoshi/blob/main/crates/tanoshi/src/application/worker/downloads.rs#L109.

The DownloadWorker uses a sleep(1) to handle throttling, and sends itself a Download command each loop. But the other commands (InsertIntoQueue) don't sleep. Nor would you want them to sleep, because the queue hasn't been written to the db yet. I suggest writing an incomplete row to the queue table with just the chapter ids wanted, then fill in the page ids when the queue picks it up. Like, different kids of "want"s in the queue table - 1. Get this chapter id and stuff the page ids into the download queue. 2. Get this page id and download the image