macearl/Wallhaven-Downloader

Not downloading all images

flallnatural opened this issue · 4 comments

I run into an issue where the downloader does not download every image available when I use a query for a specific tag. Say a tag has 100 wallpapers associated to it. On the first pass I'll get like 60 of them. Then I have to do subsequent passes to get it to download most of them. A lot of times it wont get all the wallpapers (like 80-90% of them). I know it looks for duplicates but there are flat out unique images that it misses no matter how many subsequent runs I make. This issue occurs with or without parallels.
Am I missing some setting here?

I just tried it with the tag "#pencils" ("id:1668") which currently has 179 Images, after the script finished i have 179 images + the downloaded.txt file in my directory.

It seems to work for me.

Do you get an error for the images that are not downloaded? Could you post your settings?

I used:

WPNUMBER=480
STARTPAGE=1
TYPE=search
CATEGORIES=111
FILTER=111
MODE=date_added
QUERY="id:1668"

without setting an aspect ratio or resolution:

I'm almost matching those settings plus a few extras (also for query I'm leaving it blank in file and instead using -q '' when initiated it from terminal:
WPNUMBER=480
STARTPAGE=1
TYPE=search
CATEGORIES=101
FILTER=111
MODE=date_added
ORDER=desc
COLLECTION=""
QUERY=""
COLOR=""
SUBFOLDER=1
USR=""
PARALLEL=1
THUMBS=24

Is the thumbs parameter correct? Should I use parallels or not?

I would like to mention that i remember when i first downloaded the script months ago it worked as you described but for some reason for the past few months it hasn't worked the same way.

The options seem fine.
THUMBS has to be set to the same value you chose in your account as Thumbs Per Page ( https://wallhaven.cc/settings/browsing )

Try it with PARALLEL=0 as enabling it can lead to problems with cloudflare.
(although for me it also works with it enabled)

If you don't receive any errors you could also try removing the -q in

wget -q --header="$httpHeader" --keep-session-cookies \
and posting the output of a run, maybe there is something in there, that points to the problem

I tried it again with your settings and ./wallhaven.sh -q "id:1668" and it grabbed all available wallpapers.

You know what. That might have been it. My Thumbs was set to 24 in the script but was 32 on my account. I turned off parallels and set the correct Thumbs and it worked. With Parallel on it looks like it still misses a few but that might be because of cloudfare like you mentioned.

Thanks for your help! Turns out it was something simple