L4cache/kemono-dl

[Bug] Failed to download

Closed this issue · 9 comments

tezrilet commented

Version

Version: 2024.02.02

Your Command

python3 /root/kemono-dl/kemono-dl.py --cookies /root/kemono-dl/cookies-kemono-party.txt --inline --content --comments --json --dms --icon --banner --dirname-pattern \"{username}\\{service}\" --filename-pattern \"[{id}] ({published}) {title}\\{filename}.{ext}\" --inline-filename-pattern \"[{id}] ({published}) {title}\other\inline\{filename}.{ext}\" --other-filename-pattern \"[{id}] ({published}) {title}\other\{filename}.{ext}\" --date-strf-pattern \"%Y-%m-%d\" --from-file download.txt --archive archive.txt

The download.txt contains URLs of posts, 1 per line.

Description of bug

It's able to download the post, but some attachments result in an error. I have also tried giving full permissions (chmod 777) to the script, but that did not change anything.

How To Reproduce

Run the above command. Only certain posts will produce the error, but it's rare.

Error messages and tracebacks

ERROR:Failed to download: <ommitted>
Traceback (most recent call last):
  File "/root/kemono-dl/src/main.py", line 429, in download_attachments
    self.download_file(file, retry=self.retry, postid=post['post_variables']['id'])
  File "/root/kemono-dl/src/main.py", line 597, in download_file
    with open(part_file, 'wb' if resume_size == 0 else 'ab') as f:
PermissionError: [Errno 1] Operation not permitted: '<ommitted>'

Additional comments

I can provide specific URLs and full debug logs privately.

L4cache commented

I think I need the URLs and logs.
Is GPG ok?

-----BEGIN PGP PUBLIC KEY BLOCK-----

mDMEZs4zahYJKwYBBAHaRw8BAQdA8TwApK1RLGHAdLsAFbdi4DEhbDsD1Lxn89GG
kjniOvK0B2w0Y2FjaGWImQQTFgoAQRYhBF3LLv/z9Yf1FdTgsqnTkFJ1QgqkBQJm
zjNqAhsDBQkFo5qABQsJCAcCAiICBhUKCQgLAgQWAgMBAh4HAheAAAoJEKnTkFJ1
QgqkqtcA/jY2BI8g2Bc8Dp2tuYiDUVsUG1hJp3ik29EbLmMwEuk8AQCOmuKn9L3z
iSOnjuMIakkW1gAmSq5tTOpxwwUbkgzjArg4BGbOM2oSCisGAQQBl1UBBQEBB0C0
IIOa2bNBT1LGN5UGEgtVJaXyPpF2daGwh8hxF78bPwMBCAeIfgQYFgoAJhYhBF3L
Lv/z9Yf1FdTgsqnTkFJ1QgqkBQJmzjNqAhsMBQkFo5qAAAoJEKnTkFJ1QgqkKc8A
/iwfqpWqsQFkK4F3VwM5zOtghx5snedkc+FiiDNik6cmAP91OnNsiNeh0ZgWutmS
eHgGmb/L4xX8skCFOkPdqmSkDg==
=Vm3/
-----END PGP PUBLIC KEY BLOCK-----

L4cache commented

I got error when I tried to decrypt it, what happened? (I'm not very familiar with GPG)

$ gpg --output log.log --decrypt log.txt
gpg: no valid OpenPGP data found.
gpg: decrypt_message failed: Unknown system error

Maybe you should ascii armor it when you upload it as a txt file?

tezrilet commented

Sorry about that. I updated the gist, please try again.

L4cache commented

It is decryptable now. I have an idea but I'll have to inspect it later (won't be a very long time)

L4cache commented

I can't reproduce the error...
This problem is probably specific to your setup.
Can you elaborate on the environment you using? Assuming it's Linux, what distro are you using and what's the version of it? Are you using root user?

tezrilet commented

I think you're right, since my setup is a bit weird. It's Alpine Linux v3.17 in a virtual machine, but it outputs to a Windows filesystem through a shared directory. It can probably be reproduced on a Windows machine. I believe the issue might have to do with filenames that are too long, which happens with some image URLs.

I was also able to fix it by using {hash}.{ext} for --filename-pattern and --inline-filename-pattern, but a new problem occurs. The extractor sees 2 file attachments for the same file, but one without an extension so it downloads a duplicate, one {hash}.{ext} and one {hash}. You can see this in the metadata for the first URL in my log file. I tried to see if I could use special formatting, like {filename!s:.50}.{ext}, but the same thing happens.

If you're not able to reproduce/fix the issue it's fine. I double-checked some of the posts and everything appears to be there despite the errors, so it's probably a false alarm.

L4cache commented

I think the problem is due to the lack of Windows "long paths" support in your shared directory, or you didn't enable it in your host machine, the filename length is limited to 255 bytes in kemono-dl which is fine by both Windows and Linux.
I tested the post you mentioned in Windows and Linux VMs and didn't reproduce the issue.

But Windows by default only allows the absolute path to be up to 256 characters (or 260, including the "D:" thing and terminating null character), in Windows 10 1607 and above the system can be configured to extend that limitation, by doing THIS or install python and the installer will prompt you to enable that (my prefered way LOL, I'll be using python anyways)

tezrilet commented

True, though unfortunately I'm on Windows 8.1. I really appreciate the help though, and of course for the continued maintenance of this tool. Since I'm able to work around the issue, I think this one can be closed. Thanks again!