bibanon/tubeup

Bug report: Channels having YouTube shorts cause Tubeup to fail

Closed this issue · 4 comments

animeavi commented

Checklist

  • I have fully updated tubeup, yt-dlp and internetarchive along with their associated dependencies.
  • I've included the full and unredacted URL and console output (with the exception of site usernames and passwords, or IP addresses of the machine doing the download). Hiding URLs will get issue closed.
  • I've checked that all provided URLs are alive and playable in a browser
  • I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • I've searched the issues (closed or open) for similar bug reports including closed ones
  • [ X I am not submitting a bug report about S3 timeouts (we have no control over Internet Archive outages or per-user throttling)
  • I've checked the Twitter account of Internet Archive (@internetarchive) for any announced outages or planned infrastructure maintenance that would affect performance of uploads.
  • I have properly indented with triple backticks before and after the console output full terminal terminal output from the line where the command was run to where I was returned to command prompt, and am not trying to obscure item identifiers or URLs used in the creation of the bug (we need this to recreate the bug or investigate what happened).

Dependency versions

Please provide version information from core dependencies:

ia --version: 3.0.2
yt-dlp --version: 2023.02.17

Expected behavior
I'd expect tubeup to be able to download the channel's videos, but that doesn't happen.

Additional context
Command used with example channel
tubeup -d https://www.youtube.com/@zaetiar

At the end of the process I get this:

[DEBUG] 2023-03-01 13:11:21,114 - tubeup.TubeUp - DEBUG - [debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[DEBUG] 2023-03-01 13:11:21,138 - tubeup.TubeUp - DEBUG - [debug] Default format spec: bestvideo*+bestaudio/best
[DEBUG] 2023-03-01 13:11:21,139 - tubeup.TubeUp - DEBUG - [download] Downloading item 6 of 6
[DEBUG] 2023-03-01 13:11:21,140 - tubeup.TubeUp - DEBUG - [youtube] Extracting URL: https://www.youtube.com/shorts/BrdR6TfTNps
[DEBUG] 2023-03-01 13:11:21,140 - tubeup.TubeUp - DEBUG - [youtube] BrdR6TfTNps: Downloading webpage
[DEBUG] 2023-03-01 13:11:21,484 - tubeup.TubeUp - DEBUG - [youtube] BrdR6TfTNps: Downloading android player API JSON
[DEBUG] 2023-03-01 13:11:21,587 - tubeup.TubeUp - DEBUG - [debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[DEBUG] 2023-03-01 13:11:21,587 - tubeup.TubeUp - DEBUG - [debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[DEBUG] 2023-03-01 13:11:21,607 - tubeup.TubeUp - DEBUG - [debug] Default format spec: bestvideo*+bestaudio/best
[DEBUG] 2023-03-01 13:11:21,609 - tubeup.TubeUp - DEBUG - [download] Finished downloading playlist: zae tiar - Shorts
[DEBUG] 2023-03-01 13:11:21,609 - tubeup.TubeUp - DEBUG - [download] Finished downloading playlist: zae tiar

An exception just occured, if you found this exception isn't related with any of your connection problem, please report this issue to https://github.com/bibanon/tubeup/issues
Traceback (most recent call last):
  File "/home/iota/.local/lib/python3.10/site-packages/tubeup/__main__.py", line 99, in main
    for identifier, meta in tu.archive_urls(URLs, metadata,
  File "/home/iota/.local/lib/python3.10/site-packages/tubeup/TubeUp.py", line 417, in archive_urls
    downloaded_file_basenames = self.get_resource_basenames(
  File "/home/iota/.local/lib/python3.10/site-packages/tubeup/TubeUp.py", line 176, in get_resource_basenames
    if check_if_ia_item_exists(entry) == 0:
  File "/home/iota/.local/lib/python3.10/site-packages/tubeup/TubeUp.py", line 112, in check_if_ia_item_exists
    infodict['display_id']))
KeyError: 'display_id'
vxbinaca commented

yt-dlp --version: 0.0.35

Please run yt-dlp --version again, you gave us the Tubeup version, Tubeup does not rip the video, yt-dlp does.

What version of yt-dlp are you using?

vxbinaca commented

Try running this:

tubeup https://www.youtube.com/@zaetiar/videos

/shorts should work too. I know because I can upload /shorts. /shorts aren't the problem.

yt-dlp needs to be pointed to the correct type of upload, you can't just point it at a channel and it ingests everything, hasn't worked that way with yt-dlp in a while now.

Let me know the result.

animeavi commented

yt-dlp --version: 0.0.35

Please run yt-dlp --version again, you gave us the Tubeup version, Tubeup does not rip the video, yt-dlp does.

What version of yt-dlp are you using?

2023.02.17, edited the post.

Try running this:

tubeup https://www.youtube.com/@zaetiar/videos

/shorts should work too. I know because I can upload /shorts. /shorts aren't the problem.

yt-dlp needs to be pointed to the correct type of upload, you can't just point it at a channel and it ingests everything, hasn't worked that way with yt-dlp in a while now.

Let me know the result.

That makes sense, perhaps should be documented somewhere (if it isn't), I thought you could just point tubeup to a channel and archive everything.

vxbinaca commented

Tubeup doesn't download video it just tells yt-dlp where to go, yt-dlp does not parse base YT channel URLs you have to select what category you want.

I'm taking it this issue is resolved for you.

  • 👍1