NicoNicoLive resulting in corrupt Video
Opened this issue · 6 comments
Checklist
- I'm reporting a broken site support
- I've verified that I'm running youtube-dl version 2020.12.14
- I've checked that all provided URLs are alive and playable in a browser
- I've checked that all URLs and arguments with special characters are properly quoted or escaped
- I've searched the bugtracker for similar issues including closed ones
Verbose log
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-u', 'PRIVATE', '-p', 'PRIVATE', '-v', 'https://live2.nicovideo.jp/watch/lv329535579']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.12.14
[debug] Git HEAD: 8ec12ee48
[debug] Python version 3.8.0 (CPython) - Windows-10-10.0.18362-SP0
[debug] exe versions: ffmpeg 4.2
[debug] Proxy map: {}
[NiconicoLive] Logging in
[NiconicoLive] lv329535579: Downloading webpage
[NiconicoLive] Detected post-March 2019 HLS-based stream
[NiconicoLive] lv329535579: Downloading m3u8 information
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34'
[download] Destination: hololive IDOL PROJECT 1st Live.『Bloom,』-lv329535579.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.135 Safari/537.36
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
" -i "https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34" -c copy -f mp4 "file:hololive IDOL PROJECT 1st Live.『Bloom,』-lv329535579.mp4.part"
ffmpeg version 4.2 Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 9.1.1 (GCC) 20190807
configuration: --enable-gpl --enable-version3 --enable-sdl2 --enable-fontconfig --enable-gnutls --enable-iconv --enable-libass --enable-libdav1d --enable-libbluray --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libtheora --enable-libtwolame --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libzimg --enable-lzma --enable-zlib --enable-gmp --enable-libvidstab --enable-libvorbis --enable-libvo-amrwbenc --enable-libmysofa --enable-libspeex --enable-libxvid --enable-libaom --enable-libmfx --enable-amf --enable-ffnvcodec --enable-cuvid --enable-d3d11va --enable-nvenc --enable-nvdec --enable-dxva2 --enable-avisynth --enable-libopenmpt
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[tcp @ 000001bc66e6d700] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc66e6d700] Successfully connected to 133.152.55.85 port 443
[hls @ 000001bc66e6a180] Skip ('#EXT-X-VERSION:3')
[hls @ 000001bc66e6a180] Skip ('#EXT-X-START:TIME-OFFSET=0.0')
[hls @ 000001bc66e6a180] Skip ('#STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#CURRENT-POSITION:0.0')
[hls @ 000001bc66e6a180] Skip ('#DMC-STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#DMC-CURRENT-POSITION:0.0')
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/5000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[hls @ 000001bc66e6a180] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/5000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tcp @ 000001bc673d0f00] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc673d0f00] Successfully connected to 133.152.55.85 port 443
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/10000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[hls @ 000001bc66e6a180] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/10000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tcp @ 000001bc67471140] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc67471140] Successfully connected to 133.152.55.85 port 443
[h264 @ 000001bc67870dc0] Reinit context to 1280x720, pix_fmt: yuv420p
Input #0, hls, from 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34':
Duration: N/A, start: 6.000000, bitrate: N/A
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp
Metadata:
variant_bitrate : 0
Stream #0:1: Video: h264 (High), 1 reference frame ([27][0][0][0] / 0x001B), yuv420p(tv, bt709, left), 1280x720 [SAR 1:1 DAR 16:9], 30 tbr, 90k tbn, 60 tbc
Metadata:
variant_bitrate : 0
Output #0, mp4, to 'file:hololive IDOL PROJECT 1st Live.『Bloom,』-lv329535579.mp4.part':
Metadata:
encoder : Lavf58.29.100
Stream #0:0: Video: h264 (High), 1 reference frame (avc1 / 0x31637661), yuv420p(tv, bt709, left), 1280x720 (0x0) [SAR 1:1 DAR 16:9], q=2-31, 30 tbr, 90k tbn, 90k tbc
Metadata:
variant_bitrate : 0
Stream #0:1: Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp
Metadata:
variant_bitrate : 0
Stream mapping:
Stream #0:1 -> #0:0 (copy)
Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
Automatically inserted bitstream filter 'aac_adtstoasc'; args=''
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/15000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc673cbb80] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/15000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tcp @ 000001bc67874500] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc67874500] Successfully connected to 133.152.55.85 port 443
[hls @ 000001bc66e6a180] Skip ('#EXT-X-VERSION:3')
[hls @ 000001bc66e6a180] Skip ('#EXT-X-START:TIME-OFFSET=0.0')
[hls @ 000001bc66e6a180] Skip ('#STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#CURRENT-POSITION:0.0')
[hls @ 000001bc66e6a180] Skip ('#DMC-STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#DMC-CURRENT-POSITION:0.0')
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/20000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc67688940] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/20000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[https @ 000001bc67a72640] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
.
.
.
SKIP TO ERROR
.
.
.
[tcp @ 000001bc685a9c80] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc685a9c80] Successfully connected to 133.152.55.85 port 443
[hls @ 000001bc66e6a180] Skip ('#EXT-X-VERSION:3')
[hls @ 000001bc66e6a180] Skip ('#EXT-X-START:TIME-OFFSET=0.0')
[hls @ 000001bc66e6a180] Skip ('#STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#CURRENT-POSITION:1340.0')
[hls @ 000001bc66e6a180] Skip ('#DMC-STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#DMC-CURRENT-POSITION:1340.0')
[hls @ 000001bc66e6a180] skipping 43 segments ahead, expired from playlists
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1345000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc6741dfc0] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1345000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tls @ 000001bc67b48940] Error in the pull function.
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1350000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc673cbb80] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1350000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tls @ 000001bc67b48940] The specified session has been invalidated for some reason.
[AVIOContext @ 000001bc67e0bb80] Statistics: 1310574 bytes read, 0 seeks
[hls @ 000001bc66e6a180] keepalive request failed for 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1350000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' when opening url, retrying with new connection: I/O error
[hls @ 000001bc66e6a180] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1350000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tcp @ 000001bc67426c00] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc67426c00] Successfully connected to 133.152.55.85 port 443
[mpegts @ 000001bc66e715c0] PES packet size mismatch
[https @ 000001bc67f097c0] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[AVIOContext @ 000001bc67e0c4c0] Statistics: 894 bytes read, 0 seeks
[tcp @ 000001bc67684680] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc67684680] Successfully connected to 133.152.55.85 port 443
[hls @ 000001bc66e6a180] Skip ('#EXT-X-VERSION:3')
[hls @ 000001bc66e6a180] Skip ('#EXT-X-START:TIME-OFFSET=0.0')
[hls @ 000001bc66e6a180] Skip ('#STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#CURRENT-POSITION:1510.0')
[hls @ 000001bc66e6a180] Skip ('#DMC-STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#DMC-CURRENT-POSITION:1510.0')
[hls @ 000001bc66e6a180] skipping 32 segments ahead, expired from playlists
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1515000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc6741dfc0] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1515000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tls @ 000001bc67b48940] Error in the pull function.
[https @ 000001bc67f097c0] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/playlist.m3u8?start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[hls @ 000001bc66e6a180] Skip ('#EXT-X-VERSION:3')
[hls @ 000001bc66e6a180] Skip ('#EXT-X-START:TIME-OFFSET=0.0')
[hls @ 000001bc66e6a180] Skip ('#STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#CURRENT-POSITION:1525.0')
[hls @ 000001bc66e6a180] Skip ('#DMC-STREAM-DURATION:11511.402')
[hls @ 000001bc66e6a180] Skip ('#DMC-CURRENT-POSITION:1525.0')
[hls @ 000001bc66e6a180] skipping 2 segments ahead, expired from playlists
[hls @ 000001bc66e6a180] HLS request for url 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1530000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34', offset 0, playlist 0
[https @ 000001bc673cbb80] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1530000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tls @ 000001bc67b48940] The specified session has been invalidated for some reason.
[AVIOContext @ 000001bc67e0ba40] Statistics: 454510 bytes read, 0 seeks
[hls @ 000001bc66e6a180] keepalive request failed for 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1530000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' when opening url, retrying with new connection: I/O error
[hls @ 000001bc66e6a180] Opening 'https://pd085040959.dmc.nico/hlsarchive/ht2_nicolive/nicolive-hamster-lv329535579_main_fa6a09a6656b7a7a5aa2bbf3ec536bb31aa8289be434b09f5dd205e041dce1db/1/ts/1530000.ts?start=0.0&start_time=-573738078816&ht2_nicolive=116342196.spa72fzrgy_qosaiu_stzlnkmnap34' for reading
[tcp @ 000001bc67685440] Starting connection attempt to 133.152.55.85 port 443
[tcp @ 000001bc67685440] Successfully connected to 133.152.55.85 port 443
Description
When using both the latest official release and the latest master commit from the GitHub-fork trying to download anything from NicoNicoLive are resulting in incomplete/corrupt video files.
The video files are technically playable, but are missing a lot of chunks and keeps skipping in the audio.
A 3 hour stream for example results in somewhere between a 150 and 500 MB MP4-file, varying between each run how many chunks are actually skipped, which is far smaller than they used to be.
Looking at the logs (provided an example above) it seems to be an issue with the keep-alive of the connection that fails, resulting in an some errors (error in the pull function/PES packet size mismatch) and causing whole HLS segments be skipped.
Can something have change on NNL's side as this used to work a few weeks ago?
Can you do me a favor and check how fast ffmpeg is downloading the stream (specifically the speed value when downloading the stream. ffmpeg log doesn't show it which isn't useful)
This needs to be at x1.0 to stop the stream from skipping. NND is hostile towards clients and prevents them from downloading faster than x1.0, and deliberately makes video chunks missing if you're not downloading fast enough (because the server assumes you're going to restart the stream and buffer if you're downloading too slow)
If this is the case, then unfortunately there is no fix for this other than getting a better connection to NNDs servers. I rent a server in a datacenter which I use to download NND streams from so I don't have this issue, but I have heard cases of people who have otherwise decent internet not able to download NND streams exactly as you describe.
Can you do me a favor and check how fast ffmpeg is downloading the stream (specifically the speed value when downloading the stream. ffmpeg log doesn't show it which isn't useful)
This needs to be at x1.0 to stop the stream from skipping. NND is hostile towards clients and prevents them from downloading faster than x1.0, and deliberately makes video chunks missing if you're not downloading fast enough (because the server assumes you're going to restart the stream and buffer if you're downloading too slow)
If this is the case, then unfortunately there is no fix for this other than getting a better connection to NNDs servers. I rent a server in a datacenter which I use to download NND streams from so I don't have this issue, but I have heard cases of people who have otherwise decent internet not able to download NND streams exactly as you describe.
How do I do that? As you say, nothing in the logs mention speed.
@redbatz It should still show in the terminal as it's downloading. It just doesn't persist in the logs
@bbepis
I have now verified that it does happen when the speed drops low. It seems if it below x0.7 it causes the error in the pull function, which in turn causes the whole corrupt segment thing.
Sorry for opening an incorrect ticket, but maybe this behaviour should be noted somewhere in the documentation?
Sorry for opening an incorrect ticket, but maybe this behaviour should be noted somewhere in the documentation?
You didn't do anything wrong. This is an obscure issue within NND and it's okay to ask questions about it.
I'll write something on the README later, and leave this ticket open until it's done. Thanks for suggesting it