Traceback when unsupported URL
swoopla opened this issue · 4 comments
When i use a wrong URL (like this youtube URL https://www.youtube.com/watch?9A-HLSvtBWc, URL whitout 'v=')
I have a big error, it's break my container. I need to destroy it and i have to create a new one.
docker logs -f
(...)
Started download thread
Bottle v0.12.16 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.
Added url https://www.youtube.com/watch?9A-HLSvtBWc to the download queue
AA.BB.CC.DD - - [17/Jun/2019 17:08:46] "POST /youtube-dl/q HTTP/1.1" 200 106
[generic] watch?9A-HLSvtBWc: Requesting header
[redirect] Following redirect to https://www.youtube.com/
[generic] www.youtube: Requesting header
WARNING: Falling back on generic information extractor.
[generic] www.youtube: Downloading webpage
[generic] www.youtube: Extracting information
ERROR: Unsupported URL: https://www.youtube.com/
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 529, in extract
ie_result = self._real_extract(url)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/extractor/generic.py", line 3320, in _real_extract
raise UnsupportedError(url)
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.youtube.com/
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "./youtube-dl-server.py", line 60, in dl_worker
download(url, options)
File "./youtube-dl-server.py", line 106, in download
ydl.download([url])
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 2008, in download
url, force_generic_extractor=self.params.get('force_generic_extractor', False))
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 807, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 868, in process_ie_result
extra_info=extra_info)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 819, in extract_info
self.report_error(compat_str(e), e.format_traceback())
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 624, in report_error
self.trouble(error_message, tb)
File "/usr/local/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 594, in trouble
raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: Unsupported URL: https://www.youtube.com/
Added url https://www.youtube.com/watch?v=9A-HLSvtBWc to the download queue
AA.BB.CC.DD - - [17/Jun/2019 17:09:12] "POST /youtube-dl/q HTTP/1.1" 200 107
Added url https://www.youtube.com/watch?v=9A-HLSvtBWc to the download queue
AA.BB.CC.DD - - [17/Jun/2019 17:10:27] "POST /youtube-dl/q HTTP/1.1" 200 107
Added url https://www.youtube.com/watch?v=9A-HLSvtBWc to the download queue
AA.BB.CC.DD - - [17/Jun/2019 17:12:39] "POST /youtube-dl/q HTTP/1.1" 200 107
Added url https://www.youtube.com/watch?v=9A-HLSvtBWc to the download queue
AA.BB.CC.DD - - [17/Jun/2019 17:15:00] "POST /youtube-dl/q HTTP/1.1" 200 107
I'm getting the same thing... and its not necessarily just a wrong URL either! Even if you give it the right URL, whenever you cause an error and cause a Traceback... any subsequent URLs sent are added to a queue and not executed at all.
In my case, I gave it a valid URL... but the youtube-dl application was out of date, and so the error was caused because it couldn't parse the information it had received. After upgrading youtube-dl it works perfectly fine for any urls (that aren't region locked). However, while developing a Firefox addon so that I can easily just click a button and send a request... I accidentally gave it the wrong URL during development and it also caused it to crash, and thus anything that goes into the queue isn't processed.
This was annoying for me because it meant I had to keep restarting the container each time I made a bad request. But considering a valid URL can also cause it to crash if youtube-dl is out of date... then it makes using my plugin unreliable because http://127.0.0.1:5000/youtube-dl/q doesn't give you an error because as far as its concerned, it got added to the queue. There needs to be a way for the server to detect the crash and reboot the main process again.
Any thoughts?
In your code, i change the download function and i add Exception Handling (Try and Except).
I'm testing it during july and if it works, i will create a pull request.
i add this lines:
def download(url, request_options):
with youtube_dl.YoutubeDL(get_ydl_options(request_options)) as ydl:
try:
ydl.download([url])
except:
print("Error for URL: ",url)
So, when i use the wrong URL: https://www.youtube.com/watch?9A-HLSvtBWc
docker-compose logs -f
Attaching to site-youtube-dl_1
youtube-dl_1 | Started download thread
youtube-dl_1 | Bottle v0.12.16 server starting up (using WSGIRefServer())...
youtube-dl_1 | Listening on http://0.0.0.0:8080/
youtube-dl_1 | Hit Ctrl-C to quit.
youtube-dl_1 |
youtube-dl_1 | Added url https://www.youtube.com/watch?9A-HLSvtBWc to the download queue
youtube-dl_1 | 172.19.0.6 - - [30/Jul/2019 14:23:08] "POST /youtube-dl/q HTTP/1.1" 200 98
youtube-dl_1 | [generic] watch?9A-HLSvtBWc: Requesting header
youtube-dl_1 | [redirect] Following redirect to https://www.youtube.com/
youtube-dl_1 | [generic] www.youtube: Requesting header
youtube-dl_1 | WARNING: Falling back on generic information extractor.
youtube-dl_1 | [generic] www.youtube: Downloading webpage
youtube-dl_1 | [generic] www.youtube: Extracting information
youtube-dl_1 | ERROR: Unsupported URL: https://www.youtube.com/
youtube-dl_1 | Error for URL: https://www.youtube.com/watch?9A-HLSvtBWc
youtube-dl_1 | Added url https://www.youtube.com/watch?v=9A-HLSvtBWc to the download queue
youtube-dl_1 | 172.19.0.6 - - [30/Jul/2019 14:29:08] "POST /youtube-dl/q HTTP/1.1" 200 100
youtube-dl_1 | [youtube] 9A-HLSvtBWc: Downloading webpage
youtube-dl_1 | [youtube] 9A-HLSvtBWc: Downloading video info webpage
youtube-dl_1 | WARNING: Unable to extract video title
youtube-dl_1 | [download] Destination: /youtube-dl/_ [9A-HLSvtBWc].f137.mp4
Confirmed adding an unsupported url no longer prevents future requests.