e-alfred/ocdownloader

Unable to retrieve true YouTube video URL

geosphere opened this issue · 21 comments

geosphere commented

I'm use Nextcloud 18 in container with Aria2 and youtube-dl in some container. Aria2 work fine, but youtube-dl work only manually.

Steps to reproduce

  1. Trying to download a Youtube video and catch error "Unable to retrieve true YouTube video URL"
  2. No matching log entries
  3. Open console and connect to docker: docker exec -it cloud sh
  4. Trying to download manually: /usr/local/bin/youtube-dl https://..... - and this work fine

Expected behaviour

Actual behaviour

Server configuration detail

Operating system: Linux 5.5.0-amd64 #1 SMP Debian 5.5.17-1 (2020-04-21) x86_64

Webserver: nginx/1.17.8 (fpm-fcgi)

Database: pgsql PostgreSQL 9.6.5 on x86_64-pc-linux-gnu, compiled by gcc (Debian 7.2.0-1) 7.2.0, 64-bit

PHP version:

7.3.14
Modules loaded: Core, date, libxml, openssl, pcre, sqlite3, zlib, ctype, curl, dom, fileinfo, filter, gd, hash, iconv, json, mbstring, SPL, PDO, pdo_sqlite, bz2, posix, readline, Reflection, session, SimpleXML, standard, tokenizer, xml, xmlreader, xmlwriter, mysqlnd, cgi-fcgi, apcu, Phar, exif, ftp, gmp, imap, intl, ldap, mysqli, pcntl, pdo_mysql, pdo_pgsql, pgsql, zip, imagick, inotify, redis, smbclient, libsmbclient, Zend OPcache

Nextcloud version: 18.0.4 - 18.0.4.2

Updated from an older Nextcloud/ownCloud or fresh install: updated

Where did you install Nextcloud from: unknown

Signing status

Array
(
)

List of activated apps 
Enabled:
 - accessibility: 1.4.0
 - activity: 2.11.0
 - admin_audit: 1.8.0
 - analytics: 2.2.3
 - announcementcenter: 3.7.0
 - apporder: 0.9.0
 - bookmarks: 3.0.10
 - breezedark: 18.0.14
 - bruteforcesettings: 1.6.0
 - calendar: 2.0.3
 - camerarawpreviews: 0.7.3
 - checksum: 0.4.4
 - cloud_federation_api: 1.1.0
 - comments: 1.8.0
 - contacts: 3.3.0
 - cospend: 0.4.9
 - dav: 1.14.0
 - deck: 0.8.2
 - drawio: 0.9.5
 - external: 3.5.0
 - federatedfilesharing: 1.8.0
 - federation: 1.8.0
 - files: 1.13.1
 - files_accesscontrol: 1.8.1
 - files_downloadactivity: 1.7.0
 - files_external: 1.9.0
 - files_frommail: 0.4.0
 - files_markdown: 2.2.0
 - files_pdfviewer: 1.7.0
 - files_retention: 1.7.0
 - files_rightclick: 0.15.2
 - files_sharing: 1.10.1
 - files_texteditor: 2.11.0
 - files_trackdownloads: 1.7.0
 - files_trashbin: 1.8.0
 - files_versions: 1.11.0
 - files_videoplayer: 1.7.0
 - firstrunwizard: 2.7.0
 - flowupload: 0.1.8
 - gpxedit: 0.0.13
 - gpxmotion: 0.0.11
 - gpxpod: 4.2.1
 - groupfolders: 6.0.6
 - impersonate: 1.5.0
 - issuetemplate: 0.6.0
 - logreader: 2.3.0
 - lookup_server_connector: 1.6.0
 - mail: 1.3.4
 - maps: 0.1.6
 - metadata: 0.11.1
 - music: 0.14.0
 - nextcloud_announcements: 1.7.0
 - notes: 3.3.0
 - notifications: 2.6.0
 - oauth2: 1.6.0
 - occweb: 0.0.7
 - ocdownloader: 1.7.7
 - ocsms: 2.1.7
 - passman: 2.3.5
 - password_policy: 1.8.0
 - phonetrack: 0.6.2
 - photos: 1.0.0
 - previewgenerator: 2.3.0
 - privacy: 1.2.0
 - provisioning_api: 1.8.0
 - qownnotesapi: 20.1.0
 - quota_warning: 1.7.0
 - recommendations: 0.6.0
 - serverinfo: 1.8.0
 - settings: 1.0.0
 - sharebymail: 1.8.0
 - sharingpath: 0.2.2
 - smb_test: 0.2.4
 - sociallogin: 2.4.6
 - socialsharing_diaspora: 2.0.0
 - socialsharing_email: 2.0.0
 - socialsharing_facebook: 2.0.0
 - socialsharing_twitter: 2.0.0
 - spreed: 8.0.8
 - support: 1.1.0
 - survey_client: 1.6.0
 - systemtags: 1.8.0
 - tasks: 0.12.2
 - telephoneprovider: 1.0.3
 - text: 2.0.0
 - theming: 1.9.0
 - theming_customcss: 1.5.0
 - twofactor_backupcodes: 1.7.0
 - twofactor_totp: 4.1.3
 - updatenotification: 1.8.0
 - viewer: 1.2.0
 - w2g2: 3.0.1
 - workflow_script: 1.3.1
 - workflowengine: 2.0.0
Disabled:
 - appointments
 - circles
 - dropit
 - encryption
 - files_automatedtagging
 - files_fulltextsearch
 - fulltextsearch
 - richdocuments
 - sharerenamer
 - user_ldap
 - user_saml
 - user_usage_report
Configuration (config/config.php) 
{
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "apps_paths": [
        {
            "path": "\/nextcloud\/apps",
            "url": "\/apps",
            "writable": false
        },
        {
            "path": "\/apps2",
            "url": "\/apps2",
            "writable": true
        }
    ],
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0,
        "dbindex": 1
    },
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "logtimezone": "Europe\/Kiev",
    "logdateformat": "Y-m-d H:i:s",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
    "overwritehost": "***REMOVED SENSITIVE VALUE***",
    "overwriteprotocol": "https",
    "dbtype": "pgsql",
    "version": "18.0.4.2",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "mail_smtpmode": "smtp",
    "mail_smtpauthtype": "LOGIN",
    "mail_smtpauth": 1,
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
    "mail_smtpsecure": "tls",
    "mail_smtpport": "587",
    "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
    "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
    "maintenance": false,
    "updater.release.channel": "stable",
    "loglevel": 0,
    "data-fingerprint": "ae85ed2460dfc8f1a3100b51adc85aa0",
    "ldapIgnoreNamingRules": false,
    "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
    "log_type": "file",
    "log_rotate_size": 1048576,
    "debug": false,
    "customclient_desktop": "https:\/\/nextcloud.com\/install\/#install-clients",
    "customclient_android": "https:\/\/play.google.com\/store\/apps\/details?id=com.nextcloud.client",
    "allow_user_to_change_display_name": true,
    "theme": "",
    "enabledPreviewProviders": [
        "OC\\Preview\\Image",
        "OC\\Preview\\MP3",
        "OC\\Preview\\TXT",
        "OC\\Preview\\MarkDown",
        "OC\\Preview\\Epub",
        "OC\\Preview\\FB2",
        "OC\\Preview\\PDF"
    ],
    "has_rebuilt_cache": true
}

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Operating system: Linux

Logs

Web server error log 
No similar entries
Nextcloud log 
No similar entries
Browser log

No similar entries

MoucceuWildfire commented

Similar error. FreeNAS-11.3-U1 + Nextcloud 18.0.4 + ARIA2. A couple of weeks ago was fine. Youtube-dl latest 2020.05.03 version and ocDownloader v1.7.7. PHP7.2
Manual downloading from command line works fine.

  • 👍1
e-alfred commented

Did you set the right path in the admin setting section of ocDownloader for youtube-dl?

  • 👎1
geosphere commented

/usr/local/bin/youtube-dl
This is the default path. And yes, this path corresponds to the entry in the admin section.

  • 👍1
geosphere commented

It looks strange, but the download from YT works when I use the plugin for Firefox ocDownloader v1.6.2. Global settings Audio/Video Format are ignoring, but the download is successful. And the same link in UI /apps/ocdownloader/add brings an error

e-alfred commented

Can you try to revert to 1.7.6 to test if this PR (#158) might be the problem? It is strange because on a non-docker installation using the UI or extension works flawlessly.

Emiliaaah commented

I am running Nextcloud with ocdownloader on ubuntu 20.04 LTS but I also have the same issue.
Could be a newer version thats causing this since I just installed it on a seperate VM instead of in a Docker container.

EDIT: I am running this in combination with nextcloud and when trying to download via the chrome extension I get the same error

MoucceuWildfire commented

Similar error. FreeNAS-11.3-U1 + Nextcloud 18.0.4 + ARIA2. A couple of weeks ago was fine. Youtube-dl latest 2020.05.03 version and ocDownloader v1.7.7. PHP7.2
Manual downloading from command line works fine.

I downgraded app to ver 1.7.6 and this version works fine. Forget to mention - Nextcloud works in Jail of FreeNAS.

e-alfred commented

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: fc18d43

Emiliaaah commented

I'm running 1.7.7 but how do I revert the a earlier version?

MoucceuWildfire commented

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: fc18d43

I have to edit this controller/lib/youtube.php file in my installation according to link above?

MoucceuWildfire commented

Can you try to revert this change if you have ocDownloader 1.7.7 installed and retry a Youtube download: fc18d43

I have to edit this controller/lib/youtube.php file in my installation according to link above?

this didn`t help

e-alfred commented

@RelyIz You can download earlier versions from https://github.com/e-alfred/ocdownloader/releases and extract them to the apps folder of your Nextcloud installation.

@MoucceuWildfire Yes, trying to change lines 72-76 in this file. As it didn't help, I have to look further into this.

MoucceuWildfire commented

Hi e-alfred. Want to thank you for this app at first. Maybe you can tell me how to change the source to add some logging. Just for debugging purposes. I want to help to find the problem. I successfully can use 1.7.6 ver but...

e-alfred commented

To debug the Youtube download link extraction, please insert throw new \Exception( "\$Output = $Output" ); at line 61 in controller/lib/youtube.php and post the results from your Nextcloud log here to see if the link is extracted correctly.

Please also check the following settings to see if they match with yours:

image

MoucceuWildfire commented

Well my steps:

  1. Upgrade ocDownloader to 1.7.7
  2. Write the proper value in YouTube DL Audio Format bestaudio[abr<=75]
  3. Write in field YouTube DL Video Format best[width<=1280]
  4. App works well now.

Mentioned field was empty before.
I also restarted my instance of Nextcloud and App still works fine.
So problem was in empty field YouTube DL Audio Format and YouTube DL Video Format in "Additional settings" of Nextcloud. OcDownloader failed to fill this field with default value.
Thanks for very nice app Alfred!

If you want me to help investigate the bug further, let me know how.

e-alfred commented

The values for youtube-dl weren't correctly set initially for local downloads, this explains why it worked with the API. Fixed by cb6ced0.

NewRedsquare commented

Hello and sorry for re-opening this issue, i added the line to get debug output and here is what i get ( i have the Unable to retrieve true YouTube video URL problem ) :

{"reqId":"24wmOJ7J9VnKyfJWJ6SF","level":3,"time":"2020-06-09T16:40:57+00:00","remoteAddr":"***HIDDEN IP***","user":"***HIDDEN USER***","app":"index","method":"POST","url":"/apps/ocdownloader/ytdownloader/add","message":{"Exception":"Exception","Message":"$Output = ","Code":0,"Trace":[{"file":"/var/www/cloud/apps/ocdownloader/controller/ytdownloader.php","line":150,"function":"getVideoData","class":"OCA\\ocDownloader\\Controller\\Lib\\YouTube","type":"->","args":[]},{"file":"/var/www/cloud/lib/private/AppFramework/Http/Dispatcher.php","line":170,"function":"add","class":"OCA\\ocDownloader\\Controller\\YTDownloader","type":"->","args":[]},{"file":"/var/www/cloud/lib/private/AppFramework/Http/Dispatcher.php","line":100,"function":"executeController","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\ocDownloader\\Controller\\YTDownloader"},"Add"]},{"file":"/var/www/cloud/lib/private/AppFramework/App.php","line":137,"function":"dispatch","class":"OC\\AppFramework\\Http\\Dispatcher","type":"->","args":[{"__class__":"OCA\\ocDownloader\\Controller\\YTDownloader"},"Add"]},{"file":"/var/www/cloud/lib/private/AppFramework/Routing/RouteActionHandler.php","line":47,"function":"main","class":"OC\\AppFramework\\App","type":"::","args":["YTDownloaderController","Add",{"__class__":"OC\\AppFramework\\DependencyInjection\\DIContainer"},{"_route":"ocdownloader.YTDownloader.Add"}]},{"function":"__invoke","class":"OC\\AppFramework\\Routing\\RouteActionHandler","type":"->","args":[{"_route":"ocdownloader.YTDownloader.Add"}]},{"file":"/var/www/cloud/lib/private/Route/Router.php","line":297,"function":"call_user_func","args":[{"__class__":"OC\\AppFramework\\Routing\\RouteActionHandler"},{"_route":"ocdownloader.YTDownloader.Add"}]},{"file":"/var/www/cloud/lib/base.php","line":1007,"function":"match","class":"OC\\Route\\Router","type":"->","args":["/apps/ocdownloader/ytdownloader/add"]},{"file":"/var/www/cloud/index.php","line":37,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"/var/www/cloud/apps/ocdownloader/controller/lib/youtube.php","Line":61,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.142 Safari/537.36","version":"19.0.0.12"}

So apparently no issue but i get an infinite loading while adding youtube video with webUI.
With webextension i have the same Unable to retrieve true YouTube video URL.

My config :
The last 1.7.8 version of Ocdownloader
Ubuntu 18.04.4 LTS
Mariadb Ver 15.1 Distrib 10.1.44-MariaDB
Nextcloud 19.0.0.12

blob42 commented

Hi,

I have same issue as @NewRedsquare !

EDIT: Fixed the issue. The log was showing the wrong youtube-dl path /usr/bin/youtube-dl even though on the admin interface it looked good by default.

The fix:

  • Delete the youtube-dl path field and click outside to save the setting
  • Refill the path again
  • Works !

config:
ocDownloader 1.7.8
Nextcloud 19 official docker image

  • 👍1
xspeed1989 commented

Hi,

I have same issue as @NewRedsquare !

EDIT: Fixed the issue. The log was showing the wrong youtube-dl path /usr/bin/youtube-dl even though on the admin interface it looked good by default.

The fix:

  • Delete the youtube-dl path field and click outside to save the setting
  • Refill the path again
  • Works !

config:
ocDownloader 1.7.8
Nextcloud 19 official docker image

same as u
but it doesn't work fine
Unable to retrieve true YouTube video URL

  • 👍1
thiccpad commented

Also have this issue on
Ubuntu 20.04
Nextcloud 20.0.4
PHP 7.4.3
ocdownoader 1.7.10

Delete the youtube-dl path field and click outside to save the setting
Refill the path again
Works !

Didn't work. Also tried installing it with pip instead of apt-get and manually.

etyrnal commented

this app should be taken out of the Nextcloud store. There are SO MANY reports of this not working. It's clear that it's not mature, and is barely an alpha version.

  • 👍1