InvalidUrlException
Opened this issue · 22 comments
Hello, I just noticed that all my video embeds have suddenly stopped working.
I'm trying to embed this video --> https://www.youtube.com/watch?v=VvfT2wvs3h0 (which is public on YouTube)
Here's the Craft debug error:
Embed\Exceptions\InvalidUrlException: Invalid url 'https://www.youtube.com/watch?v=VvfT2wvs3h0' (Status code 429) in /srv/app/vjh-bighorn/htdocs/vendor/embed/embed/src/Embed.php:142
Stack trace:
#0 /srv/app/vjh-bighorn/htdocs/vendor/embed/embed/src/Embed.php(81): Embed\Embed::process(Object(Embed\Http\Url), Array, Object(Embed\Http\CurlDispatcher))
#1 /srv/app/vjh-bighorn/htdocs/vendor/mikestecker/craft-videoembedder/src/services/VideoEmbedderService.php(38): Embed\Embed::create(Object(Embed\Http\Url), Array)
#2 /srv/app/vjh-bighorn/htdocs/vendor/mikestecker/craft-videoembedder/src/services/VideoEmbedderService.php(216): mikestecker\videoembedder\services\VideoEmbedderService->getInfo('https://www.you...')
#3 /srv/app/vjh-bighorn/htdocs/vendor/mikestecker/craft-videoembedder/src/variables/VideoEmbedderVariable.php(47): mikestecker\videoembedder\services\VideoEmbedderService->getEmbedUrl('https://www.you...', Array)
#4 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Extension/Core.php(1619): mikestecker\videoembedder\variables\VideoEmbedderVariable->getEmbedUrl('https://www.you...')
#5 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/helpers/Template.php(73): twig_get_attribute(Object(craft\web\twig\Environment), Object(Twig_Source), Object(mikestecker\videoembedder\variables\VideoEmbedderVariable), 'getEmbedUrl', Array, 'method', false, false)
#6 /srv/app/vjh-bighorn/htdocs/storage/runtime/compiled_templates/43/436e075031d7e5673229fd1999870b2d37dc680fcafa020bff431fc5ba257e03.php(23): craft\helpers\Template::attribute(Object(craft\web\twig\Environment), Object(Twig_Source), Object(mikestecker\videoembedder\variables\VideoEmbedderVariable), 'getEmbedUrl', Array, 'method')
#7 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(390): __TwigTemplate_0f219985ff513a66ff3d8734ab47a7aa5f578ba776295f5ec4f7c1601d1bae0e->doDisplay(Array, Array)
#8 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(49): Twig_Template->displayWithErrorHandling(Array, Array)
#9 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(367): craft\web\twig\Template->displayWithErrorHandling(Array, Array)
#10 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(31): Twig_Template->display(Array, Array)
#11 /srv/app/vjh-bighorn/htdocs/storage/runtime/compiled_templates/c8/c8e45289308b00ec8c5235eb826496e06fdfb6b0711c4dbb2be3fde126278cf1.php(54): craft\web\twig\Template->display(Array)
#12 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(189): __TwigTemplate_9c2bf03b7241e24cc9b54b59ee50783347425916c52ec340ff2ab74c8441ea4a->block_content(Array, Array)
#13 /srv/app/vjh-bighorn/htdocs/storage/runtime/compiled_templates/3e/3e703181660c7199a1af705d1b3519d5f6a9acbb17c1df406fce063a85979145.php(156): Twig_Template->displayBlock('content', Array, Array)
#14 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(390): __TwigTemplate_e7cca79f9c8b45d03269112611398a5ee6b9a9cf6ce4f0dbfdf78874ba50be2d->doDisplay(Array, Array)
#15 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(49): Twig_Template->displayWithErrorHandling(Array, Array)
#16 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(367): craft\web\twig\Template->displayWithErrorHandling(Array, Array)
#17 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(31): Twig_Template->display(Array, Array)
#18 /srv/app/vjh-bighorn/htdocs/storage/runtime/compiled_templates/c8/c8e45289308b00ec8c5235eb826496e06fdfb6b0711c4dbb2be3fde126278cf1.php(29): craft\web\twig\Template->display(Array, Array)
#19 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(390): __TwigTemplate_9c2bf03b7241e24cc9b54b59ee50783347425916c52ec340ff2ab74c8441ea4a->doDisplay(Array, Array)
#20 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(49): Twig_Template->displayWithErrorHandling(Array, Array)
#21 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(367): craft\web\twig\Template->displayWithErrorHandling(Array, Array)
#22 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/twig/Template.php(31): Twig_Template->display(Array, Array)
#23 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Template.php(375): craft\web\twig\Template->display(Array)
#24 /srv/app/vjh-bighorn/htdocs/vendor/twig/twig/lib/Twig/Environment.php(289): Twig_Template->render(Array)
#25 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/View.php(331): Twig_Environment->render('homepage.html', Array)
#26 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/View.php(378): craft\web\View->renderTemplate('homepage.html', Array)
#27 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/Controller.php(156): craft\web\View->renderPageTemplate('homepage.html', Array)
#28 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/controllers/TemplatesController.php(78): craft\web\Controller->renderTemplate('homepage.html', Array)
#29 [internal function]: craft\controllers\TemplatesController->actionRender('homepage.html', Array)
#30 /srv/app/vjh-bighorn/htdocs/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#31 /srv/app/vjh-bighorn/htdocs/vendor/yiisoft/yii2/base/Controller.php(157): yii\base\InlineAction->runWithParams(Array)
#32 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/Controller.php(104): yii\base\Controller->runAction('render', Array)
#33 /srv/app/vjh-bighorn/htdocs/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('render', Array)
#34 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/Application.php(282): yii\base\Module->runAction('templates/rende...', Array)
#35 /srv/app/vjh-bighorn/htdocs/vendor/yiisoft/yii2/web/Application.php(103): craft\web\Application->runAction('templates/rende...', Array)
#36 /srv/app/vjh-bighorn/htdocs/vendor/craftcms/cms/src/web/Application.php(271): yii\web\Application->handleRequest(Object(craft\web\Request))
#37 /srv/app/vjh-bighorn/htdocs/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#38 /srv/app/vjh-bighorn/htdocs/web/index.php(21): yii\base\Application->run()
#39 {main}
Any ideas?
We're running into a similar issue on one of our site and the exception content Google is sending back here for use:
Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot.
This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the Terms of Service. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services.
This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible. Learn more
Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.
(Which we got by dumping the exception https://github.com/oscarotero/Embed/blob/master/src/Embed.php#L142)
Looks like something changed on YouTube's end that has started causing just right now.
A possible solution would be to get the embed code through the YouTube API but that would have to be added to the plugin.
Just ran into this also. Weird one and horrible timing
Yep, also happening to me. Vimeo embeds are fine.
I've not got anything in my logs, but getEmbedUrl is returning null for all YouTubes
The site had this issue with is now working again, but that's entirely down to YouTube and the traffic they are getting. We'll probably try & move them away from this plugin to something custom that gets the embed code from the Youtube API.
I was having the same problem. All pages where we show a YouTube video would not render. Now it is working fine... so not likely to be the plugin. I was experiencing similar issues with other plugins.
I "think" but can't be sure at this point that it might have something to do with a sitemap update/submission?
My hypothesis is that if there is a lot of sitemap update requests you get blocked as @janhenckens described. Are others using the SEOmatic Plugin?
We’re also using SEOmatic on that site.
Good theory - I am using SEOMatic on a site that's generating a lot of sitemaps.
There's an option in SEOMatic to not index videos and images, so turning those off might solve the issue... or help prove the theory.
On our site it started working again a couple of days ago without changing anything so working theory here is that it's to much requests to a non-api/non-authenticated endpoint.
I "think" but can't be sure at this point that it might have something to do with a sitemap update/submission?
My hypothesis is that if there is a lot of sitemap update requests you get blocked as @janhenckens described. Are others using the SEOmatic Plugin?
SEOmatic will ping Google to let it know that a particular URL or sitemap has changed:
https://github.com/nystudio107/craft-seomatic/blob/v3/src/services/Sitemaps.php#L175
https://github.com/nystudio107/craft-seomatic/blob/v3/src/services/Sitemaps.php#L216
But I don't think either results in your site getting banned, no matter how frequently the endpoints are pinged.
I could also add a "debounce" to make sure pings aren't sent more than X times per minute or such, but I'd want some more evidence that this is actually an issue.
Same thing happening here on our server. Also using SEOMatic on this particular website, and the YouTube URL's aren't working anymore.
The thing I'm unclear on is whether sending a ping to Google, which they recommend doing here:
https://support.google.com/webmasters/answer/183668?hl=en
Use the "ping" functionality to ask us to crawl your sitemap. Send an HTTP GET request like this:
http://www.google.com/ping?sitemap=<complete_url_of_sitemap>
for example:
http://www.google.com/ping?sitemap=https://example.com/sitemap.xml
Could have any causation here. It seems a little strange to me that doing what they recommend (pinging an endpoint), however frequently, would result in YouTube temporarily banning the URLs from being embedded.
Why YouTube, and only YouTube?
I'm open to the idea of debouncing these pings, but I don't want to introduce additional complexity without a reasonable idea that this is actually causing the problem.
Just noticed this again on our site. I managed to download my queue and 404 log files if you are interested @khalwat ?
Absolutely, get 'em along to me andrew@nystudio107.com
Anyone managed to bypass this? And if so, how? We looked at other plugins, but as long as they don't use Youtube's API, they will all give the same error eventually.
I had this error come up just now and adding a YouTube v3 enabled API key into the plugin settings got us back up and running. Hopefully, that helps someone else.
I had this error come up just now and adding a YouTube v3 enabled API key into the plugin settings got us back up and running. Hopefully, that helps someone else.
Which plugin are you using? I think you're mixing up this videoembedder plugin with a different one, because there are no plugin settings where you can enter Youtube API keys :)
I'm curious though which plugin you are using that uses the Youtube API, because I'm still in need for a fix.
The thing I'm unclear on is whether sending a ping to Google, which they recommend doing here:
https://support.google.com/webmasters/answer/183668?hl=en
Use the "ping" functionality to ask us to crawl your sitemap. Send an HTTP GET request like this:
http://www.google.com/ping?sitemap=<complete_url_of_sitemap>
for example:
http://www.google.com/ping?sitemap=https://example.com/sitemap.xmlCould have any causation here. It seems a little strange to me that doing what they recommend (pinging an endpoint), however frequently, would result in YouTube temporarily banning the URLs from being embedded.
Why YouTube, and only YouTube?
I'm open to the idea of debouncing these pings, but I don't want to introduce additional complexity without a reasonable idea that this is actually causing the problem.
Is there perhaps a way to disable these 'pings' in the SEOmatic settings? That way we could try and see for now if that 'fixes' this Youtube ratelimit issue
Sounds reasonable to me @thijskaspers -- added it here: nystudio107/craft-seomatic@673154d
Set your semver to dev-develop and do a composer update to pull it down and try it out.
Sounds reasonable to me @thijskaspers -- added it here: nystudio107/craft-seomatic@673154d
Set your semver to
dev-developand do acomposer updateto pull it down and try it out.
That was fast Andrew, thanks! Will try and let you know how it goes
@thijskaspers Anything to report regarding this? We just had the issue reappear on our website.
@sjcallender Can you provide a bit more detail on which plugin you are adding the YouTube v3 enabled API key?
@nearkingdom @khalwat at first I thought it was working, the videos were working for a long time with the SEOmatic setting (sitemap ping) disabled, but unfortunately after clearing all craft cache, the videos immediately broke again on the first call to YouTube. It's still nice to have the ping setting, so thanks for that Andrew!
We 'fixed' it now by removing the videoembedder plugin and using regular YouTube embeds with the 'plyr' video player in combination with a custom twig filter to get the video thumbnails where we need them. YouTube won't block you using embeds on your site, or regular requests to their images. This plugin uses a different request method that YouTube/Google isn't fond of apparently.
Edit: of course this plugin is nice as it also supports Vimeo, but since we only use YouTube we decided to remove it because of the rate limit issue.
@thijskaspers thanks for the update! :)