Athlon1600/SerpScraper

New Google Cookie popup is Stopping this Working

Opened this issue · 5 comments

It seems like a new Google Cookie popup is stopping this working, see screenshot.
image

That's what you get when you echo the response.

Noticed exactly the same thing, and have just been doing some testing, and got around the problem by setting a cookie that states the cookie policy has been agreed - i.e.

$google->setPreference('results_per_page', 100);
$google->setPreference('google_domain', 'google.co.uk');

// Set A Cookie To State GDPR acceptance
$browser = $google->getBrowser();
$browser->setCookies( ".google.co.uk\tTRUE\t\\\tTRUE\t2145916800\tCONSENT\tYES+shp.gws-20210421-0-RC1.en+FX+836\r\n" );

Then do your standard do while resonse

@JamesSimpson

Although the rendered page has a Cookie Popup modal, the SERP results still exist in the DOM *

The inability to parse the results is a failing of the preg_match regex formula this library uses, setting cookie fixes the symptoms but not the cause.

  • although this does depend on your user agent string, I've tested a great number of them and some return a dom with results + pop up and some return just the pop up.

So for now this still seems broken and unusable, even if you add the cookies manually?

So for now this still seems broken and unusable, even if you add the cookies manually?

I can't replicate any of this from my end.

That's what you get when you echo the response.

that may be, but the $searchResponse->results should still return an array of all the results for you. Is that array empty on your end? Post your code and the outputs you're getting

that may be, but the $searchResponse->results should still return an array of all the results for you. Is that array empty on your end? Post your code and the outputs you're getting

This only happens when you using EUROPE Based proxy or IP.!