tasos-py/Search-Engines-Scraper

Suggestion: Collect ad based and organic links

Closed this issue · 4 comments

jatlantic commented

Hi,

Love this package and was wondering if it would be possible to add a feature where one could collect the ad based and organic urls/link results (for Bing for example)?

Cheers,
jatlantic

tasos-py commented

Hi Jatlantic. While ads should be present in HTTP response bodies, the css selectors ere designed to ignore them. So, we should be able to change the selectors and get them, but I don't see the need to add this feature. Of course, you could just create a fork and add it yourself. And if you choose to do so I'll be glad to help

  • 👍1
jatlantic commented

Hi tasos-py, thanks a lot for your swift answer. And yes it makes absolute sense. I'll do a fork to capture organic and ad results for Bing and will go from there.

  • 👍1
tasos-py commented

That's awesome! To help you get started, you'll find SearchEngine objects in "/search_engines/engines/". Each of them has all required selectors in a ._selectors() method that you'd have to edit. Other than that you could maybe add a flag or argument in the parent object to chose between ads and organic results

  • 👍1
jatlantic commented

Good advice, adapted things for a Bing search here: https://github.com/jatlantic/Search-Engines-Scraper. Likely you would do it more elegantly:) Happy to build in your suggestions.

  • 🎉1