1.SEO: Discover the Technical Challenges to get the Number of Indexed Pages on Google and Sitemap.
The aim: This Python script performs a search to check the number of indexed pages on Google for multiple sites using Selenium, bs4 and Python. And compare that with the real indexed pages from Sitemap-parser.
This is not exact at all. But it is good enough to have an idea of the competitor’s index sizes when building an SEO strategy.
You can use python for SEO by leveraging APIs, automating the boring tasks and by implementing machine learning algorithms.
Why making this script then? Because a lot of times you just want to have an idea of how committed a site is in a market. Around how many pages does the business deal with 200 or 200M.