Scrape Ebay product information. Ebay lists its products under specific categories. For example, https://www.ebay.com/b/Mens-Shoes/93427/bn_61999 lists all the products under "Mens-Shoes". This is a paginated list with 48 products per page.
The scraper takes 3 arguments as mentioned below:
$ python3 ebay_scraper.py <url> <pagecount> <outfile>- url - Base URL (similar to the one given above)
- pagecount - Number of pages you want to scrape
- outfile - Output json file without extension
I am using residential proxies from Oxylabs which will return the HTML for the URL. Their proxies are efficient when compared to free proxies. You can check the code in utils.py
Free proxies are available over the Internet from websites like SSLProxies, Free-proxy-list etc. In getproxies.py I have implemented scraper to extract the list of proxies and use them for Ebay scraper.