crawl all laptops from amazon and save them in xml files in python using Scrapy, scrapy-user-agents, scrapy-rotating-proxies
install scrapy using the following command : pip install Scrapy
See the Documentation here : https://doc.scrapy.org/
install the package using the following command: pip install scrapy-user-agents
See the Documentation here : https://github.com/hyan15/crawler-demo/tree/master/crawling-basic/scrapy_user_agents
install the package using the following command: pip install scrapy-rotating-proxies
See the Documentation here : https://github.com/TeamHG-Memex/scrapy-rotating-proxies
navigate to Amazon-Crawler/crawler/crawler then run the command : scrapy crawl amazon
after the spider finishes, you can find the xml files insind Amazon-Crawler/crawler/crawler/product-xml-files and the links of the crawled products in Amazon-Crawler/crawler/crawler/laptops.txt
you will have to paste the link of the department you want to crawl inside start_urls array which is located in Amazon-Cralwer/crawler/crawler/spiders/amazon.py