Assignment for SE role
- Started by creating a cluster in MongoDB Atlas
- Trying to connect over MongoDB using python
- Scrapped the webpages and then stores it in a json file
- Using Scrapy Mongodb Pipelines uploading data to mongoDB
- Performing Queries
First I installed scrapy and pymongo. After installing scrapy I was not able to initialize the repository properly. After trying more than 10 times, I got success from using Anaconda environment. After this I followed the documentation and successfully scraped quotes tot scrap website. After this I changed the requirements accordingly and followed similar steps for https://www.net-a-porter.com/.
Current issues in the code : Some values are changed to None. No sale price available.
Reasons for Delay:
- Scrapy Installation took time
- Trying to find other way around but it didn't worked