Scraper to fetch Oil prices from different websites and updates them to spreadsheet.
Project Contract
·
Client - Ian Markowitz
·
Status - On Going
- Prerequisite
- Unpacking
- Installation Dependencies
- Understanding Project File
- Running The Script
- Show Your Support
- Contact Me
- Windows/MacOS
- Python 3.6 or above
- Chrome Webdriver
Get Chrome WebDriver according to your Google Chrome Version
- Unzip the OilPricesAggregator-Scrapper.zip in desired location. Let's assume that we unzip the file in location Documents/OilPricesAggregator
- After this the path of the Project lookes like Documents/OilPricesAggregator
- The project folder should look something like this:
- Start terminal and type below command in terminal to point to the project folder:
~$ cd Documents/OilPricesAggregator
- Now we need to download all the dependencies required to run the script. For this we will type below command in terminal:
~$ pip3 install -r requirements.txt
All outpoutl files will be stored in this folder.
This file contains all the dependencies that we need to install on our system. You can delete this file but its Ok to keep it there and forget that it exists.
This file is the main script that we need to run to get the desired output. Please never touch this file.
This is a chome webdriver. This is version 88.0.4324.146 and is supported by MacOs. In case you have different version of Chrome installed in your system or you have OS other than MacOs then, please replace it with the one that you will get from here. Make sure you download the driver that supports your OS and have same version as that of chrome installed on your system.
This file needs to be edited everytime you run the script and so it needs some explation...
-
pathToScript => Path where script.exe is stored on your system.
-
pathToChromeDriver => Path where chromedriver.exe is stored.
-
google_sheet_cred => Path to the json file having credentials for google sheets.
-
google_sheet_name => Name of the sheet inside spreadsheet in which the data is to be saved.
-
website => This is the list of websites from which data is to be fetched.
Please watch this video and follow below timestamps:
-
0:10-0:30 => you create a new spreadsheet. This is the spreadsheet where the oil prices will be updated. Rename this spreadsheet as 'Oil_Prices'.
-
0:31-3:00 => setting up google drive credentials and enabling google sheet in your google account.
-
At 2:30 => you have created a json file with credentials. download this file and rename this to google_sheet_cred.json and place it inside GoogleSheet_Cred folder
-
3:30-3:53 => Giving permission to the google drive to access the spreadsheet.
-
3:53-* => You can skip rest part.
So before running the script please make sure of two things:
- You have followed Google Sheet Stup properly.
- The pointer in terminal is pointing to the project forlder. If not then use below code:
~$ cd Documents/OilPricesAggregator
Now we need to enter below code to execute the script.
~$ python3 script.py
Now just grab yourself a pint of 🍺 and let the script do its task.
If you are happy with my work then please give me ⭐⭐⭐⭐⭐ rating and also leave really nice recommendation/feedback on upwork. This will help me a lot in getting more project. A small and happy bonus is always appreciated 🤩. Also kindly rememeber me if you have any such project or any scraping projects.
Thank You for giving me opportunity to work on this project.
Siddhant Shah - Please feel free to connect to me in case there is any issue in the script or any changes are required. You can contact on below mentioned connects