Sends scrapy output to Google Sheet
- Head to Google Developers Console and create a new project (or select the one you already have).
- In the box labeled
Search for APIs and Services
search forGoogle Drive API
and enable it. - In the box labeled
Search for APIs and Services
, search forGoogle Sheets API
and enable it.
- Enable API Access for a Project if you haven’t done it yet.
- Go to “APIs & Services > Credentials” and choose “Create credentials > Service account key”.
- Fill out the form
- Click “Create” and “Done”.
- Press “Manage service accounts” above Service Accounts.
- Press on ⋮ near recently created service account and select “Manage keys” and then click on “ADD KEY > Create new key”.
- Select JSON key type and press “Create”.
- Go to your spreadsheet and share it with a client_email from the step above. Just like you do with any other Google account. If you don’t do this, you’ll get a gspread.exceptions.SpreadsheetNotFound exception when trying to access this spreadsheet from your application or a script.
- Move the downloaded file to
~/.config/gspread/service_account.json
. Windows users should put this file to%APPDATA%\gspread\service_account.json
.
- Install Python3
- Install the dependencies
pip install -r requirements.txt
- Scrapy for scraping
- Pandas for cleaning up raw data
- Gspread for updating Google sheet
Set the following in main.py file
WORKSHEET_NAME = 'results'
GSHEET_URL = 'https://docs.google.com/spreadsheets/d/...'
CREDENTIALS_FILE = "xyz.json"
Set up crontab to run every Monday at 10 a.m. by using these lines:
0 10 * * mon sh /path/to/runner.sh