Local or Google Cloud Function for pulling Nozzle ranking data into BigQuery
- Create a New Project in GCP. We used
nozzle-ranking-data
. - Activate BigQuery and Pubsub APIs.
- Create Service Account with admin roles for Big Query. Make sure to send the service account email to Nozzle to add to your datasets.
- Generate JSON key for service account.
- Create Pubsub topic. We used
nozzle-trigger-run
. - Create BigQuery dataset. We used
client_ranking_data
. - Create table
config
in dataset and map to Google sheet with columns: client, nozzle_domain, and nozzle_dataset. - Create Python 3.7 Cloud Function. Set your Pubsub topic as the trigger. Timeout: 270s.
- Add contents of nozzle2bg.py file into main.py of the function.
Set
run_job
as function handler. - A requirements below to requrements.txt.
- Add your Service Account JSON as service_account.json.
- Function structure: /main.py /requirements.txt /service-account.json
- Save Cloud Function
- Set a Cloud Schedule CRON tast k to send
{'job': 'run all'}
to your Pubsub topic on any interval.0 1 * * *
is daily at 1am.
google-cloud-bigquery
google-cloud-pubsub
pandas
tqdm