/load-data-2-bigquery

A Collection of BigQuery Productivity Scripts

Primary LanguageJupyter Notebook

BigQuery Productivity Scripts

A collection of scripts written in Python for BigQuery to boost productivity.

Before executing the script, please following instructions from BigQuery and establish the connection to the BigQuery API:

import os
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = "<your.json file path>"

Upload CSV from link to BigQuery (csv-link-2-bigquery.py)

As the name implies, with the download link to the CSV file, this script will upload to your dataset automatically. Could've overcomplicated the script with all the clean up scripts but I figured it would probably be better to it on BigQuery with SQL.

Inputs:

  1. Link to CSV
  2. Project ID and Dataset ID in project_id.dataset_id format
  3. Name the Table ID
  4. Done!

API json to BigQuery (json-link-2-bigquery.ipynb)

Upload directly from APIs to BigQuery, modify the .ipynb to you liking.

To-dos / Ideas:

  • Excel convestion to CSV then upload
  • JSON Upload
  • Check folder for new CSV/JSON file, then upload.
  • After all the above, modularise(idk if that's a word)