/bq-table-copy-automation

Copy table from one dataset to another in google big query using bash script

Primary LanguageShellMIT LicenseMIT

bq-table-copy-automation

Bash script to read a mapping Yaml file & copy table from one dataset to another dataset for Google BigQuery

Working on

  • Linux
  • OSX

Getting Started

Prerequisites

  1. Google Cloud SDK
  2. Make sure that Python 2.7 is installed on your system:

    python -V

  3. Initialize the GCloud SDK.

Usage

  1. Copy the yaml_parser.sh, mapping.yaml & bq_cp_util.sh files locally/on the server where gcloud is installed & authenticated.
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/yaml_parser.sh
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/mapping.yaml
wget https://github.com/khichar-anil/bq-table-copy-automation/blob/master/bq_cp_util.sh

  1. Modify the mapping.yaml as per your source & target datasets. mapping.yaml

    Yaml Attribute Sample Value Description
    sourceProjectId source-project Source table gcp project id.
    targetProjectId target-project Target table gcp project id. Source & target project Ids can be same, if we want to copy table withing same project.
    sourceDataset source_ds Source table BQ dataset name.
    targetDataset target_ds Target BQ dataset name. Source & target dataset can be same, if we want to copy table withing same dataset.
    tables > source source_tbl_1 Source table name.
    tables > target target_tbl_1 Target table name.

  2. ./bq_cp_util.sh

Credits