/redshift-migrator

Migrate data between redshift clusters

Primary LanguageElixirApache License 2.0Apache-2.0

redshift-migrator

Migrate data between redshift clusters.

How it works

  1. Create the any schemas and tables of the source cluster in the destination cluster.
  2. Unload each table from the source cluster into a s3 bucket.
  3. Copy each table from the s3 bucket into the destination cluster.
  4. Delete any duplicate data in each table using the id column as uniqueness key.

Requirements

  • elixir 1.13+
  • aws cli
  • Source and Destination Redshift Clusters with the same user name and database name.
  • Add a role to both clusters to read/write to the s3 bucket that the data will be unloaded
  • Create the admin View to get the DDL for a table on the source cluster.

Usage

$ elixir migrator.exs -s source-redshift-cluster -d destination-redshift-cluster -r "arn:aws:iam::$ACCOUNT_ID:role/service-role/$ROLE_NAME" -u $DB_USER -n $DB_NAME -b $S3_BUCKET