aws-redshift-to-rds: KeyError

Question

aws-redshift-to-rds: KeyError

rimusz opened this issue 8 years ago · 12 comments

trying to use cabal-sandbox/bin/aws-redshift-to-rds --from source_db --from-table schema_name.table_name --to destination_db --to-table schema_name.table_name command, but getting the error aws-redshift-to-rds: KeyError "analytics.type" where analytics is the DB in redshfit and RDS.

aws-redshift-to-rds.config has correct connection settings.

Answer 1 · 2017-03-19T00:35:10.000Z

could you copy the content (masked) of aws-redshift-to-rds.config? Seems like possibly a formatting issue.

Answer 2 · 2017-03-19T15:29:57.000Z

here we go:

redshift_name {
  type = "redshift"
  connection = "user='root' password='password' host='redshift-xx.xx.io' port=5439 dbname='analytics'"
}

rds_name {
  type = "rds"
  connection = "user='root' password='password' host='analytics.db.xxxx.io' port=5432 dbname='analytics'"
}

Answer 3 · 2017-03-19T15:44:20.000Z

hmm I don't see any problem with the config, my guess is that it was because you used

cabal-sandbox/bin/aws-redshift-to-rds --from source_db --from-table schema_name.table_name --to destination_db --to-table schema_name.table_name

with source_db and destination_db as "analytics". If so, can you try the command again with source_db as redshift_name and destination_db as rds_name?

Answer 4 · 2017-03-19T15:49:41.000Z

ok, will check that now

Answer 5 · 2017-03-19T15:59:42.000Z

hmm, what do you mean by: with source_db as redshift_name and destination_db as rds_name?
what I have to use there?

Answer 6 · 2017-03-19T16:13:12.000Z

also tried:

.cabal-sandbox/bin/aws-redshift-to-rds \
  --from-type redshift --from-db "${REDSHIFT_URL}" --from-table aaaa_aaaa.users \
  --to-type rds --to-db "${DATABASE_URL}" --to-table aaaa_aaaa.users

and getting an error:

mirroring from redshift to RDS: xxxx_xxxx.users -> xxxx_xxxx.users...
aws-redshift-to-rds: SqlError {sqlState = "08001", sqlExecStatus = FatalError, sqlErrorMsg = "could not establish connection", sqlErrorDetail = "could not translate host name \"redshift-xxxx.xxx.io\" to address: Name or service not known\n", sqlErrorHint = ""}

pinging to redshift-xxxx.xxx.io shows the proper IP.

Answer 7 · 2017-03-20T01:12:49.000Z

You will have to use exactly "redshift_name" and "rds_name" as you put in the config file :)

Answer 8 · 2017-03-22T08:37:10.000Z

@rimusz did you manage to get it resolved?

Answer 9 · 2017-03-22T15:51:32.000Z

was urgently needing to get it to work, so found another solution for now

Answer 10 · 2017-03-22T16:03:53.000Z

Okay, I'd close this issue for now, as I've just tested again and it works for me.

Answer 11 · 2017-03-22T16:07:03.000Z

can you give me an example "redshift_name" and "rds_name", please?
next time I know it should work.

Answer 12 · 2017-03-22T16:13:11.000Z

The config looks like this:

  type = "redshift"
  connection = "user='z' password='z' host='z' port=z dbname='z'"
}

dave_rds_singapore {
  type = "rds"
  connection = "user='z' password='z' host='z' port=z dbname='z'"
}

The execution looks like this

    sudo docker run aws-redshift-to-rds .cabal-sandbox/bin/aws-redshift-to-rds \
    --from dave_redshift --from-table $table \
    --to dave_rds_singapore --to-table $table --cascade