awsdocs/amazon-rds-user-guide

pg_transport Missing Documentation and pitfalls

Ilhicas opened this issue · 10 comments

Ilhicas commented

I'm currently running into an issue with pg_tranport that only outputs the following message: ERROR: failed to download file data
SQL state: XX000

There is no documentation on errors that are outputted from the function, and given the closed source nature of the function, there is simply no way to understand what might be missing.

miketwenty1 commented

I'm seeing this issue all over the place with running pg_transport multiple times. First time works. Consecutive times with other databases don't work. Must something going on because pg_transport doesn't work the second time unless I reboot the DB.

  • 👍1
baniol commented

Also experiencing the issue. My workaround was to create a new migration database in target RDS for each source database and running SELECT there once.

OzRgz commented

I'm experiencing same issue, with a database around 330 GB always fails at 9-10% with 'failed to download file data' error. Any update?

matti-saemi commented

I am facing the same issue as well. I was hoping to use this extension. Will there be any fix for this anytime soon? or should I go back to using pg_dump :(?

OswaldoRGZ commented

Have anyone tried with this new Postgresql 12 on AWS RDS? my last atempt was with postgresql 10.x

Ilhicas commented

This was tried with a 11.5 version, haven't tried since, as we've resorted to using pg_dump and restore automated with good old ansible trying to mimic the behaviour by suspending write on the source to avoid inconsistencies.

miketwenty1 commented

This aws blog has some parameter suggestions for things like "num_workers" and "worker_mem".
I noticed bumping resources for the DBs involved and setting parameters helps, but doesn't solve the issue completely.

OswaldoRGZ commented

Somehow its working now, tested on PostgreSQL 12.4.
400 GB in 20 mins with two m5.xlarge.

randyurbano commented

Thanks for the feedback. I'm closing this old issue. If changes are needed to the documentation in the RDS User Guide, re-open the issue and specify the topic and the changes that are needed.

santiagortiiz commented

Hi, I have encountered this problem recently.

What I notice is that the pg_transport.num_workers must be configured according to the database instance type and its capabilities.
I have been using Postgres version 14 on a db.t3.micro with 20GB of storage.

When I set pg_transport.num_workers > 3, I get the same error (Failed to download file data) at some point (50%, 30%).

So in the end I solve it by setting the workers to 2.

  • 👍1