PGRES_COPY_BOTH
Closed this issue · 1 comments
Getting this error after doing the initial full table sync, during the first incremental sync, about 1.5min after the "Starting Logical Replication" INFO message:
psycopg2.DatabaseError: error with status PGRES_COPY_BOTH and no message from the libpq
Not sure what other information to include. Seems like "COPY_BOTH" is a type from Postgres that only appears during logical replication. psycopg2 added support for it in 2016 or so. Not sure why there's an issue here.
Running Postgres 11.4 on RDS.
full error:
00:40:09 CRITICAL connection already closed 00:40:09 Traceback (most recent call last): 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/sync_strategies/logical_replication.py", line 353, in sync_tables 00:40:09 msg = cur.read_message() 00:40:09 psycopg2.DatabaseError: error with status PGRES_COPY_BOTH and no message from the libpq 00:40:09 During handling of the above exception, another exception occurred: 00:40:09 Traceback (most recent call last): 00:40:09 File "/usr/local/bin/tap-postgres", line 10, in <module> 00:40:09 sys.exit(main()) 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/__init__.py", line 705, in main 00:40:09 raise exc 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/__init__.py", line 702, in main 00:40:09 main_impl() 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/__init__.py", line 696, in main_impl 00:40:09 do_sync(conn_config, args.catalog.to_dict() if args.catalog else args.properties, args.config.get('default_replication_method'), state) 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/__init__.py", line 671, in do_sync 00:40:09 state = sync_logical_streams(conn_config, list(streams), state, end_lsn) 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/__init__.py", line 581, in sync_logical_streams 00:40:09 state = logical_replication.sync_tables(conn_config, logical_streams, state, end_lsn) 00:40:09 File "/usr/local/lib/python3.7/site-packages/tap_postgres/sync_strategies/logical_replication.py", line 377, in sync_tables 00:40:09 pass # recalculate timeout and continue 00:40:09 psycopg2.InterfaceError: connection already closed
I figured this out. Apparently that means the connection timed out and was disconnected. RDS sets wal_sender_timeout to 30 seconds by default and apparently tap-postgres took about 1.5min to use the connection again. I increased wal_sender_timeout to 10min and this hasn't occurred again.