pgsync on daemon mode dont insert into opensearch
Cbudano opened this issue · 1 comments
PGSync version: 2.5.0
Postgres version: Aurora 14.5
Elasticsearch version: Opensearch 2.3
Redis version: 5.0.7
Python version: 3.8.10
Problem Description:
If i run pgsync everything works, i get the data inserted into opensearch, but if i try to run it using the --daemon flag, pgsync detect the changes on the rds, but it never insert them into opensearch
the only thing i see is:
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
i tried deleting the construct file and start the process again, still same issue.
i also followed this steps:
Did you run bootstrap successfully?
did you alter the structure of the schema.json
There is a checkpoint file which maintains the state.
Try deleting .construct_pg_posts and rerun
if i restart pgsync on deamon mode, it inserts the rows that were pending from the previous run, but then any new row is not inserted.
the schema definition im using is:
[
{
"database": "client_facing",
"index": "blue-sky-rules-schema",
"nodes": {
"table": "v_security_stats",
"schema": "ref_data",
"columns":
[
"source",
"status",
"secid",
"comp_id",
"cusip",
"issuer_name",
"symbol"
],
"primary_key": ["secid"]
}
}
]
Error Message (if any):
My guess is this has to do with AWS Aurora. PGSync has customs modifications for AWS RDS.
can you please run this query on your db
SELECT name FROM pg_settings ORDER BY name