toluaina/pgsync

pgsync on daemon mode dont insert into opensearch

Cbudano opened this issue · 1 comments

PGSync version: 2.5.0

Postgres version: Aurora 14.5

Elasticsearch version: Opensearch 2.3

Redis version: 5.0.7

Python version: 3.8.10

Problem Description:
If i run pgsync everything works, i get the data inserted into opensearch, but if i try to run it using the --daemon flag, pgsync detect the changes on the rds, but it never insert them into opensearch

the only thing i see is:

Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...
Sync client_facing:blue-sky-rules-schema Xlog: [1] => Db: [1] => Redis: [total = 1 pending = 0] => OpenSearch: [0]...

i tried deleting the construct file and start the process again, still same issue.

i also followed this steps:

Did you run bootstrap successfully?
did you alter the structure of the schema.json
There is a checkpoint file which maintains the state.
Try deleting .construct_pg_posts and rerun

if i restart pgsync on deamon mode, it inserts the rows that were pending from the previous run, but then any new row is not inserted.

the schema definition im using is:

[

{
"database": "client_facing",
"index": "blue-sky-rules-schema",
"nodes": {
"table": "v_security_stats",
"schema": "ref_data",
"columns":
[
"source",
"status",
"secid",
"comp_id",
"cusip",
"issuer_name",
"symbol"
],
"primary_key": ["secid"]
}
}
]

Error Message (if any):



My guess is this has to do with AWS Aurora. PGSync has customs modifications for AWS RDS.

can you please run this query on your db

SELECT name FROM pg_settings ORDER BY name