permitio/opal-fetcher-postgres

database update

alazarbeyeneazu opened this issue · 8 comments

alazarbeyeneazu commented

when I update the databases data it is not correct the values of OPA to the latest values

orweis commented

Hi, it's hard to say exactly what the issue is without further details;
But If I have to guess - let me ask did you trigger an update event?
These are not automatic - you need to send an event to OPAL to let it know the database was update.
The data fetcher only takes care of fetching the data not tracking the changes.

alazarbeyeneazu commented

I was looking if there was an automatic way to trigger it as it looks for changes on policies update.

orweis commented

There's no automatic magic way :) Postgres doesn't support webhooks by default.
You need to do one of:

  1. Have the code writing to Postgres trigger an update
  2. Have another service polling on the DB and triggering an update
  3. Have the Postgres SQL query use the notify command to send an event out
alazarbeyeneazu commented

There's no automatic magic way :) Postgres doesn't support webhooks by default. You need to do one of:

1. Have the code writing to Postgres trigger an update

2. Have another service polling on the DB and triggering an update

3. Have the Postgres SQL query use the notify command to send an event out

thank you I will use one of them. you are helpful.

lcarlos10 commented

Hi friends, @alazarbeyeneazu / @orweis, Could anyone of you explain how you did it? How can I trigger an update to OPAL to say it "Hey, I did an insertion on postgresql, refetch the data". Thanks!

orweis commented

Hi friends, @alazarbeyeneazu / @orweis, Could anyone of you explain how you did it? How can I trigger an update to OPAL to say it "Hey, I did an insertion on postgresql, refetch the data". Thanks!

Just follow the guide here: https://docs.opal.ac/tutorials/trigger_data_updates
And for Postgres Specifically here: https://github.com/permitio/opal-fetcher-postgres

lcarlos10 commented

Thanks for your answer orweis. I checked those links. When I followed option 2 OPAL is replacing all the data with the new PUT call body. The intention is just notify to OPAL to refetch same query. My scenario: I'm fetching policies from postgresql query. Other service is in charge of inserting a new record there and then I want to say OPAL for retrieving the new query including the new record. I can't call the endpoint with all the policies, including the new one.

Thanks in advance @orweis

orweis commented

Thanks for your answer orweis. I checked those links. When I followed option 2 OPAL is replacing all the data with the new PUT call body. The intention is just notify to OPAL to refetch same query. My scenario: I'm fetching policies from postgresql query. Other service is in charge of inserting a new record there and then I want to say OPAL for retrieving the new query including the new record. I can't call the endpoint with all the policies, including the new one.

Thanks in advance @orweis

You need to set different sub-keys for the dst_path param;
or use the PATCH save method - https://docs.opal.ac/tutorials/trigger_data_updates#using-patch-save-method.
You could also create a custom fetch provider to further customize this flow