opensanctions/yente

Implement incremental scans

Closed this issue · 1 comments

pudo commented

cc @everplays - because you mentioned it.

There's two tiers to this:

  1. Implement query parameters for the /search and /match APIs that allow a since= search. They would probably just filter on since > entity.last_changed. This gives you the ability to do incremental queries with the state managed externally.
  2. Implement some sort of state in the API itself. In this scenario, a client would submit a /match-style entity and a callback mechanism (e.g. a webhook) and the API would then store this query, execute it at a set interval and notify the client whenever a new result is available. This could also be implemented a REST primitive, a la POST /alerts and GET /alerts?since=xxx.

I'm curious: which of these did you have in mind?

everplays commented

I think the first option would be less work and also more flexible. The second solution would require more infrastructure:

  • queue system (with a backend: redis, rabbitmq, sqs, ...)
  • storing match entries (probably just elasticsearch/opensearch)

About the first solution: do you think the client would be responsible for keeping track of since or we can expose a bit more info in /catalog?