itpp-labs/sync-addons

Implement pagination

Closed this issue · 0 comments

Dysta commented

Hello, we are using your modules to add a way to extract Odoo data to Airbyte with a custom connector. It works very well.

The problem is that the amount of data in our models is growing rapidly, causing each query to become slower and slower. This also causes some latency in the Odoo application when a large amount of data is extracted.

A good way to prevent slowdowns from happening when we do an extract is to paginate our extractor with Airbyte like this
image

Unfortunately, the module doesn't seem to provide a way to request just a few data in a paginated request.
The API provide a way to request the data in a paginated way with the call route. Unfortunately, this endpoint use a PATCH method that isn't supported by AirByte.
A workaround to this can be to add POST method in https://github.com/itpp-labs/sync-addons/blob/16.0/openapi/controllers/api.py#L194 and https://github.com/itpp-labs/sync-addons/blob/16.0/openapi/controllers/api.py#L173

In the screenshot above, the "page size" is the returned input number (like 100, 200, 1000, etc.) and the "start of page" is the page number (like 1, 2, etc.)

The method to pass these parameters is not just in the URL as a request, Airbyte supports a few other methods.
image

Should you be able to implement a paging system in a future release?

Thank you 😄