long2ice/meilisync

Postgres `get_full_data` is slow for large datasets

MattExact opened this issue · 1 comments

I found that get_full_data was unusably slow for datasets >1m rows.
I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.

I think a better implementation would be to use a server-side cursor and use fetchmany to fetch size number of rows at a time.
See Psycopg docs for more.

OK, could you please make a PR?