Postgres `get_full_data` is slow for large datasets
MattExact opened this issue · 1 comments
MattExact commented
I found that get_full_data
was unusably slow for datasets >1m rows.
I believe the culprit is the use of offset/limit pagination, which is not performant for large offsets.
I think a better implementation would be to use a server-side cursor and use fetchmany
to fetch size
number of rows at a time.
See Psycopg docs for more.
long2ice commented
OK, could you please make a PR?