JackBister/logsuck

Speed up search by not using OFFSET for paginating when filtering results

JackBister opened this issue · 0 comments

When searching for events, they are retrieved in batches of 1000 using OFFSET/LIMIT. The reason why we can't retrieve all results in one query is because SQLite locks itself while the query is running, so we can't write the results to the JobResults table while we have the search rowset open.

The current implementation becomes slower and slower the more results we find because OFFSET gets more expensive the bigger the offset is. By using https://stackoverflow.com/questions/14468586/efficient-paging-in-sqlite-with-millions-of-records we should be able to speed up the search significantly. Testing directly against the database with ~1M events, retrieving page 200 went from taking 600ms to 3ms.