Slow to clear search result
Closed this issue · 14 comments
Hi,
I am working on a large code base. If I search in CodeQuery gui (windows 7) for a symbol and get like 20.000 results then the search is very fast around 1 second. But any search I do after getting such a large result takes around 1 minute. If I make the same search (that just took 1 minute) again then it only takes 1 sec (if the search give few hits). When having a result with many hits like the 20.000 I restart the gui (which takes a few sec), then any search is fast again.
My conclusion is that clearing the current result when making a new search is very slow somehow (if it contains many hits). Is that something you can do anything about?
Best regards,
Ivan
I'll take a look. Thanks.
Are you able to reproduce?
Hi @Masterivan
So far I couldn't reproduce it yet with a few thousand results. Need to find a bigger source base.
If I make a search that returns 2316 results it takes 3 sec first time, then if I perform the same search again it takes 10 sec. Search time probably also is affected by how big you you codequery.db is. In my case it is 1.1 GB. (my tags file is ~ 77 MB, and cscope.out is 228 MB.
Hi @Masterivan
I've tried with the Linux 3.10 kernel source code (excluding arch) and it generated a codequery.db file of size 1.7GB. I don't have performance issues. I suspect that it is related to sqlite.
Now I'm using the latest sqlite. Try with the latest CodeQuery v0.19.0 on Windows. It's using sqlite 3.17.0.
Let me test it further. Maybe it could be an issue.
@Masterivan OK. I could reproduce it now and I know the root cause.
Hi Ruben, nice to hear that you can reproduce now and even found the root cause. What was the difference between when it was slow and fast?
Looking forward to see if you are able to do mitigate the issue.
Hi @Masterivan
The problem was with the clearing of the QTreeWidget. Turns out that I had been using it wrongly all these years. Although the QTreeWidget adds pointers to new'ed QTreeWidgetItems, there was no need to separately maintain and delete QTreeWidgetItems manually. It is deleted automatically by Qt when the QTreeWidget is cleared. It was the manual deleting of a large list (> 8K items) of these pointers that took a long time. With around 40K items, it would simply hang.
Now I just let Qt do it. With the commit to fix this, clearing a previous list of 835K items and replacing it with a new list of 835K items, takes only 3.2s on my PC. This excludes the time taken to query the database, which is about 4.6s on my PC. In this commit, I also changed some counting variables from unsigned int to (at least) long because unsigned int can only handle 65K while the Linux kernel 3.10 source base can produce millions of results depending on what you search for.
I think I can improve the performance further if I do the database query in one thread and the clearing of the previous list in another thread, but that would probably save only one second for 835K items which is not much of a gain.
@Masterivan Please try it out and let me know if you're satisfied. Thanks.
Hi Ruben,
Will there then be a 0.20 anytime soon to take advantage of your nice new improvements?
Or how can I try it out before?
Hi @Masterivan
I will release it on Saturday. For now, you can try out a beta version here (download from here):
https://bitbucket.org/ruben2020/codequery/downloads/cqsetup_001901_win32.exe
Thank you.
This site (bitbucket.org) seems slow and I did not verify checksums.
Hi Ruben,
I made the same 2 searches in the old 0.19 and the new 1901 version
In the old (0.19) it took 99 sec to finish the second search.
In the new 1901 it took about 1 sec, an I am very satisfied with the solution.
Br, Ivan
Great. Thanks, Ivan! This will be released in 0.20.0 within 24 hours.