counsel-projectile-find-file extremely sluggish in larger repositories.
dminuoso opened this issue · 6 comments
When switching the project to https://github.com/NixOS/nixpkgs (~25,000 files), or calling M-x counsel-projectile-find-file inside the same repository, typing a few letters (say "postfix") causes emacs/ivy to hang for a few seconds on my laptop until the ivy minibuffer refreshes and displays the filtered strings. In comparison, projectile-find-file is much more snappy, which seems to respond within a hundred milliseconds or so.
I also tried to set counsel-projectile-find-file-more-chars to 2 to no avail.
My counsel/ivy/projectile config is completely left at default.
Here's the output of CPU and memory profilers:
CPU profiler
- command-execute 4889 57%
- call-interactively 4889 57%
- funcall-interactively 4770 55%
- counsel-projectile-switch-project 4719 55%
- ivy-read 4717 55%
- ivy-call 4622 53%
- counsel-projectile-switch-project-action 4622 53%
- counsel-projectile-switch-project-by-name 4622 53%
- #<lambda 0xf428ca0> 4620 53%
- counsel-projectile 4620 53%
- ivy-read 4617 53%
- read-from-minibuffer 4521 52%
- ivy--queue-exhibit 4471 52%
- ivy--exhibit 4471 52%
- ivy--update-minibuffer 4464 52%
- ivy--filter 3017 35%
+ counsel-projectile--matcher 1719 20%
+ ivy--recompute-index 1297 15%
ivy--regex-plus 1 0%
+ ivy--format 1447 16%
+ ivy--insert-minibuffer 6 0%
+ minibuffer-inactive-mode 2 0%
+ timer-event-handler 2 0%
undo-auto--add-boundary 1 0%
+ ivy--reset-state 64 0%
+ projectile-project-p 1 0%
+ projectile-prepend-project-name 1 0%
+ hack-dir-local-variables-non-file-buffer 2 0%
+ read-from-minibuffer 75 0%
+ ivy--reset-state 5 0%
+ projectile-project-p 1 0%
+ projectile-project-root 1 0%
+ execute-extended-command 51 0%
+ byte-code 119 1%
- ... 3673 42%
Automatic GC 3673 42%
+ evil-repeat-post-hook 1 0%
+ evil--jump-hook 1 0%
+ direnv--maybe-update-environment 1 0%
+ redisplay_internal (C function) 1 0%
+ timer-event-handler 1 0%
undo-auto--add-boundary 1 0%
Memory profiler
- command-execute 3,380,453,797 99%
- call-interactively 3,380,453,797 99%
- funcall-interactively 3,377,042,321 99%
- counsel-projectile-switch-project 3,373,287,970 99%
- ivy-read 3,373,284,421 99%
- ivy-call 3,373,145,049 99%
- counsel-projectile-switch-project-action 3,373,145,049 99%
- counsel-projectile-switch-project-by-name 3,373,145,049 99%
- #<lambda 0xf428ca0> 3,373,122,051 99%
- counsel-projectile 3,373,122,051 99%
- ivy-read 3,373,111,234 99%
- read-from-minibuffer 3,313,649,682 98%
- ivy--queue-exhibit 3,313,561,434 98%
- ivy--exhibit 3,313,561,434 98%
- ivy--update-minibuffer 3,313,181,018 98%
+ ivy--format 1,742,675,931 51%
- ivy--filter 1,570,505,087 46%
- ivy--recompute-index 1,502,558,612 44%
- ivy--completing-fname-p 1,502,557,556 44%
- counsel-projectile--project-buffers-and-files 1,502,557,556 44%
- projectile-current-project-files 1,498,212,421 44%
- projectile-project-files 1,498,035,757 44%
- projectile-dir-files-alien 1,498,035,757 44%
- projectile-files-via-ext-command 1,463,755,470 43%
- shell-command 1,400,129,372 41%
call-process-shell-command 1,400,084,492 41%
make-temp-file 3,152 0%
+ delete-file 3,048 0%
+ split-string 61,078,085 1%
string-trim 94,191 0%
generate-new-buffer 28,821 0%
+ projectile-get-sub-projects-files 134,115 0%
+ projectile-project-vcs 16,276 0%
+ projectile-acquire-root 176,664 0%
+ counsel-projectile--project-buffers 3,170,679 0%
remove 946,176 0%
+ projectile-project-root 183,804 0%
file-relative-name 44,476 0%
+ ivy--preselect-index 1,056 0%
+ counsel-projectile--matcher 67,944,371 2%
+ ivy--regex-plus 2,104 0%
+ ivy--insert-minibuffer 318,648 0%
+ ivy-set-text 59,672 0%
ivy--input 2,096 0%
+ redisplay_internal (C function) 36,424 0%
+ command-execute 6,128 0%
+ direnv--maybe-update-environment 3,168 0%
+ ivy--reset-state 59,435,965 1%
+ ivy--update-prompt 1,024 0%
+ projectile-prepend-project-name 8,779 0%
+ projectile-project-p 1,488 0%
+ projectile-maybe-invalidate-cache 550 0%
+ hack-dir-local-variables-non-file-buffer 21,929 0%
+ read-from-minibuffer 105,273 0%
+ ivy--reset-state 29,600 0%
+ ivy--update-prompt 1,024 0%
abbreviate-file-name 2,048 0%
+ projectile-prepend-project-name 585 0%
+ projectile-project-p 458 0%
+ projectile-project-root 458 0%
+ execute-extended-command 3,754,351 0%
+ byte-code 3,411,476 0%
+ redisplay_internal (C function) 18,740 0%
evil-repeat-pre-hook 8,188 0%
... 0 0%
The immediate problem appears to be
counsel-projectile/counsel-projectile.el
Line 1632 in 06b03c1
This collection function is called each time a new character is added to the input (but not when characters are removed, I guess ivy-read simply caches the filtered results).
In a non-trivial repository this can incur several hundred milliseconds on a single character stroke, so if one rapidly types in foo.nix, this will call the function 6 times in rapid succession and hanging for a few seconds.
I am experiencing the same issue. I use counsel-projectile to open file, when I type the file name, the issue occurs. But if I just paste the file name without typing, I get the result quickly.
Hi, thanks for reporting. I guess you are right that the issues comes from the use of a collection function. I didn't notice it because I don't use large repositories. It is probably worth trying to change this, but unfortunately it will probably take a while until I can look at it, sorry about that (PRs welcome of course).
Hello,
I just want to say that I'm having the exact same issue.
The counsel-projectile command is very laggy with a folder of about 200 files while projectile-find-file feels very responsive.
Hi,
I also see this issue on macOS. If I watch active processes on my machine whilst running counsel-projectile-find-file, counsel-projectile is causing Projectile to re-index the project for every(?) keystroke I do in the mini-buffer. When Projectile indexes a project it will call external commands like git, find, etc. and I believe this is where the slowness comes from. The CPU profile above doesn't reveal this, probably because Emacs is simply waiting for external commands to finish. shell-command does show up in the memory profile, however. I would guess the root cause is the constant re-indexing, but I didn't look into the code to figure out why that happens.
As a workaround I found that if I enable caching of project files in Projectile (with (setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar to projectile-find-file, at least for the projects I work on (few thousand files).
As a workaround I found that if I enable caching of project files in Projectile (with
(setq projectile-enable-caching t)), it stops calling external commands and responsiveness becomes similar toprojectile-find-file, at least for the projects I work on (few thousand files).
Great tip, thanks for sharing! I recommend reading the docs on caching to understand how/when you would need to manually purge the cache.
Another thing I noticed that it was still sluggish when switching between projects with counsel-projectile-switch-project. Turns out that this is due to the default action being counsel-projectile-switch-project-action which also adds buffers to the set of completion candidates. Setting it to counsel-projectile-switch-project-action-find-file instead makes it snappy, too.