Speed up filesystem queries (parallel `WalkDir` ?)
maxlandon opened this issue · 4 comments
Request
Currently, the ActionFiles seems to read directory contents in a sequential way.
Reading large numbers of directory entries thus significantly increases the time to wait for completions.
An example is the /usr/bin directory, where I could test on:
- A system with ~500 file entries in. Completion processing is immediate.
- A system with ~1800 file entries. Completion takes (on ZSH) a bit less than a second, and the latter will disable color processing.
Proposed solution
Enable parallelizing the queries for file entries, then sort them if needed.
Anything else?
No response
I got close to 4000 in /usr/bin and it takes about 54ms to create the values in carapace.
Likely that this can be improved but to my knowledge the bigger performance impact at the moment is the shell.
zsh is especially bad with its interpretion engine and all the regex matching but even in elvish the color processing is a bit slow.
Okay thanks for the quick answer, and yes obviously I didn't do any benchmarking in the shell before opening.
Most of the overhead is on my own code then !
Nah, it certainly is an issue. I've noticed it during the "which" completion but so far it was only happening there for me (other folders usually don't have that much content).
After a bit of work on my code I realized I was very guilty too.
Regexp checks and replacements for color in those stuff quickly gets costly when you don't pay attention indeed.