live_grep freezes and eats a lot of memory.

[painhardcore]

Description

I've got not very large monorepo(167mb) with code, when I'm typing in live_grep - at some point the result freezes and I see that RAM consumption goes to the moon.

Neovim version

NVIM v0.9.5
Build type: Release
LuaJIT 2.1.1710088188

Operating system and version

macos 13.6.4 (22G513)

Telescope version / branch / rev

0.1.6 and master

checkhealth telescope

==============================================================================
telescope: require("telescope.health").check()

Checking for required plugins ~
- OK plenary installed.
- OK nvim-treesitter installed.

Checking external dependencies ~
- OK rg: found ripgrep 14.1.0
- OK fd: found fd 9.0.0

===== Installed extensions ===== ~

Telescope Extension: `fzf` ~
- OK lib working as expected
- OK file_sorter correctly configured
- OK generic_sorter correctly configured

Telescope Extension: `ui-select` ~
- No healthcheck provided

Steps to reproduce

1. Just live_grep in the folder
2. Start typing

Expected behavior

1. When I type it should cancel previous chars search.
2. If there is too much results it should stop at some point.

Actual behavior

1. freezes at some letter when typing
2. Consumes a lot of memory and CPU
3. After some time unfreezes and shows the result

If I just type "a" it would eat all memory and die.

Funny thing that if i quit the live_grep search without waiting for it to complete - it will hold the memory and wont release it.

Minimal config

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvim/site]]
local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'
local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      {
        'nvim-telescope/telescope.nvim',
        requires = {
          'nvim-lua/plenary.nvim',
          { 'nvim-telescope/telescope-fzf-native.nvim', run = 'make' },
        },
      },
      -- ADD PLUGINS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
      display = { non_interactive = true },
    },
  }
end
_G.load_config = function()
  require('telescope').setup()
  require('telescope').load_extension('fzf')
  -- ADD INIT.LUA SETTINGS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
end
if vim.fn.isdirectory(install_path) == 0 then
  print("Installing Telescope and dependencies.")
  vim.fn.system { 'git', 'clone', '--depth=1', 'https://github.com/wbthomason/packer.nvim', install_path }
end
load_plugins()
require('packer').sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua load_config()]]

[jamestrew]

Sorry I can't reproduce this. My guess is its some build issue or something. Maybe reinstalling ripgrep or neovim could help. Hard to say for certain but I don't think it's telescope's code that's directly responsible for consuming GBs of memory in a modest sized codebase.

[painhardcore]

@jamestrew I've tried kubernetes repo, but cannot reproduce the freeze.
But consumption of the memory - works. On the different scale, but still.

Now, I need to find what exactly causing this. Because in my repo this memory consumption happens much faster.

[painhardcore]

I also tried reinstalling ripgrep and neovim with master branch version - the same result

[jamestrew]

maybe it's some sort of weird edge case related to a filename in that specific repo.
if you're fine with sharing the repo, I can try on my side.

[painhardcore]

@jamestrew can't share the exact repo. The hanging memory should be reproducible with https://github.com/kubernetes/kubernetes repo.

I've tried randomly deleting files and seems to be lt caused by testdata folders with csv and json files.
And apparently file_ignore_patterns don't fully ignore them, only deletion helps.

[jamestrew]

With the kubernetes repo, it's not exactly blazing fast and takes a few gigs of memory but I'm not getting any leaks.

And apparently file_ignore_patterns don't fully ignore them, only deletion helps.

be aware that file_ignore_patterns uses lua patterns and not extended regex or globs. Unrelated to the issue, but where possible, I try to avoid file_ignore_patterns and use things like .gitignore/.rgignore or ripgrep's --glob. They'll be much faster.

[painhardcore]

@jamestrew thank you for the suggestions - .rgignore fixed it(not sure how to tweak ripgrep's --glob in telescope).

Do you know the other way to limit the output/time of execution so it won't consume a lot of memory? So experience while searching would be similar to other IDE.

[jamestrew]

live_grep can take an option called additional_args. You can use it for globs like require'telescope.builtin'.live_grep { additional_args = { '--glob', '!.git' } }.

Do you know the other way to limit the output/time of execution so it won't consume a lot of memory? So experience while searching would be similar to other IDE.

Not really that I know of. I don't think a true timeout of execution is possible without preemptive multitasking.

If I'm being honest though, if you're frequently working in large codebases, something like fzf-lua might work better for you. It's generally more performant than telescope due to various trade offs.

[painhardcore]

@jamestrew thank you.