Low performance on very large files like `all-packages.nix` on nvim/nvim-lspconfig/nvim-cmp setup

[doronbehar]

Hello,

Thanks for creating this program, it has more features in comparison to rnix-lsp and the diagnostic messages are better as well IMO. However, I have to reside to using rnix-lsp never the less, due to performance issues when I edit all-packages.nix, in Neovim, using nvim-lspconfig. If I turn on completion using nvim-cmp, it becomes so slow it's impossible to edit a single line.

Due to my suspicion that this is an issue with one of these neovim plugins, I created a repository with minimal configuration that reproduces the experience:

https://github.com/doronbehar/nvim-issues-reproduce

What's frustrating, is that I can't seem to turn nil off when I edit large files such as all-packages.nix - this is more of a nvim-lspconfig issue I guess, which I have asked about here. In anycase, with rnix-lsp I don't experience this issue so that's what I am doing now.

[oxalica]

Could you confirm in top or htop that which process is consuming lots of CPU and/or memory? Is it nil or nvim? I suspect it's neovim native lsp and/or nvim-cmp having troubles handling large responses from nil.

I'm daily using coc-nvim on neovim and editing all-packages.nix is not a problem for me.

[doronbehar]

Oh you are right, it is nvim consuming all the CPU.

[doronbehar]

Is it possible to make nil return smaller responses?

[oxalica]

Turns out it's mostly caused by neovim requesting semantic tokens for the whole document (textDocument/semanticTokens/full) after every update, that is, every keystroke in insert mode. We support range requests that only renders visible lines, but neovim doesn't support it yet. You can follow neovim/neovim#23026.

You can turn off semantic tokens in neovim by following this reddit link, adding these into your neovim config:

vim.api.nvim_create_autocmd("LspAttach", {
  callback = function(args)
    local client = vim.lsp.get_client_by_id(args.data.client_id)
    client.server_capabilities.semanticTokensProvider = nil
  end,
});

It is incorrect to truncate responses for a full-requests, because they are supposed to be cached, maintained and reused by the client. I don't think there are anything we can do in nil side.

[doronbehar]

You can follow neovim/neovim#23026.

Thanks a lot for the links and the investigations! That's much appreciated.

You can turn off semantic tokens in neovim by following this reddit link, adding these into your neovim config:

This is not bad, though I've found an even better solution:

v = "nil"
lsp[v].setup{
	on_attach = function(client, bufnr)
		if bufIsBig(bufnr) then
			client.server_capabilities.semanticTokensProvider = nil
		end
		-- ...
	end,
}

I needed the buffer number that is available in on_attach function's signature, in order to check whether the file is too big.

It is incorrect to truncate responses for a full-requests, because they are supposed to be cached, maintained and reused by the client. I don't think there are anything we can do in nil side.

I agree. However, I would have never reached this resolution without you, so that's a lot I got from nil's side! Thanks again 🙏.

[jdrouhard]

Turns out it's mostly caused by neovim requesting semantic tokens for the whole document (textDocument/semanticTokens/full) after every update, that is, every keystroke in insert mode. We support range requests that only renders visible lines, but neovim doesn't support it yet. You can follow neovim/neovim#23026.

I posted a comment on that linked neovim issue but thought I'd pop in here too--the lagginess caused by the large full semantic token responses should be resolved in neovim HEAD and the 0.9.1 release. Shouldn't need to disable semantic tokens for large files or to try to get better input latency.

Range still doesn't work but at least the editor shouldn't lag when using full requests. It might just take a bit longer for the colors to pop in.