Tons of duplicate results that don't show up in `rg`

[mcmillion]

Not sure if this is a ripgrep 13.0 thing, but I've noticed I have a ton of duplicate results.

I can confirm running the same search in cli with rg --vimgrep --no-heading --no-config --max-columns 4096 returns a total of 15 results, while quickfix has over 70

[wincent]

Weird. Any clues for how this might be reproduced? (I'm also using v13.0.0). What's your search query?

[mcmillion]

😅

So the culprit is me using globs, which is probably just my misunderstanding:

:Ack thing path/to/thing works
:Ack thing path/to/thing/** shows duplicates

Both are the same output in rg though, so I assumed it should be working.

[wincent]

Yeah, I think this is just a confusing artifact of the way globs work in Vim, as documented in :h expand() (specifically, see the stuff on ** in the last paragraph):

expand({string} [, {nosuf} [, {list}]])				*expand()*
		Expand wildcards and the following special keywords in 
		{string}.  'wildignorecase' applies.

		If {list} is given and it is |TRUE|, a List will be returned.
		Otherwise the result is a String and when there are several
		matches, they are separated by <NL> characters.

		If the expansion fails, the result is an empty string.  A name
		for a non-existing file is not included, unless {string} does
		not start with '%', '#' or '<', see below.

		When {string} starts with '%', '#' or '<', the expansion is
		done like for the |cmdline-special| variables with their
		associated modifiers.  Here is a short overview:

			%		current file name
			#		alternate file name
			#n		alternate file name n
			<cfile>		file name under the cursor
			<afile>		autocmd file name
			<abuf>		autocmd buffer number (as a String!)
			<amatch>	autocmd matched name
			<sfile>		sourced script file or function name
			<slnum>		sourced script line number or function
					line number
			<sflnum>	script file line number, also when in
					a function
			<SID>		"<SNR>123_"  where "123" is the
					current script ID  |<SID>|
			<cword>		word under the cursor
			<cWORD>		WORD under the cursor
			<client>	the {clientid} of the last received
					message |server2client()|
		Modifiers:
			:p		expand to full path
			:h		head (last path component removed)
			:t		tail (last path component only)
			:r		root (one extension removed)
			:e		extension only

		Example: >
			:let &tags = expand("%:p:h") .. "/tags"
<		Note that when expanding a string that starts with '%', '#' or
		'<', any following text is ignored.  This does NOT work: >
			:let doesntwork = expand("%:h.bak")
<		Use this: >
			:let doeswork = expand("%:h") .. ".bak"
<		Also note that expanding "<cfile>" and others only returns the
		referenced file name without further expansion.  If "<cfile>"
		is "~/.cshrc", you need to do another expand() to have the
		"~/" expanded into the path of the home directory: >
			:echo expand(expand("<cfile>"))
<
		There cannot be white space between the variables and the
		following modifier.  The |fnamemodify()| function can be used
		to modify normal file names.

		When using '%' or '#', and the current or alternate file name
		is not defined, an empty string is used.  Using "%:p" in a
		buffer with no name, results in the current directory, with a
		'/' added.

		When {string} does not start with '%', '#' or '<', it is
		expanded like a file name is expanded on the command line.
		'suffixes' and 'wildignore' are used, unless the optional
		{nosuf} argument is given and it is |TRUE|.
		Names for non-existing files are included.  The "**" item can
		be used to search in a directory tree.  For example, to find
		all "README" files in the current directory and below: >
			:echo expand("**/README")
<

So, taking Ferret's own repo as an example, note that :echo expand('autoload/**') prints:

autoload/ferret
autoload/ferret/private
autoload/ferret/private/async.vim
autoload/ferret/private/nvim.vim
autoload/ferret/private/shared.vim
autoload/ferret/private/vanilla.vim
autoload/ferret/private.vim
autoload/ferret.vim

So, when you search for :Ack thing autoload/**, it's as though you were running this command:

rg --vimgrep --no-heading --no-config --max-columns 4096 thing autoload/ferret autoload/ferret/private autoload/ferret/private/async.vim autoload/ferret/private/nvim.vim autoload/ferret/private/shared.vim autoload/ferret/private/vanilla.vim autoload/ferret/private.vim autoload/ferret.vim

Which will yield these results, with duplicates:

autoload/ferret/private/async.vim:116:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private/nvim.vim:167:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private.vim:81:7:  " Nothing special about 1829; it's just the version I am testing with as I
autoload/ferret/private.vim:369:20:" (Note: there's nothing specific to Ack in this function; it's just named this
autoload/ferret/private.vim:964:27:      " (Not valid, but nothing we can do about it here).
autoload/ferret/private.vim:968:33:  " Didn't get to a filename; nothing to complete.
autoload/ferret/private/async.vim:116:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private/nvim.vim:167:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private/async.vim:116:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private/nvim.vim:167:56:  " Job may have been canceled with cancel_async. Do nothing in that case.
autoload/ferret/private.vim:81:7:  " Nothing special about 1829; it's just the version I am testing with as I
autoload/ferret/private.vim:369:20:" (Note: there's nothing specific to Ack in this function; it's just named this
autoload/ferret/private.vim:964:27:      " (Not valid, but nothing we can do about it here).
autoload/ferret/private.vim:968:33:  " Didn't get to a filename; nothing to complete.

I don't know exactly how we're going to fix this because Ferret itself is not calling expand() anywhere; something in Vim is causing the glob expansion to occur magically, so we'll have to figure out what that is and try to avoid it, I think... The "right" way to use globs with rg anyway is with the -g flag, so stopping Vim from doing it's own confusing layer of globbing on top of that is probably the right thing to do (although at the same time I can see how people might be relying on it, so maybe it's one of those things that needs to be gated behind an option).

[wincent]

something in Vim is causing the glob expansion to occur magically

Minor correction to that... we are calling glob() here — need to think a bit about what is the right behavior... 🤔

[wincent]

Did some digging to try to figure out why I added that glob() call. It's been in Ferret since the initial import (b270d44). Looking in the prior history in my dotfiles, I can see that I added it in wincent/wincent@acf2cb3 with the following commit message:

vim: make `:Ack something ~/foo` work

Pass path arguments through `glob()` to allow expansion to occur.

[mcmillion]

Totally makes sense, thanks for the explanation!