universal-ctags/citre

citre cause error "Creating pipe: Too many open files" with args (prefix)

haoisli9 opened this issue · 18 comments

haoisli9 commented

win10, emacs 29.0.50
when using company to auto-completion, it always cause error "Creating pipe: Too many open files" with args (prefix); when I exit emacs, I can see readtags processing in background, like:
readtags stderr -- open readtags-stderr -- Main (serial port ?)
readtags stderr<1> -- open readtags-stderr-254200 -- Main (serial port ?)
readtags stderr<2> -- open readtags-stderr-376378 -- Main (serial port ?)

This problem happens recently after last citre update. Old citre version does not have this problem.

Can you check it? Thanks.

AmaiKinono commented

How did you get Emacs 29.0.50 on Windows? If you downloaded it somewhere, please offer me the URL; If you build it from source, please give instructions.

haoisli9 commented
haoisli9 commented

hi, how about this issue? need more information?

AmaiKinono commented

I haven't got time to work on it.

AmaiKinono commented

Sadly I can't reproduce this.

I'm on

  • Windows 10.

  • Emacs 29.0.50 (downloaded from https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-29/, emacs-29.0.50-snapshot-2022-04-02.zip).

  • Citre is on the latest version on master.

  • Company configuration are just 2 lines:

    (setq company-minimal-prefix-length 1)
(setq company-backends '(company-capf))

Tested in the Linux kernel. Typing is smooth, the error you mentioned didn't happen, and I didn't see any Citre related processes by list-processes or quitting Emacs.

I could suggest a patch that you can try. In citre-common.el, there's this function:

(defun citre-kill-process-buffer (buffer)
  "Delete the process in BUFFER and kill BUFFER.
This doesn't run `kill-buffer-hook' and
`kill-buffer-query-functions' so it should be faster."
  (let ((kill-buffer-hook nil)
        (kill-buffer-query-functions nil))
    (when-let ((proc (get-buffer-process buffer)))
      (delete-process proc))
    (kill-buffer buffer)))

Try change it to:

(defun citre-kill-process-buffer (buffer)
  "Delete the process in BUFFER and kill BUFFER.
This doesn't run `kill-buffer-hook' and
`kill-buffer-query-functions' so it should be faster."
  (let ((kill-buffer-hook nil)
        (kill-buffer-query-functions nil))
    (when-let ((proc (get-buffer-process buffer)))
      (signal-process proc 'sighup)
      (delete-process proc))
    (kill-buffer buffer)))

maybe we can discuss with "**话"?

Yes, but for other maintainers' sake, I have to attach an English version of everything we are saying. I'm OK with that but using English saves developer's time.

masatake commented

How does M-x list-processes say when getting the error?

haoisli9 commented
haoisli9 commented
masatake commented

readtags doesn't open much files.
So I guessed too many readtags processes run at the same time. However, list-processes printed only three readtags processes.

So I guess "Creating pipe: Too many open files" implies there are too many running applications (how do you call them on Windows?) other than readtags or there are some? applications (other than readtags) opening too many files.

@AmaiKinono

I wonder why

readtags stderr --      open     *readtags-stderr*        --           Main         (serial port ?)

is kept being running. Is it doing "large" searching?

masatake commented

I don't know and use Windows.
Is there way to list files opened now?
On GNU/Linux, lsof is the tool for the purpose.

How about windows?
What I found with web-searching:

https://www.nirsoft.net/utils/opened_files_view.html

haoisli9 commented
AmaiKinono commented

@masatake Let me explain the big picture.

@haoisli9 uses company, basically you can think of it as a minor mode that calls completion-at-point-functions after every keystroke.

The capf function offered by Citre can be expensive to run (when the tags file is large). To eliminate this problem, citre-capf--get-collection wraps some time-consuming part using while-no-input (this depends on citre-capf-optimize-for-popup), so the user could always interrupt readtags process by typing, which translates to a local quit by while-no-input.

So we must do cleanup when a local quit arrives. This is done in an unwind-protect form in citre-get-output-lines, which calls citre-destruct-process and it does all the jobs: kill the process, the stderr pipe process, and the stderr buffer. It seems part of (or the whole) cleanup mechanism doesn't work for @haoisli9.

I can't reproduce this so I can't inspect it further. I have several guesses that may work or not.

  • Some commit on Emacs master breaks / changes the behavior of unwind-protect.
  • citre-destruct-process is not called. It's easy to inspect, just insert some message in it and typing with company-mode on.
  • The stderr pipe process is not killed. Figure out how to kill a pipe process in Windows and replace the relevant part in citre-destruct-process with it. I think you can get the process object or buffer somehow from list-processes buffer.
  • Maybe the company-tabnine backend is interfering with the capf backend. Disable it and try again.
  • When nothing works, try reproduce using $ emacs -Q. Maybe the problem will disappear and then it would be easy to figure out what's wrong.
masatake commented

@AmaiKinono It looks that the failures of killing processes keep too many processes alive.

error in process sentinel: citre-kill-process-buffer: Cannot signal process readtags stderr<2>

I wonder we cannot send a signal.

This message comes from https://github.com/emacs-mirror/emacs/blob/master/src/process.c#L7091 .

      pid = XPROCESS (process)->pid;
      if (pid <= 0)
	error ("Cannot signal process %s", SDATA (XPROCESS (process)->name));

https://github.com/emacs-mirror/emacs/blob/c4e93b67c456b2a7cfc57b13c5d8070eb2b6d167/src/process.h#L122

    /* Process ID.  A positive value is a child process ID.
       Zero is for pseudo-processes such as network or serial connections,
       or for processes that have not been fully created yet.
       -1 is for a process that was not created successfully.
       -2 is for a pty with no process, e.g., for GDB.  */
    pid_t pid;
AmaiKinono commented

I wonder why we cannot send a signal.

My assumption is we cannot signal a pipe process. They should belong to "pseudo-processes such as network or serial connections".

(process-id (make-pipe-process :name "pipe"))
;; => nil (is nil defined as 0 in C?)

Citre uses delete-process to kill the pipe process. I'm suspecting it doesn't work well on Windows (though it works well on my Windows machine). SIGHUP seems to work well on Windows on readtags processes while other methods sometimes fail, so I suggested to try SIGHUP, but seems we can't signal a pipe process.

haoisli9 commented
AmaiKinono commented

Guessing doesn't help if we can't reproduce the problem.

You are on your own. Try advices in #129 (comment), or give a recipe that starts from $ emacs -Q and reproduces the problem.

haoisli9 commented
AmaiKinono commented

Close as developers can't reproduce the problem, and we haven't receive a reproduce guide from $ emacs -Q.

Feel free to reopen if you have a solution.