eradman/entr

'entr: setrlimit cannot set rlim_cur to 61440: Invalid argument' on macOS

marktsuchida opened this issue · 23 comments

marktsuchida commented

I get this error on macOS Sonoma 14.2.1 (23C71) and entr 5.5 (Homebrew).
entr 5.4 (built from source) works without error.

With current master:

$ cc --version
Apple clang version 15.0.0 (clang-1500.1.0.2.5)
Target: arm64-apple-darwin23.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
$ ./configure
+ cp Makefile.macos Makefile
$ make test
cat /dev/null  > compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c entr.c
cc  -D_MACOS_PORT -o entr compat.o entr.o
ls entr.1 | ./entr -zns 'wc $0'
entr: setrlimit cannot set rlim_cur to 61440: Invalid argument
make: *** [test] Error 1
eradman commented

Thanks for reporting. It seems as if every upgrade to MacOS breaks something around file limits!
This error does not occur with Ventura 13.6.3. I'll try upgrading

eradman commented

I updated to Mac OS 14.3, but could not replicate this problem

$ git clean -xdf
$ cc --version
Apple clang version 12.0.5 (clang-1205.0.22.9)
Target: arm64-apple-darwin23.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
eradman@macmini-m1 entr % ./configure
+ cp Makefile.macos Makefile
$ make test
cat /dev/null  > compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c entr.c
cc  -D_MACOS_PORT -o entr compat.o entr.o
ls entr.1 | ./entr -zns 'wc $0'
     184     928    5130 /Users/eradman/git/entr/entr.1
zsh returned exit code 0

Are you using clang from homebrew? I see my compiler is older

marktsuchida commented

Thanks for the quick response!

My compiler is from the latest CommandLineTools. I get the same Apple clang version 15.0.0 (clang-1500.1.0.2.5) if I point xcode-select to Xcode 15.1 (and make test gives the same error).

I could try upgrading to macOS 14.3 and see if it makes a difference, unless you have ideas for other things to test while I'm still on 14.2.1.

eradman commented

Upgraded developer's tools, and still works okay

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install

I don't think this relates to the version of MacOS. What values do you have for maxfiles?

$ sysctl -a kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 30720
kern.maxfilesperproc: 10240
marktsuchida commented 
$ sysctl -a kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 122880
kern.maxfilesperproc: 61440
eradman commented

Try compiling and running the following program

#include <err.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/resource.h>

/* sysctlbyname */
#include <sys/types.h>
#include <sys/sysctl.h>

int main() {
    struct rlimit rl;
    size_t len;
    unsigned open_cur;
    unsigned open_max;

    if (getrlimit(RLIMIT_NOFILE, &rl) == -1)
        err(1, "getrlimit");
    open_cur = rl.rlim_cur;

    len = sizeof(open_max);
    sysctlbyname("kern.maxfilesperproc", &open_max, &len, NULL, 0);

    rl.rlim_cur = (rlim_t)open_max;
    if (setrlimit(RLIMIT_NOFILE, &rl) != 0)
        err(1, "setrlimit cannot set rlim_cur to %u", open_max);

    printf("setrlimit: %d -> %d\n", open_cur, open_max);
}

Output should look like this

$ cc setlimit.c
$ ./a.out
setrlimit: 256 -> 10240
eradman commented

Looking at the man page for sysctlbyname(3), I see that the value it modifies is signed

           kern.maxproc                 int32_t                 yes
           kern.maxprocperuid           int32_t                 yes

This may not be the root cause, but I adjusted the type to match in commit 67e1c01

marktsuchida commented

Here you go:

Try compiling and running the following program

$ pbpaste >setlimit.c
$ cc setlimit.c 
$ ./a.out
a.out: setrlimit cannot set rlim_cur to 61440: Invalid argument

I adjusted the type to match in commit 67e1c01

$ git checkout 67e1c01
[...snip...]
HEAD is now at 67e1c01 Use int32_t for max_open and set EV_TRACE for make test
$ git clean -dxf
$ ./configure
+ cp Makefile.macos Makefile
$ make test
cat /dev/null  > compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c entr.c
cc  -D_MACOS_PORT -o entr compat.o entr.o
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
entr: setrlimit cannot set rlim_cur to 61440: Invalid argument
make: *** [test] Error 1
eradman commented

Goodness. Perhaps there is a problem setting setrlimit() to a large number. Try setting the system maxfilesperproc limit to a smaller value, then bump it up to see where the error occurs

sudo sysctl -w kern.maxfilesperproc=5000
make test
sudo sysctl -w kern.maxfilesperproc=10000
make test
# ...
marktsuchida commented

Aha!

$ sudo sysctl -w kern.maxfilesperproc=1024
kern.maxfilesperproc: 1025 -> 1024
$ make test
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
open_max: 1024
     184 entr.1
$ sudo sysctl -w kern.maxfilesperproc=1025
kern.maxfilesperproc: 1024 -> 1025
$ make test
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
entr: setrlimit cannot set rlim_cur to 1025: Invalid argument
make: *** [test] Error 1

(This was with 67e1c01.)

eradman commented

1024 is a very small hard limit!

As far as I know setting these limits is no longer an option at boot time, so they will always reset to Apple defaults.

Here's another program to try:

#include <stdio.h>
#include <unistd.h>
#include <limits.h>
#include <sys/resource.h>

int main() {
        struct rlimit rl;
        getrlimit(RLIMIT_NOFILE, &rl);

        printf("rlim_cur: %ld\n", (long)rl.rlim_cur);
        printf("rlim_max: %ld\n", (long)rl.rlim_max);
        printf("_SC_OPEN_MAX: %ld\n", (long)sysconf(_SC_OPEN_MAX));
        printf("OPEN_MAX: %d\n", OPEN_MAX);
}

The output on my mac:

rlim_cur: 256
rlim_max: 9223372036854775807
_SC_OPEN_MAX: 256
OPEN_MAX: 10240
eradman commented

Interesting section in the man page for setrlimit(2)

COMPATIBILITY
     setrlimit() now returns with errno set to EINVAL in places that historically
     succeeded.  It no longer accepts "rlim_cur = RLIM_INFINITY" for RLIM_NOFILE.
     Use "rlim_cur = min(OPEN_MAX, rlim_max)".

Perhaps this formula should be used instead of querying kern.maxfilesperproc

marktsuchida commented

Here's what I get.

rlim_cur: 1024
rlim_max: 1024
_SC_OPEN_MAX: 1024
OPEN_MAX: 18446744073709551615

(This is under kern.maxfilesperproc: 61440.)

eradman commented

I have no idea why, but the hard limit really is 1024 on your system.
Looks like we just need to follow the man page for setrlimit() to solve this problem

Then how to properly raise it on Mac OS is a separate issue...

marktsuchida commented

I would be totally happy if you just said that macOS 14.2 is bad and people should upgrade to 14.3 (perhaps after confirming that my laptop fixes itself after upgrading). On the other hand, if you do want to add a workaround for this, I'm happy to keep testing things before upgrading.

marktsuchida commented

BTW note again that entr 5.4 does not encounter any errors. Not sure if that is because it is successfully raising the limit or giving up silently.

eradman commented

5.4 would fail to raise the limit to a high number of available, so that's not surprising.

eradman commented

Pushed what I think is the fix in commit 6fa963e

$ make test
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
open_max: 10240
     184 entr.1
marktsuchida commented

6fa963e works for me, too (well, as long as I don't have too many files, I guess, but that is looking like a macOS 14.2 issue):

$ make test
cat /dev/null  > compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c entr.c
cc  -D_MACOS_PORT -o entr compat.o entr.o
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
open_max: 1024
     184 entr.1

Thanks!

eradman commented

Excellent! I'd be curious to know what happens what you upgrade, but this appears to be a solution that should work reliably.

marktsuchida commented

Somewhat surprisingly, upgrading to macOS 14.3 (23D56) did not change much:

$ make test
cat /dev/null  > compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c compat.c
cc  -D_MACOS_PORT -DRELEASE=\"5.5\" -c entr.c
cc  -D_MACOS_PORT -o entr compat.o entr.o
ls entr.1 | EV_TRACE=1 ./entr -zn wc -l entr.1
open_max: 1024
     184 entr.1

FWIW:

$ sysctl -a kern.maxfiles kern.maxfilesperproc
kern.maxfiles: 122880
kern.maxfilesperproc: 61440

However, the OPEN_MAX from your test program above did change (previously it was UINT64_MAX):

rlim_cur: 1024
rlim_max: 1024
_SC_OPEN_MAX: 1024
OPEN_MAX: 10240

Then I searched around a little more and found this comment. And sure enough, I had the line ulimit -u 512 -n 1024 in my bash startup. (Apologies for not realizing this earlier!)

Interestingly, I get this:

$ ulimit -n 1025
-bash: ulimit: open files: cannot modify limit: Operation not permitted

It looks like ulimit can decrease, but not increase, the limit. Not sure if it was always this way, but it could have been. I know I had the ulimit in my .bashrc to increase the limit, but it's probably been there for well over a decade....

If I start without the ulimit I get results matching yours and can see the effect of ulimit:

# No previous ulimit
$ ./a.out
rlim_cur: 256
rlim_max: 9223372036854775807
_SC_OPEN_MAX: 256
OPEN_MAX: 10240
$ ulimit -n 1234
$ ./a.out
rlim_cur: 1234
rlim_max: 1234
_SC_OPEN_MAX: 1234
OPEN_MAX: 10240

Of course your fixes did in fact fix entr for people who have a small-ish rlim_max, whether via ulimit or otherwise. Appreciate all your help!

marktsuchida commented

Oh, and a further twist is that I normally run bash from Homebrew (including above).

macOS /bin/zsh gives the following:

% ./a.out
rlim_cur: 256
rlim_max: 9223372036854775807
_SC_OPEN_MAX: 256
OPEN_MAX: 10240
% ulimit -n 1234
% ./a.out       
rlim_cur: 1234
rlim_max: 9223372036854775807
_SC_OPEN_MAX: 1234
OPEN_MAX: 10240

(But macOS /bin/bash behaves identically to Homebrew bash.)

eradman commented

Thanks for the update! Your .bashrc setting did uncover a real bug. Interested that you can lower, but not raise the soft limit.