Open files limit is restricted by supervisord

[yahor-filipchyk]

Nomad version: Nomad v0.9.6 (1f8eddf2211d064b150f141c86e30d9fceabec89)
AWS nomad module version: v0.5.0
OS: 16.04.6 LTS (Xenial Xerus) 

We started running into the "Too many files open" issue on our client nodes due to some increased IO usage by some of our services. The erroring process (JVM) has been holding a little over 4000 file descriptors. Given the default soft limit of 1024 we've tried increasing it to 10000 by updating /etc/security/limits.conf. Even though the new config seemed to be picked up by the system we continued to experience the issue.

Then we saw that cat /proc/<pid_id>/limits was showing

Limit                     Soft Limit           Hard Limit           Units     
Max open files            4096                 4096                 files

which made us learn about the whole chain of starting an allocation by nomad, realizing that the process is inheriting the limits from the parent process. And given that the allocation is started by nomad and nomad is started by supervisord we found that the actual limit is set by supervisord.

The solution for us was to add another config file in /etc/supervisor/conf.d that included

[supervisord]
minfds=8192

See the docs for minfds here http://supervisord.org/configuration.html#supervisord-section-settings

To prevent surprises for other users, I'd suggest that either the default supervisord config should be updated in this module (perhaps with an optional argument passed to the install script) or at least this fact should be mentioned in documentation so the users know how to act.

[brikis98]

To prevent surprises for other users, I'd suggest that either the default supervisord config should be updated in this module (perhaps with an optional argument passed to the install script)

I'd vote to keep the default as-is, but to expose a parameter to make it easy to tweak this setting, and add docs explaining when you might want to do so. Would you be up for a PR to add that?