Error 24: Too many open files
ArthurDents opened this issue · 6 comments
Hi
I have cross compiled the uftpd for use on my embedded arm platform, and I have started doing some stress tests. The test script repeatedly:
- logs in to the ftp server
- runs “get testfile.txt”
- close the connection.
When the script reaches approximately round 1015 the uftpd server generates an error message:
Failed opening data server socket. Error 24: Too many open files
and eventually hangs with a never ending:
Failed accepting FTP client connection. Error 24: Too many open files
My system has a limit on 1024 open files.
I have not been able to figure out why this happens, but I did add some extra debug output around fopen and fclose:
fp = fopen(path, "rb");
INFO("fopen fp no:%d.", fileno(fp));
INFO("Closing fp no:%d.", fileno(ctrl->fp));
fclose(ctrl->fp);
The fopen and fclose seems to work, but the fileno keeps increasing util it reaches the 1k limit.
uftpd version 2.12. Options used:
/sbin/uftpd_terminal_compile_org/uftpd -n -l debug -o ftp=9013,tftp=0 /mnt/ramdisk
Any ideas on how to fix this?
BR
AD
From the debug output (original code) when it starts to fail:
12604> Created new client session as PID 13623
13623> Client connection from 192.168.1.213
13623> Sent: 220 uftpd (2.12) ready.
13623> Recv: SYST
13623> Sent: 215 UNIX Type: L8
13623> Recv: USER Testuser
13623> Sent: 331 Login OK, please enter password.
13623> Recv: PASS Testuser
13623> User Testuser login from 192.168.1.213
13623> Sent: 230 Guest login OK, access restrictions apply.
13623> Recv: TYPE I
13623> Sent: 200 Type set to I.
13623> Recv: PASV
13623> Failed opening data server socket. Error 24: Too many open files
13623> Sent: 426 Internal server error.
13623> Recv: QUIT
13623> Sent: 221 Goodbye.
13623> Client exiting, bye
13623> FTP Client session ended.
12604> Created new client session as PID 13624
13624> Client connection from 192.168.1.213
13624> Sent: 220 uftpd (2.12) ready.
13624> Recv: SYST
13624> Sent: 215 UNIX Type: L8
13624> Recv: USER Testuser
13624> Sent: 331 Login OK, please enter password.
13624> Recv: PASS Testuser
13624> User Testuser login from 192.168.1.213
13624> Sent: 230 Guest login OK, access restrictions apply.
13624> Recv: TYPE I
13624> Sent: 200 Type set to I.
13624> Recv: PASV
13624> Failed opening data server socket. Error 24: Too many open files
13624> Sent: 426 Internal server error.
13624> Recv: QUIT
13624> Sent: 221 Goodbye.
13624> Client exiting, bye
13624> FTP Client session ended.
12604> Created new client session as PID 13625
13625> Client connection from 192.168.1.213
13625> Sent: 220 uftpd (2.12) ready.
13625> Recv: SYST
13625> Sent: 215 UNIX Type: L8
13625> Recv: USER Testuser
13625> Sent: 331 Login OK, please enter password.
13625> Recv: PASS Testuser
13625> User Testuser login from 192.168.1.213
13625> Sent: 230 Guest login OK, access restrictions apply.
13625> Recv: TYPE I
13625> Sent: 200 Type set to I.
13625> Recv: PASV
13625> Failed opening data server socket. Error 24: Too many open files
13625> Sent: 426 Internal server error.
13625> Recv: QUIT
13625> Sent: 221 Goodbye.
13625> Client exiting, bye
13625> FTP Client session ended.
12604> Created new client session as PID 13626
13626> Client connection from 192.168.1.213
13626> Sent: 220 uftpd (2.12) ready.
13626> Client exiting, bye
13626> FTP Client session ended.
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
12604> Failed accepting FTP client connection. Error 24: Too many open files
…
Interesting, never tested this before! I'll give you a bit of background and some facts, maybe we can track this one down together:
- For every new FTP connection the server accept()s it on a new client socket
- Sockets are also files in UNIX (same shared file descriptor space)
- For each new client connection a new child is forked (new PID)
- Each client has a linger feature, if a client never disconnects we kill the connection/child after inactivity timeout
My guess is that in your case you also have a lot of uftpd child processes. Possibly the children are not "reaped", or their client socket is not closed properly. It could of course be something else, but let's start there.
Thanks for your fast reply.
I have run:
pgrep -P <pid of uftpd>
which only shows one or zero child processes.
I have added my test scripts:
- ftp_get_testfile.sh is the loop
- ftp_get_testfile includes the ftp commands
ftp_get_testfile.sh:
#!/bin/bash
x=1
while [ 1 ]
do
echo "Get testfile.txt round: $x"
./ftp_get_testfile
x=$(( $x + 1 ))
#sleep 1
echo "-----------------------------"
done
ftp_get_testfile:
#!/bin/bash
ftp -n 192.168.1.131 9013 <<END
verbose on
user Testuser Testuser
bin
get testfile.txt
bye
END
Test script output (when ok):
-----------------------------
Get testfile.txt round: 43
Verbose mode on.
331 Login OK, please enter password.
230 Guest login OK, access restrictions apply.
local: testfile.txt remote: testfile.txt
227 Entering Passive Mode (192,168,1,131,232,77)
125 Data connection already open; transfer starting.
226 Transfer complete.
5 bytes received in 0.0019 secs (2.64 Kbytes/sec)
221 Goodbye.
-----------------------------
Get testfile.txt round: 44
BR
AD
Thanks for the scripts! I've reproduced the problem now. Looking into it.
There, fixed! Thank you for the report and the really great scripts to help reproduce this! :)
I'll see about getting a v2.13 out soonish
v2.13 released now
Thank you for fixing this so quickly. It’s greatly appreciated!
BR
AD