"426 Failure reading network stream" responses when using FTPS to some backend servers
jackmoot opened this issue · 7 comments
We compiled the latest mod_proxy version as shared module using debian proftpd package 1.3.6-4+deb10u5 (buster), but also on bullseye with proftpd 1.3.7a-1, and we noticed that after every file upload there is this error on the client side:
426 Failure reading network stream.
This is from the simple ftp-ssl client, but it happens with all, even in filezilla there's a 426 code shown.
Lftp says more, it gives a message "Delaying before reconnect", so it seems it can't resume the session
Note that file uploads are transferred successfully. Also note there's no error when downloading files, it's only at the end of each upload.
We use mod_proxy in forward mode with passive transfers (ProxyDataTransferPolicy passive). We have tried different FTP servers for the server behind (pureftpd, vsftpd on different linux versions).
We thought it could be TLS 1.3 related, so we tried enforcing the TLS protocol to the other server, by using:
ProxyTLSProtocol TLSv1.2
But we noticed it is still logging this:
[tls.info] connecting: TLSv1.3 read encrypted extensions
[tls.msg] received (null) 'Finished' Handshake message (52 bytes)
[tls.info] connecting: SSLv3/TLS read finished
[tls.msg] sent protocol record message (content_type = ApplicationData, len = 69)
[tls.msg] sent (null) 'Finished' Handshake message (52 bytes)
[tls.info] connecting: SSLv3/TLS write finished
[tls.info] ok: SSL negotiation finished successfully
[tls.info] connecting: SSL negotiation finished successfully
Server: CN = XXXXXX
TLSv1.3 connection created, using cipher TLS_AES_256_GCM_SHA384 (256 bits)
Is it normal that it seems it can't resume the session after an upload? (maybe work in progress? as we saw a lot of tls 1.3 related commits)
Also is it normal that it is logging that it connects with TLS 1.3 to the other server when we try to enforce 1.1 or 1.2 ?
Could you provide the output from proftpd -V, and the mod_proxy configuration you're using, so that I can attempt to reproduce this behavior locally?
I did a fresh install with the proftpd + proftpd-mod-proxy packages in debian-testing:
Compile-time Settings:
Version: 1.3.7a (maint)
Platform: LINUX [Linux 4.15.XX x86_64]
Built: Thu Jul 23 2020 05:53:17 UTC
Built With:
configure '--build=x86_64-linux-gnu' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--disable-dependency-tracking' '--prefix=/usr' '--with-includes=/usr/include/postgresql:/usr/include/mariadb:/usr/include/mariadb/mysql' '--mandir=/usr/share/man' '--sysconfdir=/etc/proftpd' '--localstatedir=/run' '--libexecdir=/usr/lib/proftpd' '--enable-sendfile' '--enable-facl' '--enable-dso' '--enable-autoshadow' '--enable-ctrls' '--enable-ipv6' '--enable-nls' '--enable-memcache' '--with-lastlog=/var/log/lastlog' '--enable-pcre' '--disable-strip' '--enable-redis' '--build' 'x86_64-linux-gnu' '--with-shared=mod_unique_id:mod_site_misc:mod_load:mod_ban:mod_quotatab:mod_sql:mod_sql_mysql:mod_sql_postgres:mod_sql_sqlite:mod_sql_odbc:mod_dynmasq:mod_quotatab_sql:mod_ldap:mod_quotatab_ldap:mod_ratio:mod_tls:mod_rewrite:mod_radius:mod_wrap:mod_wrap2:mod_wrap2_file:mod_wrap2_sql:mod_quotatab_file:mod_quotatab_radius:mod_facl:mod_ctrls_admin:mod_copy:mod_deflate:mod_ifversion:mod_geoip:mod_exec:mod_sftp:mod_sftp_pam:mod_sftp_sql:mod_shaper:mod_sql_passwd:mod_ifsession:mod_auth_otp:mod_tls_redis:mod_wrap2_redis:mod_redis:mod_memcache:mod_tls_memcache:mod_readme:mod_snmp:mod_digest:mod_ident:mod_log_forensic:mod_qos:mod_statcache:mod_tls_fscache:mod_tls_shmcache' 'build_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -fdebug-prefix-map=/build/proftpd-dfsg-ynkZ3Q/proftpd-dfsg-1.3.7a=. -fstack-protector-strong -Wformat -Werror=format-security' 'LDFLAGS=-Wl,-z,relro' 'CPPFLAGS=-Wdate-time -D_FORTIFY_SOURCE=2' 'CXXFLAGS=-g -O2 -fdebug-prefix-map=/build/proftpd-dfsg-ynkZ3Q/proftpd-dfsg-1.3.7a=. -fstack-protector-strong -Wformat -Werror=format-security'
CFLAGS: -g2 -g -O2 -fdebug-prefix-map=/build/proftpd-dfsg-ynkZ3Q/proftpd-dfsg-1.3.7a=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -fno-omit-frame-pointer -Werror=implicit-function-declaration
LDFLAGS: -L$(top_srcdir)/lib -L$(top_builddir)/lib -Wl,-z,relro -rdynamic -L/usr/lib/x86_64-linux-gnu/ -L/usr/lib/x86_64-linux-gnu
LIBS: -lacl -lpcreposix -lpcre -lssl -lcrypto -lsodium -lcap -lpam -lsupp -lattr -lnsl -lresolv -lresolv -lcrypt -ldl -lhiredis -lmemcachedutil -lmemcached -pthread
Files:
Configuration File:
/etc/proftpd/proftpd.conf
Pid File:
/run/proftpd.pid
Scoreboard File:
/run/proftpd.scoreboard
Header Directory:
/usr/include/proftpd
Shared Module Directory:
/usr/lib/proftpd
Info:
+ Max supported UID: 4294967295
+ Max supported GID: 4294967295
Features:
+ Autoshadow support
+ Controls support
+ curses support
- Developer support
+ DSO support
+ IPv6 support
+ Largefile support
+ Lastlog support
+ Memcache support
+ ncursesw support
+ NLS support
+ OpenSSL support (OpenSSL 1.1.1g 21 Apr 2020)
+ PCRE support
+ POSIX ACL support
+ Redis support
+ Sendfile support
+ Shadow file support
+ Sodium support
+ Trace support
+ xattr support
Tunable Options:
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_ENV_MAX = 2048
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_LOGIN_MAX = 256
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_PATH_MAX = 4096
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 10
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10
Here's the config i added:
<IfModule mod_proxy.c>
ProxyEngine on
ProxyLog /var/log/proftpd/proxy.log
ProxyTables /var/log/proftpd
ProxyTLSEngine on
ProxyTLSProtocol TLSv1.2
ProxyTLSOptions EnableDiags
ProxyRole forward
ProxyForwardMethod user@host
ProxyForwardTo servers [NC]
ProxyDataTransferPolicy passive
</IfModule>
<Class forward-proxy>
From 0.0.0.0/0
ProxyForwardEnabled on
</Class>
In the tls config i also have:
TLSEngine on
TLSLog /var/log/proftpd/tls.log
TLSProtocol TLSv1 TLSv1.1 TLSv1.2
TLSOptions NoSessionReuseRequired AllowClientRenegotiations EnableDiags
TLSRequired on
The rest should be pretty much default
And i have those results for forwarded servers:
proftpd on stretch -> OK
vsftpd on stretch -> 426 Failure reading network stream
pureftpd on stretch -> OK
pureftpd on buster -> File transfer failed
(those errors appear on filezilla and some other cli)
(files transfer fine)
Forwarded servers with default config and debian packages + TLS enforced
Thanks for that info, @jackmoot ! I'm now able to reproduce the "426 Failure reading network stream" scenario with vsftpd.
In my local setup, I see the following in the /var/log/vsftpd.log file:
Sun Oct 25 00:26:23 2020 [pid 546] [docker] DEBUG: Client "::ffff:127.0.0.1", "DATA connection terminated without SSL shutdown. Buggy client! Integrity of upload cannot be asserted."
Sun Oct 25 00:26:23 2020 [pid 547] [docker] FAIL UPLOAD: Client "::ffff:127.0.0.1", "/home/docker/test.dat", 13 bytes, 2.54Kbyte/sec
This suggests that mod_proxy is not performing proper TLS shutdown on the data connection, for uploads.
I've confirmed that this same behavior, using the exact same mod_proxy and vsftpd configurations, does not happen for downloads/directory listings, only uploads. (I wonder if it's the same for the pureftpd case you mentioned as well? We'll see; I'll work through the vsftpd scenario first.)
@jackmoot would you mind filing a separate ticket for the TLSv1.2/TLSv1.3 issue you noticed? I think it's a valid concern, but I'd like to use this ticket to focus on those upload errors. Thanks!
I just installed pure-ftpd, and tested uploads over FTPS to it via mod_proxy; they succeed as well. (I'm using an Ubuntu 20.04 Docker image for testing this, if it helps.)
@jackmoot actually, I just found the TLSv1.3 issue; it's a small change, so I've added it to the existing PR.
This should now be fixed in master (the 426 upload error, and the unexpected use of TLSv1.3 when TLSv1.2 is configured). Thanks!