High memory consumption for service hitch reload
iammeken opened this issue · 3 comments
I am on a 24core server with 256G memory, using ubuntu 20.04 +Hitch 1.7.0/1.7.1+openssl 1.1.1k, ocsp stappling on.
I am using nginx + varnish +php 8.0 + mariadb behind.
Certs about 3000 at pem-dir, LETSENCRYPT.
Total memory consumption (including nginx etc.) will be 16G.
When service hitch reload, total memory goes to 110G+ at peak, then sometimes down to 16G, but sometimes maintain at 110G+, I have to run service hitch restart to keep it back to normal.
Or ocsp-verify-staple = off
Any advice? Thank you.
backend = "[127.0.0.1]:6086"
frontend = "[]:443"
pem-dir-glob = ".pem"
pem-dir = "/root/.acme.sh/hitch"
pem-file = "/root/.acme.sh/domain.com/hitch-bundle.pem"
user = "_hitch"
group = "_hitch"
ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
prefer-server-ciphers = on
session-cache=600
#sni-nomatch-abort = on
tls-protos = TLSv1.2 TLSv1.3
tcp-fastopen = off
alpn-protos = "h2,http/1.1"
write-proxy-v2 = on
ocsp-connect-tmo = 4
ocsp-resp-tmo = 4
ocsp-dir = "/var/lib/hitch/"
ocsp-verify-staple = on
workers = 48 # number of CPU cores
syslog = off
quiet = on
log-level = 1
keepalive = 30
backlog = 1024
I may know the reason:
I have thousands of lets certs, and usually renew hundreds of them in a batch. It will cause service hitch reload some seconds to scan whole directory to find and load changed one, longer than next command of service hitch reload (post renew hook).
Using Ubuntu 20.04+ hitch 1.7.1, there is no hitch.service in /usr/lib/systemd/system,
My /etc/init.d/hitch:
#!/bin/sh
BEGIN INIT INFO
Provides: hitch
Required-Start: $local_fs $network $remote_fs $syslog
Required-Stop: $local_fs $network $remote_fs $syslog
Default-Start: 2 3 4 5
Default-Stop: 0 1 6
Short-Description: scalable TLS proxy
Description: hitch is a network proxy that terminates TLS/SSL
connections and forwards the unencrypted traffic to some
backend. It's designed to handle 10s of thousands of
connections efficiently on multicore machines.
END INIT INFO
Author: Stig Sandbeck Mathisen ssm@debian.org
Do NOT "set -e"
PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="hitch"
NAME=hitch
DAEMON=/usr/local/sbin/hitch
DAEMON_ARGS="--daemon --pidfile=/run/hitch.pid --user _hitch --group _hitch --config=/etc/hitch/hitch.conf"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME
Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0
Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME
Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
Define LSB log_* functions.
Depend on lsb-base (>= 3.2-14) to ensure that this file is present
and status_of_proc is working.
. /lib/lsb/init-functions
Function that starts the daemon/service
do_start()
{
# Return
# 0 if daemon has been started
# 1 if daemon was already running
# 2 if daemon could not be started
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null
|| return 1
start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --
$DAEMON_ARGS
|| return 2
# The above code will not work for interpreted scripts, use the next
# six lines below instead (Ref: #643337, start-stop-daemon(8) )
#start-stop-daemon --start --quiet --pidfile $PIDFILE --startas $DAEMON
# --name $NAME --test > /dev/null
# || return 1
#start-stop-daemon --start --quiet --pidfile $PIDFILE --startas $DAEMON
# --name $NAME -- $DAEMON_ARGS
# || return 2
# Add code here, if necessary, that waits for the process to be ready
# to handle requests from services started subsequently which depend
# on this one. As a last resort, sleep for some time.
}
Function that stops the daemon/service
do_stop()
{
# Return
# 0 if daemon has been stopped
# 1 if daemon was already stopped
# 2 if daemon could not be stopped
# other if a failure occurred
start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME
RETVAL="$?"
[ "$RETVAL" = 2 ] && return 2
# Wait for children to finish too if this is a daemon that forks
# and if the daemon is only ever run from this initscript.
# If the above conditions are not satisfied then add some other code
# that waits for the process to drop all resources that could be
# needed by services started subsequently. A last resort is to
# sleep for some time.
start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
[ "$?" = 2 ] && return 2
# Many daemons don't delete their pidfiles when they exit.
rm -f $PIDFILE
return "$RETVAL"
}
Function that sends a SIGHUP to the daemon/service
do_reload() {
#
# If the daemon can reload its configuration without
# restarting (for example, when it is sent a SIGHUP),
# then implement that here.
#
start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
return 0
}
case "$1" in
start)
[ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
do_start
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
do_stop
case "$?" in
0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
esac
;;
status)
status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
;;
reload|force-reload)
#
# If do_reload() is not implemented then leave this commented out
# and leave 'force-reload' as an alias for 'restart'.
#
log_daemon_msg "Reloading $DESC" "$NAME"
do_reload
log_end_msg $?
;;
restart)
#
# If the "reload" option is implemented then remove the
# 'force-reload' alias
#
log_daemon_msg "Restarting $DESC" "$NAME"
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
*)
#echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
exit 3
;;
esac
:
Is there a way to avoid/fix this?
Thank you.
It seems hitch reload will only add small amount memory (300M) under Ubuntu 22.04, with 6500 ssl certs
hitch 1.7.2 + openssl 3.0.2
frontend = "[]:443"
backend = "[127.0.0.1]:6086"
pem-dir = "/root/.acme.sh/hitch/"
pem-dir-glob = ".pem"
syslog-facility = "daemon"
daemon = on
user = "_hitch"
group = "_hitch"
ssl-engine = ""
ciphers = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
prefer-server-ciphers = on
session-cache=100
sni-nomatch-abort = on
tls-protos = TLSv1.2 TLSv1.3
tcp-fastopen = on
alpn-protos = "h2,http/1.1"
write-proxy-v2 = on
#ocsp-connect-tmo = 4
#ocsp-resp-tmo = 4
ocsp-dir = "/var/lib/hitch/"
ocsp-verify-staple = on
workers = 6 # number of CPU cores
syslog = off
quiet = on
log-level = 1
keepalive = 100
backlog = 1024
KillMode=none
The above may be the solution.