Dangling sockets in a CLOSE_WAIT state
xjewer opened this issue · 0 comments
xjewer commented
Prometheus can't scrapes metrics from the exporter, due to lots of CLOSE_WAIT opened sockets, thus exporter can't accept any further http request.
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 5ms
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 10ms
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 20ms
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 40ms
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 80ms
Dec 13 19:00:35 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:35 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 160ms
Dec 13 19:00:36 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:36 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 320ms
Dec 13 19:00:36 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:36 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 640ms
Dec 13 19:00:37 localhost prometheus-clickhouse-exporter[61675]: 2018/12/13 19:00:37 http: Accept error: accept tcp 127.0.0.1:3000: accept4: too many open files; retrying in 1s
:~$ sudo netstat -ntp | grep 3010 | grep CLOSE_WAIT | wc -l
3912