Provide context out of API boundaries
Closed this issue · 7 comments
Freki crashes with error in connHandler
error: [user.tcp] accept tcp [::]:5000: accept4: too many open files
I think it is because of not closing connections passed to the user out of its boundary.
Ref: https://github.com/kung-foo/freki/blob/master/userconnhandler_tcp.go#L58
Do you have a stack trace?
No, as far as I know the error we are getting only occurs when server is not closing connections.
I encounter this problem again today:
Initially it hits on a lot of the message 'too many open files' and eventually the log size growing until 52GB ++ on my test machine. It consumed my whole disk space and I press a "Ctrl + Z", thus lead to this stack error:
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
Failed to write to log, write /home/glutton/log/glutton.log: no space left on device
panic: interface conversion: error is *os.PathError, not *exec.ExitError
goroutine 24 [running]:
panic(0x9492e0, 0xc4211a3fc0)
/usr/local/go/src/runtime/panic.go:500 +0x1a1
github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables.(*IPTables).runWithOutput(0xc421161680, 0xc4223a76c0, 0xe, 0x1a, 0x0, 0x0, 0x0, 0x0)
/home/glutton/go/src/github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables/iptables.go:251 +0x4ad
github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables.(*IPTables).run(0xc421161680, 0xc42015c180, 0xc, 0xc, 0xc421396000, 0x8)
/home/glutton/go/src/github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables/iptables.go:221 +0x5b
github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables.(*IPTables).Delete(0xc421161680, 0x9e0901, 0x3, 0x9ebc13, 0xa, 0xc421396000, 0x8, 0x8, 0x0, 0x44)
/home/glutton/go/src/github.com/kung-foo/freki/vendor/github.com/coreos/go-iptables/iptables/iptables.go:136 +0x172
github.com/kung-foo/freki.(*iptrule).Delete(0xc4212d8be8, 0xc421161680, 0x0, 0x482733)
/home/glutton/go/src/github.com/kung-foo/freki/freki.go:52 +0x6e
github.com/kung-foo/freki.(*Processor).resetIPTables(0xc421364960, 0xc421a7a600, 0x1)
/home/glutton/go/src/github.com/kung-foo/freki/freki.go:145 +0xce
github.com/kung-foo/freki.(*Processor).cleanup(0xc421364960, 0xc4212d8e38, 0x48215d)
/home/glutton/go/src/github.com/kung-foo/freki/freki.go:248 +0xf7
github.com/kung-foo/freki.(*Processor).Shutdown.func1()
/home/glutton/go/src/github.com/kung-foo/freki/freki.go:240 +0x4a
sync.(*Once).Do(0xc4213649a0, 0xc4212d8e78)
/usr/local/go/src/sync/once.go:44 +0xdb
github.com/kung-foo/freki.(*Processor).Shutdown(0xc421364960, 0x0, 0x0)
/home/glutton/go/src/github.com/kung-foo/freki/freki.go:241 +0x71
github.com/gento/glutton.(*Glutton).Shutdown(0xc42137f2c0, 0xc4212d8f20, 0x1)
/home/glutton/go/src/github.com/gento/glutton/glutton.go:169 +0x2f
main.main.func1()
/home/glutton/go/src/github.com/gento/glutton/app/server.go:78 +0x171
main.main.func2()
/home/glutton/go/src/github.com/gento/glutton/app/server.go:83 +0x26
main.onInterruptSignal.func1(0xc42137ea80, 0xc421387520)
/home/glutton/go/src/github.com/gento/glutton/app/server.go:28 +0x4a
created by main.onInterruptSignal
/homeglutton/go/src/github.com/gento/glutton/app/server.go:29 +0xbf
glutton@honeypot:~/log$ ls -lht
total 52G
-rw------- 1 glutton glutton 52G May 6 01:33 glutton.log
glutton@honeypot:~/log$
@glaslos I think https://github.com/mushorg/glutton/blob/master/glutton.go#L141-L161 needs a bit of a rework.
It is important to note that when freki invokes the user-provided conn handler, it is done in a separate goroutine. This means unless you have a special use case, you don't need to spawn one yourself.
Second, freki won't explicitly close the connection except in the case of a panic. And if I added an explicit close, it would still fail with glutton since you return (after spawning a goroutine).
So I'd recommend that you add a defer conn.Close() and then just call protocolHandler(conn).
ah, my bad... I couldn't catch the right issue. I will fix this soon.
@furusiyya can you open an issue on the glutton side so we can close this one?