Memoryleak on reconnecting
Closed this issue · 2 comments
Hi @42wim , matterircd got an odd issue last night, an OOM condition. I have about 180k lines, seems Go dumps all the routines in an OOM event heheh
Let me know if that's all needed, the gzipped blob has 500K, I can attach here.
I'm running matterircd from master, at commit 9bfde7c . Also, seems matterircd was facing issue #302 while the OOM condition happened. Right after the issue, I saw the following, might be enough to understand the problem - thanks in advance!
`time="2020-08-20T04:04:30+02:00" level=debug msg="addUserToChannelWorker&{security 7jiftgd7ii8bdksnxfips49bza sqmc4sz45prypmkfctwynm5yjr false false}" module=matterircd
time="2020-08-20T04:04:30+02:00" level=debug msg="Adding channel &bridge.ChannelInfo{Name:"matterircd", ID:"uw577mmmgfbtjg9i6bz9wk97or", TeamID:"sqmc4sz45prypmkfctwynm5yjr", DM:false, Private:false}" module=matterircd
fatal error: runtime: out of memory
runtime stack:
runtime.throw(0xc3f225, 0x16)
/home/guilherme/app/go/src/runtime/panic.go:1116 +0x72
runtime.sysMap(0xc068000000, 0x4000000, 0x12a7038)
/home/guilherme/app/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0x1292100, 0x400000, 0x1292108, 0x10)
/home/guilherme/app/go/src/runtime/malloc.go:715 +0x1cd
runtime.(*mheap).grow(0x1292100, 0x10, 0x0)
/home/guilherme/app/go/src/runtime/mheap.go:1286 +0x11c
runtime.(*mheap).allocSpan(0x1292100, 0x10, 0x100, 0x12a7048, 0x4175d6)
/home/guilherme/app/go/src/runtime/mheap.go:1124 +0x6a0
runtime.(*mheap).alloc.func1()
/home/guilherme/app/go/src/runtime/mheap.go:871 +0x64
runtime.(*mheap).alloc(0x1292100, 0x10, 0x101, 0x200)
/home/guilherme/app/go/src/runtime/mheap.go:865 +0x81
runtime.largeAlloc(0x1fe00, 0x7f98be600101, 0x4648ba)
/home/guilherme/app/go/src/runtime/malloc.go:1152 +0x92
runtime.mallocgc.func1()
/home/guilherme/app/go/src/runtime/malloc.go:1047 +0x46
runtime.systemstack(0x7ffe6a069f60)
/home/guilherme/app/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
/home/guilherme/app/go/src/runtime/proc.go:1041
goroutine 46246 [running]:
runtime.systemstack_switch()
/home/guilherme/app/go/src/runtime/asm_amd64.s:330 fp=0xc062751910 sp=0xc062751908 pc=0x466a50
runtime.mallocgc(0x1fe00, 0xb0e6e0, 0x66a001, 0xc062751a10)
/home/guilherme/app/go/src/runtime/malloc.go:1046 +0x895 fp=0xc0627519b0 sp=0xc062751910 pc=0x40e835
runtime.makeslice(0xb0e6e0, 0xfe00, 0x1fe00, 0xc067e71dc8)
/home/guilherme/app/go/src/runtime/slice.go:49 +0x6c fp=0xc0627519e0 sp=0xc0627519b0 pc=0x44dc4c
encoding/json.(*Decoder).refill(0xc067e71ce0, 0x65, 0x0)
/home/guilherme/app/go/src/encoding/json/stream.go:159 +0x157 fp=0xc062751a48 sp=0xc0627519e0 pc=0x66dc87
encoding/json.(*Decoder).readValue(0xc067e71ce0, 0x0, 0x0, 0xbe4000)
/home/guilherme/app/go/src/encoding/json/stream.go:140 +0x1e8 fp=0xc062751aa0 sp=0xc062751a48 pc=0x66d9a8
encoding/json.(*Decoder).Decode(0xc067e71ce0, 0xae69a0, 0xc067f59500, 0x7f98e677e470, 0xc067f594e0)
/home/guilherme/app/go/src/encoding/json/stream.go:63 +0x79 fp=0xc062751ae0 sp=0xc062751aa0 pc=0x66d489
github.com/mattermost/mattermost-server/v5/model.UserListFromJson(...)
/home/guilherme/go/pkg/mod/github.com/mattermost/mattermost-server/v5@v5.25.2/model/user.go:734
github.com/mattermost/mattermost-server/v5/model.(*Client4).GetUsersInChannel(0xc063db3900, 0xc064606920, 0x1a, 0x0, 0xc8, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/home/guilherme/go/pkg/mod/github.com/mattermost/mattermost-server/v5@v5.25.2/model/client4.go:934 +0x3e8 fp=0xc062751c10 sp=0xc062751ae0 pc=0x942b18
github.com/42wim/matterircd/bridge/mattermost.(*Mattermost).GetChannelUsers(0xc0000f1dd0, 0xc064606920, 0x1a, 0x785bac, 0xb06e20, 0x122a2a0, 0xb06e20, 0x122a2a0)
/home/guilherme/src/matterircd/bridge/mattermost/mattermost.go:411 +0x197 fp=0xc062751cd8 sp=0xc062751c10 pc=0xa025d7
github.com/42wim/matterircd/mm-go-irckit.(*User).syncChannel(0xc00016cc30, 0xc064606920, 0x1a, 0xc06462c3a0, 0x2)
/home/guilherme/src/matterircd/mm-go-irckit/userbridge.go:507 +0x69 fp=0xc062751da8 sp=0xc062751cd8 pc=0xa7f899
github.com/42wim/matterircd/mm-go-irckit.(*User).createSpoof(0xc00016cc30, 0xc063fc3840, 0xc000bd1f78)
/home/guilherme/src/matterircd/mm-go-irckit/userbridge.go:405 +0x16d fp=0xc062751e28 sp=0xc062751da8 pc=0xa7e4ad
github.com/42wim/matterircd/mm-go-irckit.(*User).addUserToChannelWorker(0xc00016cc30, 0xc0648a0660, 0xc0642497c0)
/home/guilherme/src/matterircd/mm-go-irckit/userbridge.go:417 +0x13d fp=0xc062751fc8 sp=0xc062751e28 pc=0xa7e7ed
runtime.goexit()
/home/guilherme/app/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc062751fd0 sp=0xc062751fc8 pc=0x468b61
created by github.com/42wim/matterircd/mm-go-irckit.(*User).addUsersToChannels
/home/guilherme/src/matterircd/mm-go-irckit/userbridge.go:369 +0x2c3
`
Yes, the problem is goroutine leak when reconnecting, but that should be A LOT of reconnects going on then.
My matterircd uses 13MB of RAM at the moment (after 3 days)
This should be fixed now in master.