Any Idea to improve GC?
yushengery opened this issue · 10 comments
Greetings,
I am deploying the websocket service for about 70K concurrent users, however, I find the GC cause too much STW, which reached up to 2 seconds
Then I pprof the program, the most malloc comes from net/http.newBufioReader net/http.newBufioWriteSize and the net/http.(*conn).hijackLocked, I found that the hijack breaks the put to the buffer pool when a connection ended, besides, the hijacked connection has to malloc a pair of new read/write buf, is there any way to improve this ?
Best Regards.
@yushengery What version of Go are you running?
@elithrar thanks for you reply
I tried both go 1.7.5 and go 1.8
they come out the same result.
I have some issue
GC is clear your memory，your 70K uers are shot client or long client?
how many messages([]byte) create in 1 second?
I think the problem with 2 second GC almost be large memory，so it from?
and why accumulate so much then start GC
sorry for my pool engilsh
@zcdj1987
All the users are long connection, however there are not so many messages in short time , only about 1 msg / per 10 seconds.
The rss will reach up to be 7GB at most when the server keeps running for one day, thus this can easily tigger OOM...
@yushengery
can you show the memory pprof？
How often GC works?every 2 minutes?
it seem a big problem
I found that the hijack breaks the put to the buffer pool when a connection ended
Because the net/http package returns a new bufio.ReadWriter from the hijack method, the original bufio.Reader and bufio.Writer can be returned to the buffer pool. This is an issue for the net/http package.
the hijacked connection has to malloc a pair of new read/write buf, is there any way to improve this ?
Yes, the upgrade method should attempt to use the slices from the bufio.ReadWriter returned from the hijack method.
My previous comment is not correct. The bufio.Reader is returned from hijack and is therefore not pooled on connection cleanup.
The http connection's bufio.Writer is not returned in Hijack and can be returned to the buffer pool.