Leaking file descriptors

[kirillrdy]

When doing http requests in golang, you need to close body or it will leave open socket connections.
https://github.com/iamatypeofwalrus/logspout-loggly/blob/master/loggly/loggly.go#L106
The limit of open files on linux is 1024.
The issue can be verified using lsof util.

Fix is very simple
https://github.com/kirillrdy/logspout-loggly/commit/5cf3073428805c751038d510a0a27884fbabcd68

For references:
https://golang.org/pkg/net/http/
The client must close the response body when finished with it:

[iamatypeofwalrus]

You only need to defer resp.Body.Close() if you actually open and use the body. The code only uses Status code which is on the Response struct

From the http docs:

resp, err := http.Get("http://example.com/")
if err != nil {
    // handle error
}
defer resp.Body.Close()

// Body is actually being used
body, err := ioutil.ReadAll(resp.Body)
// ...

Examples of requests where body is not accessed and thus not closed

resp, err := http.Get("http://example.com/")
...
resp, err := http.Post("http://example.com/upload", "image/jpeg", &buf)
...
resp, err := http.PostForm("http://example.com/form",
    url.Values{"key": {"Value"}, "id": {"123"}})

[kirillrdy]

Hi,
I can reproduce the issue with this small go app.
It should be equivalent to what your are doing in your code.
Here is my env:

go version
go version go1.4.3 linux/amd64

uname -a
Linux yao-local 4.2.3-300.fc23.x86_64 #1 SMP Mon Oct 5 15:42:54 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

ulimit -a
1024

Limit of open files per process on linux is 1024

package main

import (
    "bytes"
    "log"
    "net/http"
)

func pan(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

func doRequest() {
    js := []byte{1, 2, 3, 4, 5}
    req, err := http.NewRequest("POST", "http://google.com/", bytes.NewBuffer(js))
    pan(err)
    _, err = client.Do(req)
    pan(err)
}

var client http.Client

func main() {
    for i := 0; i < 2015; i++ {
        log.Println(i)
        doRequest()
    }
}

End of output:

2016/02/11 09:05:21 986
2016/02/11 09:05:21 987
2016/02/11 09:05:21 988
2016/02/11 09:05:21 989
2016/02/11 09:05:21 990
2016/02/11 09:05:22 991
2016/02/11 09:05:27 992
2016/02/11 09:05:27 993
2016/02/11 09:05:27 994
2016/02/11 09:05:27 995
2016/02/11 09:05:32 996
2016/02/11 09:05:33 997
2016/02/11 09:05:33 998
2016/02/11 09:05:33 999
2016/02/11 09:05:33 1000
2016/02/11 09:05:33 1001
2016/02/11 09:05:34 1002
2016/02/11 09:05:34 1003
2016/02/11 09:05:34 1004
2016/02/11 09:05:34 1005
2016/02/11 09:05:34 1006
2016/02/11 09:05:35 1007
2016/02/11 09:05:35 1008
2016/02/11 09:05:35 1009
2016/02/11 09:05:35 1010
2016/02/11 09:05:35 1011
2016/02/11 09:05:36 1012
2016/02/11 09:05:36 1013
2016/02/11 09:05:36 1014
2016/02/11 09:05:36 1015
2016/02/11 09:05:36 1016
2016/02/11 09:05:37 1017
2016/02/11 09:05:37 1018
2016/02/11 09:05:37 1019
2016/02/11 09:05:37 1020
2016/02/11 09:05:37 Post http://google.com/: dial tcp: lookup google.com: too many open files

You can also see number of open files growing using lsof util.

[iamatypeofwalrus]

Confirmed via your code along with this link and this other one. Will patch tonight along with your other issue.

Thanks for these!

[iamatypeofwalrus]

fixed in 8943d77