Go tracing and monitoring (Prometheus) for `net.Conn`

Prometheus monitoring and x/net/trace tracing wrappers net.Conn, both inbound (net.Listener) and outbound (net.Dialer).

Why?

Go standard library does a great job of doing "the right" things with your connections: http.Transport pools outbound ones, and http.Server sets good Keep Alive defaults. However, it is still easy to get it wrong, see the excellent The complete guide to Go net/http timeouts.

That's why you should be able to monitor (using Prometheus) how many connections your Go frontend servers have inbound, and how big are the connection pools to your backends. You should also be able to inspect your connection without ssh and netstat.

How to use?

All of these examples can be found in example/server.go:

Conntrack Dialer for HTTP DefaultClient

Most often people use the default http.DefaultClient that uses http.DefaultTransport. The easiest way to make sure all your outbound connections monitored and trace is:

http.DefaultTransport.(*http.Transport).DialContext = conntrack.NewDialContextFunc(
    conntrack.DialWithTracing(),
    conntrack.DialWithDialer(&net.Dialer{
        Timeout:   30 * time.Second,
        KeepAlive: 30 * time.Second,
    }),
)

Dialer Name

Tracked outbound connections are organised by dialer name (with default being default). The dialer name is used for monitoring (dialer_name label) and tracing (net.ClientConn.<dialer_name> family).

You can pass conntrack.WithDialerName() to NewDialContextFunc to set the name for the dialer. Moreover, you can set the dialer name per invocation of the dialer, by passing it in the Context. For example using the ctxhttp lib:

callCtx := conntrack.DialNameToContext(parentCtx, "google")
ctxhttp.Get(callCtx, http.DefaultClient, "https://www.google.com")

Conntrack Listener for HTTP Server

Tracked inbound connections are organised by listener name (with default being default). The listener name is used for monitoring (listener_name label) and tracing (net.ServerConn.<listener_name> family). For example, a simple http.Server can be instrumented like this:

listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
listener = conntrack.NewListener(listener, 
    conntrack.TrackWithName("http"), 
    conntrack.TrackWithTracing(),
    conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
httpServer.Serve(listener)

Note, the TrackWithTcpKeepAlive. The default http.ListenAndServe adds a tcp keep alive wrapper to inbound TCP connections. conntrack.NewListener allows you to do that without another layer of wrapping.

TLS server example

The standard lobrary http.ListenAndServerTLS does a lot to bootstrap TLS connections, including supporting HTTP2 negotiation. Unfortunately, that is hard to do if you want to provide your own net.Listener. That's why this repo comes with connhelpers package, which takes care of configuring tls.Config for that use case. Here's an example of use:

listener, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
listener = conntrack.NewListener(listener, 
    conntrack.TrackWithName("https"), 
    conntrack.TrackWithTracing(),
    conntrack.TrackWithTcpKeepAlive(5 * time.Minutes))
tlsConfig, err := connhelpers.TlsConfigForServerCerts(*tlsCertFilePath, *tlsKeyFilePath)
tlsConfig, err = connhelpers.TlsConfigWithHttp2Enabled(tlsConfig)
tlsListener := tls.NewListener(listener, tlsConfig)
httpServer.Serve(listener)

Status

This code is being tried out in staging environments of Improbable's HTTP frontending stack ATM.

Additional tooling will be added if needed, and contributions are welcome.

#License

go-conntrack is released under the Apache 2.0 license. See the LICENSE file for details.

fabxc/go-conntrack

Go tracing and monitoring (Prometheus) for net.Conn