Exclude healthcheck & other http spans from ochttp

The ochttp wrapper in

Line 882 in 7e0ee57

adminServer.Handler = &ochttp.Handler{Handler: s.AdminRouter}

Line 858 in 7e0ee57

server.Handler = &ochttp.Handler{Handler: s.Router}

creates a ton of spans which we could do without on a jaeger/zipkin ui.

I'm working on adding StartOptions to the wrapper that can remove these requests (mostly kube-probe and prometheus) from the console. Without it, it will look like this:

.

I'm thinking we could also use this to ratelimit other traces, and perhaps even remove traces from canary tests.

We could either sample only a set of requests that we need and exclude the rest, or just find a few of the spans that we dont want and keep them out. I've got the latter working like this

https://github.com/fnproject/fn/blob/7c132f3840ffba1dd7568e73a07f52c813d6b317/api/server/server.go#L858:L867

		server.Handler = &ochttp.Handler{
			Handler: s.Router,
			GetStartOptions: func(r *http.Request) trace.StartOptions {
				startOptions := trace.StartOptions{}
				if r.UserAgent() == "Prometheus/2.7.1" || r.UserAgent() == "kube-probe/1.11" {
					startOptions.Sampler = trace.NeverSample()
				}
				return startOptions
			},
		}

We could add a list of useragents, paths that could all be excluded, seems the easiest way. This would also save on overhead for those using AlwaysSample for all spans? Also I'm guessing its easier to name the traces you won't need than to keep those that you do.