Defining stats and views for dynamically created endpoints

[thecodejunkie]

Hi!

I have a small service that has dynamically registered HTTP endpoints (internally I'm using Fiber for the route handling). I'd like to add some metrics to each of these endpoints but I'm not entirely sure that stats can be reused across several view instances, while still recording stats for unique resources? Basically I am doing something very similar to this, during route registration

func createRoute(name string) fiber.Handler {
	metricRequestCounterTotal := stats.Int64("req_counter", "total request counter", "1")

	view.Register(&view.View{
		Name:        fmt.Sprintf("myapplication/request_total/%s", name),
		Description: "The number of requests sent to the handler handler",
		Measure:     metricRequestCounterTotal,
		Aggregation: view.Count(),
	})

	return func(c *fiber.Ctx) error {
		stats.Record(ctx, metricRequestCounterTotal.M(1))
		return nil
	}
}

1. Create a stat req_counter to count the number of requests for the endpoint in question
2. Create a view for the stat, with a unique name for the endpoint in the format of myapplication/request_total/foo where foo would be the path of my endpoint (i.e /foo)
3. In the route handler func, that is being returned, I increase the stat by 1 for every call

However, when I did some testing using two routes, and then visualized the metrics in GCP Cloud Monitoring, they always seem to get the same values no matter which of the routes that I call. I.e if I call /foo a couple of times, it seems to register those requests on the view for both /foo and /bar. It could be my visualization that is wrong (added the metrics as a rate visualization using a count alignment function), but I think it looks alright on that end

Thanks!

[punya]

Hi @thecodejunkie! If I understand your question correctly, you're expecting each invocation of stats.Int64 to return a totally unique stat, which you're treating as scoped to the route. Unfortunately that's not how stats work; there can only be one stat with a given name across your whole application. The first call creates the stat and subsequent calls return the existing value from a registry.

Would it make sense to include the route as a tag on your measurement, and then use the tag to pull apart the aggregates? Something like

var metricRequestCounterTotal = stats.Int64("req_counter", "total request counter", "1")
var routeKey = tag.NewMustKey("route")

func init() {
	view.Register(&view.View{
		Name:        "myapplication/request_total",
		Description: "The number of requests sent to the handler handler",
		Measure:     metricRequestCounterTotal,
		Aggregation: view.Count(),
		TagKeys: []tag.Key{routeKey},
	})
}

func createRoute(name string) fiber.Handler {
	return func(c *fiber.Ctx) error {
		ctx := tag.New(c.Context(), tag.Insert(routeKey, name))
		stats.Record(ctx, metricRequestCounterTotal.M(1))
		return nil
	}
}

[punya]

There's a more complete/robust implementation of this idea built into OpenCensus, but against Go's net/http library: https://github.com/census-instrumentation/opencensus-go/blob/master/plugin/ochttp/route.go. Since fiber builds on fasthttp rather than net/http, you may need to reimplement some bits to suit your needs. I recommend using the tag and metric names from ochttp for the sake of compatibility.

[andreas-hakansson-ingka]

@punya Thanks! Yeah, that makes sense. I'll take a look at using a tag to uniquely identify each recorded stat (like in your example, we do have the name of the component that registers a route). I assume it would be possible to pull them apart in Cloud Monitoring and show individual graphs?

[punya]

Yes, an OpenCensus tag gets mapped to a Google Cloud Monitoring label - see more details at https://cloud.google.com/monitoring/custom-metrics/open-census#opencensus-vocabulary.