jschaf/gcp-monitoring-resource-weirdness

Experiment with GCP monitoring custom metrics and monitored resources.

GCP Monitoring: Monitored Resource Weirdness

In my prod repo, it seems that adding a Monitored Resource to CreateTimeSeries causes GCP Monitoring to drop the TimeSeries. It also seems impossible to set the allowed monitored resources when calling CreateMetricDescriptor.

References

• Custom Metrics: GCP Monitored Resources: For custom metrics, GCP Monitoring limits the monitored resource; includes k8s_container and generic_task.

• GCP Monitoring API - CreateTimeSeries

• StackOverflow: One or more points were written more frequently than the maximum sampling period configured for the metric The problem is that calling CreateTimeSeries without a set resource uses the global monitored resource. That means if you have multiple servers, they all update the same TimeSeries. Therefore, it's important to set the resource so that each server updates a separate TimeSeries.