Need to block checking if metric descriptors exist to prevent rate limiting by Google
Closed this issue · 0 comments
Is your feature request related to a problem? Please describe.
When a large number of clients are started at the same time (e.g. 1000), the calls to the Google Cloud Monitoring API to check if metric descriptors exist will result in the clients being rate limited. Until a client successfully completes the metric descriptor check, it will be blocked from sending metrics. This causes some clients to fail to send metrics for some number of minutes. Relatively short-lived clients, such as those that are only running for 5-10 minutes, may never be able to send metrics. This rate limit is also a project-level rate limit so any other client in the project that tries to get or create metric descriptors will also be rate limited.
Describe the solution you'd like
Other projects that interact with the Google Cloud Metrics API have an option to assume metric descriptors already exist. It would be useful to have that option in this library such that short lived clients could come and go without contributing to the rate limit.
Additional context
I will submit a PR shortly (it is a pretty small change).