Improve Dataproc collector's accuracy
Opened this issue · 1 comments
Describe the problem/challenge you have
The current way of retrieving a Dataproc cluster status is through its Status.State. The issue with this approach is that the default interval (5m) might possibly take some small and quick jobs out of the account and erroneously consider the cluster idle.
Describe the solution you'd like
I'd like to consider the Dataproc Jobs statuses besides the cluster status as a whole.
There is no way of listing Jobs by Cluster =/
We would have to perform two different loops which wouldn't be efficient. Perhaps diminishing the default interval between scrapes could be more effective.
Currently, the application takes ~6s to assemble all metrics and respond to the request. I believe we have room to set an interval of 60 seconds here.