Prometheus exceeded maximum resolution of 11,000 points or empty result

[iamatulsingh]

It's an awesome work I would say. I tried this and everything working but if you try to get the metrics of past 2 days or later, it will eventually fail with the error "Prometheus exceeded maximum resolution of 11,000 points or empty result" for some metrics such as "Total Download" or "Total Upload".

[olmo1501]

What is failing? Grafana? If so: what is the selected timeframe?

[iamatulsingh]

I think it's Prometheus and hence Grafana. Timeframe is 2 days.

[olmo1501]

How often do you poll?

But after all this seems to be a hardcoded Prometheus limit. See prometheus/prometheus#2253

[iamatulsingh]

Here you can check the actual detail.

[sberk42]

the dashboard normally has a hard coded limit of max datapoints 10000 to avoid this issue

check the panels query options - probably it is messed up

[olmo1501]

I think the main "problem" is within Prometheus scrape interval. More than 11000 points in a 2 day timeframe would mean scraping about every 10 - 15 seconds which is way too much. My observation is that there are values a Fritzbox is updating every 3 minutes so tht would be a good starting point.

Also remember Pormetheus' hardcoded limit of 11000 datapoints per query as mentioned earlier on.

[iamatulsingh]

hardcoded limit is a problem but may be some other way to fix this. I tried rate but that's not good.