comparison binary operator doesn't work in Cloud Monitoring

[hzhang4-sc]

Hi team, I'm trying to create alert in Cloud Monitoring with PromQL, see below query.

count(histogram_quantile(0.99,sum by (le, queue_id)(increase(cloudtasks_googleapis_com:queue_task_attempt_delays_bucket{monitored_resource="cloud_tasks_queue", queue_id=~"rules-.*|rule-sets.*"}[60s]))) / 1000 > 60) > 2

Basically I want to monitor on the number task queues latency over 1 minute. I believe the Cloud Monitoring console treats the > as alarm binary operator rather than PromQL filtering operator, and thus the query failed. I would suggest supporting escape character with \ so that these special character can keep its semantics in PromQL.

[lyanco]

Could you send me the project ID that you're using? You can email it to me at (my github username) at (the company I work for's domain). Thanks!

[lyanco]

And did you run this query in the alerting panel or the metrics explorer panel?

[hzhang4-sc]

The metrics explorer works great, it's the alerting panel that confuses the metrics threshold with the filtering

[EranGabber]

I verified that nested comparison operators work as expected (they all perform filtering). The result of your PromQL query is non-empty when the inner threshold is less than 60.

For example, when I executed this query:

count(histogram_quantile(0.99,sum by (le, queue_id)(increase(cloudtasks_googleapis_com:queue_task_attempt_delays_bucket{monitored_resource="cloud_tasks_queue", queue_id=~"rules-.*|rule-sets.*"}[60s]))) / 1000 > 32) > 2

at Fri Feb 23 05:07:43 AM UTC 2024, the result was:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {},
        "value": [
          1708664743,
          "16"
        ]
      }
    ]
  }
}

[EranGabber]

I may have misunderstood your original question. If you want the outer ">" operator to be a numeric comparison instead of a filter, you should write it as "> bool". See the explanation here. The > operator always performs filtering regardless of its position in the query.

Here is how I verified that the > operator performs filtering. I evaluated a sequence of progressively more complex PromQL queries and verified their output. It is easy to determine whether an operation is a filter or a comparison based on their output. The output of a comparison operation is either the constant 0 or the constant 1.

1. The output of the query vector(60) > 60 is empty.
2. The output of the query vector(61) > 60 is 61.
3. The output of the query count(vector(60) > 60) is empty, because the input of the count aggregation operator is empty.
4. The output of the query count(vector(61) > 60) is 1.
5. The output of count(vector(60) > 60) > 0 is empty.
6. The output of count(vector(61) > 60) > 0 is 1.

[hzhang4-sc]

Ok I reran the query and it succeeded this time! I guess there is indentation issue when I copy paste the query from metrics explorer to alerting panel, I was seeing error "left parenthesis is not closed". This is a false alarm, thank you for looking into it!