unable to validate rule group payload

[lesaux]

Describe the bug
Sample prometheusrules from this repo doesn't work.

To Reproduce
Steps to reproduce the behavior:

1. Deploy loki through distributed helm chart.
2. Set prometheusRule.enable to true i.e.

 prometheusRule:
  enabled: true
  namespace: loki
  labels:
    alloy: loki
  groups:
    - name: loki_rules
      rules:
        - expr: |-
            histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m]))
            by (le, cluster, job))
          record: cluster_job:loki_request_duration_seconds:99quantile
          labels:
            loki: true
          annotations:
            loki: true

3. Grafana alloy operator tries to create the rule but receives an error from the loki ruler:

could not parse expression for record 'cluster_job:loki_request_duration_seconds:99quantile' in group 'loki_rules': parse error at line 1, col 1: syntax error: unexpected IDENTIFIER

grafana-alloy config

loki.rules.kubernetes "alloyrules" {
  address = "http://loki-loki-distributed-ruler.loki.svc.cluster.local:3100"
  rule_selector {
    match_labels = {
      alloy = "loki",
    }
  }
}

loki ruler configuration:

ruler:
  storage:
    type: gcs
    gcs:
      bucket_name: dev-loki-ruler
  ring:
    kvstore:
      store: memberlist
  rule_path: /tmp/loki/scratch
  alertmanager_url: http://mimir-nginx.mimir.svc.cluster.local/alertmanager
  external_url: http://mimir-nginx.mimir.svc.cluster.local/alertmanager
  enable_api: true
  enable_alertmanager_v2: true
  remote_write:
    enabled: true
    clients:
      fake:
        url: http://mimir-nginx.mimir.svc.cluster.local/api/v1/push
    add_org_id_header: true
  wal:
    dir: /tmp/loki/ruler-wal

Expected behavior
The rule gets created in the loki ruler properly.

Additionally, I am confused that metric concerning loki is stored in mimir. How does the loki ruler know how to query data out of loki or mimir depending on the query??

Environment:

• Infrastructure: GKE
• Deployment tool: helm distributed chart