Query-Frontends return "too short binary" to Grafana 11.3.0 on /detected_labels API

[diranged]

Describe the bug
Original Slack Thread: https://grafana.slack.com/archives/CEPJRLQNL/p1729782706272119

We're experimenting with the Grafana 11.3.0 "Explore Logs" tab but we're getting errors in the UI when we load it up. These errors come from a 500 returned by Grafana when it makes a call to the /loki/api/v1/detected_labels API.

Here is the Grafana log:

logger=tsdb.loki endpoint=CheckHealth endpoint=callResource pluginId=loki dsName=Loki dsUID=loki uname... t=2024-10-24T15:06:26.992054397Z level=error msg="Failed resource call from loki" err="too short binary" url="/loki/api/v1/detected_labels?query=%7Bservice_name%3D%60proxy%60%7D&start=2024-10-24T14%3A51%3A21.610Z&end=2024-10-24T15%3A06%3A21.610Z

Here's the log from the Loki query-frontend pod:

level=warn ts=2024-10-24T15:04:48.967315256Z caller=logging.go:128 orgID=fake|dev|staging|production msg="GET /loki/api/v1/detected_labels?query=%7Bservice_name%3D%60app%60%7D&start=2024-10-24T14%3A49%3A48.843Z&end=2024-10-24T15%3A04%3A48.843Z (500) 6.670356ms Response: \"too short binary\" ws: false; 

To Reproduce

1. Grafana 11.3.0 + Loki 3.2.0
2. Loki in deploymentMode: Distributed
3. Loki configured with multiple tenants
4. Grafana configured to send multiple tentants via the headers (orgID=fake|dev|staging|production)
5. Load up the Explore Logs page, see error:

Expected behavior
I expect it to work. :)

Environment:

• Grafana 11.3.0
• Loki 3.2.0 via the Loki chart at https://grafana.github.io/helm-charts - version 6.18.0

[trevorwhitney]

@diranged which encoding are you using for the frontend? Is this reproducible everytime you curl /loki/api/v1/detected_labels.

cc @shantanualsi who also has context on the detected labels endpoint.