grafana/k8s-monitoring-helm

401 Unauthorized Error Despite Correct Credentials

JensvanZutphen opened this issue · 5 comments

JensvanZutphen commented

Issue: Unauthorized Error Despite Correct Credentials

Description:
I'm encountering an "Unauthorized" error even though I've verified that the credentials in my secrets file are correct.

Steps to Reproduce:

  • Apply the provided kustomization.yaml to deploy Grafana monitoring.
  • Monitor the logs for authentication errors.
  • Confirm that the credentials in monitoring-secrets.yaml are correct.

Expected Behavior:

  • Authentication should succeed with the provided credentials.

Actual Behavior:

Authentication fails with a 401 "Unauthorized" error.

Additional Information:

When attempting to authenticate with the provided credentials, the server returns HTTP status 401 "Unauthorized".
The secrets are encrypted using SOPS.
Even with the secrets in plain text, the issue persists.

Environment:

Kubernetes cluster managed with fluxcd and k3s.
Grafana version: 1.0.0
Attachments:

kustomization.yaml
monitoring-secrets.yaml
namespace.yaml
release.yaml
repository.yaml

Notes:
This issue seems to be related to the authentication process despite providing the correct credentials. Any insights or suggestions on how to resolve this would be appreciated.

apps/grafana-monitoring/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: grafana-monitoring

resources:
- namespace.yaml
- release.yaml
- repository.yaml
- monitoring-secrets.yaml

apps/grafana-monitoring/monitoring-secrets.yaml

apiVersion: v1
kind: Secret
metadata:
    name: monitoring-secrets
type: Opaque
stringData:
    externalservices-username: ENC[AES256_GCM,data:xbQFHUe5IA==,iv:C3kCkVRErQcg+kDkk/EiO3wKzviRV9BXrRdJUhxkGT8=,tag:VO1xZvfvrSVl+xJvlWZWOQ==,type:str]
    loki-username: ENC[AES256_GCM,data:o0UE7qvM,iv:q97451GJDbzPulafLJ4zfxx1SyL1/k2UTHoZ0AshCIM=,tag:9/1Vw4NvdRq1WEsZnZO8Eg==,type:str]
    externalservices-password: ENC[AES256_GCM,data:CLLpI7Xh1q1KTLocd6wHx1NaFCwhvzDCZLnbEe9Q2LK2Og4k/0TEq17Fdx1aNBM36ZNLJjuw6gGYeE38Z7dJG9D8sj8HEZbS69UZ6+ctHuEdHZgASy1MNz1SgBXzM44cc9YEOW2z+Bl7xqFZ0FkMbc+SCAb/p/Erbs8kgHgWjhFSgmzPOpVvItYcJ4j1UL+HJJbwbgnqvthD6XxacSDe9mq55to=,iv:T9hNI3ktTxwxDspLONo5vXpI6kVTuu8ERyMSdVeyrJ8=,tag:Qhh3wJ18EiV7TKvs12esHw==,type:str]
    loki-password: ENC[AES256_GCM,data:anIZaCnOXN0Ulu1UB4L+j4LsYdBA9N7Pvhh7hibN6g+k6KoHwgszpL//JmB0/UBJec+Yzd4e8w670KdmTcrDmf2S1UspmIxsKJPvDKIEPKFgBNIeQ36JW27gXUaSOU0E2B+FK9XG6zBZ5ECm/qWe5E7ZTtrgLrh/AkP0OIHfGzUwqCJIwGxI0Mav4c8a1/9AENrWbeIunu3Sk/7MMxIu/FAdbwU=,iv:rCirKrWumioh1Jgncynd6qQy+f2civuIOoccVTuhITg=,tag:iZDyn6koawFCxYauMxOMfA==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2024-04-30T09:05:03Z"
    mac: ENC[AES256_GCM,data:jefx7uhk35PXHM4c+icEwkChP9ni0uwjSDtGyqmGM1pv+/Bv011daGLZqA6h7WGtqz5h3xEp8akrAK203+yVe6QF4YW7977WxmUsTKkwSLEVtYEaRXJTD600mgxrNxN2bqACjXMJZuXRI8BbHQVKDXMhpjdMFBTWcLOyh+t76J4=,iv:Bi/9tGhS55rnhQWMp8o5xFmL9X/Kfqi8nkmzD9b24Ns=,tag:dj6wnlIIBBz57Pkx2uwXrg==,type:str]
    pgp:
        - created_at: "2024-04-30T09:05:03Z"
          enc: |-
            -----BEGIN PGP MESSAGE-----

            hQIMAyQjoAaCsIHPAQ//UZy1z3Ie8ar669s+3VWR4NZSxw2w30u9hipa8fxa9qEC
            5Sy3YgSCtD6G6H4yQmWi9SYzCqMg5caIiN7F/KFMP8jmyohVUBtFSDrTNhQLESNs
            8L1iYAYjaL3RIls+AsiedXee69kLz0hLoeSJccIJHUoJQiVmutZEv5L7J4i4LJAg
            9fmmjBERoNhfd8KjulSNDSbuqxVTrLCt6HxLCXMbo7rSHYYY52ZoM1SleGBrOfRb
            HZPFOR+OFhN6/T7gMsQkK80PFIgASw4ZTLTg+NqlbMk4xKdw2+21eJe1Zr71i/1K
            7kkbujYTfFzsR7wyzzPEdaH6TybsO9Ybi3bjGaFLA6oDkyzt7zcrVpyo0KNN70mO
            DolWZ3215ZwwkC5bbKmBQmcdeOSvV0P9w0MAErDqSmSUqzn8SXjsI2pEqCfxdoyi
            0/OcHRaLucHG8F1OcXN65ej/8CeE52xO6q/LOlWB2J1Lmbela8PDEO4mrYgOPZSH
            ZBMEd18SGzmdxXlGw7mgM93orBvoJWeL3g4EsODXjq7fyC0DmPI0kxNWpiNAiRwO
            gST/pNcvsHFNIG7OtWeH14RWG6I+kzROp04RIzi7HLE5CY5MNSHVLRn+L3UsSX5B
            lZZWHXlMNIpGazyAoAzBOmkXK7JAdGnpxhmj/NWLzRxAcpGCYUHg3grvOZV83fzS
            XgFOnQVFyqLTRz3xMVYebYebs7Q1ylO4At0p8s9xH10+Kr7MJNqAqZ1eQZBAbXob
            Rzy7mvH2dN8fvVWiVHMJjyJnH3d0XvW8MNtBpjChBgYA55og93RaH5+IsmPvh1Y=
            =xbU4
            -----END PGP MESSAGE-----
          fp: EA30182F2B668EB7063485CC9C28CDADBBC5A84B
    encrypted_regex: ^(data|stringData)$
    version: 3.8.1

apps/grafana-monitoring/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: grafana-monitoring

apps/grafana-monitoring/release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: grafana-k8s-monitoring
spec:
  interval: 15m
  chart:
    spec:
      version: "1.0.0"
      chart: k8s-monitoring
      sourceRef:
        kind: HelmRepository
        name: grafana
      interval: 60m
  values:
    cluster:
      name: z121
    externalServices:
      prometheus:
        host: https://prometheus-prod-24-prod-eu-west-2.grafana.net
        basicAuth:
          existingSecret: monitoring-secrets
          username: externalservices-username
          password: externalservices-password
      loki:
        host: https://logs-prod-012.grafana.net
        authMode: "basic"
        basicAuth:
          existingSecret: monitoring-secrets
          username: loki-username
          password: loki-password
    metrics:
      enabled: true
      cost:
        enabled: false
      node-exporter:
        enabled: true
    logs:
      enabled: true
      pod_logs:
        enabled: true
      cluster_events:
        enabled: true
    traces:
      enabled: false
    receivers:
      grpc:
        enabled: false
      http:
        enabled: false
      zipkin:
        enabled: false
    opencost:
      enabled: false
    kube-state-metrics:
      enabled: true
    prometheus-node-exporter:
      enabled: true
    prometheus-operator-crds:
      enabled: true
    alloy: {}
    alloy-logs: {}

apps/grafana-monitoring/repository.yaml

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: grafana
spec:
  interval: 5m
  type: default
  url: https://grafana.github.io/helm-charts
JensvanZutphen commented

ps:
I've also used usernameKey and passwordKey to no avail

skl commented

@JensvanZutphen from the available values, assuming you want to use your existing secret, could you try something like the following?

externalServices:
  prometheus:
    host: https://prometheus-prod-24-prod-eu-west-2.grafana.net
    basicAuth:
      usernameKey: "externalservices-username"
      passwordKey: "externalservices-password"

    secret:
      create: false
      name: "monitoring-secrets"
      namespace: "grafana-monitoring"

  loki:
    host: https://logs-prod-012.grafana.net
    basicAuth:
      usernameKey: "loki-username"
      passwordKey: "loki-password"

    secret:
      create: false
      name: "monitoring-secrets"
      namespace: "grafana-monitoring"
JensvanZutphen commented

Yes this worked thank you :)

now i get this: alloy ts=2024-04-30T10:25:20.414001119Z level=warn msg="error sending batch, will retry" component_path=/ component_id=loki.write.logs_service component=client host="" status=-1 tenant="" error="Post "/loki/api/v1/push": unsupported protocol scheme """

but i'll firstt look around if i can find a fix

skl commented

@JensvanZutphen I think, when using your own secret, you also need to put the host in the secret (see this helm template).

Add the prometheus and loki hosts to your secret so it looks something like (but replace <...> with your values):

apiVersion: v1
kind: Secret
metadata:
    name: monitoring-secrets
type: Opaque
stringData:
    externalservices-host: <host>
    externalservices-username: <username>
    externalservices-password: <password>
    loki-host: <host>
    loki-password: <username>
    loki-username: <password>

Set your hostKey for prometheus and loki in the helm values to match the above secret:

externalServices:
  prometheus:
    host: https://prometheus-prod-24-prod-eu-west-2.grafana.net
    hostKey: "externalservices-host"
    basicAuth:
      usernameKey: "externalservices-username"
      passwordKey: "externalservices-password"

    secret:
      create: false
      name: "monitoring-secrets"
      namespace: "grafana-monitoring"

  loki:
    host: https://logs-prod-012.grafana.net
    hostKey: "loki-host"
    basicAuth:
      usernameKey: "loki-username"
      passwordKey: "loki-password"

    secret:
      create: false
      name: "monitoring-secrets"
      namespace: "grafana-monitoring"
JensvanZutphen commented

Everything works now, thank you very much :)

  • 🎉1