Please clarify how to collect metrics from RabbitMQ with auth

Question

Please clarify how to collect metrics from RabbitMQ with auth

rerime opened this issue 2 years ago · 20 comments

Hi!
Just installed netdata. I want to collect queue information from RabbitMQ. As I understand it is auto enabled by sd if port and image name matches the pattern. Should I add rabbitmq: yes in modules conf? How to pass credentials for collector to auth in rabbit?

rabbitmq pod info

containers:
    - name: rabbitmq
      image: artifactory.example.org/docker/rabbitmq:3.8-management
      ports:
        - name: http
          containerPort: 15672
          protocol: TCP
        - name: amqp
          containerPort: 5672
          protocol: TCP

Install:

helm upgrade -i netdata netdata/netdata \
--namespace netdata \
--create-namespace \
--wait \
--atomic

netdata-conf-parent

data:
  health: |
    SEND_EMAIL="NO"
    SEND_SLACK="YES"
    SLACK_WEBHOOK_URL=""
    DEFAULT_RECIPIENT_SLACK=""
    role_recipients_slack[sysadmin]="${DEFAULT_RECIPIENT_SLACK}"
    role_recipients_slack[domainadmin]="${DEFAULT_RECIPIENT_SLACK}"
    role_recipients_slack[dba]="${DEFAULT_RECIPIENT_SLACK}"
    role_recipients_slack[webmaster]="${DEFAULT_RECIPIENT_SLACK}"
    role_recipients_slack[proxyadmin]="${DEFAULT_RECIPIENT_SLACK}"
    role_recipients_slack[sitemgr]="${DEFAULT_RECIPIENT_SLACK}"
  netdata: |
    [global]
      memory mode = dbengine

    [plugins]
      cgroups = no
      tc = no
      enable running new plugins = no
      check for new plugins every = 72000
      python.d = no
      charts.d = no
      go.d = no
      node.d = no
      apps = no
      proc = no
      idlejitter = no
      diskspace = no
  stream: |
    [11111111-2222-3333-4444-555555555555]
      enabled = yes
      history = 3600
      default memory mode = dbengine
      health enabled by default = auto
      allow from = *

netdata-conf-child

data:
  go.d: |
    modules:
      pulsar: no
      prometheus: yes
  kubelet: |
    update_every: 1
    autodetection_retry: 0
    jobs:
      - url: http://127.0.0.1:10255/metrics
      - url: https://localhost:10250/metrics
        tls_skip_verify: yes
  kubeproxy: |
    update_every: 1
    autodetection_retry: 0
    jobs:
      - url: http://127.0.0.1:10249/metrics
  netdata: |
    [global]
      memory mode = ram
    [health]
      enabled = no
  stream: |
    [stream]
      enabled = yes
      destination = netdata:19999
      api key = 11111111-2222-3333-4444-555555555555
      timeout seconds = 60
      buffer size bytes = 1048576
      reconnect delay seconds = 5
      initial clock resync iterations = 60

netdata-child-sd-config-map

name: kubernetes
discovery:
  k8s:
    - tags: unknown
      role: pod
      local_mode: true
tag:
  - name: "Control-Plane"
    selector: unknown
    tags: -unknown control_plane
    match:
      - tags: kube_scheduler
        expr: '{{ glob .Image "k8s.gcr.io/kube-scheduler:*" }}'
      - tags: kube_controller_manager
        expr: '{{ glob .Image "k8s.gcr.io/kube-controller-manager:*" }}'
  - name: "Applications"
    selector: unknown
    tags: -unknown applications
    match:
      - tags: activemq
        expr: '{{ and (eq .Port "8161") (glob .Image "**/activemq*") }}'
      - tags: apache
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "httpd*" "**/httpd*") }}'
      - tags: bind
        expr: '{{ and (eq .Port "8653") (glob .Image "**/bind*") }}'
      - tags: cockroachdb
        expr: '{{ and (eq .Port "8080") (glob .Image "**/cockroach*") }}'
      - tags: consul
        expr: '{{ and (eq .Port "8500") (glob .Image "consul*" "**/consul*") }}'
      - tags: coredns
        expr: '{{ and (eq .Port "9153") (glob .Image "**/coredns*") }}'
      - tags: elasticsearch
        expr: '{{ and (eq .Port "9200") (glob .Image "elasticsearch:*" "**/elasticsearch:*") }}'
      - tags: fluentd
        expr: '{{ and (eq .Port "24220") (glob .Image "fluentd*" "**/fluentd*") }}'
      - tags: freeradius
        expr: '{{ and (eq .Port "18121") (glob .Image "**/freeradius*") }}'
      - tags: hdfs
        expr: '{{ and (eq .Port "50070") (glob .Image "**/hdfs*") }}'
      - tags: lighttpd
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "**/lighttpd*") }}'
      - tags: lighttpd2
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "**/lighttpd2*") }}'
      - tags: logstash
        expr: '{{ and (eq .Port "9600") (glob .Image "logstash*" "**/logstash*") }}'
      - tags: mysql
        expr: '{{ and (eq .Port "3306") (glob .Image "mysql*" "**/mysql*" "mariadb*" "**/mariadb*") }}'
      - tags: nginx
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "nginx*" "**/nginx*") }}'
      - tags: openvpn
        expr: '{{ and (eq .Port "7505") (glob .Image "**/openvpn") }}'
      - tags: phpfpm
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "**/phpfpm*" "**/php-fpm*") }}'
      - tags: rabbitmq
        expr: '{{ and (eq .Port "15672") (glob .Image "rabbitmq*" "**/rabbitmq*") }}'
      - tags: solr
        expr: '{{ and (eq .Port "8983") (glob .Image "solr*" "**/solr*") }}'
      - tags: tengine
        expr: '{{ and (eq .Port "80" "8080") (glob .Image "**/tengine*") }}'
      - tags: unbound
        expr: '{{ and (eq .Port "8953") (glob .Image "**/unbound*") }}'
      - tags: vernemq
        expr: '{{ and (eq .Port "8888") (glob .Image "**/vernemq*") }}'
      - tags: zookeeper
        expr: '{{ and (eq .Port "2181") (glob .Image "zookeeper*" "**/zookeeper*") }}'
  - name: "Prometheus Generic Applications"
    selector: unknown
    tags: -unknown prometheus_generic
    match:
      - tags: prometheus_generic
        expr: |
          {{ $scrapeOK := eq (get .Annotations "prometheus.io/scrape") "true" -}}
          {{ $portOK := eq (default .Port (get .Annotations "prometheus.io/port")) .Port -}}
          {{ $imageOK := not (glob .Image "netdata/netdata*" "**pulsar*" "**telegraf*") -}}
          {{ and $scrapeOK $portOK $imageOK }}
build:
  - name: "Control-Plane"
    selector: '!unknown control_plane'
    tags: file
    apply:
      - selector: kube_scheduler
        template: |
          - module: prometheus
            name: prometheus-{{.TUID}}
            url: http://{{.PodIP}}:{{default "10251" .Port}}/metrics
            app: '{{.ContName}}'
            update_every: 10
            max_time_series: 1000
      - selector: kube_controller_manager
        template: |
          - module: prometheus
            name: prometheus-{{.TUID}}
            url: http://{{.PodIP}}:{{default "10252" .Port}}/metrics
            app: '{{.ContName}}'
            update_every: 10
            max_time_series: 2000
  - name: "Prometheus Generic Applications"
    selector: '!unknown prometheus_generic'
    tags: file
    apply:
      - selector: prometheus_generic
        template: |
          {{ $path := default "/metrics" (get .Annotations "prometheus.io/path") -}}
          - module: prometheus
            name: prometheus-{{.TUID}}
            url: http://{{.Address}}{{$path}}
            app: '{{.ContName}}'
            update_every: 10
            max_time_series: 4000
  - name: "Applications"
    selector: '!unknown applications'
    tags: file
    apply:
      - selector: activemq
        template: |
          - module: activemq
            name: activemq-{{.TUID}}
            url: http://{{.Address}}
      - selector: apache
        template: |
          - module: apache
            name: apache-{{.TUID}}
            url: http://{{.Address}}/server-status?auto
      - selector: bind
        template: |
          - module: bind
            name: bind-{{.TUID}}
            url: http://{{.Address}}/json/v1
      - selector: cockroachdb
        template: |
          - module: cockroachdb
            name: cockroachdb-{{.TUID}}
            url: http://{{.Address}}/_status/vars
      - selector: consul
        template: |
          - module: consul
            name: consul-{{.TUID}}
            url: http://{{.Address}}
      - selector: coredns
        template: |
          - module: coredns
            name: coredns-{{.TUID}}
            url: http://{{.Address}}/metrics
      - selector: elasticsearch
        template: |
          - module: elasticsearch
            name: elasticsearch-{{.TUID}}
            url: http://{{.Address}}
      - selector: fluentd
        template: |
          - module: fluentd
            name: fluentd-{{.TUID}}
            url: http://{{.Address}}
      - selector: freeradius
        template: |
          - module: freeradius
            name: freeradius-{{.TUID}}
            address: {{.PodIP}}
            port: {{.Port}}
      - selector: hdfs
        template: |
          - module: hdfs
            name: hdfs-{{.TUID}}
            url: http://{{.Address}}/jmx
      - selector: lighttpd
        template: |
          - module: lighttpd
            name: lighttpd-{{.TUID}}
            url: http://{{.Address}}/server-status?auto
      - selector: lighttpd2
        template: |
          - module: lighttpd2
            name: lighttpd2-{{.TUID}}
            url: http://{{.Address}}/server-status?format=plain
      - selector: logstash
        template: |
          - module: logstash
            name: logstash-{{.TUID}}
            url: http://{{.Address}}
      - selector: mysql
        template: |
          - module: mysql
            name: mysql-{{.TUID}}
            dsn: 'netdata@tcp({{.Address}})/'
      - selector: nginx
        template: |
          - module: nginx
            name: nginx-{{.TUID}}
            url: http://{{.Address}}/stub_status
      - selector: openvpn
        template: |
          - module: openvpn
            name: openvpn-{{.TUID}}
            address: {{.Address}}
      - selector: phpfpm
        template: |
          - module: phpfpm
            name: phpfpm-{{.TUID}}
            url: http://{{.Address}}/status?full&json
      - selector: rabbitmq
        template: |
          - module: rabbitmq
            name: rabbitmq-{{.TUID}}
            url: http://{{.Address}}
      - selector: solr
        template: |
          - module: solr
            name: solr-{{.TUID}}
            url: http://{{.Address}}
      - selector: tengine
        template: |
          - module: tengine
            name: tengine-{{.TUID}}
            url: http://{{.Address}}/us
      - selector: unbound
        template: |
          - module: unbound
            name: unbound-{{.TUID}}
            address: {{.Address}}
            use_tls: false
      - selector: vernemq
        template: |
          - module: vernemq
            name: vernemq-{{.TUID}}
            url: http://{{.Address}}/metrics
      - selector: zookeeper
        template: |
          - module: zookeeper
            name: zookeeper-{{.TUID}}
            address: {{.Address}}
export:
  file:
    - selector: file
      filename: "/export/go.d.yml"

Answer 1 · 2022-05-17T14:48:08.000Z

Hi, @rerime. How do you set credentials for RabbitMQ container?

Answer 2 · 2022-05-17T15:08:24.000Z

Via definitions.json
example:

{
 "rabbit_version": "3.6.6",
 "users": [
  {
   "name": "user1",
   "password_hash": "pass1",
   "hashing_algorithm": "rabbit_password_hashing_sha256",
   "tags": ""
  },
  {
   "name": "adminuser",
   "password_hash": "adminpass",
   "hashing_algorithm": "rabbit_password_hashing_sha256",
   "tags": "administrator"
  }
 ],
 "vhosts": [
  {
   "name": "\/vhost1"
  },
  {
   "name": "\/vhost2"
  }
 ],
 "permissions": [
  {
   "user": "user1",
   "vhost": "\/vhost1",
   "configure": ".*",
   "write": ".*",
   "read": ".*"
  }
 ],
 "parameters": [],
 "policies": [],
 "queues": [],
 "exchanges": [],
 "bindings": []
}

Answer 3 · 2022-05-17T15:10:50.000Z

And this file is mounted to the rabbitmq container or do you pass some values via env variables?

Answer 4 · 2022-05-17T15:16:00.000Z

Only as file, I can pass as env also if it helps netdata

Answer 5 · 2022-05-17T15:20:59.000Z

Passing user/pass as env var will do. You can create a secret and set env vars from it. The service discovery thing collects all env vars keys/values and we can use them in netdata-child-sd-config-map.

Answer 6 · 2022-05-18T09:13:44.000Z

So I provide, but it didn't change anything.

      env:
        - name: RABBITMQ_DEFAULT_USER
          valueFrom:
            secretKeyRef:
              name: rabbitmq
              key: RABBITMQ_DEFAULT_USER
        - name: RABBITMQ_DEFAULT_PASS
          valueFrom:
            secretKeyRef:
              name: rabbitmq
              key: RABBITMQ_DEFAULT_PASS

Would be nice, if you add more information about service discovery mechanics
https://github.com/netdata/helmchart#configure-service-discovery

Answer 7 · 2022-05-19T10:16:09.000Z

@rerime yes, it doesn't work right now. We need to agree on env var names and I will update https://github.com/netdata/helmchart/blob/master/charts/netdata/sdconfig/child.yml

We need to agree on env var names

I think we need generic names we could use for all applications that support user/pass auth.

Answer 8 · 2022-05-19T10:51:59.000Z

@ilyam8 Thx for update!
I'm always create user netdata for server monitoring. May be add it and update docs.

Answer 9 · 2022-05-19T10:57:55.000Z

@rerime can you test #280?

NETDATA_SD_AUTH_USER env var value is used as username for HTTP basic auth.
NETDATA_SD_AUTH_PASS env var value is used as password for HTTP basic auth.

If it works I will update other jobs that support HTTP basic auth.

Answer 10 · 2022-05-19T11:29:36.000Z

@ilyam8
Works like a charm)
I've added:

        - name: NETDATA_SD_AUTH_USER
          valueFrom:
            secretKeyRef:
              name: rabbitmq
              key: RABBITMQ_DEFAULT_USER
        - name: NETDATA_SD_AUTH_PASS
          valueFrom:
            secretKeyRef:
              name: rabbitmq
              key: RABBITMQ_DEFAULT_PASS

Also tested that it is not broken if auth is not needed.

But still not very happy that netdata grub all secrets.

Answer 11 · 2022-05-19T11:44:09.000Z

But still not very happy that netdata grub all secrets.

Secrets are basically ConfigMaps with encoded values, using them to store sensitive info is not secure in general.

Using this method is an option, I think it is acceptable. If there are better ways we can add them.

Answer 12 · 2022-05-19T12:08:03.000Z

Works like a charm)

Great, I will:

update other collectors' jobs to use those variables.
update documentation.

Answer 13 · 2022-05-19T12:15:08.000Z

@ilyam8 Off topic... whats about redis module in sd config? Should I create another issue?

Answer 14 · 2022-05-19T12:38:00.000Z

Ahh, the config has no Redis. Actually nice things you mention the problem, I will add it and you will help with testing 😄

If we go with the current approach of identifying applications that is guessing by the image name... Can you share the image name of your Redis container?

Answer 15 · 2022-05-19T14:10:35.000Z

I've added:

containers:
    - name: redis
      image: artifactory.org/redis:6-alpine
      command:
        - redis-server
        - '--requirepass'
        - $(REDIS_PWD)
      ports:
        - name: redis
          containerPort: 6379
          protocol: TCP
      envFrom:
        - secretRef:
            name: redis
      env:
        - name: NETDATA_SD_AUTH_PASS
          valueFrom:
            secretKeyRef:
              name: redis
              key: REDIS_PWD

and in netdata-child-sd-config-map

      - tags: redis
        expr: '{{ and (eq .Port "6379") (glob .Image "redis*" "**/redis*") }}'
        
# ....

      - selector: redis
        template: |
          - module: redis
            name: redis-{{.TUID}}
            url: http://{{.Address}}
            password: "{{ get .Env "NETDATA_SD_AUTH_PASS" }}"

No success.

Seems, it accept password in url string
https://github.com/netdata/go.d.plugin/blob/master/config/go.d/redis.conf

Answer 16 · 2022-05-19T14:21:16.000Z

Try

          - module: redis
            name: redis-{{.TUID}}
            address: 'redis://:{{ get .Env "NETDATA_SD_AUTH_PASS" }}@{{.Address}}'

address string format is https://github.com/netdata/go.d.plugin/tree/master/modules/redis#configuration
service discovery available fields we can use is https://github.com/netdata/agent-service-discovery#pod-role

Answer 17 · 2022-05-19T15:01:36.000Z

Thx, it is working, but without single quotes ;)

      - selector: redis
        template: |
          - module: redis
            name: redis-{{.TUID}}
            address: redis://:{{ get .Env "NETDATA_SD_AUTH_PASS" }}@{{.Address}}

But for consistency, better

      - selector: redis
        template: |
          - module: redis
            name: redis-{{.TUID}}
            address: redis://{{ get .Env "NETDATA_SD_AUTH_USER" }}:{{ get .Env "NETDATA_SD_AUTH_PASS" }}@{{.Address}}

Answer 18 · 2022-05-19T15:16:51.000Z

@rerime thanks for help/testing, not really sure why it doesn't work with single quotes 🤔 If you want you can make a PR with Redis btw (after #280 is merged).

Answer 19 · 2022-05-20T13:43:12.000Z

@ilyam8 Will wait merge and make PR.
I guess #280 should be expanded to all http basic auth modules.

Answer 20 · 2022-05-20T15:36:12.000Z

I think all that support user/pass authentication - I gave a generic name to those env var on purpose.