Monitoring Ingress HTTP traffic from all namespaces.

Question

Monitoring Ingress HTTP traffic from all namespaces.

chefcook opened this issue 3 years ago · 2 comments

I found the following item on the Netdata documentation page.
https://learn.netdata.cloud/docs/agent/collectors/python.d.plugin/httpcheck/

After that I tried to list this plugin in my Values.yaml and start a helm upgrade.

helm upgrade -f values.yaml netdata netdata/netdata,
             --namespace monitoring,
             --set parent.nodeSelector."beta\.kubernetes\.io/os"=linux,
             --set child.nodeSelector."beta\.kubernetes\.io/os"=linux,
             --set parent.database.volumesize=4Gi,
             --set-file sd.child.configmap.from.value=./child.yml

Unfortunately without success. Do you have any suggestions how to achieve this goal ?

Eventuel also interesting to know. My service discovery looks like this.

Possibly also interesting to know. My service discovery looks like this.

> kubectl -n monitoring logs netdata-child-5zfjs -c sd

{"level":"info","component":"pipeline manager","time":"2021-06-15 22:18:54","message":"instance is started"}
{"level":"info","component":"k8s config provider","time":"2021-06-15 22:18:54","message":"instance is started"}
{"level":"info","component":"export manager","time":"2021-06-15 22:18:55","message":"registered: '[file exporter (/export/go.d.yml)]'"}
{"level":"info","component":"discovery manager","time":"2021-06-15 22:18:55","message":"registered: [k8s discovery manager]"}
{"level":"info","component":"pipeline manager","time":"2021-06-15 22:18:55","message":"received a new config, starting a new pipeline ('k8s/cmap/monitoring/netdata-child-sd-config-map:config.yml')"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"export manager","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"discovery manager","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"file export","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"k8s discovery manager","time":"2021-06-15 22:18:55","message":"registered: [k8s pod discovery]"}
{"level":"info","component":"k8s pod discovery","time":"2021-06-15 22:18:55","message":"instance is started"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"received '37' group(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/tenant-service-747c4f5577-9qdd9' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/event-service-66c6d8f5fd-dgmm7' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/operations-center-545c97df4c-5lxdf' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/inventory-service-74c66fc788-5grrj' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/event-service-76874db675-sfdbt' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-cni-networkmonitor-pwbnc' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/aux-service-6f8869d8c7-mqwnk' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/nginx-ingress-stg-ingress-nginx-controller-64fb8c84f8-fkpf7' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/mqtt-69d8ff6cb7-hm5fz' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/mqtt-5fdb4b8599-5cxk9' with 0 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/config-service-7cdbc9c5b4-kkjtv' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/gatekeeper-system/gatekeeper-audit-6dcf64cf84-ll62h' with 2 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/event-service-84bbdc796c-v7sfc' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/kube-proxy-szn2z' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/gatekeeper-system/gatekeeper-controller-6dd6b7f4c9-5swjf' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/aux-service-655b8bd78f-bqq7m' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/config-service-7fbb46876d-rmwtt' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/config-service-554578664b-r6r4z' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/mqtt-6fd7796798-rzjnw' with 3 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/tenant-service-79469ddb56-zm426' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/coredns-9d6c6c99b-79rnl' with 3 target(s)"}
{"level":"info","component":"build manager","time":"2021-06-15 22:19:00","message":"built 1 config(s) for target 'kube-system_coredns-9d6c6c99b-79rnl_coredns_tcp_9153'"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"group 'k8s/pod/kube-system/coredns-9d6c6c99b-79rnl': new/stale config(s) 1/0"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/operations-center-7f69589fd7-6k9qp' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-ip-masq-agent-bdrgd' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/monitoring/netdata-child-5zfjs' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/azure-policy-798f4cbf69-868hf' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/operations-center-6657f65596-zh6sb' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/nginx-ingress-dev-ingress-nginx-controller-5b665f8d9c-h8j8m' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/identity-service-6cbcb4665f-mbxhl' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/stg/inventory-service-66d6bc89b-5v5nz' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/internal/dev-service-7b7b6d4bf9-nnrlb' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/identity-service-85644d4f95-475qk' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/config-service-554578664b-8bjks' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/nginx-ingress-qa-ingress-nginx-controller-6dfb6b55c4-f6nk4' with 5 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/inventory-service-64bf4bd878-2696m' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/qa/operations-center-7f69589fd7-cznff' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/dev/identity-service-899796b88-992pn' with 1 target(s)"}
{"level":"info","component":"pipeline","time":"2021-06-15 22:19:00","message":"processing group 'k8s/pod/kube-system/omsagent-fn2tc' with 2 target(s)"}
{"level":"info","component":"file export","time":"2021-06-15 22:19:01","message":"wrote 1 config(s) to '/export/go.d.yml'"}

Thanks in advance for any help!

Answer 1 · 2021-06-28T11:44:32.000Z

Hi @chefcook

There is a brief description of how service discovery works:

parent instance has no service discovery.
children instances have a service-discovery sidecar that works in local mode; has role pod enabled => it supports discovering only the services that are running on the Node. ❗ If the resource/service is not local to the Node (e.g.: external database, ingress) we don't discover it (role pod + local mode doesn't suppose to discover them).

We plan to improve it:

add support for collecting data from external services, ingresses, etc.
make it much easier to enable any collector (e.g.: httpcheck).
improve service discovery related documentation.

Closing this issue in favor of https://community.netdata.cloud/t/monitoring-ingress-http-traffic-from-all-namespaces/1435/3, let's continue our discussion in there.

Answer 2 · 2021-06-28T11:50:57.000Z

This issue has been mentioned on the Netdata Community. There might be relevant details there:

https://community.netdata.cloud/t/monitoring-ingress-http-traffic-from-all-namespaces/1435/4