megaease/easeagent

health check, reflect different metrics collected by the agent, support different endpoint, like Prometheus

Closed this issue · 3 comments

akwei commented

Health check api should output response status code 200 for ready to start accepting traffic.

Kubernetes

The kubelet uses liveness probes to know when to restart a container, and uses readiness probes to know when a container is ready to start accepting traffic. Any code greater than or equal to 200 and less than 400 indicates success. Any other code indicates failure.
Reference: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

SpringBoot 2.3

It supports K8s Probes (/actuator/health/liveness, /actuator/health/readiness)

Reference: https://docs.spring.io/spring-boot/docs/2.3.9.RELEASE/reference/html/production-ready-features.html#production-ready-kubernetes-probes

Prometheus

It use static_configs to config pull api. The api should response status 200 for ready to start accepting traffic. Additionall the api should output metric info(Format as Prometheus Data Model)

Currentlly, agent use Jolokia for JMX over HTTP. It output json format, But Prometheus can not recognizes json.

akwei commented

Agent should provide two api for health check:

  • /healthy
  • /prometheus

Usually we can use /healthy。 /prometheus for Prometheus, because Prometheus data format is not json, we must process specificlly

/liveness, /readiness for k8s. Usually we can use /liveness, /readiness.

zhao-kun commented

Can we distinguish it with the HTTP standard header Accept ? If a client accept application/json we return JSON, if client need a graphite we returned as it wish

akwei commented

Good idea。
Prometheus was unable to recognize the response because Jolokia packed the result.
I'm trying to find a solution or replace 'Jolokia' with other's.