Cannot get APM server to work on ECK
ColtonClark opened this issue · 1 comments
i start with
kubectl create -f https://download.elastic.co/downloads/eck/2.11.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.11.0/operator.yaml
then apply my elastic, kibana, fleet and agent manifest
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server-quickstart
namespace: default
spec:
version: 8.12.0
kibanaRef:
name: kibana-quickstart
elasticsearchRefs:
- name: elasticsearch-quickstart
mode: fleet
fleetServerEnabled: true
policyID: eck-fleet-server
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent-quickstart
namespace: default
spec:
version: 8.12.0
kibanaRef:
name: kibana-quickstart
fleetServerRef:
name: fleet-server-quickstart
mode: fleet
policyID: eck-agent
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
volumes:
- name: agent-data
emptyDir: {}
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-quickstart
namespace: default
spec:
version: 8.12.0
count: 1
elasticsearchRef:
name: elasticsearch-quickstart
config:
xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-quickstart-es-http.default.svc:9200"]
xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.default.svc:8220"]
xpack.fleet.packages:
- name: system
version: latest
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
xpack.fleet.agentPolicies:
- name: Fleet Server on ECK policy
id: eck-fleet-server
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: fleet_server-1
id: fleet_server-1
package:
name: fleet_server
- name: Elastic Agent on ECK policy
id: eck-agent
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: system-1
id: system-1
package:
name: system
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-quickstart
namespace: default
spec:
version: 8.12.0
nodeSets:
- name: default
count: 3
config:
node.store.allow_mmap: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- pods
- nodes
- namespaces
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
- apiGroups: ["apps"]
resources:
- replicasets
verbs:
- list
- watch
- apiGroups: ["batch"]
resources:
- jobs
verbs:
- list
- watch
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: default
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
and whatever i seem to do after this i cannot get the apm server or some of the other integrations to work. I tried adding the kubernetes integration and it said the agent was installed but failed to confirm incoming data. sometimes it says the APM server says working but i can never access the metrics. ive tried applying an apm yaml. as well as just adding the integration in kibana.
also getting this error when i try to launch the java apm agent
2024-01-26 15:06:41,848 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.AbstractIntakeApiHandler - Error trying to connect to APM Server at http://localhost:8200/intake/v2/events. Although not necessarily related to SSL, some related SSL configurations corresponding the current connection are logged at INFO level.
2024-01-26 15:06:41,849 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICSET_JSON_WRITER with this error: Connection refused: no further information
2024-01-26 15:06:41,850 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.AbstractIntakeApiHandler - Backing off for 0 seconds (+/-10%)
Hi @ColtonClark, I think that this should this be moved to the elastic/cloud-on-k8s repo, as it looks like an issue with k8s, not fleet-server or the apm-server