The extension monitors events and the state of Kubernetes or OpenShift clusters, records attributes of resources: pods, endpoints, daemon sets, replica sets, deployments and nodes. The data is received via Kubernetes API at a configurable interval and is sent to the AppDynamics Analytics Events API. Metrics are automatically created and stored under a desired Application Tier. Metrics can be viewed in the Metrics Browser under Application -> Metric Browser -> Application Infrastructure Performance -> Tier Name -> Custom Metrics -> Cluster Stats.
The extension aggregates metrics at the cluster level with further categorization by node and namespace.
If you want to monitor events only, use Kubernetes Events Extension. If you are interested in automated metric collection for events and other cluster resources, this extensions is self-sufficient. The event collection is enabled by default.
The extension automatically creates the dashboard below. The first attempt to create the dashboard is made after the number of seconds defined in the dashboardCheckInterval setting has passed since the first run of the extension.
Other specialized dashboards with the metrics collected can be built in AppDynamics -> Dashboards & Reports.
For each exposed metric, a named ADQL query is created by the extension along with the dashboard. Double-clicking on a dashboard widget will open the corresponding query. These automatically created queries can also be accessed under Analytics -> Searches. The search names are built in the following format:
<Cluster Name. Metric Name>
- This extension requires the Java Machine Agent.
- The AppDynamics platform needs the Events Service set up.
- REST API credentials. The account can be created under Administration -> Users. The user account must have rights to create dashboards and saved searches.
- You will need one or more Transaction Analytics/APM Peak licenses to consume the raw data. Viewing metrics and the dashboard does not require PEAK licenses.
- The number of collected metrics depends on the size of the cluster in terms of namespaces/projects and nodes. It may be necessary to increase the threshold for metric ingestion in the machine agent configuration:
-Dappdynamics.agent.maxMetrics=2000
The current metric collection rate is:
- Cluster-specific: 52
- Node-specific: 15
- Namespace-specific: 39
By default, the extension will collect the cluster-level metrics only. To monitor specific nodes or namespaces add the names of the desired nodes or namespaces
to the node and namespace arrays of the config.yml respectively. See specific instruction below in the Optional settings section.
If you request metric collection for some or all nodes or namespaces, make sure to adjust the maxMetric parameter accordingly.
Either Download the Extension from the latest Github release or Build from Source.
- Deploy the
KubernetesSnapshotExtension-<VERSION>.zipfile into the<machine agent home>/monitorsdirectory.
> unzip KubernetesSnapshotExtension-<VERSION>.zip -d <machine agent home>/monitors/
- Set up
config.yml.
Required settings
# Prefix for metrics. Replace <appTierName> with the value in appTierName setting.
metricPrefix: "Server|Component:<appTierName>|Custom Metrics|Cluster Stats|"
# Path to your kubectl client configuration. A typical location is "$HOME/.kube/config",
# but it may differ on your machine. Can be left blank, if apiMode is set to "cluster"
kubeClientConfig:
# Name of the application. It can be an existing application or a new application.
# All collected metrics will be associated with it
# The extension will first look for APPLICATION_NAME environmental variable.
# The application name in the machine agent configuration must match this value
appName: ""
# Name of the tier where metrics will be stored. The tier will be associated
# with the application configured earlier
# The extension will first look for TIER_NAME environmental variable.
# The tier name in the machine agent configuration must match this value.
appTierName: ""
# Events API Key obtained from AppDynamics --> Analytics --> Configuration API Keys --> Add
# The API Key you create needs to be able to Manage and Publish Custom Analytics Events
# EVENT_ACCESS_KEY environmental variable can be used to populate the field
eventsApiKey: ""
# Global Account Name obtained from
# AppDynamics --> Settings --> License --> Accounts --> Global Account Name
# **GLOBAL_ACCOUNT_NAME** environmental variable can be used to populate the field
accountName: ""
# REST API credentials. The account must have rights to login, create dashboards and saved searches
# REST_API_CREDENTIALS environmental variable can be used to populate the field
controllerAPIUser: ""
# Controller URL to access REST API
controllerUrl: "http://example.appdynamics.com/controller/"
# Absolute path to the json template for the default dashboard
dashboardTemplatePath: "<full path>/monitors/KubernetesSnapshotExtension/templates/k8s_dashboard_template.json"
Note: node name parameters -Dappdynamics.agent.tierName and -Dappdynamics.agent.nodeName must be defined for the machine agent startup, as shown in the example start-up script below.
When running MA with this extension in Kubernetes, do not deploy as Daemon Set. Make it a 1 replica Deployment. Set apiMode config variable to cluster. See detailed instructions below.
Optional settings:
# List of resources that will be monitored. Comment out individual items to exclude
#from monitoring
entities:
- type: "pod"
- type: "node"
- type: "deployment"
- type: "daemon"
- type: "replica"
- type: "event"
- type: "endpoint"
# Proxy host. Env Var: APPD_PROXY_HOST
proxyHost: ""
# Proxy port. Env Var: APPD_PROXY_PORT
proxyPort: ""
# Proxy user name. Env Var: APPD_PROXY_USER
proxyUser: ""
# Proxy password. Env Var: APPD_PROXY_PASS
proxyPass: ""
# list of nodes to collect metrics for. If all nodes need to be monitored, set name to "all"
nodes:
#- name:
# list of namespaces to collect metrics for. If all namespaces need to be monitored, set name to "all"
namespaces:
#- name:
# Dashboard name suffix
dashboardNameSuffix: "SUMMARY"
# Time in seconds between the checks if the default dashboard needs to be recreated
dashboardCheckInterval: "600"
# Number of records in a batch posted to AppD events API
batchSize: "100"
# Events Service Endpoint. These Default settings are for SaaS Users. Change if you are on Premise
eventsUrl: "https://analytics.api.appdynamics.com"
# Name of the Pod schema
podsSchemaName: "k8s_pod_snapshots"
# Name of the Node schema
podsSecuritySchemaName: "k8_node_snapshots"
# Name of the Deployment schema
deploySchemaName: "k8s_deploy_snapshots"
# Name of the Daemon Set schema
daemonSchemaName: "k8s_daemon_snapshots"
# Name of the Replica Set schema
rsSchemaName: "k8s_rs_snapshots"
# Name of the EndPoint schema
endpointSchemaName: "k8s_endpoint_snapshots"
# Name of the Events schema
eventsSchemaName: "k8s_events"
- Configure frequency of updates in monitor.xml:
<execution-style>periodic</execution-style>
<execution-frequency-in-seconds>200</execution-frequency-in-seconds>
- Restart the Machine Agent
A sample startup script for the machine agent with an elevated metrics threshold:
#!/bin/bash
SVM_PROPERTIES="-Dappdynamics.controller.hostName=${CONTROLLER_HOST}"
SVM_PROPERTIES+=" -Dappdynamics.controller.port=${CONTROLLER_PORT}"
SVM_PROPERTIES+=" -Dappdynamics.agent.applicationName=${APPLICATION_NAME}" # must match the appName configuration value
SVM_PROPERTIES+=" -Dappdynamics.agent.tierName=${TIER_NAME}" # must must match the appTierName configuration value
SVM_PROPERTIES+=" -Dappdynamics.agent.nodeName=${TIER_NAME}_node1" # must be defined for the metrics to be accepted
SVM_PROPERTIES+=" -Dappdynamics.agent.accountName=${ACCOUNT_NAME}"
SVM_PROPERTIES+=" -Dappdynamics.agent.accountAccessKey=${ACCOUNT_ACCESS_KEY}"
SVM_PROPERTIES+=" -Dappdynamics.agent.uniqueHostId=Master-${APPLICATION_NAME}"
SVM_PROPERTIES+=" -Dappdynamics.controller.ssl.enabled=true"
SVM_PROPERTIES+=" -Dappdynamics.sim.enabled=true"
SVM_PROPERTIES+=" -Dappdynamics.agent.maxMetrics=2000"
#SVM_PROPERTIES+=" -Dmetric.http.listener=true"
#SVM_PROPERTIES+=" -Dmetric.http.listener.host=0.0.0.0"
./bin/machine-agent ${SVM_PROPERTIES} -d -p ./pid.txt
- Stop the machine agent
- Delete the existing default dashboard and the automatically generated searches in Analytics. They will be re-created by the extension.
- Deploy the new version of the extension as described in the Installation section.
- Clone this repository
- Run
mvn -DskipTests clean install - The
KubernetesSnapshotExtension-<VERSION>.zipfile can be found in thetargetdirectory
| Directory/File | Description |
|---|---|
| src/main/resources/conf | Contains monitor.xml and config.yml |
| src/main/resources/templates | Contains the default dashboard template k8s_dashboard_template.json |
| src/main/java | Contains source code for the Kubernetes Snapshot extension |
| src/test/java | Contains test code for the Kubernetes Snapshot extension |
| target | Only obtained when using maven. Run 'maven clean install' to get the distributable .zip file. |
| pom.xml | maven build script to package the project (required only if changing Java code) |
The extension bundled with the machine agent can run in the cluster as a deployment.
- Download the desired version of the machine agent from The official downloads site
- Unzip the contents of the archive in the deployment/artifacts folder so that the artifacts folder becomes the machine agent root directory
- Download the Extension from the latest Github release
- Unzip its contents into the artifacts/monitors folder.
The resulting directory structure will look as follows artifacts
...
machineagent.jar
...
monitors
KubernetesSnapshotExtension
config.yml
monitor.yml
templates
k8s_dashboard_template.json
- Navigate to deployment directory and run the command below to create the image. Provide the desired image name and tag
docker build -t <image-name> .
- Create a secret for the necessary access keys.
- Obtain controller access key License - Account - Access key
- Obtain events API key AppDynamics --> Analytics --> Configuration API Keys --> Add
The API Key needs to be able to Manage and Publish Custom Analytics Events
- Create a new user for rest API access Administration - Users - New When creating the account for rest API access, you can setup a role with the following permissions and assign the user to it
- Run the following command to create a secret
oc create secret generic appd-secret --from-literal=ACCOUNT_ACCESS_KEY=<controller access key> --from-literal=EVENT_ACCESS_KEY=<event api key> --from-literal=REST_API_CREDENTIALS=<username@accountname:password>
- Open ma-config.yaml and update the settings with the information specific to your controller and save
- Open ma-sa-rbac.yaml and update ClusterRoleBinding subject namespace with the namespace where k8s-monitor-sa account was created
- Open machine-agent.yaml and update the image reference
- Run the deployment
kubectl create -f specs/
Always feel free to fork and contribute any changes directly via GitHub.
-
Verify Machine Agent Data: Please start the Machine Agent without the extension and make sure that it reports data. Verify that the machine agent status is UP and it is reporting Hardware Metrics.
-
config.yml: Validate the file here
-
Check Logs: There could be some obvious errors in the machine agent logs. Please take a look.
-
The config cannot be nullerror. This usually happens when on a windows machine in monitor.xml you give config.yaml file path with linux file path separator /. Use Windows file path separator ** e.g. monitors\Monitor\config.yaml. For Windows, please specify the complete path -
Collect Debug Logs: Edit the file, /conf/logging/log4j.xml and update the level of the appender com.appdynamics and com.singularity to debug. Let it run for 5-10 minutes and attach the logs to a support ticket.
Support is provided by the author. Please post your inquiries and bug reports in the Issues area.