AKS Periscope

Quick troubleshooting for your Azure Kubernetes Service (AKS) cluster.

Overview

Hopefully most of the time, your AKS cluster is running happily and healthy. However, when things do go wrong, AKS customers need a tool to help them diagnose and collect the logs necessary to troubleshoot the issue. It can be difficult to collect the appropriate node and pod logs to figure what's wrong, how to fix the problem, or even to pass on those logs to others to help.

AKS Periscope allows AKS customers to run initial diagnostics and collect and export the logs (like into an Azure Blob storage account) to help them analyze and identify potential problems or easily share the information to support to help with the troubleshooting process with a simple az aks kollect command. These cluster issues are many times are caused by wrong configuration of their environment, such as networking or permission issues. This tool will allow AKS customers to run initial diagnostics and collect logs and custom analyses that helps them identify the underlying problems.

Raw Logs and metrics from AKS cluster are collected and basic diagnostic signals are generated by analyzing these raw data. More advanced diagnostic signals can be further generated by analyzing multiple basic diagnostic signals.

Data Privacy and Collection

AKS Periscope runs on customer's agent pool nodes, and collect VM and container level data. It is important that the customer is aware and gives consent before the tool is deployed/information shared. Microsoft guidelines can be found in the link below:

https://azure.microsoft.com/en-us/support/legal/support-diagnostic-information-collection/

Compatibility

AKS Periscope currently only work on Linux based agent nodes. Please use the script https://github.com/Azure/aks-engine/blob/master/scripts/collect-windows-logs.ps1 for Windows based agent nodes.

Current Feature Set

It collects the following logs and metrics:

Container logs (by default all containers in the kube-system namespace, can be config to take other namespace/containers).
Docker and Kubelet system service logs.
Network outbound connectivity, include checks for internet, API server, Tunnel, Azure Container Registry and Microsoft Container Registry.
Node IP tables.
All node level logs (by default cluster provision log and cloud init log, can be config to take other logs).
VM and Kubernetes cluster level DNS settings.
Describe Kubernetes objects (by default all pods/services/deployments in the kube-system namespace, can be config to take other namespace/objects).
Kubelet command arguments.
System performance (kubectl top nodes and kubectl top pods).

It also generates the following diagnostic signals:

Network outbound connectivity, reports the down period for a specific connection.
Network configuration, includes Network Plugin, DNS, and Max Pods per Node settings.

User Guide

AKS Periscope can be deployed by using Azure Command-Line tool (CLI). The steps are:

If CLI extension aks-preview has been installed previously, uninstall it first.
```
az extension remove --name aks-preview
```
Install CLI extension aks-preview.
```
az extension add --name aks-preview
```
Run az aks kollect command to collect metrics and diagnostic information, and upload to an Azure storage account. Use az aks kollect -h to check command details. Some useful examples are also listed below:
1. Using storage account name and a shared access signature token with write permission
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster \
--storage-account MyStorageAccount \
--sas-token "MySasToken"
```
2. Using the resource id of a storage account resource you own.
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster \
--storage-account "MyStorageAccountResourceId"
```
3. Using a pre-setup storage account in diagnostics settings for your managed cluster.
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster
```
4. Customize the container logs to collect. Its value can be either all containers in a namespace, for example, kube-system, or a specific container in a namespace, for example, kube-system/tunnelfront.
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster \
--container-logs "mynamespace1/mypod1 myns2"
```
5. Customize the kubernetes objects to collect. Its value can be either all objects of a type in a namespace, for example, kube-system/pod, or a specific object of a type in a namespace, for example, kube-system/deployment/tunnelfront.
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster \
--kube-objects "mynamespace1/service myns2/deployment/deployment1"
```
6. Customize the node log files to collect.
```
az aks kollect \
-g MyResourceGroup \
-n MyManagedCluster \
--node-logs "/var/log/azure-vnet.log /var/log/azure-vnet-ipam.log"
```

All collected logs, metrics and node level diagnostic information is stored on host nodes under directory:

/var/log/aks-periscope.

This directory is also mounted to container as:

/aks-periscope.

After export, they will also be stored in Azure Blob Service under a container with its name equals to cluster API server FQDN. A zip file is also created for easy download.

Alternatively, AKS Periscope can be deployed directly with kubectl. See instructions in Appendix.

Programming Guide

To locally build this project from the root of this repository:

CGO_ENABLED=0 GOOS=linux go build -mod=vendor github.com/Azure/aks-periscope/cmd/aks-periscope

Tip: In order to test local changes, user can build the local image via Dockerfile and then push it to your local hub. This way, user should be able to reference this test image in the deployment\aks-periscope.yaml containers property image attribute reference to your published test docker image.

For example:

docker build -f ./builder/Dockerfile -t <some_docker_repo_name>/<aks-periscope-user-selected-test-name> .
docker push <some_docker_repo_name>/<aks-periscope-user-selected-test-name>

Dependent Consuming Tools and Working Contract

az-cli and vscode both consume the aks-periscope.yaml file. If the aks-periscope.yaml file is changed, you will introduce breaking changes to az-cli and vscode.

Debugging Guide

This section intend to add some tip for debugging logs for the aks-periscope. This will help beginner to locally debug the pod logs.

Scenario, where user A uses expired sas-token and converts into base64 to be used in the deployment file.

In the scenario above, the kubectl apply -f deployment-file.yaml will show no error but the output which will look like the one below.

❯ kubectl apply -f deployment/aks-periscope.yaml
namespace/aks-periscope created
serviceaccount/aks-periscope-service-account created
clusterrole.rbac.authorization.k8s.io/aks-periscope-role unchanged
clusterrolebinding.rbac.authorization.k8s.io/aks-periscope-role-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/aks-periscope-role-binding-view unchanged
daemonset.apps/aks-periscope created
secret/azureblob-secret created
configmap/containerlogs-config created
configmap/kubeobjects-config created
configmap/nodelogs-config created
customresourcedefinition.apiextensions.k8s.io/diagnostics.aks-periscope.azure.github.com unchanged

In order to debug the logs of pods of the aks-periscope namespace deployed in users cluster.

The following command will come handy:

To get the pods in aks-periscope namespace.
- kubectl get pods -n aks-periscope
To check the logs in of the each deployed pod, this command will come handy:
- kubectl logs <name-of-pod> -n aks-periscope

Feel free to contact aksperiscope@microsoft.com or open an issue with any feedback or questions about AKS Periscope. This is currently a work in progress, but look out for more capabilities to come!

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

happy2048/aks-periscope