must-gather is a tool built on top of OpenShift must-gather
that provides the scripts for an OpenStack control plane logs and data collection.
oc adm must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gatherThe command above will create a local directory where logs, configs and status of the OpenStack control plane services are dumped.
In particular the openstack-must-gather will get a dump of:
- Service logs: Retrieved by the output of the pods (and operators) associated to the deployed services
- Services config: Retrieved for each component by the deployed
ConfigMapsandSecrets - Status of the services deployed in the OpenStack control plane
- Deployed CRs and CRDs
CSVs,pkgmanifests,subscriptions,installplans,operatorgroupPods,Deployments,Statefulsets,ReplicaSets,Service,Routes,ConfigMaps, (part of / relevant)Secrets- Network related info (
Metallbinfo,IPAddressPool,L2Advertisements,NetConfig,IPSet) - SOS reports for OpenShift nodes that are running OpenStack service pods.
Some openstack-must-gather collectors can be configured via environmental
variables to behave differently. For example SOS gathering can be disabled
passing an empty SOS_SERVICES environmental variable.
To provide environmental variables we'll need to invoke the gathering command manually like this:
oc adm must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather -- SOS_SERVICES= gatherThis is the list of available environmental variables:
CONCURRENCY: Must gather runs many operations, so to speed things up we run them in parallel with a concurrency of 5 by default. Users can change this environmental variable to adjust to its needs.SOS_SERVICES: Comma separated list of services to gather SOS reports from. Empty string skips sos report gathering. Eg:cinder,glance. Defaults to all of them.SOS_ONLY_PLUGINS: List of SOS report plugins to use. Empty string to run them all. Defaults to:block,cifs,crio,devicemapper,devices,iscsi,lvm2, memory,multipath,nfs,nis,nvme,podman,process,processor,selinux,scsi,udev.SOS_EDPM: Comma separated list of edpm nodes to gather SOS reports from, empty string skips sos report gathering. Accepts keyword all to gather all nodes. eg:edpm-compute-0,edpm-compute-1SOS_EDPM_PROFILES: List of sos report profiles to use. Empty string to run them all. Defaults to:system,storage,virtSOS_EDPM_PLUGINS: List of sos report plugins to use. This is optional.OPENSTACK_DATABASES: comma separated list of OpenStack databases that should be dumped. It is possible to set it toALLand dump all databases. By default this env var is unset, hence the database dump is skipped.ADDITIONAL_NAMESPACES: comma separated list of additional namespaces where we want to gather the associated resources.DO_NOT_MASK: This is an option for CI only purposes. It's set to 0 by default (and preserves the default behavior required in a production environment). However, if set to 1, it dumps secrets and services config files without masking sensitive data.
openstack-must-gather is capable of getting both the kubernetes resources
defined in the collection-scripts, and the sos-reports associated with both the
CoreOS nodes and the EDPM ones.
When the openstack-must-gather execution ends, a directory containing all the
gathered resources is generated, and in general it contains:
-
Global resources: useful to get some context about the status of the openshift cluster and the openstack deployed resources. These resources include
crds,apiservices,csvs,packagemanifests,webhooksandnetworkrelated informations likenncp,nnce,IPAddressPool, and so forth -
Namespaced resources: critical to get the status of the
OpenStackcluster and troubleshoot any problematic situation -
sos-reports: gathered from both the
CoreOSnodes that are part of theOpenShiftcluster, and theEDPMnodes in case are part of the cluster; the information to connect to the EDPM nodes is retrieved by theOpenStackDataplaneNodeSetsCR, and the resulting sos-report is retrieved from the remote nodes and downloaded in the current must-gather directory -
OpenStack Ctlplane Services: commands run through the
openstack-clito check the relevant resources generated within the OpenStack cluster (endpoint list,networks,subnets,registered services, etc)
A generic output of the openstack-must-gather execution looks like the
following:
+-----------------------------------+
| . | +-----------------------------+
| ├── apiservices | | ctlplane/neutron/ |
| ├── crd | | ├── agent_list |
| ├── csv | (control plane resources) | ├── extension_list |
| ├── ctlplane |------------------------------------| ├── floating_ip_list |
| │ ├── neutron | | ├── network_list |
| │ ├── nova |----------------- | ├── port_list |
| │ └── placement | | | ├── router_list |
| ├── dbs | +---------------------------+ | ├── security_group_list |
| ├── namespaces | | namespaces/openstack/ | | └── subnet_list |
| │ ├── cert-manager | | ├── all_resources.log | +-----------------------------+
| │ ├── openshift-machine-api | | ├── buildconfig |-----------------------------------
| │ ├── openshift-nmstate | | ├── configmaps | |
| │ ├── openstack | | ├── cronjobs | +--------------------------------------------------------------------+
| │ └── openstack-operators | | ├── crs | | namespaces/openstack/secrets/glance/ |
| ├── network | | ├── daemonset | | ├── cert-glance-default-public-route.yaml |
| │ ├── ipaddresspools | | ├── deployments | | ├── glance-config-data.yaml |
| │ ├── nnce | | ├── events.log | | ├── glance-config-data.yaml-00-config.conf |
| │ └── nncp | | ├── installplans | | ├── glance-default-single-config-data.yaml |
| ├── nodes | | ├── jobs | | ├── glance-default-single-config-data.yaml-00-config.conf |
| ├── sos-reports | | ├── nad.log | | ├── glance-default-single-config-data.yaml-10-glance-httpd.conf |
| │ ├── _all_nodes | | ├── pods | | ├── glance-default-single-config-data.yaml-httpd.conf |
| │ ├── barbican | | ├── pvc.log | | ├── glance-default-single-config-data.yaml-ssl.conf |
| │ ├── ceilometer | | ├── replicaset | | └── glance-scripts.yaml |
| │ ├── glance | | ├── routes | +--------------------------------------------------------------------+
| │ ├── keystone | | ├── secrets | |
| │ ├── neutron | | ├── services | +--------------------------------------------------------------------+
| │ ├── nova | | ├── statefulsets | | Note: if DO_NOT_MASK is passed in CI, secrets are dumped without |
| │ ├── ovn | | └── subscriptions | | hiding any sensitive information. |
| │ ├── ovs | +---------------------------+ +--------------------------------------------------------------------+
| │ ├── placement |
| │ └── swift |
| └── webhooks |
| ├── mutating |
| └── validating |
+-----------------------------------+In a troubleshooting session, however, it's critical to check and analyze not
only Secrets and services config files, but also the CRs associated with
each service and the Pod logs.
These are still namespaced resources, and they can be found in the CRs and Pods
directories.
Other than that, for each namespace some generic informations is collected. In
particular the openstack-must-gather tool is able to retrieve:
Eventsrecorded for the current namespaceNetwork Attachment DefinitionsPVCsattached to the deployed Pods- A picture of the namespaces in terms of deployed resources (
all_resources.log)
+---------------------------+
| namespaces/openstack/ | ------------------------------------
| ├── buildconfig | |
| ├── cronjobs | +--------------------------------------------------------+
| ├── crs | | namespaces/openstack/crs/ |
| ├── daemonset | | ├── barbicanapis.barbican.openstack.org |
| ├── deployments | | ├── barbicankeystonelisteners.barbican.openstack.org |
| ├── events.log | | ├── barbicans.barbican.openstack.org |
| ├── installplans | | ├── barbicanworkers.barbican.openstack.org |
| ├── jobs | | ... |
| ├── nad.log | | ... |
| ├── pods | | ├── glanceapis.glance.openstack.org |
| ├── all_resources.log | | └── glance-default-single.yaml |
| ├── configmaps | | ├── glances.glance.openstack.org |
| ├── pvc.log | | └── glance.yaml |
| ├── replicaset | | ├── keystoneapis.keystone.openstack.org |
| ├── routes | | ├── keystoneendpoints.keystone.openstack.org |
| ├── secrets | | ├── keystoneservices.keystone.openstack.org |
| ├── services | | ... |
| ├── statefulsets | | ├── telemetries.telemetry.openstack.org |
| └── subscriptions | | └── transporturls.rabbitmq.openstack.org |
+---------------------------+ +--------------------------------------------------------+As depicted in the schema above, the same pattern applies to the Pod resources.
For each Pod the openstack-must-gather tool is able to retrieve the
description and the associated logs (including -previous in case the Pod is
in a CrashLookBackoff status).
+---------------------------+
| namespaces/openstack/ | ------------------------------------
| ├── buildconfig | |
| ├── cronjobs | +-----------------------------------------------------------+
| ├── crs | | namespaces/openstack/pods/glance-dbpurge-28500481-f4jk9 |
| ├── daemonset | | ├── glance-dbpurge-28500481-f4jk9-describe |
| ├── deployments | | └── logs |
| ├── events.log | | └── glance-dbpurge.log |
| ├── installplans | | namespaces/openstack/pods/glance-default-single-0 |
| ├── jobs | | ├── glance-default-single-0-describe |
| ├── nad.log | | └── logs |
| ├── pods | | ├── glance-api.log |
| ├── all_resources.log | | ├── glance-httpd.log |
| ├── configmaps | | └── glance-log.log |
| ├── pvc.log | | namespaces/openstack/pods/glance-default-single-1 |
| ├── replicaset | | ├── glance-default-single-1-describe |
| ├── routes | | └── logs |
| ├── secrets | | ├── glance-api.log |
| ├── services | | ├── glance-httpd.log |
| ├── statefulsets | | └── glance-log.log |
| └── subscriptions | | namespaces/openstack/pods/glance-default-single-2 |
+---------------------------+ | ├── glance-default-single-2-describe |
| └── logs |
| ├── glance-api.log |
| ├── glance-httpd.log |
| └── glance-log.log |
+-----------------------------------------------------------+You can build the image locally using the Dockerfile included.
A Makefile is also provided. To use it, you must pass:
- an image name using the variable
MUST_GATHER_IMAGE. - an image registry using the variable
IMAGE_REGISTRY(default is quay.io/openstack-k8s-operators) - an image tag using the variable
IMAGE_TAG(default islatest)
The targets for make are as follows:
check-image: Check if theMUST_GATHER_IMAGEvariable is setbuild: build the image with the supplied name and pushes itcheck: Run sanity check against the script collectionpytest: Run sanity check and unit tests against the python script collectionpodman-build: builds the must-gather imagepodman-push: pushes an already-builtmust-gatherimage
One possible workflow that can be used for development is to run the openstack must-gather in debug mode:
$ oc adm must-gather --image=quay.io/openstack-k8s-operators/openstack-must-gather -- gather_debug
[must-gather ] OUT Using must-gather plug-in image: quay.io/openstack-k8s-operators/openstack-must-gather:latest
When opening a support case, bugzilla, or issue please include the following summary data along with any other requested information:
ClusterID: 6ffe21e8-7dc9-4719-926f-f34ec33e6916
ClusterVersion: Stable at "4.12.35"
ClusterOperators:
All healthy and stable
[must-gather ] OUT namespace/openshift-must-gather-b9fpw created
[must-gather ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-mq6st created
[must-gather ] OUT pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created
[must-gather-mxc7k] POD 2023-09-29T09:56:24.995284210Z Must gather entering debug mode, will sleep until file /tmp/rm-to-finish-gathering is deleted
[must-gather-mxc7k] POD 2023-09-29T09:56:24.995284210ZRunning in debug mode makes the must gather container just sit waiting for a
file to be removed, allowing us to go into the container and test our scripts.
In the above case, where the namespace is openshift-must-gather-b9fpw and the
pod name is must-gather-mxc7k we would enter the container with:
oc -n openshift-must-gather-b9fpw rsh must-gather-mxc7kAnd then if we were debugging a bash script called gather_trigger_gmr, we
would run it in debug mode:
sh-5.1# bash -x /usr/bin/gather_trigger_gmrAnd once we had the script working as intended we would copy the file from a terminal:
oc cp openshift-must-gather-b9fpw/must-gather-mxc7k:usr/bin/gather_trigger_gmr collection-scripts/gather_trigger_gmrAnd finally, from inside the container shell we let the oc adm must-gather
command complete, optionally running everything to ensure we haven't
inadvertently broken anything in the process:
sh-5.1# gather
sh-5.1# rm /tmp/rm-to-finish-gatheringBesides the generic OpenShift etcd objects that must-gather scripts are currently gathering there's also component specific code that gather other information, so when adding or improving a component we should look into:
-
collection-scripts/common.sh: Services are in theOSP_SERVICESvariable to indicate that the script must gather its config maps, secrets, additional information, trigger Guru Mediation Reports, etc. -
collection-scripts/gather_services_status: Runs OpenStack commands to gather additional information. For example for Cinder it gather state of the services, volume types, qos specs, volume transfer requests, summary of existing volumes, pool information, etc. -
collection-scripts/gather_trigger_gmr: Triggers Guru Meditation Reports on the services so they are present in the logs when those are gathered afterwards.
(optional) Build and push the must-gather image to a registry:
git clone ssh://git@github.com/openstack-k8s-operators/openstack-must-gather.git
cd openstack-must-gather
IMAGE_TAG=<tag> IMAGE_REGISTRY=<registry> MUST_GATHER_IMAGE=openstack-must-gather make buildOn a machine where you have oc adm access, do the following:
oc adm must-gather --image=<registry>/openstack-must-gather:<tag>When generation is finished, you will find the dump in the current directory
under must-gather.local.XXXXXXXXXX.
It is possible to gather debugging information about specific features by adding
to the oc adm must-gather command the --image-stream argument.
The must-gather tool supports multiple images, so you can gather data about more
than one feature by running a single command.
Adding --image-stream assumes that an image has been imported in an OpenShift
namespace and the associated imagestream is available.
To import an image as an imagestream, run the following commands:
oc project <namespace>
oc import-image <registry>/<image>:<tag> --confirmand double check the imagestream exists in the specified namespace.
At this point, run the oc adm must-gather command passing one or more
imagestream parameters.
For instance, assuming we import the kubevirt must-gather image:
within the existing openstack namespace:
oc project openstack
oc import-image kubevirt-must-gather --from=quay.io/kubevirt/must-gather:latest --confirmwe can combine, in a single command, three different containers executions that gather different aspects of the same environement.
oc adm must-gather --image-stream=openstack/kubevirt-must-gather \
--image-stream=openshift/must-gather \
--image=quay.io/openstack-k8s-operators/openstack-must-gather:latestThe command above will create three pods associated to the existing imagestream
objects that point to openstack, openshift and kubevirt must-gather container
images.
[must-gather] OUT pod for plug-in image quay.io/openstack-k8s-operators/openstack-must-gather:latest created
[must-gather] OUT pod for plug-in image quay.io/kubevirt/must-gather@sha256:501e30ac7d5b9840a918bb9e5aa830686288ccfeee37d70aaf99cd2e302a2bb0 created
[must-gather] OUT pod for plug-in image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e9601b492cbb375f0a05310efa6025691f8bba6a97667976cd4baf4adf0f244c created
...
...To gather config files and logs from hosts we need to rely on the sos tool (former sosreport).
We can obtain logs from both OCP cluster nodes and compute nodes.
First we need to gather cluster host names
$ oc get nodes -o name
node/master-0
node/master-1
node/master-2Then we need to login into a node via a debug container
$ oc debug node/master-0
Warning: would violate PodSecurity "restricted:v1.24": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), privileged (container "container-00" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (container "container-00" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container-00" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volume "host" uses restricted volume type "hostPath"), runAsNonRoot != true (pod or container "container-00" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "container-00" must not set runAsUser=0), seccompProfile (pod or container "container-00" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
Starting pod/master-0-debug-lpvnn ...
To use host binaries, run `chroot /host`
Pod IP: 192.168.111.20
If you don't see a command prompt, try pressing enter.
sh-4.4#We need to chroot into the node itself in order to have access to all commands
sh-4.4# chroot /host
sh-5.1#We need to spawn a toolbox container to use sos tool.
sh-5.1# toolbox
Trying to pull registry.redhat.io/rhel9/support-tools:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob bf237f774da8 done
Copying blob 4b36affe1d29 done
Copying config d184fda91f done
Writing manifest to image destination
Storing signatures
d184fda91f0aa5c6deed433f984bd393754f707a285445a83637aaa13b8b7e86
Spawning a container 'toolbox-root' with image 'registry.redhat.io/rhel9/support-tools'
Detected RUN label in the container image. Using that as the default...
57305547f44861347e453b535fd59a56b4d9c0a3f472b7f1d86c2f246c94a5ea
toolbox-root
Container started successfully. To exit, type 'exit'.
[root@master-0 /]#Finally we can collect all required config/logs
[root@master-0 /]# sos report -k crio.all=on -k crio.logs=on
<snip output>
Finished running plugins
Creating compressed archive...
Your sosreport has been generated and saved in:
/host/var/tmp/sosreport-master-0-2023-11-15-mvgdmxo.tar.xz
Size 82.68MiB
Owner root
sha256 81b356f0069b4bc35dc4ae016e2c25369c9cefa214c75973cde8b8470ffa4516
Please send this file to your support representative.
[root@master-0 /]#You can also add the flag --all-logs to the sos command to retrieve further configuration file and logs
For more information see the official OCP documentation here
First we need to gather compute node info from OpenstackDataPlaneNodeSet resources.
$ oc get openstackdataplanenodesets -o name
openstackdataplanenodeset.dataplane.openstack.org/openstack-edpm-ipam
$ oc get openstackdataplanenodeset.dataplane.openstack.org/openstack-edpm-ipam -o json | jq -r '.spec.nodes[] | [.hostName, .ansible.ansibleHost] | @csv'
"edpm-compute-0","192.168.122.100"
"edpm-compute-1","192.168.122.101"
"edpm-compute-2","192.168.122.102"Then you have to login into nodes via ssh. In the OpenstackDataplaneNodeSet resource you can find the user used by ansible and the private ssh, that we can extract from the secret.
# user
$ oc get openstackdataplanenodeset.dataplane.openstack.org/openstack-edpm-ipam -o json | jq .spec.nodeTemplate.ansible.ansibleUser
"zuul"
# private ssh key
$ oc get openstackdataplanenodeset.dataplane.openstack.org/openstack-edpm-ipam -o json |jq .spec.nodeTemplate.ansibleSSHPrivateKeySecret
"dataplane-ansible-ssh-private-key-secret"
# save private ssh key to a file and fix its permissions
$ oc get secret/dataplane-ansible-ssh-private-key-secret -o go-template='{{ index .data "ssh-privatekey" | base64decode }}' > ~/.ssh/compute.key
$ chmod 0600 ~/.ssh/compute.keyFinally login into compute node
$ ssh -i ~/.ssh/compute.key zuul@192.168.122.100
Register this system with Red Hat Insights: insights-client --register
Create an account or view all your systems at https://red.ht/insights-dashboard
Last login: Wed Nov 15 10:02:11 2023 from 192.168.111.1
[zuul@compute-0 ~]$And launch the sos command just like we did in the other example, but with different flags.
sudo sos report -p system,storage,virt,openstack_edpmAs you can see in the example above, there's a new profile for the sos tool, openstack_edpm, which enables the gathering from the plugins running on the node.
The output file /var/tmp/sosreport-$hostname-$date-$hash.tar.xz will contains all the required files from paths introduced in the new containerized services, like /var/lib/openstack/, /var/lib/edpm-config, /var/log/containers/ and so on.
If the image is pushed to quay.io registry, make sure it's set to public,
otherwise it can't be consumed by the must-gather tool.