This repository contains a very simple Kubernetes Operator that uses VMware's govmomi to return some simple virtual machine information through the status field of a Custom Resource (CR), which is called VMInfo. This will require us to extend Kubernetes with a new Custom Resource Definition (CRD). The code shown here is for education purposes only, showing one way in which a Kubernetes controller / operator can access the underlying vSphere infrastructure for the purposes of querying resources.
You can think of a CRD as representing the desired state of a Kubernetes object or Custom Resource, and the function of the operator is to run the logic or code to make that desired state happen - in other words the operator has the logic to do whatever is necessary to achieve the object's desired state.
This is the second in the series of Kubernetes Operators to query the status of vSphere resources. The first tutorial was built to query ESXi resources (called HostInfo). Details about that operator tutorial can be found here. Another operator tutorial was built to query information from First Class Disks (FCDs). FCDs are used to back Kubernetes Persistent Volumes when the PVs are provisioned by the vSphere CSI driver. You can find the details about the FCD operator tutorial here.
In this example, we will create a CRD called VMInfo. VMInfo will contain the name of an virtual machine in its specification, possibly a Kubernetes Node. When a Custom Resource (CR) is created and subsequently queried, we will call an operator (logic in a controller) whereby some details about the virtual machine will be returned via the status fields of the object through govmomi API calls.
The following will be created as part of this tutorial:
-
A Customer Resource Definition (CRD)
- Group:
Topology- Kind:
VMInfo - Version:
v1 - Specification will include a single item:
Spec.Nodename
- Kind:
- Group:
-
One or more VMInfo Custom Resource / Object will be created through yaml manifests, each manifest containing the nodename of a virtual machine that we wish to query. The fields which will be updated to contain the relevant information from the VM (when the CR is queried) are:
Status.GuestIdStatus.TotalCPUStatus.ResvdCPUStatus.TotalMemStatus.ResvdMemStatus.PowerStateStatus.IPAddressStatus.HWVersionStatus.PathToVM
-
An Operator (or business logic) to retrieve virtual machine information specified in the CR will be coded in the controller for this CR.
The assumption is that you already have a working Kubernetes cluster. Installation and deployment of a Kubernetes is outside the scope of this tutorial. If you do not have a Kubernetes cluster available, consider using Kubernetes in Docker (shortened to Kind) which uses containers as Kubernetes nodes. A quickstart guide can be found here:
The assumption is that you also have a VMware vSphere environment comprising of at least one ESXi hypervisor with at least one virtual machine which is managed by a vCenter server. While the thought process is that your Kubernetes cluster will be running on vSphere infrastructure, and thus this operator will help you examine how the underlying vSphere resources are being consumed by the Kubernetes clusters running on top, it is not necessary for this to be the case for the purposes of this tutorial. You can use this code to query any vSphere environment from Kubernetes.
If this sounds even too daunting at this stage, I strongly recommend checking out the excellent tutorial on CRDs from my colleague, Rafael Brito. His RockBand CRD tutorial uses some very simple concepts to explain how CRDs, CRs, Operators, spec and status fields work, and is a great way to get started on Kubernetes Operators.
You will need the following components pre-installed on your desktop or workstation before we can build the CRD and operator.
- A git client/command line
- Go (v1.15+) - earlier versions may work but I used v1.15.
- Docker Desktop
- Kubebuilder
- Kustomize
- Access to a Container Image Repositor (docker.io, quay.io, harbor)
- A make binary - used by Kubebuilder
If you are interested in learning more about Golang basics, which is the code used to create operators, I found this site very helpful.
The CRD is built using kubebuilder. I'm not going to spend a great deal of time talking about KubeBuilder. Suffice to say that KubeBuilder builds a directory structure containing all of the templates (or scaffolding) necessary for the creation of CRDs and controllers. Once this scaffolding is in place, this turorial will show you how to add your own specification fields and status fields, as well as how to add your own operator logic. In this example, our logic will login to vSphere, query and return virtual machine information via a Kubernetes CR / object / Kind called VMInfo, the values of which will be used to populate status fields in our CRs.
The following steps will create the scaffolding to get started.
mkdir vminfo
$ cd vminfoNext, define the Go module name of your CRD. In my case, I have called it vminfo. This creates a go.mod file with the name of the module and the Go version (v1.15 here).
$ go mod init vminfo
go: creating new go.mod: module vminfo$ ls
go.mod$ cat go.mod
module vminfo
go 1.15Now we can proceed with building out the rest of the directory structure. The following kubebuilder commands (init and create api) creates all the scaffolding necessary to build our CRD and operator. You may choose an alternate domain here if you wish. Simply make note of it as you will be referring to it later in the tutorial.
kubebuilder init --domain corinternal.comWe must now define a resource. To do that, we again use kubebuilder to create the resource, specifying the API group, its version and supported kind. My API group is called topology, my kind is called VMInfo and my initial version is v1.
kubebuilder create api \
--group topology \
--version v1 \
--kind VMInfo \
--resource=true \
--controller=trueThe operator scaffolding (directory structure) is now in place. The next step is to define the specification and status fields in our CRD. After that, we create the controller logic which will watch our Custom Resources, and bring them to desired state (called a reconcile operation). More on this shortly.
Customer Resource Definitions CRD are a way to extend Kubernetes through Custom Resources. We are going to extend a Kubernetes cluster with a new custom resource called VMInfo which will retrieve information about the virtual machine whose name is specified in a Custom Resource. Thus, I will need to create a field called nodename in the CRD - this defines the specification of the custom resource. We also add status fields, as these will be used to return information from the Virtual Machine.
This is done by modifying the api/v1/vminfo_types.go file. Here is the initial scaffolding / template provided by kubebuilder:
// VMInfoSpec defines the desired state of VMInfo
type VMInfoSpec struct {
// INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
// Important: Run "make" to regenerate code after modifying this file
// Foo is an example field of VMInfo. Edit VMInfo_types.go to remove/update
Foo string `json:"foo,omitempty"`
}
// VMInfoStatus defines the observed state of VMInfo
type VMInfoStatus struct {
// INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
// Important: Run "make" to regenerate code after modifying this file
}
// +kubebuilder:object:root=trueThis file is modified to include a single spec.nodename field and to return various status fields. There are also a number of kubebuilder fields added, which are used to do validation and other kubebuilder related functions. The shortname "ch" will be used later on in our controller logic. Also, when we query any Custom Resources created with the CRD, e.g. kubectl get vminfo, we want the output to display the nodename of the virtual machine.
Note that what we are doing here is for education purposes only. Typically what you would observe is that the spec and status fields would be similar, and it is the function of the controller to reconcile and differences between the two to achieve eventual consistency. But we are keeping things simple, as the purpose here is to show how vSphere can be queried from a Kubernetes Operator. Below is a snippet of the vminfo_types.go showing the code changes. The code-complete vminfo_types.go is here.
// VMInfoSpec defines the desired state of VMInfo
type VMInfoSpec struct {
Nodename string `json:"nodename"`
}
// VMInfoStatus defines the observed state of VMInfo
type VMInfoStatus struct {
GuestId string `json:"guestId"`
TotalCPU int64 `json:"totalCPU"`
ResvdCPU int64 `json:"resvdCPU"`
TotalMem int64 `json:"totalMem"`
ResvdMem int64 `json:"resvdMem"`
PowerState string `json:"powerState"`
HwVersion string `json:"hwVersion"`
IpAddress string `json:"ipAddress"`
PathToVM string `json:"pathToVM"`
}
// +kubebuilder:validation:Optional
// +kubebuilder:resource:shortName={"ch"}
// +kubebuilder:printcolumn:name="Nodename",type=string,JSONPath=`.spec.nodename`
// +kubebuilder:object:root=true
// +kubebuilder:subresource:statusWe are now ready to create the CRD. There is one final step however, and this involves updating the Makefile which kubebuilder has created for us. In the default Makefile created by kubebuilder, the following CRD_OPTIONS line appears:
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:trivialVersions=true"This CRD_OPTIONS entry should be changed to the following:
# Produce CRDs that work back to Kubernetes 1.11 (no version conversion)
CRD_OPTIONS ?= "crd:preserveUnknownFields=false,crdVersions=v1,trivialVersions=true"Now we can build our CRD with the spec and status fields that we have place in the api/v1/vminfo_types.go file.
make manifests && make generateThe CRD is not currently installed in the Kubernetes Cluster.
$ kubectl get crd
NAME CREATED AT
antreaagentinfos.clusterinformation.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
antreacontrollerinfos.clusterinformation.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
clusternetworkpolicies.security.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
traceflows.ops.antrea.tanzu.vmware.com 2020-11-18T17:14:03ZTo install the CRD, run the following make command:
make installNow check to see if the CRD is installed running the same command as before.
$ kubectl get crd
NAME CREATED AT
antreaagentinfos.clusterinformation.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
antreacontrollerinfos.clusterinformation.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
clusternetworkpolicies.security.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
traceflows.ops.antrea.tanzu.vmware.com 2020-11-18T17:14:03Z
vminfoes.topology.corinternal.com 2021-01-18T11:25:20ZOur new CRD vminfoes.topology.corinternal.com is now visible. Another useful way to check if the CRD has successfully deployed is to use the following command against our API group. Remember back in step 2 we specified the domain as corinternal.com and the group as topology. Thus the command to query api-resources for this CRD is as follows:
$ kubectl api-resources --api-group=topology.corinternal.com
NAME SHORTNAMES APIGROUP NAMESPACED KIND
vminfoes ch topology.corinternal.com true VMInfoAt this point, we can do a quick test to see if our CRD is in fact working. To do that, we can create a manifest file with a Custom Resource that uses our CRD, and see if we can instantiate such an object (or custom resource) on our Kubernetes cluster. Fortunately kubebuilder provides us with a sample manifest that we can use for this. It can be found in config/samples.
$ cd config/samples
$ ls
topology_v1_vminfo.yaml$ cat topology_v1_vminfo.yaml
apiVersion: topology.corinternal.com/v1
kind: VMInfo
metadata:
name: vminfo-sample
spec:
# Add fields here
foo: barWe need to slightly modify this sample manifest so that the specification field matches what we added to our CRD. Note the spec: above where it states 'Add fields here'. We have removed the foo field and added a spec.nodename field, as per the api/v1/vminfo_types.go modification earlier. Thus, after a simple modification, the CR manifest looks like this, where tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298 is the name of the virtual machine that we wish to query. It is in fact a Tanzu Kubernetes worker node. It could be any virtual machine in your vSphere infrastructure.
$ cat topology_v1_vminfo.yaml
apiVersion: topology.corinternal.com/v1
kind: VMInfo
metadata:
name: tkg-worker-1
spec:
# Add fields here
nodename: tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298To see if it works, we need to create this VMInfo Custom Resource.
$ kubectl create -f topology_v1_vminfo.yaml
vminfo.topology.corinternal.com/tkg-worker-1 created$ kubectl get vminfo
NAME NODENAME
tkg-worker-1 tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298Note that the nodename field is also printed, as per the kubebuilder directive that we placed in the api/v1/vminfo_types.go. As a final test, we will display the CR in yaml format.
$ kubectl get vminfo -o yaml
apiVersion: v1
items:
- apiVersion: topology.corinternal.com/v1
kind: VMInfo
metadata:
creationTimestamp: "2021-01-18T12:20:45Z"
generation: 1
managedFields:
- apiVersion: topology.corinternal.com/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:nodename: {}
manager: kubectl
operation: Update
time: "2021-01-18T12:20:45Z"
- apiVersion: topology.corinternal.com/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:guestId: {}
f:hwVersion: {}
f:ipAddress: {}
f:pathToVM: {}
f:powerState: {}
f:resvdCPU: {}
f:resvdMem: {}
f:totalCPU: {}
f:totalMem: {}
manager: manager
operation: Update
time: "2021-01-18T12:20:46Z"
name: tkg-worker-1
namespace: default
resourceVersion: "28841720"
selfLink: /apis/topology.corinternal.com/v1/namespaces/default/vminfoes/tkg-worker-1
uid: 2c60b273-a866-4344-baf5-0b3b924b65a5
spec:
nodename: tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298
kind: List
metadata:
resourceVersion: ""
selfLink: ""This appears to be working as expected. However there are no Status fields displayed with our VM information in the yaml output above. To see this information, we need to implement our operator / controller logic to do this. The controller implements the desired business logic. In this controller, we first read the vCenter server credentials from a Kubernetes secret passed to the controller (which we will create shortly). We will then open a session to my vCenter server, and get a list of virtual machines that it manages. We will then look for the virtual machine that is specified in the spec.nodename field in the CR, and retrieve various information for this virtual machine. Finally we will update the appropriate status fields with this information, and we should be able to query it using the kubectl get vminfo -o yaml command seen previously.
Once all this business logic has been added in the controller, we will need to be able to run it in the Kubernetes cluster. To achieve this, we will build a container image to run the controller logic. This will be provisioned in the Kubernetes cluster using a Deployment manifest. The deployment contains a single Pod that runs the container (it is called manager). The deployment ensures that my Pod is restarted in the event of a failure.
Note: The initial version of this code was not very optomized as it placed the vSphere session login in the controller reconcile code, and logging into vCenter Server for every reconcile request is not ideal. The login function has now been moved out of the reconcile request, and is now in main.go. This is the vlogin fucntion that I created in main.go:
//
// - vSphere session login function
//
func vlogin(ctx context.Context, vc, user, pwd string) (*vim25.Client, error) {
//
// Create a vSphere/vCenter client
//
// The govmomi client requires a URL object, u.
// You cannot use a string representation of the vCenter URL.
// soap.ParseURL provides the correct object format.
//
u, err := soap.ParseURL(vc)
if u == nil {
setupLog.Error(err, "Unable to parse URL. Are required environment variables set?", "controller", "VMInfo")
os.Exit(1)
}
if err != nil {
setupLog.Error(err, "URL parsing not successful", "controller", "VMInfo")
os.Exit(1)
}
u.User = url.UserPassword(user, pwd)
//
// Session cache example taken from https://github.com/vmware/govmomi/blob/master/examples/examples.go
//
// Share govc's session cache
//
s := &cache.Session{
URL: u,
Insecure: true,
}
//
// Create new client
//
c := new(vim25.Client)
//
// Login using client c and cache s
//
err = s.Login(ctx, c, nil)
if err != nil {
setupLog.Error(err, " login not successful", "controller", "VMInfo")
os.Exit(1)
}
return c, nil
}This is where the vlogin function is called in the main function:
//
// Retrieve vCenter URL, username and password from environment variables
// These are provided via the manager manifest when controller is deployed
//
vc := os.Getenv("GOVMOMI_URL")
user := os.Getenv("GOVMOMI_USERNAME")
pwd := os.Getenv("GOVMOMI_PASSWORD")
//
// Create context, and get vSphere session information
//
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
c, err := vlogin(ctx, vc, user, pwd)
if err != nil {
setupLog.Error(err, "unable to get login session to vSphere")
os.Exit(1)
}Finally, this is the modified Reconciler call, which now includes the vSphere session information:
//
// Add a new field, VC, to send session info to Reconciler
//
if err = (&controllers.VMInfoReconciler{
Client: mgr.GetClient(),
VC: c,
Log: ctrl.Log.WithName("controllers").WithName("VMInfo"),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "VMInfo")
os.Exit(1)
}The code complete main.go is available here.
Let's next turn our attention to the controller code. This is what kubebuilder provides as controller scaffolding - it is found in controllers/vminfo_controller.go - we are most interested in the VMInfoReconciler function:
func (r *VMInfoReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
_ = context.Background()
_ = r.Log.WithValues("vminfo", req.NamespacedName)
// your logic here
return ctrl.Result{}, nil
}Considering the business logic that I described above, this is what my updated VMInfoReconciler function looks like. Hopefully the comments make is easy to understand, but at the end of the day, when this controller gets a reconcile request (something as simple as a get command will trigger this), the status fields in the Custom Resource are updated for the specific VM in the spec.nodename field. Note that I have omitted a number of required imports that also need to be added to the controller. Refer to the code for the complete vminfo_controller.go code.
func (r *VMInfoReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
ctx := context.Background()
log := r.Log.WithValues("VMInfo", req.NamespacedName)
ch := &topologyv1.VMInfo{}
if err := r.Client.Get(ctx, req.NamespacedName, ch); err != nil {
// add some debug information if it's not a NotFound error
if !k8serr.IsNotFound(err) {
log.Error(err, "unable to fetch VMInfo")
}
return ctrl.Result{}, client.IgnoreNotFound(err)
}
msg := fmt.Sprintf("received reconcile request for %q (namespace: %q)", ch.GetName(), ch.GetNamespace())
log.Info(msg)
//
// Create a view manager
//
m := view.NewManager(r.VC)
//
// Create a container view of VirtualMachine objects
//
v, err := m.CreateContainerView(ctx, r.VC.ServiceContent.RootFolder, []string{"VirtualMachine"}, true)
if err != nil {
msg := fmt.Sprintf("unable to create container view for VirtualMachines: error %s", err)
log.Info(msg)
return ctrl.Result{}, err
}
defer v.Destroy(ctx)
//
// Retrieve summary property for all VMs
//
var vms []mo.VirtualMachine
err = v.Retrieve(ctx, []string{"VirtualMachine"}, []string{"summary"}, &vms)
if err != nil {
msg := fmt.Sprintf("unable to retrieve VM summary: error %s", err)
log.Info(msg)
return ctrl.Result{}, err
}
//
// Print summary for host in VMInfo specification info
//
for _, vm := range vms {
if vm.Summary.Config.Name == ch.Spec.Nodename {
ch.Status.GuestId = string(vm.Summary.Guest.GuestId)
ch.Status.TotalCPU = int64(vm.Summary.Config.NumCpu)
ch.Status.ResvdCPU = int64(vm.Summary.Config.CpuReservation)
ch.Status.TotalMem = int64(vm.Summary.Config.MemorySizeMB)
ch.Status.ResvdMem = int64(vm.Summary.Config.MemoryReservation)
ch.Status.PowerState = string(vm.Summary.Runtime.PowerState)
ch.Status.HwVersion = string(vm.Summary.Guest.HwVersion)
ch.Status.IpAddress = string(vm.Summary.Guest.IpAddress)
ch.Status.PathToVM = string(vm.Summary.Config.VmPathName)
}
}
if err := r.Status().Update(ctx, ch); err != nil {
log.Error(err, "unable to update VMInfo status")
return ctrl.Result{}, err
}
return ctrl.Result{}, nil
}With the controller logic now in place, we can now proceed to build and deploy the controller / manager.
At this point everything is in place to enable us to deploy the controller to the Kubernete cluster. If you remember back to the prerequisites in step 1, we said that you need access to a container image registry, such as docker.io or quay.io, or VMware's own Harbor registry. This is where we need this access to a registry, as we need to push the controller's container image somewhere that can be accessed from your Kubernetes cluster.
The Dockerfile with the appropriate directives is already in place to build the container image and include the controller / manager logic. This was once again taken care of by kubebuilder. You must ensure that you login to your image repository, i.e. docker login, before proceeding with the make commands. In this case, I am using the quay.io repository, e.g.
$ docker login quay.io
Username: cormachogan
Password: ***********
WARNING! Your password will be stored unencrypted in /home/cormac/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$Next, set an environment variable called IMG to point to your container image repository along with the name and version of the container image, e.g:
export IMG=quay.io/cormachogan/vminfo-controller:v1Next, to create the container image of the controller / manager, and push it to the image container repository in a single step, run the following make command. You could of course run this as two seperate commands as well, make docker-build followed by make docker-push if you so wished.
make docker-build docker-push IMG=quay.io/cormachogan/vminfo-controller:v1The container image of the controller is now built and pushed to the container image registry. But we have not yet deployed it. We have to do one or two further modifications before we take that step.
Kubebuilder provides a manager manifest scaffold file for deploying the controller. However, since we need to provide vCenter details to our controller, we need to add these to the controller/manager manifest file. This is found in config/manager/manager.yaml. This manifest contains the deployment for the controller. In the spec, we need to add an additional spec.env section which has the environment variables defined, as well as the name of our secret (which we will create shortly). Below is a snippet of that code. Here is the code-complete config/manager/manager.yaml).
spec:
.
.
env:
- name: GOVMOMI_USERNAME
valueFrom:
secretKeyRef:
name: vc-creds
key: GOVMOMI_USERNAME
- name: GOVMOMI_PASSWORD
valueFrom:
secretKeyRef:
name: vc-creds
key: GOVMOMI_PASSWORD
- name: GOVMOMI_URL
valueFrom:
secretKeyRef:
name: vc-creds
key: GOVMOMI_URL
volumes:
- name: vc-creds
secret:
secretName: vc-creds
terminationGracePeriodSeconds: 10Note that the secret, called vc-creds above, contains the vCenter credentials. This secret needs to be deployed in the same namespace that the controller is going to run in, which is vminfo-system. Thus, the namespace and secret are created using the following commands, with the environment modified to your own vSphere infrastructure obviously:
$ kubectl create ns vminfo-system
namespace/vminfo-system created$ kubectl create secret generic vc-creds \
--from-literal='GOVMOMI_USERNAME=administrator@vsphere.local' \
--from-literal='GOVMOMI_PASSWORD=VMware123!' \
--from-literal='GOVMOMI_URL=192.168.0.100' \
-n vminfo-system
secret/vc-creds createdWe are now ready to deploy the controller to the Kubernetes cluster.
To deploy the controller, we run another make command. This will take care of all of the RBAC, cluster roles and role bindings necessary to run the controller, as well as pinging up the correct image, etc.
make deploy IMG=quay.io/cormachogan/vminfo-controller:v1Now that our controller has been deployed, let's see if it is working. There are a few different commands that we can run to verify the operator is working.
The deployment should be READY. Remember to specify the namespace correctly when checking it.
$ kubectl get rs -n vminfo-system
NAME DESIRED CURRENT READY AGE
vminfo-controller-manager-79d6756854 1 1 1 37m
$ kubectl get deploy -n vminfo-system
NAME READY UP-TO-DATE AVAILABLE AGE
vminfo-controller-manager 1/1 1 1 37mThe deployment manages a single controller Pod. There should be 2 containers READY in the controller Pod. One is the controller / manager and the other is the kube-rbac-proxy. The kube-rbac-proxy is a small HTTP proxy that can perform RBAC authorization against the Kubernetes API. It restricts requests to authorized Pods only.
$ kubectl get pods -n vminfo-system
NAME READY STATUS RESTARTS AGE
vminfo-controller-manager-79d6756854-b8jdq 2/2 Running 0 72sIf you experience issues with the one of the pods not coming online, use the following command to display the Pod status and examine the events.
kubectl describe pod vminfo-controller-manager-79d6756854-b8jdq -n vminfo-systemIf we query the logs on the manager container, we should be able to observe successful startup messages as well as successful reconcile requests from the VMInfo CR that we already deployed back in step 5. These reconcile requests should update the Status fields with CPU information as per our controller logic. The command to query the manager container logs in the controller Pod is as follows:
kubectl logs vminfo-controller-manager-79d6756854-b8jdq -n vminfo-system managerLast but not least, let's see if we can see the CPU information in the status fields of the VMInfo object created earlier.
$ kubectl get vminfo tkg-worker-1 -o yaml
apiVersion: topology.corinternal.com/v1
kind: VMInfo
metadata:
creationTimestamp: "2021-01-18T12:20:45Z"
generation: 1
managedFields:
- apiVersion: topology.corinternal.com/v1
fieldsType: FieldsV1
fieldsV1:
f:spec:
.: {}
f:nodename: {}
manager: kubectl
operation: Update
time: "2021-01-18T12:20:45Z"
- apiVersion: topology.corinternal.com/v1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:guestId: {}
f:hwVersion: {}
f:ipAddress: {}
f:pathToVM: {}
f:powerState: {}
f:resvdCPU: {}
f:resvdMem: {}
f:totalCPU: {}
f:totalMem: {}
manager: manager
operation: Update
time: "2021-01-18T12:20:46Z"
name: tkg-worker-1
namespace: default
resourceVersion: "28841720"
selfLink: /apis/topology.corinternal.com/v1/namespaces/default/vminfoes/tkg-worker-1
uid: 2c60b273-a866-4344-baf5-0b3b924b65a5
spec:
nodename: tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298
status:
guestId: vmwarePhoton64Guest
hwVersion: vmx-17
ipAddress: 192.168.62.45
pathToVM: '[vsanDatastore] 4d56b55f-11db-8822-6463-246e962f4914/tkg-cluster-1-18-5b-workers-kc5xn-dd68c4685-5v298.vmx'
powerState: poweredOn
resvdCPU: 0
resvdMem: 0
totalCPU: 2
totalMem: 4096Success!!! Note that the output above is showing various status fields as per our business logic implemented in the controller. How cool is that? You can now go ahead and create additional VMInfo manifests for different virtual machines in your vSphere environment managed by your vCenter server by specifying different nodenames in the manifest spec, and all you to get status from those VMs as well.
To remove the vminfo CR, operator and CRD, run the following commands.
$ kubectl delete vminfo tkg-worker-1
vminfo.topology.corinternal.com "tkg-worker-1" deletedDeleting the deployment will removed the ReplicaSet and Pods associated with the controller.
$ kubectl get deploy -n vminfo-system
NAME READY UP-TO-DATE AVAILABLE AGE
vminfo-controller-manager 1/1 1 1 2d8h$ kubectl delete deploy vminfo-controller-manager -n vminfo-system
deployment.apps "vminfo-controller-manager" deletedNext, remove the Custom Resource Definition, vminfoes.topology.corinternal.com.
$ kubectl get crds
NAME CREATED AT
antreaagentinfos.clusterinformation.antrea.tanzu.vmware.com 2021-01-14T16:31:58Z
antreacontrollerinfos.clusterinformation.antrea.tanzu.vmware.com 2021-01-14T16:31:58Z
clusternetworkpolicies.security.antrea.tanzu.vmware.com 2021-01-14T16:31:59Z
vminfoes.topology.corinternal.com 2021-01-14T16:52:11Z
traceflows.ops.antrea.tanzu.vmware.com 2021-01-14T16:31:59Z$ make uninstall
go: creating new go.mod: module tmp
go: found sigs.k8s.io/controller-tools/cmd/controller-gen in sigs.k8s.io/controller-tools v0.2.5
/home/cormac/go/bin/controller-gen "crd:preserveUnknownFields=false,crdVersions=v1,trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl delete -f -
customresourcedefinition.apiextensions.k8s.io "vminfoes.topology.corinternal.com" deleted$ kubectl get crds
NAME CREATED AT
antreaagentinfos.clusterinformation.antrea.tanzu.vmware.com 2021-01-14T16:31:58Z
antreacontrollerinfos.clusterinformation.antrea.tanzu.vmware.com 2021-01-14T16:31:58Z
clusternetworkpolicies.security.antrea.tanzu.vmware.com 2021-01-14T16:31:59Z
traceflows.ops.antrea.tanzu.vmware.com 2021-01-14T16:31:59ZThe CRD is now removed. At this point, you can also delete the namespace created for the exercise, in this case vminfo-system. Removing this namespace will also remove the vc_creds secret created earlier.
One thing you could do it to extend the VMInfo fields and Operator logic so that it returns even more information about the virtual machine. . There is a lot of information that can be retrieved via the govmomi VirtualMachine API call.
You can now use kusomtize to package the CRD and controller and distribute it to other Kubernetes clusters. Simply point the kustomize build command at the location of the kustomize.yaml file which is in config/default.
kustomize build config/default/ >> /tmp/vminfo.yamlThis newly created vminfo.yaml manifest includes the CRD, RBAC, Service and Deployment for rolling out the operator on other Kubernetes clusters. Nice, eh?