Application to test resilience in services deployed in Kubernetes.
The program must runs inside the cluster, interacts with the kube-apiserver, and deletes on pod at random in a particular namespace on a schedule.
It is assumed that a schedule is a repetitive event over the time with a specific frequency.
| Parameter | Type | Default value | Description |
|---|---|---|---|
targetNamespace |
String | workloads |
Namespace used to remove pods. |
excludeNamespaces |
String | kube-system |
Namespaces were pods can't be removed. |
scheduler |
String | 5s |
Scheduler to delete a random pod. e.g 10s, 2m, 4h. |
labelSelector |
String | "" |
Label selector to filter the list of pods. |
gracePeriod |
Int64 | 0 |
Grace period to remove the pod. |
- Have container management tool installed.
- Have a kubernetes cluster created.
- Have Kubectl installed.
- Have Helm installed.
- Create Kubernetes cluster
- Deploy pod examples into workloads namespace
- Deploy podchaosmonkey app
- Check container image vulnerabilities
Kind can be used to create a Kubernetes cluster if you don't have your own cluster.
kind version
Output
kind v0.15.0 go1.19 linux/amd64
Create a new cluster with kind.
kind create cluster
Output
Creating cluster "kind" ...
✓ Ensuring node image (kindest/node:v1.25.0) 🖼
✓ Preparing nodes 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a nice day! 👋
Check cluster status.
kubectl get nodes
Output
NAME STATUS ROLES AGE VERSION
kind-control-plane Ready control-plane 62s v1.25.0
Create workloads namespace
kubectl create namespace workloads
Output
namespace/workloads created
Deploy example-1.
kubectl --namespace workloads apply \
--filename https://raw.githubusercontent.com/mmorejon/erase-una-vez-k8s/main/deployments/deploy-01.yaml
Output
deployment.apps/deploy-example-1 created
Deploy example-2.
kubectl --namespace workloads apply \
--filename https://raw.githubusercontent.com/mmorejon/erase-una-vez-k8s/main/deployments/deploy-02.yaml
Output
deployment.apps/deploy-example-2 created
List all pods created in the workloads namespace.
kubectl --namespace workloads get pods
Output
NAME READY STATUS RESTARTS AGE
deploy-example-1-7bd69c4c97-7sts9 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-88k6v 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-br4xf 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-jht7s 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-jj88k 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-kb6g6 1/1 Running 0 4m15s
deploy-example-1-7bd69c4c97-l25vf 1/1 Running 0 4m15s
deploy-example-2-5d6ffd8d74-5zpj2 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-82mwz 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-8z58n 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-9fr7p 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-9ws6h 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-d9x7x 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-nbtrr 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-pzzgw 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-s4mv6 1/1 Running 0 96s
deploy-example-2-5d6ffd8d74-vdbhs 1/1 Running 0 96s
The application use the default values defined in the parameter section, but these parameter can be changed in the value.yaml file.
Clone github repository.
{
git clone git@github.com:mmorejon/podchaosmonkey.git
cd podchaosmonkey
}
Deploy podchaosmonkey app using helm.
helm upgrade --install podchaosmonkey \
--namespace podchaosmonkey --create-namespace \
chart/podchaosmonkey
Output
Release "podchaosmonkey" does not exist. Installing it now.
NAME: podchaosmonkey
LAST DEPLOYED: Wed Sep 14 11:55:46 2022
NAMESPACE: podchaosmonkey
STATUS: deployed
REVISION: 1
TEST SUITE: None
Check podchaosmonkey pod deployment.
kubectl --namespace podchaosmonkey get pod
Output
NAME READY STATUS RESTARTS AGE
podchaosmonkey-9c9bc4586-l68rc 1/1 Running 0 8s
See podchaosmonkey logs.
kubectl --namespace podchaosmonkey logs \
--selector app.kubernetes.io/name=podchaosmonkey --follow
Output
Starting chaos process ...
Pods in the namespace workloads will be removed every 5s.
Waiting for the next schedule.
It is time to remove a new pod ...
Number of pods available 17
The pod deploy-example-1-7bd69c4c97-88k6v was removed.
Waiting for the next schedule.
It is time to remove a new pod ...
Number of pods available 17
The pod deploy-example-2-5d6ffd8d74-nbtrr was removed.
Waiting for the next schedule.
It is time to remove a new pod ...
Number of pods available 17
The pod deploy-example-1-7bd69c4c97-jj88k was removed.
Trivy can be used to detect vulnerabilities in the podchaosmonkey image.
trivy image ghcr.io/mmorejon/podchaosmonkey:v0.1.0
Output
2022-09-14T12:15:12.510+0200 INFO Need to update DB
2022-09-14T12:15:12.510+0200 INFO DB Repository: ghcr.io/aquasecurity/trivy-db
2022-09-14T12:15:12.510+0200 INFO Downloading DB...
33.86 MiB / 33.86 MiB [------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 17.17 MiB p/s 2.2s
2022-09-14T12:15:16.177+0200 INFO Vulnerability scanning is enabled
2022-09-14T12:15:16.177+0200 INFO Secret scanning is enabled
2022-09-14T12:15:16.177+0200 INFO If your scanning is slow, please try '--security-checks vuln' to disable secret scanning
2022-09-14T12:15:16.177+0200 INFO Please see also https://aquasecurity.github.io/trivy/v0.31.2/docs/secret/scanning/#recommendation for faster secret detection
2022-09-14T12:15:19.121+0200 INFO Number of language-specific files: 1
2022-09-14T12:15:19.121+0200 INFO Detecting gobinary vulnerabilities...