"service e2e-test-webhook" not found cause delete failed

[shiguowang]

Thank you for your excellent script! It really help me a lot!
I do a k8s e2e test, after that I have two namesapces which can't be delete

knsk's output:
Kubernetes NameSpace Killer

Checking if kubectl is configured... ok

Checking for unavailable apiservices... not found

Checking for stuck namespaces... found
.: Checking resources in namespace webhook-7327... found

configmap/kube-root-ca.crt... dry-run
timeout 15 kubectl -n webhook-7327 --grace-period=0 --force=true delete configmap/kube-root-ca.crt
timeout 15 kubectl -n webhook-7327 patch configmap/kube-root-ca.crt -p '{"metadata":{"finalizers":null}}'
.: Checking resources in namespace webhook-9779... found
configmap/kube-root-ca.crt... dry-run
timeout 15 kubectl -n webhook-9779 --grace-period=0 --force=true delete configmap/kube-root-ca.crt
timeout 15 kubectl -n webhook-9779 patch configmap/kube-root-ca.crt -p '{"metadata":{"finalizers":null}}'

my /tmp/webhook-7327.json's content:
{
"apiVersion": "v1",
"kind": "Namespace",
"metadata": {
"creationTimestamp": "2021-03-17T12:51:52Z",
"deletionTimestamp": "2021-03-17T12:52:49Z",
"labels": {
"e2e-framework": "webhook",
"e2e-run": "bd69f395-1e7b-4b4a-b268-9e40135e2b64",
"webhook-7327": "true"
},
"managedFields": [
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:metadata": {
"f:labels": {
".": {},
"f:e2e-framework": {},
"f:e2e-run": {},
"f:webhook-7327": {}
}
},
"f:status": {
"f:phase": {}
}
},
"manager": "e2e.test",
"operation": "Update",
"time": "2021-03-17T12:51:52Z"
},
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:status": {
"f:conditions": {
".": {},
"k:{"type":"NamespaceContentRemaining"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{"type":"NamespaceDeletionContentFailure"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{"type":"NamespaceDeletionDiscoveryFailure"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{"type":"NamespaceDeletionGroupVersionParsingFailure"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
},
"k:{"type":"NamespaceFinalizersRemaining"}": {
".": {},
"f:lastTransitionTime": {},
"f:message": {},
"f:reason": {},
"f:status": {},
"f:type": {}
}
}
}
},
"manager": "kube-controller-manager",
"operation": "Update",
"time": "2021-03-17T12:52:54Z"
}
],
"name": "webhook-7327",
"resourceVersion": "1502908",
"uid": "2ddf4230-d197-4f50-ba0f-28e099a6797b"
},
"spec": {
"finalizers": [

    ]
},
"status": {
    "conditions": [
        {
            "lastTransitionTime": "2021-03-17T21:23:28Z",
            "message": "All resources successfully discovered",
            "reason": "ResourcesDiscovered",
            "status": "False",
            "type": "NamespaceDeletionDiscoveryFailure"
        },
        {
            "lastTransitionTime": "2021-03-17T12:52:54Z",
            "message": "All legacy kube types successfully parsed",
            "reason": "ParsedGroupVersions",
            "status": "False",
            "type": "NamespaceDeletionGroupVersionParsingFailure"
        },
        {
            "lastTransitionTime": "2021-03-17T12:52:54Z",
            "message": "Failed to delete all resource types, 1 remaining: Internal error occurred: failed calling webhook \"deny-unwanted-configmap-data.k8s.io\": Post \"https://e2e-test-webhook.webhook-7327.svc:8443/configmaps?timeout=10s\": service \"e2e-test-webhook\" not found",
            "reason": "ContentDeletionFailed",
            "status": "True",
            "type": "NamespaceDeletionContentFailure"
        },
        {
            "lastTransitionTime": "2021-03-17T12:52:54Z",
            "message": "All content successfully removed",
            "reason": "ContentRemoved",
            "status": "False",
            "type": "NamespaceContentRemaining"
        },
        {
            "lastTransitionTime": "2021-03-17T12:52:54Z",
            "message": "All content-preserving finalizers finished",
            "reason": "ContentHasNoFinalizers",
            "status": "False",
            "type": "NamespaceFinalizersRemaining"
        }
    ],
    "phase": "Terminating"
}

}

my /tmp/webhook-7327.json.log 's conteng:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "the body of the request was in an unknown format - accepted media types include: application/json, application/yaml, application/vnd.kubernetes.protobuf",
"reason": "UnsupportedMediaType",
"code": 415

my kube-apiserver's log:
Apr 26 20:19:37 master1 kube-apiserver[1168]: E0426 20:19:37.557723 1168 dispatcher.go:159] Internal error occurred: failed calling webhook "deny-unwanted-configmap-data.k8s.io": Post "https://e2e-test-webhook.webhook-7327.svc:8443/configmaps?timeout=10s": service "e2e-test-webhook" not found
Apr 26 20:19:37 master1 kube-apiserver[1168]: W0426 20:19:37.597769 1168 dispatcher.go:134] Failed calling webhook, failing closed deny-unwanted-configmap-data.k8s.io: failed calling webhook "deny-unwanted-configmap-data.k8s.io": Post "https://e2e-test-webhook.webhook-9779.svc:8443/configmaps?timeout=10s": service "e2e-test-webhook" not found

[shiguowang]

I think the problem has relation with ""e2e-test-webhook" which I really don't have

[thyarles]

Hi @shiguowang,

Thanks for taking time to talk about the script, I'm happy that it helps... I use it once a month on my four kubernetes clusters to see if there any mess.

Could you please tell what is the version of your kubernetes? Is it running on minikube, microk8s or rke?