Can not manipulate with `deployment` resources with k8s version 1.11.6

[kolach]

Problem:

Can not manipulate with deployment resources. Getting error service unavailable. It's enough to run deployment unit test to reproduce. See bellow.

Kubernetes Version

1.11.6-gke.2 Google K8s Engine

Affected Resource(s)

• deployment

Expected Behavior

TESTARGS="-run TestAccKubernetesDeployment_minimal" make testacc
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run TestAccKubernetesDeployment_minimal -timeout 120m
?       github.com/sl1pm4t/terraform-provider-kubernetes        [no test files]
=== RUN   TestAccKubernetesDeployment_minimal
=== PAUSE TestAccKubernetesDeployment_minimal
=== CONT  TestAccKubernetesDeployment_minimal
--- PASS: TestAccKubernetesDeployment_minimal (5.89s)
PASS

Actual Behavior

TESTARGS="-run TestAccKubernetesDeployment_minimal" make testacc
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test $(go list ./... |grep -v 'vendor') -v -run TestAccKubernetesDeployment_minimal -timeout 120m
?       github.com/sl1pm4t/terraform-provider-kubernetes        [no test files]
=== RUN   TestAccKubernetesDeployment_minimal
=== PAUSE TestAccKubernetesDeployment_minimal
=== CONT  TestAccKubernetesDeployment_minimal
--- FAIL: TestAccKubernetesDeployment_minimal (2.05s)
        testing.go:518: Step 0 error: Error applying: 1 error(s) occurred:

                * kubernetes_deployment.test: 1 error(s) occurred:

                * kubernetes_deployment.test: an error on the server ("service unavailable") has prevented the request from succeeding
FAIL
FAIL    github.com/sl1pm4t/terraform-provider-kubernetes/kubernetes     2.127s
make: *** [testacc] Error 1

Steps to Reproduce

TESTARGS="-run TestAccKubernetesDeployment_minimal" make testacc

[kolach]

Can somebody help me pointing in source code where we do an API call?

[kolach]

We found the issue happened because of metrics service found in failed state. The cure was to remove it. The system restarted it and all started work again.

[kolach]

To check metrics service status:

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

in case of trouble expect the output like:

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml                                           
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","kubernetes.io/cluster-service":"true"},"name":"v1beta1.metrics.k8s.io","namespace":""},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-server","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: 2018-09-04T01:15:27Z
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: v1beta1.metrics.k8s.io
  resourceVersion: "222666330"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 020981a2-afe0-11e8-8855-42010a800015
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2019-03-07T17:15:11Z
    message: 'no response from https://10.20.9.29:443: Get https://10.20.9.29:443:
      net/http: request canceled while waiting for connection (Client.Timeout exceeded
      while awaiting headers)'
    reason: FailedDiscoveryCheck
    status: "False"
    type: Available

get the pod name with:

kubectl get pods --namespace=kube-system --selector=k8s-app=metrics-server

and delete it with kubectl delete <pod>

It'll be immediately restarted by the system. Check status again. It should now be:

apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apiregistration.k8s.io/v1beta1","kind":"APIService","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","kubernetes.io/cluster-service":"true"},"name":"v1beta1.metrics.k8s.io","namespace":""},"spec":{"group":"metrics.k8s.io","groupPriorityMinimum":100,"insecureSkipTLSVerify":true,"service":{"name":"metrics-server","namespace":"kube-system"},"version":"v1beta1","versionPriority":100}}
  creationTimestamp: 2019-03-07T22:28:22Z
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
  name: v1beta1.metrics.k8s.io
  resourceVersion: "159125"
  selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.metrics.k8s.io
  uid: 50ce9e7d-4128-11e9-923c-42010a800070
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100
status:
  conditions:
  - lastTransitionTime: 2019-03-08T18:17:19Z
    message: all checks passed
    reason: Passed
    status: "True"
    type: Available