Cluster Upgrade failed
Opened this issue · 0 comments
sohel2020 commented
When I apply terraform change, I'm getting the following error
2024-06-11T18:05:12.288+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="W0611 18:05:12.285659 37605 builder.go:232] failed to digest image "public.ecr.aws/aws-ec2/aws-node-termination-handler:v1.18.3": GET https://public.ecr.aws/v2/aws-ec2/aws-node-termination-handler/manifests/v1.18.3: DENIED: Your authorization token has expired. Reauthenticate and try again."
2024-06-11T18:05:15.882+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="W0611 18:05:15.881657 37605 builder.go:232] failed to digest image "public.ecr.aws/eks-distro/kubernetes-sigs/aws-iam-authenticator:v0.5.9-eks-1-23-3": GET https://public.ecr.aws/v2/eks-distro/kubernetes-sigs/aws-iam-authenticator/manifests/v0.5.9-eks-1-23-3: DENIED: Your authorization token has expired. Reauthenticate and try again."
module.kubernetes.kops_cluster_updater.this: Still modifying... [id=k8s.data-staging.gg, 20s elapsed]
2024-06-11T18:05:23.231+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:23.231157 37605 executor.go:111] Tasks: 0 done / 139 total; 57 can run"
2024-06-11T18:05:23.858+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:23.857623 37605 executor.go:111] Tasks: 57 done / 139 total; 26 can run"
2024-06-11T18:05:24.375+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:24.375051 37605 executor.go:111] Tasks: 83 done / 139 total; 26 can run"
2024-06-11T18:05:25.230+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:25.230449 37605 executor.go:111] Tasks: 109 done / 139 total; 6 can run"
2024-06-11T18:05:25.524+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:25.524526 37605 dnsname.go:118] AliasTarget for "api.k8s.data-staging.gg." is "api-k8s-data-staging-gg-2cqvvl-bc2c9142b3686c13.elb.eu-west-1.amazonaws.com.""
2024-06-11T18:05:25.791+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:25.790691 37605 dnsname.go:118] AliasTarget for "api.k8s.data-staging.gg." is "api-k8s-data-staging-gg-2cqvvl-bc2c9142b3686c13.elb.eu-west-1.amazonaws.com.""
2024-06-11T18:05:26.139+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:26.139101 37605 executor.go:111] Tasks: 115 done / 139 total; 8 can run"
2024-06-11T18:05:26.586+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:26.586270 37605 executor.go:111] Tasks: 123 done / 139 total; 4 can run"
2024-06-11T18:05:26.974+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:26.974564 37605 executor.go:111] Tasks: 127 done / 139 total; 12 can run"
2024-06-11T18:05:27.274+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:27.274537 37605 executor.go:111] Tasks: 139 done / 139 total; 0 can run"
2024-06-11T18:05:27.698+0200 [WARN] unexpected data: registry.terraform.io/terraform-kops/kops:stderr="I0611 18:05:27.697754 37605 dns.go:235] Pre-creating DNS records"
2024-06-11T18:05:28.783+0200 [INFO] provider.terraform-provider-kops_v1.28.7: 2024/06/11 18:05:28 (will retry): unexpected error during validation: error listing nodes: Get "https://api-k8s-data-staging-gg-2cqvvl-bc2c9142b3686c13.elb.eu-west-1.amazonaws.com:8443/api/v1/nodes": tls: failed to verify certificate: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, api.k8s.data-staging.gg, api.internal.k8s.data-staging.gg, not api-k8s-data-staging-gg-2cqvvl-bc2c9142b3686c13.elb.eu-west-1.amazonaws.com: timestamp=2024-06-11T18:05:28.783+0200
module.kubernetes.kops_cluster_updater.this: Still modifying... [id=k8s.data-staging.gg, 30s elapsed]Anyone know why it is using AWS ELB endpoint instead of api.k8s.data-staging.gg ?
When I use cli it works fine.
$ kops validate cluster k8s.data-staging.
Validating cluster k8s.data-staging.gg
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-eu-west-1a ControlPlane t3.xlarge 1 1 eu-west-1a
master-eu-west-1b ControlPlane t3.xlarge 1 1 eu-west-1b
master-eu-west-1c ControlPlane t3.xlarge 1 1 eu-west-1c
nodes Node t3.2xlarge 4 6 eu-west-1a,eu-west-1b,eu-west-1c
NODE STATUS
NAME ROLE READY
i-03f177f4b1ccd1e9e control-plane True
i-05901f4825a44a9ca node True
i-05c9edab031dc33d5 node True
i-06b9be93eaf4f9658 node True
i-092cb4ac8aaa8b059 control-plane True
i-0a7e5f8ff0ca5deda node True
i-0f806bd3cf57e2717 control-plane True
Your cluster k8s.data-staging.gg is ready
$ kops version
Client version: 1.28.5 (git-v1.28.5)