JX3 setup failing. Kubernetes cluster unreachable: invalid configuration.
serhiykrupka opened this issue · 11 comments
Summary
JX3 setup failed on AWS
Steps to reproduce the behavior
-
generate repo from https://github.com/jx3-gitops-repositories/jx3-terraform-eks/generate (commit https://github.com/jenkins-x/terraform-aws-eks-jx/tree/af1182fc3d07881af6a96a2335255ef7b0dce7f6)
-
generate repo from https://github.com/jx3-gitops-repositories/jx3-eks-vault/generate
-
configure terraform variables
-
execute:
terraform apply
Expected behavior
Boot job stated without errors.
Actual behavior
module.eks-jx.module.cluster.module.iam_assumable_role_cluster_autoscaler.aws_iam_role_policy_attachment.custom[0]: Creating...
module.eks-jx.module.cluster.module.iam_assumable_role_pipeline_visualizer.aws_iam_role_policy_attachment.custom[0]: Creating...
module.eks-jx.module.cluster.module.iam_assumable_role_external_dns.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-external-dns-20210121104622556100000019]
module.eks-jx.module.cluster.module.iam_assumable_role_controllerbuild.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-build-ctrl-2021012110462263690000001a]
module.eks-jx.module.cluster.module.iam_assumable_role_cm_cainjector.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-cert-manager-cert-manager-cainjector-2021012110462269650000001b]
module.eks-jx.module.cluster.module.iam_assumable_role_cluster_autoscaler.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-cluster-autoscaler-cluster-autoscaler-2021012110462283150000001c]
module.eks-jx.module.cluster.module.iam_assumable_role_pipeline_visualizer.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-jx-pipelines-visualizer-2021012110462286320000001d]
module.eks-jx.module.cluster.module.iam_assumable_role_tekton_bot.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-tekton-bot-2021012110462288510000001f]
module.eks-jx.module.cluster.module.iam_assumable_role_cert_manager.aws_iam_role_policy_attachment.custom[0]: Creation complete after 1s [id=jx-test-cert-manager-cert-manager-2021012110462288200000001e]
module.eks-jx.module.cluster.null_resource.kubeconfig (local-exec): Updated context arn:aws:eks:eu-central-1:xxx:cluster/jx-test in /Users/serhiykrupka/.kube/config
module.eks-jx.module.cluster.null_resource.kubeconfig: Creation complete after 1s [id=5757698152048227794]
module.eks-jx.module.cluster.helm_release.jx-git-operator[0]: Creating...
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Terraform version
v0.13.3
Module version
Operating system
MacOS 11.1
Same here
- terraform version
v0.14.5
- OS
CentOS Linux release 7.9.2009 (Core)
- helm
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}
Note that I can actually reach the cluster:
# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-node-7vdqj 1/1 Running 0 11m
aws-node-fksdg 1/1 Running 0 11m
aws-node-nhmgr 1/1 Running 0 11m
coredns-59b69b4849-hslfw 1/1 Running 0 16m
coredns-59b69b4849-wdzds 1/1 Running 0 16m
kube-proxy-bl7vm 1/1 Running 0 11m
kube-proxy-chw85 1/1 Running 0 11m
kube-proxy-jfktn 1/1 Running 0 11m
# eksctl get cluster
NAME REGION EKSCTL CREATED
eks-devops eu-central-1 False
This is the last part of the apply
with debug enabled:
-----------------------------------------------------: timestamp=2021-01-24T17:54:46.619Z
2021-01-24T17:54:46.620Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [DEBUG] [aws-sdk-go] {"ContinuousBackupsDescription":{"ContinuousBackupsStatus":"ENABLED","PointInTimeRecoveryDescription":{"PointInTimeRe
coveryStatus":"DISABLED"}}}: timestamp=2021-01-24T17:54:46.619Z
2021/01/24 17:54:46 [WARN] Provider "registry.terraform.io/hashicorp/aws" produced an unexpected new value for module.eks-jx.module.vault.aws_dynamodb_table.vault-dynamodb-table[0], but we are tolerating it because it is using the le
gacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .write_capacity: was cty.NumberIntVal(2), but now cty.NumberIntVal(0)
- .read_capacity: was cty.NumberIntVal(2), but now cty.NumberIntVal(0)
module.eks-jx.module.vault.aws_dynamodb_table.vault-dynamodb-table[0]: Modifications complete after 2s [id=vault-unseal-eks-devops-Y9yKtOQW]
module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]: Reading... [id=3649945265]
2021-01-24T17:54:46.686Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.686Z
2021-01-24T17:54:46.686Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.686Z
2021-01-24T17:54:46.686Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.686Z
2021-01-24T17:54:46.686Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.686Z
2021-01-24T17:54:46.687Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.686Z
2021-01-24T17:54:46.687Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.687Z
2021-01-24T17:54:46.687Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.687Z
2021-01-24T17:54:46.687Z [INFO] plugin.terraform-provider-aws_v3.25.0_x5: 2021/01/24 17:54:46 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-01-24T17:54:46.687Z
module.eks-jx.module.vault.data.aws_iam_policy_document.vault_iam_user_policy_document[0]: Read complete after 0s [id=3649945265]
2021/01/24 17:54:46 [DEBUG] After incorporating new values learned so far during apply, module.eks-jx.module.vault.aws_iam_policy.aws_vault_user_policy[0] change has become NoOp
2021-01-24T17:54:46.711Z [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-24T17:54:46.715Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/3.25.0/linux_amd64/terraform-provider-aws_v3.25.0_x5 pid=26563
2021-01-24T17:54:46.715Z [DEBUG] plugin: plugin exited
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
2021-01-24T17:54:46.734Z [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-24T17:54:46.734Z [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-24T17:54:46.734Z [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-01-24T17:54:46.737Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/helm/2.0.2/linux_amd64/terraform-provider-helm_v2.0.2_x5 pid=26543
2021-01-24T17:54:46.737Z [DEBUG] plugin: plugin exited
2021-01-24T17:54:46.739Z [DEBUG] plugin: plugin process exited: path=/usr/bin/terraform pid=26385
2021-01-24T17:54:46.739Z [DEBUG] plugin: plugin exited
2021-01-24T17:54:46.739Z [DEBUG] plugin: plugin process exited: path=/usr/bin/terraform pid=26288
2021-01-24T17:54:46.739Z [DEBUG] plugin: plugin exited
Also note that before trying jx3 I've tried jx2, which is using the same module, and it worked, so I guess the issue is related the jx3 part triggered by is_jx2
.
I did some testing and apparently if I set to false
the option install_kuberhealthy
I get the error only one time:
Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
If I set it back to true
, I get 2 errors like that, as you can see in both mine and OP log, so (maybe) that option is related.
Hope it helps.
we fixed the kuberhealthy issue here: jenkins-x/terraform-jx-health@9d99da3 it was caused by a new kuberhealthy chart breaking
I wonder if you could remove your .terraform
folder and rerun terraform plan && terraform apply
?
Sure, currently destroying, will start over and update, thanks! :)
same issue for me, tried on both macos and ubuntu
Update: Unfortunately I'm getting the same behaviour starting with a new deployment (no .terraform
)
/assign
I have been doing what @patrickleet suggested in the slack message. Let me check why this is breaking, will report back.
i had the same problem. I fixed that by adding helm provider config
provider "helm" {
kubernetes {
config_path = "./kubeconfig_jenkins"
}
}
it will fail on first call, but it will success on second call, after kubeconfig_jenkins is created
it will fail on first call, but it will success on second call, after kubeconfig_jenkins is created
So the reason this happened was because, the helm provider uses outdated kube config on the first apply, then when you apply the second time, it has access to the correct credentials and applies successfully.
@dephee Would be nice to give my branch a try, and see if the error goes away - wont need the provider helm block anymore in the main.tf
file.