cloud-provider-aws does not Prefer CLI Arguments for Configuring Kubernetes
aauren opened this issue · 7 comments
What happened:
I attempted to pass --master and --kubeconfig to cloud-provider-aws and didn't see any change to the output of the provider. This resulted in cloud-provider-aws not being configured correctly in my cluster.
This resulted in the following error:
F0128 01:25:22.339300 26 main.go:84] unable to execute command: unable to load configmap based request-header-client-ca-file: Get "https://10.96.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication": dial tcp 10.96.0.1:443: i/o timeout
In the above, 10.96.0.1 is what the kube-apiserver would be reachable at, if my CNI was up and proxying requests to it's ClusterIP.
What you expected to happen:
I expected it to adhere to the CLI arguments that I passed.
How to reproduce it (as minimally and precisely as possible):
- Run the DaemonSet as documented
- Configure the arguments in the daemonset to attempt to use either
--kubeconfigor--masterand see thataws-cloud-controller-managerdoes not follow the defined configuration
Anything else we need to know?:
In my case, I found that this was caused by aws-cloud-controller-manager preferring the environment variables that Kubernetes sets:
KUBERNETES_SERVICE_HOSTKUBERNETES_SERVICE_PORT
instead of the CLI parameters passed.
This will be surprising for most users as other CLI tooling from Kubernetes like kubectl prefer the CLI arguments over environment variables.
It also makes it incredibly difficult to bootstrap a new cluster where the CNI is dependent on information from the cloud provider plugin in order to function correctly. As it will prefer the ClusterIP of the Kubernetes API server rather than the configuration that the user is attempting to override it with and result in a race condition.
Environment:
- Kubernetes version (use
kubectl version): v1.28.2 - Cloud provider or hardware configuration: AWS -
aws-cloud-controller-managerv1.29.1 - OS (e.g. from /etc/os-release):
Ubuntu 22.04.3 LTS - Kernel (e.g.
uname -a):6.2.0-1018-aws - Install tools:
- Others:
/kind bug
This issue is currently awaiting triage.
If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
aws cloud provider uses the upstream kubernetes cloud library and the error is coming from the upstream code. It is using a different flag to override the kube config . Can you adding additional flag authentication-kubeconfig and verify if it works. We can create an issue upstream and see how they would like to proceed.
Adding --authentication-kubeconfig seemed to make the configuration work as well.
Can I ask why cloud-provider-aws has two separate flags for this? --kubeconfig is the normal flag so I wasn't expecting to find another flag for passing the kubeconfig again.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle rotten - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Reopen this issue with
/reopen - Mark this issue as fresh with
/remove-lifecycle rotten - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
@k8s-triage-robot: Closing this issue, marking it as "Not Planned".
In response to this:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied- After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied- After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closedYou can:
- Reopen this issue with
/reopen- Mark this issue as fresh with
/remove-lifecycle rotten- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.