Cannot enable instance metadata access to tags due to tag format
adamacosta opened this issue · 6 comments
What happened:
See terraform-aws-modules/terraform-aws-eks#1785
If you deploy an ASG with the launch template set to allow instances access to tags via the metadata service, the API request to AWS will return an error:
InvalidRequestException: 'k8s.io/cluster-autoscaler/enabled' is not a valid tag key. Tag keys must match pattern ([0-9a-zA-Z\-+=,.@:]{1,255}), and must not be a reserved name ('.', '..', 'index')
Note this example is for autoscaler, but it's the same error for both cloud-provider-aws and cluster-autoscaler. The issue is simply having '/' in a tag.
What you expected to happen:
It should be possible to allow metadata access to tags when using CCM (or cluster-autoscaler or both).
How to reproduce it (as minimally and precisely as possible):
Attempt to deploy any ASG with more than 0 instances using the tag specifications from either cloud-provider-aws or cluster-autoscaler with metadata access to tags set to enabled. It is not necessary to actually install Kubernetes. Launching an instance at all fails.
Anything else we need to know?:
I'm guessing this is just a consequence of the fact that the metadata service lets you query "http://169.254.169.254/latest/meta-data/tags/instance/{tag-key}" and doesn't allow '/' in the tag key, even though this is valid for an AWS tag in general, because it gets interpreted by the API backend for hierarchical routing and can't be part of an API resource name. Their regex appears to be pretty close to the RFC 3986 path segment definition, removing '~', adding '+' and '=', and adding a 255 character limit.
This feels more like a bug on the part of AWS and they shouldn't allow '/' in tags at all, at least not for EC2 instances, but I'm fairly certain they won't care or do anything about it. Kubernetes projects associated with AWS, on the other hand, could at least not require tags that won't be allowed. I don't expect you'll do anything, either, given this is a fairly standard way of namespacing annotations in Kubernetes, but documentation could at least be added that metadata service access to tags isn't possible if you use cloud-provider-aws.
Environment:
- Kubernetes version (use
kubectl version): - Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
/kind bug
This issue is currently awaiting triage.
If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
This is a documented limitation of tags-on-IMDS:
If you enable instance tags in instance metadata, instance tag keys can only use letters (a-z, A-Z), numbers (0-9), and the following characters:
+ - = . , _ : @. Instance tag keys can't contain spaces or/, and can't comprise only.(one period),..(two periods), or_index.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
I do not expect this to change.
/close not-planned
@cartermckinnon: Closing this issue, marking it as "Not Planned".
In response to this:
This is a documented limitation of tags-on-IMDS:
If you enable instance tags in instance metadata, instance tag keys can only use letters (a-z, A-Z), numbers (0-9), and the following characters:
+ - = . , _ : @. Instance tag keys can't contain spaces or/, and can't comprise only.(one period),..(two periods), or_index.https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictions
I do not expect this to change.
/close not-planned
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
Yeah, so did you see I said in my report that I realize this is a limitation of tags on IMDS? I was suggesting you add documentation to the examples of how to use this provider saying you can't enable metadata tag access. I'm sure most people will figure that out anyway, but you'd be saving them a few minutes to a few hours of figuring it out. Making things a tiny bit easier on users is a pleasant thing developers can do to be friendly.
@adamacosta feel free to open a PR if you think a doc would help, probably should go in here: https://github.com/kubernetes/cloud-provider-aws/blob/master/docs/prerequisites.md
this is highly disappointing as the automatically generated tags for EKS break tags in metadata. One feature in AWS breaking another feature does seem to be a bit odd.