Develop integration into Talos K8s token provisioning once available
Closed this issue · 12 comments
- The 0.15 milestone (now 1.0, delayed though) of Talos includes a change to auto-provision Talos API credentials in K8s (siderolabs/talos#4422). Talos-vmtoolsd can make use of this feature to simplify installation since no more config bootstrapping is needed.
- We can change the setup instructions to extend Talos' "extraManifests" config key if the auto-provisioning part works.
- Talos 1.0+ includes a system extensions feature that offers complete access to machined and thus the relevant part of Talos' API.
Hi @mologie! Just for the note. I've just seen that additional functionality was added to Talos 1.0 => https://www.talos.dev/v1.0/learn-more/extension-services/ Not sure if that will help implementing Vmware tools in a better way. In addition, to the dependant issue: "smira removed this from the v0.15 milestone on 21 Feb".
Hi @dimatha, I am aware and in contact with the Sidero folks via Slack. So far I determined that talos-vmtoolsd can be implemented as Talos extension and thereby reduce configuration effort for the user, but I have not made any code or preview public yet. I will update this issue to reflect this. Appreciate your input regardless, thanks :)!
Hey @mologie ! I just wanted to get back to this topic and check if you will continue your excellent work on the Vmware tools integration ?
Yes, in fact there is an open pull request at #8 by @bnason, and a working system extension in his fork of this tool. The only thing that blocks merging so far is extensive testing at my company, which is well underway. (Sorry for the lack of status updates, Brandon!)
If you'd like to please give his extension a shot with your cluster too. You can find a discussion about how to use it at:
https://taloscommunity.slack.com/archives/CPW6V498D/p1680742717739549
Just to edit in relevant bits:
Created and tested talos-v1.3.7-vmtoolsd.ova using make image-vmware IMAGER_SYSTEM_EXTENSIONS="ghcr.io/bnason/talos-vmtoolsd-system-extension:v1.0.0"
So one thing to watch out for is upgrading. Having the extension installed in the ova doesn't add it to the machine config so an upgrade will remove the extension. I had to add it to the configPatches sections for my capi install. Ideally, there would be a mechanism for the machine to add that in itself since it's already installed.
Hi @mologie, thanks for making talos easier to use with VMware. I've run into this same issue where VMware can't get the IP for my deployed VM's early enough.
I can't access the slack channel listed above. Was there any more instruction there on how to make this system extension work that you can share?
Hi @bobbled, the Talos Slack is unfortunately still a free instance limited to a 10K message history and I frankly don't remember. As far as I recall, "make image-vmware" with a custom extension path was sufficient.
Note however bnason's build likely has compatibility issues with Talos 1.5 due to #9, so I'd advise to wait until next week when his changes are integrated here.
I won't manage to develop a ready-made system extension image by next week, but have enough included to start playing with it in this issue (or a new one, this is about token stuff, not system images).
@mologie , that's great to hear!
I did manage to figure out how to incorporate his extension into talos v1.6.1 using the talos imager tool and load it up on a v7.x vcenter, but as you mention above, I think there are either compatability issues with the newer version of talos, or the version of vmtools is just too old.
It reports "Running, version:2147483647 (Guest Managed)" when I look at the vm's summary. The only reference I can find to that version is now about 5 years old, so I'm not sure even if it did work properly with the new version of talos, that it would work with current versions of vsphere.
At any rate, thanks again.
EDIT: I should have read the issue #9 you mentioned first. That was the issue I was seeing. I attempted to build bnason's container, but couldn't figure out how to do so. I assume it's not using the latest unstable version you linked in #9. If it were, I might have no isseus.
This repo has changed hands since last comment (see #9). Our vision/focus would be to have talos-vmtoolsd run as a system extension, in the first place. We are going to fix up the tool for 1.6 compatibility, and dust of Brandon's work with the extension.
Given that all that succeeds and everything works, would there still be an interest to deploy talos-vmtoolsd in Kubernetes (not as an extension) and if so, what would be your use case?
This repo has changed hands since last comment (see #9). Our vision/focus would be to have talos-vmtoolsd run as a system extension, in the first place. We are going to fix up the tool for 1.6 compatibility, and dust of Brandon's work with the extension.
That's great news! It would solve many headaches.
Given that all that succeeds and everything works, would there still be an interest to deploy talos-vmtoolsd in Kubernetes (not as an extension) and if so, what would be your use case?
I have no specific use case that requires the deploy of talos-vmtoolsd not as an extension. The extension perfectly suits the scenario I plan to use it in.
Our vision/focus would be to have talos-vmtoolsd run as a system extension
With 1.6, the extension should also start as early as in maintenance mode reporting data back
Having this as a system extension would be great, do you have any timelines in mind as when we can use this?
Expect a system extention release in the next couple of days.