Image builds should use ansible from upstream kubernetes
Closed this issue · 4 comments
I've been looking through the demo repository and looking at was to improve it. Rather than maintaining a set of in-tree ansible playbooks to build images, it would make more sense to rely on the ansible roles provided in upstream at https://github.com/kubernetes/contrib/tree/master/ansible The ansible there looks to be more complete and flexible. It supports installing on different distro flavors, installing kubernetes from packages or source checkouts, and provides lots of tweakable config variables to customize deployments. That said, I am not certain (yet) that would be reusable in for image building as opposed to running against a live system.
If the long-term plan of this repo is to showcase the many moving parts of the kubernetes ecosystem functioning together, it would seem appropriate it integrate on as much upstream stuff as possible.
With regard to potentially using this repo as a test harness for a future integration test suite, the ability to use source checkouts would easily allow us to build and test images for proposed patches to things running in the image (k8s, etcd, etc.).
The upstream ansible and the ansible in this demo repo don’t necessarily tie into any larger approach, just vestigial leftovers. Everything is still in flux basically.
While there was some initial work on creating images with ansible/packer in this demo repo, the focus was never specifically on producing images, that is just an artifact of trying to get going quickly (I used to spin up and tear down lots of clusters manually and this was just the fastest method for my own use).
If the long-term plan of this repo is to showcase the many moving parts of the kubernetes ecosystem functioning together, it would seem appropriate it integrate on as much upstream stuff as possible.
Not necessarily, ./kube-up.sh for example is no longer relevant and kubeadm doesn’t use ansible, the approaches are still evolving and it depends on what kind of cluster we want to bring up and why.
"Therefore, the scope for kubeadm is to be easily extendable, modular and very easy to use. Right now, with this v1.5 release we have, kubeadm can only do the “full meal deal” for you. In future versions that will change as kubeadm becomes more componentized,"
Would love to use kubeadm or whatever upstream provides when it is ready, it wasn’t around when I wrote my ansible bit.
The CI stuff and Demo stuff are unrelated and may or may not require different types of provisioning, the demo can/should run on any cluster regardless of how exactly it was spun up.
@hh will elaborate here on the CI in a bit.
Hi @gandelman-a
TLDR:
We’ve moved away from image creation and are focusing on deploy time customization via cloud-init using terraform. There are 50+ k8s deployment approaches out there. We have focused intentionally on ENV driven multi-cloud deploys of cncf project artifacts directly from upstream CI / release binaries and cloud-images. As @Zilman said, everything is in a bit of flux, and we welcome your feedback and appreciate your interest in contributing.
Long Version:
cloud-image curation
It supports installing on different distro flavors, installing kubernetes from packages or source checkouts, and provides lots of tweakable config variables to customize deployments.
Our evolving approach so far has been to focus on combining existing upstream cncf project artifacts (including ci builds) with stock cloud provider images at deploy time. There is a cncf-mailing list with some initial conversations on creating baseline cloud-images. As participation in that conversation grows it might make sense to start using those image artifacts.
kubernetes deployment ecosystem diversity
If the long-term plan of this repo is to showcase the many moving parts of the kubernetes ecosystem functioning together, it would seem appropriate it integrate on as much upstream stuff as possible.
In the migration away from cluster/kube-up.sh we’ve seen over 50 new approaches from the community.
It’s the early days for the Vendor/Community-Maintained cncf-k8s-conformance group and we need to plan ahead to enable swapping out components of the pipeline with as different ‘conforming’ approaches.
I think it would be great to support contrib/ansible as one of those approaches, but it would need to be underpinned by cross-cloud provisioning of the underlying infrastructure.
Many of the evolving approaches for creating cross-cloud infra are using terraform. We are looking at several of those as we try to quickly ramp up the cncf/demo to support more clouds.
One of those approaches we are exploring a bit more closely is kops, however we wanted to go ahead and get some stakes in the ground for our initially supported clouds before diving too deep into that code base.
“The scope for kops is to automate full cluster operations: installation, reconfiguration of your cluster, upgrading kubernetes, and eventual cluster deletion.... kops (unlike kubeadm) handles provisioning of resources for you. kops aims to be the ultimate out-of-the-box experience on AWS (and perhaps other providers in the future). In the future kops will be adopting more and more of kubeadm for the bootstrapping phases that exist.”
We have initially focused on plain templated cloud-init on existing cloud images to reduce our dependency vendor specific provisioning / image creation.
cncf/demo as an artifact test harness
With regard to potentially using this repo as a test harness for a future integration test suite, the ability to use source checkouts would easily allow us to build and test images for proposed patches to things running in the image (k8s, etcd, etc.).
Our intention so far was to not create more artifacts (particularly images), but to simply combine existing outputs/releases from the cloud providers and cncf projects.
I like the idea of having source builds for everything longer term and looking to integrate individual artifacts published by various CI systems or cloud providers at deploy/trigger/commit time.
multiple clouds per commit
We are currently refactoring a bit to support multiple deployments per cloud at the same time using the same creds. If we start factoring in creating an AMI or other cloud specific image per deploy I feel it may create unnecessary complexity.
Our current approach usually brings up k8s on our supported cloud providers in under 5 minutes and we are working to ensure that all artifacts are driven from environment variables. Ie TF_VAR_x
Hi guys, thanks for all the insights.. its quite valuable. I've been trying to grok where this repo is heading and I'll admit it's been a bit challenging, but this helps clear up a lot.
If I'm understanding both replies here correctly, it sounds like kubernetes provisioning/images/clouds/etc are all just a means to an end and all we really need is a kubernetes endpoint on which to run the demo/etc? I had originally thought that image builds would allow for build-time customization and expansive test matrix that could be tweaked to test different permutations of the underlying cluster infrastructure, but it sounds like that may be out of scope, at least ATM.
My original interest in this repo came from its end-to-end nature (build, provision, deploy, validate workload) and the possibilities that opens up for doing interesting testing of the whole stack. But with the lower level components out of scope, it simplifies things a bit and it might be easier to get something up and running that tests changes to CNCF projects using the demo as a validation. I'm curious if you guys hang out anywhere like IRC where we might be able to bounce around ideas, or if here or a mailing list is a better place?
Cheers
The repo is intended to be used to provision, deploy, and validate workloads.
I think the only difference in what you are describing is that we have shifted to customizing the stock cloud images at provisioning time, rather than saving a customized cloud image.
We currently gzip up all the configuration for kubernetes services + certs into cloud-config which we feed to user_data on each cloud provider.
It's meant to all be driven by TF_VARS_* which can override per project deployment sources and versions as well as the per cloud image and OS.
Our current focus is a cross-cloud / cross-project test matrix for head of each project on top of infrastructure laid down at each cloud provider. It's all meant to be tied together via a CI system which will likely be one of the components we demo.
We'll update the repo/readme this week as we finish up the kubernetes cross-cloud provisioning and move into the CI portion.
We've primarily been using github issues for conversation so far, but also hang out at:
http://chat.ii.coop/cncf/channels/town-square
See you there!