cncf/demo

e2e CI setup (WIP)

Opened this issue · 2 comments

namliz commented

Manifest / Version controlled config

  • Bake into container # 1

    • Terraform files that are used to spin up the cluster
    • and the version of Terraform binary

  • Kubernetes binaries, stick to major releases for now, just reference in TF_vars for above

  • Overlay deployment (kubectl apply -f {Weave, Flannel, Calico}.yml)

    • These are all pod manifests and images maintained upstream. Whether or not we care to override minor versions specified inside them is an open question

  • Prometheus & Pushgateway - not currently baked anywhere, bake and specify image in TF_vars

  • CoreDNS / KubeDNS - former is not yet currently baked anywhere, bake and specify image in TF_vars

Terraform should generate appropriately the yaml files for all of the above and place in /etc/kubernetes/manifests.

  • Runner wrapper for Demo/CI
    • Helm / Self-deploying pod with kubectl embedded / some other way - TBD
    • Dump the outer manifest via TF_vars into a configMap to be accessible within the cluster (experimental)

To deploy in a sane manner we end up with a bundle, but at the same time, we want to somehow build on every subcomponent of the bundle as well. This requires multiple CI pipelines, trigger new bundle builds whenever one of the components get updated, start new clusters on every new bundle.

Additional reading:

Templates+Parameterization proposal has been stuck for a long time, no plans until after(?) 1.7.
kubernetes/kubernetes#23896 (comment) - "Helm and Templates are different tools and have different strengths".

Artifactory / Container Registry / Build pipeline - TBD

Trigger run on every new Terraform, provisioning files changes, overlay image, Prometheus, Demo/CI artifacts.

Process:

Input: Manifest file with all the versions and desired flags
Ouput: Submitted to cncfdemo.io backend (which generates a uuid) and saved as statically hosted file
Cluster deployed with uuid, pings endpoint after successful bootstrap, appends information during run

Run successfully completes on final event or times out between pings.

namliz commented

Using Google Cloud Container Builder on a fork of this repo to the side as a temporary measure.