kubernetes/kubeadm

tracking issue for kubeadm init phases

neolit123 opened this issue · 39 comments

neolit123 commented

this tracking issue goes outside of the scope for 1.13 AKA we have items for post 1.13.

this post will summarize the work that has to be done to graduate the kubeadm phases to beta (ga?).
the issue will track PRs, assignees, but also have extra actions that have to be handled.
if you wish to work on a certain item please add a comment about it.

discussion happened around this gdoc in the 17oct/2018 kubeadm office hours meeting:
https://docs.google.com/document/d/10ZQ5erWhab3GJH-CjCBT6NgvLJQygyqPkQUsx_y5snQ/edit?usp=sharing

notes:

  • we are keeping the kubeadm alpha command for now, but moving all viable phases in the bellow list to kubeadm init phase
  • some phases are going to be collapsed into a parent phase, but optionally sub-phases to be exposed over time.
  • some phases are shuffled around or removed.
  • we have plans to support aliases where a sub-phase can call the same code as another sub-phase.
  • some phases are going to be hidden which implies that they are going to be called by the backend in a specific order, but the user would not be able to call them from CLI or skip them.

list of phases (ordered)

currently this phase has sub-phases, the idea is to collapse them into a single phase. shuffle folders around etc. this however removes the control if the user wants to write a kubelet config, modify it and then start. so we might want to expose it eventually based on user feedback.

should be a straight forward one, move all cert sub commands as sub phases.
renew needs to not be a sub-phase of certs, extra item bellow.

same as certs,
user however should be kept as an utility under kubeadm alpha kubeconfig user and not being a phase.

contains sub-phases for the control plane components. under api-server we also need to handle audit-policy (currently feature gated as a separate phase). there are plans to add aliases for certs here and eventually also have sub-sub-commands under each control plane component to 1) write config 2) write cert

similar to control-plane, optionally enable 1) write config 2) write cert as sub phases here eventually too.

hidden phase!

this should including uploading the kubelet config (currently under kubelet phase) and the kubeadm config. both should be exposed as sub-phases.

annotate-cri-socket should be part of uploading the kubelet config

renamed from mark-master.

should contain all of these, but collapsed and not exposed as sub-phases.

create (the function is CreateOrUpdate...)
allow-post-rbac-csr
auto-approve-node-tokens
auto-approve-node-cert-rotation
create-bootstrap-config-map 
create-cluster-info-rbac

should contain dns and proxy as subphases exposed in that order.

other action items:

this needs to handle the case where the creation of a token was skipped.

it's feature gated.

to be decided....
the new home for this should be "k8s.io/cluster-bootstrap/token/util"
https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/cluster-bootstrap/token/util/helpers.go

fabriziopandini commented

fyi
This was the list of phases proposed some time ago

  preflight                  Run master pre-flight checks
  certs|certificates         Generates all PKI assets necessary to establish the control plane
    /ca                        Generates a self-signed kubernetes CA to provision identities for components of the cluster
    /apiserver                 Generates an API server serving certificate and key
    /apiserver-kubelet-client  Generates a client certificate for the API server to connect to the kubelets securely
    /etcd-ca                   Generates a self-signed CA to provision identities for etcd
    /etcd-server               Generates an etcd serving certificate and key
    /etcd-peer                 Generates an etcd peer certificate and key
    /etcd-healthcheck-client   Generates a client certificate for liveness probes to healthcheck etcd
    /apiserver-etcd-client     Generates a client certificate for the API server to connect to etcd securely
    /sa                        Generates a private key for signing service account tokens along with its public key
    /front-proxy-ca            Generates a front proxy CA certificate and key for a Kubernetes cluster
    /front-proxy-client        Generates a front proxy CA client certificate and key for a Kubernetes cluster
  kubeconfig                 Generates all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
    /admin                     Generates a kubeconfig file for the admin to use and for kubeadm itself
    /kubelet                   Generates a kubeconfig file for the kubelet to use. Please note that this should be used *only* for bootstrapping purposes.
    /controller-manager        Generates a kubeconfig file for the controller manager to use
    /scheduler                 Generates a kubeconfig file for the scheduler to use
  audit-policy               Generates the audit-policy to use in the API server configuration (featureGated)
  controlplane               Generates all static Pod manifest files necessary to establish the control plane
    /apiserver                 Generates the API server static Pod manifest.
    /controller-manager        Generates the controller-manager static Pod manifest.
    /scheduler                 Generates the scheduler static Pod manifest.
  etcd                       Generates the static Pod manifest file for a local, single-node etcd instance
  init-kubeletconfig         (featureGated) Writes base configuration of kubelets to disk
  upload-kubeletconfig       (featureGated) Uploads kubelet config to a configMap and use it as a configSource for this node
  upload-config              Uploads the currently used configuration for kubeadm to a ConfigMap
  mark-master                Marks a node as master
  bootstrap-token            Makes all the bootstrap token configurations and creates an initial token
    /token                     Creates an initial bootstrap token to be used for node joining
    /cluster-info|clusterinfo  Uploads the cluster-info ConfigMap from the given kubeconfig file
    /allow-post-csr            Configures RBAC to allow node bootstrap tokens to post CSR in order for nodes to get long term certificate credentials
    /allow-auto-approve        Configures RBAC rules to allow the CSR approver controller automatically approve CSR from a node bootstrap token
  addons                     Installs required addons for passing Conformance tests
    /dns                       Installs the dns addon to a Kubernetes cluster
    /proxy                     Installs the proxy addon to a Kubernetes cluster
  selfhosting|self-hosting   Makes a kubeadm cluster self-hosted (featureGated)
  • 👍2
timothysc commented

This looks good, but I think we might want to change some of the names, and obviously move selfhosting into it's own space.

fabriziopandini commented

@neolit123 could you kindly change this issues into a tracking issue /help-wanted that we can use to assign part of this effort to volunteers...

I think that @ereslibre, @RA489, @Klaven can help on this effort once the PR with the first phase implementation merges; ideally who takes in charge a phase should implement related subphases as well

  • 👍1
neolit123 commented

are we going to discuss the list of phases this weeks kubeadm office hour? we need a solid list...

ereslibre commented

I won't be able to join tomorrows kubeadm office hours :(, but count me in for any ongoing effort!

  • 👍1
neolit123 commented

now that we have the list of phases i will update this tracking issue tomorrow.

  • 👍1
fabriziopandini commented

@neolit123 thanks for this write-up. It is great starting point
@ereslibre, @RA489, @Klaven time for volunteering!

neolit123 commented

a couple of requests for volunteers :)

  • please, don't mention this issue in commit messages, because it will spam it on commit rebases.
  • in PR descriptions use, Ref #this-issue-num instead of Fixes #this-issue-num, so that we don't close it on merge.

thanks.

ereslibre commented

For starting I can take the certs one!

  • 🎉3
neolit123 commented

@fabriziopandini
btw i'm not exactly sure how we can sync the work here, unless phases are implemented in the presented order. the code in runInit() would still have to be present and we have to hold PRs that implement say mark-control-plane in the new fashion. otherwise we are going to break master.

fabriziopandini commented

@neolit123 I'm confident we can sync via slack/this issue

BTW, I'm taking phase kubelet-start

neolit123 commented

@fabriziopandini ok, sounds good.

kubelet-start has a bit of a blocker in terms of the "stop" part as per @chuckha 's PR here:
kubernetes/kubernetes#69985
we need to discuss what to do with that.

fabriziopandini commented

taking kubeconfig

neolit123 commented

taking control-plane but really want the PR that removes placeholder docs merged...

neolit123 commented

taking etcd

fabriziopandini commented

@RA489 takin mark-control-plane

  • 👍2
yuexiao-wang commented

taking certs/pkiutil

  • 👍1
ereslibre commented

@fabriziopandini @neolit123 Started kubernetes/kubernetes#70331, still in WIP, but you can modify the comment linking to this PR, thanks!

yuexiao-wang commented

@neolit123 my task has done in kubernetes/kubernetes#70336.
where is certs/upgrade which i cannot find?

  • 👍1
neolit123 commented

@yuexiao-wang

my task has done in kubernetes/kubernetes#70336.

will review it next week. thanks.

where is certs/upgrade which i cannot find?

i meant phases/upgrade:

https://github.com/kubernetes/kubernetes/tree/master/cmd/kubeadm/app/phases/upgrade

yuexiao-wang commented

@neolit123 thanks
taking app/phases/upgrade

yagonobre commented

I can take alpha phase certs renew

yuexiao-wang commented

taking printing the join command

  • 👍1
yagonobre commented

alpha phase certs renew was done in kubernetes/kubernetes#70333, I can take bootstrap-token and addons.

bart0sh commented

I can take this: "app/phases/upgrade needs to be moved to generic kubeadm utils (not a phase)"

bart0sh commented

@yagonobre thanks for pointing it out. won't take it then. I thought it's not assigned:

 app/phases/upgrade needs to be moved to generic kubeadm utils (not a phase)

assigned: TODO
neolit123 commented

sorry, just updated it.

yagonobre commented

What is the expected subphases on bootstrap-token? I'm taking upload-config in the meantime.

yagonobre commented

taking enable-dynamic-kubelet-config

neolit123 commented

@yagonobre

What is the expected subphases on bootstrap-token? I'm taking upload-config in the meantime.

as commented there, the listed sub-phases should be just collapsed in a single phase.

should contain all of these, but collapsed and not exposed as sub-phases.

  • 👍1
yagonobre commented

ok, taking bootstrap-token again

neolit123 commented

PSA, the phases need to be merged in the above order as we don't want to break e2e tests.

fabriziopandini commented

@neolit123 annotateCRI IMO doesn't deserve a first level phase because it ideally should go away as soon as possible.. What about having annotateCRI as part of the upload kubelet config subphases? (Run Upload kubelet config writes the config map and annotates CRI)
@yagonobre wdyt?

  • 👍1
neolit123 commented

@fabriziopandini
SGTM, i will update the list.
responded here too: kubernetes/kubernetes#70405 (comment)

neolit123 commented

@RA489 hi, do you have a PR in the works for mark-control-plane?
we might be able to merge it this week if so.

neolit123 commented

the work here is done!

the only remaining items are these, but we can track then individually:
kubernetes/kubernetes#70346
kubernetes/kubernetes#69979

we did a pretty nice organization here and managed to tackle the list of tasks pretty well!!
thank you @fabriziopandini @yagonobre @yuexiao-wang @ereslibre @RA489

i will also send a shoutout to everyone for the community meeting on Thursday.

  • 🎉3
  • ❤️3