vmware-archive/wardroom

Updates to Calico CNI Deployment

Opened this issue · 8 comments

joshrosso commented

I'd like to make the following changes to how wardroom does its Calico deployments.

Please let me know if they sound good and I will make the changes.

  1. Increase the version to 3.6.X from 3.3.X

  2. Store the calico manifest as a jinja template

  3. Add variables to ensure calico respects the variables set for the cluster's podSubnet as its IPV4 pool.

  4. Set IP-in-IP to crossSubnet (https://docs.projectcalico.org/v3.5/usage/configuration/ip-in-ip#configuring-crosssubnet-ip-in-ip) rather than the default always.

    a. This will ensure that IP-in-IP encapsulation is not used for all traffic, causing unnecessary overhead.

/cc @craigtracey @scottslowe

scottslowe commented

No objections to any of the suggested changes. Are you going to make these changes against master, and then we'll port over to the v1.13.x branch?

craigtracey commented

I am all for bumping versions of Calico. Which branches would you like to apply this to?

As for storing the Calico manifests as Jinja templates, I would rather that we not do that, as it will make things a lot more rigid/confusing than they are today. And, it would be another burden for us to maintain.

The repo does have a mechanism to manipulate any applied manifest (called modify_manifest), and I would prefer that we take that route...by documenting modify_manifest options that a user may employ.

craigtracey commented

As for podSubnet, we intentionally do not set this today, as this will vary wildly between deployments. Again, I think we should very clearly document how this modify_manifest library can be leveraged.

An example:
https://github.com/heptiolabs/wardroom/blob/master/swizzle/examples/calico.yml#L15-L21

joshrosso commented
joshrosso commented
craigtracey commented

Wardroom does support setting podCIDR arbitrarily, but there is no explicit check to ensure that Calico also sets that same CIDR. It has always been the intention that the user needs to validate their configuration before deployment. If we want to write a warning about the mismatch, I am fine with that.

Carrying a template means that we need to fast-follow releases from Calico, with no guarantee that our variables will stay in lock step with theirs. This leaves the user in a spot where a configuration that worked once, may not work again....and probably fails silently, as it is just a template. While modify_manifest does not have strict enforcement currently, it would be somewhat trivial to add a parameter that forces all rules to be evaluated and modified.

joshrosso commented

Wardroom does support setting podCIDR arbitrarily, but there is no explicit check to ensure that Calico also sets that same CIDR.

Are there cases where podCIDR for the k8s cluster should not be the CIDR used for Calico? Why not just reuse this variable in the Calico template (or always set it via modify_manifest?

Carrying a template means that we need to fast-follow releases from Calico

When a new Calico release happens, the new yaml needs to be copied over and the variables re-established (or someone does diff work). Having the template-tized manifest makes the ansible easier to reason with. This overhead is not substantial and better as having config management pull a manifest from the sky and run a dynamic mod on it probably won't fly in some cases / environments.

Again, just my view of the world. I am OK with strictly using modify_manifest. I do however feel more strongly about the podCIDR 😄 .

joshrosso commented

As part of this ticket, I'll provide documentation around modify_manifest.

We do need to find consensus around our philosophy for plumbing podCIDR.

@craigtracey @scottslowe