Investigate implications of new Packer licensing

HashiCorp recently changed the license on some of their open source projects to BSL 1.1. This may or may not present legal issues to the use of Packer in the image-builder project.

Let's communicate with the CNCF to see if there's an official position on the new license, and determine whether there are implications for image-builder.

image-builder currently uses Packer ~~v1.9.1 and is updating to~~ v1.9.2, the last version under the MPL. The project will stay pinned there until we have a resolution to this issue.

image-builder/images/capi/hack/ensure-packer.sh

Lines 23 to 24 in d2a2af9

    
           # **DO NOT** change the Packer version: v1.9.2 is the last release under the MPL v2.0 license. 
        
           _version="1.9.2"

See also:

Some points of note:

Security fixes will be backported under MPL 2.0 through December 31, 2023.

Taken from https://www.hashicorp.com/license-faq
If you need further clarification with respect to a particular use case, you can email licensing@hashicorp.com. Custom licensing terms are also available to provide more clarity and enable use cases beyond the BSL limitations.

Taken from https://www.hashicorp.com/license-faq
It seems some libraries and SDKs are still ok but doesn’t seem to include anything for Packer as far as I understand it. This would rule out switching to our own binary leveraging the Packer Go modules (e.g. resurrecting the cli project we recently got rid of)
From what I understand, I don’t think the image-builder project itself would be considered a competitor but many of our users that run image-builder to create their images would be (e.g. Microsoft building for CAPZ). Maybe it would be possible for us as an OSS project to build and provide the images for the CAPI projects but that wouldn’t be ideal (cost, inflexible, maintenance burden).

/cc @abhay-krishna

thread in the cncf on this topic: cncf/foundation#617

Recommendations from CNCF: https://github.com/cncf/foundation/blob/main/source-available-recommendations.md

We've already pinned the version but thats a short term solution at best. I don't believe their is an alternative available to Packer that we could switch to, at least not without substantial work needing to be done.

As discussed in this slack thread: https://cloud-native.slack.com/archives/C0MP69YF4/p1692111612303149?thread_ts=1692108926.999039&cid=C0MP69YF4

We should submit a request for license exception from the CNCF Governing Board (not sure how to do that yet) just to be on the safe side. It's possible we won't need the exception but its best to check to be sure.

If no one else gets to it first I'll try and get the request in Thurs/Fri this week.

We should submit a request for license exception from the CNCF Governing Board (not sure how to do that yet) just to be on the safe side.

From https://github.com/cncf/foundation/blob/main/source-available-recommendations.md:

In either case, while a project's maintainers may request a license exception from the CNCF Governing Board, it is extremely unlikely that one will be granted for the use of a dependency under a source available, non-open source license.

The CNCF has granted exceptions in the past for hashi libraries etc being under MPL, but that is still an "open source" license rather than "source available". I don't think they're likely to approve one here, the CNCF approved license list (no exceptions necessary) are all relatively free use license (ASLv2, MIT, ... https://github.com/cncf/foundation/blob/main/allowed-third-party-license-policy.md) and BUSL is pretty much the opposite.

That said, to file a request you file an issue at https://github.com/cncf/foundation/issues such as cncf/foundation#187

Has anyone looked at potential alternatives to packer? I was not aware of any but did a quick search and found linuxkit. Any ideas if it could replace packer? (Looks like they used to build OS images for Kubernetes with it in this project but it hasn't been updated in 5 years.)

See also kubernetes-sigs/cluster-api#9181.

Opened exception request: cncf/foundation#625

Linuxkit is used by docker desktop (see adopters list) which ships kubernetes.

It seems like switching would be a lot of work though.

It seems like switching would be a lot of work though.

Exactly this. Any alternative that's not api-compatible with Packer would need some serious consideration and input from many users before a decision is made.

It appears that Packer 1.9.x patches will keep the MPL 2.0 license, although BUSL is already in the main branch. See #1269.

The FAQ notes:

Security fixes will be backported under MPL 2.0 through December 31, 2023.

I'm wondering if towards the end of the year there might be more efforts to fork projects like packer similarly to OpenTofu

I think if there were going to be a similar effort for projects other than terraform there would've been some indication of that by now.

We did have alternative approaches in this project that did not require packer, though they were removed in #1175 because they were not well maintained (I was one of the guilty parties!)

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

The exception request has been rejected: cncf/foundation#625 (comment)

Yeah. Quite disappointing. 😞

I'm talking with the other maintainers to decide how we proceed. Once we have something we'll update.

👋 Following this discussion, one point we should clarify: are we bundling Packer today at all and redistributing it?

From what I can recall, for image-builder Packer is used as the primary mean to create artifacts, but it shouldn't be shipped as an artifact itself. While the license exception makes sense it was closed, I wonder if using Packer as a tool, and not embedding it in any form categorizes as fair use.

Not a lawyer, but would like to get some clarification if the above is also out of scope.

@BenTheElder do you know any project using almost-closed-source dependencies in a similar way?

Yes, we are packaging it and distribute it in the container image that we build and provider for others to use. As far as I understand this counts.

Myself and the other maintainers have been discussing this and have decided that for right now we're not going to be making any changes. We're still pinned at the latest version that uses the old license. This is obviously not a long term solution but it works for now and gives us a chance to not rush in to anything.

There has been some rumours that the IBM purchase of HashiCorp could possibly lead to them reverting the license change back to a real open source license. The likelihood of this is admittedly very low I think but it would be ideal if that is the outcome so we're hoping we can "buy some time" until that is either confirmed or not.

Failing that, the most likely course of action would be to remove Packer from the container image (and possibly not provide a container image at all) and have end users need to bring their own Packer binary that would then be compliant with the license. This isn't ideal as it impacts a lot of our users who would then need to build their own images and it would cause problems for testing and automated CI/CD building of images.

For completeness - replacing Packer with an alternative is almost certainly not going to happen. Rewriting image-builder to use an alternative (if there even was such a tool) or building something custom would be a very large development undertaking and not one we have the resources for. As we don't even have consistency across all providers and OSs in image-builder it would also be incredibly difficult to build something new that didn't break something for some use case.
We also discussed the possibility of forking Packer with the open source license like OpenTofu but similarly we don't have the resources to manage that and it seems no one else in the community are interested in taking this on.

I hope that clears some things up. None of this is "set in stone" and we're trying to be flexible but realistic where possible. 😄

replacing Packer with an alternative is almost certainly not going to happen

I agree with all of what @AverageMarcus says above. In particular, I don't think it's likely we could create a fully functional replacement tool that would support all of the use cases represented by the 180 build- targets in the Makefile. Instead I can imagine ~~two~~ three scenarios:

image-builder continues to use Packer v1.9.5, switching to a fork so we can apply our own patches if critical CVEs arise
image-builder continues to use Packer v1.9.5, but providers begin to add alternate, supported methods of building the same images
image-builder switches to a "bring your own Packer" model to avoid actually bundling it in the project

The first scenario still implies a fair amount of work making sure the fork is supported and up-to-date, and that we somehow can be made aware of security issues in upstream Packer and are able to fix them without running afoul of Hashicorp licensing.

The second scenario would involve something like changing the build-azure-sig-azurelinux-3 target to call the confusingly named Azure Image Builder service (for example) with new JSON or Bicep scripts to produce an image. This could help us move forward, but would be kind of a free-for-all that implies increased support from each CAPI provider, which may not be realistic. It's also not really a community, open source approach.

The third scenario could work, but raises the bar of entry to use image-builder. We also would need to get CNCF legal to agree that was adequate.

Thanks for the context folks. One question: why are we distributing a container image with packer bundled within? Can we stop publishing it and instead provide guidance and documentation for the intended use?

Not including Packer in the container image would mean the container isn't actually usable as Packer is required for image-builder to function. We're considering this but it would impact our end users that rely on this image to make use of image-builder so we're reluctant to do so until needed.

Once way I think we might be able to get around this (though I'm not 100% sure if this avoids the license or not) is if we have the dependencies fetched at runtime when launching the container image. This could possibly have implications for users that run it in more locked down environments and could cause the image to stop working if an external dependency isn't available at runtime.

I would avoid any sort of runtime dependency that does not meet CNCF licensing requirements, regardless of distribution or linkage.

So:

tool we ship to user fetches at runtime ❌
tool we run in CI for development purposes fetches at runtime ✅

(e.g. we fetch shellcheck in kubernetes CI, but users of the Kubernetes release are not expected to use this, and we do NOT host it)

but the ultimate authority on this is CNCF / LF legal.

There has been some rumours that the IBM purchase of HashiCorp could possibly lead to them reverting the license change back to a real open source license. The likelihood of this is admittedly very low I think but it would be ideal if that is the outcome so we're hoping we can "buy some time" until that is either confirmed or not.

We're not depending on this elsewhere in Kubernetes and I wouldn't bet on it, given no intention has been announced. It also may be quite some time before the deal closes even if there were clear intent. ("end of 2024" is the target per the IBM announcement)

Forking pre-BUSL packer could be acceptable but sounds difficult to sustain and probably should be discussed as part of a larger effort under the CNCF or similar.

Maybe I'm misunderstanding @BenTheElder but it sounds like you're suggesting that we should stop our reliance on Packer completely, is that correct?

I don't think that's the only valid option but I also don't think

Once way I think we might be able to get around this (though I'm not 100% sure if this avoids the license or not) is if we have the dependencies fetched at runtime when launching the container image.

Is a workable approach.

The CNCF recommends freezing at the current version as a short term approach but I think either an exception or a fork will be necessary for long term:
https://github.com/cncf/foundation/blob/main/source-available-recommendations.md#recommendations

If there aren't sufficient resources for a fork (or switching tools), you could request an exception (see doc above) and maybe also seek clarification if the "we download it at runtime" approach is acceptable.

If there aren't sufficient resources for a fork (or switching tools), you could request an exception

We already tried that and it was rejected. 😔 See cncf/foundation#625 (comment)

I tried to give my 2 cents here

TL;DR;
I think the usage the project is doing of packer should not be classified a runtime dependency, because it can lead to confusion.

AFAIK, the project is using packer as a build tool, and packer is not included at all in any of the images (OVA, ami etc) which are the ultimate output of the project

The "commodity docker image" which is publiished as a release artifact, instead is just a commodity to avoid to install dependency locally, not the core output of the project; I understand that this is not ideal, but in case this can help in addressing the issue, IMO the image builder project can consider stopping publishing those "commodity images". Users can always install the required tools locally, or eventually build the "commodity image" locally if they prefer.

Yes that sounds reasonable to me, if the docker image is just used at development time we can stop pre-installing packer when a new [BUSL] version is needed for patch reasons (... and start pre-installing it again if the license changes [back]).

It looks like we're only dodging a ~10-14MB download.
Contributors in an airgapped environment can import the binaries alongside the container image. See e.g. the shellcheck scripts in k/k that will install in CI if the binary isn't available / the right version.

	# DO NOT change the Packer version: v1.9.2 is the last release under the MPL v2.0 license.
	_version="1.9.2"