Late binding of extra spfs layers for spk packages

[jrray]

We had a case where someone published a package containing configuration files and this package was used widely across many different DCCs and baked into spawn targets. Then, the package maintainer wanted to push out a change to this configuration file, for example to change a server name. They didn't realize that it wasn't possible to retroactively modify baked targets and shouldn't have put the config file into the spk package.

However, it would be possible to support something like this and not require users to find some external way to manage configuration. The idea is similar to docker volumes but these volumes would be spfs layers that also get published to the spfs central repo. The desired workflow goes like this:

1. Create an spfs layer containing configuration and tag it with some name.
2. Add the name of the spfs tag to a new place in the install section of an spk recipe.
3. Whenever this spk package is included in a runtime, the additional spfs layers defined in the install section are pulled and added to the runtime layer stack. Mechanism to accomplish this t.b.d., maybe involving annotation layers.
4. If changes to configuration are needed, the spfs tag can be updated with different content.

We don't want to introduce any kind of late re-solving of packages, but this would provide a way for a baked environment to still contain some amount of dynamic content. This isn't necessarily specific to spawn either; anyone attempting to use the package in question in spk env or otherwise would need to receive the configuration too.

[rydrman]

From the meeting today:

• basically a question of how config-type data/files should be managed in spk
• the goal is not to need to re-invoke the solver
• this is probably not only going to be used for configs, so we need to lean into that
• can we work this in as another publishable data type in spk? maybe even something that can be generated for you with a single command invocation but still generates an artifact in the repo that can be tracked.
• final question is how do these get into the env? is it declared in the package and handled by spk (eg baking a tag instead of a digest) or in spfs through some kind of dynamic setup

[rydrman]

From the meeting today:

• are config files another section or component in an existing package?
• maybe there's some part of the build process that generates config data?
• feels like we need a baked env spec of some kind in spk first, and if that spec could "float" packages and allow them to change then we would have all the foundational pieces needed to build this workflow and possibly add new metadata and api types for configs specifically