Share dependencies between bundles

[troy0820]

Is your feature request related to a problem? Please describe.

During today's community meeting a problem surfaced from how bundles do not share dependencies. As each installation is its own autonomous unit of "bundle", sharing dependencies could show merits from allowing the installations to not re-install and/or share components to one another.

e.g:

Bundle A is a front-end web app that has a mysql backend
Bundle B is a back-end that does mysql backend things that needs to do those things with the Bundle A's mysql backend

Question is: shouldn't this just be one bundle, perhaps but this contrived example is to provide a point that if it isn't, can we share dependencies between two bundles.

Describe the solution you'd like

The ability to share dependencies between the bundles meaning that we can utilize a flag --weak-dependencies to not install the second bundles dependency but make sure the dependency is installed. Whatever surface (layer) it needs to interact with that is the bundles expectation.

Should this be a mixin? Can a mixin handle this?

Describe alternatives you've considered

Looking at the PEP for Advanced Dependencies, that may be something we can do but is a longer term solution that can provide value but not specifically for this use case.

Additional context

Realizing that the advanced Dependencies PEP may resolve this, this community ask was a way to get around that implementation currently not built.

[edespong]

An issue I am having with the PEP is that I have a use-case that is only mentioned in passing but which I think could be relevant here. In essence the PEP mostly considers dependencies and sharing thereof as a concern at bundle creation time, whereas I would stress the need for it to be (also) an install-time concern.

Consider the scenario:

• I have three bundles,A, B, and MySQL that makes up my application.
• A and B are developed by different teams (i.e. I do not want a single bundle)
• A and B both need a database
• My application is deployed in many instances for customers of very varying sizes
  • For very small installations, I want A and B to share a single MySQL instance
  • For larger installations, A and B should have separate MySQL instances

I do not see that it would be possible to model this at bundle creation time, except if I create different bundles e.g. A_small, A_large, which feels very wrong as the number of combinations will quickly explode.

But I said that the PEP "mostly considers", because there is this (emphasis mine):

When an installation is created to represent a dependency, the sharing mode and group should be copied from the bundle definition to the installation so that we can quickly query for suitable candidates without evaluating every bundle definition of every installation. The sharing mode can be edited by the user, to allow for "break glass" situations that may arise where the user needs to accomplish something we didn't anticipate.

So given this, could a step toward implementing the whole PEP be to start with this special case? That you can specify the sharing mode/group on the command-line instead of the bundle definition, e.g.:

porter install mybundle --sharing-mode group --sharing-group mygroup

If two bundles have the same group, the dependency is only created on first. For different groups, or if not specified the dependency is not shared.

[reuzel]

Just a thought...

Going back to the Szyperski definition of components (independent unit of deployment, explicit interfaces, 3rd party composition), you may wonder if this use-case should solved with an extra '3rd party composition layer' that describes how bundle installations are linked (leveraging the interface they expose or consume). Bit similar to how docker-compose relates to docker. Such a composition manifest would reference the different bundles and could 'override' configurations, such as references to instance names. For @edespong use case, composition manifests could be created for the different types of customers. I can think of more (related) use-cases here: specific compositions for test or development environments, HA variants for production, etc. etc.

[edespong]

Well, @reuzel I could not agree more. What you are describing is very much the route we are currently heading down, with a slight modification. We are leveraging the custom section in porter.yaml to specify dependencies slighly differently that Porter's current approach. Then we have a separate manifest, the system definition, that does the referencing and the wiring. Finally, we are creating a program that can read, validate, and apply the system definition.

The reason why we need the custom section in the porter.yaml is because in our case the bundle authors and the creator/maintainer of the system definition are not the same. It is up to the bundle authors to define the dependencies, but up to the system definition creator to wire them up.

[schristoff]

Hey @edespong and @reuzel - thank you for all this amazing information. I'm currently working on implementing shared dependencies, and I would love to get more information on your use case/what you're doing with Porter/etc. If you ever have time, please feel very much encouraged to hop in our community call :)

[schristoff]

Solved in #2940