Repo structure when adding more sources
embano1 opened this issue · 9 comments
I'm currently working on creating a VMware Horizon source adapter (using the tested code base from the VEBA project).
The current repository structure is set up assuming there's only one source (vSphere). How do we want to handle new/multiple VMware sources in this repo?
My gut feeling tells me that we should have separate repositories for different sources, that will allow us to release them independently and also follow the sample source template: https://github.com/knative-sandbox/sample-source
Would it be too complicated to rename this repo to vsphere source?
My concern is that additional repos adds additional overhead whenever considering automation and processes, even including manual processes like "check to see if there are any open issues from users".
I'd be inclined to package things as follows:
- One "combined source" manifest which installs all the different types and a single controller that manages all of them.
- Standalone "only X" manifests for each source type for users that only want a single source and don't want the extra resources on the cluster.
Sharing a repo also makes it easier to share things like libraries; if we split them into many repos, we end up needing N+1 repos (one for each source, plus one for "pkg". I'm not sure that model worked great for Knative; it certainly has ended up producing a need for a lot more movement and machinery. The initial goal was that each component could stand alone, but the real flexibility comes in versioning -- it's possible to release a patch fix for one eventing component without needing to CI the rest of the system.
I'm not sure that this repo will end up being big enough to have that problem, and I'm willing to face it when the time comes.
In terms of creating an "all in one" plus individual releases, I'd probably end up creating one directory for each source (and a separate directory for each binding), and then have a collected directory alongside them that symlinks in the correct files.
I agree with @evankanderson (especially around the overhead around repo management - also from user's/contributor's pov).
Regarding the manifests, all-in-one as well as individual components make sense (as done in other kn repos). My question was more around the source code, e.g. everything under /cmd and /pkg.
I think having the container entrypoints under cmd makes sense. I don't have a strong feeling about pkg either way. https://github.com/golang-standards/project-layout#pkg suggests that this is a directory for "Library code that's ok to use by external applications". I might prefer to put things in /internal or just at the top level. This blog post seems to have more details: https://travisjeffery.com/b/2019/11/i-ll-take-pkg-over-internal/
Note that Kubernetes and a number of other projects use pkg, so the big argument in favor of putting things in that directory is making things more familiar for contributors from those projects. I'm not sure it's a big win either way, but I can see the arguments for keeping pkg so that other content like docs, testing, configs and so forth sit "next to" the code, rather than "interspersed with" it.
Sorry, I do not want to change pkg or any of the other folders. The question I had was how to structure multiple sources under these folders then. E.g. /pkg/<source>/[api|injection|...]?
If it helps, in Knative RabbitMQ Eventing we have exactly that, maybe it could be prettier like having each thing in its own separate kind of module, but there in the link you can take a look
I think what RabbitMQ is doing makes sense. The key is that the go compiler will "pull" from the cmd/$containername through the rest of pkg. In some cases, like pkg/reconciler/$typename, it's clear how to separate out the different sources. In other cases, like (hypothetical) pkg/cloudevents/normalize, we may end up having a common library that doesn't fit into a per-source model.
I like that and I'm in favor of following that approach.
Update:
Based on the above conversation I'm proposing the following structure (based on the existing directory layout) using vsphere and horizon Source as example:
cmd
|- vsphere-adapter
|- vsphere-controller
|- horizon-adapter
|- horizon-controller
|- horizon-webhook
config [will include all source definitions in flat folder]
plugins
|- vsphere
|- horizon
pkg
|- apis [will include all definitions in existing structure]
|- client [auto-generated]
|- reconciler
|- vspheresource
|- vspherebinding
|- horizonsource
|- vsphere [vsphere shared libs]
|- horizon [horizon shared libs]Thoughts?