Drastic reduction of bv_maker

[sapetnioc]

Now that we are using containers and that we moved code to Git, we have the possibility to remove a huge amount of code from the project. I believe we may completely get rid of bv_makerand bv_env. I made some proof of concept tests in a personal GitHub project where I used git submoduleto manage sources. I also created a branch to modify brainvisa-cmake to be able to add a CMakeLists.txt in my new project and do a compilation with only CMake. It worked with a small side effect in projects that define a CMake use file. The relative location of this use file is hardcoded and must be changed. In these projects, I added a if ( NO_BV_MAKER ) clause to keep them compatible with actual bv_maker.There is still things to do for docs and tests but I believe it is not a huge work (it has taken me two days for the compilation POC).

To summarize, I propose to remove bv_envand keep a very small bv_maker` where :

• Each distro would be in a git project using git submodule for components selection and sources management
• bv_makercommands would simply call git, CMake or CTest and eventually summarize output.

[denisri]

It will probably be a good path to reduce code maintenance, yes...
But please don't go too fast, because

1. un-containerized builds will not be possible any longer (this is not dramatic),
2. svn support will be dropped, whereas several users (including myself) are still using personal projects on the bioproj/svn server, and need time to move their code to another server or to convert to git.

[sapetnioc]

For now the only user visible modification is the modified CMake files in some projects (backward compatible). The rest is in a dedicated bv_maker_killer branch and in an experimental project. So I can continue working slowly on this. Even if I ever end up quickly with something usable, there is no rush to adopt these changes.

[ylep]

There are three separate aspects to this:

bv_env

I feel like we could get rid of this one, since we now support only builds in controlled environments (container / VM). However, the bv_env executable and bv_env.sh will need to stay for compatibility with user scripts, we can just make them a no-op.

bv_maker sources

When we discussed the integration of Git in bv_maker around 2014, we ruled out submodules because a submodule has to be pointing to exact revisions of a repository (referred to by its sha1). As far as I know there is no way to point to e.g. the master branch of a subproject. This means that we cannot have a superproject “distro” repository that automatically tracks the latest updates from each project, without writing additional code. Submodules are good for freezing the status of a distro, not for day-to-day incremental work.

On the other hand, I feel like the current implementation of Git repositories in bv_maker sources / bv_maker status is quite robust and maintainable ; at least I spent a great deal of time and effort to make it so 😅. The components_definition.py mechanism could definitely be modernized a bit (e.g. rewritten to declare projects in JSON or YAML?).

There are community tools out there that could maybe do the job, such as vcstool, see brainvisa/brainvisa.github.io#12 for a discussion of requirements, pros and cons.

bv_maker configure

Most of the logic is already written in CMake, apart from the top-level logic (mostly listing the components of a distro) which is written in Python as part of bv_maker. I am not sure I like the idea of moving that logic from Python to the mess of a scripting language that CMake is, but if you are willing to do so, I don’t mind...

[sapetnioc]

Submodule are defined as URL and branches. For instance here is a sample of the content of .gitmodules file that define the module in the main project (this file is managed via git submodule command):

[submodule "aims-free"]
	path = aims-free
	url = https://github.com/brainvisa/aims-free
[submodule "aims-gpl"]
	path = aims-gpl
	url = https://github.com/brainvisa/aims-gpl
[submodule "anatomist-free"]
	path = anatomist-free
	url = https://github.com/brainvisa/anatomist-free
[submodule "anatomist-gpl"]
	path = anatomist-gpl
	url = https://github.com/brainvisa/anatomist-gpl
[submodule "axon"]
	path = axon
	url = https://github.com/brainvisa/axon
[submodule "brainvisa-cmake"]
	path = brainvisa-cmake
	url = https://github.com/brainvisa/brainvisa-cmake
	branch = bv_maker_killer

I have been able to use git submodule to pull, and also commit, in all sub-repositories. But I noticed that any change in a sub-repository (e.g. git pull) is seen as a change in the main project that can be put in a commit. I suppose that is is connected with what you said about the sha1 used to store sub-modules status.

Therefore, I believe that using submodule is still an option but I am not against using something with more features like vcstool. For me the important point would be to put components definition out of brainvisa-cmake, whatever the format and underlying implementation we choose. I see several customization advantages to have a repository (or repository+branch) per distro that would contain, among other things, the list of component sources.

Another thing that is important to me is the quality of the sub-repositories status report currently displayed by bv_maker sources. So we may also choose to keep current implementation for that.

[sapetnioc]

With the use of casa-distro, I was finally able to use a much more simpler version of bv_maker and to use it for months. I still think it is a good idea but I also believe we do not have the energy to work on this now. I prefer to close this issue. It (or another one) will be reopened later if necessary.