Compile and deploy only the required binaries
Opened this issue · 9 comments
Hello,
QiBuild relies on the qiproject.xml files to know what are the dependencies between projects, and the CMake API helps using them in practice for C++.
This is good, but when compiling projects implies compiling fully every other dependency, whereas only a some parts are actually required in the CMake. This is too bad since CMake actually lets you build only what is needed for your target.
As a consequence, a qibuild make, install, deploy and package will take uselessly more time, and bring unused binaries in the result. Let's fix this!
I need suggestions on what would be the best way to provide this feature.
My initial idea is to have an option in qibuild configure to allow the generation of specific CMake code that would put together all the targets, I call it the "one cmake" option. Is it cool?
This is too bad since CMake actually lets you build only what is needed for your target.
Beware, I think this assertion is untrue/imprecise.
On the one hand, CMake will let you build only what is needed to build another target: you don't need to "make all", just "make my_test", and the minimal subset will be built.
On the other hand, the CMake install rules seem to rely on the fact that "make all" is called beforehand. (that is what I recall from the last time I looked into it, and talked about it with Dimitri).
My initial idea is to have an option in qibuild configure to allow the generation of specific CMake code that would put together all the targets, I call it the "one cmake" option. Is it cool?
So, one would only need to call qibuild configure
- when the worktree is initialized
- when projects are added/removed
right?
I think it is cool :
- it reduces the scope of qibuild (which would only be called once before cmake).
- it will lead to faster parallel builds: your proposal would let -j do all the job without qibuild's -J (which is buggy and coarser grained).
By the way, I think this is what catkin does.
That being said, before digging into qibuild work, I think it is important to see what raw cmake enables these days (especially regarding export targets).
So, one would only need to call qibuild configure
- when the worktree is initialized
- when projects are added/removed
Also when you update toolchains (with added / removed projects), or change a config.
You may need to reconfigure to build in release, too.
But you are right that we are stuck at the installation level (unless we make new tricks with CMake), since make install does not systematically select what we want to install...
CMake export looks interesting to distinguish "staging" from "installing", but in practice it does not help having select targets in make install, in the case where you really want to install them.
However I see that there is a notion of components in cmake that can be installed with alternative commands (from http://stackoverflow.com/questions/9190098/for-cmakes-install-command-what-can-the-component-argument-do):
cmake -DCOMPONENT=my_component -P {your_build_dir}/cmake_install.cmake
Since we already use custom CMake code in qi_stage_lib, we could update it to systematically produce new components (one per target staged) and include the dependencies in them.
As a result, joint with a single CMake file regrouping the projects, you could install "only what's necessary" to your project. What do you think of that?
Note that qiBuild already uses CMake components: for buildtime/runtime/testtime notions
So it would be a lot of refactoring if you want to use components for targets
This issue lies deep within qibuild's design: when you depend on a qibuild project you can only choose between these 3 types of dependencies (with the qiproject syntax).
There isn't any finer-grain control, the purpose of qiprojects dependencies is to abstract the CMake layer (projects can be other languages not built by CMake)
I remember we tried the possibility of a common top CMakeLists (Dimitri probably had a way of generating it)
So in theory yes that could be interesting
So, one would only need to call qibuild configure
- when the worktree is initialized
- when projects are added/removed
edit: Yes, you wouldn't need qibuild anymore, only CMake but that would bring another set of drawbacks:
You may need sometimes to work on CMake/debug CMake code, and then having a really heavy configure time could be painful.
When you checkout/ change branches, CMake detects the CMakeLists.txt dates changed and re-runs configure 😦
Whereas with a small qiproject, even if the install is longer because of useless targets in the dependencies, you're guaranteed to have a quick incremental re-configure and re-install.
Just my 2 cents 
CMake export looks interesting to distinguish "staging" from "installing",
I suspect it would enable us to drop lots of qibuild-specific code and make qibuild compatible with raw-cmake upstream targets.
I also suspect it would help us drop the cmake cache hacks/tweaks/optimizations in qibuild. (Which may in turn make incremental calls to cmake faster).
but in practice it does not help having select targets in make install, in the case where you really want to install them.
This is my understanding too.
However I see that there is a notion of components in cmake
Yes. This is how qibuild distinguishes what it needs to install for runtime vs test vs devel.
Since we already use custom CMake code in qi_stage_lib, we could update it to systematically produce new components (one per target staged) and include the dependencies in them.
As a result, joint with a single CMake file regrouping the projects, you could install "only what's necessary" to your project.
There will be tricks.
For instance, if libhelloworld depends on libhhello, you still don't know if libhelloworld-dev depends on libhello-dev or just libhello (depending if the use of libhello is private or expose in libhelloworld public headers). (well, target_link_libraries has a PRIVATE|PUBLIC|INTERFACE option that may be related). An alternative would be to let the user explicitly declare the components.
What do you think of that?
Maybe this is going too far from the original intended use of cmake component. I think the use case they had in mind was to work with package managers (like dpkg) but not to replace them.
A debian source package produces several debian binary packages. I think components were intended to pick in which (single) binary package a file would land. Dependencies would then be installed by the package manager as their own package.
So to summarize, even if possible, my gut feeling is that having a file listed in several component may lead to troubles.
I also think we should reduce the scope of qibuild, not enlarge it. Having a single cmake projects goes in that direction. Adding smartness to qi_stage_lib, not that much.
I see that it's not easily feasible, nor so compatible with current design of qibuild, thanks guys.
Also, looking at what Catkin shows in the logs, I wonder whether they are able to do it, with their different design. Do you know that?
@victor, it's not really an answer to your question but you might be interested in reading those articles
http://design.ros2.org/articles/ament.html
http://design.ros2.org/articles/build_tool.html
sorry, what question?
Ament is indeed interesting! But it sure they are able to do the trick catkin did. Anyway, it is worth trying for new projects.