Non-JOSS Feedback (traversaro)
traversaro opened this issue · 4 comments
This issue provides feedback that are not following the JOSS review checklist in openjournals/joss-reviews#2752 , so feel free to ignore them. They are just some user pain points that I experienced while trying o80 as an external Academic/R&D user, so if you are interested in getting people outside your lab to use o80, I hope they could be useful to you.
Problem with versioning and documentation of changes in each version
From the files in http://people.tuebingen.mpg.de/mpi-is-software/o80/older/, it seems that there were a lot of versions of o80 released in the past months. However, it is not clear to the users what changed between this versions, and to which specific git commit each version corresponds, even because the only tag on the repo is the v1.0 and the version in package.xml is still set to 1.0.0 . In my experience to track this I found it useful to have a small CHANGELOG.md file checked in the repo that contains the user-relevant changes for each version, following the format described in https://keepachangelog.com/en/1.0.0/ . For an example in the wild of such a file in an "academic" repo maintained by PhD students/PostDoc, see for example https://github.com/robotology/wearables/blob/master/CHANGELOG.md (full disclosure: it is a project from my lab). In my experience it is critical to keep the CHANGELOG complexity at the minimum, otherwise people do not keep it up to date. : )
Providing a clear versioning is particularly important for users external to your lab. In my experience, for members of the same lab/team, do not have any versioning scheme and just "Live at Head", i.e. always get the latest master version can work fine, because for every problem or incompatibility the members of the lab/team can ask to the expert (that I assume it is you @vincentberenz : ) ) to fix the problem directly, or in any case to get direct support. Actually, "Live at Head" is great in that context as any new problem is catched quickly by the team. The situation is quite different for external users: if I have two identical experimental setups created at different times and one does not work due to a new version of o80 that introduces some incompatibility, we can't just come to your desk and ask for help, so in practice external users would prefer to use a fixed version of o80, and upgrade their machines only once they tested that a new version works fine.
Providing a clear versioning is also helpful for users interested in strict Reproducibility in Scientific Software (see for example https://doi.org/10.1109/LRA.2018.2878604, https://www.nature.com/articles/d41586-020-02462-7, https://ieeexplore.ieee.org/document/8036322).
Default installation is quite invasive
In my experience, one of the major source of software problems in PhD students and Researchers laptop or system in general are software that is installed in system locations but that is not managed by the distribution proper package manager (i.e. sudo pip3 install ... , sudo make install ). For the sake of simplicity o80 default installation process uses both, so I think it would be worth to clearly warn user about this. I feel the poor guy that in the future will need to debug a tricky problem that appeared on machine, to realize only after a while that the o80 subtly shadowed the PySide2 apt installation by silently installing the pip3 PySide2 installation outside a virtual environment. Beside the warning on the dangers of installing on system locations, I think possible longer term solution could be (in order of difficulties):
- Document uninstallation process: not a problem for apt dependencies that by design should not created problems, but it would be cool to document the the uninstall process of o80 with the
uninstall_o80script, and furthermore document how to uninstall the pip packages (that in my experience are the major source of problems due to incompatibilities with apt-installed packages) - Install required pip packages and o80 in non system location: it would be cool to also support the installation of the o80 binaries and the pip packages in a non system location (for pip using virtualenv) and then providing a
setup.shscript that the user can run to activate the o80 installation. - Long term, it would be cool to just install o80 using a user-level package manager distro such as spack, conda-forge or nix, so that you can install o80 and all its dependencies at a user level without worrying of any possible system-wide incompatibility. I am a probably a bit biased on that as I am part of the RoboStack project, that is trying to bring that kind of experience to research software for robotics.
Thanks a lot for having taking the time to discuss this. Even if these points are not part of the "official checklist", I am grateful I can address them during this review.
The versioning is currently confusing because everytime an update over any dependencies of o80 is updated, the continuous integration upload new tar balls if the compilation and tests are successful. So currently, even updating a typo in the documentation triggers a new "version". I must improve this.
I believe the issues related to installation a mostly a lack of documentation.
Before calling "make" or "make install", the configure script is called and set the installation path. It is possible for user to set then user paths, but this is not documented. Also: if an virtual environment is active, then by default the python libraries will be installed in it (also not documented).
For the pip3 dependencies installation, it also seem to me that if a virtual environment is active, it will install for it (but I will double check, and possibly document).
I will look at these points carefully and make updates. I will not be available the next two weeks, so this may take a bit. Thanks for your patience and for your suggestions.
I believe the issues related to installation a mostly a lack of documentation.
Before calling "make" or "make install", the configure script is called and set the installation path. It is possible for user to set then user paths, but this is not documented. Also: if an virtual environment is active, then by default the python libraries will be installed in it (also not documented).
For the pip3 dependencies installation, it also seem to me that if a virtual environment is active, it will install for it (but I will double check, and possibly document).
Great, I imagine that, good to know that this is the case.
Installation
I updated the document regarding the installation, to point out how to change the installation prefix, or target user installation (or virtual environment installation) for python (in comments of the sections "From binaries" and "From source"):
http://people.tuebingen.mpg.de/mpi-is-software/o80/docs/o80/doc/02.installation.html
versioning
Until recently, the download page was cluttered with a lots of version, with no log of the modifs. This was due to the fact that the continuous integration was uploading new tar files at each successful run, and runs would occur for any minor change (e.g. fix of a typo in the documentation of one of the dependency repo, etc).
So, I introduced this file:
https://github.com/intelligent-soft-robots/o80/blob/master/version.txt
and the continuous integration will continue to run for every tiny change, but will upload only if the version number has been changed.
There is currently only 1 version available, but I will introduce a file logging the updates introduced for each version.
Great @vincentberenz, thanks a lot!