Version update breaks most nnunet-based packages
lassoan opened this issue · 9 comments
We have been benefitting from using nnunet, especially via TotalSegmentator, but there are many other projects where it is applied successfully. We are very grateful for all your efforts!
The success of nnunet also means that if anything goes wrong then that unfortunately impacts many people. Recent backward-incompatible update and releasing as a new minor version broke all existing nnunet versions, which is therefore a major problem.
Just one example: in the 3D Slicer project 3 developers spent 4 hours on figuring out what just happened and how to address it (https://discourse.slicer.org/t/totalsegmentator-failing-with-error-120/34434/9). We implemented a quick workaround and will need to remove it in the future, then test everything, so it will take a bit more time before everything is smoothed out.
Probably hundreds of other projects are/will be impacted similarly. To minimize the damage that this incompatibility may cause, you might consider doing the followings:
- Yank the 0.3.0 and 0.4.0 releases
- Make the changes backward compatible (to not break nnunet versions released in the past few years)
To prevent similar situations in the future when making backward-incompatible changes in this package, you could:
- Declare a versioning scheme (if not using semantic versioning)
- Update nnunet's version requirements to prevent nnunet picking up incompatible versions in the future. For example, specify maximum supported version to be the current major version (see for example as it is done here).
- Keep all changes in this package backward compatible for at least 1-2 years (until the existing nnunet versions - that do not cap the maximum version of dynamic-network-architectures - get old enough that people don't expect them to work anymore)
Thank you again for all your efforts!
Hey there,
thanks for bringing this to our attention. This is something that slipped during the updating process and we apologize for any issues this has caused.
Indeed we forgot to add the restriction <=0.2 in nnU-Net version 2.2.1 and this will currently causes 2.2.1 installs to break. Since I cannot replace pypi packages inplace I have created v2.2.2 which fixes that dependency problem (no other changes).
nnUNet v2.3 and the current master are fully backwards compatible with previous versions of nnU-Net and they already have the proper requirements for dynamic-network-architectures set in the pyproject.toml file. V3 will bring non-backwards compatible changes.
We will work on improving our release procedures in the future! Thanks for the feedback!
Best,
Fabian
Thane for your reply. Now that we are aware of this issue, having a new nnunet release fixes the problem for our latest Slicer application releases, which the large majority of the community uses. However, TotalSegmentator, MEMOS, HD-BET, and all other nnunet-based Slicer extensions will be broken when somebody installs a previous application version. It'll cause frustration for hundreds users who don't want, cannot, or does not know that should upgrade to the latest Slicer version, and 5-10 of them will contact us and we'll need to spend time with asking what their problem is exactly, what Slicer version they are using, ask them to try with the latest version, etc. This would all take us quite a bit of time.
There are thousands of users of TotalSegmentator alone. There are probably tens of thousandsof users of nnunetv2 and even if just 10% are not tracking the latest version, that's still a lot of people that will be annoyed and filling bug reports to various applications, various nnunet-based models, and maybe even to nnunet. The cost of this is really high (can easily go up to a million dollar that the community will absorb, but would e better to spend elsewhere). You can query your Python package download stats to get more accurate numbers but the magnitude of my estimates should be about right.
So, if you want to save your user community from significant amount of frustration and lost time, I would recommend to yank the backward-incompatible releases and upload backward-compatible releases for the next 1-2 years (that should be enough for the current nnunetv2 - that uses unbounded dynamic-network-architectures version - to taper out).
The silver lining is that the magnitude of the issue is entirely due to the success and wide adoption of your work.
this will currently causes 2.2.1 installs to break
Just to get some numbers, I queried nnunetv2 downloads in the past 3 days:
row | count | version
-- | -- | -- | --
1 | 36 | 2.0 |
2 | 198 | 2.1 |
3 | 35 | 2.1.2 |
4 | 42 | 2.2 |
5 | 309 | 2.2.1 |
6 | 96 | 2.2.2 |
7 | 383 | 2.3
As you can see, there were 200+ downloads per day for nnunetv2<=2.2.1, so hundreds of people spending a few hours each, trying to figure out what's wrong (they not all use nnunetv2 directly but they may have it as indirect dependency, so the investigation can take a while).
ill do it later today, I am just one guy with more work than the day has hours
All the 'new' versions are yanked, I will make sure everything stays backwards compatible and rerelease nnunet v2.3 plus the non-breaking changes to dynamic-network-architectures, I'll post here once this is done
Awesome, thank you!
ill do it later today, I am just one guy with more work than the day has hours
I know this feeling very well... thanks again!
Hey so things should be back to normal now. dynamic network architecture is backwards compatible and I tested the nnUNetv2 2.2.1 and 2.3.1 releases to ensure everything works as expected. Please let me know if you run into any further issues.
I would like to warmly thank you for taking the time to explain the problem and pointing towards sensible solutions! As you see there is a lot I still need to learn and I am grateful for your patience :-)
@FabianIsensee I probably would have suggested that upon yanking 0.3 and 0.4, that an updated version 0.5 would be the one that restored backwards compatibility rather than the creation of 0.3.1. This is because you should not release another next version of 0.4 because that version has already been released to the public and is a known yanked version. nnUNetV2 changes will need to be made compatible with whatever your next steps may be.
Seems like I did not push the last changes to the master, which should have taken down the 0.4 requirement. My bad, I will do this tomorrow (currently traveling). You can just install the latest version of nnunet (https://github.com/MIC-DKFZ/nnUNet/releases/tag/v2.3.1) and that should work in the meantime