Automate updates of all dependencies
mhdawson opened this issue · 30 comments
PR's like this are really hard to validate and should probably be done through automation.
@RafaelGSS,@facutuesca is that something you could add to your do list?
https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-root-certs.md)
FYI if you are not aware there is a PR to add a workflow for updating the timezone information: nodejs/node#43988.
Updating ICU itself is sometimes done together with V8 updates if V8 has bumped ICU versions.
@richardlau thanks for pointing that out.
@RafaelGSS, @facutuesca maybe what we should focus on is looking at all of the dependencies, if upates are automated and if not identify which ones should be, and prioritize which ones we'd want to automate. (For example I think we had some discusssions around openSSL, but I think we should track/work on them as an overall program to ensure progress).
If that makes sense to you two I might update the title of this issue to be more about doing it in general for all of the dependencies.
That makes sense to me.
@mhdawson I'll start working on this (looking at all dependencies and see which ones we could update with a script). Should we change the title of the issue to match that?
@facutuesca thanks, I've updated the title.
I updated the first part of the issue to have the list of deps along with checkboxes. We can track progress there in terms of which ones we have automated versus not so far.
@BethGriggs if you have any insight/suggestions of what we might need/want to include in the automation based on what you have leared about SALSA, that info would be good to factor into how we do the automation.
The following dependencies are already updated automatically via a Github action:
The following have a script + docs on how to update them (but no GH Action):
The following have only docs on how to update them:
- base64 (docs)
- cjs-module-lexer (docs)
- icu-small (docs)
- ngtcp2 (docs)
- OpenSSL (docs)
- V8 (docs)
- zlib (docs)
Finally, the following don't have any docs/scripts/etc:
- acorn
- brotli
- googletest
- histogram
- uv
I'd go with the ones that are often updated, such as OpenSSL / ICU / zlib
ICU doesn't update very often (about once a year), although we do now have an automated workflow for updating the timezone information (which updates more often) in the ICU data file.
I believe the npm team have their own automation to push npm releases into Node.js core.
I'd agree that starting with the ones we update most often would be good, and in particular OpenSSL. I think we have a few starting points. I'd written up nodejs/node#42395 and I think that @RafaelGSS had also done some work on that front as well.
I also think that working on the list for which we have no instructions is also a priority as I see not having that documented as a risk we might get it wrong if we do have to do an update.
@facutuesca and thanks for the good categorization, it's good to be able to look at the overall list like that.
Root certificates is another thing we could add to the list: nodejs/node#45477
Update process: https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-root-certs.md
FWIW Adoptium have some automation for something similar for Termurin Java builds (thanks @sxa for the pointer 🙇): https://github.com/adoptium/temurin-build/blob/master/.github/workflows/ca-cert-updater.yml
I believe their version of mk-ca-bundle.pl differs from ours and they also have less information in their commit messages regarding the certificates removed/added or NSS version the update is based on.
@richardlau thanks for pointing that out. I think starting with automating the root cert updates would be a good thing to start with.
So far we have added:
- acorn: nodejs/node#45357
- base64: nodejs/node#45300
- libuv: nodejs/node#45362
Currently on review:
- OpenSSL: nodejs/node#45605
cc @RafaelGSS
@facutuesca thanks for progressing this and the update.
we can update the list since nghttp2 is now updated with action with this pr nodejs/node#46700
@marco-ippolito I've selected llhttp as completed, but IIRC we use a different major of llhttp on v16/v14, so this automation wouldn't solve all the lines.
Maybe bring this discussion up in the next security-wg call? We might find some ideas.
Note, the case of llhttp happens to OpenSSL and I assume to other libraries as well.
Let's discuss about it in the next security-wg I'm happy to work on it
feel free to edit if I made a mistake
update on dependecy status:
The following dependencies are already updated automatically via a Github action:
- eslint
- undici
- cares
- corepack
- llhttp
- nghttp2
- npm
- base64
- ada
- brotli
- uv
- acorn
- ngtcp2
- OpenSSL
- zlib
- cjs-module-lexer
- googletest
- uvwasi
- V8 (patch)
The following have only docs on how to update them:
- icu-small
Finally, the following don't have any docs/scripts/etc:
- histogram (version header file missing I've opened a PR to include it)
Root certificates is another thing we could add to the list: nodejs/node#45477 Update process: https://github.com/nodejs/node/blob/main/doc/contributing/maintaining-root-certs.md
Opened a PR for updating the root certificates: nodejs/node#47425
@richardlau thanks! Great to see that automated.
Hi everyone! As @marco-ippolito suggested, I'm currently working into uvwasi automation
histogram (version header file missing I've opened a PR to include it)
FWIW, the version header file is not necessary. You can take a look at my approach from nodejs/node#47482, which uses some git magic to figure out if anything changed between the latest upstream version and what's in the node repository. It doesn't actually matter what version is in the node repo. The only difference to nodejs/node#47482 is that for googletest, we follow HEAD, whereas for HdrHistogram_c we probably want to use the latest published release.
Updated by github action:
- acorn
- ada
- base64
- brotli
- c-ares
- cjs-module-lexer
- corepack
- googletest
- icu-small
- llhttp
- minimatch
- nghttp2
- ngtcp2
- npm
- openssl
- postject
- simdutf
- undici
- uv
- uvwasi
- V8
- zlib
- icu-small (wip: nodejs/node#47727)
Finally, the following don't have any docs/scripts/etc:
- histogram (version header file missing I've opened a PR to include it)
PR for dependencies overview: nodejs/node#47589
One left to go 🥳
There appears to be an issue with nghttp3 updates, see nodejs/node#47576 (comment).
Last dependency update automation nodejs/node#48171
and it's merged 🎉🎉🎉🎉🎉🎉 all the dependencies are automated 🥳
Bravo!!
