chore: Dependency updates
SgtPooki opened this issue · 2 comments
This issue is intended to track & discuss any and all issues related to old dependencies:
Issues tracking dependency work
Core dependency updates needed (priority)
- #1840
- ipfs/ipld-explorer-components#329
- PR is out for this: ipfs/ipld-explorer-components#330
- #2045
- #2046
- #2047
- #2048
Other work
Nice to have but not required
Items blocked by outdated deps in one way or another
Issues blocking dependency updates
Investigating dependencies
npm outdated --all > npm-outdated.raw.log
gives us this: npm-outdated.raw.log
Then we want to remove duplicates, because npm outdated doesn't do that for us for some reason (echo "$(head -n 1 npm-outdated.raw.log && tail -n +2 npm-outdated.raw.log | sort | uniq -d)" > npm-outdated.uniq.log
): npm-outdated.uniq.log
cat npm-outdated.raw.log | awk '{print $6}' | sort | uniq -c | sort > most-impactful-dependencies.log
To see which dependencies are mentioned the most in the depended by column, we can run cat npm-outdated.uniq.log | awk '{print $6}' | sort | uniq -c | sort > most-impactful-dependencies.log
to get: most-impactful-dependencies.log
and then to get the top 10 most impactful dependency updates, tail -n 10 most-impactful-dependencies.log
:
8 ipfs-http-client
9 @babel/core
9 @storybook/theming
9 css-loader
10 jest-cli
11 ipfs-core-utils
11 ipfs-utils
11 react-dev-utils
15 @storybook/components
51 @babel/preset-env
So which of those versions do we 'want'?
> tail -n 10 most-impactful-dependencies.log | awk '{print $2}' | xargs -n1 -I% sh -c 'cat npm-outdated.uniq.log | grep "^%"'
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/helper-compilation-targets
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/helper-create-class-features-plugin
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-class-properties
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-dynamic-import
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-export-namespace-from
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-logical-assignment-operators
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-nullish-coalescing-operator
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-numeric-separator
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-optional-chaining
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-proposal-private-methods
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-syntax-flow
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-syntax-typescript
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-transform-flow-strip-types
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-transform-modules-commonjs
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/plugin-transform-typescript
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @babel/preset-typescript
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @jest/transform
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core @svgr/plugin-jsx
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core istanbul-lib-instrument
@babel/core 7.15.5 7.18.6 7.18.6 node_modules/@babel/core jest-snapshot
@storybook/theming 5.3.21 5.3.21 6.5.9 node_modules/@storybook/theming @storybook/components
ipfs-core-utils 0.10.5 0.10.5 0.15.1 node_modules/ipfs-core-utils ipfs-http-client
@babel/preset-env 7.12.11 7.18.6 7.18.6 node_modules/@babel/preset-env @svgr/webpack
And why are they there? tail -n 10 most-impactful-dependencies.log | awk '{print $2}' | xargs -n1 -I% sh -c 'npm explain %' > explain.txt
: explain.txt
That's super hard to understand, so let's aggregate again... cat explain.txt | grep 'from the root project' | awk '{$1=$1};1' | sort | uniq -c | sort
:
1 dev @babel/core@"^7.15.5" from the root project
1 dev @storybook/addon-links@"^5.3.19" from the root project
1 dev @storybook/addons@"^5.3.19" from the root project
1 ipld-explorer-components@"^2.4.1" from the root project
2 dev @svgr/cli@"^5.4.0" from the root project
4 ipfs-http-client@"49.0.2" from the root project
5 dev @storybook/addon-a11y@"^5.3.19" from the root project
6 dev @storybook/addon-actions@"^5.3.19" from the root project
6 dev @storybook/addon-knobs@"^5.3.19" from the root project
29 dev @playwright/test@"^1.12.1" from the root project
106 dev ipfs@"0.58.3" from the root project
117 dev ipfsd-ctl@"^7.2.0" from the root project
190 dev jest@"^26.6.3" from the root project
213 dev @storybook/react@"^5.3.19" from the root project
301 dev jest-playwright-preset@"^1.7.0" from the root project
754 react-scripts@"^4.0.3" from the root project
Breakdown of the above:
- List all outdated npm modules
- find the outdated modules that are mentioned the most
- find and count the npm modules that depend on the most-mentioned out-of-date modules.
- These are the "most impactful dependencies"
- get npm to tell us why these most impactful dependencies are in our repo.
- filter through step4 above to tell us which module we actually depend on is at fault for all the above
- find the most-mentioned module from step5 that we depend on
- The module with the most entries should be the first target for our upgrade plan.
So what does this mean?
Using react-scripts as an example: there are 754 listings of react-scripts being the ROOT CAUSE. react-scripts
is the package we depend on, that is listed in the npm-explain output, when trying to determine why out-of-date modules are listed in our dependency graph.
Why did you do all this?
I attempted to upgrade some (see main...SgtPooki:ipfs-webui:fix/node-14-windows) to resolve the build errors showing at https://github.com/SgtPooki/ipfs-webui/actions/runs/2605014389, (see #1961) and was consistently blocked by npm because of semver conflicts of wanted packages. In order to move forward.. since there is no npm-dependency-graph-resolver upgrade tool, I figured I would need all of this data.
@lidel Can we chat about what this story looks like by tying ourselves specifically to Kubo? e.g. Pulling out ipfs-http-client entirely from webui, and replacing with https://github.com/ipfs/js-kubo-rpc-client.
on the latest main:
> cat npm-install.log | sort | uniq | grep 'npm WARN deprecated' | wc -l
52
> npm i &> npm-install.log
> cat npm-install.log | perl -pe 's/npm WARN deprecated //g' | perl -pe 's/(^[^:]+)@[^:]+/$1/g' | sort | uniq | perl -ne 'print if /ipfs|cid|multi|ipld|libp2p/'
cids: This module has been superseded by the multiformats module
interface-ipld-format: This module has been superseded by the multiformats module
ipfs-block-service: This module has been merged into ipfs
ipld-dag-cbor: This module has been superseded by @ipld/dag-cbor and multiformats
ipld-dag-pb: This module has been superseded by @ipld/dag-pb and multiformats
ipld-ethereum: This module is no longer maintained
ipld-git: This module is no longer maintained
ipld-raw: This module has been superseded by the multiformats module
ipld: Please use the multiformats module instead
multibase: This module has been superseded by the multiformats module
multicodec: This module has been superseded by the multiformats module
multihashing-async: This module has been superseded by the multiformats module
All of these packages listed need to be removed/updated prior to us being able to safely&sanely keep up with new kubo feature development.
cat npm-install.log | perl -pe 's/npm WARN deprecated //g' | perl -pe 's/(^[^:]+)@[^:]+/$1/g' | sort | uniq | perl -ne 'print if /ipfs|cid|multi|ipld|libp2p/' | awk -F: '{ print $1 }' | xargs -tn1 npm explain &> npm-explain.log
Callouts
Of those 52 deprecated packages, 12 are core packages.
Of those 12 core packages, 7 require replacing fully with the multiformats package.
Of those 12 core packages, 1 was merged into ipfs.
Of those 12 core packages, 2 are no longer maintained
Of those 12 core packages, 2 require replacing with a mixture of @ipld/pkg + multiformats.