ENOTCACHED / cache mode is 'only-if-cached' but no cached response available

Question

ENOTCACHED / cache mode is 'only-if-cached' but no cached response available

Opened this issue 4 years ago · 35 comments

I'm not sure this isn't related to npmlock2nix, but now that my build works I get this:

[...]
building
90m..................] / rollbackFailedOptional: verb npm-session 7c5d0f9d32bc49c
> myproject@0.0.1 preinstall /build/myproject
> /build/myproject/node_modules/.hooks/preinstall
npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
npm ERR! request to https://registry.npmjs.org/polka failed: cache mode is 'only-if-cached' but no cached response available.

npm ERR! A complete log of this run can be found in:
npm ERR!     /build/myproject/.npm/_logs/2020-12-26T15_18_40_514Z-debug.log

builder for '/nix/store/fxhblr5b6rbzbz6mxj5y414f8b1pcz60-TODO-0.0.1.drv' failed with exit code 1
error: build of '/nix/store/fxhblr5b6rbzbz6mxj5y414f8b1pcz60-TODO-0.0.1.drv' failed

Is it possible that this is related to svanderburg/node2nix#106 (and also svanderburg/node2nix#70) and that this is because of bundled package-lock.json files in some packages in the dependency tree?

If this is so, I guess npmlock2nix needs to do something like svanderburg/node2nix#106 (comment)

Answer 1 · 2020-12-26T18:36:46.000Z

I think is might be related with the other projects but neither of their solutions are likey to work here. Can you pust the relevant parts of your package.json and package-lock.json? My guess is that it is just another variant of the git references. I've multiple branches that add better/different support of them but all of them require massive cleanup.

Answer 2 · 2020-12-26T19:18:40.000Z

You can reproduce this with the Sapper template: npx degit "sveltejs/sapper-template#rollup" sappertemplate

I used a vanilla npmlock2nix shell.nix like this:

{ pkgs ? import <nixpkgs> {}, npmlock2nix ? import <npmlock2nix> { inherit pkgs; } }:
npmlock2nix.shell {
  src = ./.;
  nodejs = pkgs.nodejs-14_x;
}

Based on the error message in the post above, it seems that polka is the culprit.

Answer 3 · 2021-01-10T08:01:25.000Z

Any ideas about even a hacky/temporary workaround for this? I currently cannot use npmlock2nix for my project anymore due to this issue...

Answer 4 · 2021-08-06T01:57:04.000Z

This issue is unfortunately fairly common, in my brief experience so far with npmlock2nix. The really bad bit is that it can hit you out of the blue when you add a new dependency, meaning that everything might work fine for a few weeks and then, suddenly, you've hit a a dead end.

Answer 5 · 2021-08-06T02:03:43.000Z

On that note, I'm not sure whether this is the same issue or not, but if you try to set up Tailwind CSS for use with a create-react-app app as advised in the Tailwind docs here, you get the following error:

error: builder for '/nix/store/0pn16y1r4r02sfp8hfk81ca4g33j8r8p-react-nix-craco-0.1.0.drv' failed with exit code 1;
       last 10 log lines:
       > 90m..................] / rollbackFailedOptional: verb npm-session fa56488e4e6b8c7
       > > react-nix-craco@0.1.0 preinstall /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0
       > > /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0/node_modules/.hooks/preinstall
       > patching script interpreter paths in .
       > npm ERR! code ENOTCACHEDrver: sill fetchPackage[0m
       > npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
       > 
       > npm ERR! A complete log of this run can be found in:
       > npm ERR!     /private/tmp/nix-build-react-nix-craco-0.1.0.drv-0/.npm/_logs/2021-08-06T01_52_58_999Z-debug.log
       > 
       For full logs, run 'nix log /nix/store/0pn16y1r4r02sfp8hfk81ca4g33j8r8p-react-nix-craco-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/l0dca36c585yy6y20cagbh7ll9ksfcc9-nix-shell-env.drv' failed to build

Unfortunately, I don't know how to work around this, as create-react-app requires this special compatibility release of tailwindcss.

Answer 6 · 2021-08-10T05:49:38.000Z

I ran into the same error on my project, it seems to be a dependency that was pulled out of npm.
If you remove your package-lock.json and recreate it (with npm install), for me it removed the problematic dependency.

unfortunately I got the same error with another dependency after that.

Answer 7 · 2021-08-13T22:05:45.000Z

I may have gained some insight into the internals of this, but not much. Nag me to take some notes at some point.
https://github.com/npm/pacote appears to be related (and is part of an npm checkout).
Do note that this code has churned a bit between node 6 and 7.

Here's some log from where we got with peeking at the above mentioned issue today, but it doesn't seem to be the same as a related looking issue we've been fighting;

38 verbose stack Error: request to https://registry.npmjs.org/polka failed: cache mode is 'only-if-cached' but no cached response available.
38 verbose stack     at /nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/make-fetch-happen/index.js:133:21
38 verbose stack     at async Arborist.[nodeFromEdge] (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:934:19)
38 verbose stack     at async Arborist.[buildDepStep] (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:880:11)
38 verbose stack     at async Arborist.buildIdealTree (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/build-ideal-tree.js:218:7)
38 verbose stack     at async Promise.all (index 1)
38 verbose stack     at async Arborist.reify (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/node_modules/@npmcli/arborist/lib/arborist/reify.js:136:5)
38 verbose stack     at async Install.install (/nix/store/dpn8hvlq7v703kz3x3npx6522x89kb57-nodejs-16.3.0/lib/node_modules/npm/lib/install.js:150:5)

This log is using a lock file generated by npm 6 so that npmlock2nix can handle it, and then we use npm 7 because the output seems easier to make sense of and use in debugging.

Answer 8 · 2021-08-13T23:20:39.000Z

FYI, I have yet to encounter this with https://github.com/serokell/nix-npm-buildpackage (edit not with npm v6, anyway; see my later comment below).

Answer 9 · 2021-08-18T20:51:35.000Z

Unfortunately, I have encountered this issue with nix-npm-buildpackage, at least using NPM v7.

Under the thread for serokell/nix-npm-buildpackage#33, @yorickvP linked npm/cli#2160, which seems to point to a fix for pacote being pushed with npm/pacote@500a34f.

Based on the versions associated with nodejs/npm, this fix should be available as part of pacote 11.3.2 / nodejs 7.11.0 if I understand correctly.

Still totally lost with where that leaves us with all these projects, as they seem to be mostly up to date as far as nodejs/npm versions go.

Answer 10 · 2021-08-18T21:21:00.000Z

Unfortunately, I have encountered this issue with nix-npm-buildpackage, at least using NPM v7.

Yes, I have as well with npm v7, but not with npm v6.

Answer 11 · 2021-08-19T07:43:55.000Z

Hi! When I looked into it, I think my conclusion was that npm v7 now also requires the cache to exist during the npm prune step. Copying the cache into that step might help.

Answer 12 · 2021-09-02T21:27:32.000Z

good news : ) should be fixed in #93 (comment)

Answer 13 · 2021-09-02T23:47:32.000Z

good news : ) should be fixed in #93 (comment)

I tried your branch here: milahu@3970a1b

I don't get ENOTCACHED anymore, but I do get this failure when trying to build a React project:

 nix-build -A packages.x86_64-darwin.frontend
warning: Git tree '/Users/dhess/git/primer-frontend-web' is dirty
these 2 derivations will be built:
  /nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv
  /nix/store/lwsbgvhylbvw2hfpa12py680f091xvc3-primer-frontend-web-0.1.0.drv
building '/nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv'...
patching sources
configuring
no configure script, doing nothing
building
run 'npm install' ...
[ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree
error: builder for '/nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv' failed with exit code 1;
       last 6 log lines:
       > patching sources
       > configuring
       > no configure script, doing nothing
       > building
       > run 'npm install' ...
       > [ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree
       For full logs, run 'nix log /nix/store/aafdmp6zklgan5nw863w29q33lp0m94j-primer-frontend-web-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/lwsbgvhylbvw2hfpa12py680f091xvc3-primer-frontend-web-0.1.0.drv' failed to build

The project works with nix-npm-buildpackage.

Answer 14 · 2021-09-03T00:06:00.000Z

maybe thats too silent. can you test d0c89d1?

Answer 15 · 2021-09-03T00:47:25.000Z

maybe thats too silent. can you test d0c89d1?

Sorry, no luck — back to ENOTCACHED:

 nix-build -A packages.x86_64-darwin.frontend                                                        ~/git/primer-frontend-web
warning: Git tree '/Users/dhess/git/primer-frontend-web' is dirty
these 2 derivations will be built:
  /nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv
  /nix/store/xs4aakg1v6580sa0kxr428fvaghi58rl-primer-frontend-web-0.1.0.drv
building '/nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv'...
patching sources
configuring
no configure script, doing nothing
building
90m..................] / rollbackFailedOptional: verb npm-session 707adbebf5cc376
> primer-frontend-web@0.1.0 preinstall /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0
> /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/node_modules/.hooks/preinstall
patching script interpreter paths in .
npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.

npm ERR! A complete log of this run can be found in:
npm ERR!     /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/.npm/_logs/2021-09-03T00_46_27_051Z-debug.log

error: builder for '/nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv' failed with exit code 1;
       last 10 log lines:
       > 90m..................] / rollbackFailedOptional: verb npm-session 707adbebf5cc376
       > > primer-frontend-web@0.1.0 preinstall /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0
       > > /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/node_modules/.hooks/preinstall
       > patching script interpreter paths in .
       > npm ERR! code ENOTCACHED install loadAllDepsIntoIdealTree
       > npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.
       > 
       > npm ERR! A complete log of this run can be found in:
       > npm ERR!     /private/tmp/nix-build-primer-frontend-web-0.1.0.drv-0/.npm/_logs/2021-09-03T00_46_27_051Z-debug.log
       > 
       For full logs, run 'nix log /nix/store/gskhdrfcz08fhkq5bm03ny52flprrw62-primer-frontend-web-0.1.0.drv'.
error: 1 dependencies of derivation '/nix/store/xs4aakg1v6580sa0kxr428fvaghi58rl-primer-frontend-web-0.1.0.drv' failed to build

Answer 16 · 2021-09-03T00:54:15.000Z

can you share your package.json and package-lock.json files?

Answer 17 · 2021-09-03T01:11:13.000Z

can you share your package.json and package-lock.json files?

Not for that particular project, but here's the one it's based on, where I first encountered this issue. With milahu@3970a1b (and master), this commit works fine: https://github.com/hackworthltd/react-nix-craco-public/commit/d184c089e36f40188d9f5f6b4b493b2f736226c8

But once you try to add Tailwind & its dependencies, it breaks with the same error that I showed above: https://github.com/hackworthltd/react-nix-craco-public/commit/e4dd29ae364d0625c81ee0abfc61e96397bd7e56

Answer 18 · 2021-09-03T14:19:22.000Z

im getting a broken logfile : /

loa^M[ .................] - fetchMetadata: sill install loadAllDepsIntoIdealTree^M[ ................] \ loadDep:csstype: sill fetchPackageMetaData error for tai^M[ ..............] \ loadDevDep:typescript-language-server: sill fetchPackage^M

fix: npm --progress=false, optional: npm --loglevel=verbose

now i also get

npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.

but the main problem here, is that requires are not patched

to debug: add cat package-lock.json | ${jq}/bin/jq

    "yargs": {
      "dev": true,
      "integrity": "sha512-aePbxDmcYW++PaqBsJ+HYUFwCdv4LVvdnhBy78E57PIor8/OVvhMrADFFEDh8DHDFRv/O9i3lPhsENjO7QX0+A==",
      "requires": {
        "cliui": "^6.0.0",
        "decamelize": "^1.2.0"
      },
      "resolved": "file:///nix/store/c4lkadnl5rqrlmr742101gcigj1106xi-yargs-15.4.1.tgz",
      "version": "15.4.1"
    },

intead of "cliui": "^6.0.0", it should say "cliui": "file:///nix/store/xxxxxxxxxxxx-cliui-6.0.0.tgz",

Answer 19 · 2021-09-03T14:48:03.000Z

intead of "cliui": "^6.0.0", it should say "cliui": "file:///nix/store/xxxxxxxxxxxx-cliui-6.0.0.tgz",

Hmm, are you sure that's a good solution? @andir and I discovered that Nix store paths were being written into the package-lock.json here: #85 (comment) and that didn't seem like a good idea, a) because of the 211-character path length limitation (which can apparently be worked around), and b) because it wouldn't be a very useful package-lock.json file for people who aren't using Nix.

Answer 20 · 2021-09-03T15:00:44.000Z

Hmm, are you sure that's a good solution?

no ... the problem started with tailwindcss which is declared as

    "tailwindcss": "npm:@tailwindcss/postcss7-compat@^2.2.9",

    "tailwindcss": {
      "version": "npm:@tailwindcss/postcss7-compat@2.2.9",
      "resolved": "https://registry.npmjs.org/@tailwindcss/postcss7-compat/-/postcss7-compat-2.2.9.tgz",

in the patched lockfile

    "tailwindcss": {
      "resolved": "file:///nix/store/zcggqgvhw919c209c7f4hpfq3ki6a1bw-postcss7-compat-2.2.9.tgz",
      "version": "@tailwindcss/postcss7-compat@2.2.9"
    },

the patched version should be 2.2.9 to make it work (workaround, maybe its a bug in npm)

          echo patch lockfile for debug
          #sed -i 's|"@tailwindcss/postcss7-compat@^2.2.9"|"npm:@tailwindcss/postcss7-compat@2.2.9"|' package.json
          #sed -i 's|"version":"@tailwindcss/postcss7-compat@2.2.9"|"version":"npm:@tailwindcss/postcss7-compat@2.2.9"|' package-lock.json
          #npm verb stack Error: request to https://registry.npmjs.org/@tailwindcss%2fpostcss7-compat failed: cache mode is 'only-if-cached' but no cached response available.

          sed -i 's|"@tailwindcss/postcss7-compat@^2.2.9"|"2.2.9"|' package.json
          sed -i 's|"version":"@tailwindcss/postcss7-compat@2.2.9"|"version":"2.2.9"|' package-lock.json
          # works

          echo "use npm version $(npm --version) in $(${which}/bin/which npm)"
          # use npm version 6.14.14 in /nix/store/pvnvfvimzldwbx08pj53iaf3z1wmwl94-nodejs-14.17.5/bin/npm
          echo "run 'npm install' ..."
          npm install --offline --nodedir=${nodeSource nodejs} --progress=false # --loglevel=verbose

it wouldn't be a very useful package-lock.json file for people who aren't using Nix.

the patched json files are only used in npmlock2nix

Answer 21 · 2021-09-03T16:03:46.000Z

The Tailwind version string comes straight from the documentation here: https://tailwindcss.com/docs/guides/create-react-app

It works fine with nix-npm-buildpackage.

Answer 22 · 2021-09-03T17:30:31.000Z

please try 47e3003

with "patching dependencies" i mean the function patchDependency in npmlock2nix/internal.nix

Answer 23 · 2021-09-03T18:20:37.000Z

please try 47e3003

Closer, but still no cigar. It works fine with https://github.com/hackworthltd/react-nix-craco-public/commit/aff06f32d4befcca2a633c95a4295fec1f332d82

However, with the more complicated (private) repo I tested yesterday, it fails with this log: https://gist.github.com/dhess/084878f17db5859c0bf70f251e235731

(Edit I recreated the package-lock.json file from scratch and the error is the same.)

(Edit #2 Exactly the same problem with npm 7 (using nodejs-16_x from nixpkgs-unstable).)

Answer 24 · 2021-09-03T18:45:18.000Z

probably another bug in npmlock2nix

a different version of babel-loader was detected higher up in the tree

-> why is version 8.1.0 not installed deep in /build/node_modules/react-scripts/node_modules/

possible reason: a github version github:owner/repo is replaced with wildcard version *

to debug, add this to npmlock2nix/internal.nix

        cat package-lock.json | ${jq}/bin/jq -r | grep -C3 babel-loader
        cat package.json | ${jq}/bin/jq -r | grep babel-loader

in the first buildPhase, before npm install --offline
add jq (for json pretty-printing) to dependencies in internal.nix line 1

then compare with your input files package-lock.json and package.json

# nix-shell -p jq
cat package-lock.json | jq -r | grep -C3 babel-loader
cat package.json | jq -r | grep babel-loader

Answer 25 · 2021-09-03T21:57:37.000Z

I guess there's no real harm in making the package.json and package-lock.json files from the project available to you, which would make this a lot easier, right? :) So here you go:

package.json: https://gist.github.com/dhess/0ba93bd8685b14f88672ae8068a775b2

package-lock.json: https://gist.githubusercontent.com/dhess/1cced868803bbeda16d1b27a02216dbc/raw/346a84a241d4c2e15460df731199977d3eba4071/gistfile1.txt

Answer 26 · 2021-09-04T17:20:39.000Z

could be a bug in npm 6

patched lockfile, as produced by npmlock2nix

{
  "dependencies": {
    "react-scripts": {
      "version": "4.0.3",
      "resolved": "file:///nix/store/jfkjd4w2nfzwa93q97h4db6r47m06b3a-react-scripts-4.0.3.tgz",
      "dependencies": {
        "babel-loader": {
          "version": "8.1.0",
          "resolved": "file:///nix/store/873g019wn2x550f4sdssjdl7nlgy9n3g-babel-loader-8.1.0.tgz"
        },
      },
      "requires": {
        "babel-loader": "8.1.0"
      }
    }
  }
}

workaround in npmlock2nix:

in react-scripts.requires, replace "babel-loader": "8.1.0"
with "babel-loader": "file:///nix/store/873g019wn2x550f4sdssjdl7nlgy9n3g-babel-loader-8.1.0.tgz"

thats how i fixed the ENOTCACHED bug with my github dependencies

Answer 27 · 2021-09-04T17:46:39.000Z

Thanks for your hard work on this!

I think I'll continue using nix-npm-buildpackage for the time being, since it works without any workarounds required.

Answer 28 · 2021-09-04T20:06:58.000Z

weird, with nix-npm-buildpackage i also get the warning from react-scripts

this warning "should" be fixed by npm install --legacy-bundling
which disables all deduplication and creates a "deep" node_modules

i say "should" cos npm install --legacy-bundling does not work >: (
i still get a "flat" node_modules, with a global /node_modules/babel-loader

"dear npm, are my commands a joke to you?" ...

your case with "flat" node_modules

= deduplication on, note the "deduped" annotations

++ npm ls babel-loader
primer-frontend-web@0.1.0 /build/react-nix-craco-public
+-- @storybook/addon-essentials@6.3.7
| `-- @storybook/addon-docs@6.3.7
|   `-- @storybook/builder-webpack4@6.3.7
|     `-- babel-loader@8.2.2  deduped
+-- @storybook/react@6.3.7
| +-- @storybook/core@6.3.7
| | `-- @storybook/core-server@6.3.7
| |   `-- @storybook/manager-webpack4@6.3.7
| |     `-- babel-loader@8.2.2  deduped
| `-- @storybook/core-common@6.3.7
|   `-- babel-loader@8.2.2
`-- react-scripts@4.0.3
  `-- babel-loader@8.1.0

"global" version of babel-loader, shared by @storybook/builder-webpack4 and @storybook/manager-webpack4

++ head node_modules/babel-loader/package.json
{
  "name": "babel-loader",
  "version": "8.2.2",

"local" version of babel-loader, only for react-scripts

++ head node_modules/react-scripts/node_modules/babel-loader/package.json
{
  "name": "babel-loader",
  "version": "8.1.0",

~~we install packages by symlinks, so we have no need for deduplication (and its side-effects)~~
nope, currently we copy packages to node_modules

maybe this can be solved with buildInputs for dependencies and nativeBuildInputs for devDependencies
we could use hardlinks like pnpm or fallback to copy if hardlinks dont work

Answer 29 · 2022-03-01T02:32:44.000Z

I'm getting this issue with the wrappy package (v1.0.2). It looks like #93 was merged, so it doesn't appear to solve the issue in my case. nix-npm-buildpackage also fails with the same error.

Here is my lockfile.

Answer 30 · 2023-03-09T13:31:43.000Z

I'm also getting the issue while trying to install gatsby-cli on aarch64-darwin with nixpkgs-22.05-darwin.

Answer 31 · 2023-03-16T11:05:46.000Z

Yup, running into this due to wrappy. @jmgilman , did you find a solution or work-around?

Answer 32 · 2023-03-16T16:27:30.000Z

9/10 times it's related to package locks that are missing hashes. This is especially prevalent in older versions of npm. I'd have to go track down the issue again, but I did eventually find out that, against all common sense, hashes are technically optional in lock files.

Answer 33 · 2023-03-16T17:09:03.000Z

Thanks! My work-around was using node2nix, which somehow did the job.

Answer 34 · 2023-10-20T00:25:06.000Z

Happened with me:

  npm.v1.node_modules { src = inputs.simplecrypto; } + /node_modules;

package-lock.json

npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/aes-js failed: cache mode is 'only-if-cached' but no cached response available.

Answer 35 · 2024-08-23T03:08:08.000Z

any updates on this? still actual in 2024 on 24.05