mixu/npm_lazy

npm_lazy doesn't cache some packages "Error: URL is not in the npm_lazy cache"

Closed this issue · 7 comments

mostr commented

I've setup npm_lazy on my localhost and set my global registry pointing to it. Then I did clean npm install on my project (with node_modules removed prior to that) and it went fine. Then I switched off my network connection so npm install could use cached versions and while some of the packages were being fetched from cache I was getting an error for others:

In npm install log:

npm http 500 http://localhost:8888/connect-livereload/-/connect-livereload-0.2.0.tgz
npm ERR! fetch failed http://localhost:8888/connect-livereload/-/connect-livereload-0.2.0.tgz

In npm_lazy log:

app debug cache get https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app debug [GET] https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app debug [GET] https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app debug [GET] https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app debug [GET] https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app debug [GET] https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz
app error [500] Error:  [Error: URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): https://registry.npmjs.org/connect-livereload/-/connect-livereload-0.2.0.tgz]

It looks like some packages are not being cached by npm_lazy. Why is it so? Did I miss something in config? (Changed port for localhost only). I'm using npm to manage my dev-dependencies only as it's grunt-based build for frontend app.

mixu commented

If some of the packages were fetched from the cache and others are not, then npm probably didn't actually request all the packages from npm_lazy when the network was up. npm_lazy is lazy so it won't cache anything that it was not asked for and npm does some internal caching so it doesn't always do a HTTP request for every single package.

Did you also run npm cache clean to flush npm's local cache before doing the whole rm -rf ./node_modules and reinstall?

Could you try this:

  • enable networking
  • do wget http://localhost:8888/connect-livereload/-/connect-livereload-0.2.0.tgz
  • kill networking
  • do wget http://localhost:8888/connect-livereload/-/connect-livereload-0.2.0.tgz

This should work, and not be affected by whether npm asks for the packages. If it fails, then this is a npm_lazy bug, otherwise it's just npm not requesting every module because of internal caching or because of the state of the local node_modules folder (e.g. installed packags are not re-fetched).

brunowego commented

I get a similar issue:

npm ERR! fetch failed http://localhost:8080/assemble/-/assemble-0.4.42.tgz
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 502
npm ERR! fetch failed http://localhost:8080/assemble/-/assemble-0.4.42.tgz
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 502

I not use the address http://localhost:8080, I set in config file external url for http://10.200.1.128:8081/ but this settings not persist.

Any idea where I wrong? Thanks!

mixu commented

Hey so I'd guess you have an old npm cache and need to run npm cache clean, or perhaps you need to audit your config settings for consistency since you shouldn't be seeing any accesses against http://localhost:8080 - they should all be targeting http://10.200.1.128:8081/. This can be because of caching in npm or the specific command you ran.

I'm trying to write a bit more of a guide to debugging given that we've gotten a bunch of issues related to this but I haven't figured out how to reproduce the issues myself, and no repro = no insight into what the issue is.

Debugging steps:

  1. run npm cache clean before attempting any npm commands. npm's local-file caching behavior has become more aggressive in recent versions, so if ANY packages are still in the local ~/.npm cache, it may hold on to old URLs. If you change the port, host or external URL for npm_lazy, you will need to clear your npm cache. Each user has their own npm cache, so make sure you are clearing the npm cache for the user that will be running npm commands.

  2. when debugging, add --loglevel verbose to npm commands to get useful debugging information. Set the registry using --registry http://localhost:8080. For example: npm install --loglevel verbose --registry http://localhost:8080 socket.io

  3. make sure your externalUrl, remoteUrl, port and host are consistent. externalUrl's port and the bind port must match. When host is 0.0.0.0 npm_lazy listens on all local adapters which is usually OK.

 // external url to npm_lazy, no trailing /
externalUrl: 'http://localhost:8080',
// registry url with trailing /
remoteUrl: 'https://registry.npmjs.org/',
// bind port and host
port: 8080,
host: '0.0.0.0',

You must be able to access http://externalUrl/socket.io directly, e.g curl -vvv http://externalUrl/socket.io should work when run (replace externalUrl) from any computer that should use npm_lazy. Firewalls etc. may interfere here so it's worth giving it a try.

  1. please include the following information:
  • include the specific npm commands you ran
    • this should begin with a npm cache clean
    • this should include all commands you did for the issue to occur.
    • use the --registry option (rather than a custom .npmrc setting) for easy reproducibility
  • include the config externalUrl, remoteUrl, port and host values you are using in npm_lazy
  • whether you are running the commands on the npm_lazy host itself, or on another computer.
jsumners commented

Local computer:

$ npm cache clean
$ curl -vvv http://my.internal.server:5000/cas-server-theme
# shows the correct package.json
$ npm install --registry http://my.internal.server:5000 --loglevel verbose cas-server-theme
...
npm info attempt registry request try #1 at 3:58:12 PM
npm http fetch GET http://my.internal.server:5000/cas-server-theme/-/cas-server-theme-0.1.0.tgz
npm http fetch 404 http://my.internal.server:5000/cas-server-theme/-/cas-server-theme-0.1.0.tgz
npm ERR! fetch failed http://my.internal.server:5000/cas-server-theme/-/cas-server-theme-0.1.0.tgz
npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 404

After having done the following on my internal server:

$ npm cache clean
$ npm_lazy -c my_config.js

And the server reports:

app debug cache get https://registry.npmjs.org/cas-server-theme/-/cas-server-theme-0.1.0.tgz
app debug [GET] https://registry.npmjs.org/cas-server-theme/-/cas-server-theme-0.1.0.tgz
app error [404] Error:  { [Error: URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): https://registry.npmjs.org/cas-server-theme/-/cas-server-theme-0.1.0.tgz]
  statusCode: 404,
  contentStream: null,
  content: '<!doctype html>\n<html lang="en" prefix="og: http://ogp.me/ns#">\n<head>\n  <meta charset="utf-8">\n  <script nonce=\'eb4ea55eb6781f57d3bfac47ba6e61ba76f259cd26fc6096db89ea21b792c19d03e18a4649c09d5afc261d29928dbe1ab886b0b429449161ebbcbc794d0e389d'...
...
...

//
//
//
module.exports = {
  //
  //
  //
  cacheDirectory: homePath + '/.npm_lazy',
  cacheAge: 0,
  httpTimeout: 10000,
  maxRetries: 5,
  rejectUnauthorized: true,
  externalUrl: 'http://my.internal.server:5000',
  remoteUrl: 'https://registry.npmjs.org/',
  port: 5000,
  host: '172.25.50.6'

  // note: system has another IP that can actually reach npmjs.org
};
jsumners commented

Looks like my problem was a result of npmjs.com screwing up -- http://status.npmjs.org/incidents/x726c1qdddf6 . I'm one of the 23 that received an email.

alexindigo commented

Having similar issue:

app debug [GET] http://registry.npmjs.org/sinon
app debug [GET] http://registry.npmjs.org/samsam
app debug [GET] http://registry.npmjs.org/sinon
app debug [GET] http://registry.npmjs.org/samsam
app debug [GET] http://registry.npmjs.org/samsam
app debug [GET] http://registry.npmjs.org/sinon
app debug [GET] http://registry.npmjs.org/samsam
app debug [GET] http://registry.npmjs.org/sinon
app error [500] Error:  { [Error: URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): http://registry.npmjs.org/samsam]
  message: 'URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): http://registry.npmjs.org/samsam',
  contentStream: null,
  content: '{ "error": "Unknown error" }' }
app debug [GET] http://registry.npmjs.org/samsam
app error [500] Error:  { [Error: URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): http://registry.npmjs.org/sinon]
  message: 'URL is not in the npm_lazy cache, and it cannot be fetched (max retries exhausted): http://registry.npmjs.org/sinon',
  contentStream: null,
  content: '{ "error": "Unknown error" }' }
app debug [GET] http://registry.npmjs.org/sinon
app debug [GET] http://registry.npmjs.org/samsam
app debug [GET] http://registry.npmjs.org/sinon
app debug [GET] http://registry.npmjs.org/samsam

Not sure if it might be related to this:
npm/npm#12196

mixu commented

looks like these were both related to npmjs.com outages / bugs. Closing to clean up the issues list a bit.