mixu/npm_lazy

cache isn't working via npm install behind proxy

Closed this issue · 8 comments

Hotell commented

Hi, I'm trying to setup npm_lazy on corporate server ( runs on centOS ) but without any luck.
I've configured the proxy settings in config file.

When I run npm_lazy:

[root@ITAP11 ~] npm_lazy -c ./npm_lazy.config.js
app info npm_lazy at 0.0.0.0 port 9000
app info npm_lazy cache directory: /root/.npm_lazy
app info Using http://171.27.1.254:3128 to proxy http requests.
app info Using https://171.27.1.254:3128 to proxy https requests.

now when i run npm install --registry http://localhost:9000 lodash
this is what I get:

npm http 502 http://localhost:9000/lodash
npm ERR! registry error parsing json
npm ERR! SyntaxError: Unexpected token <
npm ERR! <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
npm ERR! <html><head>
npm ERR! <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
npm ERR! <title>ERROR: The requested URL could not be retrieved</title>
npm ERR! <style type="text/css"><!--
npm ERR!  /*
npm ERR!  Stylesheet for Squid Error pages
npm ERR!  Adapted from design by Free CSS Templates
npm ERR!  http://www.freecsstemplates.org
npm ERR!  Released for free under a Creative Commons Attribution 2.5 License
npm ERR! */
...
...
npm ERR! System Linux 2.6.32-431.5.1.el6.centos.plus.x86_64
npm ERR! command "node" "/usr/bin/npm" "i" "--registry" "http://localhost:9000/" "lodash"
npm ERR! cwd /root/tmp
npm ERR! node -v v0.10.25
npm ERR! npm -v 1.4.6
npm ERR! type unexpected_token
npm ERR!
npm ERR! Additional logging details can be found in:
npm ERR!     /root/tmp/npm-debug.log
npm ERR! not ok code 0

but when I run request via wget, it works and correctly saves package to ~/.npm_lazy/
wget http://localhost:9000/lodash

any ideas?

thanks in advance

mixu commented

Do you mean wget http://registry.npmjs.org/lodash works? I'm not sure why there would be a need to use a proxy for localhost but I'm not familiar with your setup,

In any case can you run the wget that works with the -d option which will print request and response headers and post them here? It's probably a difference between the headers npm_lazy is sending vs wget and if wget works then that's encouraging.

Hotell commented

Well all our local servers are behind a firewall so when you wanna access internet you have to go through proxy, that's why.

about wget. Nope, I meant calling npm_lazy server, which should check if that package is in npm_lazy cache. If not it calls http://registry.npmjs.org/lodash. So I'm calling wget localhost:9000/lodash to "try" if npm_lazy server is working.
wget-npm_lazy

here is the response with -d option that you've requested.

 wget -d localhost:9000/lodash
DEBUG output created by Wget 1.12 on linux-gnu.

--2014-04-04 09:19:02--  http://localhost:9000/lodash
Resolving localhost... ::1, 127.0.0.1
Caching localhost => ::1 127.0.0.1
Connecting to localhost|::1|:9000... Closed fd 3
failed: Connection refused.
Connecting to localhost|127.0.0.1|:9000... connected.
Created socket 3.
Releasing 0x0000000002107390 (new refcount 1).

---request begin---
GET /lodash HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: localhost:9000
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Fri, 04 Apr 2014 07:19:02 GMT
Connection: close

---response end---
200 OK
Length: unspecified
Saving to: “lodash.1”

    [ <=>                                                                                      ] 69,989      --.-K/s   in 0.001s

Closed fd 3
2014-04-04 09:19:02 (111 MB/s) - “lodash.1” saved [69989]

thanks

mixu commented

ok, thanks for the clarification. So the issue is that npm_lazy hangs when trying to connect externally in response to your request against localhost. Can you run wget -d http://registry.npmjs.org/lodash and post it here. If that works correctly (check that the file is as expected) through your proxy then I should be able to compare the npm_lazy code with whatever works in your environment.

I am assuming .wgetrc is already configured with the proxy info but if wget -d http://registry.npmjs.org/lodash doesn't work could you figure out the correct wget settings that work for external requests and then post an example of a working set of request headers?

Hotell commented 
wget -d registry.npmjs.org/lodash
DEBUG output created by Wget 1.12 on linux-gnu.

--2014-04-04 10:15:41--  http://registry.npmjs.org/lodash
Connecting to 171.27.1.254:3128... connected.
Created socket 3.
Releasing 0x0000000001990bd0 (new refcount 0).
Deleting unused 0x0000000001990bd0.

---request begin---
GET http://registry.npmjs.org/lodash HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: */*
Host: registry.npmjs.org

---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Age: 3399
Via: 1.1 varnish, 1.1 varnish, 1.0 localhost (squid/3.1.6), 1.0 171.27.1.254 (McAfee Web Gateway 7.4.1.1.0.17024)
Date: Fri, 04 Apr 2014 08:15:41 GMT
ETag: "AJSV8HHNVCYLT3DO196P8YWMI"
Vary: Accept
Server: CouchDB/1.5.0 (Erlang OTP/R14B04)
X-Cache: MISS, MISS
X-Cache: MISS from localhost
X-Timer: S1396595942.354797840,VS0,VS42,VE223,VE3399068
X-Served-By: cache-v41-ASH, cache-am72-AMS
Content-Type: application/json
X-Cache-Hits: 0, 0
Accept-Ranges: bytes
Cache-Control: max-age=1
Content-Length: 70809
X-Cache-Lookup: MISS from localhost:8080
Proxy-Connection: Close

---response end---
200 OK
Length: 70809 (69K) [application/json]
Saving to: “lodash”

100%[===================================================================================================================>] 70,809      --.-K/s   in 0.1s

Closed fd 3
2014-04-04 10:15:41 (632 KB/s) - “lodash” saved [70809/70809]
mixu commented

Nothing looks obviously wrong in the headers. Now that I think of it, are you using the npm proxy setting right now on that server? What does: npm config get proxy and npm config get https-proxy say?

Because it would make a lot of sense that npm install --registry http://localhost:9000 lodash would not work IF npm itself is trying to access localhost:9000 through a proxy - which, now that I think of it is what the output looks like e.g. a squid generated page for a local http request.

it looks like this stuff also uses environment variables: https://www.npmjs.org/doc/misc/npm-config.html so maybe try:

HTTP_PROXY= HTTPS_PROXY= npm config get proxy
HTTP_PROXY= HTTPS_PROXY= npm config get https-proxy

those should both return null. If that works then: HTTP_PROXY= HTTPS_PROXY= npm install --registry http://localhost:9000 lodash might work as expected (e.g. no proxy via npmrc, no proxy via env vars, and please use npm_lazy)

Edit: it would kind of make sense that npm does not special case localhost requests when doing proxying, as having the client and the registry on the same machine is not a common case.

Hotell commented

ha! works. man, you're the best. Thanks so much! 🎉

summary

yeah I've had proxy config in .npmrc, and also in env variables

  • so I've commented that out ( so installing packages will be possible only via npm_lazy )
  • .npmrc
;proxy = http://171.27.1.254:3128/
;https-proxy = https://171.27.1.254:3128/
strict-ssl = false
;registry = http://registry.npmjs.org/
registry = http://localhost:9000
  • npm_lazy.config.js
 proxy: {
    'http' : 'http://my-proxy-ip:port',
    'https' : 'https://my-proxy-ip:port'
}
  • installing packages:
    HTTP_PROXY= HTTPS_PROXY= npm install lodash
mixu commented

awesome, I'll add a note to the readme about this. thanks for your help in debugging this!

kozmaz87 commented

This is all well and good until you install a package like imagemin. Which brings in pngquant, which goes to github via https and cannot download stuff.

⚠ connect ETIMEDOUT
⚠ jpegtran pre-build test failed
ℹ compiling from source
⚠ connect ETIMEDOUT
⚠ optipng pre-build test failed
ℹ compiling from source
⚠ Request to https://raw.github.com/imagemin/pngquant-bin/v3.0.0/vendor/linux/x64/pngquant failed
⚠ pngquant pre-build test failed
ℹ compiling from source
⚠ connect ETIMEDOUT
⚠ gifsicle pre-build test failed
ℹ compiling from source
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete
✖ Error: connect ETIMEDOUT
at errnoException (net.js:904:11)
at Object.afterConnect as oncomplete

I am afraid it is not possible to run npm_lazy on localhost because of the utterly stupid way npm handles requests to localhost. There is not a single use-case in this quadrant of our galaxy where requests to localhost aka 127.0.0.1 is supposed to be sent off to a proxy.