npm/pacote

[BUG] gzip archives (from git) have inconsistent checksums across Linux distributions

Closed this issue · 7 comments

ThiefMaster commented

What / Why

This is very similar to #62 and possibly npm/cli#2846 but since it's happening reliably on the same machine (ie no architectural differences) in separate docker containers and comments are more likely to be overlooked, I preferred to open a new issue.

I can reliably reproduce this bug using ubuntu and archlinux Docker images using these two Dockerfiles (just for the sake of reproducing it easily) - simply run builds using docker build -f Dockerfile.XXX . and check the different lock files:

How

FROM ubuntu:focal
RUN set -ex && \
        apt update && \
        apt install -y curl git && \
        curl -fsSL https://deb.nodesource.com/setup_15.x | bash - && \
        apt install -y nodejs
RUN npm i -g npm@7.10.0
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json
{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-X/WC4DGdoiLof0cK/nTywyNzBwTNsEwH7Ky6ndwn5SUgsNmZDNnfugNhpfMX1y3Jh+GG6O9UxSMaFH/3pcffHQ==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}
FROM archlinux/base:latest
RUN set -ex && \
        pacman -Sy --noconfirm && \
        pacman -S --noconfirm nodejs npm git && \
        pacman -Syu --noconfirm
RUN npm i -g npm@7.10.0
RUN mkdir /test
WORKDIR /test
RUN npm i --verbose 'github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af'
RUN cat package-lock.json
{
  "name": "test",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "dependencies": {
        "jqplot": "github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
      }
    },
    "node_modules/jqplot": {
      "version": "1.0.9",
      "resolved": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "license": "(MIT AND GPL-2.0)"
    }
  },
  "dependencies": {
    "jqplot": {
      "version": "git+ssh://git@github.com/jqplot/jqplot.git#d96a669fbb729f4f51e2214688e54320411219af",
      "integrity": "sha512-hMjKgDiIZ2RWZOe0wOUk9V1kWwyuvpNoqIoDT1hJ/1RmzKnYIfKM1BUPdJAo4gXr/LgmEF6GxGPZ1uXn7cfVBw==",
      "from": "jqplot@github:jqplot/jqplot#d96a669fbb729f4f51e2214688e54320411219af"
    }
  }
}

When running npm pack in node_modules/jqplot manually I also get the different integrity hashes, but when I gunzip those files the .tar files have the same checksum so it's clearly related to gzip producing different output.

On my Gentoo system I get the same hash as on archlinux; I just used arch in the dockerfile because it's faster than compiling nodejs manually on a Gentoo image ;)

luxaritas commented

I think we're running into this as well - we rely on github repo-based packages, and having installs fail due to integrity issues based on who's installing is rather problematic.

  • 👍1
ahuth commented

I think we're running into this, too.

The package-lock.json is generated on linux dev environments. But pushes to Heroku fail for the integrity checksum mismatch.

  • 👍1
  • 👀1
akselikap commented

I just ran into this issue and exactly the same variant as @ahuth. It seems like there hasn't been any updates to this as I'm using the newest version of npm (7.22.0) and it's still occurring.

nevcos commented

Same happening between Windows and Linux machines, very frustrating:

npm pack "git+ssh://git@github.com/jhiesey/idb-kv-store.git#109ccad165fd6470e12fd66025da9e4743a46043"
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Windows 10 with NPM 7.21.1
# integrity:     sha512-4IWjabwqciNun[...]aXD5qGvf1Yrnw==   # Windows 10 with NPM 6.14.15
# integrity:     sha512-qpMpLBvXY0w8J[...]J8drdjr3X9/2w==   # Ubuntu WSL with NPM 7.21.1
# integrity:     sha512-pu/VZ6qKntixi[...]iPaYSOpTaEgrA==   # Ubuntu WSL with NPM 6.14.15
  • 👍1
jameskerr commented

I have a package that contains a prepare script to build a go binary when installed. When I run npm pack <my git dependency> I get a different integrity value each time I run it on the same machine. With npm 6 it was stable.

Npm 6 -- stable

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-DwGGh2q5ta6An[...]F71cfQHzLWyKQ==

$ npm -v
6.14.15

Npm 7 -- different every time

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-/i5QW944neOZe[...]PlmRMHSvk0hZA==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-kh664omwauvXp[...]UDd6RbkQKYW9A==

$ npm pack "git+https://github.com/brimdata/zed.git#0b6233ba43b8ae48585af9e1f4e7b93c97195411" |& grep integrity
npm notice integrity:     sha512-iZW1VW06yVaQG[...]eytlUvcwkkR+g==

$ npm -v
7.24.2

Note on reproducing: This is the public Zed repo so you can use the commands above, but you'll need go and make installed for it to succeed.

ahuth commented

Can confirm this is fixed for me.