npm/cli

[BUG] npx does not attempt to get newer versions

jsg2021 opened this issue · 36 comments

jsg2021 commented

Current Behavior:

I have been using npx <some-workflow-helper> over the past year to run the current version of a command without having to manage updating it. npm7 does not pull the latest version if I have run it once before... i suspect because its no longer a temp install...

Expected Behavior:

npx some-package always runs the latest published version.

Steps To Reproduce:

  1. npx <test-package>
  2. publish a new version of
  3. npx <test-package>

Environment:

  • OS: Fedora 33
  • Node: 14.15.0
  • npm: 7.1.0
ljharb commented

If it’s installed locally or globally, it won’t fetch a version. If you want to force that, add @latest

jsg2021 commented

If I never "install" it and only use npx <> to run it, on npm6 you would always get the latest without extra args. Also, it wasn't actually installed, it was only temporarily installed... after running it was discarded. We lost that on npm7.

  • 👍3
  • 👀4
jsg2021 commented

I should amend ... yes, if the command exists it should obviously prefer that one. However, a command installed by npx should not be considered "installed"

  • 👍1
leafac commented

I ran into this issue as well (see #2329).

Here’s some more information I gathered:

  1. If you don’t have a command installed, then npx installs the package into the npm cache (on my machine that cache lives in ~/.npm).
  2. If you run npx <command>@latest then npx fetches the latest version. But the next time you run just npx <command> again it’ll get the older version.
  3. One workaround to force the package to update is to destroy the cache: rm -rf ~/.npm.

I think the best solution would be to mimic the behavior of running commands installed via npm install --global:

  1. Provide some sort of npx --udpate <command>.
  2. If the command is in the cache and you run npx <command>@latest, then the next time you run npx <command> it should choose the latest.
  3. If the command is in the cache and you run just npx <command> then it should not check if there are updates available. First because you may not want to use the latest version, but more importantly you may not want to slow down running the command by going to the registry look for newer versions.
  • 👍3
jsg2021 commented

I think the best solution would be to mimic the behavior of running commands installed via npm install --global:

  1. Provide some sort of npx --udpate <command>.
  2. If the command is in the cache and you run npx <command>@latest, then the next time you run npx <command> it should choose the latest.
  3. If the command is in the cache and you run just npx <command> then it should not check if there are updates available. First because you may not want to use the latest version, but more importantly you may not want to slow down running the command by going to the registry look for newer versions.

I respectfully disagree. Unless actually installed (globally, or locally) the cache should only be used if the current match matches, otherwise it should pull a new one (and in npm7's case prompt, unless --yes is present). The whole reason for npx is to run commands in the npm context that you may not have nor care to keep track of. (for example internal CI or workflow tools that you rather not spend time updating across all the runners.)

For your use case, you would be better off by installing the command and running with the --no-install flag.

  • 👍2
leafac commented

You make good points. Thanks for reading my proposal and for the feedback.

The whole reason for npx is to run commands in the npm context that you may not have nor care to keep track of. (for example internal CI or workflow tools that you rather not spend time updating across all the runners.)

For me, the whole reason for npx is somewhat different. It’s being able to tell people: “If you have Node.js installed, go ahead and use my tool simply by running npx obs-cli (for example); no installation necessary.” That’s only half-true, of course, because npx is installing the thing, but it doesn’t feel like it because it’s just on the cache; it doesn’t even show up on npm --global ls. The reduced friction of trying something without having to “install” it is appealing to some people (myself included).

  • 👍1
jsg2021 commented

@leafac I believe we are saying the same thing at this point.

However, where I think we differ is the point of updating. I believe the convenance of not installing should come with the burden of always getting the latest... only falling back to cache if the latest is cached.

leafac commented

You make a good point…

darcyclarke commented

I believe #2592 should fix this issue once landed & is a duplicate of, if not related to, #2395

  • ❤️1
wraithgar commented

v7 should once again match v6 behavior in the next release (v7.5.2). Changing it further should probably involve an RFC discussion.

  • 🎉2
leafac commented

@jsg2021: It seems like you got your way 😃

SrBrahma commented

I am using npm 7.11.1. I still uses an outdated package of mine.

When using npx myPackage, it uses v1.2.2, using npx myPackage@latest, uses v1.2.4, as expected.

What the hell? I don't have it globally installed. I don't want to tell my users to append @latest when calling my command. I don't want to append @latest to my npx calls.

npx should always use the latest version of the package. Unless maybe if there is no connection and there is a cached version that may or not be the latest one.

SrBrahma commented

Maybe, leave the devs an option to allow or not for the package to be executed at older versions in package.json.

I chose a 3 letter name for my package to be easy to call. I really don't want to add @latest every time.

ljharb commented

As a user, I never want to install something I already have available. I would much rather type @latest every time then be forced to implicitly incur a download cost just because the author deigned to publish a new version.

If I wanted that, I'd npm uninstall the package, so that it was downloaded every time.

SrBrahma commented

As a user, I like to use the latest versions by default because I don't really bother waiting 3-30s for a new version (that won't always happen) that may fix a breaking bug I don't know about and I may fall into it or add a cool new feature, and really dislike having to add @latest every single time because I don't know if I already have that package cached in a lower version.

For me, as an user, seems way better to use @current (or even @c) if for some reason I've never had during those years of npx is around, want to use the outdated cached version.

If I really want to use regularly a package and at a specific version, I can just npm i -g it. The npx philosophy as I understand, is to run a code you don't need to have in your machine. Basically running codes from cloud into your PC.

If this is just a matter of personal choice, it should be configurable and it won't be bad to have a poll on what should be the default value.

If I wanted that, I'd npm uninstall the package, so that it was downloaded every time.

I don't get this. How could I npm uninstall something I don't have really installed? I run my package with npx. Just tried npm uninstall myPackage, and as expected, nothing was removed.

Also, wasn't it fixed in 7.5.2, AFAIK? What about not only this issue itself, but as I found out now, #2641 and facebook/create-react-app#10601?

This behaviour seems way too prone for user errors, like the issues above. People, like me, will have to find out that their npx call is not using the latest package version and that may be leading to unexpected results. I am glad that I added the require("../package.json").version to the error messages, else, I don't really know how much longer it would take me to find that out.

  • 👍2
itstheandre commented

Hey there! I know this has been closed, but I am now facing this issue again.

I created a package a while ago ironlauncher and when i do npx ironlauncher i am getting version 0.9.6. However in npm the current version is something like 0.20+ already. I know i dont have ironlauncher installed globally or done via npm link.

So what is the reason for this?

Anyone knows how to fix it?

  • 👍2
SrBrahma commented

I insist to the npm devs: Add a new option to the package.json like "npxCheckLatest: true". Being it true and this package being npx'cuted, it will check if the cached version is the latest available one and will fetch the new one if required. This will leave the choice to the developers of the package, so the desired and intended behaviour of the respective npx package will be reached.

ljharb commented

That seems like a choice that should be up to the consumers of the package only - just like it already is in a package.json dependencies entry.

itstheandre commented

@ljharb maybe i am misreading it, but how would you code this in a way that every consumer of my package gets the latest version?

I always assume that npx gives me the latest version if i dont have something installed globally

SrBrahma commented

That seems like a choice that should be up to the consumers of the package only - just like it already is in a package.json dependencies entry.

Again, I have my specific case: I ALWAYS want for my users to use the latest version of my package without having to type @latest. I chose a 3 letter package name to be quickly written and executed. It's a specific case that isn't impossible for other projects to have the same intention. In case, my comment above has 2 likes, so I ain't a crazy single one with that request.

It's a project boilerplate starter and every update is a improved / fixed version of the previous version. The philosophy of it is to always use the latest version, mainly because it's a project I made for myself. Like it or not, that's what I want for my package. If an user for some strange reason don't want to wait 5s to download a new version, he can just not use my project (and creating the boilerplate can take some minutes, so 5s to download the newer version is like nothing), I don't care.

So, the use case exists, I am one. How a project is used isn't up to the npm dev team, with all the respect. Your personal opinion shouldn't have priority over what the developers want for their packages.

ljharb commented

ftr I think the default behavior of npx should be to always grab latest - but either way I don't think a package author's intention is relevant here.

How a project is used is also not up to the package author - it's up to the consumers.

  • 👍1
SrBrahma commented

ftr I think the default behavior of npx should be to always grab latest - but either way I don't think a package author's intention is relevant here.

How a project is used is also not up to the package author - it's up to the consumers.

It's up to the consumers to use or not a package that its philosophy is to use the latest version. I am both a regular user and a dev of my package. I already explained my use case, it exists and it's valid. It's simple, as an user I always want to use the latest version of this package when using npx without having to type @latest every single time. There isn't such an option for that. As a dev, I want all my users to be able to do that. If they don't like it, they can just don't use it at all.

ljharb commented

Such an option would only work for npx tho, and npx foo is supposed to be conceptually identical to npm install -g foo (or whatever the package name is) and then foo.

yw662 commented

and npx foo is supposed to be conceptually identical to npm install -g foo

Actually not. The wanted behavior of npx foo, should be conceptually identical to:

  • foo, with the package already installed globally or locally,
    otherwise,
  • npm install foo, then foo, then npm uninstall foo, or uninstall conceptually and hopefully the cache can be useful.

It is important to do the uninstall step. npx should keep it clean as if it never runs, except the side effects caused by the command itself. People expect npx to work this way.

ljharb commented

@yw662 "conceptually identical" encompasses the steps you described.

yw662 commented

But 'npm i -g' would introduce a side effect that 'npx' should not. It is not identical.

jsg2021 commented

I think this issue should probably be locked. (It's already closed)

npx is operating as expected (as far as I can tell)

Bugs happen.

This issue was originally to document a perceived departure from v6 to v7. That has been clarified and rectified.

TomasHubelbauer commented

Seeing as the issue is not locked, yet, I'd like to add my own comment in support of NPX always installing the latest version of the package no matter what. The current behavior of being stuck on whatever version happened to be the latest the first time the user has run npx package seems really odd to me. The user is not choosing a specific version when using npx package the first time they run it - they want the latest. So why keep reusing that version? It is not special just because it happened to be latest the first time the user ran npx command. What is special is the actual current latest version. I'd like to see NPX always execute that one.

jsg2021 commented

npx is designed to always pull latest if the command is not already present. If you are getting an old one when running the command you are experiencing a bug.

SrBrahma commented

For other devs, here is my implementation to always use the latest version in my cli program:

const spinner = ora().start('Ensuring latest version');
const latestVer = await latestVersion('gev');
if (compareSemver(VERSION, latestVer) === -1) {
  spinner.info(`The current version of gev [${chalk.keyword('brown')(VERSION)}] is lower than the latest available version [${chalk.yellow(latestVer)}]. Recalling gev with @latest...`);
  const rawProgramArgs = process.argv.slice(2);
  await execa('npx', ['gev@latest', '--no-check-latest', ...rawProgramArgs], { env: {
    npm_config_yes: 'true', // https://github.com/npm/cli/issues/2226#issuecomment-732475247
  } });
  return;
} else { // Same version. We are running the latest one!
  spinner.succeed();
}

Anyway, there is indeed a bug there.

Running npx gev@latest won't update the cached version. Just tested with my program. Running npx gev after npx gev@latest will still use the older cached version.

image

(the auto @latest wasn't present in the 1.x.x)

npm cache clear --force is not fixing this.

edit: I removed ~/.npm/_npx and npx gev -v is now returning the @latest one.

yw662 commented

#3430 This may solve some issues I guess. Anyone tried it after version 7.18.0 ?

SrBrahma commented

#3430 This may solve some issues I guess. Anyone tried it after version 7.18.0 ?

My comment before happened on npm 7.15.x. The same happened now on 7.19.1.

itstheandre commented

For other devs, here is my implementation to always use the latest version in my cli program:

const spinner = ora().start('Ensuring latest version');
const latestVer = await latestVersion('gev');
if (compareSemver(VERSION, latestVer) === -1) {
  spinner.info(`The current version of gev [${chalk.keyword('brown')(VERSION)}] is lower than the latest available version [${chalk.yellow(latestVer)}]. Recalling gev with @latest...`);
  const rawProgramArgs = process.argv.slice(2);
  await execa('npx', ['gev@latest', '--no-check-latest', ...rawProgramArgs], { env: {
    npm_config_yes: 'true', // https://github.com/npm/cli/issues/2226#issuecomment-732475247
  } });
  return;
} else { // Same version. We are running the latest one!
  spinner.succeed();
}

Anyway, there is indeed a bug there.

Running npx gev@latest won't update the cached version. Just tested with my program. Running npx gev after npx gev@latest will still use the older cached version.

image

(the auto @latest wasn't present in the 1.x.x)

npm cache clear --force is not fixing this.

edit: I removed ~/.npm/_npx and npx gev -v is now returning the @latest one.

The workaround is quite smart. I think ill implement this for my cli then.

SrBrahma commented
iamjaredwalters commented

This issue still appears to be present in 7.19.0. Running npx myPackage -V confirms an outdated version of the package is being preferred by npx, rather than the latest version.

wraithgar commented

The bug as written has been fixed.

With a clean cache:

~/D/npm $ npx tap@15.0.8 -v
Need to install the following packages:
  tap@15.0.8
Ok to proceed? (y) 
15.0.8
~/D/npm $ npx tap -v
Need to install the following packages:
  tap
Ok to proceed? (y) 
15.0.9

I am going to lock this issue because further comments are not going to get anyone's problems troubleshooted. If you feel there is a bug, please open a new issue but please be aware that if you have a package globally installed, npx will find and use that before trying to install something else, and it will not automatically fetch the latest version in that case.