"yarn cache clean <package-name>" doesn't work

[danshome]

I've tried using 1.3.2 and 1.4.0 of yarn.

This has to be a bug, if it's by design...well, let hope it's not. :)

What is the expected behavior?

If I run "yarn cache clean <package-name>" there should be no way possible for an old copy of a local package to be installed, especially if I'm installing from an updated local package using "yarn add file:pathto/package-0.1.0.tgz".

Please mention your node.js, yarn and operating system version.
Node v9.5.0
Yarn 1.3.2 or 1.4.0
OSX 10.13.3 (17D47)

[danshome]

yarncachetmpbug.sh.txt

[danshome]

I've attached a README/Shell script that reproduces the problem. In short, after a "yarn cache clean <package-name>" there is a copy of the old version left in the cache/.tmp directory that will keep causing the old copy of a locally installed package to get installed. Not until you manually wipe the cache/.tmp folder or completely whip your entire cache using "yarn cache clean" will you be able to install the updated version of your local package. I'm not sure why you would even use the cache if I'm using "yarn add file:pathto/package-0.1.0.tgz" anyways. It's already local, why would you spend anytime caching it to begin with?

[rally25rs]

Thanks for reporting this and for all the hard work on the repro script! 🎉
Seems really odd 🤔 I'll try to dig into it some tonight and report what I find...

I suspect that the file: prefix is somewhat irrelevant. I think anything that ends with *.tgz will use the tarball-resolver in the code, which would also be used if you were fetching an archive from online, like http://joes/cool/repo/great_tool.tgz so that's probably why it's using cache at all; as a place to extract the repo after it's been downloaded.

As a workaround, you might be able to do yarn upgrade <package-name> and it might overwrite the cache... or it'll do the same thing. Not sure off hand, but, something to try anyway.

Anyway, I'll let you know if I figure anything out...

[rally25rs]

Yeah so it looks like tarball-resolver extracts the archive to a temp directory. The temp dir just happens to be under the cache dir by default.

  async resolve(): Promise<Manifest> {
    ...

    // generate temp directory
    const dest = this.config.getTemp(crypto.hash(url));

    if (await this.config.isValidModuleDest(dest)) {
      // load from local cache
      ({package: pkgJson, hash, registry} = await this.config.readPackageMetadata(dest));
    } else {
      // delete if invalid
      await fs.unlink(dest);

      const fetcher = new TarballFetcher(
        ...

The call to this.config.getTemp() gets the path {cache_dir}/v1/.tmp. So basically yarn is assuming that the file at the same URL has already been downloaded and extracted. This is probably a bad assumption.

@yarnpkg/core what do you guys thing about just removing that code branch? Have tarball-resolver always re-download and extract the file? Or other thoughts on how to handle this? I'm not sure if we can use "browser-style" caching here with like an etag or something to have the server tell us if the file changed... that might not be supported by the request library though.

edit

Just remembered this is a file: url so the etag idea wouldn't work anyway. The easiest solution here is to just remove the check. It might have a negative effect on performance, but it's probably worth looking at the cache too, since I assume it would check the cache first before re-extracting the archive, so maybe performance wouldn't really be affected.

[arcanis]

I'm be a bit concerned about degrading the perf in order to fix something that only affect a very small part of our users 😐 Maybe a compromise would be to always remove the temporary directory when people run cache clean?

[rally25rs]

@arcanis my thought (untested) is that it would check the normal cache first, then fall into this logic if not already cached. So potentially the performance hit would be limited to "not in cache but already downloaded" which would basically only occur if you clear the cache anyway.

[rainmanhhh]

same problem. what i did:

1. delete {cache_dir}/tmp manually
2. yarn remove xxx
3. yarn cache clean xxx
4. delete xxx.tgz
   but yarn add xxx.tgz still success...

[ascott18]

Any updates on this? This is really problematic for us that there's a 'hidden' cache dir that we can't clean without going directly to the filesystem. The compromise to remove the tmp folder on any run of yarn cache clean would work fine for us.

[mweststrate]

Same here.

Edit: For others stumbling into this:

My use case is: build, pack and locally install a package. (And no, link: doesn't work because that duplicates transitive deps)

I'm now using the following work-around script:

set -e -x

NR=`date "+%s"`
rm -rf node_modules/my-package
yarn --cwd ../../../ build
yarn --cwd ../../../ pack --filename my-package$NR.tgz
yarn add file:./my-package$NR.tgz
rm my-package$NR.tgz

[oscarotero]

What do you think about an option to remove also the .tmp dir?
For example: yarn cache clean <package-name> --force

[Crashthatch]

+1 for deleting cache's .tmp directory on yarn cache clean <package-name>.

It is incredibly confusing to have a hidden cache that isn't cleaned when the cache clean <package> command is run.

[grebenyuksv-preply]

just ran into this. Would love an update!

[noahw3]

Just hit this as well. Seems like the .tmp directory should be removed automatically? Currently I'm having to resort to removing it manually.

Similar use case to mweststrate above. Every time the local package is rebuilt I have to clear the cache and kill the .tmp directory.

 yarn cache clean package-name && rm -rf $(yarn cache dir)/.tmp && yarn add file:../shared/package-name.tgz

[Ch-sriram]

Instead of using a file:<path>/filename, use link:<path>:filename.
Should resolve the issue that's occurring. Since using file:... and yarn link creates a symbolic link between the file path in local and the package inside node_modules.
But with link:..., it creates a hard link, so the total dependence of the package, has to be on the local file/path mentioned in the link. So yarn cannot use the cache.

[sytolk]

@Ch-sriram it appears that link protocol cannot create link for the package-name.tgz it used only for dir: https://yarnpkg.com/features/protocols#why-is-the-link-protocol-recommended-over-aliases-for-path-mapping

btw link protocol not exist in classic yarn v1