recipes.py ignores --no-fetch and --deps-path

Question

recipes.py ignores --no-fetch and --deps-path

Opened this issue 8 years ago · 11 comments

eg. in the Skia repo:

$ mkdir /tmp/recipe_deps
$ infra/bots/recipes.py --deps-path=/tmp/recipe_deps fetch
...
$ infra/bots/recipes.py --deps-path=/tmp/recipe_deps --no-fetch simulation_test train
(clones recipe_engine into infra/bots/.recipe_deps without looking for it in /tmp/recipe_deps)

The clone is taking a long time on our raspberry pi bots, and we'd like to package the recipe_deps as a CIPD package, and then run recipes.py in our swarming task with --no-fetch. CIPD doesn't like to install to skia/infra/bots/.recipe_deps, so we have to install it elsewhere and use --deps-path to point recipes.py to the right place, but because of this bug we clone the recipe engine unnecessarily even when the user requested us not to.

Answer 1 · 2016-10-03T20:25:17.000Z

The recipe engine wrapper you have in your repo (copy in this repo is at doc/recipes.py) automatically tries to fetch the recipe engine to .recipe_deps. It doesn't try to detect the --deps-path argument, which is a bug. Will fix

Answer 2 · 2016-10-03T20:37:23.000Z

"CIPD doesn't like to install to skia/infra/bots/.recipe_deps"

what do you mean by that?

Answer 3 · 2016-10-03T20:50:18.000Z

Now that I look again, it seems to be isolate failing to write files after CIPD has written into skia/infra/bots/.recipe_deps: https://chromium-swarm.appspot.com/user/task/31a4a3c1b2bbc310. Anyway, the nesting causes problems.

Answer 4 · 2016-10-03T20:51:23.000Z

Yeah, isolate probably expects clean directory :-/

Answer 5 · 2016-10-03T21:08:21.000Z

FYI, I'm adding a 'bundle' command to recipes which will generate a stand-alone hermetic version of the recipes in a given repo. It is designed with these sorts of uses in mind (fetching in a single location, totally offline operation, compatibility with isolate). I have a preliminary CL up here: https://chromiumcodereview.appspot.com/2352163003/

Answer 6 · 2016-10-03T21:19:29.000Z

Sounds great!

Answer 7 · 2017-01-10T20:39:27.000Z

Friendly ping. This is causing us a lot of pain and we'd like to get rid of the transparent fetching.

Answer 8 · 2017-01-11T01:55:55.000Z

I think the CL (https://chromiumcodereview.appspot.com/2352163003/) is good to go. Pinged it

Answer 9 · 2017-01-11T03:11:17.000Z

unfortunately that CL is not good to go yet, and I stalled on it. it's still on my radar though and I'll be actually fixing it this quarter.

Answer 10 · 2017-03-28T19:54:02.000Z

This is actually fixed now (forgot to reference this bug!):

Answer 11 · 2017-04-03T13:34:37.000Z

As of this morning, recipes.py is still ignoring --no-fetch:

$ infra/bots/recipes.py --deps-path=/tmp/recipe_deps --no-fetch simulation_test train
Cloning into '/usr/local/google/home/borenet/go/src/go.skia.org/infra/infra/bots/.recipe_deps/recipe_engine'...
remote: Total 6188 (delta 3847), reused 6188 (delta 3847)
Receiving objects: 100% (6188/6188), 6.78 MiB | 0 bytes/s, done.
Resolving deltas: 100% (3847/3847), done.
WARNING:root:(Not Bad) Using custom deps path: /tmp/recipe_deps
......
----------------------------------------------------------------------
Trained 6 tests in 0.237s (ran 6 tests)