Use of 'git checkout' causes intermittent failure due to git index.lock contention
russdill opened this issue · 3 comments
When doing multiple simultaneous builds, the shards install step sometimes fails with the below message:
I: Resolving dependencies
I: Installing crc (1.0.1)
E: Failed git --work-tree=/home/russd/src/foo-service/worktrees/5a1e87e36c16413334b5818f596002c80af1243e/lib/crc checkout refs/tags/v1.0.1 -- . (). Maybe a commit, branch or file doesn't exist?
An examination of strace output shows more detail:
[pid 44168] write(2, "fatal: Unable to create '/home/russd/.cache/shards/github.com/postmodern/crystal-crc.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n", 393) = 393
The changeset here: #435 causes the issue. Shards either needs to figure out how to make git archive work on windows, utilize a git shallow clone, retry git checkout on failures, or have some other way to obtain the git source.
Setting the environmental variable GIT_INDEX_FILE to an lib/.<dependency>.index may also work in this situation.
Yeah, git really isn't supposed to be run in parallel. But shards should be. So we'll have to manage.
What do you mean with "make git archive work on windows"? You're apparently not on Windows, right? (EDIT: Ah I realize it's commenting on the reasoning for #435)
I don't understand what a shallow clone would help here. The error has nothing to do with cloning, it happens when checking out a work tree.
Using GIT_INDEX_FILE looks like a good idea. shards checks out a shared git repo to (potentially) multiple work trees. So every work tree needs its own index. That how git worktree operates as well. There it's not just index because it's also used for making changes, but shards only needs to read.
As a side effect, keeping a local index per worktree checkout should probably improve performance for updates.
A clone would not touch the index of the bare repo so would not run into this issue. Making the clone shallow just avoids copying a whole bunch of unnecessary data.
But yes, I think the GIT_INDEX_FILE is the way to go. I've been testing that locally and things seem to be working. Although I'm not sure if shards using GIT_* variables from the environment when cloning dependencies and copying their work-trees is desired behavior or not. It's at least helpful in this case.