xorpaul/g10k

g10k does not deploy module correctly when -retrygitcommands is set

Closed this issue · 2 comments

We have a big environment with 380 modules.
When deploying a new environment with -maxworker 2 -maxextractworker 2 -retrygitcommands on 2 nodes I don't get consistent, neither satisfying results :

  • on first node : 371 out of 380 have been deployed
  • on second node : 367 :/

When digging, it seems that when we use -retrygitcommands g10k will retry once, and then ignore silently if the second try fails too :

2019/06/06 10:27:13 Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git remote update --prune took 0.27376s
WARN: git command failed: git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git remote update --prune deleting local cached repository and retrying...
2019/06/06 10:27:13 DEBUG purgeDir(): Trying to remove: /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git called from doMirrorOrUpdate, because git command failed, retrying
2019/06/06 10:27:13 DEBUG executeCommand(): Executing git clone --mirror git@git.company.net:PP-Puppet/common_dnsmasq.git /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git
2019/06/06 10:27:14 Executing git clone --mirror git@git.company.net:PP-Puppet/common_dnsmasq.git /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git took 0.88200s
WARN: git repository git@git.company.net:PP-Puppet/common_dnsmasq.git does not exist or is unreachable at this moment!
2019/06/06 10:27:22 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git rev-parse --verify 'production^{object}'
2019/06/06 10:27:22 Executing git --git-dir /var/cache/g10k/modules/git@git.company.net-PP-Puppet_common_dnsmasq.git rev-parse --verify 'production^{object}' took 0.00156s

But at the end the run does not raise any error :

Synced /etc/g10k.yaml with branch test_g10k_deploy with 313 git repositories and 68 Forge modules in 17.4s with git (6.9s sync, I/O 10.1s) and Forge (2.0s query+download, I/O 0.4s) using 2 resolv and 2 extract workers

but some modules are missing

So, checked the code, it seems the behavior is not directly related to the retrygitcommands option.

As I understand it, issue is that

g10k/git.go

Line 81 in d9d50df

doMirrorOrUpdate(url, workDir, privateKey, gm.ignoreUnreachable, 1)
does not check return value from doMirrorOrUpdate

So we have no way to detect that environment has deployed correctly (unless we grep in logs)

Please check if v0.6.1 works for you: https://github.com/xorpaul/g10k/releases/tag/v0.6.1