xorpaul/g10k

`doMirrorOrUpdate` does not respect maxworker

Closed this issue · 4 comments

maxenced commented

Hi,

I'm trying to debug why our git sometimes break ssh connection even with maxworker set to 1.

It seems that, for some reason, the remote update --prune triggered bu

g10k/git.go

Line 82 in 2a1a22e

success := doMirrorOrUpdate(url, workDir, privateKey, gm.ignoreUnreachable, 1)
does not respect maxworker.

Log :

$ /usr/local/bin/g10k -config /etc/g10k.yaml -branch production -maxworker 1 -maxextractworker 1 -retrygitcommands -debug
[...]
2019/08/26 10:54:23 DEBUG resolveGitRepositories(): Resolving 317 Git modules with 1 workers
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_beanstalk.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_dehydrated.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_apache.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_nodejs.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_ssh.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_openattic.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_supervisor.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_varnish5.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_fail2ban.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_jolokia.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_couchdb.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_munin.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_kafka.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_pip.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_mysql.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_sphinx.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_innodb_cluster.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_web_request.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_openvpn.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_statsd.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_nsq.git remote update --prune
2019/08/26 10:54:23 DEBUG executeCommand(): Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_logstash.git remote update --prune
[...]
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_apache.git remote update --prune took 4.31376s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_dehydrated.git remote update --prune took 4.62823s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_openattic.git remote update --prune took 4.65086s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_nodejs.git remote update --prune took 4.68310s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_beanstalk.git remote update --prune took 4.78185s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_varnish5.git remote update --prune took 4.75063s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_supervisor.git remote update --prune took 4.80052s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_innodb_cluster.git remote update --prune took 4.76119s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_ssh.git remote update --prune took 4.85105s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_fail2ban.git remote update --prune took 4.80471s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_sphinx.git remote update --prune took 4.80825s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_kafka.git remote update --prune took 4.89245s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_pip.git remote update --prune took 4.91080s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_jolokia.git remote update --prune took 4.95046s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_statsd.git remote update --prune took 4.93188s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_munin.git remote update --prune took 4.96203s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_zeromq.git remote update --prune took 4.87405s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_mysql.git remote update --prune took 4.97041s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_openvpn.git remote update --prune took 4.97957s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_web_request.git remote update --prune took 4.98709s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_graphite.git remote update --prune took 4.91951s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_acmedns.git remote update --prune took 4.90618s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_redis.git remote update --prune took 4.94386s
2019/08/26 10:54:28 Executing git --git-dir /var/cache/g10k/modules/git@git.adm.fr.clara.net-PP-Puppet_common_graylog.git remote update --prune took 4.97207s

And running a while true;do ps aux|egrep [g]it.adm && echo '---';done I can effectively see many git command at same time.

dekonnection commented

I encountered the same issue today with the latest release :)

maxenced commented

I encountered the same issue today with the latest release :)

should be fixed by #141

dekonnection commented

Thanks, I will try that today.

xorpaul commented

You can use v.0.7.1 which has merged #141