g10k delete all modules when Puppetfile repo is modified

We are switching to g10k for some big environment (380 modules per env, ~ 10 env).

It seems to work quite well, except for a thing.
When we push some modifications on one of the 380 modules, we trigger a run of 👍

/usr/local/bin/g10k -config /etc/g10k.yaml -branch production -maxworker 2 -retrygitcommands -maxextractworker 2 -module somemodule

This works correctly, except if we modified the repository which contains the Puppetfile (even if we do not modify the Puppetfile itself) before.

In that case, g10k seems to detect the main (ie the Puppetfile) repository has changed, which is good, but it then delete all the modules from environment, before updating the one we just modified.

g10k.conf is simple :

---
# Puppet Controlled #
# DO NOT MODIFY     #
#
:cachedir: '/var/cache/g10k'
use_cache_fallback : false

# A list of git repositories to create
sources:
    puppetfile:
        remote  : git@git.company.net:Puppet/puppetfile.git
        basedir : /etc/puppetlabs/code/environments/

So, after this happened again I checked the log / code, so here is what happens :

We start in

g10k/git.go

Line 125 in d9d50df

func syncToModuleDir(srcDir string, targetDir string, tree string, allowFail bool, ignoreUnreachable bool, correspondingPuppetEnvironment string) bool {
A sync is needed, so we call

g10k/git.go

Line 170 in d9d50df

createOrPurgeDir(targetDir, "syncToModuleDir()")

which simply drop & recreate the environment directory:

g10k/helper.go

Line 144 in a4554d7

if err := os.RemoveAll(dir); err != nil {

g10k/helper.go

Line 148 in a4554d7

os.MkdirAll(dir, 0777)

So at this point we have no more environment. We then go through preparePuppetFile(), readPuppetfile() and resolvePuppetfile(). Here, because the -module option is use, only 1 module is setup, others are ignored.

So, for me, the issue is that

g10k/git.go

Line 170 in d9d50df

createOrPurgeDir(targetDir, "syncToModuleDir()")

should not be called here

The problem is that g10k can't be sure what changed inside the control / environment repository. Was a file removed or modified or added?

The easiest way to archive a "clean" environment is to simply delete the environment directory and repopulate it again, which is what I did chose while developing g10k at the beginning.

I don't know how r10k tracks the control / environment repository changes, so that it detects that it only needs to resync one module. I'll have a look at the r10k code.

You're talking about using the -module parameter, but I don't see that parameter in the original post. Can you please clarify what you mean with:

Here, because the -module option is use, only 1 module is setup, others are ignored.

You're talking about using the -module parameter, but I don't see that parameter in the original post.

Oh ... you're right, forgot it in the first copy / paste it seems. Fixed the original post.
AFAIK r10k does a diff between the modules currently existing on filesystem and the Puppetfile, and remove the deleted modules.

I'm fine with the delete & recreate, but then a full environment deployment should be forced, no matter if -module is specified or not (I think).

r10k do this :
https://github.com/puppetlabs/r10k/blob/1dafec0d8b61eb0812587a6c7718a906eadf30a5/lib/r10k/util/purgeable.rb#L62

any news on this issue ? As we create a lot of environments it would be really useful to at least ignore the -branch when a change is detected in puppetfile repo :)

any news on this issue ? As we create a lot of environments it would be really useful to at least ignore the -branch when a change is detected in puppetfile repo :)

You mean ignore the -module parameter?

As I understand it you would be fine with purging and recreating/populating the complete Puppet environment, but your whole problem is that the -module parameter then only populates the control/Puppetfile repository with that specified Puppet module. Is this correct?

In the long-term I definitely want to emulate the r10k behavior for control/Puppetfile repository changes, but simply ignoring the -module parameter would be something, which I could provide you with much faster.

yeah sorry, -module is the correct thing. Agree with emulating r10k behavior in long term, but at least ignoring -module when puppetfile repo changes would make g10k more consistent and should be much easier to implement.

Okay, I've implemented the purge_levels from r10k and stopped purging the whole Puppet environment directory if the control repository was changed.

Please test https://github.com/xorpaul/g10k/releases/tag/v0.7.0