Removals aren't actioned until environment is redeploy
Opened this issue · 17 comments
When there is a file removal and g10k deploys the environment
The removed file is still present until the environment is purged and re-deployed.
Can you give me an example so that I can reproduce it?
Do you meant a file is removed from the control repository by hand?
We make use of a mix of forge modules and locally hosted git modules.
How to reproduce:
Create a new git feature branch in one of the locally hosted git repos.
Do a git rm <filename>
Commit and push that change.
Merge the feature branch into a (pre-production) branch in your workflow (testing/staging/canary)
Have g10k deploy the change to the environment via the command
/usr/local/bin/g10k -verbose -maxworker 50 -maxextractworker 20 -config /etc/g10k.yaml -branch <environment name>
After the command has completed the removed file is still on disk on the puppet server for that environment in the basedir
If I remove the environment and redeploy it, (either by deleting the environment directory or passing the -force flag) the expected behaviour happens, meaning the removed file is not re-deployed
Below is the contents of our /etc/g10k.yaml
---
# Puppet Controlled #
# DO NOT MODIFY #
#
:cachedir: '/etc/puppetlabs/puppet/environments/g10k_cache'
use_cache_fallback : false
# A list of git repositories to create
sources:
local:
remote : git@gitlab.host-h.net:puppet/control
basedir : /etc/puppetlabs/puppet/environments
I can't reproduce your bug.
I've created a control repository branch called single_remove and added an empty foobar file into it: https://github.com/xorpaul/g10k-environment/tree/189b8f9e6f44658f2b8d3939a60467a3f08b72a1
Then I forked a branch of this called single_remove_branch and did a git rm foobar and pushed this branch. (xorpaul/g10k-environment@8720074)
After that I merged single_remove_branch into single_remove (also xorpaul/g10k-environment@8720074)
Then I synced the control repository branch single_remove again:
$ g10k -config test.yaml -environment example_single_remove
Need to sync /tmp/example/example_single_remove
Removing unmanaged path /tmp/example/example_single_remove/foobar
As you can see the merged removal of the file foobar was removed.
Are you using the most recent version of g10k? https://github.com/xorpaul/g10k/releases
Ah, I've found it.
This is not a bug, if you want g10k to purge your control repository content from unmanaged files you need to set the purge_level to at least contain 'environment'.
See r10k documentation, which I'm trying to emulate: https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#purge_levels
The default value is
['deployment', 'puppetfile']
I'm setting these purge_level for my test g10k config file, that's why I couldn't reproduce your bug previously: https://github.com/xorpaul/g10k/blob/master/test.yaml#L5-L6
I've verified that r10k does exactly the same.
Without the 'environment' in the purge_level it leaves unmanaged files in the control repository untouched.
Thank you very much, I'll adjust our configs appropriately
Hi, in our setup we run g10k with the following flags
-quiet -maxworker 50 -maxextractworker 20 -config /etc/g10k.yaml -branch staging
which then checks out the specified branch which correlates to a Puppet environment into /etc/puppetlabs/puppet/environments
ie. /etc/puppetlabs/puppet/environments/staging
Based on the code I see here it looks like the cleanup only works when making use of the -environment flag?
Would it be possible to adjust the code to also purge unmanaged content when making use of the -branch flag?
Nope, it doesn't matter if you use the -environment or the -branch parameter for the unmanaged content to be purged:
$ touch /tmp/example/example_single_remove/foobar
$ g10k -config test.yaml -branch single_remove -info
Removing unmanaged path /tmp/example/example_remove
Synced test.yaml with branch remove with 0 git repositories and 0 Forge modules in 0.5s with git (0.0s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
Maybe you think that if you're missing the Removing unmanaged path line in your output the file also doesn't get removed, but this output only happens when you're using -info.
The file also does get deleted without -info or with -quiet:
$ touch /tmp/example/example_single_remove/foobar
$ g10k -config test.yaml -branch single_remove -quiet
$ ls -l /tmp/example/example_single_remove/foobar
ls: cannot access '/tmp/example/example_single_remove/foobar': No such file or directory
@xorpaul thanks for all the help thus far. We're still struggling to get the desired outcome.
The prescribed behaviour works for us on modules that don't exist in the puppetfile, but not on files inside modules/repos that are cloned when updating ie.
Bad module/file added to module path:
$ touch /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/badmodule
$ /usr/local/bin/g10k -info -config /etc/g10k.yaml -branch WK_g10_cleanup
Removing unmanaged path /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/badmodule
Synced /etc/g10k.yaml with branch WK_g10_cleanup with 45 git repositories and 84 Forge modules in 5.2s with git (1.2s sync, I/O 0.0s) and Forge (0.1s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
File removed/untracked from module repository - simulated:
$ touch /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/ssh/badmodulefile
$ /usr/local/bin/g10k -info -config /etc/g10k.yaml -branch WK_g10_cleanup
Synced /etc/g10k.yaml with branch WK_g10_cleanup with 45 git repositories and 84 Forge modules in 5.6s with git (1.2s sync, I/O 0.0s) and Forge (0.1s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
The latter example is more of a concern for us since we want the contents of the cloned repos to not contain untracked files.
How did a file inside a Puppetfile managed module get modified/created outside of r10k/g10k?
Currently g10k and r10k do not check the content of a module that was not updated by r10k/g10k in the current run:
$ touch /tmp/example/example_single_remove/external_modules/inifile/foobar
$ r10k deploy environment -c ./test.yaml example_single_remove -p
$ ls -l /tmp/example/example_single_remove/external_modules/inifile/foobar
-rw-rw-r-- 1 andpaul andpaul 0 Dez 4 16:34 /tmp/example/example_single_remove/external_modules/inifile/foobar
I did that just to illustrate the point, let me illustrate with the actual use case:
The workflow we are trying to get to is that files that are removed from modules that get deployed by g10k get removed from the code base.
I just ran the following test to illustrate again:
In my own git based puppet module repository (called hieradata as an example) and created on the puppetserver by g10k under /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata
I perform the following steps in my local checked out repo:
$ touch fqdn/test_removal.yaml (and commit)
$ git push
On the puppetserver:
run g10k and check that the file exists:
$ ll /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml
-rw-rw-r-- 1 root root 0 Dec 5 10:41 /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml
Great it's there, g10k made sure that the new file is present in the codebase
Now the removal of the file in the module repo
In my local checked out repo:
$ git rm fqdn/test_removal.yaml (and commit)
$ git push
On the puppetserver:
Run g10k again, but the file is not removed
$ ll /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml
-rw-rw-r-- 1 root root 0 Dec 5 10:41 /etc/puppetlabs/puppet/environments/WK_g10_cleanup/modules/hieradata/fqdn/test_removal.yaml
The only way to ensure that it is removed, is by either removing it behind g10k's back or using force to remove all environments and recreate them. Neither of these seem like feasible options.
I can't reproduce your bug.
When I'm adding such a file in my test module here: https://github.com/xorpaul/g10k_testmodule/tree/update_master
and also added a fqdn/test_removal.yaml xorpaul/g10k_testmodule@51aa7f6
Then deployed the environment via g10k:
$ ./g10k -config test.yaml -environment example_single_own_modules -info
Need to sync /tmp/example/example_single_own_modules/modules/test
Synced test.yaml with 2 git repositories and 0 Forge modules in 1.2s with git (0.8s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
$ ll /tmp/example/example_single_own_modules/modules/test/hieradata/
total 4,0K
drwxr-xr-x 2 andpaul andpaul 4,0K Dez 5 13:53 fqdn/
$ ll /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/
total 0
-rw-rw-r-- 1 andpaul andpaul 0 Dez 5 13:53 test_removal.yaml
Then I removed the test file again:
$ git rm fqdn/test_removal.yaml ; git ci -m "remove fake hieradata" ; git push
rm 'hieradata/fqdn/test_removal.yaml'
and triggered the g10k run:
$ ./g10k -config test.yaml -environment example_single_own_modules -info
Need to sync /tmp/example/example_single_own_modules/modules/test
Removing unmanaged path /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn
Removing unmanaged path /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/test_removal.yaml
Synced test.yaml with 2 git repositories and 0 Forge modules in 1.3s with git (0.7s sync, I/O 0.0s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
$ ls -l /tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/
ls: cannot access '/tmp/example/example_single_own_modules/modules/test/hieradata/fqdn/': No such file or directory
Can you check your g10k yaml config again if you're missing the 'puppetfile' in purge_level?
https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#purge_levels
I'm running into this issue currently as well -- I'm not yet able to get cleanup of removed controlrepo files using v0.8.8.
My config looks like this:
:cachedir: '/tmp/g10k'
use_cache_fallback : true
deploy:
purge_levels: [ 'deployment', 'environment', 'puppetfile' ]
sources:
puppet:
remote : ssh://user@example.org/controlrepo.git
basedir : /etc/puppetlabs/code/staging/environments
exit_if_unreachable : true
invalid_branches : correct
interestingly -- I just noticed my master branch is getting cleaned up, however my other branch that has its name corrected is not -- possibly issue with purging and corrected branch names?
This could be a possibility that I'm missing the autocorrected Puppet environment names.
Although I don't know if the issue creator is also having his branch name corrected as well.
I'll have a look at this later
Something to note here: If you had the purge settings incorrect and then updated them, you've missed the boat.
The purge settings must be correct on the commit you are removing content. If it's not, it will never be noticed as unmanaged.
IE: I rolled my 'production' branch back to before a large delete as 'delete_test' - with purging turned on as I want. It did the right thing and generated types stayed around, etc.
Then I merged in production, which then deleted a bunch of files. g10k removed those files.
However, in the local copy of production, they still exist , because g10k is not aware they disappeared. (The "environment" option to purge_levels was added to my config after the change was already deployed live)
I haven't looked at the code recently, so I'm not sure what the correct fix to this is.