Branch filtering doesn't work as expected

Question

Branch filtering doesn't work as expected

JoshuaSmeda opened this issue 4 years ago · 7 comments

Hello,

Branch filtering within the g10k config file doesn't work as expected:

Using this for reference:
https://github.com/EncoreTechnologies/r10k/blob/feature/branch-filtering/doc/dynamic-environments/configuration.mkd

g10k Config File

  clientfire:
    remote: 'git@github.com:Nclose-ZA/clientfire_frontend.git'
    basedir: '/etc/puppetlabs/code/environments/production/modules/'
    exit_if_unreachable: true
    warn_if_branch_is_missing: true
    prefix: true
    branch_filter: 'develop'
    ignore_branch_prefixes:
      - 'staging'

This would threotically pull only the develop branch but it ends up pulling everything.

I can get it to work as expected if I pass the --branch develop parameter with the g10k command.

I unfortunately need this functionality to work since I use multiple branches in one Puppet environment.

root@pm1:/etc/puppetlabs/code/environments/production/modules# g10k -config /etc/g10k/config.yaml --debug --verbose
2020/07/22 15:40:45 DEBUG main(): Using as config file: /etc/g10k/config.yaml
2020/07/22 15:40:45 DEBUG readConfigfile(): Trying to read g10k config file: /etc/g10k/config.yaml
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir from g10k config /etc/g10k/config.yaml: /etc/g10k
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir: /etc/g10k
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir/forge: /etc/g10k/forge
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir/modules: /etc/g10k/modules
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir/environments: /etc/g10k/environments
2020/07/22 15:40:45 DEBUG readConfigfile(): detected deploy configration hash, which takes precedence over the non-deploy scoped settings
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as cachedir configured value: /etc/g10k
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir for source clientfire: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir for source sharepoint_guide: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG Puppet environment: clientfire ({Remote:git@github.com:Nclose-ZA/clientfire_frontend.git Basedir:/etc/puppetlabs/code/environments/production/modules Prefix:true PrivateKey: ForceForgeVersions:false WarnMissingBranch:true ExitIfUnreachable:true AutoCorrectEnvironmentNames:correct_and_warn})
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG Puppet environment: sharepoint_guide ({Remote:git@github.com:Nclose-ZA/sharepoint_guide.git Basedir:/etc/puppetlabs/code/environments/production/modules Prefix:true PrivateKey: ForceForgeVersions:false WarnMissingBranch:true ExitIfUnreachable:true AutoCorrectEnvironmentNames:correct_and_warn})
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir for source opendxl: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG Puppet environment: opendxl ({Remote:git@github.com:Nclose-ZA/opendxl_mcafee-epo.git Basedir:/etc/puppetlabs/code/environments/production/modules Prefix:true PrivateKey: ForceForgeVersions:false WarnMissingBranch:true ExitIfUnreachable:true AutoCorrectEnvironmentNames:correct_and_warn})
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git remote update --prune
2020/07/22 15:40:45 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git remote update --prune
2020/07/22 15:40:45 DEBUG checkDirAndCreate(): Using as basedir: /etc/puppetlabs/code/environments/production/modules
2020/07/22 15:40:45 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git remote update --prune
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git remote update --prune took 3.51329s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git branch
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git branch took 0.00450s
2020/07/22 15:40:49 DEBUG 1(): Resolving environment opendxl_develop of source opendxl
WARNING: Couldn't find specified branch '' anywhere in source 'opendxl' (git@github.com:Nclose-ZA/opendxl_mcafee-epo.git)
2020/07/22 15:40:49 DEBUG 1(): Resolving environment opendxl_master of source opendxl
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git rev-parse --verify 'develop^{object}'
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git rev-parse --verify 'master^{object}'
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git rev-parse --verify 'develop^{object}' took 0.01368s
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git rev-parse --verify 'master^{object}' took 0.01343s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git ls-tree --full-tree -r -t --name-only develop
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/opendxl.git ls-tree --full-tree -r -t --name-only master
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git remote update --prune took 3.52607s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git branch
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git ls-tree --full-tree -r -t --name-only develop took 0.00438s
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch opendxl_develop because /etc/puppetlabs/code/environments/production/modules/opendxl_develop/Puppetfile does not exist
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/opendxl.git ls-tree --full-tree -r -t --name-only master took 0.00769s
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch opendxl_master because /etc/puppetlabs/code/environments/production/modules/opendxl_master/Puppetfile does not exist
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git branch took 0.00488s
2020/07/22 15:40:49 DEBUG 1(): Resolving environment sharepoint_guide_develop of source sharepoint_guide
WARNING: Couldn't find specified branch '' anywhere in source 'sharepoint_guide' (git@github.com:Nclose-ZA/sharepoint_guide.git)
2020/07/22 15:40:49 DEBUG 1(): Resolving environment sharepoint_guide_master of source sharepoint_guide
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git rev-parse --verify 'develop^{object}'
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git rev-parse --verify 'master^{object}'
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git rev-parse --verify 'develop^{object}' took 0.00417s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git ls-tree --full-tree -r -t --name-only develop
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git rev-parse --verify 'master^{object}' took 0.00645s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git ls-tree --full-tree -r -t --name-only master
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git ls-tree --full-tree -r -t --name-only develop took 0.00352s
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch sharepoint_guide_develop because /etc/puppetlabs/code/environments/production/modules/sharepoint_guide_develop/Puppetfile does not exist
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/sharepoint_guide.git ls-tree --full-tree -r -t --name-only master took 0.00407s
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch sharepoint_guide_master because /etc/puppetlabs/code/environments/production/modules/sharepoint_guide_master/Puppetfile does not exist
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git remote update --prune took 3.56706s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git branch
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git branch took 0.00374s
2020/07/22 15:40:49 DEBUG 1(): Resolving environment clientfire_delete-dashboard-under-dashboard-section of source clientfire
2020/07/22 15:40:49 DEBUG 1(): Resolving environment clientfire_develop of source clientfire
2020/07/22 15:40:49 DEBUG 1(): Resolving environment clientfire_master of source clientfire
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'develop^{object}'
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'master^{object}'
2020/07/22 15:40:49 DEBUG 1(): Resolving environment clientfire_rison of source clientfire
Renaming branch delete-dashboard-under-dashboard-section to delete_dashboard_under_dashboard_section
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'delete-dashboard-under-dashboard-section^{object}'
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'rison^{object}'
WARNING: Couldn't find specified branch '' anywhere in source 'clientfire' (git@github.com:Nclose-ZA/clientfire_frontend.git)
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'develop^{object}' took 0.00356s
Need to sync /etc/puppetlabs/code/environments/production/modules/clientfire_develop
2020/07/22 15:40:49 DEBUG checkDirAndCreate(): Using as git dir: /etc/puppetlabs/code/environments/production/modules/clientfire_develop
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive develop
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'master^{object}' took 0.00581s
Need to sync /etc/puppetlabs/code/environments/production/modules/clientfire_master
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'rison^{object}' took 0.00491s
Need to sync /etc/puppetlabs/code/environments/production/modules/clientfire_rison
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'delete-dashboard-under-dashboard-section^{object}' took 0.00638s
2020/07/22 15:40:49 DEBUG checkDirAndCreate(): Using as git dir: /etc/puppetlabs/code/environments/production/modules/clientfire_rison
2020/07/22 15:40:49 DEBUG checkDirAndCreate(): Using as git dir: /etc/puppetlabs/code/environments/production/modules/clientfire_master
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive rison
Need to sync /etc/puppetlabs/code/environments/production/modules/clientfire_delete_dashboard_under_dashboard_section
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive master
2020/07/22 15:40:49 DEBUG checkDirAndCreate(): Using as git dir: /etc/puppetlabs/code/environments/production/modules/clientfire_delete_dashboard_under_dashboard_section
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive delete-dashboard-under-dashboard-section
2020/07/22 15:40:49 DEBUG unTar(): Discarded 4096 bytes of trailing data from tar
2020/07/22 15:40:49 DEBUG unTar(): Discarded 2048 bytes of trailing data from tar
2020/07/22 15:40:49 syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive rison took 0.24903s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'rison^{object}'
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'rison^{object}' took 0.03497s
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Writing to deploy file /etc/puppetlabs/code/environments/production/modules/clientfire_rison/.g10k-deploy.json
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch clientfire_rison because /etc/puppetlabs/code/environments/production/modules/clientfire_rison/Puppetfile does not exist
2020/07/22 15:40:49 DEBUG unTar(): Discarded 1024 bytes of trailing data from tar
2020/07/22 15:40:49 syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive develop took 0.34426s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'develop^{object}'
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'develop^{object}' took 0.00523s
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Writing to deploy file /etc/puppetlabs/code/environments/production/modules/clientfire_develop/.g10k-deploy.json
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch clientfire_develop because /etc/puppetlabs/code/environments/production/modules/clientfire_develop/Puppetfile does not exist
2020/07/22 15:40:49 DEBUG unTar(): Discarded 1024 bytes of trailing data from tar
2020/07/22 15:40:49 syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive delete-dashboard-under-dashboard-section took 0.35444s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'delete-dashboard-under-dashboard-section^{object}'
2020/07/22 15:40:49 DEBUG unTar(): Discarded 1024 bytes of trailing data from tar
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'delete-dashboard-under-dashboard-section^{object}' took 0.00429s
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Writing to deploy file /etc/puppetlabs/code/environments/production/modules/clientfire_delete_dashboard_under_dashboard_section/.g10k-deploy.json
2020/07/22 15:40:49 syncToModuleDir(): Executing git --git-dir /etc/g10k/environments/clientfire.git archive master took 0.36694s
2020/07/22 15:40:49 DEBUG executeCommand(): Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'master^{object}'
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch clientfire_delete-dashboard-under-dashboard-section because /etc/puppetlabs/code/environments/production/modules/clientfire_delete_dashboard_under_dashboard_section/Puppetfile does not exist
2020/07/22 15:40:49 Executing git --git-dir /etc/g10k/environments/clientfire.git rev-parse --verify 'master^{object}' took 0.00483s
2020/07/22 15:40:49 DEBUG syncToModuleDir(): Writing to deploy file /etc/puppetlabs/code/environments/production/modules/clientfire_master/.g10k-deploy.json
2020/07/22 15:40:49 DEBUG 1(): resolvePuppetEnvironment(): Skipping branch clientfire_master because /etc/puppetlabs/code/environments/production/modules/clientfire_master/Puppetfile does not exist
2020/07/22 15:40:49 DEBUG resolveGitRepositories(): uniqueGitModules[] is empty, skipping...
2020/07/22 15:40:49 DEBUG resolveForgeModules(): empty ForgeModule[] found, skipping...
2020/07/22 15:40:49 DEBUG timeTrack(): resolveGitRepositories() took 0.00010s
2020/07/22 15:40:49 DEBUG timeTrack(): resolveForgeModules() took 0.00010s
2020/07/22 15:40:49 DEBUG main(): Forge response JSON parsing took 0.0000 seconds
2020/07/22 15:40:49 DEBUG main(): Forge modules metadata.json parsing took 0.0000 seconds
Synced /etc/g10k/config.yaml with 8 git repositories and 0 Forge modules in 4.0s with git (0.0s sync, I/O 1.3s) and Forge (0.0s query+download, I/O 0.0s) using 50 resolve and 20 extract workers
2020/07/22 15:40:49 DEBUG executeCommand(): Executing /bin/bash /etc/g10k/add_files_dir.sh
2020/07/22 15:40:49 Executing /bin/bash /etc/g10k/add_files_dir.sh took 0.01571s
2020/07/22 15:40:49 DEBUG checkForAndExecutePostrunCommand(): postrun command '/bin/bash /etc/g10k/add_files_dir.sh' terminated with exit code 0

Answer 1 · 2020-07-23T11:02:39.000Z

Yeah, I can tell you right off the bat, that I've never heard of this feature before your ticket 😉
https://github.com/EncoreTechnologies/r10k/blob/feature/branch-filtering/doc/dynamic-environments/configuration.mkd#branch_filter
Let me have a look how I can implement this in g10k

Answer 2 · 2020-07-23T11:24:38.000Z

I can't find your branch_filter setting in the official r10k documentation anymore.

Did they remove it and possibly replace it with filter_command?
https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#filter_command

Answer 3 · 2020-07-23T14:20:26.000Z

You can try this link: https://github.com/EncoreTechnologies/r10k/blob/feature/branch-filtering/doc/dynamic-environments/configuration.mkd#filter-branches---match-branches-exactly Works for me in incognito mode as well.

…

On Thu, Jul 23, 2020 at 1:24 PM Andreas Paul ***@***.***> wrote: I can't find your branch_filter setting in the official r10k documentation anymore. Did they remove it and possibly replace it with filter_command? https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#filter_command — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJRBWDWUP6TCGXIP7TGPDFTR5AMYNANCNFSM4PEYBHCA> .

Answer 4 · 2020-07-23T14:59:02.000Z

Yeah, I saw this link, but that is a fork of the official r10k repository in a branch that is multiple years old.

I would rather implement the filter_command feature https://github.com/puppetlabs/r10k/blob/master/doc/dynamic-environments/configuration.mkd#filter_command

This would seem to achieve the same result you expect:

filter_command: 'echo $R10K_BRANCH | grep -qE "^(develop)$"'

or

filter_command: '[[ $R10K_BRANCH =~ ^(develop)$ ]]'

Right?

You can also simply test it in your shell:

$ echo $R10K_BRANCH | grep -qE "^(develop)$" && echo matched
matched
$ [[ $R10K_BRANCH =~ ^(develop)$ ]] && echo matched
matched

Answer 5 · 2020-07-28T16:40:19.000Z

Checkout my examples here: #167

Would that work for you?

I could not get the previous filter commands to work as I would need to reimplement the Bash piping or shell keywords like [[, but you could use a simple bash wrapper script for this like https://github.com/xorpaul/g10k/blob/add_branch_filtering/tests/branch_filter_command.sh

I can still rename the filter_regex to the branch_filter settings name.

Answer 6 · 2020-07-28T19:20:02.000Z

Thanks Andreas, Will check this out and get back to you.

…

On Tue, 28 Jul 2020, 18:40 Andreas Paul, ***@***.***> wrote: Checkout my examples here: #167 <#167> Would that work for you? I could not get the previous filter commands to work as I would need to reimplement the Bash piping or shell keywords like [[, but you could use a simple bash wrapper script for this like https://github.com/xorpaul/g10k/blob/add_branch_filtering/tests/branch_filter_command.sh I can still rename the filter_regex to the branch_filter settings name. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJRBWDXETLWYN55COFRPHILR535QLANCNFSM4PEYBHCA> .

Answer 7 · 2020-08-21T16:27:42.000Z

I've added the branch filtering stuff to the new release here:

https://github.com/xorpaul/g10k/releases/tag/v0.8.12