allow `-target` to accept globs
tamsky opened this issue · 50 comments
Suppose one declares several concurrent environments in a single tf config.
To limit cli actions to a single environment at a time, it would be convenient if a way existed, via -target to glob-match modules, module_paths, resource_types and resource_names:
terraform apply -target=aws*.staging* # target aws* resource_types
or similarly, for all modules:
terraform apply -target=*.*.staging* # target all modules and resource_types
or similarly, for all module_paths:
terraform apply -target=**.staging* # star star = target all modules and submodules
Hi,
Any plans to fix this?
Was just wondering if this is possible. Would be a very useful feature!
There are other requests I've seen for "exclude" logic, so it may be useful to have this use regex instead of glob to take care of both requests.
Where is the issue for "exclude" logic? that it'll resolve me many problems
$ terraform plan -exclude=mymodule.resource_not_important
Would love this feature as well !
Just hit it today... Would love this feature times 1000s
++ This feature would be excellent to have
Wow this is a very old request for a very useful feature.
Great idea, our use case would be to use target to specify the creation of template resources to create fully populated python files for Lambdas which have interpolated their vars from terraform so that we can do local unit and lint testing before interacting with the AWS API.
I am also voting for this feature...
Would it be correct to relate with Issue #2253
The feature request is for a different option parameter, but the nature of the feature to have an "exclude" pattern instead of just "apply" could be interesting.
Considering they are both trying to achieve somewhat similar results. By supporting also the exclude and this pattern feature, it would make it that much more flexible and powerful.
++ would love to have this feature.
this feature would make things a lot better
+1
+1
+1
Please do not post "+1" comments here, since it creates noise for others watching the issue and ultimately doesn't influence our prioritization because we can't actually report on these. Instead, react to the original issue comment with
Not only the option target, the "Data" ressource block as well should be able to target a ressource name="prefix-*". Ressources often needs to have dynamic name to be able to spawn in parallel to have zero downtime with lifecycle "create_before_destroy", it makes them pretty hard to target sometimes.
Any schedule on this feature? It helps on managing multiple modules. E.g. modify resources under specific module instead of all resources among modules
we are managing 700+ domains (plus records) with terraform in cloudflare. our resource naming is perfect for prefix wildcarding (if that's easier to implement than full-on globbing):
terraform apply -target=cloudflare_record.my_domain_com__*
some zones have hundreds of records so it would be incredibly tedious to list them all as targets. but all the records are prefixed with the zone name.
so definitely
Second highest voted open issue and still no progress.
Please do not post "+1" comments here, since it creates noise for others watching the issue and ultimately doesn't influence our prioritization because we can't actually report on these. Instead, react to the original issue comment with +1, which we can and do report on.
Hi,
this request is almost four years old and seems to be rather important. Any progress on that?
Thanks and best regards,
Josef
Any updates for this feature? We use terraform to manage our Kafka Topics. This feature is will be really useful for us to automate manage our topics since our topics are prefixed by a common cluster name. This wildcard or RegEx match enable us to only apply to a subset of topics that need to update.
As a workaround I came up with https://github.com/schneidexe/tfp. Yet, it's a kinda low-tech solution that might not cover all cases ... ;) But maybe it's helpful...
Also ended up making a little cli app to help with this. https://github.com/phzietsman/tft/releases.
terraform plan | tft -pattern=aws_s3_bucket -mode=include
Its crude, but it works.
still waiting for this feature years later, either wildcarding or exclude would work
Returning to this after a while, would being able to "Tag" or group resources be another way to help? I'm thinking of building a canary group for deployments but to make deploying this easier you could prefix all the resources in a canary group with a common name that can then be targeted.
The same could be achieved with a grouping convention. This could help with any staged deployments as you could deploy certain artifacts in groups with manual triggers between them.
Returning to this after a while, would being able to "Tag" or group resources be another way to help? I'm thinking of building a canary group for deployments but to make deploying this easier you could prefix all the resources in a canary group with a common name that can then be targeted.
The same could be achieved with a grouping convention. This could help with any staged deployments as you could deploy certain artifacts in groups with manual triggers between them.
It would work for groupings that you define ahead of time, but we generally face this issue when something needs to be changed on some long list of production hosts but not on others. The tagging would resolve staged deployments or groupings that you know before running into issues. The regex target and exclusion would help in many more cases.
It would appear this is still relevant years later. Did the thumbsup to the initial issue comment.
Context: Got here looking for a better way to find terraform plan -target=module.FeatureName.aws_thing_here*.* without having to specify each target. I see there are cli workarounds, those would have to be integrated with atlantis workflows, which is possible, I'll check them out. Here's hoping regex becomes a more desirable feature. Still relevant years later. Thanks for all you do.
What workarounds are people using? Because this is a very important feature.
There were at least two implemented/mentioned by @phzietsman and me a few comments above. But I guess there is more out there in the wild...
Not sure as of what version but you can terraform plan -target=module.my-module which will plan for all resources within the module. Not quite the same thing but its at least a step in the right direction. This did solve most of my use cases and can live with until there is more featureful pattern matching.
That only gets you so far though. Imagine you have a complex setup containing modules for k8s clusters on AWS with lambdas, ELBs, databases, DNS records, etc. and just wanna update your ELBs. Then you need scope it to something like terraform plan -target=**.aws_elb.*. In such a case you'd now need to search for all aws_elb entries in your state and then add them as individual -targets which can become quite cumbersome.
I hear, at work we use CI (jenkins) to help alleviate some of the tedious bits associated. A state list job allows an optional grep term to filter. Our plan jobs allows a comma separated target list. Not perfect, a bit of CI glue can reduce the pain until its enhanced.
Yeah, exactly that was also my approach and it's pretty much what my little wrapper project does (see above).
@schneidexe Instead of parsing the human-readable console output of terraform plan, you could render the plan to a file and then use terraform show on it. With some jq magic, you'd have something less brittle. Although the terraform output is most likely not going to change anytime soon.
Here's my low-tech workaround. Add this function to your .bashrc:
terraform-targets () {
sed 's/\x1b\[[0-9;]*m//g' | grep -o '# [^( ]* ' | sed " s/^# /-target '/; s/ $/'/; "
}You can use it it in a variety of ways:
terraform plan | terraform-targets
terraform plan | terraform-targets | grep 'some pattern'
terraform plan | terraform-targets | grep 'some pattern' | xargs -r terraform apply(The -r flag to xargs means don't run the command if the list is empty.)
Thanks @alahijani . I've added an additional grep to the line because we have bash scripts with comments, and those comments are incorrectly shown as -target otherwise.
terraform-targets () {
sed 's/\x1b\[[0-9;]*m//g' | grep -o '# [^( ]* ' | grep '\.' | sed " s/^# /-target '/; s/ $/'/; "
}terraform plan | terraform-targets | grep 'some pattern' | xargs -r terraform apply
When I try your last command for the "yes" prompt, something else gets passed and I get action cancelled. Am I doing something wrong?
terraform plan | terraform-targets | grep 'some pattern' | xargs -r terraform apply
When I try your last command for the "yes" prompt, something else gets passed and I get action cancelled. Am I doing something wrong?
I run terraform plan | terraform-targets | grep 'some pattern' | xargs -r terraform apply -auto-approve when I am positive it's what I want.
I have another workaround for this, but I think this will apply only for resource that already in state
for i in $(terraform state list | grep "some-pattern");
do
terraform apply -target=$i -auto-approve;
donenote: be careful with -auto-approve, do terraform plan first insteaad of terraform apply directly
@suryastef keep in mind that will run a separate plan/apply for each resource matching which could take a while esp. if you have e.g. a big remote state file in s3. Also in some cases the state might chnage and resources in the list would be removed as dependecies to others... so it would be beneficial to chain the -target .... -target for each $i and run the command only once.
@schneidexe thank you for your concern, I didn't think that way before
Also in some cases the state might chnage
The problem is the lock acquisition and release time for each call, and if they were route53 targets every call could take 3 minutes. (grumble about route53 (not terraform) taking so long to do ANYTHING...). so if I wanted to destroy 120 route53 entries
(which I actually do at least weekly), the loop method could take up to 6 hours!!!
I have another workaround for this, but I think this will apply only for resource that already in state
for i in $(terraform state list | grep "some-pattern"); do terraform apply -target=$i -auto-approve; donenote: be careful with
-auto-approve, doterraform planfirst insteaad ofterraform applydirectly
+1
Based on the solution provided by @suryastef at #2182 (comment), I think this makes more sense as it only makes use of one terraform call. (terraform allows us to define multiple targets at the same apply call)
targets=$(for x in `terraform state list | grep -e "azurerm_key_vault*.*"`; do echo "-target=$x "; done);
echo $targets | xargs terraform apply -var-file=... # add here your own options;Unfort. that does not cover for targets that were not yet deployed and hence do not yet exist in the state...
@schneidexe figured that out after commenting it in here.
Getting the approval prompt with xargs can be accomplished with --open-tty:
terraform plan | terraform-targets | grep 'some pattern' | xargs -r --open-tty terraform apply