- Bash (v3+)
- Terraform
- AWS CLI (only for aws_bootstrap command)
Terraformsh is a Bash script that makes it easier to run Terraform by performing common steps for you. It also makes it easy to keep your configuration DRY and deploy infrastructure based on a directory hierarchy of environments. See DRY_CODE.md for more details.
Unlike Terragrunt, this script includes no DSL or code generation. All it does is make it easier to call Terraform. See PHILOSOPHY.md for more details.
Terraformsh will detect and use Terraform -var-files and -backend-config
configuration files across a directory hierarchy. It also has its own
configuration file so you don't have to remember any command other than
terraformsh itself (removing the need for a Makefile, though you can
still use one if you want).
You can override any options with environment variables, command-line options and config files. Good conventions like using .plan files for changes are the default.
Change to the directory of a Terraform module and run terraformsh with any
Terraform commands and arguments you'd normally use.
$ cd root-modules/aws/common/
$ terraformsh plan
$ terraformsh apply
Terraformsh will run dependent Terraform commands when necessary. If you run
terraformsh plan, Terraformsh will first run terraform validate, but before
that terraform get, but before that terraform init. Terraformsh passes
relevant options to each command as necessary, and you can also override
those options.
When certain commands are run (plan, apply, plan_destroy, destroy)
Terraformsh will use the appropriate options to create a plan file. This way
you can be sure that an apply or destroy operation will only happen on
a plan that has been saved to a file and reviewed. (You can disable this
automatic behavior by setting USE_PLANFILE=0 as an environment or configuration
variable)
The plan files are, by default, written to the directory where you ran Terraformsh,
with a naming convention like tfsh.92h39d9hd9.plan. You can override this by
setting environent or configuration variable TF_PLANFILE and TF_DESTROY_PLANFILE.
You can pass multiple Terraform commands as options and it'll run them in the order you specify.
Not sure what that looks like? Use the dry-run mode:
$ ./terraformsh -N plan apply
./terraformsh: WARNING: No -b option passed! Potentially using only local state.
+ terraform init -input=false -reconfigure -force-copy
+ terraform get -update=true
+ terraform validate
+ terraform plan -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan
+ terraform init -input=false -reconfigure -force-copy
+ terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/tf.104900abc1.plan
You can tell Terraformsh to change to a module's directory before running commands so you don't have to do it yourself (later versions of Terraform have an option for this, but earlier ones don't):
$ ./terraformsh -C ../../../root-modules/aws/common/ plan
You can pass Terraform configuration files using the -f or -b options.
$ terraformsh -C ../../../root-modules/aws/common/ \
-f terraform.tfvars.json \
-f override.auto.tfvars.json \
-b backend.tfvars \
-b backend-key.tfvars \
plan approve apply
To make this even simpler, if you pass any argument to Terraformsh after the
initial OPTIONS, and they match TFVARS ('*.backend.tfvars', '*.backend.sh.tfvars',
'*.tfvars.json', '*.tfvars', '*.sh.tfvars.json', '*.sh.tfvars'), they will
be automatically loaded with the -f and -b options.
# Assuming you already have 'something.tfvars' and 'something.backend.tfvars'
# in your current working directory, run the following:
$ terraformsh -C ../../../root-modules/aws/common/ \
*.tfvars \
plan approve apply
Finally, if in any parent directory of where you ran Terraformsh, there are
files named backend.sh.tfvars, terraform.sh.tfvars.json, or terraform.sh.tfvars,
those will also be loaded automatically (you can disable this with the -I option).
$ mkdir -p some/configs/here
$ cd some
$ touch terraform.sh.tfvars
$ cd configs
$ touch backend.sh.tfvars
$ cd here
$ touch terraform.sh.tfvars
$ terraformsh -N plan apply
+ terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars
+ terraform get -update=true
+ terraform validate -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars
+ terraform plan -var-file /home/vagrant/git/PUBLIC/terraformsh/some/terraform.sh.tfvars -var-file /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/terraform.sh.tfvars -input=false -out=/home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan
+ terraform init -input=false -reconfigure -force-copy -backend-config /home/vagrant/git/PUBLIC/terraformsh/some/configs/backend.sh.tfvars
+ terraform apply -input=false /home/vagrant/git/PUBLIC/terraformsh/some/configs/here/tf.019c25e289.plan
Don't want to remember what options to pass to terraformsh? You don't have to! You can capture anything you want Terraformsh to do in a config file that is automatically loaded.
The config file format is just a bash script. Therefore you can do things like 'export' arbitrary environment variables for Terraform to load, or even run custom code.
It's highly recommended that you do not set environment variables like
Terraform's TF_VAR_*, otherwise you will have a mix of variables set in both
config files and environment variables, and it will make it difficult to track
down where/how a variable is being set. Stick to static variables in
*.tfvars or *.tfvars.json files, and load dynamic variables from Terraform
with a data source.
You can set the following variables in a config file (any of:
/etc/terraformsh, ~/.terraformshrc, .terraformshrc, terraformsh.conf),
or set them as environment variables before you call Terraformsh:
DEBUG=1 # Enable bash tracing
TERRAFORM=terraform # The name of the terraform executable
TF_PLANFILE= # Automatically populated by terraformsh
TF_DESTROY_PLANFILE= # Automatically populated by terraformsh
TF_BOOTSTAP_PLANFILE= # Automatically populated by terraformsh
PUSH_ERRORED_TFSTATE=0 # Don't push errored.tfstate on failed apply
USE_PLANFILE=0 # Don't use a plan file for each apply/destroy
INHERIT_TFFILES=0 # Don't inherit tfvars files in parent directories
NO_DEP_CMDS=1 # Don't run dependent commands automatically
NO_CLEANUP_TMP=1 # Don't clean up temporary TF_DATA_DIR
DRYRUN=1 # Enable dry-run mode
CD_DIR= # The directory to change to before running terraform commands
The environment variable TF_DATA_DIR is automatically overridden by Terraformsh.
A new temporary directory is created for the data dir, based on both the name of
the directory you ran Terraformsh from, and the Terraform module directory
you run terraform against (the -C option). If you pass your own TF_DATA_DIR
environment variable, Terraformsh will use that instead.
The following can be set in the Terraformsh config file as Bash arrays, or you
can set them by passing them to -E, such as -E "PLAN_ARGS=(-no-color -input=false)".
VARFILES=() # files to pass to -var-file
BACKENDVARFILES=() # files to pass to -backend-config
CMDS=() # the commands for terraformsh to run
PLAN_ARGS=(-input=false) # the arguments for 'terraform plan'
APPLY_ARGS=(-input=false) # the arguments for 'terraform apply'
PLANDESTROY_ARGS=(-input=false) # arguments for 'plan -destroy'
DESTROY_ARGS=(-input=false) # arguments for 'terraform destroy'
REFRESH_ARGS=(-input=false) # arguments for 'terraform refresh'
INIT_ARGS=(-input=false -reconfigure -force-copy) # arguments for 'terraform init'
OH12UPGRADE_ARGS=(-yes) # arguments for '0.12upgrade'
OH13UPGRADE_ARGS=(-yes) # arguments for '0.13upgrade'
IMPORT_ARGS=(-input=false) # arguments for 'terraform import'
GET_ARGS=(-update=true) # arguments for 'terraform get'
STATE_ARGS=() # arguments for 'terraform state'
VALIDATE_ARGS=() # arguments for 'terraform validate'
WORKSPACE_ARGS=() # arguments for 'terraform workspace'
CONSOLE_ARGS=() # arguments for 'terraform console
OUTPUT_ARGS=() # arguments for 'terraform output'
TAINT_ARGS=() # arguments for 'terraform taint'
UNTAINT_ARGS=() # arguments for 'terraform untaint'
FORCEUNLOCK_ARGS=(-force) # arguments for 'terraform forceunlock'
SHOW_ARGS=() # arguments for 'terraform show'
To use the 'aws_bootstrap' command, pass the '-b FILE' option and make sure the file(s) have the following variables:
bucket - The S3 bucket your Terraform state will live in
dynamodb_table - The DynamoDB table your Terraform state will be managed in
An example file: .terraformshrc-example
Need to troubleshoot some problem by just running 'terraform' yourself? No
problem, use the shell command. It will drop you into a Bash shell after
first changing to the correct directory and running terraform init and
terraform get with all the environment variables set up for you
(including the automatic TF_DATA_DIR).
$ ./terraformsh -N -C ../../../root-modules/aws/common/ shell
+ cd "../../../root-modules/aws/common/"
./terraformsh: WARNING: No -b option passed! Potentially using only local state.
+ terraform init -input=false -reconfigure -force-copy
+ terraform get -update=true
+ bash -i -l
You can even get Terraformsh to explicitly ask you for confirmation before
moving to the next command with the approve command (since the default is
to pass -input=false to each command for easier use in automation).
Are you working in a hierarchy of config files, and want to grep all
the parent directories? Use the built-in revgrep command:
$ terraformsh revgrep -H -e "gcp_project_id"
terraformsh: Info: Found terraform command 'revgrep'
terraformsh: Warning: '-H' is not a valid command; passing as an option instead
terraformsh: Warning: 'project_id' is not a valid command; passing as an option instead
+ cd "/home/vagrant/my-repo/env/product/dev/nonprod/us-west1/tf-state/bootstrap"
/home/vagrant/git/my-repo/env/product/dev/nonprod/terraform.sh.tfvars:gcloud_project_id = "123456789"
Want to output one of Terraformsh's plan files as JSON?
$ terraformsh show -json "$(pwd)/tf.b063520160.plan"
There are many ways to use Terraformsh, whether you pass all the options via environment variables/command-line options, or keep all the commands in a configuration file and load everything automatically.
-
Run 'plan' using a
.terraformshrcfile that has all the above options, but override terraformsh's internal arguments to 'terraform plan':$ terraformsh -E 'PLAN_ARGS=("-compact-warnings" "-no-color" "-input=false")' \ plan -
Run 'plan' on a module and pass any configs found in these directories:
$ terraformsh -C root-modules/my-database/ \ *.tfvars \ env/my-database/*.tfvars \ plan -
Run 'plan' on a module, implicitly loading configuration files from parent directories:
$ pwd /home/vagrant/git/some-repo/env/non-prod/us-east-2/my-database $ echo 'CD_DIR=../../../../modules/my-database/' > terraformsh.conf $ echo 'aws_account_id = "0123456789"' > ../../terraform.sh.tfvars $ echo 'region = "us-east-2"' > ../terraform.sh.tfvars $ echo 'database_name = "some database"' > terraform.sh.tfvars $ terraformsh plan -
You've applied some Terraform using local state, and now you want to migrate it to a remote backend. After you add your new backend tf code, you'd run:
$ terraformsh -E "INIT_ARGS=()" init -force-copy -migrate-state plan apply
-
Problem: I'm using Terraformsh from two different shell sessions, in the same directory, running the same commands, but one of them is working and the other isn't. What's going on?
Solution: Something's wrong with your environment variables in one of the sessions. If both
TF_DATA_DIRandTF_TMPDIRare set to something starting with/tmp/tfsh., then you probably usedterraformsh shelland forgot to exit.
terraformsh v0.14
Usage: ./terraformsh [OPTIONS] [TFVARS] COMMAND [..]
Pass these OPTIONS before any others (see examples); do not pass them after TFVARS or COMMANDs.
-f FILE A file passed to Terraform's -var-file option.
( config: VARFILES= )
-b FILE A file passed to Terraform's -backend-config option.
( config: BACKENDVARFILES= )
-C DIR Change to directory DIR.
( config: CD_DIR= )
-c file Specify a '.terraformshrc' configuration file to load.
-E EXPR Evaluate an expression in bash ('eval EXPR').
-I Disables automatically loading any 'terraform.sh.tfvars',
'terraform.sh.tfvars.json', or 'backend.sh.tfvars' files
found while recursively searching parent directories.
( config: INHERIT_TFFILES=0 )
-P Do not use '.plan' files for plan/apply/destroy commands.
( config: USE_PLANFILE=0 )
-D Don't run 'dependency' commands (e.g. don't run "terraform
init" before "terraform apply").
( config: NO_DEP_CMDS=1 )
-N Dry-run mode (don't execute anything).
( config: DRYRUN=1 )
-n Don't remove the temporary TF_DATA_DIR.
( config: NO_CLEANUP_TMP=1 )
-v Verbose mode.
( config: DEBUG=1 )
-h This help screen.
The following are Terraform commands that terraformsh provides wrappers for (there's some Terraformsh-specific logic behind the scenes). Other Terraform commands not listed here are passed through to Terraform verbatim.
plan Run init, get, validate, `terraform plan @VARFILE_ARG -out $TF_PLANFILE`
apply Run init, get, validate, `terraform apply $TF_PLANFILE`
plan_destroy Run init, get, validate, `terraform plan -destroy -out=$TF_DESTROY_PLANFILE`
destroy Run init, get, validate, `terraform apply $TF_DESTROY_PLANFILE`
refresh Run init, `terraform refresh`
validate Run init, get, `terraform validate`
init Run clean_modules, `terraform init @BACKENDVARFILE_ARG`
get Run init, `terraform get [..]`
show Run init, `terraform show [..]`
import Run init, `terraform import [..]`
state Run init, `terraform state [..]`
taint Run init, `terraform taint [..]`
untaint Run init, `terraform untaint [..]`
output Run init, refresh, `terraform output [..]`
console Run init, `terraform console [..]`
workspace Run init, `terraform workspace [..]`
force-unlock Run init, `terraform force-unlock [..]`
0.12upgrade Run init, `terraform 0.12upgrade [..]`
0.13upgrade Run init, `terraform 0.13upgrade [..]`
The following commands are specific to terraformsh:
shell Run init, get, and `bash -i -l`
clean Remove '.terraform/modules/*', terraform.tfstate files, and .plan files
clean_modules Run `rm -v -rf .terraform/modules/*`
approve Prompts the user to approve the next step, or the program will exit with an error.
aws_bootstrap Looks for 'bucket' and 'dynamodb_table' in your '-b' file options.
If found, creates the bucket and table and initializes your Terraform state with them.
revgrep Run 'grep' on files in all parent directories
env Run 'env' command with optional arguments
All arguments after a COMMAND are evaluated for whether they match a Terraform or Terraformsh command; if they don't, they are assumed to be options and are passed to the first recognized command that precedes them.