TFlat = Terraform + Subdirectories + Ruby (ERB)

Aren't you tired of copy and pasting?

I love Terraform, but HCL really gets in the way sometimes.

How many times you wish you could just write a simple IF or CASE statement inside a .tf file? Any attempt of having minimal flow control with HCL results in a massive oneliner mess. Sometimes it feels like writing PERL one-liners. Hard to read equals hard to debug.

Also, you can't use subfolders with Terraform, so you often end up at one of the three scenarios below:

You have a few .tf files with a lot of code in it. Ugly and not organized.
You create lots of different .tf files in a single directory, which makes it really hard to stay organized.
You separate everything in modules, so you have to keep passing variables downstream. And if you try to be DRY, good luck passing 1 million variables downstream to submodules!

TFlat does 2 things to solve this problem:

Separate your Terraform code in subdirectories.
It allows you to write Ruby code in .tf files using ERB templates. Hurray!

Jump to:

Installation
Usage

How does TFlat make Terraform read subdirectories?

It doesn't. This is what happens when you run it:

Create a folder .tflat inside the current folder if it doesn't exist yet. If it does exist, delete all files from this folder non-recursively.
Make a recursive list of all files in the current directory and move one by one to the .tflat folder.
Replace all non-binary files in .tflat/ with its ERB rendered version.
Execute terraform with the arguments you passed to tflat.

For example, say you have the following file structure:

|_ config
  |_ providers.tf
  |_ state.tf
|_ ec2
  |_ files
    |_ user_data.sh
  |_ instances.tf
  |_ keypairs.tf
|_ outputs.tf
|_ variables.tf

TFlat will create the following files inside ./.tflat/

config#providers.tf
config#state.tf
ec2#files#user_data.sh
ec2#instances.tf
ec2#keypairs.tf
outputs.tf
variables.tf

Then it will cd into .tflat and run terraform with the arguments you passed to tflat.

IMPORTANT: The .terraform folder will live at .tflat/.terraform, so make sure you don't delete that folder if you are storing the terraform state locally!

TFLAT Workspaces

Say you want to run the same terraform code against multiple environments and cloud regions.

You can do that by setting the environment variable TFLAT_WORKSPACE in your shell, plus any other TF_VAR_'s that are required to achieve what you want.

For example, say I have some terraform code setup with remote state using S3 + DynamoDB as backend. I want to run it against the staging environment on ca-central-1.

Here is my state.tf:

terraform {
  backend "s3" {
    encrypt        = true
    bucket         = "my-s3-bucket"
    region         = "ca-central-1"
    dynamodb_table = "my-dynamo-table"
    key            = "<%= ENV['TFLAT_WORKSPACE'] %>/terraform.tfstate"
  }
}

And here is providers.tf:

variable "region" {}
provider "aws" {
  version = "~> 1.50"
  region  = "${var.region}"
}

And here is how I would set my shell environment:

export TF_VAR_region=ca-central-1
export TFLAT_WORKSPACE=staging/ca-central-1

When you run tflat init, you will notice that the .tflat folder is really .tflat.staging/ca-central-1, and your state is in the key staging/ca-central-1/terraform.tfstate inside of S3!

There's only one more thing you have to pay attention to: handling file references.

Handling file references

Because TFlat will actually copy and rename files to make it work with subdirectories, you need to pass file references in a different way. For example, imagine you are rendering a terraform template like this:

files/userdata.tpl

#!/bin/bash
# ...
CONSUL_ADDRESS=${consul_address}
# ...

main.tf

# ...
data "template_file" "ec2_userdata" {
  template = "${file("files/userdata.tpl")}"

  vars {
    consul_address = "${aws_instance.consul.private_ip}"
  }
}

# Create a web server
resource "aws_instance" "web" {
  # ...

  user_data = "${data.template_file.ec2_userdata.rendered}"
}
# ...

The line template = "${file("files/userdata.tpl")}" has to be written in one of the following ways to work with TFlat:

# Let Ruby load the file content using the 'file' helper method (easier to read)
template = "<%= file('files/userdata.tpl') %>"

# Let Terraform load the file content using the 'f' helper method (quoting nightmare!)
template = "${file("<%= f('files/userdata.tpl') %>")}"

IMPORTANT: The file path must be relative to the project's root folder!

It is up to you to choose what you like. Try both and look inside .tflat/main.tf to see the difference between the two ways.

You can also use <%= file_sha256('path/to_file') %> to return its SHA256.

Installation

Download and install Terraform. Make sure the terraform command is in your $PATH.
Install the gem:

$ gem install tflat

Usage

TFlat takes the same arguments from Terraform. It actually hands off the execution to Terraform after processing the files. So:

terraform plan

Becomes:

tflat plan

That's it!

Using TFVARS inside the ERB template

If you store your variables in JSON format inside a file named terraform.tfvars.json, those variables will be automatically available for you as a HASH named @variables (with string keys). For example:

# terraform.tfvars.json
{
  "app_domain": "example.com"
}

# files/swarm-stack.yml
version: "3.7"
services:
  ...
  myapp:
    ...
    environment:
      - DOMAIN_NAME=<%= @variables['app_domain'] %>
...

That way you don't have to deal with Terraform templates!

NOTE: Environment variables like TF_VAR_* are also accessible in Ruby, and will take precedence if the same value is set in the JSON variables file.

Ignoring certain folders

If you want TFlat to ignore a file or folder, just add a '#' to its name. This is useful when you want to destroy a chunk of infrastructure really quick, or when you don't want to apply something that you have been working on yet.

For example:

|_ config
  |_ providers.tf
  |_ state.tf
|_ #ec2    <--- This folder and its contents will be ignored by TFlat
  |_ files
    |_ user_data.sh
  |_ instances.tf
  |_ keypairs.tf
|_ #resources.tf  <---- This file will be ignored by TFlat
|_ outputs.tf
|_ variables.tf

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/parruda/tflat. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.

License

The gem is available as open source under the terms of the MIT License.

kanatsultan-zz/tflat