Option to load placeholder values from a file

[mike-rsi]

I would love to have the ability to load placeholders from a file as opposed to having to pass them on the command line.

I find I like to use the same .config file for multiple environments (e.g. dev, prod) and just make minor tweaks to some values (e.g. prepending stack names and tag names with the environment name, or changing the first two octets of VPC and subnet CIDRs which are passed as parameters, etc.). When you have multiple environments with multiple placeholders, this gets cumbersome.

An alternative is writing an all new .config for each environment, but in complicated .config files this makes things much less DRY. I often have very flexible CFN templates which use a lot of parameters. Those parameters don't change much between environments of the same project, (but may change a lot between projects) so it'd be nice to use the same .config for each environment in a project, with an additional file (.placeholders?) to hold small differences between environments.

A couple examples of how this could be implemented: adding a placeholder-file option, or extending the existing placeholder option to be similar to how the AWS CLI --parameters flag works, where it can take a shorthand syntax or file://... as a value.

Deploying two different environments for the same project may look like:

dev.placeholders:

{
  "{Namespace}": "Dev",
  "{CidrPrefix}": "10.0"
}

prod.placeholders:

{
  "{Namespace}": "Prod",
  "{CidrPrefix}": "10.1"
}

Shell commands to create stacks DevVpc and ProdVpc:

$ aws-cfn create --config-file project.config --environment vpc --placeholder-file dev.placeholders
$ aws-cfn create --config-file project.config --environment vpc --placeholder-file prod.placeholders

If you think this is an idea that you'd like to incorporate into the main project, I could take a first crack at implementing it and make a PR.

[carlnordenfelt]

Hi,

I just want to make sure I don't misunderstand what you need.

There is functionality to separate environments in the config file already:

{
    "default": {
        "Paramters": {
            "DefaultParameter": "DefaultValue"
        }
    },
    "dev": {
        "Parametes": {
             "Cidr": "10.0"
        }
    },
    "prod": {
          "Parameters": {
               "Cidr": "10.1
        }
    }
}

This would give CloudFormation DefaultParameter with DefaultValue and Cidr with either 10.0 or 10.1 depending on which environment config you use. Basically the idea is to make your example calls with --environment dev or --environment prod instead of --environment vpc.

Placeholders are intended to be used to replace variables that differ at deploy time, i.e. are not known in advance. For example a package version which usually consists of a build number or commit hash.

If I understand you correctly this isn't sufficient for your use case?
I am not entirely sure if I understand the use case though :)

[mike-rsi]

Ah, yes, I should have clarified. I think I abuse the environments a bit. The problem with my typical workflow is that I build environments out of multiple templates and stacks (even templates are often reused for multiple stacks as well, just with different parameters passed to them). So what you call environments is more akin to "stacks" in my workflow, and I use a separate .config file for each of my "environments". So I may have a file laid out like:

prod.config:

{
    "default": {
        "Parameters": {
            "CidrPrefix" : "10.0"
        }
    },
    "vpc": {
        ...
    },
    "nacls": {
        ...
    },
    "security-groups": {
        ...
    }
}

Then to create the entire production environment, I have a wrapper script that loops through and runs aws-cfn create ... for each environment a.k.a. stack in the .config file.

Perhaps the true feature request here is to allow environments to be made up of multiple stacks. I haven't thought that through completely though, and whether or not a placeholder file would still be beneficial if that was the case.

[carlnordenfelt]

I'm still not 100% sure I understand your use case completely and with that not entirely convinced that adding a placeholder file is a good idea. Not saying I'm against the idea, just need to wrap my head around the benefit.

The intention with placeholders from my side was to replace runtime values which aren't known and then use the environments for values that never or rarely change. I understand that you're not using it quite this way and if I understand you correctly you are missing another layer of configuration. Does that sound about right?

I'm trying to think of a way to use the current functionality in another way to support your use case :)

One thing that I have on my list is to allow UsePreviousValue in the config but I guess that won't do you any good.

[mike-rsi]

Thinking about it further, supporting multiple stacks per environment still wouldn't solve the core problem reported in this issue. Hopefully I can further illustrate my point.

The .config file for one of my projects right now is about 250 lines. It defines 14 environments (I'll use the tool's terminology here) from 12 CFN templates (one gets reused for 3 environments). Because each environment in this file is actually a separate piece of one complete infrastructure deployment (e.g. "prod"), they don't share many common parameters at all. The "default" section of the .config file can't be used effectively as you designed it because parameters in the "default" section need to exist in every environment in the .config.

In this use case, to make a deployment that has some slight tweaks (e.g. a different CIDR and name tag prefix, maybe reduce the instance count of an ASG, maybe a different AMI ID because it's for a different region) I would have to make a copy of this file and edit a number of environment parameter values. Maintaining two copies (or maybe 3 or 4) of what is essentially the same, large .config becomes a maintenance nightmare because ongoing modifications need to be propagated to every copy of the file.

Using placeholders for any of these values that change between deployments allows me to reuse just one .config, but then I have to type a bunch of them in on the command line every time, and can't effectively persist these values anywhere.

I could write a wrapper script that reads in a file and generates the --placeholder flags for me, but I thought it may be a feature worthy of being in the core. Maybe my usage of the tool is too far removed from what you intended however.