Allow including multiple files as part of a single stack

[naftulikay]

I often work with big stacks that have multiple resources:

• DNS records (Route53)
• Auto-Scaling Groups
• Security Groups
• ECS Configuration
  • Clusters
  • Services
  • Task Definitions
• S3 Buckets

All of this presently can live in a single massive awful JSON blob. Trading a single massive JSON blob for a single massive YAML blob is a little bit better, but would be amazing would be if I could have a single stack which did something like:

---
name: baby-got-stack
resources:
    - resources/launch-configs.yml
    - resources/autoscaling-groups.yml
    - resources/ecs.yml
    - resources/security-groups.yml
    - resources/dns.yml

---
AWSTemplateFormatVersion: '2010-09-09'
Description: The most amazing stack.
Resources: {} # empty dictionary which will accept a union of the resources defined above

I may invest some time in a pull request to do this. I made a homebrew YAML to CloudFormation JSON project for my company, but it lacks the niceties that stacks has (namely sane config management and Jinja2).

[vaijab]

@rfkrocktk you can actually achieve that already using jinja2 include functionality. I use that in various templates already. Here is how:

stacks/templates/snippets/instance_profile.yaml:

InstanceProfile:
  Type: AWS::IAM::InstanceProfile
  Properties:
    Roles:
      - {{ get_stack_output(cf_conn, env + '-infra', 'ComputeCoreOSRole') }}
    Path: /

stacks/templates/coreos.yaml:

---
name: {{ env }}-coreos-compute
disable_rollback: false

---
AWSTemplateFormatVersion: '2010-09-09'
Description: CoreOS Compute Cluster Stack in {{ env }} environment
Resources:
{% filter indent(2) -%}
{% include 'snippets/instance_profile.yaml' %}
{% endfilter %}

[vaijab]

@rfkrocktk I normally have multiple smaller stacks rather than one huge stack. I use cross-stack referencing to reference resources in other stacks. For example I have infra stack were I create VPC, empty security groups, subnets and so on. And then I have specific service/app stacks where I reference infra stack resources. That way I can update stacks independently.

[naftulikay]

My own homebrew solution basically had hardcoded directory lookups so that:

1. it'll find all YAML files in resources/ and union them into the Resources dictionary.
2. it'll find all YAML files in outputs/ and union them into the Outputs dictionary.
3. it'll find all YAML files in parameters/ and union them into the Parameters dictionary.
4. it'll find all YAML files in mappings/ and union them into the Mappings dictionary.

Your design heavily favors having multiple stacks and linking them together, which is nice, but for certain scenarios, it'd still be really nice to be able to split things up like this. Having a way to do this for multiple environments and multiple stacks and allowing merging between them when desired would be awesome.

If I'm able to find the time, I'll most likely be contributing changes here, though they'll probably be large and sweeping, but I'll try to keep things largely compatible with your design. One thing that is a priority for us to change is the handling of Jinja within YAML templates. I'd like to keep things compatible with YAML syntax so that a linter would be able to determine validity.

As for syntax, the main reason for Jinja (apart from simple string templating) is the loops and if conditions. For instance, rather than this:

---
Resources:
    AutoScalingGroup:
        Type: AWS::Autoscaling::Group
        Properties:
            VPCZoneIdentifier:
            {% for subnet in asg.subnets %}
                - "{{ subnet }}"
            {% endfor %}
            AvailabilityZones:
            {% for zone in asg.availability_zones %}
                - "{{ zone }}"
            {% endfor %}

we'd use the following, which is valid YAML:

---
Resources:
    AutoScalingGroup:
        Type: AWS::Autoscaling::Group
        Properties:
            VPCZoneIdentifier:
                # fill in VPCZoneIdentifier with an array where each value is the subnet id
                stacks::for:
                    header: subnet in subnets
                    body:
                        - "{{ subnet }}"
            AvailabilityZones:
                stacks::for:
                    header: zone in asg.availability_zones
                    body:
                        - "{{ zone }}"

This way, we're using Jinja in YAML like Ansible is using Jinja in YAML, in a way that doesn't invalidate YAML.

Other ideas include allowing a full Jinja2 template to be imported for things like a launch config with Jinja filters for including Ref and Fn::FindInMap statements as Jinja filters:

eg. templates/asg-launch-config.j2:

#cloud-init
hostname: {% stacks.find_in_map('MyMappingName', stacks.ref('ParameterName')) %}

This would expand into the CloudFormation literal:

{ "Fn::Join": ["\n", [
    "#cloud-init",
    { "Fn::Join": ["", ["hostname: ", { "Fn::FindInMap": ["MyMappingName", { "Ref": "ParameterName" }] }]] }
]]}

As someone who regularly has to type out this nonsense, the Jinja equivalent would save much sanity on my part.

[naftulikay]

By the way, I'm totally with you on separating things like networking infrastructure (ie: VPCs, subnets, etc.) from stacks that run in them to avoid destroying things accidentally. That makes a lot of sense.

[vaijab]

Thanks for the feedback. I try to keep stacks very lean and not add a lot of bespoke functionality to abstract away cloudformation. There are tools that allow you to write pure python to then build cloudformation json templates.

The goal of stacks is a thin layer on top of cloudformation which is yaml+jinja2 with some basic AWS API functions for getting value into your templates (mostly other resources referencing).

[naftulikay]

Thanks for your response. I might be creating my own project for this support then and will credit you with a lot of the inspiration. Thanks for creating stacks!

[vaijab]

You're welcome! :-)