aws/aws-cdk-rfcs

cdk assert: Regression test support

eladb opened this issue · 10 comments

eladb commented

Support an ability to "regress" a synthesized template against a checked-in version. Think of it as a template "lock file".

This capability allows developers to make sure that infrastructure changes are audited (code reviewed) and committed to the code repository. It also allows developers to discover unexpected infra changes before pushing them to a CI/CD pipeline (as they are represented as file diffs).

Here's a proposed usage:

$ cdk assert --lockfile ./mystack.json [stack]

This will synthesize the specified stack and will diff it with the contents of ./mystack.json.

  1. If the file doesn't exist, it will indicate that this is a new file and exit with non-zero exit code.
  2. If the file exists, and equals, it will just exit with 0
  3. If the file exists, and differs, it will print a friendly diff, update the file, and exit with a non-zero exit code.

This command is designed to be executed as a package test. In case the user pushes a change to the code that resulted in a template change, but did not run the test (which means they did not "acknowledge" the change), the test will fail with informative information. If the user runs the test locally, the first time will display the diff and modify the file, which will be surfaced when the user commits their code into the repository.

Consider if we want to merge this functionality with cdk-integ (#113, aws/aws-cdk#40, aws/aws-cdk#41).

RomainMuller commented

It's a matter of personal taste, I think, but I would rather use --lock and it emits <stack>.lock.json. I don't see a tremendous value for allowing user-configured file location.

eladb commented

--lock as a flag for which subcommand?

eladb commented

@mipearson wrote:

First, see aws/aws-cdk#1532

I've also set up "snapshot testing" via jest. This makes it easier to see the impacts of changes that I'm making to my stack without having to run cdk synth manually. It also means if I'm making a change that unexpectedly changes logical IDs or something else unexpected I'm told about it very quickly.

Snapshot testing isn't intended as "this works", but "this did/didn't change". It's also very, very easy to set up: unlike other forms of testing, it will autogenerate the expected output from the first run, and interactively ask you to confirm changes rather than assuming that they count as a "failure".

Given the boilerplate set up by cdk init sample-app, here's what I did to set up a very simple snapshot test around the cdk-stack.ts file:

Ran:

npm i ts-jest jest @types/jest

Edited package.json, added:

  "jest": {
    "transform": {
      "^.+\\.tsx?$": "ts-jest"
    },
    "testRegex": "(/__tests__/.*|(\\.|/)(test|spec))\\.(jsx?|tsx?)$",
    "moduleFileExtensions": [
      "ts",
      "tsx",
      "js",
      "jsx",
      "json",
      "node"
    ],
    "globals": {
      "ts-jest": {
        "diagnostics": {
          "warnOnly": true
        }
      },
    "testEnvironment": "node"
    }
  }

And added "test": "jest --watch", to the scripts key.

Created a lib/cdk-stack.test.ts with the following contents:

import { CdkStack } from "./cdk-stack";
import cdk = require("@aws-cdk/cdk");

test("CdkStack matches snapshot", () => {
  const stack = new CdkStack(new cdk.App(), "CdkStack", {});
  expect(stack.toCloudFormation()).toMatchSnapshot();
});

Ran npm run test and was presented with the following output:

 PASS  lib/cdk-stack.test.ts
  ✓ CdkStack matches snapshot (47ms)

 › 1 snapshot written.
Snapshot Summary
 › 1 snapshot written from 1 test suite.

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   1 written, 1 total
Time:        2.968s
Ran all test suites related to changed files.

This then creates a lib/__snapshots__/cdk-stack.test.ts.snap file containing the synthesized stack, which I add to source control.

As I add things to the stack I expect that snapshot to "break" - jest will prompt me as to whether I want to update the snapshot or treat it as a failure. If I add a new sns.Topic(this, "AnotherTopic"); like to the cdk-stack.ts, I'll then receive this:

    Received value does not match stored snapshot "CdkStack matches snapshot 1".

    - Snapshot
    + Received

    @@ -1,7 +1,10 @@
      Object {
        "Resources": Object {
    +     "AnotherTopicC20D17AD": Object {
    +       "Type": "AWS::SNS::Topic",
    +     },

Pressing u updates the snapshot for me.

btotharye commented

hey @eladb I like what you did with the snapshot testing but it appears that toCloudFormation no longer exists in the stack class/module how are you doing this now out of curiosity?

btotharye commented

@eladb awesome thanks

JReinhold commented

I think this would be a great feature. I thought that I could use snapshot testing for most of my stack, but currently we can't run snapshot tests through CI, because some of the generated id's are different for each environment, and thus snapshots won't be the same on CI and locally. This results in jest failing, because it expects exact matching snapshots.

hupe1980 commented

@JReinhold I had the same problem. I solved it with property matchers:
https://github.com/hupe1980/jest-cdk-snapshot#propertymatchers

eladb commented

Duplicate with #31

inakianduaga commented

Would be nice to have YAML conversion for the snapshots out of the box. We are currently defining a custom Jest extension that is doing the equivalent of

import { SynthUtils } from '@aws-cdk/assert'
import * as jsYaml from 'js-yaml'

expect(
        jsYaml.dump(SynthUtils.synthesize(stack).template)
      ).toMatchSnapshot()

which we provide as a

expect(stack).toMatchCdkSnapshot(selector?: (artifact: CloudFormationStackArtifact) => object)

where the selector can be used to pick a specific part of the template (or do any sort of transformation) that we end up snapshotting.

This is much more readable that dumping plain JS diffs (yaml > JS in this case)