buildkite/elastic-ci-stack-for-aws

v6.15.0 is missing AWSRegion2AMI mappings for all regions except us-east-1

Closed this issue · 9 comments

Describe the bug
I tried to upgrade from v6.13.0 of the stack to v6.15.0 but my CloudFormation rolled back early in the process.
In the CloudFormation events were the following errors:

Template error: Unable to get mapping for AWSRegion2AMI::ap-southeast-2::windows

If I look at the template as downloaded via https://s3.amazonaws.com/buildkite-aws-stack/v6.15.0/aws-stack.yml then I see this:

  # Generated from Makefile via build/mappings.yml
  AWSRegion2AMI:
    us-east-1 : { linuxamd64: ami-0d533485c83fbfd81, linuxarm64: ami-0147f31e9ef0f02eb, windows: ami-044673d5deecd80aa }

While if I download https://s3.amazonaws.com/buildkite-aws-stack/v6.14.0/aws-stack.yml then I see this:

  # Generated from Makefile via build/mappings.yml
  AWSRegion2AMI:
    us-east-1 : { linuxamd64: ami-0183caef313aa57a8, linuxarm64: ami-005b1246b7c312d75, windows: ami-0887d14cbefc8ae3d }
    us-east-2 : { linuxamd64: ami-00de9036a36f73520, linuxarm64: ami-0db9ecf56f4a9313e, windows: ami-0e310d0db6452f793 }
    us-west-1 : { linuxamd64: ami-0defbc65ceaa32dd1, linuxarm64: ami-0820aec2c904d15e1, windows: ami-0cdc69b863dbc63ed }
    us-west-2 : { linuxamd64: ami-0b30ab2eea949e93e, linuxarm64: ami-0a0810ea0d35dfa04, windows: ami-07069601abe24b0c6 }
    af-south-1 : { linuxamd64: ami-09a95859b666c385c, linuxarm64: ami-023856db3ff9ffebf, windows: ami-0aaf725c9d56214a5 }
    ap-east-1 : { linuxamd64: ami-033d34c189faaa3ea, linuxarm64: ami-0e7a59eb736084415, windows: ami-0b853d3ff4837fb8d }
    ap-south-1 : { linuxamd64: ami-0c1e25e1e786bf5ab, linuxarm64: ami-0f37afba54066bd60, windows: ami-019ca0ce695b202d5 }
    ap-northeast-2 : { linuxamd64: ami-06d8f125ea12c73a4, linuxarm64: ami-01d26356d8601112a, windows: ami-08149bf6171985f3f }
    ap-northeast-1 : { linuxamd64: ami-0337fff9cddb16928, linuxarm64: ami-06bd64a2473145d48, windows: ami-06f7d221ae9077f91 }
    ap-southeast-2 : { linuxamd64: ami-0b27fff20c46df4a0, linuxarm64: ami-0495bb42e2c5ad16b, windows: ami-0c46183f41186a75f }
    ap-southeast-1 : { linuxamd64: ami-052adc4e5f2b331d3, linuxarm64: ami-0eeb8dfc3642c4ff9, windows: ami-031add0bdca9cea1a }
    ca-central-1 : { linuxamd64: ami-0ec8ad5aa316be9c6, linuxarm64: ami-0fd3c285174fd4aa6, windows: ami-03624e0a25b90a666 }
    eu-central-1 : { linuxamd64: ami-0ea202151b77cd132, linuxarm64: ami-0d87234a4153da86d, windows: ami-08b686742f62d0c52 }
    eu-west-1 : { linuxamd64: ami-07ee5053aa22fcc0e, linuxarm64: ami-01a6b461c70f92a37, windows: ami-0125a5a3c5a5e09bb }
    eu-west-2 : { linuxamd64: ami-0cea94b9f4862f155, linuxarm64: ami-00963a2bbcdc29e6c, windows: ami-00cf533b2d4fff822 }
    eu-south-1 : { linuxamd64: ami-0fa3808b03f7467c2, linuxarm64: ami-0fae08dd40e29217d, windows: ami-0e2abc82c9ba72992 }
    eu-west-3 : { linuxamd64: ami-02b949b8b4659ff01, linuxarm64: ami-009c57fae557b81c3, windows: ami-0a82cb3a3e802818e }
    eu-north-1 : { linuxamd64: ami-05a2dbe3cca823d1b, linuxarm64: ami-03c222dd8c1c9d76c, windows: ami-060242282b5e2b9b7 }
    me-south-1 : { linuxamd64: ami-0e909c7019ac9d74a, linuxarm64: ami-061aa8c89e3e2dcee, windows: ami-0408bcd45277ac97d }
    sa-east-1 : { linuxamd64: ami-071f30371b0aeaed3, linuxarm64: ami-0332cfe794d7db30b, windows: ami-0704755918feea0c8 }

Steps To Reproduce
Steps to reproduce the behavior:
Try to install the stack in any region other than us-east-1 and I'm sure you'll see the error.

Expected behavior
The stack should upgrade.

Actual behaviour
The stack had to roll back.

Stack parameters (please complete the following information):

  • AWS Region: ap-southeast-2
  • Version v6.15.0

Additional context
I think I've covered it all.

Confirming this issue.

It also appears to be a regression.

#757

Interestingly latest is not showing the issue (and notes it is v6.15.0).


  # Generated from Makefile via build/mappings.yml
  AWSRegion2AMI:
    us-east-1 : { linuxamd64: ami-0183caef313aa57a8, linuxarm64: ami-005b1246b7c312d75, windows: ami-042afc44e66ce87e5 }
    us-east-2 : { linuxamd64: ami-055c3faafbca08196, linuxarm64: ami-00cf20791d56435f0, windows: ami-06ddadbcfa4c8d3a1 }
    us-west-1 : { linuxamd64: ami-0556ebadfc00fd54d, linuxarm64: ami-06058a638754410c8, windows: ami-00465593433099006 }
    us-west-2 : { linuxamd64: ami-0e0599c52a684b6bb, linuxarm64: ami-0d08c283de76a8b51, windows: ami-0e3287fc37a328c60 }
    af-south-1 : { linuxamd64: ami-0ce7e0c1fda35084e, linuxarm64: ami-0002b331d444852db, windows: ami-0b807bbc754c67d32 }
    ap-east-1 : { linuxamd64: ami-03509f882fb5aebfb, linuxarm64: ami-0a009c13db263f6d2, windows: ami-0b574a3ab483ff4cd }
    ap-south-1 : { linuxamd64: ami-09faa840f14b7dccb, linuxarm64: ami-0f559c37677a6add1, windows: ami-0086f348589a9765c }
    ap-northeast-2 : { linuxamd64: ami-02b0f1ee7a05244f6, linuxarm64: ami-0d30857a06ce94d0c, windows: ami-094aabf2b62c86751 }
    ap-northeast-1 : { linuxamd64: ami-0524bfea13f6c566c, linuxarm64: ami-00d382a5c0f4b15d3, windows: ami-035a22c11802f59a0 }
    ap-southeast-2 : { linuxamd64: ami-0dfbc573b01df5b7b, linuxarm64: ami-029a286424dfe3c98, windows: ami-03651b9fa2f2acc8b }
    ap-southeast-1 : { linuxamd64: ami-07e622d396787084d, linuxarm64: ami-0fbde30ec2a2d75e9, windows: ami-096fc38e097c52e48 }
    ca-central-1 : { linuxamd64: ami-09e1f16af0558577d, linuxarm64: ami-007ce148abc0d7c96, windows: ami-0b03bf91c5fb63efd }
    eu-central-1 : { linuxamd64: ami-0b5daf927a2883c52, linuxarm64: ami-0c848a269a6cd2169, windows: ami-0960daaa4fc7be5b0 }
    eu-west-1 : { linuxamd64: ami-009ead2e9445e71d1, linuxarm64: ami-08ad8f0670fd81161, windows: ami-0c7c75db5d9196add }
    eu-west-2 : { linuxamd64: ami-0db9c06081658fcee, linuxarm64: ami-05dbe98fabcad573d, windows: ami-054da271e36554df5 }
    eu-south-1 : { linuxamd64: ami-0e71f615952657992, linuxarm64: ami-0fa494daee735cc86, windows: ami-033a3df0cfbb24f0d }
    eu-west-3 : { linuxamd64: ami-0ad94f19599441ecc, linuxarm64: ami-0c56129301f7e395a, windows: ami-066e77a9a222a74c0 }
    eu-north-1 : { linuxamd64: ami-00f3913a69c71b807, linuxarm64: ami-019dbec60f6def3b1, windows: ami-0a76a2bef3fe542dc }
    me-south-1 : { linuxamd64: ami-0fda8c2674dc0ff37, linuxarm64: ami-0025908adae20f679, windows: ami-0d4c506d1ae013351 }
    sa-east-1 : { linuxamd64: ami-0d0fc147f40628556, linuxarm64: ami-01449121016ddf072, windows: ami-07e104784f1cdbc2e }

Related: #716 (comment)

Thanks for raising this @jim-barber-he, @jufemaiz. Earlier today, we noticed that the latest template references AMIs that DO NOT have the fix for CVE-2024-21626. We are trying to rectify this, but as part of that process, a template for v6.15.0 was published that only contains AMIs for us-east-1. We're in the process of copying over the AMIs to the other regions now.

Ok, looks like both latest and v6.15.0 have the correct AMIs for all regions now, and the templates should have the same contents. Please let us know if you're finding otherwise.

Success! Cheers @triarius!

Would it be worth having a smoke test environment that automatically validates amis in all regions prior to official release?

We've got some plans to improve the release automation, I think it would be good to have such a validation in there before the GitHub release is made.

Confirmed that this is fixed at my end too.