CloudFormation did not receive a response from your Custom Resource
Closed this issue · 19 comments
Hi, we seem to be stuck at this last piece of trying to deploy. The cognito fixes you implemented worked great and we no longer see that issue, but we have been having problems from the start with these custom resources: NonceSigningSecret, HttpHeadersHandlerCodeUpdate, and StaticSite. On all the other resources, we do see a status of Resource creation initiated and the resource itself created, but for those three, we never get to a point where it initiates the resource creation. The actual error we get for those three is: "CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [85c5c344-6284-404c-9a0a-6b4f772d6c05]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version."
We've looked through this and cannot determine why we consistently get this error. The code doesn't appear to be using Python, instead using TypeScript so maybe it's a generic error? Any help you can provide to get us over this last bit of issue is greatly appreciated. We've tried Googling the error, but cannot find anything useful.
Can't explain it, it is really weird that some custom resources would time-out while others don't. All of them have been coded similarly and handle uncaught exceptions. They don't need permission to callback to the CloudFormation signed URL, so that can't be it either.
You'll have to look in the CloudWatch logs of the concerned Lambda functions and see what that says. Log group names you'll need to look in should be similar to:
- ...RandomValueGenerator...
- ...LambdaCodeUpdateHandler...
- ...StaticSiteHandler...
Tried looking in Cloudwatch, but since it doesn't initiate the creation of those custom resources, there's nothing in Cloudwatch to point us in a direction. This is literally the only thing holding up the full deployment at this stage from what I can see.
Can you paste a screenshot here, of the CloudFormation events tab that shows the errors?
You could try navigating to these lambda functions and just running them with a test payload (doesn't have to be meaningful) just to see what you get yourself if you try to invoke them. There might be a very fundamental error, that prevents CloudFormation from running them, you should see that yourself then too.
- ...RandomValueGenerator...
- ...LambdaCodeUpdateHandler...
- ...StaticSiteHandler...
Hi Otto,
Here's the screenshot you requested
The problem with trying a test payload against them is that you can't actually get into the resources in Lambda because they aren't being created. There's just placeholder names there, but they're not clickable to get into them. See this screenshot:
So, whatever is going on, is preventing these final 3 resources to be created. We see it get to the Create Initiated step but it never actually creates them, then fails after an hour when it exhausts its retry limit. Since it never starts the creation, there's no logs that I can find either.
As always, I appreciate your help and feedback on this. You're awesome!
As always, I appreciate your help and feedback on this. You're awesome!
Cheers mate!
I see e.g. "StaticSiteHandler" is blue and a valid link? That's what I'd be after, that takes you to the Lambda that implements the "StaticSite". If you run that Lambda manually, with a dummy payload, what happens? It should fail because that dummy payload is invalid (expected) but not because of some other error (like import errors, or code parse errors).
Note, that Lambda function should write logs to CloudWatch (that you can navigate to from within the Lambda function, in the monitoring tab). The Lambda function might be cleaned up by CloudFormation when the stack fails to deploy (unless you use --disable-rollback), but the logs should remain -- they must hold the answer to what goes wrong.
Ok, I'll try and get you that data today after all my morning meetings :) Yeah, I used the --disable-rollback function since we're testing in our nonprod environment just for that reason. I've just not been able to find anything in cloudwatch related to why it's not initiating the creation of those 3 custom resources. Hope to have more info for you soon!
So, one this I see is this (just trying to click through to Cloudwatch from StaticSiteHandler): Log group does not exist. The specific log group: /aws/lambda/uopx-nonprod-myresource-StaticSiteHander-xxxxxxxxxx does not exist in this account or region. That's not what I should see correct?
Just for giggles, I ran a test using the default test data (no passing in anything really) and this is the error I see:
START RequestId: d25396a4-70d8-4d9c-8747-9b52139735bb Version: $LATEST
2022-03-16T16:10:57.830Z undefined ERROR Uncaught Exception {"errorType":"Runtime.HandlerNotFound","errorMessage":"index.handler is undefined or not exported","stack":["Runtime.HandlerNotFound: index.handler is undefined or not exported"," at Object.module.exports.load (/var/runtime/UserFunction.js:246:11)"," at Object. (/var/runtime/index.js:43:30)"," at Module._compile (internal/modules/cjs/loader.js:1085:14)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)"," at Module.load (internal/modules/cjs/loader.js:950:32)"," at Function.Module._load (internal/modules/cjs/loader.js:790:12)"," at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)"," at internal/main/run_main_module.js:17:47"]}
2022-03-16T16:10:57.835Z undefined ERROR (node:9) Warning: Accessing non-existent property 'handler' of module exports inside circular dependency
(Use node --trace-warnings ... to show where the warning was created)
2022-03-16T16:10:57.998Z undefined ERROR Uncaught Exception {"errorType":"Runtime.HandlerNotFound","errorMessage":"index.handler is undefined or not exported","stack":["Runtime.HandlerNotFound: index.handler is undefined or not exported"," at Object.module.exports.load (/var/runtime/UserFunction.js:246:11)"," at Object. (/var/runtime/index.js:43:30)"," at Module._compile (internal/modules/cjs/loader.js:1085:14)"," at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)"," at Module.load (internal/modules/cjs/loader.js:950:32)"," at Function.Module._load (internal/modules/cjs/loader.js:790:12)"," at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)"," at internal/main/run_main_module.js:17:47"]}
2022-03-16T16:10:58.002Z undefined ERROR (node:9) Warning: Accessing non-existent property 'handler' of module exports inside circular dependency
(Use node --trace-warnings ... to show where the warning was created)
END RequestId: d25396a4-70d8-4d9c-8747-9b52139735bb
REPORT RequestId: d25396a4-70d8-4d9c-8747-9b52139735bb Duration: 112.46 ms Billed Duration: 113 ms Memory Size: 2048 MB Max Memory Used: 12 MB
Unknown application error occurred
Runtime.HandlerNotFound
Well there's the culprit.
Not sure I understand the error completely. Was compiling the .ts files into .js files successful? (npm run build or npx tsc directly)
Good question. Let me rerun the build and see what happens. It's been a minute. Stand by!
So, no, it's not compiling looks like (using npm run build):
cloudfront-authorization-at-edge@2.0.18 build /Users/xxx/Desktop/Projects/cloudfront-authorization-at-edge-master 4
npm run tsc && npm run webpack
cloudfront-authorization-at-edge@2.0.18 tsc /Users/xxx/Desktop/Projects/cloudfront-authorization-at-edge-master 4
npx tsc -b
npx: installed 1 in 1.389s
This is not the tsc command you are looking for
To get access to the TypeScript compiler, tsc, from the command line either:
- Use npm install typescript to first add TypeScript to your project before using npx
- Use yarn to avoid accidentally running code from un-installed packages
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! cloudfront-authorization-at-edge@2.0.18 tsc:npx tsc -b
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the cloudfront-authorization-at-edge@2.0.18 tsc script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/xxx/.npm/_logs/2022-03-16T16_27_18_319Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! cloudfront-authorization-at-edge@2.0.18 build: npm run tsc && npm run webpack
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the cloudfront-authorization-at-edge@2.0.18 build script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm WARN Local package.json exists, but node_modules missing, did you mean to install?
npm ERR! A complete log of this run can be found in:
npm ERR! /Users/xxx/.npm/_logs/2022-03-16T16_27_18_349Z-debug.log
Looks like you need to still run npm install in the root of the repo first, that should install typescript.
Ok, I did npm install and then npm run build, and that all worked, so cheers on that! Should I do another sam deploy now and see if we get different results? Sorry for all the stupid questions, I'm just still very new with all this.
Should I do another sam deploy now
Indeed. Basically follow these instructions closely:
https://github.com/aws-samples/cloudfront-authorization-at-edge#deployment
Yeah, I thought I did, but apparently not, so I feel a little foolish possibly wasting your time. I appreciate your patience! I'm going to clean up my current deploy (since it'll want to delete those 3 failed resources anyway) but that requires another team since they handicap our ability to delete resources (frustrating). I'll get back to you with updated results later today.
Sure thing, and no worries -- debugging Custom Resources can be a pain unfortunately.
Hi Omar!
Well, it MOSTLY worked this time!! All the custom resources created, the only thing that failed was the actual S3 bucket creation with this error: CREATE_FAILED AWS::S3::Bucket S3Bucket Value of property Tags must be of type List
That's something we should be able to work through I think. I just have to figure out where it's failing on that. We added some Tags: properties within the S3 bucket resource because it wasn't creating them and getting purged by our sentinel policies. Thanks so much. I'll let you know when we're 100% successful, but we're almost there :)
We finally got a full deploy done. Looks like everything is good now (we fixed the cloudfront log ACL issue on our end and added in Tags in the S3 bucket resource). Thank again for your help!!
Great! Our perseverance payed off in the end