aws-powertools/powertools-lambda-java

AbstractCustomResourceHandler - Unable to send response

drullar opened this issue · 2 comments

drullar commented

What were you trying to accomplish?

I have a CloudFormation template, which invokes a lambda. The particular lambda doesn't do anything on Stack create, other than return Response.success(physicalResourceID), but rather should perform an action on Stack deletion, i.e when the delete method of AbstractCustomResourceHandler is invoked.

Expected Behavior

The expected behavior is sometimes noticed, although rarely and that is on cloudformation stack creation the lambda should return the success status without an issue, and the stack should continue provisioning.

Current Behavior

Currently on stack creation when the lambda is invoked the return statement throws a SocketTimeoutException and doesn't return the response to CloudFormation, thus with no response received the CloudFormation provisioning can't continue and times out after the given timeout period.

Possible Solution

Steps to Reproduce (for bugs)

  1. Create a Java class, which inherits AbstractCustomResourceHandler and use the following peace of code as the implementation for the create method:
override Response create(CloudFormationCustomResourceEvent event, Context context) {
        return Response.success(UUID.randomUUID().toString())
    }
  1. Create a CloudFormation stack with a couple of resource, one of them being invocation of the lambda. The part of the cloudformation template when you invoke the lambda should look something like this:
    "InvokeLambdaCustomResource": {
      "DependsOn": "SomeOtherResourceBuiltPriorToThis",
      "Type": "AWS::CloudFormation::CustomResource",
      "Properties": {
        "ServiceToken": "arn:aws:lambda:reamaing-arn-of-your-lambda"
      }
    }
  1. Deploy the cloudformation template.

Environment

  • Powertools for AWS Lambda (Java) version used: 1.18.0
  • Packaging format (Layers, Maven/Gradle): Maven
  • AWS Lambda function runtime: Java 17 | x86_64
  • Debugging logs
[main] ERROR software.amazon.lambda.powertools.cloudformation.AbstractCustomResourceHandler - Unable to send response [Status = SUCCESS,NoEcho = false,PhysicalResourceId = dc-c76342f1-9809-4563-8969-374b1b8238ea ,JSON = null] to https://cloudformation-custom-resource-response-useast1.s3.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A223498081460%3Astack/SC-223498081460-pp-oho3ejxety3ze/b7bd5070-cf1e-11ee-8b45-0e54464a22e9%7CInvokeDataCleanupLambda%7C7290c4ac-71c5-4d79-bd89-178175b83757?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20240219T120425Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIA6L7Q4OWTX2SQBBEW%2F20240219%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=cd15b4a720afb59125f17d0f2f2fbd8c9c47f37a57c14dcaa55bd96a236a9526.
java.net.SocketTimeoutException: Connect timed out
at [sun.nio.ch](http://sun.nio.ch/).NioSocketImpl.timedFinishConnect(Unknown Source) ~[?:?]
at [sun.nio.ch](http://sun.nio.ch/).NioSocketImpl.connect(Unknown Source) ~[?:?]
at [java.net](http://java.net/).SocksSocketImpl.connect(Unknown Source) ~[?:?]
at [java.net](http://java.net/).Socket.connect(Unknown Source) ~[?:?]
at sun.security.ssl.SSLSocketImpl.connect(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).NetworkClient.doConnect(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.http.HttpClient.openServer](http://www.http.httpclient.openserver/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.http.HttpClient.openServer](http://www.http.httpclient.openserver/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.https.HttpsClient](http://www.protocol.https.httpsclient/).<init>(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.https.HttpsClient.New](http://www.protocol.https.httpsclient.new/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient](http://www.protocol.https.abstractdelegatehttpsurlconnection.getnewhttpclient/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.http.HttpURLConnection.plainConnect0](http://www.protocol.http.httpurlconnection.plainconnect0/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.http.HttpURLConnection.plainConnect](http://www.protocol.http.httpurlconnection.plainconnect/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.https.AbstractDelegateHttpsURLConnection.connect](http://www.protocol.https.abstractdelegatehttpsurlconnection.connect/)(Unknown Source) ~[?:?]
at [sun.net](http://sun.net/).[www.protocol.https.HttpsURLConnectionImpl.connect](http://www.protocol.https.httpsurlconnectionimpl.connect/)(Unknown Source) ~[?:?]
at software.amazon.awssdk.http.urlconnection.UrlConnectionHttpClient$RequestCallable.call(UrlConnectionHttpClient.java:296) ~[task/:?]
at software.amazon.lambda.powertools.cloudformation.CloudFormationResponse.send(CloudFormationResponse.java:114) ~[task/:?]
at software.amazon.lambda.powertools.cloudformation.AbstractCustomResourceHandler.handleRequest(AbstractCustomResourceHandler.java:74) [task/:?]
at software.amazon.lambda.powertools.cloudformation.AbstractCustomResourceHandler.handleRequest(AbstractCustomResourceHandler.java:1) [task/:?]
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:202) [aws-lambda-java-runtime-interface-client-2.4.1-linux-x86_64.jar:2.4.1]
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$2.call(EventHandlerLoader.java:905) [aws-lambda-java-runtime-interface-client-2.4.1-linux-x86_64.jar:2.4.1]
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:245) [aws-lambda-java-runtime-interface-client-2.4.1-linux-x86_64.jar:2.4.1]
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntime(AWSLambda.java:197) [aws-lambda-java-runtime-interface-client-2.4.1-linux-x86_64.jar:2.4.1]
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:187) [aws-lambda-java-runtime-interface-client-2.4.1-linux-x86_64.jar:2.4.1]
jeromevdl commented

@mriccia any idea ?

drullar commented

Hey, Please ignore this issue. The issue was with miss configured VPC settings of my lambda. I thought it was a bug because it didn't happen all the time. The lambda had two subnets assigned to it and one of them had issues connecting to Internet that was the main issue.