MSK Configuration Example

[vishnu-wednesday]

AWS::MSK::Configuration seems relatively new. Can anyone help with a cloudformation template/syntax for the YAML of serverproperties. I seem to be always running into 400 errors while creating. If not something answerable please point me to the proper place to ask this. Syntax tried and failed.

Resources:
  ServerlessMSKConfiguration:
    Type: AWS::MSK::Configuration
    Properties:
      ServerProperties: !Base64 |
        auto.create.topics.enable=false
        default.replication.factor=2
        min.insync.replicas=2
        num.io.threads=8
        num.network.threads=8
        num.replica.fetchers=1
        num.partitions=10
        replica.lag.time.max.ms=10000
        socket.receive.buffer.bytes=102400
        socket.request.max.bytes=104857600
        socket.send.buffer.bytes=102400
        unclean.leader.election.enable=true
        zookeeper.session.timeout.ms=18000

[pmkuny]

I'm trying to play around with this to see if I can get you an answer. I can confirm that this is not creating properly with YAML and CloudFormation
It looks like a manual creation using the aws cli works just fine:

Using your code above, I got the same error trying to create this in CloudFormation:

Resource handler returned message: "[ClientRequestToken: b9338ec9-4c5c-a649-1a50-75dec0033b11] Invalid request body (Service: Kafka, Status Code: 400, Request ID: e2ee50a9-cecc-4f7a-a479-ec5ede54895c, Extended Request ID: null)" (RequestToken: b9338ec9-4c5c-a649-1a50-75dec0033b11, HandlerErrorCode: InvalidRequest)

I then tried to take the properties and pass them to the AWS CLI:

1. Create properties file with server properties and base64 encode it

cat server.properties | base64 > server.properties.base64

2. Create MSK Configuration

aws kafka create-configuration --name test --server-properties "YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZT1mYWxzZQpkZWZhdWx0LnJlcGxpY2F0aW9uLmZhY3Rv
cj0yCm1pbi5pbnN5bmMucmVwbGljYXM9MgpudW0uaW8udGhyZWFkcz04Cm51bS5uZXR3b3JrLnRo
cmVhZHM9OApudW0ucmVwbGljYS5mZXRjaGVycz0xCm51bS5wYXJ0aXRpb25zPTEwCnJlcGxpY2Eu
bGFnLnRpbWUubWF4Lm1zPTEwMDAwCnNvY2tldC5yZWNlaXZlLmJ1ZmZlci5ieXRlcz0xMDI0MDAK
c29ja2V0LnJlcXVlc3QubWF4LmJ5dGVzPTEwNDg1NzYwMApzb2NrZXQuc2VuZC5idWZmZXIuYnl0
ZXM9MTAyNDAwCnVuY2xlYW4ubGVhZGVyLmVsZWN0aW9uLmVuYWJsZT10cnVlCnpvb2tlZXBlci5z
ZXNzaW9uLnRpbWVvdXQubXM9MTgwMDAK"

Results in:

{
    "Arn": "arn:aws:kafka:us-west-2:XXXXXXXXX:configuration/test/d26b5d8a-3c31-4a45-bb97-16b1ed71efca-11",
    "CreationTime": "2022-03-28T14:09:36.553000+00:00",
    "LatestRevision": {
        "CreationTime": "2022-03-28T14:09:36.553000+00:00",
        "Revision": 1
    },
    "Name": "test",
    "State": "ACTIVE"
}

I also confirmed in CloudTrail that the Base64 encoding being sent is slightly off from the one in my file:
File:

YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZT1mYWxzZSBkZWZhdWx0LnJlcGxpY2F0aW9uLmZhY3Rvcj0yIG1pbi5pbnN5bmMucmVwbGljYXM9MiBudW0uaW8udGhyZWFkcz04IG51bS5uZXR3b3JrLnRocmVhZHM9OCBudW0ucmVwbGljYS5mZXRjaGVycz0xIG51bS5wYXJ0aXRpb25zPTEwIHJlcGxpY2EubGFnLnRpbWUubWF4Lm1zPTEwMDAwIHNvY2tldC5yZWNlaXZlLmJ1ZmZlci5ieXRlcz0xMDI0MDAgc29ja2V0LnJlcXVlc3QubWF4LmJ5dGVzPTEwNDg1NzYwMCBzb2NrZXQuc2VuZC5idWZmZXIuYnl0ZXM9MTAyNDAwIHVuY2xlYW4ubGVhZGVyLmVsZWN0aW9uLmVuYWJsZT10cnVlIHpvb2tlZXBlci5zZXNzaW9uLnRpbWVvdXQubXM9MTgwMDAK```

CloudTrail (coming from the CloudFormation Request):

    "requestParameters": {
        "serverProperties": "YXV0by5jcmVhdGUudG9waWNzLmVuYWJsZT1mYWxzZSBkZWZhdWx0LnJlcGxpY2F0aW9uLmZhY3Rvcj0yIG1pbi5pbnN5bmMucmVwbGljYXM9MiBudW0uaW8udGhyZWFkcz04IG51bS5uZXR3b3JrLnRocmVhZHM9OCBudW0ucmVwbGljYS5mZXRjaGVycz0xIG51bS5wYXJ0aXRpb25zPTEwIHJlcGxpY2EubGFnLnRpbWUubWF4Lm1zPTEwMDAwIHNvY2tldC5yZWNlaXZlLmJ1ZmZlci5ieXRlcz0xMDI0MDAgc29ja2V0LnJlcXVlc3QubWF4LmJ5dGVzPTEwNDg1NzYwMCBzb2NrZXQuc2VuZC5idWZmZXIuYnl0ZXM9MTAyNDAwIHVuY2xlYW4ubGVhZGVyLmVsZWN0aW9uLmVuYWJsZT10cnVlIHpvb2tlZXBlci5zZXNzaW9uLnRpbWVvdXQubXM9MTgwMDA="
    },

The last letter is an = instead of a K

My only guess is that the Base64 function is somehow working differently than the Linux command I'm using, or some form of transformation is happening.

Because this may be a bug and not a documentation issue, can you also open up this issue in
https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues ?

[vishnu-wednesday]

Thanks for the investigation into @pmkuny. I actually got it to work. I use the Serverlerless Framework for deployment, I didn't use the base64 function. I just simply referenced the file using extrapolated ${file(...)} syntax. Posting a link to a SO answer I made. I will make my serverless repo public soon and update with a link here for any future use cases.

https://stackoverflow.com/questions/71238399/cloudformation-cannot-create-resource-of-type-awsmskconfiguration/71499172#71499172

I think a little more clarity on the docs would be wonderful though. Maybe a small example too. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-configuration.html.

[joshbean]

Closing this issue or pull request in advance of archiving this repo. For more information about the decision to archive this repo (and others in the 'awsdocs' org), see the announcement on the AWS News Blog.