[aws-kendra] Generated CfnIndex construct's DocumentMetadataConfigurations property is of the wrong type

Question

[aws-kendra] Generated CfnIndex construct's DocumentMetadataConfigurations property is of the wrong type

Closed this issue 4 years ago · 9 comments

Hi,

I've been using CDK to define a kendra index, and am trying to define custom attributes in the document metadata.

new kendra.CfnIndex(this, 'MyIndex', {
    edition: 'DEVELOPER_EDITION',
    name: `${props.stage}-MyIndex`,
    roleArn: kendraIndexRole.roleArn,
    documentMetadataConfigurations: {
        documentMetadataConfigurationList: [
            {
                name: 'myAttribute',
                type: 'STRING_VALUE',
                search: {
                    displayable: true,
                    searchable: false,
                    sortable: false,
                    facetable: false,
                },
            },
        ],
    },
});

This all builds fine, but when I synth and deploy, I get an error:

5:35:40 PM | UPDATE_FAILED        | AWS::Kendra::Index | KendraWebCrawlerIndex
Model validation failed (#/DocumentMetadataConfigurations: expected type: JSONArray, found: JSONObject)

However this doesn't match the cloudformation documentation, which states that DocumentMetadataConfigurations is an object with a single property DocumentMetadataConfigurationsList.

See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kendra-index.html#cfn-kendra-index-documentmetadataconfigurations and https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfigurationlist.html

The CDK CfnIndex construct types in kendra.generated.d.ts:

    /**
     * `AWS::Kendra::Index.DocumentMetadataConfigurations`.
     *
     * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kendra-index.html#cfn-kendra-index-documentmetadataconfigurations
     * @external
     */
    readonly documentMetadataConfigurations?: CfnIndex.DocumentMetadataConfigurationListProperty | cdk.IResolvable;

/**
 * A CloudFormation `AWS::Kendra::Index`.
 *
 * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kendra-index.html
 * @external
 * @cloudformationResource AWS::Kendra::Index
 */
export declare namespace CfnIndex {
    /**
     * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfigurationlist.html
     * @external
     */
    interface DocumentMetadataConfigurationListProperty {
        /**
         * `CfnIndex.DocumentMetadataConfigurationListProperty.DocumentMetadataConfigurationList`.
         *
         * @see http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfigurationlist.html#cfn-kendra-index-documentmetadataconfigurationlist-documentmetadataconfigurationlist
         * @external
         */
        readonly documentMetadataConfigurationList?: Array<CfnIndex.DocumentMetadataConfigurationProperty | cdk.IResolvable> | cdk.IResolvable;
    }
}

I first thought that this might be a problem with the mapping of CFN to kendra api calls, so raised this issue: aws-cloudformation/aws-cloudformation-resource-providers-kendra#77

However it might be a problem somewhere in CDK or CloudFormation!

I'm able to work around this issue with:

new kendra.CfnIndex(this, 'MyIndex', {
    // Change this to ENTERPRISE_EDITION for use in production
    edition: 'DEVELOPER_EDITION',
    name: `${props.stage}-MyIndex`,
    roleArn: kendraIndexRole.roleArn,
});

(this.node.findChild('MyIndex') as cdk.CfnResource).addPropertyOverride('DocumentMetadataConfigurations', [
    {
        name: 'myAttribute',
        type: 'STRING_VALUE',
        search: {
            displayable: true,
            searchable: false,
            sortable: false,
            facetable: false,
        },
    },
]);

Thanks,
Jack

Reproduction Steps

import * as kendra from '@aws-cdk/aws-kendra';

new kendra.CfnIndex(this, 'MyIndex', {
    edition: 'DEVELOPER_EDITION',
    name: `${props.stage}-MyIndex`,
    roleArn: kendraIndexRole.roleArn,
    documentMetadataConfigurations: {
        documentMetadataConfigurationList: [
            {
                name: 'myAttribute',
                type: 'STRING_VALUE',
                search: {
                    displayable: true,
                    searchable: false,
                    sortable: false,
                    facetable: false,
                },
            },
        ],
    },
});

What did you expect to happen?

Expected a valid cloudformation template to be generated, like:

...
    "KendraWebCrawlerIndex": {
      "Type": "AWS::Kendra::Index",
      "Properties": {
        ...
        "DocumentMetadataConfigurations": [
          {
            "name": "myAttribute",
            "type": "STRING_VALUE",
            "search": {
              "displayable": true,
              "searchable": false,
              "sortable": false,
              "facetable": false
            }
          }
        ]
      },
      ...
    },
...

What actually happened?

A template like the following was generated:

...
    "KendraWebCrawlerIndex": {
      "Type": "AWS::Kendra::Index",
      "Properties": {
        ...
        "DocumentMetadataConfigurations": {
          "DocumentMetadataConfigurationList": [
            {
              "Name": "myAttribute",
              "Search": {
                "Displayable": true,
                "Facetable": false,
                "Searchable": false,
                "Sortable": false
              },
              "Type": "STRING_VALUE"
            }
          ]
        }
      },
      ...
    },
...

Which then trigged an error at deploy time:

5:35:40 PM | UPDATE_FAILED        | AWS::Kendra::Index | KendraWebCrawlerIndex
Model validation failed (#/DocumentMetadataConfigurations: expected type: JSONArray, found: JSONObject)

Environment

CDK CLI Version : 1.72.0 (build c2f38e8)
Framework Version: 1.71.0
Node.js Version: v12.18.2
OS : macOS Mojave (10.14.6)
Language (Version): TypeScript (3.9.7)

Other

Cloudformation Documentation - states that the DocumentMetadataConfigurations property is an object with a single property which is a list.
CDK Documentation - matches this
Generated Typescript types - also match this
aws-cloudformation-resource-providers-kendra defines the property as simply a list. See the definition of the top level property on the index and the definition of its type.

This is 🐛 Bug Report

Answer 1 · 2020-11-11T00:24:22.000Z

Thanks cogwirrel@ for raising this! Just adding a bit more info here as a developer on the Kendra team. As cogwirrel@ points out, https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-kendra/blob/master/aws-kendra-index/aws-kendra-index.json is where the CloudFormation schema is defined. In that file we define DocumentMetadataConfigurations here https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-kendra/blob/master/aws-kendra-index/aws-kendra-index.json#L259.

It's odd that cogwirrel@ needs to define a property called documentMetadataConfigurationList in the CDK as he did in

new kendra.CfnIndex(this, 'MyIndex', {
    edition: 'DEVELOPER_EDITION',
    name: `${props.stage}-MyIndex`,
    roleArn: kendraIndexRole.roleArn,
    documentMetadataConfigurations: {
        documentMetadataConfigurationList: [
            {
                name: 'myAttribute',
                type: 'STRING_VALUE',
                search: {
                    displayable: true,
                    searchable: false,
                    sortable: false,
                    facetable: false,
                },
            },
        ],
    },
});

Is this a bug? JSON/YAML CloudFormation templates certainly do not need to define this property - as cogwirrel@ points out this actually causes an error. For example, the following CloudFormation is valid:

AWSTemplateFormatVersion: 2010-09-09
Resources:
  IntegTestIndexLogicalID:
    Type: AWS::Kendra::Index
    Properties:
      Name: "IndexWithDocumentMetadata"
      Description: "Test document metadata"
      Edition: "DEVELOPER_EDITION"
      RoleArn:
        Fn::ImportValue: IntegrationTestSimpleIndexIAMRoleExport
      DocumentMetadataConfigurations:
        - Name: 'NameOne'
          Type: 'STRING_VALUE'
        - Name: '_source_uri'
          Type: 'STRING_VALUE'

Outputs:
  IndexID:
    Value:
      Ref: 'IntegTestIndexLogicalID'

I'm not sure where/why documentMetadataConfigurationList becomes necessary in CDK.

Let me know how I can help out from the Kendra side.

Thanks,
Andrew

Answer 2 · 2020-11-12T07:15:18.000Z

@angranamzn - the CDK generated code for Kendra's level 1 constructs (prefixed with Cfn*) is driven by the CloudFormation resource specification

which currently defines:

"DocumentMetadataConfigurations": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-kendra-index.html#cfn-kendra-index-documentmetadataconfigurations",
          "Required": false,
          "Type": "DocumentMetadataConfigurationList",
          "UpdateType": "Mutable"
        },


"AWS::Kendra::Index.DocumentMetadataConfigurationList": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfigurationlist.html",
      "Properties": {
        "DocumentMetadataConfigurationList": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfigurationlist.html#cfn-kendra-index-documentmetadataconfigurationlist-documentmetadataconfigurationlist",
          "ItemType": "DocumentMetadataConfiguration",
          "Required": false,
          "Type": "List",
          "UpdateType": "Mutable"
        }
      }
    },


"AWS::Kendra::Index.DocumentMetadataConfiguration": {
      "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfiguration.html",
      "Properties": {
        "Name": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfiguration.html#cfn-kendra-index-documentmetadataconfiguration-name",
          "PrimitiveType": "String",
          "Required": true,
          "UpdateType": "Mutable"
        },
        "Relevance": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfiguration.html#cfn-kendra-index-documentmetadataconfiguration-relevance",
          "Required": false,
          "Type": "Relevance",
          "UpdateType": "Mutable"
        },
        "Search": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfiguration.html#cfn-kendra-index-documentmetadataconfiguration-search",
          "Required": false,
          "Type": "Search",
          "UpdateType": "Mutable"
        },
        "Type": {
          "Documentation": "http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-kendra-index-documentmetadataconfiguration.html#cfn-kendra-index-documentmetadataconfiguration-type",
          "PrimitiveType": "String",
          "Required": true,
          "UpdateType": "Mutable"
        }
      }
    },

I believe the spec is being translated the way we would expect in the CDK.
I don't believe this is defined in the way it's intended however. It looks like the DocumentMetadataConfigurations type should be a List with an ItemType of DocumentMetadataConfiguration.

Although we can patch this from the CDK side, I think it should be rectified in the resource specification so that all consumers of the specification can benefit from it.

Answer 3 · 2020-11-12T20:32:42.000Z

Thanks shivlaks@. I agree this looks to be an issue with how the resource specification is generated on the CloudFormation side. I'll sync up with the CloudFormation folks.

Answer 4 · 2020-11-12T20:35:24.000Z

copy @PatMyron

Answer 5 · 2021-01-25T19:26:13.000Z

@angranamzn I believe this should be fixed in recent CDK versions, which included changes to the CloudFormation resource specification for the AWS Kendra resources (more specifically, in version 23.0.0, which was released in CDK version 1.85.0).

Let me know if this fixes your issue!

Thanks,
Adam

Answer 6 · 2021-02-02T06:27:49.000Z

This issue has not received a response in a while. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

Answer 7 · 2021-02-03T03:35:14.000Z

Tried with CDK version 1.86.0 and it works :)

Now defined without the documentMetadataConfigurationList key :)

new CfnIndex(this, 'MyIndex', {
  edition: 'DEVELOPER_EDITION',
  name: 'MyIndex',
  roleArn: kendraIndexRole.roleArn,
  documentMetadataConfigurations: [
    {
      name: 'myAttribute',
      type: 'STRING_VALUE',
      search: {
        displayable: true,
        searchable: false,
        sortable: false,
        facetable: false,
      },
    },
  ],
});

Thanks all!

Answer 8 · 2021-02-03T18:41:09.000Z

That's great to hear @cogwirrel !

Resolving, let us know if you need anything else from our side for this issue!

Answer 9 · 2021-02-03T18:41:36.000Z

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.