s3 v1beta1 NoSuchTagSet when providing a tagSet

[holgerjh]

What happened?

We are provisioning S3 buckets using crossplane. We make use of the tagSet property to add tags to the buckets. After upgrading the provider from v0.44.2 to v0.47.1, we cannot create new S3 buckets due to an error NoSuchTagSet.
Full output:

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-create-pending: "2024-04-12T13:28:26Z"
    crossplane.io/external-create-succeeded: "2024-04-12T13:28:26Z"
    crossplane.io/external-name: some-unused-s3-bucket-name
  creationTimestamp: "2024-04-12T13:28:26Z"
  finalizers:
  - finalizer.managedresource.crossplane.io
  generation: 2
  name: some-unused-s3-bucket-name
  resourceVersion: "1980576943"
  uid: c475e509-dcdd-4d16-b86b-b5444db1ef41
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: eu-central-1
    paymentConfiguration:
      payer: BucketOwner
    publicAccessBlockConfiguration:
      blockPublicAcls: true
      blockPublicPolicy: true
      ignorePublicAcls: true
      restrictPublicBuckets: true
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    tagging:
      tagSet:
      - key: some-example-key
        value: some-example-value
  managementPolicies:
  - '*'
  providerConfigRef:
    name: PROVIDERCONFIG
status:
  atProvider:
    arn: arn:aws:s3:::some-unused-s3-bucket-name
  conditions:
  - lastTransitionTime: "2024-04-12T13:28:26Z"
    reason: Creating
    status: "False"
    type: Ready
  - lastTransitionTime: "2024-04-12T13:28:27Z"
    message: 'update failed: cannot create or update: api error NoSuchTagSet: The
      TagSet does not exist'
    reason: ReconcileError
    status: "False"
    type: Synced

How can we reproduce it?

Install version v0.47.1 and apply the following manifest to reproduce the error (replace PROVIDERCONFIG):

apiVersion: s3.aws.crossplane.io/v1beta1
kind: Bucket
metadata:
  annotations:
    crossplane.io/external-name: some-unused-s3-bucket-name
  name: some-unused-s3-bucket-name 
spec:
  deletionPolicy: Delete
  forProvider:
    acl: private
    locationConstraint: eu-central-1
    serverSideEncryptionConfiguration:
      rules:
      - applyServerSideEncryptionByDefault:
          sseAlgorithm: AES256
    tagging:
      tagSet:
      - key: some-example-key
        value: some-example-value
  providerConfigRef:
    name: PROVIDERCONFIG

What environment did it happen in?

Crossplane version: v1.14.7-up.1
Provider Version: v0.47.1

Misc Observations

When looking for recent changes w.r.t. tagging I noted that commit eb1df4999e82ee276765508d820bd77e3858b8e3 introduced a caching mechanism for tags which does a lookup on AWS side. It is used amongst others in the Observe function and in the CreateOrUpdate function.
Maybe this calls the API before the bucket gets its first tags (In Observe e.g. it it replaced a call to client.PutBucketTagging).

[holgerjh]

Fixed in v0.47.2