PodidentityAssociation ready state is false
Opened this issue · 5 comments
ashishvaishno commented
Hello,
I have upgrade my crossplane deployment to xpkg.upbound.io/upbound/provider-aws-eks:v1.0.0. I have started to use Podidentityassociation.
Crossplane was successfully able to create the association and synced, but the ready state is false 😕 Below is the error of the managed resource. I dont have any previous managed resource / already existing association with an association id a-stubassocid123456
async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-stubassocid123456"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: f9cb61ba-2e55-4c0e-a8f7-962bfd8302bc, ResourceInUseException: Association already exists: a-dt1bedfjc72dlwp5n
candonov commented
Hi @ashishvaishno, can you provide the Crossplane manifest(s) you used so I reproduce this?
ashishvaishno commented
@candonov Sure. But the issue persists only on few PIA. I tried to clean them up and re-deploy them as well. But its still the same 😢
Composition
# XRD to create the s3 bucket and the associated irsa
---
apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
name: xs3withpodidentitys.crossplane.xyz.com
spec:
claimNames:
kind: S3WithPodIdentity
plural: s3withpodidentitys
group: crossplane.xyz.com
names:
kind: XS3WithPodIdentity
plural: xs3withpodidentitys
versions:
- name: v1alpha1
served: true
referenceable: true
schema:
openAPIV3Schema:
properties:
spec:
description: S3WithPodIdentitySpec defines the desired state of S3WithPodIdentity
properties:
resourceConfig:
description: ResourceConfig defines general properties of this AWS resource.
properties:
deletionPolicy:
description: Defaults to Delete
enum:
- Orphan
type: string
default: "Orphan"
bucketName:
description: Name of the bucket
type: string
serviceAccountName:
description: Name of the service account which will also be used to generate the k8s sa
type: string
serviceNamespace:
description: Namespace of the service where its running
type: string
bucketVersioningStatus:
description: Bucket versioning status
type: string
default: "Disabled"
enum:
- Disabled
- Suspended
- Enabled
bucketNonCurrentVersionExpirationDays:
description: Cleanup Old Objects older than x days
type: number
default: 90
bucketCleanupOldObjectStatus:
description: Cleanup Old Objects
type: string
default: "Disabled"
enum:
- Disabled
- Enabled
bucketCleanupOldObjectExpirationDays:
description: Cleanup Old Objects older than x days
type: number
default: 180
awsBackup:
description: Setup awsbackup to exclude for s3 buckets
type: string
default: "exclude"
tags:
items:
properties:
key:
type: string
value:
type: string
required:
- key
- value
type: object
type: array
required:
- bucketName
- serviceAccountName
- serviceNamespace
type: object
required:
- resourceConfig
type: object
status:
description: S3WithPodIdentityStatus defines the observed state of S3WithPodIdentity
properties:
bucketArn:
type: string
bucketVersioningStatus:
type: string
bucketNonCurrentVersionExpirationDays:
type: number
bucketCleanupOldObjectStatus:
type: string
bucketCleanupOldObjectExpirationDays:
type: number
roleArn:
type: string
awsAccountID:
type: string
policyArn:
type: string
type: object
type: object
XRD :
apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
name: dev.xs3withpodidentity.crossplane.xyz.com
labels:
environment: dev
spec:
compositeTypeRef:
apiVersion: crossplane.xyz.com/v1alpha1
kind: XS3WithPodIdentity
patchSets:
- name: deletion-policy-field
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.deletionPolicy
toFieldPath: spec.deletionPolicy
- name: common-fields
patches:
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
strategy: string
string:
fmt: "aws-provider-config-%s"
toFieldPath: spec.providerConfigRef.name
- name: tags
patches:
- type: FromCompositeFieldPath
fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
toFieldPath: spec.forProvider.tags.Environment
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceAccountName
toFieldPath: spec.forProvider.tags.serviceAccountName
- name: region
patches:
- type: FromCompositeFieldPath
fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
toFieldPath: spec.forProvider.region
transforms:
- type: map
map:
dev: eu-central-1
stage: eu-west-1
prod: eu-west-1
- name: s3-extra-tag
patches:
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.awsBackup
toFieldPath: spec.forProvider.tags.AWSBackup
resources:
- name: s3-bucket
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: Bucket
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: deletion-policy-field
- type: PatchSet
patchSetName: tags
- type: PatchSet
patchSetName: region
- type: PatchSet
patchSetName: s3-extra-tag
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketName
toFieldPath: metadata.annotations[crossplane.io/external-name]
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketName
toFieldPath: spec.forProvider.tags.Name
- type: ToCompositeFieldPath
fromFieldPath: status.atProvider.arn
toFieldPath: status.bucketArn
- name: s3-bucketpublicaccessblock
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketPublicAccessBlock
spec:
forProvider:
blockPublicAcls: true
blockPublicPolicy: true
ignorePublicAcls: true
restrictPublicBuckets: true
bucketSelector:
matchControllerRef: true
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: deletion-policy-field
- type: PatchSet
patchSetName: region
- name: s3-bucketserversideencryptionconfiguration
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketServerSideEncryptionConfiguration
spec:
forProvider:
bucketSelector:
matchControllerRef: true
rule:
- applyServerSideEncryptionByDefault:
- sseAlgorithm: AES256
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: deletion-policy-field
- type: PatchSet
patchSetName: region
- name: s3-bucketversioning
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketVersioning
spec:
forProvider:
bucketSelector:
matchControllerRef: true
versioningConfiguration:
- status: ""
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: deletion-policy-field
- type: PatchSet
patchSetName: region
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketVersioningStatus
toFieldPath: spec.forProvider.versioningConfiguration[0].status
- type: ToCompositeFieldPath
fromFieldPath: status.atProvider.versioningConfiguration[0].status
toFieldPath: status.bucketVersioningStatus
- name: s3-bucketlifecycleconfiguration
base:
apiVersion: s3.aws.upbound.io/v1beta1
kind: BucketLifecycleConfiguration
spec:
forProvider:
bucketSelector:
matchControllerRef: true
rule:
- id: "Cleanup of Non Current Versions"
noncurrentVersionExpiration:
- noncurrentDays:
status: ""
- id: "Cleanup of Old Objects"
expiration:
- days:
status: ""
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: deletion-policy-field
- type: PatchSet
patchSetName: region
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketVersioningStatus
toFieldPath: spec.forProvider.rule[0].status
transforms:
- type: map
map:
Disabled: Disabled
Suspended: Enabled
Enabled: Enabled
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketNonCurrentVersionExpirationDays
toFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectStatus
toFieldPath: spec.forProvider.rule[1].status
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.bucketCleanupOldObjectExpirationDays
toFieldPath: spec.forProvider.rule[1].expiration[0].days
- type: ToCompositeFieldPath
fromFieldPath: spec.forProvider.rule[0].status
toFieldPath: status.bucketVersioningStatus
- type: ToCompositeFieldPath
fromFieldPath: spec.forProvider.rule[0].noncurrentVersionExpiration[0].noncurrentDays
toFieldPath: status.bucketNonCurrentVersionExpirationDays
- type: ToCompositeFieldPath
fromFieldPath: spec.forProvider.rule[1].status
toFieldPath: status.bucketCleanupOldObjectStatus
- type: ToCompositeFieldPath
fromFieldPath: spec.forProvider.rule[1].expiration[0].days
toFieldPath: status.bucketCleanupOldObjectExpirationDays
- name: iamrole
base:
apiVersion: iam.aws.upbound.io/v1beta1
kind: Role
spec:
deletionPolicy: Delete
forProvider:
assumeRolePolicy: |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "pods.eks.amazonaws.com"
},
"Action": [
"sts:TagSession",
"sts:AssumeRole"
]
}
]
}
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: tags
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
- fromFieldPath: spec.resourceConfig.serviceAccountName
strategy: string
string:
fmt: "k8s-%s-%s"
toFieldPath: metadata.annotations[crossplane.io/external-name]
- type: ToCompositeFieldPath
fromFieldPath: status.atProvider.arn
toFieldPath: status.roleArn
- name: iam-policy
base:
apiVersion: iam.aws.upbound.io/v1beta1
kind: Policy
spec:
deletionPolicy: Delete
forProvider:
description: ""
policy: ""
patches:
- type: PatchSet
patchSetName: common-fields
- type: PatchSet
patchSetName: tags
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
- fromFieldPath: spec.resourceConfig.serviceAccountName
strategy: string
string:
fmt: "k8s-%s-%s"
toFieldPath: metadata.annotations[crossplane.io/external-name]
- type: ToCompositeFieldPath
fromFieldPath: status.atProvider.arn
toFieldPath: status.policyArn
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: spec.resourceConfig.bucketName
strategy: string
string:
fmt: "IAM Policy for bucket %s created by crossplane"
toFieldPath: spec.forProvider.description
- type: CombineFromComposite
toFieldPath: spec.forProvider.policy
policy:
fromFieldPath: Required
combine:
variables:
- fromFieldPath: spec.resourceConfig.bucketName
- fromFieldPath: spec.resourceConfig.serviceAccountName
- fromFieldPath: spec.resourceConfig.bucketName
- fromFieldPath: spec.resourceConfig.serviceAccountName
strategy: string
string:
fmt: |
{
"Version": "2012-10-17",
"Statement": [
{
"Action": "s3:ListBucket",
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::%s"
],
"Condition": {
"StringLike": {
"aws:PrincipalTag/kubernetes-pod-name": "%s*"
}
}
},
{
"Action": [
"s3:ListObjects",
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::%s/*"
],
"Condition": {
"StringLike": {
"aws:PrincipalTag/kubernetes-pod-name": "%s*"
}
}
}
]
}
- name: policy-attachment-1
base:
apiVersion: iam.aws.upbound.io/v1beta1
kind: RolePolicyAttachment
spec:
deletionPolicy: Delete
forProvider:
policyArnSelector:
matchControllerRef: true
roleSelector:
matchControllerRef: true
patches:
- type: PatchSet
patchSetName: common-fields
- name: pod-identity-association
base:
apiVersion: eks.aws.upbound.io/v1beta1
kind: PodIdentityAssociation
spec:
deletionPolicy: Delete
forProvider:
clusterName: "go-main-dev"
region: "eu-central-1"
namespace: ""
serviceAccount: ""
roleArnSelector:
matchControllerRef: true
patches:
- type: PatchSet
patchSetName: common-fields
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceNamespace
toFieldPath: spec.forProvider.namespace
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceAccountName
toFieldPath: spec.forProvider.serviceAccount
- name: service-account
base:
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
spec:
deletionPolicy: Delete
forProvider:
manifest:
apiVersion: v1
kind: ServiceAccount
metadata:
name: ""
namespace: ""
imagePullSecrets:
- name: registrypullsecret
patches:
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
strategy: string
string:
fmt: "kubernetes-provider-config-%s"
toFieldPath: spec.providerConfigRef.name
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceNamespace
toFieldPath: spec.forProvider.manifest.metadata.namespace
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceAccountName
toFieldPath: spec.forProvider.manifest.metadata.name
- name: remote-configmap
base:
apiVersion: kubernetes.crossplane.io/v1alpha1
kind: Object
spec:
deletionPolicy: Delete
forProvider:
manifest:
apiVersion: v1
kind: ConfigMap
metadata:
name: ""
namespace: ""
data:
roleArn: ""
policyArn: ""
bucketArn: ""
serviceAccountName: ""
region: ""
bucketVersioningStatus: ""
patches:
- type: CombineFromComposite
combine:
variables:
- fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
strategy: string
string:
fmt: "kubernetes-provider-config-%s"
toFieldPath: spec.providerConfigRef.name
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceNamespace
toFieldPath: spec.forProvider.manifest.metadata.namespace
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceAccountName
toFieldPath: spec.forProvider.manifest.metadata.name
- type: FromCompositeFieldPath
fromFieldPath: status.roleArn
toFieldPath: spec.forProvider.manifest.data.roleArn
- type: FromCompositeFieldPath
fromFieldPath: status.policyArn
toFieldPath: spec.forProvider.manifest.data.policyArn
- type: FromCompositeFieldPath
fromFieldPath: status.bucketArn
toFieldPath: spec.forProvider.manifest.data.bucketArn
- type: FromCompositeFieldPath
fromFieldPath: spec.resourceConfig.serviceAccountName
toFieldPath: spec.forProvider.manifest.data.serviceAccountName
- type: FromCompositeFieldPath
fromFieldPath: status.bucketVersioningStatus
toFieldPath: spec.forProvider.manifest.data.bucketVersioningStatus
- type: FromCompositeFieldPath
fromFieldPath: metadata.labels[crossplane.io/claim-namespace]
toFieldPath: spec.forProvider.manifest.data.region
transforms:
- type: map
map:
dev: eu-central-1
stage: eu-west-1
prod: eu-west-1
Claim :
---
apiVersion: crossplane.xyz.com/v1alpha1
kind: S3WithPodIdentity
metadata:
name: test-s3with-podidentity
namespace: dev
spec:
compositionSelector:
matchLabels:
environment: dev
resourceConfig:
bucketName: test-s3with-podidentity
serviceAccountName:service-account-name
serviceNamespace: default
Error :
Message: async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-dzwb0z8xhym98oebr"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 428d429c-75f8-4d31-bf88-cdfa1f756370, ResourceInUseException: Association already exists: a-dzwb0z8xhym98oebr
On AWS is do see the association a-dzwb0z8xhym98oebr created by crossplane. Not sure where is the drift 🤷♂️
ashishvaishno commented
@candonov Were you able to test this out?
candonov commented
Hi @ashishvaishno , I tried testing but the provider names are different than the ones in this repository. I know it's been a while, did you get this working or should we dig in?
ashishvaishno commented
@candonov Sorry for the late reply. This is still an issue. It also complains for already existing resources
async create failed: resource creation call returned error diags: creating Amazon EKS (Elastic Kubernetes) Pod Identity Association ("a-wlcylmg69wzd65sp9"): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9: operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 409, RequestID: 558cbce6-04a6-4e15-bf01-a047e2c3ffb3, ResourceInUseException: Association already exists: a-wlcylmg69wzd65sp9
- Can I force crossplane reconciliation if the PIA already exists somehow?