Access Denied Error: Failed to create mount process in AWS China region

[wttat]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened?

Driver:

E0311 06:04:17.300386 1 driver.go:96] GRPC error: rpc error: code = Internal desc = Could not mount "alphafold2-dataset-bjs" at "/var/lib/kubelet/pods/efb8c26a-e4f0-44e6-8685-a739dcb82c81/volumes/kubernetes.io~csi/s3-pv/mount": Mount failed: Failed to start service output: Error: Failed to create S3 client Caused by: 0: initial ListObjectsV2 failed for bucket alphafold2-dataset-bjs in region cn-north-1 1: Client error 2: Forbidden: Access Denied Error: Failed to create mount process
What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

REGION=cn-north-1
CLUSTER_NAME=EKS-s3-csi-test
aws configure set default.region $REGION

cat > cluster-config.yaml <<EOF
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: $CLUSTER_NAME
  region: $REGION
  version: "1.29"

managedNodeGroups:
  - name: ng-1-workers
    labels: { role: workers }
    instanceType: m5.large
    desiredCapacity: 1
    volumeSize: 80
    privateNetworking: true
  - name: ng-2-builders
    labels: { role: builders }
    instanceType: m5.large
    desiredCapacity: 2
    volumeSize: 100
    privateNetworking: true
EOF

eksctl create cluster -f cluster-config.yaml

aws eks update-kubeconfig --region $REGION --name $CLUSTER_NAME

eksctl utils associate-iam-oidc-provider --cluster $CLUSTER_NAME --approve

eksctl create iamserviceaccount --name s3-csi-driver-sa \
--namespace kube-system \
--cluster $CLUSTER_NAME \
--role-name s3-csi-driver-role \
--attach-policy-arn arn:aws-cn:iam::aws:policy/AmazonS3FullAccess \
--approve 

kubectl apply -k "github.com/awslabs/mountpoint-s3-csi-driver/deploy/kubernetes/overlays/stable/"

kubectl get pods -n kube-system -l app.kubernetes.io/name=aws-mountpoint-s3-csi-driver

wget https://raw.githubusercontent.com/awslabs/mountpoint-s3-csi-driver/main/examples/kubernetes/static_provisioning/static_provisioning.yaml

sed 's/- region us-west-2/- region cn-north-1/g; s/bucketName: s3-csi-driver/bucketName: alphafold2-dataset-bjs/g' static_provisioning.yaml > static_provisioning_wt.yaml

kubectl apply -f static_provisioning_wt.yaml

kubectl describe pod s3-app

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): 1.29
• Driver version: 1.4.0

[jjkr]

Thanks for your patience here. There should be nothing special about the China region for this use case. I ran through the reproduction steps in a brand new China account (I appreciate the through report) and was not able to reproduce, but think I do see the issue.

The s3-csi-driver-sa service account needs a special annotation to work with IAM roles for service accounts (IRSA). You can add this after sa creation with a command like this (ensuring the role arn and service account name matches your setup):

kubectl annotate serviceaccount --overwrite -n kube-system s3-csi-driver-sa eks.amazonaws.com/role-arn=arn:aws-cn:iam::ACCOUNT_NUMBER:role/s3-csi-driver-role

Your role will also need oidc trust policies, which eksctl should do for you when creating the service account. It also should set this annotation though, so double check those and the AmazonS3FullAccess policy are there on the role you want to use. There is some good documentation on IRSA here which could be helpful: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[seanfariaparetointel]

@wttat I had this problem and solved it today. It is due to the documentation missing the addon step. You are supposed to create an addon before creating the service account.

eksctl create addon --name aws-mountpoint-s3-csi-driver --cluster --service-account-role-arn arn:aws:iam::1111111111:role/AmazonEKS_S3_CSI_DriverRole --force

Note: you will need to recreate the service account once the addon has been added.

Additionally you do need the annotation mentioned above @jjkr

[igor-golubovich]

I had the same issue. I created addon and also add annotation in service account. But in my case the reason was in Trusted entities for IAM role - "Condition": {"StringEquals": {}/. In documentation https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html - "system:serviceaccount:kube-system:s3-csi-*" - I got an error. I altered it to full name - "system:serviceaccount:kube-system:s3-csi-driver-sa" - It solved my issue

[dannycjones]

I had the same issue. I created addon and also add annotation in service account. But in my case the reason was in Trusted entities for IAM role - "Condition": {"StringEquals": {}/. In documentation https://docs.aws.amazon.com/eks/latest/userguide/s3-csi.html - "system:serviceaccount:kube-system:s3-csi-*" - I got an error. I altered it to full name - "system:serviceaccount:kube-system:s3-csi-driver-sa" - It solved my issue

I suspect you needed to use the full name since you may have missed the step to replace StringEquals with StringLike, so the wildcard would not work.

[dannycjones]

@wttat, are you still experiencing an issue here? Please let us know if we can help.

[easy1481437320]

i met the same problem follow the guide : https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/s3-csi.html

output: Error: Failed to create S3 client  Caused by:     0: initial ListObjectsV2 failed for bucket xxxx in region xxx     1: Client error     2: Forbidden: Access Denied Error: Failed to create mount process

i attached the AmazonS3CSIDriverPolicy into my AmazonEKSNodeRole which use for EC2 nodegroup to resolved this problem.

[dannycjones]

I'm going to assume that the issue is resolved for the original author and close the issue.

If anyone else is stuck with this issue, please do open a new issue and we can take a look.

[peterbosalliandercom]

i attached the AmazonS3CSIDriverPolicy into my AmazonEKSNodeRole which use for EC2 nodegroup to resolved this problem.

The reason why node permission worked is that pod level is not working.