3scale install fails on Azure RedHat OpenShift [ARO] 4.5
ezYakaEagle442 opened this issue · 9 comments
Following RedHat 3scale install doc on OpenShift, it is mentionned
- Persistent volumes
-
- RWO (ReadWriteOnce) persistent volumes for Redis and MySQL persistence
-
- RWX (ReadWriteMany) persistent volume for CMS and System-app Assets
- "Note: OCP 4 supports deployment of 3scale using the operator only"
The 3Scale APIM Supported Configurations shows
- OCP 4.5 & 4.6 are supported, ==> is ARO 4.5 & 4.6 also supported ?
- 3scale API Management 2.9 System component has been tested and it’s supported with the following file storage configurations:
File storage
- Persistent Volume Claims (PVC) with RWX access mode* ( NFS is not a suitable storage for production environments.)
- Amazon Simple Storage Service (S3) compatible storages
When installing it through the Operator Hub in Azure RedHat OpenShift [ARO] 4.5, the installation fails as the System-Storage PVC has been created in RWO, failing with error : "Failed to provision volume with Storage Class "managed-premium: invalid AccessModes [ReadWriteMany]": only AccessModes [ReadWriteOnce] are supported"
This managed-premium storage-class is the default one created in ARO leveraing the in-tree Azure disk plugin which does not support RWX as far as I know :
provisioner: kubernetes.io/azure-disk
parameters:
kind: Managed
storageaccounttype: Premium_LRS
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
ARO4 supports the CSI drivers and I have already sucessfully tested them, see my repo at https://github.com/ezYakaEagle442/aro-pub-storage
The Azure Disk CSI Driver supports Multi-Node Read Write with RWX :
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-csi
provisioner: disk.csi.azure.com
parameters:
skuname: Premium_LRS # Currently shared disk only available with premium SSD
maxShares: "2"
cachingMode: None # ReadOnly cache is not available for premium SSD with maxShares>1
reclaimPolicy: Delete
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-azuredisk
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 256Gi # minimum size of shared disk is 256GB (P15)
volumeMode: Block
storageClassName: managed-csi
The CSI Azure File drivers and the Azure Blob Storage CSI driver support as well RWX .
So could you please fix the installation failure on ARO :
- Stating that the CSI drivers are required, therefore its installation is needed as a pre-req
- Update the System-Storage PVC to managed-csi or file.csi.azure.com or blob.csi.azure.com
Hi @ezYakaEagle442,
If your default PVC storageClass (https://kubernetes.io/docs/concepts/storage/storage-classes/) in your cluster does not support RWX access mode we allow specifying the PVC storage class desired to be used for System's FileStorage PVC through the .spec.system.fileStorage.persistentVolumeClaim.storageClassName attribute for the APIManager custom resource (https://github.com/3scale/3scale-operator/blob/3scale-2.9.1-GA/doc/apimanager-reference.md#SystemPVCSpec). In that way, you can point to a storage class that allows RWX PVCs.
I have created this page which describes all the steps to create to Storage Classes with Azure File & also with CSI Drivers :
https://github.com/ezYakaEagle442/aro-pub-storage/blob/master/3scale.md
Could you please have a look at it , test & verify on 3Scale side if all looks to run fine ?
At the end I did run a check with :
oc describe pvc system-redis-storage
Access Modes: RWO ==> should be RWX
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/azure-disk (this is the default storage class in ARO)
This is obviously because it was also necessary to to set the System's FileStorage PVC through the .spec.systemredis.fileStorage.persistentVolumeClaim.storageClassName
Same as well for the .spec.backendredis.fileStorage.persistentVolumeClaim.storageClassName
Hi @ezYakaEagle442,
system-redis-storage and backend-redis-storage PVCs don't require RWX access mode. They work with RWO access.
The only PVC that requires RWX access mode is System's FileStorage PVC.
The attributes .spec.systemredis.fileStorage.persistentVolumeClaim.storageClassName and .spec.backendredis.fileStorage.persistentVolumeClaim.storageClassName you wrote in your previous comment don't exist with those path.
ARO doesn't support the Azure File or Azure Blob CSI drivers. The Azure File CSI driver is 9+ months away from being supported in OCP. Please close/don't merge this pull request.
Note: I assume the issue described here will be hit on any OCP installation where RWX storage hasn't been configured in advance. If anything, the only bug here is updating the OperatorHub entry to more clearly point to pre-requisites for installing 3Scale.
@luciddreamz , thanks for your reply, then Azure File ( in-tree plugin, not th CSI driver) is tha way to go
Correct. Again, I think the main issue here is either: 1) Update OperatorHub listing to clearly state pre-requisite of RWX storage; or 2) Remove need for RWX storage (unlikely).
3Scale's API manager works fine with azure files, however the developer portal always returns a 404 error.
If I remember correctly, the developer portal's CMS is trying to change file permissions but this fails and causes the error (I might be wrong since I saw the error more than 1 year ago on Openshift 3.11)
Apparently this is due to a limitation in the way azure files mount to Linux and one work around is to scope the "system-storage" PVC to the project in which 3Scale is installed as explained here[1].
I tested this on ARO 4.5.16 with 3Scale 2.9 installed from the operator hub and the developer portal seems to be working fine now.
Please see latest supported: https://access.redhat.com/articles/2798521
Please open another issue if still relevant