Scaling beyond 7 replicas causes failure in MongoDB Kubernetes Cluster with Mongo Community Operator

[raihimanshu]

Environment:

Kubernetes version: 1.28
MongoDB Operator version: 0.90
MongoDB Version: 7.0.0

Description:
When attempting to scale the MongoDB replica set beyond 7 members using the MongoDBCommunity operator resource in the Kubernetes cluster, the existing replicas encounters a failure, specifically exhibiting a 'readiness probe failed' error for each existing pod. Additionally, any newly created pod does not get added to the replica set.

However, when the replica set is limited to 7 or fewer members, the cluster operates as expected without any issues.

Steps to Reproduce:

Deploy the MongoDBCommunity resource with the following configuration:
apiVersion: mongodbcommunity.mongodb.com/v1 kind: MongoDBCommunity metadata: name: mongodb namespace: mongodb spec: members: 7 type: ReplicaSet version: "7.0.0" security: authentication: modes: ["SCRAM"] users: - name: my-user db: admin passwordSecretRef: # a reference to the secret that will be used to generate the user's password name: mongodb-password roles: - name: clusterAdmin db: admin - name: userAdminAnyDatabase db: admin - name: dbAdminAnyDatabase db: admin - name: readWriteAnyDatabase db: admin scramCredentialsSecretName: my-scram statefulSet: spec: template: spec: # resources can be specified by applying an override # per container name. containers: - name: mongod resources: limits: cpu: "8" memory: "16Gi" requests: cpu: "8" memory: "16Gi" additionalMongodConfig: storage.wiredTiger.engineConfig.journalCompressor: zlib

Scale the replica set beyond 7 members:

spec: members: 8

Expected Behavior:
The MongoDB replica set should scale beyond 7 members without any failures.

Actual Behavior:
When scaling beyond 7 replicas, all additional replicas fail.

Logs:
I have checked in the logs but can't see any error

Impact:
This issue is preventing us from scaling our MongoDB replica set to meet increased demand.

Request:
I will appreciate any guidance or updates on this issue.