load test: BTPSO deleted the servicebinding object from k8s cluster, but the servicebinding still exists on BTP side

[lin1161]

I had a try with BTPSO multi-tenancy features, I created 200 subaccounts (lindsay-subaccount-0 ... lindsay-subaccount-199) , map each to a namespace (lindsay-subaccount-0 ... lindsay-subaccount-199) , created 200 service manager instances (lindsay-subaccount-0-service-manager ... lindsay-subaccount-199-service-manager) in each subaccount，and created an auditlog serviceinstance with binding under each subaccount.
Then deleted all auditlog services and their bindings in parallel, one of the auditlog instance showed DeleteFailed Status，with error ：ServiceInstance deletion failed: could not delete instance due to 1 existing bindings.
I checked there is no servicebinding in the cluster anymore. But there is still one servicebinding for the instance on BTP cockpit.
It looks like btpso removes the servicebinding from the cluster, but on the btp side the servicebinding is not successfully removed.
Sorry I forgot to check btpso's log for error msgs, and the problem may not be reproduced, but I still want to document the problems I've encountered here for future reference.

kubectl get servicebinding -A
No resources found

kubectl get serviceinstance -A
NAMESPACE                NAME       OFFERING              PLAN      SHARED   DATACENTER   STATUS         READY   AGE
lindsay-subaccount-199   auditlog   auditlog-management   default                         DeleteFailed   True    18m

kubectl describe serviceinstance auditlog -n lindsay-subaccount-199
Name:         auditlog
Namespace:    lindsay-subaccount-199
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"services.cloud.sap.com/v1","kind":"ServiceInstance","metadata":{"annotations":{},"name":"auditlog","namespace":"lindsay-sub...
API Version:  services.cloud.sap.com/v1
Kind:         ServiceInstance
Metadata:
  Creation Timestamp:             2023-10-27T08:41:03Z
  Deletion Grace Period Seconds:  0
  Deletion Timestamp:             2023-10-27T08:48:35Z
  Finalizers:
    services.cloud.sap.com/sap-btp-finalizer
  Generation:  2
  Managed Fields:
    API Version:  services.cloud.sap.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:serviceOfferingName:
        f:servicePlanName:
    Manager:      kubectl
    Operation:    Update
    Time:         2023-10-27T08:41:03Z
    API Version:  services.cloud.sap.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"services.cloud.sap.com/sap-btp-finalizer":
    Manager:      Go-http-client
    Operation:    Update
    Time:         2023-10-27T08:46:07Z
    API Version:  services.cloud.sap.com/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:hashedSpec:
        f:instanceID:
        f:observedGeneration:
        f:ready:
        f:subaccountID:
    Manager:         Go-http-client
    Operation:       Update
    Subresource:     status
    Time:            2023-10-27T08:50:43Z
  Resource Version:  19305
  UID:               e921f06e-c13b-4f5f-8049-0462381a8b29
Spec:
  External Name:          auditlog
  Service Offering Name:  auditlog-management
  Service Plan Name:      default
  User Info:
    Groups:
      system:masters
      system:authenticated
    Username:  minikube-user
Status:
  Conditions:
    Last Transition Time:  2023-10-27T08:50:43Z
    Message:               ServiceInstance deletion failed: could not delete instance due to 1 existing bindings
    Observed Generation:   2
    Reason:                DeleteFailed
    Status:                False
    Type:                  Succeeded
    Last Transition Time:  2023-10-27T08:46:08Z
    Message:              
    Reason:                Provisioned
    Status:                True
    Type:                  Ready
    Last Transition Time:  2023-10-27T08:50:43Z
    Message:               ServiceInstance deletion failed: could not delete instance due to 1 existing bindings
    Observed Generation:   2
    Reason:                DeleteFailed
    Status:                True
    Type:                  Failed
  Hashed Spec:             8f9b6d9b9de7c697690b2b6f561a08f7
  Instance ID:             a052593e-ce54-4e89-88c5-53e9b31113a1
  Observed Generation:     2
  Ready:                   True
  Subaccount ID:           e35e14f3-8a64-4384-bdcf-596dcb5e1886
Events:                    <none>

[kerenlahav]

Hi, please provide the binding id and landscape

[lin1161]

Sorry, I used a minikube cluster, and had already deleted it.

[kerenlahav]

make sure you are using the latest version of the BTP operator.
also notice that you are using the credentials of subaccount-admin and not operator-access as required, this could lead to inconsistencies.
if the issue reproduces please reopen an attach the relevant binding id