Unable to update Silo Entry
matthewbrumpton-oddsbods opened this issue · 1 comments
We have a large number of KubeMembershipTable exceptions coming from our orleans application, the errors start after the application has been running for 5-9 hours.
Message: Unable to find silo entry 172.19.16.62-50000-374433332.
CategoryName: Orleans.Clustering.Kubernetes.KubeMembershipTable
Exception type: System.InvalidOperationException
Failed method: Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext
Problem Id: System.InvalidOperationException at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext
Assembly: Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null
Call Stack: System.InvalidOperationException:
at Orleans.Clustering.Kubernetes.KubeMembershipTable+d__11.MoveNext (Orleans.Clustering.Kubernetes, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null)
Another observation is OrleansClusterVersion is deleted after a period of time and then being recreated after we restart the app.
OrleansClusterVersion after we restart the app:
apiVersion: v1
items:
apiVersion: orleans.dot.net/v1
clusterId: clusterId
clusterVersion: 4
kind: OrleansClusterVersion
metadata:
creationTimestamp: "2021-11-15T14:43:13Z"
generation: 5
name: name
namespace: namspace
resourceVersion: "148545790"
uid: d4a66cf5-18a9-4474-bf88-351d496ac61d
kind: List
metadata:
resourceVersion: ""
selfLink: ""
OrleansClusterVersion when deleted:
kubectl get clusterversions --namespace namespace -o yaml
apiVersion: v1
items: []
kind: List
metadata:
resourceVersion: ""
selfLink: ""