dotnet/orleans

Cluster Manifest Broken by v7.2.5

orionstudt opened this issue · 0 comments

We experienced an issue where after bumping from v7.2.4 to v7.2.6 we were getting from following logs from our remote cluster:

Exception reading message "Request [ sys.client/97a1e507408e496e91dda84405325844]->[S10.16.5.43:30004:0 sys.svc.manifest/10.16.5.43:30004@0] #1" from remote endpoint "10.16.5.48:52438" to local endpoint "10.16.5.43:30004"

With the following exception:

System.TypeLoadException: Unable to resolve type alias "("inv",[Orleans.Runtime.GrainReference],[Orleans.Runtime.IClusterManifestSystemTarget,Orleans.Core],"4EFCA109")".
at Orleans.Serialization.TypeSystem.TypeConverter.ResolveCompoundAliasType[TState](TupleTypeSpec input, TState& state) in //src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 574
at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter1.HandleCompoundType(TupleTypeSpec type, String assemblyName) in /_/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 233 at Orleans.Serialization.TypeSystem.RuntimeTypeNameRewriter.TypeRewriter1.ApplyInner(TypeSpec input, String assemblyName) in /
/src/Orleans.Serialization/TypeSystem/RuntimeTypeNameRewriter.cs:line 77
at Orleans.Serialization.TypeSystem.TypeConverter.ParseInternal(TypeSpec parsed, Type& type) in //src/Orleans.Serialization/TypeSystem/TypeConverter.cs:line 345
at Orleans.Serialization.TypeSystem.TypeCodec.TryRead[TInput](Reader1& reader) in /_/src/Orleans.Serialization/TypeSystem/TypeCodec.cs:line 72 at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadType[TInput](Reader1& reader, SchemaType schemaType) in /
/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 187
at Orleans.Serialization.Codecs.FieldHeaderCodec.ReadExtendedFieldHeader[TInput](Reader1& reader, Field& field) in /_/src/Orleans.Serialization/Codecs/FieldHeaderCodec.cs:line 169 at Orleans.Runtime.Messaging.MessageSerializer.ReadBodyObject[TInput](Message message, Reader1& reader) in //src/Orleans.Core/Messaging/MessageSerializer.cs:line 130
at Orleans.Runtime.Messaging.MessageSerializer.TryRead(ReadOnlySequence`1& input, Message& message) in /
/src/Orleans.Core/Messaging/MessageSerializer.cs:line 126
at Orleans.Runtime.Messaging.Connection.ProcessIncoming() in /_/src/Orleans.Core/Networking/Connection.cs:line 349

Followed by the following log from our client host:

Failed to fetch cluster manifest update from "GrainReference:sys.svc.manifest/10.16.5.43:30004@0:Orleans.Runtime.IClusterManifestSystemTarget".

These logs occur every minute, on every attempt to retrieve cluster manifest.

We managed to fix this by downgrading all of our Orleans references to v7.2.4. We noticed that there was some kind of change related to the cluster manifest in the v7.2.5 release that might be related?