Exception when serialize state...

[vrecluse]

I'm using Orleans.Providers.MongoDB as grain storage, but ran into this exception.

silo_1  | 2021-05-20T02:31:03.716529390Z fail: Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage[900100]
silo_1  | 2021-05-20T02:31:03.716547996Z       MongoGrainStorage.WriteStateAsync failed. Exception=Error writing object reference for 'System.Collections.Generic.Dictionary`2[System.Int32,Sazabi.World.Grains.Data.PlayerPersistentData+PlayerPersistentSave]'. Path 'SaveData'.
silo_1  | 2021-05-20T02:31:03.716552090Z       Newtonsoft.Json.JsonSerializationException: Error writing object reference for 'System.Collections.Generic.Dictionary`2[System.Int32,Sazabi.World.Grains.Data.PlayerPersistentData+PlayerPersistentSave]'. Path 'SaveData'.
silo_1  | 2021-05-20T02:31:03.716555151Z        ---> System.ArgumentException: A different value already has the Id '203029'.
silo_1  | 2021-05-20T02:31:03.716568458Z          at Newtonsoft.Json.Utilities.BidirectionalDictionary`2.Set(TFirst first, TSecond second)
silo_1  | 2021-05-20T02:31:03.716571331Z          at Newtonsoft.Json.Serialization.DefaultReferenceResolver.GetReference(Object context, Object value)

silo_1  | 2021-05-20T02:31:03.716574069Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
silo_1  | 2021-05-20T02:31:03.716576533Z          --- End of inner exception stack trace ---
silo_1  | 2021-05-20T02:31:03.716578862Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
silo_1  | 2021-05-20T02:31:03.716581317Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.WriteReferenceIdProperty(JsonWriter writer, Type type, Object value)
silo_1  | 2021-05-20T02:31:03.716584017Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.WriteObjectStart(JsonWriter writer, Object value, JsonContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1  | 2021-05-20T02:31:03.716587252Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1  | 2021-05-20T02:31:03.716590438Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1  | 2021-05-20T02:31:03.716593696Z          at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
silo_1  | 2021-05-20T02:31:03.716596234Z          at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
silo_1  | 2021-05-20T02:31:03.716598668Z          at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
silo_1  | 2021-05-20T02:31:03.716601068Z          at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
silo_1  | 2021-05-20T02:31:03.716603497Z          at Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage.<>c__DisplayClass20_0.<<WriteStateAsync>b__0>d.MoveNext()

Is it caused by circular reference or duplicated reference? But my state definition is fairly simple:

        [Serializable]
        public class PlayerPersistentSave
        {
            public string Version { get; set; }
            public bool IsSavedByDelta { get; set; }
            public byte[] Data { get; set; }
        }

        [Serializable]
        public class PlayerPersistentState
        {
            public bool NewPlayerStageDone { set; get; }
            public long SceneSequence { get; set; }
            public uint DataSequence { get; set; }
            public Dictionary<int, PlayerPersistentSave> SaveData { get; set; } = new Dictionary<int, PlayerPersistentSave>();
        }

[ReubenBond]

This sounds like something related to JSON.NET. I would guess that your dictionary is being concurrently modified and so there's a data race somewhere, but I'm not sure.

[SebastianStehle]

Yes, I think @ReubenBond is correct here.

[vrecluse]

I run into this problem again, and found that maybe it's caused by Json.Net, which is not thread-safe when reuse the same serializer:

JamesNK/Newtonsoft.Json#1452

In this driver's JsonGrainStateSerializer, it reuses JsonSerializer, and was registered as Singleton:

    public class JsonGrainStateSerializer : IGrainStateSerializer
    {
        private readonly JsonSerializer serializer;

        public JsonGrainStateSerializer(ITypeResolver typeResolver, IGrainFactory grainFactory, MongoDBGrainStorageOptions options)
        {
            var jsonSettings = OrleansJsonSerializer.GetDefaultSerializerSettings(typeResolver, grainFactory);           

            options?.ConfigureJsonSerializerSettings?.Invoke(jsonSettings);
            this.serializer = JsonSerializer.Create(jsonSettings);

            if (options?.ConfigureJsonSerializerSettings == null)
            {
                //// https://github.com/OrleansContrib/Orleans.Providers.MongoDB/issues/44
                //// Always include the default value, so that the deserialization process can overwrite default 
                //// values that are not equal to the system defaults.
                this.serializer.NullValueHandling = NullValueHandling.Include;
                this.serializer.DefaultValueHandling = DefaultValueHandling.Populate;
            }            
        }

        public void Deserialize(IGrainState grainState, JObject entityData)
        {
            var jsonReader = new JTokenReader(entityData);

            serializer.Populate(jsonReader, grainState.State);
        }

        public JObject Serialize(IGrainState grainState)
        {
            return JObject.FromObject(grainState.State, serializer);
        }
    }

But in OrleansJsonSerializer.cs, the implemetation of Serialize method, calls JsonConverter.SerializeObject, which creates a new JsonSerializer everytime, avoids thread safety issue:

    public static string SerializeObject(object value, Type type, JsonSerializerSettings settings)
    {
      JsonSerializer jsonSerializer = JsonSerializer.CreateDefault(settings);
      return JsonConvert.SerializeObjectInternal(value, type, jsonSerializer);
    }

[SebastianStehle]

Can you provide a PR for that?

[sheng-jie]

Please note that if you are using a version lower than Orleans 7.0, you should exercise caution when using the default JsonGrainStateSerializer. Specifically, if your grain state contains a large list, you may encounter exceptions such as 'Size {} is larger than 16m'. This is due to the fact that the serializer uses the ObjectCreationHandling.Auto setting, which can result in the list being reused and duplicate items being created during deserialize via the serializer.Populate(jsonReader, grainState.State) method.

To fix this, just add the following config when call AddMongoDBGrainStorage：

.AddMongoDBGrainStorageAsDefault(optionsBuilder =>
                {
                    optionsBuilder.Configure(op =>
                    {
                        op.ConfigureJsonSerializerSettings = jsonSettings =>
                        {
                            jsonSettings.NullValueHandling = NullValueHandling.Include;
                            jsonSettings.DefaultValueHandling = DefaultValueHandling.Populate;
                            jsonSettings.ObjectCreationHandling = ObjectCreationHandling.Replace;
                        };
                        //...
                    });
                })