Exception when serialize state...
vrecluse opened this issue · 5 comments
I'm using Orleans.Providers.MongoDB as grain storage, but ran into this exception.
silo_1 | 2021-05-20T02:31:03.716529390Z fail: Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage[900100]
silo_1 | 2021-05-20T02:31:03.716547996Z MongoGrainStorage.WriteStateAsync failed. Exception=Error writing object reference for 'System.Collections.Generic.Dictionary`2[System.Int32,Sazabi.World.Grains.Data.PlayerPersistentData+PlayerPersistentSave]'. Path 'SaveData'.
silo_1 | 2021-05-20T02:31:03.716552090Z Newtonsoft.Json.JsonSerializationException: Error writing object reference for 'System.Collections.Generic.Dictionary`2[System.Int32,Sazabi.World.Grains.Data.PlayerPersistentData+PlayerPersistentSave]'. Path 'SaveData'.
silo_1 | 2021-05-20T02:31:03.716555151Z ---> System.ArgumentException: A different value already has the Id '203029'.
silo_1 | 2021-05-20T02:31:03.716568458Z at Newtonsoft.Json.Utilities.BidirectionalDictionary`2.Set(TFirst first, TSecond second)
silo_1 | 2021-05-20T02:31:03.716571331Z at Newtonsoft.Json.Serialization.DefaultReferenceResolver.GetReference(Object context, Object value)
silo_1 | 2021-05-20T02:31:03.716574069Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
silo_1 | 2021-05-20T02:31:03.716576533Z --- End of inner exception stack trace ---
silo_1 | 2021-05-20T02:31:03.716578862Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
silo_1 | 2021-05-20T02:31:03.716581317Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.WriteReferenceIdProperty(JsonWriter writer, Type type, Object value)
silo_1 | 2021-05-20T02:31:03.716584017Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.WriteObjectStart(JsonWriter writer, Object value, JsonContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1 | 2021-05-20T02:31:03.716587252Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeDictionary(JsonWriter writer, IDictionary values, JsonDictionaryContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1 | 2021-05-20T02:31:03.716590438Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
silo_1 | 2021-05-20T02:31:03.716593696Z at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
silo_1 | 2021-05-20T02:31:03.716596234Z at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
silo_1 | 2021-05-20T02:31:03.716598668Z at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
silo_1 | 2021-05-20T02:31:03.716601068Z at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
silo_1 | 2021-05-20T02:31:03.716603497Z at Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage.<>c__DisplayClass20_0.<<WriteStateAsync>b__0>d.MoveNext()
Is it caused by circular reference or duplicated reference? But my state definition is fairly simple:
[Serializable]
public class PlayerPersistentSave
{
public string Version { get; set; }
public bool IsSavedByDelta { get; set; }
public byte[] Data { get; set; }
}
[Serializable]
public class PlayerPersistentState
{
public bool NewPlayerStageDone { set; get; }
public long SceneSequence { get; set; }
public uint DataSequence { get; set; }
public Dictionary<int, PlayerPersistentSave> SaveData { get; set; } = new Dictionary<int, PlayerPersistentSave>();
}
This sounds like something related to JSON.NET. I would guess that your dictionary is being concurrently modified and so there's a data race somewhere, but I'm not sure.
Yes, I think @ReubenBond is correct here.
I run into this problem again, and found that maybe it's caused by Json.Net, which is not thread-safe when reuse the same serializer:
In this driver's JsonGrainStateSerializer, it reuses JsonSerializer, and was registered as Singleton:
public class JsonGrainStateSerializer : IGrainStateSerializer
{
private readonly JsonSerializer serializer;
public JsonGrainStateSerializer(ITypeResolver typeResolver, IGrainFactory grainFactory, MongoDBGrainStorageOptions options)
{
var jsonSettings = OrleansJsonSerializer.GetDefaultSerializerSettings(typeResolver, grainFactory);
options?.ConfigureJsonSerializerSettings?.Invoke(jsonSettings);
this.serializer = JsonSerializer.Create(jsonSettings);
if (options?.ConfigureJsonSerializerSettings == null)
{
//// https://github.com/OrleansContrib/Orleans.Providers.MongoDB/issues/44
//// Always include the default value, so that the deserialization process can overwrite default
//// values that are not equal to the system defaults.
this.serializer.NullValueHandling = NullValueHandling.Include;
this.serializer.DefaultValueHandling = DefaultValueHandling.Populate;
}
}
public void Deserialize(IGrainState grainState, JObject entityData)
{
var jsonReader = new JTokenReader(entityData);
serializer.Populate(jsonReader, grainState.State);
}
public JObject Serialize(IGrainState grainState)
{
return JObject.FromObject(grainState.State, serializer);
}
}
But in OrleansJsonSerializer.cs, the implemetation of Serialize method, calls JsonConverter.SerializeObject, which creates a new JsonSerializer everytime, avoids thread safety issue:
public static string SerializeObject(object value, Type type, JsonSerializerSettings settings)
{
JsonSerializer jsonSerializer = JsonSerializer.CreateDefault(settings);
return JsonConvert.SerializeObjectInternal(value, type, jsonSerializer);
}
Can you provide a PR for that?
Please note that if you are using a version lower than Orleans 7.0, you should exercise caution when using the default JsonGrainStateSerializer
. Specifically, if your grain state contains a large list, you may encounter exceptions such as 'Size {} is larger than 16m'. This is due to the fact that the serializer uses the ObjectCreationHandling.Auto
setting, which can result in the list being reused and duplicate items being created during deserialize via the serializer.Populate(jsonReader, grainState.State)
method.
To fix this, just add the following config when call AddMongoDBGrainStorage
:
.AddMongoDBGrainStorageAsDefault(optionsBuilder =>
{
optionsBuilder.Configure(op =>
{
op.ConfigureJsonSerializerSettings = jsonSettings =>
{
jsonSettings.NullValueHandling = NullValueHandling.Include;
jsonSettings.DefaultValueHandling = DefaultValueHandling.Populate;
jsonSettings.ObjectCreationHandling = ObjectCreationHandling.Replace;
};
//...
});
})