MongoGrainStorage.WriteStateAsync failed. Exception=Error writing object reference for

[AElfBourneShi]

I'm using Orleans.Providers.MongoDB 8.0.3 as grain storage, but ran into this exception.
2024-07-11 12:36:38.152 +00:00 [ERR] MongoGrainStorage.WriteStateAsync failed. Exception=Error writing object reference for 'AeFinder.Grains.State.BlockStates.AppState'. Path 'LastIrreversibleState'. Newtonsoft.Json.JsonSerializationException: Error writing object reference for 'AeFinder.Grains.State.BlockStates.AppState'. Path 'LastIrreversibleState'. ---> System.ArgumentException: A different value already has the Id '183437'. at Newtonsoft.Json.Utilities.BidirectionalDictionary2.Set(TFirst first, TSecond second)
at Newtonsoft.Json.Serialization.DefaultReferenceResolver.GetReference(Object context, Object value)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
--- End of inner exception stack trace ---
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.GetReference(JsonWriter writer, Object value)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value, Type objectType)
at Newtonsoft.Json.Linq.JToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer)
at Newtonsoft.Json.Linq.JObject.FromObject(Object o, JsonSerializer jsonSerializer)
at Orleans.Providers.MongoDB.StorageProviders.Serializers.JsonGrainStateSerializer.Serialize[T](T state)
at Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorageCollection.WriteAsync[T](GrainId grainId, IGrainState1 grainState) at Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage.<>c__DisplayClass13_0.<<DoAndLog>b__0>d.MoveNext() --- End of stack trace from previous location --- at Orleans.Providers.MongoDB.StorageProviders.MongoGrainStorage.DoAndLog[T](String actionName, Func1 action)
`
it's caused by Json.Net, which is not thread-safe when reuse the same serializer:

JamesNK/Newtonsoft.Json#1452

In this driver's JsonGrainStateSerializer, it reuses JsonSerializer, and was registered as Singleton:
`
public class JsonGrainStateSerializer : IGrainStateSerializer
{
private readonly JsonSerializer serializer;

    public JsonGrainStateSerializer(IOptions<JsonGrainStateSerializerOptions> options, IServiceProvider serviceProvider)
    {
        var jsonSettings = OrleansJsonSerializerSettings.GetDefaultSerializerSettings(serviceProvider);
        options.Value.ConfigureJsonSerializerSettings(jsonSettings);
        serializer = JsonSerializer.CreateDefault(jsonSettings);
    }

    public T Deserialize<T>(BsonValue value)
    {
        using var jsonReader = new JTokenReader(value.ToJToken());
        return serializer.Deserialize<T>(jsonReader);
    }

    public BsonValue Serialize<T>(T state)
    {
        return JObject.FromObject(state, serializer).ToBson();
    }
}

But in OrleansJsonSerializer.cs, the implemetation of Serialize method, calls JsonConverter.SerializeObject, which creates a new JsonSerializer everytime, and old version like JsonGrainStateSerializer in Orleans.Providers.MongoDB 3.7.0 also creates a new JsonSerializer everytime, so avoids thread safety issue need to refactor JsonGrainStateSerializer like this:
public class NewJsonGrainStateSerializer: IGrainStateSerializer
{
private readonly JsonSerializerSettings jsonSettings;

public NewJsonGrainStateSerializer(IOptions<JsonGrainStateSerializerOptions> options, IServiceProvider serviceProvider)
{
    jsonSettings = OrleansJsonSerializerSettings.GetDefaultSerializerSettings(serviceProvider);
    options.Value.ConfigureJsonSerializerSettings(jsonSettings);
}

public T Deserialize<T>(BsonValue value)
{
    using var jsonReader = new JTokenReader(value.ToJToken());
    var localSerializer = JsonSerializer.CreateDefault(jsonSettings);
    return localSerializer.Deserialize<T>(jsonReader);
}

public BsonValue Serialize<T>(T state)
{
    var localSerializer = JsonSerializer.CreateDefault(jsonSettings);
    return JObject.FromObject(state, localSerializer).ToBson();
}

}
I have already refactor this, and it works well, no any exception for a long time running. and after refactor NewJsonGrainStateSerializer,we need to register it on this way:
.Configure(options => options.ConfigureJsonSerializerSettings =
settings =>
{
settings.NullValueHandling = NullValueHandling.Include;
settings.ObjectCreationHandling = ObjectCreationHandling.Replace;
settings.DefaultValueHandling = DefaultValueHandling.Populate;
})
.ConfigureServices(services => services.AddSingleton<IGrainStateSerializer,NewJsonGrainStateSerializer>())`

[AElfBourneShi]

By the way, this issue also appeared in the version before 3.7.0, but was later fixed by @SebastianStehle, the following is the related issue and the fix commit link：
JamesNK/Newtonsoft.Json#870
#86
955b156

[wassim-k]

Thanks @AElfBourneShi for raising the issue, you're right this seems like a regression.
I'm currently on holiday with no access to a computer, if you're comfortable with raising a PR then I can review it, otherwise I'll have a look when I'm back.

[AElfBourneShi]

It's better if you do. Anytime is OK.

Thanks @AElfBourneShi for raising the issue, you're right this seems like a regression. I'm currently on holiday with no access to a computer, if you're comfortable with raising a PR then I can review it, otherwise I'll have a look when I'm back.