Issue with missing grain reference when retrieving all reminders
ADringer opened this issue · 0 comments
ADringer commented
Hi,
I have an issue when I'm trying to retrieve all reminders for a grain which are persisted to DynamoDb. I have the following code:
if (Persistence.RecordExists)
await Persistence.ClearStateAsync();
var reminders = await GetReminders();
foreach(var reminder in reminders)
{
await UnregisterReminder(reminder);
}
DeactivateOnIdle();
And when it tries to get all the reminders (var reminders = await GetReminders();) it first logs the following warning:
{
"EventId": 102900,
"LogLevel": "Warning",
"Category": "Orleans.Reminders.DynamoDB.DynamoDBReminderTable",
"Message": "Intermediate error reading reminder entry [{:ServiceId -> Amazon.DynamoDBv2.Model.AttributeValue}\n{:GrainReference -> Amazon.DynamoDBv2.Model.AttributeValue}] from table MCC-reminders.",
"Exception": "Amazon.DynamoDBv2.AmazonDynamoDBException: Query condition missed key schema element: GrainReference ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Dictionary`2 lastEvaluatedKey, Boolean consistentRead) in /_/src/AWS/Shared/Storage/DynamoDBStorage.cs:line 683 at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAllAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Boolean consistentRead) at Orleans.Reminders.DynamoDB.DynamoDBReminderTable.ReadRows(GrainReference grainRef) in /_/src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:line 148"
}
And then throws the following exception:
{
"EventId": 0,
"LogLevel": "Error",
"Category": "Infrastructure.GrainExceptionFilter",
"Message": "Query condition missed key schema element: GrainReference ",
"Exception": "Amazon.DynamoDBv2.AmazonDynamoDBException: Query condition missed key schema element: GrainReference ---> Amazon.Runtime.Internal.HttpErrorResponseException: Exception of type 'Amazon.Runtime.Internal.HttpErrorResponseException' was thrown. at Amazon.Runtime.HttpWebRequestMessage.GetResponseAsync(CancellationToken cancellationToken) at Amazon.Runtime.Internal.HttpHandler`1.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Unmarshaller.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) --- End of inner exception stack trace --- at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionStream(IRequestContext requestContext, IWebResponseData httpErrorResponse, HttpErrorResponseException exception, Stream responseStream) at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleExceptionAsync(IExecutionContext executionContext, HttpErrorResponseException exception) at Amazon.Runtime.Internal.ExceptionHandler`1.HandleAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.ProcessExceptionAsync(IExecutionContext executionContext, Exception exception) at Amazon.Runtime.Internal.ErrorHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.Signer.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext) at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext) at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Dictionary`2 lastEvaluatedKey, Boolean consistentRead) in /_/src/AWS/Shared/Storage/DynamoDBStorage.cs:line 683 at Orleans.Reminders.DynamoDB.DynamoDBStorage.QueryAllAsync[TResult](String tableName, Dictionary`2 keys, String keyConditionExpression, Func`2 resolver, String indexName, Boolean scanIndexForward, Boolean consistentRead) at Orleans.Reminders.DynamoDB.DynamoDBReminderTable.ReadRows(GrainReference grainRef) in /_/src/AWS/Orleans.Reminders.DynamoDB/Reminders/DynamoDBReminderTable.cs:line 148 at Orleans.Runtime.ReminderService.LocalReminderService.GetReminders(GrainReference grainRef) in /_/src/Orleans.Runtime/ReminderService/LocalReminderService.cs:line 165 at Orleans.OrleansCodeGenReminderServiceMethodInvoker.Invoke(IAddressable grain, InvokeMethodRequest request) in /_/src/Orleans.Core/obj/Release/net6.0/Orleans.Core.orleans.g.cs:line 5058 at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 104 at Infrastructure.GrainExceptionFilter.Invoke(IIncomingGrainCallContext context) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 104"
}
Might be related, but I'll chuck it in just in case. I also have a grain exception filter where I log the method that failed and the parameters passed to it to help with troubleshooting. I do this using the following context.Arguments.Where(x => x is not Type).Select(x => JsonSerializer.Serialize(x, options)) where one of the parameters I assume is a reference to the grain that causes the exceptions above. This generates the following exception:
{
"EventId": 0,
"LogLevel": "Error",
"Category": "Infrastructure.GrainExceptionFilter",
"Message": "Failed to log error.",
"Exception": "System.InvalidOperationException: This overload of UniqueKey.PrimaryKeyToGuid cannot be used if the grain uses the primary key extension feature. at Orleans.Runtime.UniqueKey.ThrowIfHasKeyExt(String methodName) in /_/src/Orleans.Core.Abstractions/IDs/UniqueKey.cs:line 195 at Orleans.Runtime.GrainId.get_PrimaryKey() in /_/src/Orleans.Core.Abstractions/IDs/GrainId.cs:line 106 at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.JsonSerializer.WriteCore[TValue](Utf8JsonWriter writer, TValue& value, JsonTypeInfo`1 jsonTypeInfo) at System.Text.Json.JsonSerializer.WriteString[TValue](TValue& value, JsonTypeInfo`1 jsonTypeInfo) at Infrastructure.GrainExceptionFilter.<>c__DisplayClass3_0.<Invoke>b__1(Object x) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 68 at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext() at System.String.Join(String separator, IEnumerable`1 values) at Infrastructure.GrainExceptionFilter.Invoke(IIncomingGrainCallContext context) in /src/services/MonitorAndControlService/src/MonitorAndControlGrains/Infrastructure/GrainExceptionFilter.cs:line 104"
}
So I can see that the grain reference isn't being passed along to the retrieval from DynamoDb. It looks like the grain itself has lost it's own reference at some point - what could be causing this?
Thanks