gottscj/Hangfire.Mongo

Unable to clean up locks on the resource

epozsh opened this issue · 2 comments

epozsh commented

Hello using

<PackageReference Include="Hangfire" Version="1.8.6" />
<PackageReference Include="Hangfire.Core" Version="1.8.6" />
<PackageReference Include="Hangfire.Mongo" Version="1.9.15" />

this is the configuration I have used

var mongoUrlBuilder = new MongoUrlBuilder(mongoConnectionString );
var mongoClient = new MongoClient(mongoUrlBuilder.ToMongoUrl());

services.AddHangfire(configuration => configuration
	.SetDataCompatibilityLevel(CompatibilityLevel.Version_180)
	.UseSimpleAssemblyNameTypeSerializer()
	.UseRecommendedSerializerSettings()
	.UseMongoStorage(mongoClient, mongoDatabaseName, new MongoStorageOptions
	{
		MigrationOptions = new MongoMigrationOptions
		{
			MigrationStrategy = new MigrateMongoMigrationStrategy(),
			BackupStrategy = new CollectionMongoBackupStrategy()
		},
		Prefix = "hangfire.mongo",
		CheckConnection = true
	})

);

services.AddHangfireServer();

And random time I am getting below errors

Hangfire:recurring-jobs:lock - Unable to clean up locks on the resource. Details:
MongoDB.Driver.MongoWriteException: A write operation resulted in an error. WriteConcernError: { Code : "64", CodeName : "WriteConcernFailed", Message : "waiting for replication timed out", Details : "{ "wtimeout" : true, "writeConcern" : { "w" : "majority", "wtimeout" : 5000, "provenance" : "clientSupplied" } }" }.
 ---> MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors. WriteErrors:  ]. WriteConcernError: { Code : "64", CodeName : "WriteConcernFailed", Message : "waiting for replication timed out", Details : "{ "wtimeout" : true, "writeConcern" : { "w" : "majority", "wtimeout" : 5000, "provenance" : "clientSupplied" } }" }.
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass30_0.<BulkWrite>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, Func`3 bulkWrite)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, Func`3 bulkWrite)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, CancellationToken cancellationToken)
   at Hangfire.Mongo.DistributedLock.MongoDistributedLock.Cleanup()

Hangfire:locks:schedulepoller - Unable to clean up locks on the resource. Details:
MongoDB.Driver.MongoWriteException: A write operation resulted in an error. WriteConcernError: { Code : "64", CodeName : "WriteConcernFailed", Message : "waiting for replication timed out", Details : "{ "wtimeout" : true, "writeConcern" : { "w" : "majority", "wtimeout" : 5000, "provenance" : "clientSupplied" } }" }.
 ---> MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors. WriteErrors:  ]. WriteConcernError: { Code : "64", CodeName : "WriteConcernFailed", Message : "waiting for replication timed out", Details : "{ "wtimeout" : true, "writeConcern" : { "w" : "majority", "wtimeout" : 5000, "provenance" : "clientSupplied" } }" }.
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass30_0.<BulkWrite>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, Func`3 bulkWrite)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, Func`3 bulkWrite)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, DeleteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionBase`1.DeleteOne(FilterDefinition`1 filter, CancellationToken cancellationToken)
   at Hangfire.Mongo.DistributedLock.MongoDistributedLock.Cleanup()


-> Exception: MongoDB.Driver.MongoWriteConcernException: waiting for replication timed out
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, ICoreSession session, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelSource channelSource, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.WriteCommandOperation`1.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindAndModifyOperationBase`1.ExecuteAttempt(RetryableWriteContext context, Int32 attempt, Nullable`1 transactionNumber, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteOperationExecutor.Execute[TResult](IRetryableWriteOperation`1 operation, RetryableWriteContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteOperationExecutor.Execute[TResult](IRetryableWriteOperation`1 operation, IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindAndModifyOperationBase`1.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.FindOneAndUpdate[TProjection](IClientSessionHandle session, FilterDefinition`1 filter, UpdateDefinition`1 update, FindOneAndUpdateOptions`2 options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass60_0`1.<FindOneAndUpdate>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.FindOneAndUpdate[TProjection](FilterDefinition`1 filter, UpdateDefinition`1 update, FindOneAndUpdateOptions`2 options, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoJobFetcher.TryGetEnqueuedJob(String queue, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoJobFetcher.TryAllQueues(String[] queues, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoJobFetcher.FetchNextJob(String[] queues, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoConnection.FetchNextJob(String[] queues, CancellationToken cancellationToken)
   at Hangfire.Server.Worker.Execute(BackgroundProcessContext context)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)
gottscj commented

@epozsh,

examining the exception it seems there is a timeout waiting for replication timed out. it most likely is something with your infrastructure and not a Hangfire.Mongo issue. Are you using Atlas or a local setup?

Thanks

gottscj commented

closing due to inactivity