getting error hangfire.mongo for amazon document db

Question

getting error hangfire.mongo for amazon document db

depo101 opened this issue 2 years ago · 18 comments

Hi, I'm using the latest version of hangire.mongo and the related libraries. The hangfire project runs on docker container and uses amazon document db as mongodb. I get the error log messages below when the hangfire app starts.

Unhandled exception. Hangfire.Mongo.Migration.MongoMigrationException: Migration failed in Hangfire.Mongo.Migration.Steps.Version06.EnqueuedJobMigration ---> MongoDB.Driver.MongoCommandException: Command find failed: Feature not supported. at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol1.Execute(IConnection connection, CancellationToken cancellationToken) at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol1.Execute(IConnection connection, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol1 protocol, ICoreSession session, CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.CommandOperationBase1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.ReadCommandOperation1.ExecuteAttempt(RetryableReadContext context, Int32 attempt, Nullable1 transactionNumber, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.Execute[TResult](IRetryableReadOperation1 operation, RetryableReadContext context, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.ReadCommandOperation1.Execute(RetryableReadContext context, CancellationToken cancellationToken) at MongoDB.Driver.Core.Operations.FindOperation1.Execute(RetryableReadContext context, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Operations.FindOperation1.Execute(IReadBinding binding, CancellationToken cancellationToken) at MongoDB.Driver.OperationExecutor.ExecuteReadOperation[TResult](IReadBinding binding, IReadOperation1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation1 operation, CancellationToken cancellationToken)
at MongoDB.Driver.MongoCollectionImpl1.FindSync[TProjection](IClientSessionHandle session, FilterDefinition1 filter, FindOptions2 options, CancellationToken cancellationToken) t at MongoDB.Driver.MongoCollectionImpl1.<>c__DisplayClass46_01.<FindSync>b__0(IClientSessionHandle session) t at MongoDB.Driver.MongoCollectionImpl1.UsingImplicitSession[TResult](Func2 func, CancellationToken cancellationToken) at MongoDB.Driver.MongoCollectionImpl1.FindSync[TProjection](FilterDefinition1 filter, FindOptions2 options, CancellationToken cancellationToken)
at MongoDB.Driver.FindFluent2.ToCursor(CancellationToken cancellationToken) at MongoDB.Driver.IAsyncCursorSourceExtensions.ToList[TDocument](IAsyncCursorSource1 source, CancellationToken cancellationToken)
at Hangfire.Mongo.Migration.Steps.Version06.EnqueuedJobMigration.Execute(IMongoDatabase database, MongoStorageOptions storageOptions, IMongoMigrationContext migrationContext)
at Hangfire.Mongo.Migration.Strategies.MongoMigrationStrategy.ExecuteMigration(IMongoDatabase database, MongoSchema fromSchema, MongoSchema toSchema)
--- End of inner exception stack trace ---
at Hangfire.Mongo.Migration.Strategies.MongoMigrationStrategy.ExecuteMigration(IMongoDatabase database, MongoSchema fromSchema, MongoSchema toSchema)
at Hangfire.Mongo.Migration.Strategies.MongoMigrationStrategy.Execute(MongoStorageOptions storageOptions, IMongoDatabase database, MongoSchema fromSchema, MongoSchema toSchema)
at Hangfire.Mongo.Migration.MongoMigrationManager.Migrate(MongoBackupStrategy backupStrategy, MongoMigrationStrategy migrationStrategy)
at Hangfire.Mongo.Migration.MongoMigrationManager.MigrateIfNeeded(MongoStorageOptions storageOptions, IMongoDatabase database)
at Hangfire.Mongo.MongoStorage..ctor(IMongoClient mongoClient, String databaseName, MongoStorageOptions storageOptions)
at Hangfire.Mongo.MongoStorage..ctor(MongoClientSettings mongoClientSettings, String databaseName, MongoStorageOptions storageOptions)
at Hangfire.Mongo.MongoBootstrapperConfigurationExtensions.UseMongoStorage(IGlobalConfiguration configuration, MongoClientSettings mongoClientSettings, String databaseName, MongoStorageOptions storageOptions)
at Hangfire.Mongo.MongoBootstrapperConfigurationExtensions.UseMongoStorage(IGlobalConfiguration configuration, String connectionString, MongoStorageOptions storageOptions)
at Tradeport.Services.OrderImport.Application.ServiceRegistration.<>c__DisplayClass0_0.b__1(IGlobalConfiguration x) in /src/src/Services/ServiceRegistration.cs:line 49
at Hangfire.HangfireServiceCollectionExtensions.<>c__DisplayClass0_0.b__0(IServiceProvider provider, IGlobalConfiguration config)
at Hangfire.HangfireServiceCollectionExtensions.<>c__DisplayClass1_0.b__10(IServiceProvider serviceProvider)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context) at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
at System.Collections.Concurrent.ConcurrentDictionary2.GetOrAdd(TKey key, Func2 valueFactory)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetService[T](IServiceProvider provider)
at Hangfire.HangfireServiceCollectionExtensions.ThrowIfNotConfigured(IServiceProvider serviceProvider)
at Hangfire.HangfireApplicationBuilderExtensions.UseHangfireDashboard(IApplicationBuilder app, String pathMatch, DashboardOptions options, JobStorage storage)
at Program.

$(String[] args) in /src/src/Services/Program.cs:line 31
client_loop: send disconnect: Connection reset`

services.AddHangfire(x => { x.UseRecommendedSerializerSettings(); x.UseSimpleAssemblyNameTypeSerializer(); x.UseMongoStorage($"{mongoDbSettings.ConnectionString}", new MongoStorageOptions() { MigrationOptions = new MongoMigrationOptions() { MigrationStrategy = new MigrateMongoMigrationStrategy(), BackupStrategy = new CollectionMongoBackupStrategy() } }); });

Answer 1 · 2023-03-04T20:31:53.000Z

Hi @depo101

We have never tested aws document db. I'm not familiar with it.

You most likely need to write some modifications as it seems it has limited support for the mongodb driver.

Good luck! 🤞

Answer 2 · 2023-03-17T15:56:37.000Z

On documentdb just adding my two cents on what we see. we notice a mongodump/mongorestore throws these two errors.
db1: error running create command: Field 'size' is currently not supported
db2: error running create command: Field 'max' is currently not supported

Answer 3 · 2023-03-28T13:03:00.000Z

For reference after recreating the documentdb cluster with mongo 5.0.0 engine the size and max issue does not show during mongorestore but a different issue shows up.

error creating collection PaymentServices-HangfireBackground.hangfire.notifications: error running create command: Feature not supported: capped:true

Answer 4 · 2023-03-31T14:13:27.000Z

hi,

in MongoStorage try setting
CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll

Good luck!

Answer 5 · 2023-03-31T14:14:50.000Z

also you should probably use use the CosmosBootstrapperConfigurationExtensions and use CosmosStorage

Answer 6 · 2023-04-25T10:34:23.000Z

@depo101 Have you managed to resolve the issue?

Answer 7 · 2023-04-25T12:56:12.000Z

Hello everyone,
@shaliake I couldn't resolve this issue on "amazon document db". I decided to switch mongodb as running on linux vm.

Answer 8 · 2023-04-25T16:47:22.000Z

@gottscj @depo101
I have actually found a way to make it work on AWS DocumentDB. I would like to share my solution. It requires some nuget package versioning things, but it does work

Answer 9 · 2023-06-15T04:38:23.000Z

@shaliake
Hello, could you please share the steps of making it work on AWS DocumentDB?
Thank you!

Answer 10 · 2023-06-15T18:29:17.000Z

@shaliake docdb steps would be very much appreciated!

Answer 11 · 2023-06-18T19:48:51.000Z

@gottscj @depo101 I have actually found a way to make it work on AWS DocumentDB. I would like to share my solution. It requires some nuget package versioning things, but it does work

@shaliake, sounds interesting. Please share :)

Answer 12 · 2023-06-20T06:19:18.000Z

So I know that when using these packages and these particular versions:

<PackageReference Include="Hangfire" Version="1.7.34" />
<PackageReference Include="Hangfire.Mongo" Version="0.7.27" />

The following Hangfire setup works in AWS DocumentDB (Please drop any existing Hangfire collections if you do have them)

         services.AddHangfire(config =>
            {
                config
                    .SetDataCompatibilityLevel(CompatibilityLevel.Version_110)
                    .UseSimpleAssemblyNameTypeSerializer()
                    .UseRecommendedSerializerSettings()
                    .UseColouredConsoleLogProvider(LogLevel.Error)
                    .UseCosmosStorage(client, DatabaseContextDetails.DatabaseName, new CosmosStorageOptions
                    {
                        CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll,
                        MigrationOptions = new MongoMigrationOptions
                        {
                            MigrationStrategy = new DropMongoMigrationStrategy()
                        }
                    });
            });

@daxnet , @rrossouw01 , @gottscj , Please try it out on your own, let me know if something doesn't work right.

Answer 13 · 2023-06-23T17:54:24.000Z

This config is what would expect would give you the best results.
If you use CosmosStorageOptions it already defaults to CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll.

It doesnt work on the latest versions?

Answer 14 · 2023-06-25T07:03:58.000Z

@shaliake
Yes, with your code, I can run Hangfire.Mongo against AWS docdb, in case that I disable the retryWrite option (AWS docdb doesn't support that).
Here is how I constructed my Mongo Client:

// I used ssh tunnel to connect to my docdb:
var mongoUrl = new MongoUrl(@"mongodb://<user>:<pass>@localhost:27018/?authMecanism=DEFAULT&tls=true&tlsCAFile=us-east-1-bundle.pem&tlsAllowInvalidHostnames=true");
var mongoSettings = MongoClientSettings.FromUrl(mongoUrl);
mongoSettings.AllowInsecureTls = true;
mongoSettings.RetryWrites = false;
var mongoClient = new MongoClient(mongoSettings);

Answer 15 · 2023-08-09T21:34:57.000Z

@daxnet @shaliake

After using you solutions, my program is able to connect to DocumentDB, run the Hangfire migrations and even schedule jobs.
However, even using CheckQueuedJobsStrategy.Poll, the following exception is throwed after every poll interval:

MongoDB.Driver.MongoCommandException: Command find failed: Projection cannot have a mix of inclusion and exclusion..
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.ProcessResponse(ConnectionId connectionId, CommandMessage responseMessage)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol`1.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol`1 protocol, ICoreSession session, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable`1 commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action`1 postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer`1 resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.CommandOperationBase`1.ExecuteProtocol(IChannelHandle channel, ICoreSessionHandle session, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.ExecuteAttempt(RetryableReadContext context, Int32 attempt, Nullable`1 transactionNumber, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableReadOperationExecutor.Execute[TResult](IRetryableReadOperation`1 operation, RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.ReadCommandOperation`1.Execute(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.Execute(RetryableReadContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.FindOperation`1.Execute(IReadBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteReadOperation[TResult](IReadBinding binding, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, ReadPreference readPreference, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteReadOperation[TResult](IClientSessionHandle session, IReadOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.FindSync[TProjection](IClientSessionHandle session, FilterDefinition`1 filter, FindOptions`2 options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.<>c__DisplayClass46_0`1.<FindSync>b__0(IClientSessionHandle session)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSession[TResult](Func`2 func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.FindSync[TProjection](FilterDefinition`1 filter, FindOptions`2 options, CancellationToken cancellationToken)
   at MongoDB.Driver.FilteredMongoCollectionBase`1.FindSync[TProjection](FilterDefinition`1 filter, FindOptions`2 options, CancellationToken cancellationToken)
   at MongoDB.Driver.FindFluent`2.ToCursor(CancellationToken cancellationToken)
   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToList[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoConnection.GetFirstByLowestScoreFromSet(String key, Double fromScore, Double toScore, Int32 count)
   at Hangfire.Server.RecurringJobScheduler.<>c__DisplayClass18_0.<EnqueueNextRecurringJobs>b__0(IStorageConnection connection)
   at Hangfire.Server.RecurringJobScheduler.UseConnectionDistributedLock[T](JobStorage storage, Func`2 action)
   at Hangfire.Server.RecurringJobScheduler.Execute(BackgroundProcessContext context)
   at Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state)
   at Hangfire.Processing.BackgroundExecution.Run(Action`2 callback, Object state)

Did you have this problem and did you manage to solve it?

Answer 16 · 2023-10-14T15:54:00.000Z

Hi,

With my latest commit (#366), you should are able to set a flag, "SupportsCappedCollection", which will disable all logic related to capped collections.

I will release it in v1.9.11, please try it out and let me know if it works out

Thanks

Answer 17 · 2023-10-15T11:18:54.000Z

@gottscj

Amazing! It works just fine. I am able now to use Hangfire with Amazon DocumentDB without needing any tricks. Here is my final client configuration:

IServiceCollection services;
string connectionString;

// ...

var mongoUrl = new MongoUrl(connectionString);
var mongoSettings = MongoClientSettings.FromUrl(mongoUrl);
var mongoClient = new MongoClient(mongoSettings);
var mongoOptions = new MongoStorageOptions()
{
    MigrationOptions = new MongoMigrationOptions()
    {
        MigrationStrategy = new DropMongoMigrationStrategy(),
        BackupStrategy = new NoneMongoBackupStrategy(),
    },
    CheckConnection = true,
    CheckQueuedJobsStrategy = CheckQueuedJobsStrategy.Poll,
    QueuePollInterval = TimeSpan.FromSeconds(1),
    InvisibilityTimeout = TimeSpan.FromMinutes(30),
    SupportsCappedCollection = false
};
var mongoStorage = new MongoStorage(mongoClient, mongoUrl.DatabaseName, mongoOptions);

services.AddHangfire(configuration => configuration
    .UseActivator(new ContainerJobActivator())
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseSimpleAssemblyNameTypeSerializer()
    .UseRecommendedSerializerSettings()
    .UseConsole(new ConsoleOptions())
    .UseMongoStorage(mongoClient, mongoUrl.DatabaseName, mongoOptions));

JobStorage.Current = mongoStorage;

Note that my connection string already have retryWrites=false. If your connection string doesn't have this parameter, will must set it as false in your MongoClientSettings. This is required and already informed in DocumentDB docs.

Answer 18 · 2023-10-15T11:21:29.000Z

@jv-amorim,

Thats great! thanks for the feedback!