"No candidate servers found" when one member of a replica set is unpingable

[watmm]

Hi all,

I have a replica set with two data holding members and an arbiter. If any one of these members becomes unpingable the site won't work (neither reads nor writes) and i get the following:

PHP message: PHP Fatal error: Uncaught exception 'MongoConnectionException' with message 'No candidate servers found' in /var/app/current/vendor/doctrine/mongodb/lib/Doctrine/MongoDB/Cursor.php:550

Which is $cursor->getMongoCursor()->rewind();

If i test this via the mongo client on a shell i can read and write no problem.
I've tried clearing the Symfony cache, and restarting php-fpm cache while just two members were available, but it makes no difference.

Installed is the following:

php56.x86_64 5.6.26-1.128.amzn1
pecl/mongo 1.6.14
doctrine/mongodb 769aab3 - appears to be part of 1.4.0
doctrine/mongodb-odm 49f3f9a - (1 year, 2 months ago) - but unrelated to connections, right?
doctrine/mongodb-odm-bundle d66da72 - Same as above

The app config included

parameters:
    database_host: 'mongodb://IP1:27017,IP2:27017,IP3:27017/?replicaSet=SetName'
    database_name: DBName

The IPs remain fixed and are reassigned after a failover, but i tried hostnames in the clients /etc/aliases, also no joy.

Then i saw http://symfony.com/doc/current/bundles/DoctrineMongoDBBundle/config.html so i tried

doctrine_mongodb:
    connections:
        default:
            server: 'mongodb://IP1:27017,IP2:27017,IP3:27017'
            options:
                replicaSet:           SetName

Same behaviour.

Does retry_connect's default value of zero imply it retries forever?

I'm running out of ideas at this point. Any help would be much appreciated as i went to quite a bit of effort setting up AWS auto-scaling so each member comes back after a harware failure, identifies as they're supposed, discovers and joins the set.

But that side of things, and the negotiation between the DBs themselves seems to work as expected, this (Symfony/Doctrine) part is the last part of the puzzle.

If you need any more information just name it.

Regards,

--
Niall

[malarzm]

The real connection stuff is what driver is all about so it'd be good to get @jmikola's insight here

[jmikola]

In theory, "No candidate servers found" (thrown here) should only be thrown when the driver cannot find any servers matching the read preference. Since a primary read preference is used by default, you could expect this whenever a failover has occurred. I don't have an answer for why this would happen if just a secondary or arbiter is unreachable, and we're no longer actively developing the legacy driver.

You may be able to use MongoLog::setCallback() to collect information about what the driver is doing internally. That should tell us exactly how the servers are being considered for selection and ultimately why none are found usable. I'd suggest using the code example from the documentation and collecting its output into a file generated during the PHP request (or perhaps stderr during a CLI script).

The IPs remain fixed and are reassigned after a failover, but i tried hostnames in the clients /etc/aliases, also no joy.

IIRC, the hostnames in the URI are supposed to match those in the replica set configuration for the driver to function properly. If that's relevant, I expect we'll notice from the MongoLog output.

---

Ultimately, I would suggest migrating to the new extension, which will also require using alcaeus/mongo-php-adapter for Doctrine compatibility. The new driver has vastly improved connection monitoring (based on this spec) and should easily handle a missing replica set member.

[malarzm]

Closing since there was no further input from OP