Pending jobs at the bottom of the pile stay in pending forever

[aloupfor]

First I would like to thanks for this module.
I hope someone will be able to help me with the following issue.

Magento version: 2.3.2-p2
Plugin MageMojo_Cron version: 1.3.2

Configuration:

path                             value
magemojo/cron/enabled            1
magemojo/cron/jobs               3
magemojo/cron/phpproc            /usr/local/bin/php
magemojo/cron/history            1
magemojo/cron/maxload            2
magemojo/cron/consumers_timeout  30

I encounter a problem where some jobs in cron_schedule never run/stay in pending forever.
I observed this with Wyomind CronScheduler module timeline.

With Xdebug, I am able to think that this is related to :

    // \MageMojo\Cron\Model\ResourceModel\Schedule::getPendingJobs
    public function getPendingJobs() {
      $connection = $this->getConnection();
      $select = $connection->select()
            ->from($this->getTable('cron_schedule'),['max(schedule_id) as schedule_id','job_code','count(*) as job_count'])
            ->where('status = ?', 'pending')
            ->where('scheduled_at < ?', date('Y-m-d H:i:s',time()))
            ->group('job_code');
      $result = $connection->fetchAll($select);
      return $result;
    }

     // \MageMojo\Cron\Model\Schedule::service
     public function service() {
      // ...

      print "Starting Service\n";
      #Loop until killed or heat death of the universe
      while (true) {
        $this->getRuntimeParameters();
        if ($this->cronenabled == 0) {
          exit;
        }

        // ...

        #Get pending jobs
        $pending = $this->resource->getPendingJobs();
        $maxConsumerMessages = intval($this->deploymentConfig->get('cron_consumers_runner/max_messages', 10000));
        $consumersTimeout =  intval($this->resource->getConfigValue('magemojo/cron/consumers_timeout',0,'default'));
        if (!$consumersTimeout) {
          $consumersTimeout = 0;
        }
        while (count($pending) && $this->canRunJobs($jobcount, $pending)) {
          $job = array_pop($pending);
          // ...
          }
        }

        // ...

        #Take a break
        sleep(5);
      }
    }

I observe that:

$pending = $this->resource->getPendingJobs();

will give a pile of pending jobs that will be selected for run with:

$job = array_pop($pending);

My problem is that I see pending jobs at the bottom of the pile that are never returned by: array_pop($pending) as $this->resource->getPendingJobs(); will "reset" the pile by erasing $pending and putting the same pending jobs at the bottom of the pile.

Is my understanding of the behavior good ?
What can I do to guarantee that all different pending jobs will be run ?

[gnuzealot]

In general what you're describing is that there are just too many pending jobs in the queue and crons are just backing up pending processing. You could increase the amount of simultaneous jobs that are waiting to be run. I would be more concerned that you have crons that are running constantly so you may want to examine what jobs are actually taking so long to run that they are blocking everything else.

[aloupfor]

Thank you for your answer.
I will try to do further investigation to understand why there is this behavior.