No recovery from BrokerNotAvailableError: Broker not available (loadMetadataForTopics)

[deweyjose]

Questions?

Hi,

Today we saw a steady stream of errors for a sustained period of time (45min) on some of the nodejs processes in our service generated the following errors

Client Error handler:
BrokerNotAvailableError: Broker not available (loadMetadataForTopics)

Error from producer.send()
NotLeaderForPartition

Usually we see the affected nodes recover. Today however, we had to restart the affected node processes.

Are there any apis we can call on the client itself when it gets into this situation that can help expedite recovery? We know Kafka was fine, some nodes were not affected, we know that affected nodejs process started working once we restated them.

Environment

• Node version: 8.9.4
• Kafka-node version: 5.0.0
• Kafka version: 1.1.0

For specific cases also provide

• Number of Brokers: 20
• Number partitions for topic: 20

Include Sample Code to reproduce behavior

 async produce(topic: string, payloads: any[]) {
    return new Promise(resolve => {
      producer.send(
        [
          {
            topic,
            messages: JSON.stringify(payloads)
          }
        ],
        (err, data) => {
          if (err) {           
            logger.error(`KAFKA PRODUCER ERROR ${err}`);
          }
          resolve(data);
        }
      );
    });
  }

  async getProducer() {
    return (
      this.producer ||
      (this.producer = new Promise((resolve, reject) => {

        const client = new KafkaClient({
          kafkaHost: this.host,
          connectTimeout: 500
        });
        const producerOptions = {
          requireAcks: 1,
          ackTimeoutMs: 100
        };
        const newProducer = new HighLevelProducer(client, producerOptions);
        newProducer.on('error', error => {
          logger.error(`${error}.`);
          reject(error);
        });
        newProducer.on('ready', () => {
          logger.info(`Producer [${this.id}] created.`, { id: this.id });
          resolve(newProducer);
        });
      }))
    );
  }

Include output with Debug turned on

Thanks for your contribution!

[shlomi-ozon]

I had the same issue. Did you find anything?

[oys0629]

I had the same issue.
But I had found solution.

like it

try it!

I had the same issue. Did you find anything?

[shlomi-ozon]

@oys0629 I guess you want to do setInterval and not setTimeout, am I right?

[oys0629]

@oys0629 I guess you want to do setInterval and not setTimeout, am I right?

ah!
setTimeout, never mind.
It is not important!

[shlomi-ozon]

@oys0629 if it is not important so I guess I didn't understand it right. the refreshBrokerMetadata should calls after the broker comes back to be available (if it is true so the interval should be called so when the broker is available again we will call refresh)? or do I need to call it once at the start (so the setTimeout is the right one)?

[oys0629]

@oys0629 if it is not important so I guess I didn't understand it right. the refreshBrokerMetadata should calls after the broker comes back to be available (if it is true so the interval should be called so when the broker is available again we will call refresh)? or do I need to call it once at the start (so the setTimeout is the right one)?

ahhhhhh so sorry

I called the refreshBrokerMetadata after consumer error
and Because I don't want continuously call refreshBrokerMetadata, I used setTimeout

[shlomi-ozon]

hoo, now I understand it. thanks :]