Replication factor: 3 larger than available brokers: 1.

[Toruitas]

Nifty tool! Thanks for this.

I've encountered a problem when trying to run the first example in your article, in a single-broker single-zookeeper scenario.

python3 kafka_docker_composer.py -b 1 -z 1

This will create a docker-compose.yml without emitting errors.

However, running it will eventually lead to repeats of this exception org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. each time any new topic creation is attempted. This will also occur just running it without any connections, as the confluent telemetry metric itself will cause the exception to appear:

[2024-04-23 17:00:05,153] INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='_confluent-telemetry-metrics', numPartitions=12, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='max.message.bytes', value='10485760'), CreateableTopicConfig(name='message.timestamp.type', value='CreateTime'), CreateableTopicConfig(name='min.insync.replicas', value='1'), CreateableTopicConfig(name='retention.ms', value='259200000'), CreateableTopicConfig(name='segment.ms', value='14400000'), CreateableTopicConfig(name='retention.bytes', value='-1')], linkName=null, mirrorTopic=null, sourceTopicId=AAAAAAAAAAAAAAAAAAAAAA, mirrorStartOffsetSpec=-9223372036854775808, mirrorStartOffsets=[]) (kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

Steps to produce:

1. git clone and cd into this repo
2. python3 kafka_docker_composer.py -b 1 -z 1
3. docker compose up -d
4. docker logs -f kafka-1
5. Wait a little while, it doesn't happen right away, but it always happens

On the other hand, when using the Controller instead of Zookeeper, this happens instead.

org.apache.kafka.common.errors.InvalidReplicationFactorException: Unable to replicate the partition 3 time(s): The target replication factor of 3 cannot be reached because only 1 broker(s) are registered.

[sknop]

I assume you mean these entries in the log:

kafka-1 | [2024-04-30 13:08:39,135] ERROR Error checking or creating telemetry topic _confluent-telemetry-metrics (io.confluent.telemetry.exporter.kafka.KafkaExporter)
kafka-1 | org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

The telemetry metrics are non-critical, and the broker will continue to run, but I agree the error is annoying. I will investigate, thanks for pointing this out.

[sknop]

Update your clone and try again, please.

[Toruitas]

Yep! That did the trick. Thank you!