Support add/remove broker to existing cluster

[k-wall]

As an author of integration tests for Kroxylicious, I am interested in testing use-cases where the number of brokers in the cluster is changed at runtime, i.e. brokers added or broker removed. I want to be able to control this programmatically from within the test.

I am imagining an extension to io.kroxylicious.testing.kafka.api.KafkaCluster interface that lets me:

#addBroker() => nodeId
#removeBroker(int nodeId)

I want the feature to work in both kraft and zookeeper mode. In kraft mode, I only need the ability to add a broker in the role of broker. It would be reasonable if the API rejected attempts to remove the node that is the controller.

I don't need the #removeBroker API to do any partition reassignment (to evacuate the broker).

[SamBarker]

The request makes sense to me but I have some questions:

1. Should the extension prevent all brokers being removed?
2. Can and should it support brokers changing from ZK to Kraft
3. Should we actually expose addBroker and addController?
4. When should the add method return[1]? Once the broker is started? Once the broker is part of describe cluster? Once the broker is known to every peer?
5. Similar to 4 but for remove. What is the signal to return [1].

[1] Equally if we return a future when does the future complete.

[k-wall]

The request makes sense to me but I have some questions:

1. Should the extension prevent all brokers being removed?

Yes

2. Can and should it support brokers changing from ZK to Kraft

3. Should we actually expose addBroker and addController?

I don't think these use-cases are interesting from a Kroxy pov. Controllers are invisible to clients, so Kroxy, doesn't care.
However, in the junit ext may get traction outside kroxy so designing the API in such a way to support future extensions make sense. I wouldn't actually do the addController work now.

4. When should the add method return[1]? Once the broker is started? Once the broker is part of describe cluster? Once the broker is known to every peer?
5. Similar to 4 but for remove. What is the signal to return [1].

I've had no experience of actually growing/shrinking a cluster, but waiting for the new broker to be reported present (describe cluster) by all brokers, or gone by those that remains sounds sensible to me.

[1] Equally if we return a future when does the future complete.

[k-wall]

@franvila having this feature soon would help the work on kroxylicious/kroxylicious#265 and kroxylicious/kroxylicious#271. Just wondering if you have capacity to pick it up soon?

[franvila]

Sure, I will try to take it today