Add support for increasing number of partitions in a topic

[jurschel]

Given the following statefile:

#Orders
topics:
  orders:
    partitions: 5
    replication: 3

Desired state:
When I update partitions to 6 I expect the tool to increase the number of partitions.
Actual State:
Executing apply...
[SUCCESS] There are no necessary changes; the actual state matches the desired state.

Kafka Version - 2.1.1-cp1
Kafka-gitops version - 0.2.7

[devshawn]

That's correct, changing the number of partitions or changing the replication factor is not yet supported.

I should probably make that clearer in the docs.

[jurschel]

So what changes are supported?

[devshawn]

Changes to the number of partitions in a topic is a rare occurrence in my experience, and usually, something you want to have planned out (e.g. if you have keyed messages, you're now going to be distributing keys differently across the partitions).

For topics, you can currently change its configs. For example:

topics:
  test-topic:
    partitions: 6
    replication: 3
    configs:
      retention.ms: 60000

Changing a config:

topics:
  test-topic:
    partitions: 6
    replication: 3
    configs:
      retention.ms: 30000

Most of the supported changes are for services/ACLs. However, I agree that this should be supported, and I'm working on adding this. 😄

[jurschel]

Funny you should mention that...

~ [TOPIC] orders
~ retention.ms: 32536000000

[ERROR] Error thrown when attempting to update a Kafka topic config:
org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not support INCREMENTAL_ALTER_CONFIGS

[devshawn]

Ah, yes, you can run into this when using an older broker version. We rely on the IncrementalAlterConfigs API due to how the old AlterConfigs API worked:

The AlterConfigs RPC gives users a way to alter the configuration of a topic, broker, or other resource. However, the new configuration replaces any existing configuration. This makes AlterConfigs unusable in cases where the client does not know the full existing configuration before making the modification. This makes AlterConfigs less efficient, since it means that the client needs to call DescribeConfigs to retrieve the existing configuration before making any modification. It also introduces the possibility of "lost updates" when multiple clients enter a read-modify-write cycle around the same time.

Even worse, in some cases, the client may be unable to discover the existing configuration. "Sensitive" fields are not returned by DescribeConfigs. However, they will be overwritten by AlterConfigs.

I've thought about including a fallback option using the old AlterConfigs API but I found it was quite messy; it seemed like it was easy to accidentally overwrite or lose (default or user-defined) topic configurations.

Support for this API was released in Kafka 2.3.0. More information can be found here.

[jrevillard]

Hi @devshawn,

This would be indeed a very nice feature. Would you already I idea if this could be implemented soon or not ?

Best,
Jerome

[devshawn]

Hi @jrevillard, I have the code written for it, just need to finish writing tests. I hope to have it finished and released by the end of the week. Thanks!

[jrevillard]

Hi @devshawn .. I developped it this week end .... :-) ... I'm completing the tests also :-) ;.. should I continue ?

Also, I added increasing and decreasing replicas

[irisgve]

👋🏻 Hello @devshawn wondering when a release for the fix for this issue is going to be released? Thanks!