shotover/shotover-proxy

kafka flexible versions support

Closed this issue · 0 comments

rukai commented

Currently we hardcode our header versions, currently this seems fine as our rdkafka driver is a little old but it will cause issues when a more modern driver is used.
The correct way to version headers is not included in the kafka protocol documentation.

I found this which is very enlightening:
https://stackoverflow.com/questions/71794315/questions-about-the-documentation-of-the-kafka-protocol
They link to source generator, but their link is broken, I've fixed it here: https://github.com/apache/kafka/blob/698319b8e2c1f6cb574f339eede6f2a5b1919b55/generator/src/main/java/org/apache/kafka/message/ApiMessageTypeGenerator.java#L324

I then found that the header versioning disaster was introduced by something called flexible versions:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Tagged+Fields
Each message type has its own flexibleVersions range: https://github.com/apache/kafka/blob/db6beb9b0302ce99d0a768d90a125f6a084aa48b/clients/src/main/resources/common/message/FindCoordinatorResponse.json#L28

I think the best way to support this would be to add support to kafka-protocol crate.
It should generate functions:

fn request_header_version(api_key: u16, api_version: u16) -> u16 {
   /// return 1 or 2 depending on if the api_key+api_version is a flexibleVersion
}

fn response_header_version(api_key: u16, api_version: u16) -> u16 {
   /// return 0 or 1 depending on if the api_key+api_version is a flexibleVersion
}