confluentinc/kafka-rest

Is there a plan to add support for multiple clusters in the future?

punit-kulal opened this issue · 3 comments

punit-kulal commented
  • The v3 API path seems to have semantic flexibility to add support for multiple clusters. However, currently kafka-rest can act as a REST proxy for only one kafka cluster.
  • Is there any future roadmap to add support for multiple cluster in kafka-rest.
Veryfirefly commented

Yes, this feature is very helpful. When using one of the proxy nodes to get cluster information, we want to be able to return a list of available nodes for the entire Kafka-REST cluster, which is necessary for fault tolerance of clients using the proxy to write messages.
I want to add nodes to the following data nodes like this:

{
    "kind": "KafkaClusterList",
    "metadata": {
        "self": "http://localhost:9391/v3/clusters",
        "next": null
    },
    "data": [
        {
            "kind": "KafkaCluster",
            "metadata": {
                "self": "http://localhost:9391/v3/clusters/cluster-1",
                "resource_name": "crn:///kafka=cluster-1"
            },
            "cluster_id": "cluster-1",
            "nodes": [
                "http://192.168.1.1:9391/",
                "http://192.168.1.2:9391"
            ],
            "controller": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers/1"
            },
            "acls": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/acls"
            },
            "brokers": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers"
            },
            "broker_configs": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/broker-configs"
            },
            "consumer_groups": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/consumer-groups"
            },
            "topics": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics"
            },
            "partition_reassignments": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics/-/partitions/-/reassignment"
            }
        }
    ]
}
Veryfirefly commented

Yes, this feature is very helpful. When using one of the proxy nodes to get cluster information, we want to be able to return a list of available nodes for the entire Kafka-REST cluster, which is necessary for fault tolerance of clients using the proxy to write messages. I want to add nodes to the following data nodes like this:

{
    "kind": "KafkaClusterList",
    "metadata": {
        "self": "http://localhost:9391/v3/clusters",
        "next": null
    },
    "data": [
        {
            "kind": "KafkaCluster",
            "metadata": {
                "self": "http://localhost:9391/v3/clusters/cluster-1",
                "resource_name": "crn:///kafka=cluster-1"
            },
            "cluster_id": "cluster-1",
            "nodes": [
                "http://192.168.1.1:9391/",
                "http://192.168.1.2:9391"
            ],
            "controller": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers/1"
            },
            "acls": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/acls"
            },
            "brokers": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers"
            },
            "broker_configs": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/broker-configs"
            },
            "consumer_groups": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/consumer-groups"
            },
            "topics": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics"
            },
            "partition_reassignments": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics/-/partitions/-/reassignment"
            }
        }
    ]
}

Sorry, the previous data format is wrong, this is a new data format, put nodes information in ClusterDataList, I don't know what you think.

{
    "kind": "KafkaClusterList",
    "metadata": {
        "self": "http://localhost:9391/v3/clusters",
        "next": null
    },
    "nodes": {
        "self": "http://localhost:8082/",
        "available_node": [
            {
                "host": "http://192.168.1.1:8082/",
                "healthy": true,
                "weight": 1.0,
                "enabled": true,
                "metadata": {
                    "foo1": "bar1"
                }
            },
            {
                "host": "http://192.168.1.2:8082/",
                "healthy": true,
                "weight": 1.0,
                "enabled": true,
                "metadata": {
                    "foo2": "bar2"
                }
            }
        ]
    },
    "data": [
        {
            "kind": "KafkaCluster",
            "metadata": {
                "self": "http://localhost:9391/v3/clusters/cluster-1",
                "resource_name": "crn:///kafka=cluster-1"
            },
            "cluster_id": "cluster-1",
            "controller": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers/1"
            },
            "acls": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/acls"
            },
            "brokers": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/brokers"
            },
            "broker_configs": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/broker-configs"
            },
            "consumer_groups": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/consumer-groups"
            },
            "topics": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics"
            },
            "partition_reassignments": {
                "related": "http://localhost:9391/v3/clusters/cluster-1/topics/-/partitions/-/reassignment"
            }
        }
    ]
}
punit-kulal commented

Could you point me to any open apache wiki issue for adding multiple cluster support. Would be happy to contribute