[FEATURE] Pulsar to Kafka topic direct integration

[ghirpara]

We have a use case where we are running a Kafka cluster with about 500+ topics, each of which is consumed or written to by one or more microservices. We would like to transition the messaging plane to Pulsar, but we want to gradually transition the system so that some microservices can continue to use Kafka while we transition others. To that end, we need to be able to read and write from both Pulsar and Kafka topics through a Pulsar client.

We had thought about using Pulsar IO Kafka connectors of various flavors, but most of them seem to not scale to more than a handful of topics at a time. That is, we would have to create separate sink and source connectors for each topic and that's a lot of management overhead for us, since you're talking about creating a 1000+ connectors if we did one sink and one source for each of the topics.

Ideally we would deploy the connector at the client side, so that the mapping happens as close to the client as possible and does not overload the broker configuration management. If we can't do that, then a single connector source and single connector sink, that accepts a topic map of some sort, where we could specify the entire topic mapping at once (either as absolute names or regex-style mapping) for all the topics we want to map would work as well.