linkedin/brooklin

Missing few mirrormaker basics

configOps opened this issue · 3 comments

configOps commented

I have few issues :

  1. Topic metadata does not replicate to destination when brooklin datastream is setup. And neither it replicate modifications in same. What I can configure to do same ?
  2. How can I configure change in topic name in destination? Fo example, in source , I have a topic name first-topic and I have a datastream setup to mirror it in destination cluster with topic name as first-topic.global ?
upendra1024 commented

I have few issues :

  1. Topic metadata does not replicate to destination when brooklin datastream is setup. And neither it replicate modifications in same. What I can configure to do same ?
  2. How can I configure change in topic name in destination? Fo example, in source , I have a topic name first-topic and I have a datastream setup to mirror it in destination cluster with topic name as first-topic.global ?

As @configOps mention bove in 1st point, I found/have some more issues/doubt with KafkaMirror :

  1. topic partitions are not creating on destination. I used REST endpoint to create a datastream with metadata "system.destination.identityPartitioningEnabled": "true".
  2. after creating datastream, new messages only replicating to destination.
  3. if partition increased on source side then destination should also get this change.
  4. auto topic creation is disabled in our datacenter. so on destination need to create topic first before pushing data.
  5. Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old verssion?

This project is really nice, and lot of things we are learning from it.... Thanks a lot.

OneCricketeer commented

MirrorMaker 1 doesn't replicate topic metadata either (configs or partitions). It requires destination topic manually be made to match (which is easily done programmatically on your own).

I'm not aware of any rename feature in Brooklin, but I would like to think it does exist in some manner. What research have you done for the same?

Will the kafkaMirroring feature support kafka version 0.10.0.1? Is there any limitation due to this old version?

Can you please clarify your concerns here? What happens when you try it?

sgoncharov1 commented

Hi,
First - thank you for this project, it is really great.

As a follow up to this question i wanted to ask if there are plans to add the ability of changing topic names (to mirror topics to different topic names in destination cluster).
I tried lookin at the documentation and the code, but did not manage to find such ability.
We currently need to use additional clusters to overcome this issue, and it would be great if we can do this with configuration.