reugn/go-streams

Kafka-go Extension (Windows Compatibility)

gavinservai opened this issue · 6 comments

gavinservai commented

This is probably not high priority for the vast majority of users, however the use of the confluent kafka client makes it so that go-streams is not windows compatible out of the box.

The suggestion I'd like to make is to change the extension to use something like:
https://github.com/segmentio/kafka-go

reugn commented

@gavinservai yes the flaws of confluent client are well known, including librdkafka dependency. Thought about migrating to sarama library. Would you like to contribute segmentio connector as well?

weldpua2008 commented

@reugn go run this https://github.com/reugn/go-streams/blob/master/examples/fs/main.go

$ go run stream/http-stream.go 
# pkg-config --cflags  -- rdkafka
Package rdkafka was not found in the pkg-config search path.
Perhaps you should add the directory containing `rdkafka.pc'
to the PKG_CONFIG_PATH environment variable
No package 'rdkafka' found
pkg-config: exit status 1

works only after:

$ brew install librdkafka pkg-config
$ export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/lib/pkgconfig/
$ go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka

OS:

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.15.3
BuildVersion:	19D76
reugn commented

@gavinservai, @weldpua2008 fixed in v0.4.0 with migration to Sarama kafka client.

  • ❤️1
gavinservai commented

I'm closing this issue as the main point of concern was windows support.

PaulisMatrix commented

@reugn, @gavinservai, any thoughts of adding support for segmentio-kafka go client?

Afaik this is the widely used kafka go connector and I was looking for the same but noticed go-streams is using IBM sarama client.

How do you feel about adding support for one more kafka client? Is this something you would be interested in?
Do lemme know!

reugn commented

@PaulisMatrix, feel free to open a PR adding a Segmentio Kafka connector under the same module. To avoid naming conflicts, consider adding "Segmentio" to the names in your implementation. The Sarama connector names will be aligned accordingly in a separate pull request.

  • 👍1