Add additional docs for working with kafka in docker

Question

Add additional docs for working with kafka in docker

Closed this issue 5 years ago · 9 comments

I'm new to Kafka and I had a hard time understanding how to interact with the cluster that I had setup in docker-compose running on my mac. The way the CLI tools are packaged with Kafka was not intuitive to me.

I found kafka-shell when I was looking for something to make connecting and managing my local (docker) cluster a bit easier; I think it's fantastic. It would be great for my use case if you could include some documentation on what a good config might look like for working with kafka in docker for development. I think this is probably a fairly common usage.

For instance:

I found that prefixing the command with a docker exec as descried in the CONFIGURATION.md didn't quite work because the image that I'm using does not have the Kafka bin directory exported to the PATH - is there a specific image I should use for these shell scripts? I'm using wurstmeister/kafka

Answer 1 · 2019-10-26T16:59:45.000Z

Hi @blankenshipz,

I haven't used this docker image, so this is interesting. Based on that docker image alone, I don't know if you could use kafka-shell in its current state to run commands via the docker image. It does look like the files are on the path, but they end in .sh which kafka-shell currently does not support. For example, docker run -it wurstmeister/kafka kafka-topics.sh works.

My docker example is based on the Confluent docker images (confluentinc/cp-kafka), which do have the files on the path. A simple way to utilize their docker images can be found here (with ZooKeeper as well):

https://github.com/simplesteph/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml

If you do that, then it will work properly. If you'd prefer to use the wurstmeister/kafka image, you could:

Extend that Dockerfile to make it work
Run another daemon container (see my confluent-tools repository as an example)
I could extend kafka-shell with some additional configurations to add in .sh

Let me know if those options work for you. I'll probably add in an additional configuration for the .sh extension this weekend.

Answer 2 · 2019-11-23T17:43:55.000Z

Hi @blankenshipz, have you been able to work around this issue? Anything else you need help with?

Answer 3 · 2019-11-26T13:35:36.000Z

Hi @devshawn; I haven't had a chance to try it yet but switching to the image you're using should work - I'll reopen this issue if It doesn't.

Thanks!

Answer 4 · 2019-12-06T01:48:18.000Z

Just wanted to mention that I also stumbled here trying to use this tool with a kafka/zookeeper deployment into docker kubernetes (using cp-kafka). I get the same error for most commands, for example: sh: kafka-topics: command not found.

I am essentially using the example you posted above, but with docker stack deploy --orchestrator kubernetes

Answer 5 · 2019-12-06T02:37:57.000Z

@solarmosaic-kflorence - thanks for the report.

I'd be happy to assist you, but I'd need to know a bit more about your setup and how your config is set up. Feel free to open a new issue 😄. I haven't used docker stack before but I've used this tool with the kubectl prefix to run commands on running k8s containers.

Answer 6 · 2019-12-07T00:52:46.000Z

hey @devshawn -- here is an example you can use:

kafka-stack.yaml:

version: '3.3'
services:
  # https://hub.docker.com/r/bitnami/zookeeper
  zookeeper1:
    image: bitnami/zookeeper:3.4.13
    hostname: zookeeper1
    ports:
      - "2181:2181"
    environment:
      ALLOW_ANONYMOUS_LOGIN: "yes"
    volumes:
      - zookeeper1-data:/bitnami/zookeeper

  # https://hub.docker.com/r/bitnami/kafka
  kafka1:
    image: bitnami/kafka:2.2.1
    hostname: kafka1
    ports:
      - "9092:9092"
    environment:
      ALLOW_ANONYMOUS_LOGIN: "yes"
      ALLOW_PLAINTEXT_LISTENER: "yes"
      KAFKA_CFG_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_CFG_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_CFG_LISTENERS: INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092
      KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CFG_ZOOKEEPER_CONNECT: "zookeeper1:2181"
    volumes:
      - kafka1-data:/bitnami/kafka

deploy with docker stack deploy --orchestrator kubernetes -c kafka-stack.yaml kafka-stack

I can now add the following to my ~/.kafka-shell/config: command_prefix: kubectl exec svc/kafka1 --

But when I try to execute commands, I get:

> kafka-topics --list
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"kafka-topics\": executable file not found in $PATH": unknown
command terminated with exit code 126

Looks like it is for the same reason, commands are exposed in this docker image with the .sh extension.

PS: using cp-kafka the commands were fixed by using the kubectl exec svc/kafka1 -- command_prefix option -- would be nice if it were possible to set this option on execution of kafka-shell instead of having to set it in configuration (so it's easier to provide developer documentation without having people modifying their local configuration), but it's not a big deal.

Answer 7 · 2019-12-07T02:38:42.000Z

FWIW it looks like the .sh commands are defacto: https://kafka.apache.org/quickstart

Answer 8 · 2019-12-07T02:47:39.000Z

That's super helpful, thanks @solarmosaic-kflorence!

I'll add a config for using .sh commands and push a release tomorrow. :)

Hope that'll help!

Answer 9 · 2019-12-09T18:59:57.000Z

@devshawn since we are slightly off-topic on this ticket, I have created #18 to track. It would probably be useful to add documentation for this use-case at the same time as well.