Kafka POC

Validar os conceitos de segurança para o Apache Kafka.

Pontos de atenção:

Authentication using SSL;
Encryption of data in-flight using SSL / TLS

Go Client - Used for tests

go get github.com/birdayz/kaf/cmd/kaf
echo 'source <(kaf completion zsh)' >> ~/.zshrc

Without Security

Running Kafka, Zookeeper and kafdrop

docker-compose -f docker-compose.yaml up

Creating topic and message

kaf -b localhost:9092 topic create "without-security"
echo "my message" | kaf -b localhost:9092 produce without-security

http://localhost:19000/

With Security - SSL

Creating certificates

cd certificates
./docker-kafka-ssl-certs.sh
cd ..
chmod 644 certificates/certs/*

Change docker-compose-ssl.yaml

Caso for geerado um novo certificado, o mesmo deve ser atualizado no docker-compose-ssl.yaml

cat certificates/certs/docker.kafka.client.truststore.jks | base64
cat certificates/certs/docker.kafka.server.keystore.jks | base64

Running Kafka, Zookeeper and kafdrop - SSL Client

export KAFKA_SSL_SECRETS_DIR=$(pwd)/certificates/certs
docker-compose rm -f
docker-compose -f docker-compose-ssl.yaml up --force-recreate

Test with logstash

Running with docker

docker run --rm -it \
    -e "xpack.security.enabled=false" \
    -e "xpack.monitoring.enabled=false" \
    -v $(pwd)/logstash/pipeline/:/usr/share/logstash/pipeline/ \
    -v $(pwd)/certificates/certs/:/etc/kafka/secrets \
    --net=poc-kafka_broker-kafka \
    docker.elastic.co/logstash/logstash:6.8.12

Problema encontrados

Existe um bug na validação do certificação, conforme o link logstash-plugins/logstash-integration-kafka#52, por isso estamos utilizando uma versão antiga do logstash. O problema ocorre somente no plugin de output.

Security

https://github.com/addoddelgrossi/kafka-test/blob/master/security.md

addodelgrossi/kafka-test