A simple MQTT to Apache Kafka Connector - ARM64 Edition

This is a fork of the original that is specifically geared toward ARM64 compatibility.

Usage

Prerequisites

Java 17 or higher

Note that you should be able to use Java 8 or 11 with no problem, but the Docker container is built with 17 and all testing/development has been done with 17.

Configuration

Edit the application.properties file
kafka.host = [HOSTNAME|IP] - IP THROUGH WHICH THE KAFKA BROKER WILL BE REACHED
kafka.port = [INTEGER]
kafka.client.id = [STRING]
mqtt.host = [HOSTNAME|IP] - IP THROUGH WHICH THE MQTT BROKER WILL BE REACHED
mqtt.port = [INTEGER]
mqtt.client.id = [STRING]
mqtt.qos = [INTEGER] - Quality of service for MQTT - Allowed[0,1,2]
topic.mapping = [list_of_topic_mapping] - This is a semicolon-separated list that describes how topics are routed from MQTT to Kafka - (Separators >>> (between kafka/MQTT mappings) and ; (between pairs))
- Example: mqttTopicA>>>kafkaTopicA;mqttTopicB>>>kafkaTopicB;mqttTopicC>>>kafkaTopicC

More info about wildcards in MQTT topics can be found here: HiveMQ's MQTT Topics & Best Practices (pt 5)

How to build

Run: mvn clean install

How to run

Place the jar with dependencies and your edited application.properties-file in the same directory
Open a bash or CMD in the same directory as the .jar
Run: java -jar kafka-mqtt-connector-0.1.0-SNAPSHOT-jar-with-dependencies.jar

Usage with Docker

Docker Hub original and this ARM64 edition (WORK IN PROGRESS)
Run (edit environment variables to your needs!):

Most basic Docker build command:

docker build -t mikeoertli/kafka-mqtt-connector:0.1.0 -t mikeoertli/kafka-mqtt-connector:latest .

docker run -d -t -i \
  -e KAFKA_HOST='localhost' \
  -e KAFKA_PORT=9092 \
  -e KAFKA_CLIENT_ID='testing-kafka-producer-1' \
  -e MQTT_HOST='localhost' \
  -e MQTT_PORT=1883 \
  -e MQTT_CLIENT_ID='mqtt-client-1' \
  -e MQTT_QOS=2 \
  -e TOPIC_MAPPING='iot/event/#>>>iot_event;iot/status/#>>>iot_status;iot/tasking/#>>>iot_tasking'  \
  --name kafka-mqtt-connector \
  mikeoertli/kafka-mqtt-connector:latest

Usage with Docker-Compose

See docker-compose-examples

Standalone

Stand-alone container
docker-compose.yml
Run: docker-compose up -d

Full Stack

Full Stack (mqtt-broker (via mosquitto), zookeeper, kafka-broker, kafka-mqtt-connector)
docker-compose.yml
env.list
Place docker-compose.yml and env.list in the same directory
Edit env.list to your needs!
Run: docker-compose up -d

License

See LICENSE file for License

mikeoertli/simple-kafka-mqtt-connector