/storm-kafka-0.8-plus-test

Simple test project for storm-kafka-0.8-plus

Primary LanguageJavaApache License 2.0Apache-2.0

storm-kafka-0.8-plus-test

Test project for storm-kafka-0.8-plus based on information provided in:

Also contains an attempt at a sample implementation of trident state based on [Hazelcast] (http://www.hazelcast.com/)

##Environment setup with Docker

If you are using a Mac follow the instructions here to setup a docker environment.

##Build for running locally:

  • mvn clean package

##Build for running on a Storm cluster:

  • mvn clean package -P cluster

##Running the test topologies locally

  • java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.trident.SentenceAggregationTopology <kafkaZookeeper>
  • java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.KafkaSpoutTestTopology <kafkaZookeeper>
  • java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.TestTopologyStaticHosts

##Running the test topologies on a storm cluster

  • storm jar target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.trident.SentenceAggregationTopology <kafkaZookeeper> sentences <dockerIp>
  • storm jar target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.KafkaSpoutTestTopology <kafkaZookeeper> sentences <dockerIp>

The Storm UI will be available under: http://<dockerIp>:8080/

The Logviewer will be available under: http://<dockerIp>:8000/ e.g. http://<dockerIp>:8000/log?file=supervisor.log

##Producing data

To feed the topologies with data, start the StormProducer (built in local mode)

  • java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.tools.StormProducer <dockerIp>:<kafkaPort>

Alternatively use the kafka console producer from within the kafka shell (see above)

  • $KAFKA_HOME/bin/kafka-console-producer.sh --topic=storm-sentence --broker-list=<dockerIp>:<kafkaPort>

##Consuming data

To run a DRPC query, start the DrpcClient (built in local mode)

  • java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.tools.DrpcClient <dockerIp> 3772