/nifi-streaming-examples

Collection of examples integrating NiFi with stream process frameworks.

Primary LanguageJavaApache License 2.0Apache-2.0

Apache NiFi Streaming Examples

Collection of examples integrating NiFi with stream process frameworks.

Initial Setup

  • Download the latest Apache NiFi release

  • Extract the tar and create two instances of NiFi:

      
  tar xzvf nifi-1.0.0-bin.tar.gz
  mv nifi-1.0.0 nifi-edge
  tar xzvf nifi-1.0.0-bin.tar.gz
  mv nifi-1.0.0 nifi-core

  • Configure the edge instance by editing nifi-edge/conf/nifi.properties and setting the following properties:

      
  nifi.remote.input.socket.port=7088
  nifi.remote.input.secure=false
  nifi.web.http.port=7080

  • Configure the core instance by editing nifi-core/conf/nifi.properties and setting the following properties:

     
  nifi.remote.input.socket.port=8088
  nifi.remote.input.secure=false
  nifi.web.http.port=8080

  • Start both instances

     
  ./nifi-core/bin/nifi.sh start
  ./nifi-edge/bin/nifi.sh start

  • Open the UI for both instances in a browser

     
  http://localhost:7080/nifi/
  http://localhost:8080/nifi/

  • Setup initial dictionary files

    
  mkdir nifi-edge/data
  mkdir nifi-edge/data/dictionary
  mkdir nifi-core/data
  mkdir nifi-core/data/dictionary

  • In each of the above dictionary directories, create a file called levels.txt with the content:


    ERROR
    WARN

  • Import nifi-streaming-examples/templates/nifi-log-example-edge.xml into the the edge instance (http://localhost:7080/nifi)

  • Import nifi-streaming-examples/templates/nifi-log-example-core.xml into the the core instance (http://localhost:8080/nifi)

  • Start everything on the core instance (http://localhost:8080/nifi) Image

  • To start sending logs, starting everything on the edge instance (http://localhost:8080/nifi) EXCEPT the TailFile processor, the "Generate Test Logs" process group will send fake log messages Image

  • To tail a real file, stop the "Generate Test Logs" process group, configure TailFile to point to your log file of choice, and start the TailFile processor

Flink - WindowLogLevelCount - Setup

  • For local testing, run a standalone Flink streaming job

  cd nifi-flink-examples
  mvn clean package -PWindowLogLevelCount
  java -jar target/nifi-flink-examples-0.0.1-SNAPSHOT.jar

Apex - LogLevelApplicationRunner - Setup

  • For local testing, run LogLevelApplicationRunner from your favorite IDE:

  nifi-apex-examples/src/test/java/nifi/apex/examples/logs/LogLevelApplicationRunner.java

Storm - LogLevelCountTopology - Setup

  • For local testing, run a standalone local Storm topology

  cd nifi-storm-examples
  mvn clean package -PLogLevelCountTopology
  java -jar target/nifi-storm-examples-0.0.1-SNAPSHOT.jar