Collection of examples integrating NiFi with stream process frameworks.
-
Download the latest Apache NiFi release
-
Extract the tar and create two instances of NiFi:
tar xzvf nifi--bin.tar.gz mv nifi- nifi-edge tar xzvf nifi--bin.tar.gz mv nifi- nifi-core -
Configure the edge instance by editing nifi-edge/conf/nifi.properties and setting the following properties:
nifi.remote.input.socket.port=7088 nifi.remote.input.secure=false nifi.web.http.port=7080 -
Configure the core instance by editing nifi-core/conf/nifi.properties and setting the following properties:
nifi.remote.input.socket.port=8088 nifi.remote.input.secure=false nifi.web.http.port=8080 -
Start both instances
./nifi-core/bin/nifi.sh start ./nifi-edge/bin/nifi.sh start -
Open the UI for both instances in a browser
http://localhost:7080/nifi/ http://localhost:8080/nifi/ -
Setup initial dictionary files
mkdir nifi-edge/data mkdir nifi-edge/data/dictionary mkdir nifi-core/data mkdir nifi-core/data/dictionary -
In each of the above dictionary directories, create a file called levels.txt with the content:
ERROR
WARN
-
Import nifi-streaming-examples/templates/nifi-log-example-edge.xml into the the edge instance (http://localhost:7080/nifi)
-
Import nifi-streaming-examples/templates/nifi-log-example-core.xml into the the core instance (http://localhost:8080/nifi)
-
Start everything on the core instance (http://localhost:8080/nifi)
-
To start sending logs, starting everything on the edge instance (http://localhost:8080/nifi) EXCEPT the TailFile processor, the "Generate Test Logs" process group will send fake log messages
-
To tail a real file, stop the "Generate Test Logs" process group, configure TailFile to point to your log file of choice, and start the TailFile processor
- Run the Flink streaming job
cd nifi-flink-examples
mvn clean package -PWindowLogLevelCount
java -jar target/nifi-flink-examples-0.0.1-SNAPSHOT.jar