phongca22/kafka-neo4j-stream

kafka-neo4j-stream

Install

• Kafka 2.1.0 and Zookeeper included https://www.apache.org/dyn/closer.cgi?path=/kafka/2.1.0/kafka_2.11-2.1.0.tgz
• Neo4j Community Server 3.4.12 https://neo4j.com/download-center/
• Neo4j streams plugin 3.4.1 https://github.com/neo4j-contrib/neo4j-streams/releases/tag/3.4.1

Setup

Kafka

• Start ZooKeeper server:

sh bin/zookeeper-server-start.sh config/zookeeper.properties

• Start Kafka server:

sh bin/kafka-server-start.sh config/server.properties

• Create Topic:

sh bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic user-track

• Create Comsumer:

sh bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic user-track --from-beginning

• Create multiple server:

Change broker.id and log.dirs in server.properties

Neo4j

streams.source.topic.nodes.<TOPIC_NAME>=<PATTERN>
streams.source.topic.relationships.<TOPIC_NAME>=<PATTERN>
streams.source.enable=<true/false, default=true>

• Update config/neo4j.conf with <TOPIC_NAME> and <PATTERN>:

kafka.zookeeper.connect=localhost:2181
kafka.bootstrap.servers=localhost:9092
kafka.acks=1
kafka.num.partitions=1
kafka.retries=2
kafka.batch.size=16384
kafka.buffer.memory=33554432
kafka.reindex.batch.size=1000
kafka.session.timeout.ms=15000
kafka.connection.timeout.ms=10000
kafka.replication=1
kafka.linger.ms=1
kafka.transactional.id=

streams.source.topic.nodes.user-track=action{*}

Copy neo4j-streams-3.4.1.jar into plugins folder

• Start neo4j server:

bin/neo4j console

• Access Neo4j Browser via http://localhost:7474. Connect to database with neo4j/neo4j and update new password later.

Create Web UI

• Install AngularJs and Angular Material:

<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.5/angular.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.5/angular-animate.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.5/angular-route.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.5/angular-aria.min.js"></script>
<script src="https://ajax.googleapis.com/ajax/libs/angularjs/1.5.5/angular-messages.min.js"></script>
<script src="https://cdn.gitcdn.link/cdn/angular/bower-material/v1.1.5/angular-material.js"></script>
<link type="text/css" rel="stylesheet" href="https://cdn.gitcdn.link/cdn/angular/bower-material/v1.1.5/angular-material.css" />

• Install Neo4j client:

<script src="js/neo4j-web.min.js"></script>

• Connect to neo4j server:

neo4j = neo4j.v1;
var driver = neo4j.driver('bolt://127.0.0.1:7687', neo4j.auth.basic('neo4j', '123456'));
var session = driver.session();

• Create a record:

var query = `CREATE (n:action {userId: '${USER_ID}', type: '${type}'})`;
var result = session.run(query);
result.then(function(data) {
    console.log(data);
}, function(e) {
    console.log(e)
});