/TwitterDWSDemo

Twitter DWS Demo for Barcelona

TwitterDWSDemo

Colophon

Version: 0.1 : March 16, 2019 : Draft

What

This demo is to show how NiFI can serve as the Flow platform to connect an Edge Device or an external Data distributor to a local PostGreSQL (or any other database).

The key here is simplicity and speed and its designed to be run from a laptop.

Demo Pre-requisites

  1. Install NiFI

  2. Install PostGresql

  3. Download the latest version of PostGreSQL JDBC Driver

Demo Steps

  1. Check NiFI and PostGres is running.

Note
 
NiFI by default runs on 8080 , change the property nifi.web.http.port in nifi.properties inside the conf directory to assign a different port.
 
sudo netstat -ltnp | grep "LISTEN"

  1. Open NiFI and import the template from here : TwitterToLocalDB.xml

  2. Login to PostGres

    sudo -i -u postgres
psql
-- Change the default password
\password
-- List the current databases
\l

  3. Create a new database

    CREATE DATABASE dwsdemo;
-- connect to the new database
\connect dwsdemo
-- check if the db is present
\l
-- list all relations (tables etc.)
\dt+

  4. Create a new table for storing "processed" tweets

    CREATE TABLE public.tweets (tweet_id  bigint, unixtime bigint, created_time timestamp, displayname varchar(255), msg text);
    -- describe a table
\d public.tweets
--

  5. The JDBC URL for connectivity

    jdbc:postgresql://localhost:5432/dwsdemo

  6. Check the tweets that have come in the last 12 hours

    SELECT created_time, displayname, msg FROM public.tweets WHERE created_time >= NOW() - interval '12 hour';