geo-convo-server

Twitter conversation scanning and mapping tool, made with SpringBoot, Kotlin, OpenLayers3 and jQuery Mobile.

Uses websockets for continuous, low-overhead updates and geospatial bucketing for (somewhat) decluttered presentation.

This is POC-level code (comments/tests are on the to-do list).

Overview

Main UI on a typical Sunday morning:

Features

In terms of the UX:

Getting Started

Tools

New-ish Linux, Windows, or Mac OS.

Download/install Java 8 SDK.

Download/install Apache Maven.

Ensure both are in your working path.

Create Twitter App

You'll need to create a Twitter app (quick/free), then get app and user keys/secrets in order to build or run the software.

Twitter app management page with the important parts highlighted:

Create Mapbox Account

You'll need to create a Mapbox account (quick/free), then get a public token to run the software.

Mapbox account management page with the important parts highlighted:

Building

Perquisites

Update src/main/resources/application.yml

Fill out Twitter keys/secrets obtained above, per instructions at top of file.

Update src/main/resources/static/app.js

Fill out Mapbox token obtained above, per instructions at top of file.

(Actually) Build

In bash/cmd:

$ cd <path-to-source-folder>
$ mvn clean package

You may build with tests disabled if you'd rather not embed Twitter keys/secrets, as follows:

$ cd <path-to-source-folder>
$ mvn clean package -DskipTests

This generates a JAR file of the following form:

target/geo-convo-server-<version>-<snapshot-or-release>.jar

E.g. (used in examples below):

target/geo-convo-server-0.0.1-SNAPSHOT.jar

Deploying

Copy generated JAR file to wherever it's to be deployed.
Copy data folder to to the same location.
(Optionally) Copy application.yml, referenced above to the same location to override embedded settings (e.g., Twitter keys/secrets).

Running

In bash/cmd:

$ cd <path-to-run-folder>
$ java -jar geo-convo-server-0.0.1-SNAPSHOT.jar

It's fine to run from the source folder, as follows:

$ cd <path-to-source-folder>
$ java -jar target/geo-convo-server-0.0.1-SNAPSHOT.jar

Using

Navigate to the following URL:

http://localhost:8080

If all tokens/secrets have been set correctly, the map will be visible and conversations will begin to appear within a minute (expect >60/min, bandwidth depending).

Metrics

Metrics (e.g., total received Tweets) in JSON form may be obtained via the following URL:

http://localhost:8080/metrics

Username/password are specified in the application.yml file, referenced above.

Example:

{
  "mem": 1499474,
  "mem.free": 678824,
  "processors": 12,
  "instance.uptime": 771214,

// [...]

  "counter.services.publisher.points.published": 1654,
  "counter.services.links.tweets.new": 37434,
  "counter.services.status.request.enqueued": 9255,
  "counter.services.location.features.known": 1243,
  "counter.services.links.tweets.replies": 5605,
  "counter.services.trends.requests.trends": 77,
  "counter.services.links.tweets.source.location.known": 36604,
  "counter.services.links.tweets.target.location.unknown": 1014,
  "counter.services.locations.total.known": 12619,
  "counter.services.twitter.limit.waits": 1,
  "counter.services.status.request.loaded": 873,
  "counter.services.appcontroller.requests.startup": 2,
  "counter.services.links.tweets.source.location.unknown": 830,
  "counter.services.publisher.messages.published": 486,
  "counter.services.links.tweets.duplicate": 435,
  "counter.services.status.request.cached": 9731,
  "counter.services.links.total.known": 58,
  "counter.services.links.tweets.target.location.known": 336,
  "counter.services.publisher.ends.published": 200,
  "counter.services.publisher.links.published": 100,
  "counter.services.links.tweets.chained": 14,
  "counter.services.links.tweets.quotes.1": 4126,
  "counter.services.links.total.unknown": 374,
  "counter.services.trends.requests.places": 2
}

Geospatial Bucketing

In an effort to collate Tweets to/from approximate locations I used the Natural Earth Populated Places dataset and qGIS to generate the following Voronoi (Thiessen) polygons:

This represents ~2000 cities from the original dataset, selected by SCALERANK and RANK_MIN (population) values.

The LocationsService in the software matches Tweet locations against these polygons, then groups matched Tweets at the latitude/longitude of the related city.

These polygons are stored in a shapefile referenced in the application.yml file and may be modified to provide differently-distributed circles and link endpoints.

dnltsk/geo-convo-server