Tweets

This repository contains a clojure project supported by SQLite that persists and allows for the querying of tweets matching predefined hashtags.

More specifically, the service does the following:

Collects tweets matching a set of predefined hashtags from the twitter streaming api.
Persists content from these tweets in a SQLite database.
Exposes a server featuring rudimentary authentication that allows the querying of these tweets.

Use

Requirements

Running in development requires several dependencies be installed on the local machine:

leiningen
java version 8 or later
sqlite

Alternatively, docker may be used to manage dependencies and run the project.

Configuration

The twitter streaming support requires setting some environment variables:

export CONSUMER_KEY=<...>
export CONSUMER_SECRET=<...>
export ACCESS_TOKEN=<...>
export ACCESS_TOKEN_SECRET=<...>

Running the project starts the collection of tweets by the project’s consumer component. Tweets matching a set of predefined hashtags (“#tech”, “#photography”, and “#funny”) are persisted in a local sqlite database. The querying of this database is supported by the tweets server, which implements jwt authentication.

Database migration

Schema migration is managed using a Clojure library called [~ragtime~](https://github.com/weavejester/ragtime), which is integrated into the Clojure project definition.

To perform schema migration, run the following command from the root of the project directory:

lein migrate

Such schema migrations can be undone using the complementary command:

lein rollback

Development

Start the project in development using evoking ~leiningen~ from the command-line:

lein run

In the context above, the server is available at url: http//localhost:3666/. By default, the REST service runs on port 3666 on the local machine. However, the user may choose an alternative port by supplying a valid port at the command-line:

lein run 4000 # run server on http//localhost:4000/

Production

The project may alternatively by as a compiled java jar:

lein uberjar # compile
java -jar <path-to-jar>

Docker

This project features docker support.

Please be sure to pass twitter authentication environment variables into the container (using an env-file is recommended).

docker build -t tweets .
docker run --env-file <path-to-env-file-with-twitter-auth> --rm -p 3666:3666 tweets

Kill by searching and destroying tweets docker process:

docker ps
... # shows processes; identify hash
docker kill <hash>

Communicating with the Service

Please see the api description document included in docs/api.org.

Please note sample json messages are included in the samples/ directory at the root of this repository.

Testing

Tests are run from the command-line as follows:

lein test

Issues and Limitations:

Please see github issue tracker.

ds2643/tweets