/tweetsniff

Grab a Twitter user timeline for further processing (storing to Elasticsearch, highligthing, etc)

Primary LanguagePythonGNU General Public License v3.0GPL-3.0

tweetsniff.py

Grabs a user's Twitter feed and tweets with specific keywords via the Twitter API for local processing (storing to ElasticSearch, regex highlighting, etc)

Installation

The following Python modules are required:

  • twitter
  • elasticsearch
  • termcolor
  • dateutil
  • hashlib
  • urllib
  • httplib

Install them with: pip install

Valid Twitter API key & token are required, see https://apps.twitter.com/

Usage

usage: tweetsniff.py [-h] [-c CONFIG]

Display a Tweet feed

optional arguments:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        configuration file (default: /etc/tweetsniff.conf)

Expand URLs

  • Enable in config with "process_urls: True"
  • urllib2.url provides the same features but gets sometimes rejected by sites (403)
  • URLs have to be part of the twitter URL object, otherwise they do not get recognized
  • not yet in writeCEFEvent
  • ES object contains URL, expanded URL, MD5 and SHA1 of the (expanded) URL

Todo

  • Add more statistics