NOTE: This repo will be migrated to github.com/saitogroup/geotweets Ariel Kalinowski and Trevor Owens 8/24/2016 About: ---------------------------------------------------------------------- This library is composed of several tools for scraping geolocated tweets and visualizing data gleaned from these tweets. Geo-tag your tweets! -------------------- We rely on geo-tagged tweets. Please allow your location to be seen when tweeting, especially when using this application! You can modify this by logging into your main twitter account and under "Security and Privacy" check the box next to "Tweet location". THANKS! Install: ---------------------------------------------------------------------- git, python 2.7.X, pip Python packages required: tweepy, nltk, matplotlib, geopy, argparse, json On Windows: upgrade powershell (you may still have unicode problems when printing to command line) $ python -m pip install For each required package listed above run: $ pip install <package> Now we need some data, so we’ll use the nltk downloader Run a python shell from the command line: $ python $ import nltk $ nltk.download() On main page, highlight book, click download and that should be it... These are the exact packages from nltk that are required in case you want less data: 1) under corpora -> highlight stopwords 2) under corpora -> highlight treebank 3) under all packages -> highlight punkt 4) under models -> highlight averaged-perceptron-tagger This created a folder called “nltk_data” in your home folder which is used by the program Navigate to the folder where you want getweets to be git clone https://github.com/owenst/geotweets.git #THIS WILL CHANGE get consumerkeyandsecret (see below) and put that in the folder cd into folder run sample.py from the command line (see below) Consumer Key and Secret: ---------------------------------------------------------------------- The program looks for a file in the geotweets folder called consumerkeyandsecret This should have at least 2 lines, with the consumer key on the first line, the secret (the longer one) on the next and then (for streaming and posting) 2 more lines. An access token on the 3rd and the access token secret on the 4th. You can get these by going to https://apps.twitter.com in a web browser and creating an app. Then hit the button to create access tokens. You may have to set the app permissions to "read and write" if you want to use this to send tweets on your behalf. After creating the app, copy the 4 alphanumeric keys into a blank file called "consumerkeyandsecret" as described above and put this file in your "geotweets" folder. TOOLS: ---------------------------------------------------------------------- sample: ------- One tool, called 'sample' allows you to scrape and save up to 100 geolocated tweets in batch form. You can optionally search within this set for specific words or hash tags and visualize the top word frequency. See sample.py for details or from command line run: $ python sample.py --help $ python sample.py --doc USAGE : $ python sample.py [-h][-d][-v][-f FILENAME][-o OUTPUT][-vis] real time visualizer: --------------------- Another tool, called 'real_time_vis' uses the previous tool to create a word frequency distribution chart which can grow and change in near real time as more tweets are grabbed. See real_time_vis.py for details or from the command line run: $ python real_time_vis.py --help $ python real_time_vis.py --doc USAGE : $ python real_time_vis.py [-h][-d][-f FILENAME][-n NUMBER][-s][-a ADDRESS] Both files use a parameter file with geolocation and search terms. See params.txt for an example. You may have to adjust your PYTHONPATH variable to run the program from the command line. Otherwise, using the python interpreter you can run it. scan_and_respond ---------------- This tool scans tweets and asks the user to verify them before sending a tweet response. The relevant tweets are also saved to a JSON file. This requires write access, which means the consumerkeyandsecret file must contain all 4 lines. usage: scan_and_respond.py [-h] [-d] [-f FILENAME] [-a ADDRESS] [-o OUTPUT] write ----- This program classifies tweets into phrase types and produces a JSON array containing these, called phrases.json. It uses parameters from params.txt. This requires quite a bit of processing time, which can be reduced by using a lower "count". suggest_bot ----------- This is a robotically assisted poetry engine. The user can create poems using a large supplied word corpus or use their own. It can also add words to the corpus from the twitter-sphere using the search option. HELP: ---------------------------------------------------------------------- All programs can be run from the command line (a.k.a. terminal in OS X). $ The $ symbol represents a command prompt. By typing $ python <program_name> -h you will get help on the various command line tool options. By typing $ python <program_name> -d you will get the programs documentation string EXAMPLES: ---------------------------------------------------------------------- Grabbing geo-located tweets using paramter file params.txt (default), print to command line and write to output.txt (default): $ python sample.py --verbose Visualizing the data, using params.txt (default): $ python real_time_vis.py Streaming real time data to create a word frequency chart using a local address: $ python real_time_vis.py -a "175 5th Avenue NYC" -s UTILITIES: ---------------------------------------------------------------------- These modules contain methods to assist the "tools" listed above: tweeter.py utils.py geo_converter.py geosearchclass.py streamer.py The below two modules run unit tests: test_real_time_vis test_write