Stock insight engine is a stock market analyzer that mines user tweets on Twitter and performs sentiment analysis on them to show how much emotions on Twitter affect the stock prices. It is built upon tweepy, Elasticsearch and NLTK.
To be able to use stock insight engine to mine tweets from Twitter, there are a few steps need to be done. Namely,
-
Apply for a Twitter developer account here, then create a new Twitter application and generate your access credentials
-
Install Elasticsearch on your local machine. The following has been tested on Ubuntu 20.04
curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list sudo apt update sudo apt install elasticsearch
Upon successful installation, start the Elasticsearch service
sudo systemctl start elasticsearch
To test the installation is successful, do
curl -X GET 'http://localhost:9200'
which will you give you some snapshot information about the installed Elasticsearch.
-
Install Kibana. If your Elasticsearch installation is successful, then you just need to do
sudo apt install kibana
and rememeber
sudo systemctl start kibana
Once the Kibana service is started, you should be able to access the dashboard via
localhost:5601
. -
Install Python dependancies
It is recommended to use conda to manage your env. For example do
conda create -n stock python=3.8 conda activate stock pip install -r requirements.txt
-
Put your Twitter consumer key and access token in
config.py
. -
Edit the
nltk_tokens_required
andnltk_tokens_ignored
inconfig.py
to the Twitter feeds you want to mine.nltk_tokens_required
means the tweets being mined must contain at least one of the tokens innltk_tokens_required
before being added to Elasticsearch otherwise skipped, andnltk_tokens_ignored
means that if a tweet contains one of the ignored tokens then it will be skipped (not adding to Elasticsearch). The number required tokens can be set innltk_min_required
. -
To mine tweets talking about
Amazon
andJeff Bezos
, dopython get_tweet_sentiment.py -s AMZN -k 'Jeff Bezos',Bezos,Amazon,Alexa,'Blue Origin' --quiet
and to further follow URL links in tweets to perform sentiment analysis on that page
python get_tweet_sentiment.py -s AMZN -k 'Jeff Bezos',Bezos,Amazon,Alexa,'Blue Origin' -l --quiet
-
To get Amazon stock price from yahoo finance, do
python get_stockprice.py -s AMZN --quiet
Follow Kibana Visualization Tutorial to customize your data visualizations. Here I showcase mine as an illustration.
MIT