The objective of this project is to match celebrity users with their respective tweets by making use of Semantic Textual Similarity.
This app finds similar Twitter users based on their tweets. It works in two ways -
- Get a list of the most similar celebrity Twitter accounts based on a predefined Twitter celebrity list (917
Twitter celebrity accounts). - Find similarities between two Twitter users based on their tweets.
View this project on kaggle - kaggle.com/code/ahmedshahriarsakib/twitter-celebrity-matcher-sbert-tweepy
💻 Streamlit App | ☑️ How it works? | 🛠️ Project Structure | 💾 Dataset | 📷 Screenshots | 🔎 Findings | ⬇️ Built With |
---|
Add the following to your .env
file to work with tweepy:
ACCESS_KEY=YOUR_ACCESS_KEY
ACCESS_SECRET=YOUR_ACCESS_SECRET
CONSUMER_KEY=YOUR_CONSUMER_KEY
CONSUMER_SECRET=YOUR_CONSUMER_SECRET
Set up a virtual environment (preferably python 3.8+, features such as walrus
operator was used) and run:
$ pip install -r requirements.txt
Check main.py file to run the streamlit app or API.
Build -
docker build -t twitter-celebrity-matcher .
Run -
docker run -p 8501:8501 twitter-celebrity-matcher
If you have CUDA enabled GPU, you can set up pytorch (for SBERT) with pip -
$ pip install --no-cache-dir --force-reinstall torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio===0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
Static type testing -
$ mypy main.py --ignore-missing-imports
The app file is located at app/app.py
.
To run the app, run the following command after installing necessary dependencies from requirements.txt
-
$ streamlit run main.py
will start the app at default port 8501
.
TwitterCelebrityMatcher
├── api
│ ├── __init__.py
│ ├── api.py
├── app
│ ├── __init__.py
│ ├── app.py
| ├── appdata.py
│ ├── apphome.py
│ ├── appinit.py
│ ├── utils.py
├── celebrity-listing
│ ├── Top-1000-Celebrity-Twitter-Accounts.csv
├── core
│ ├── __init__.py
│ ├── dataprep.py
│ ├── matcher.py
│ ├── scraper.py
│ ├── utils.py
├── twitter-celebrity-embed-data
│ ├── twitter-celebrity-embed-data.csv
└── .env
|
config.py
|
main.py
- api - fastApi script
- app - streamlit app scripts
- app.py - main app script
- appdata.py - dataclass contains configuration variables
- apphome.py - core functionalities
- appinit.py - helper script to initialize the core objects (
scraper
andmatcher
) - utils.py - utility functions (e.g. social sharing html script)
- core - contains 3 helper scripts
- dataprep.py - scripts for all preprocessing tasks ( tweet cleaning, embedding generation etc.)
- matcher.py - core functionalities to match twitter users
- matcher.py - twitter scraper helper script
- utils.py - utility functions
- config.py - all configuration variables
- main.py - run 3 apps - console scripts, streamlit app or API
utitlities
directory contains a standalone tweepy scraper scripts - tweepy_scraper.py
and emoticon proprocessing scripts (original from ekphrasis
module)
The celebrity tweets were collected using tweepy. After preprocessing the tweets, the tweets were embedded using sentence-transformers pretrained multilingual model - paraphrase-multilingual-MiniLM-L12-v2. The similarity score between tweets embeddings is calculated using cosine similarity.
This app -
- Takes a Twitter username
- Scrapes the tweets if it's unavailable
- Generate embeddings of the given Twitter account's tweets
- Calculate the mean embedding of the tweets (limitation ~3200 tweets)
- Finds the cosine similarity between
- Given user's tweets embeddings and celebrity tweets embeddings
- Given two user's tweets embeddings
- Returns the most similar celebrity Twitter accounts based on similarity score or just score between two users
Open Source GitHub Gist - Top-1000-Celebrity-Twitter-Accounts.csv
NB:
- There are some duplicates in the dataset. (986 after filtering)
- There are some unofficial Celebrity accounts (ex - twitter.com/sonunigam) with a very small amount of tweets. We can filter those users based on their tweet count. Here is a good research paper on this topic - 25 Tweets to Know You: A New Model to Predict Personality with Social Media
I uploaded the relevant datasets on kaggle. Feel free to use the datasets -
- Top 1000 Twitter Celebrity Accounts - Kaggle
- Top 1000 Twitter Celebrity Tweets And Embeddings - Kaggle
- Emoticon Dictionary - Kaggle
The final embeddings of all celebrity tweets -
The tweets were collected and saved as bytestring to preserve the text encoding (some tweets may contain complicated symbol/emojis etc.). So first the tweets are decoded as string and then some preprocessing techniques are applied.
Current pipeline :
- Removed hashtags, urls, and mentions
- Replaced emoticons with their textual representation (
ekphrasis
) - Replaced emoji with their textual representation (
demoji
package)
Citation
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "http://arxiv.org/abs/1908.10084",
}
Word embedding dimension : 384
Download SBERT pretrained models directly from here
- Most of the celebrities are from music, film or sports industry. The similarity results in these categories are very impressive.
- The multilingual model does a great job finding region/culture specific attributes. Below is an example of bollywood celebrities
- The number of authors in this celebrity list is comparatively lower than other dominating professions (actors, singers, footballers etc.). So in this example the first few results (first two from the results -> both are above
0.94
score ) were good but then the score drops. The drop rate is higher that the results from the dominating celebrity professions where the top score is around0.96+
and first 10 or 20 from the list have very close score. This will vary but with only 917 users the performance of the pretrained model is great in this case
- From the screenshot below, it's very clear that there are few people on the list who are very active in multiple categories.
- The closest of them all (for Bill Gates) is Richard Branson although the score is comparatively lower (
0.82
) than what we see in other categories. Considering how close their roles are (both are billionaires, investor, tech) this is a very positive result. Leonardo DiCaprio
is the second placeholder because he falls into Philanthropist category, same goes forTony Robbins
- Other famous billionaires such as -
anand mahindra
andRatan N. Tata
makes it to the top 10 - Most of the top people are either founders of large organizations, activists, philanthropists or authors. (Bill Gates published few books, so this could be another factor)
- The closest of them all (for Bill Gates) is Richard Branson although the score is comparatively lower (
NB: Due to space limitation, the screenshots above displays only top 10 results, try the app to view more top results which are very similar to the query
- With CUDA enabled GPU the app runs ~5x faster than CPU.
For ~3200 tweets in my rig (AMD 2600x, 1050ti, 768 CUDA cores) -
- With the current scraping scripts, it takes ~14-16 seconds to download tweets
- With
pandas
it takes ~8 seconds to preprocess the tweets - It takes ~6 seconds to generate embeddings from processed tweets
- Takes ~2h 5m in total to preprocess and generate embedding for all users' (917 total) tweets
demoji==1.1.0
ekphrasis==0.5.1
jupyter==1.0.0
python-dotenv==0.19.2
schedule==1.1.0
sentence-transformers==2.2.0
torch==1.11.0+cu113
torchaudio==0.11.0+cu113
torchvision==0.12.0+cu113
streamlit==1.7.0
tweepy==4.7.0
- Identifying Insomnia From Social Media Posts: Psycholinguistic Analyses of User Tweets
- An Effective BERT-Based Pipeline for Twitter Sentiment Analysis: A Case Study in Italian
- SBERT Multilingual Models
- Twitter Sentiment Analysis with Deep Learning using BERT and Hugging Face
- Twitter Sentiment Analysis with Twint and Textblob