Twitter community detection
The purpose of this project was to create who-follows-whom graph based on Twitter data and detect communities using most popular community detection algorithms. The outcome of this project is a graph that has over 100k vertices and over 4 mln edges with communities that were detected using the following algorithms:
- Modularity
- Infomap
- Label propagation
- Multilevel
This repository contains set of scripts for:
- crawling Twitter users data (basic info, followers and most popular hashtags)
- creating who-follows-whom graph based on crawled data
- detecting communities in the created graph
Detailed report available at Google Docs (Polish version only)
Visualizations
Visualizations were made using Gephi.
Whole graph after community detection with modularity:
Selected communities
"Hobby" community
Most popular hashtags in this community:
Multiple small communities
Most important nodes in the graph
Data and results
Crawled data and analysis results can be found at Google drive.
Repository structure
- detect_comunities.py - script for running community detection algorithms (Infomap, Label propagation, Multilevel)
- fetch_followers_scrapper.py - script for downloading followers by scrapping mobile version of Twitter using twint library, taking an initial user, download information about who they follow. Repeat recursively.
- fetch_hashtags_api.py - script for downloading hashtags for users using twitter API
- gen_graph_csv_edge_list.py - script for generating graph in form of edge list, saving it to .csv file
- get_graph_gml.py - script for generating graph in form of nodes list + edges list, saving it to .gml file
- draw_community_histogram.py - script for drawing histograms of communities sizes for different community detection algorithms
- draw_wordmap.py - script for drawing wordmaps of hashtags for small-size, middle-size and large-size communities for different community detection algorithms
- clean_duplicated_user_files.py - script for cleaning duplicated user files (race-condition between threads)
Authors
- Karol Bartyzel,
- Mieszko Makuch