Piecing social movements together by their junctures, demonstrating how individual social movements can be interpreted as components of an encompassing narrative.
View the project here.
data procuration preprocessing/...
- Wikipedia's List of social movements is used to seed the initial movement keywords in
socialList.txt
- pytrends is used in
googleTrends.py
to retrieve Google Trends by keyword - the New York Times API is used in
nytimesTrends.py
to retrieve NYT articles by keyword
data processing preprocessing/...
- Pandas is used to merge and read CSV files aggregated through
mergegoogletrends.py
- TextBlob is used in
nytimesfeatureExtraction.py
for sentiment analysis of the aggregated NYT articles - DuckDuckGo is scraped in
nytimesfeatureExtraction.py
to textual description of the aggregated NYT articles - NLTK is used in
nytimesfeatureExtraction.py
andPageRankSummarizer.py
to tokenize strings - networkx is used in
PageRankSummarizer.py
for treating the sentences as a network - sklearn's feature_extraction is used in
PageRankSummarizer.py
to extract the features from the text - scikit's PCA is used in
reduction.py
to reduce trends into two dimensions
data visualization js/...
- queue.js is used to read the data from the
data/*.json
files - d3.js is used to parse and render the data
- jquery.selection.js is used to select text from the descriptions