Piecing social movements together by their junctures, demonstrating how individual social movements can be interpreted as components of an encompassing narrative.
data procuration preprocessing/...
- Wikipedia's List of social movements is used to seed the initial movement keywords in
socialList.txt - pytrends is used in
googleTrends.pyto retrieve Google Trends by keyword - the New York Times API is used in
nytimesTrends.pyto retrieve NYT articles by keyword
data processing preprocessing/...
- Pandas is used to merge and read CSV files aggregated through
mergegoogletrends.py - TextBlob is used in
nytimesfeatureExtraction.pyfor sentiment analysis of the aggregated NYT articles - DuckDuckGo is scraped in
nytimesfeatureExtraction.pyto textual description of the aggregated NYT articles - NLTK is used in
nytimesfeatureExtraction.pyandPageRankSummarizer.pyto tokenize strings - networkx is used in
PageRankSummarizer.pyfor treating the sentences as a network - sklearn's feature_extraction is used in
PageRankSummarizer.pyto extract the features from the text - scikit's PCA is used in
reduction.pyto reduce trends into two dimensions
data visualization js/...
- queue.js is used to read the data from the
data/*.jsonfiles - d3.js is used to parse and render the data
- jquery.selection.js is used to select text from the descriptions