This is an extended project for Building Communitarianism in Diverse Neighbourhoods : A Perspective of Communication Infrastructure Theory (honours thesis, under supervision of Duyvendak, J.W. & Hurenkamp, M. in 2021). The original paper investigats how hyperlocal social media Nextdoor facilitates community building through the lens of communication infrastructure theory (CIT, Ball-Rokeach et al., 2001) by the means of manual qualitative content analysis.
This extended project is to automate the data collection process on Nextdoor to make the procedure more efficient and allow for a multi-method approach to analyse Nextdoor.
Latest update: 15/01/2023
The scraper only works for the Dutch version of NextDoor right now!
If you wish to contribute to the scraper, please contact me on Twitter.
Alpha testing! There are few to-dos to make the scraping more efficient and complete.
The scraping of the following items can be automated:
- Reactions counts
- Expand comments (multilevel)
- Comments count and comment text
- Body texts
- Post and comment ID
- Formatted date
- Scrolling by date (with exceptions, see to-dos)
- Get post user_name and comment user_name
- Get neighbourhood data of posts' and comments' authors
- Not urgent, but can also automate login processes and allow for multiple neighbourhoods to be examined (this is not within the scope of the project)
- Scrolling by date may be at times faulty as the feed on Nextdoor is not solely ranked by post date, but also other factors. If we are to scroll by date, we may stumble upon a 2-year-old post in the middle of newer posts. This is to be resolved.
- English version of Nextdoor.
- Multimedia content
- Get reactions categorisations (sad, likes, laughs, etc.)