Social Feed Manager (SFM) harvests social media data from multiple platforms' public APIs to help archivists, librarians, and researchers to build social media collections. More information about the project itself. This is a re-architected version of an earlier Social Feed Manager which has been in use at GWU Libraries since 2012.
Social Feed Manager allows users to:
- define collections comprising sets of targeted accounts, keywords, and other search strategies appropriate to different platforms.
- authorize SFM to harvest data from platforms on the user's behalf.
- view collection information and metadata about harvests.
- extract, filter, and export the datasets to formats appropriate to the user's work.
The full Social Feed Manager is made up of several component repositories:
- sfm-ui: (this repo) User interface and datastore for collection and harvest information.
- sfm-flickr-harvester: A harvester for Flickr.
- sfm-twitter-harvester: A harvester for Twitter.
- sfm-weibo-harvester: A harvester for Sina Weibo.
- sfm-tumblr-harvester: A harvester for Tumblr.
- sfm-web-harvester: A harvester for web resources using Heritrix.
- sfm-utils: Utilities to support SFM.
- sfm-docker: Docker configuration for deploying SFM.
- sfm-elk: Proof-of-concept analytics dashboard using the ELK stack.
- Provides a user interface to set up Collection Sets, Collections, and Seeds
- Provides Django admin views to administer Credentials, Groups, and other model entities.
- Publishes harvest.start messages for flickr collections. The app schedules harvest.start messages for publication when the user updates an existing, active Collection.
- Includes a scheduler which uses apscheduler to schedule publication of harvest.start messages.
- Binds to
harvest.status.*(.*)messages and creates a Harvest object (visible in the admin views) for each harvest status message received. The message consumer is started via thestartconsumermanagement command.
Behind the scenes, SFM uses a set of carefully managed processes to harvest and and store this data, recording its actions in detail.
- Documentation: http://sfm.readthedocs.org
- Full project information: http://gwu-libraries.github.com/sfm-ui
- Project updates: Follow @SocialFeedMgr on Twitter
- Discussion: sfm-dev
- Tickets: sfm-ui is used for all ticketing.
Tickets / pull requests / discussion are welcome.
cd sfm
./manage.py test --settings=sfm.settings.test_settings
Social Feed Manager is supported by a grant from the National Historical Publications & Records Commission.