czimbortibor/rss-atom-feedscraper

Download the images and some relevant metadata from RSS and Atom feeds

overview

feed_list.json contains a list of RSS and Atom feeds which the script parses and downloads the found images in the tags and some relevant metadata. The metadata, list of feeds and the path to the images are then stored in MongoDB. scheduler.py schedules an automatic run of the script in the background

packages

• feedparser
• pymongo
• urllib3
• Beautiful Soup
• schedule

usage

run the script in every k minutes

python3 scheduler.py k

or

nohup /usr/bin/python3 scheduler.py k &