mbc-harvester
A script harvesting Mazowiecka Biblietka Cyfrowa collection"Warszawa w ilustracji prasowej XIX w.". Files are then uploaded on Wikimedia Commons.
It can crawl any e-library that is powered by OAI-compatible software, for instance dLibra.
Importer is executed via GitHub Actions cross twice a week - at 7:00 AM every Monday and Thursday.
Install
Set up Python env.
virtualenv env -ppython38
. env/bin/activate
pip install -r requirements.txt
Set up account that will used for uploads.
$ cat user-password.py
('commons', 'commons', 'Mazovian_Digital_Library_Upload', 'XXX')
Run
python harvest.py
GitHub Actions
You need to set up the following secrets in order to run the importer as a cron-triggered action:
HTTP_PROXY(e.g.socks5://example.com:12345)PYWIKIBOT_USERNAMEPYWIKIBOT_PASSWORD