This tool provides a method for scraping through NCBI's PMC publications and retrieving (downloading) the figures from open access and publicly available publications.
Please note that as of August 20, 2024, scraping figures from PMC (PubMed Central) publications is no longer permitted under their Copyright Notice (https://www.ncbi.nlm.nih.gov/pmc/about/copyright/). Attempting to do so may result in a 403 error and an IP ban from NIH/PMC. Users are fully responsible for ensuring their use of this tool complies with current PMC policies and applicable laws. Use at your own risk.
This code is maintained for educational and historical reference purposes only. While it remains functional, it is not intended for use in scraping PMC data, as this is no longer allowed under current policies. The tool was originally developed for academic research, but due to these policy changes, it is not recommended for use in this capacity anymore.
- Node.js >= 20
- RAM >= 4GB
- Internet connection with greater than 7mb/s download speed
If you would like to run or modify the publication figure web scraping tool locally, clone the repository with git by running the following command:
git clone https://github.com/AlexJSully/Publication-Figures-Web-Scraping.git
Then run npm install
then npm start
. This tool runs within your node environment. On Windows, this script needs to run in an administrator mode.
The images are downloaded then downloaded locally within this containing directory under src/data/figures/{species}/{PMC ID}.
If you would like to run against commercial use publications, you will need to download oa_comm_use_file.list.txt
from ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/ then run npm run process
. Once that is done, set index.js init function to true (await init(true);
)
The publication figure scraper will resume where you last left off. If you would like to reset the scraper, empty species-pmid-list.json, data-retrieved.json and data-empty-pubs.json to contain only just an empty JSON object ({}
).
If you would like to add more species support for publications to be scraped, add the species to species.json and then run npm start
. Currently, this JSON includes species' common aliases which are not currently being used but may be useful in the future. If you would like to scrape a single species, then change speciesList
in index.js to an array of species scientific name(s) to scrape. For example: speciesList = ['Arabidopsis thaliana']; // Or whatever species name(s) you would like to scrape
. Currently, it is set to scrape all species within the species.json file.
If in the instance that you do not have an internet connection/speed greater than 7mb/s, you will need to change all the Axios request timeouts in data-retrieval.js to a value of at least half of your speed (e.g. down speed of 10mb/s, set timeout to 5s).
We aim to make this tool as perfect as possible but unfortunately, there may be some unforeseen bugs. If you manage to find one that is not here, feel free to create a bug report so we can fix it.
- None at the moment... Help us find some!
Please read CONTRIBUTING.md for more details.
This project is currently in maintenance mode. This means that:
- Only critical bug fixes and security updates will be addressed.
- New feature requests are unlikely to be implemented.
If you want to support my work, you can through the following methods: