Provides Python code to reproduce model training, predictions, and heatmaps from the CheXNet paper that predicted 14 common diagnoses using convolutional neural networks in over 100,000 NIH chest x-rays.
Click on the launch binder button at the top of this README to launch a remote instance in your browser using binder. This requires no local configuration and lets you get started immediately. Open Explore_Predictions.ipynb, run all cells, and follow the instructions provided to review a selection of included chest x-rays from NIH.
To configure your own local instance (assumes Anaconda is installed; can be run on Amazon EC2 p2.xlarge instance if you do not have a GPU):
cd reproduce-chexnet
conda env create -f environment.yml
source postBuild
source activate reproduce-chexnet
This reproduction achieved average test set AUC 0.836 across 14 findings compared to 0.841 reported in original paper:
| retrained auc | chexnet auc | |
|---|---|---|
| label | ||
| Atelectasis | 0.8161 | 0.8094 |
| Cardiomegaly | 0.9105 | 0.9248 |
| Consolidation | 0.8008 | 0.7901 |
| Edema | 0.8979 | 0.8878 |
| Effusion | 0.8839 | 0.8638 |
| Emphysema | 0.9227 | 0.9371 |
| Fibrosis | 0.8293 | 0.8047 |
| Hernia | 0.9010 | 0.9164 |
| Infiltration | 0.7077 | 0.7345 |
| Mass | 0.8308 | 0.8676 |
| Nodule | 0.7748 | 0.7802 |
| Pleural_Thickening | 0.7860 | 0.8062 |
| Pneumonia | 0.7651 | 0.7680 |
| Pneumothorax | 0.8739 | 0.8887 |
aucs.csv: test AUCs of retrained model vs original ChexNet reported resultscheckpoint: saved model checkpointlog_train: log of train and val loss by epochpreds.csv: individual probabilities for each finding in each test set image predicted by retrained model
To explore the full dataset, download images from NIH (large, ~40gb compressed),
extract all tar.gz files to a single folder, and provide path as needed in code.
Please note: a GPU is required to train the model. You will encounter errors if you do not have a GPU available and CUDA installed and you attempt to retrain. With a GPU, you can retrain the model with retrain.py. Make sure you download the full NIH dataset before trying this. If you run out of GPU memory, reduce BATCH_SIZE from its default setting of 16.
If you do not have a GPU, but wish to retrain the model yourself to verify performance, you can replicate the model using Amazon EC2's p2.xlarge instance ($0.90/hr at time of writing) with an AMI that has CUDA installed (e.g. Deep Learning AMI (Ubuntu) Version 8.0 - ami-dff741a0). After creating and ssh-ing into the EC2 instance, follow the instructions in Getting Started above to configure your environment. If you have no experience with Amazon EC2, fast.ai's tutorial is a good place to start
I use SGD+momentum rather than the Adam optimizer as described in the original paper. I achieved better results with SGD+momentum, as has been reported in other work.
A sample of 621 test NIH chest x-rays enriched for positive pathology is included with the repo to faciliate immediate use and exploration in the Explore Predictions.ipynb notebook. The full NIH dataset is required for model retraining.
My goal in releasing this code is to increase transparency and replicability of deep learning models in radiology. I encourage you to use this code to start your own projects. If you do, please cite the repo:
author = {Zech, J.},
title = {reproduce-chexnet},
year = {2018},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/jrzech/reproduce-chexnet}}
}
With deep gratitude to researchers and developers at PyTorch, NIH, Stanford, and Project Jupyter, on whose generous work this project relies. With special thanks to Sasank Chilamkurthy, whose demonstration code was incorporated into this project. PyTorch is an incredible contribution to the research community.