This repository provides a set of Python tools to perform multimodal learning with tabular data. It contains the code used in our study:
"Integration of clinical, pathological, radiological, and transcriptomic data improves the prediction of first-line immunotherapy outcome in metastatic non-small cell lung cancer"
- lifelines (>= 0.27.4)
- matplotlib (>= 3.5.1)
- numpy (>= 1.21.5)
- pandas (= 1.5.3)
- pyyaml (>= 6.0)
- scikit-learn (>= 1.2.0)
- scikit-survival (>= 0.21.0)
- seaborn (=0.13.0)
- shap (>= 0.41.0)
- xgboost (>= 1.7.5)
Clone the repository:
git clone https://github.com/ncaptier/multipit.git
-
Early and late fusion implementations: 4 estimators compatible with scikit-learn and scikit-surv to fuse several tabular modalities in a single multimodal model.
multipit.multi_model.EarlyFusionClassifierandmultipit.multi_model.LateFusionClassifierfor binary classification.multipit.multi_model.EarlyFusionSurvivalandmultipit.multi_model.LateFusionSurvivalfor survival prediction.
-
Scripts to reproduce the experiments of our study: Scripts to perform late fusion an early fusion of clinical, radiomic, pathomic and transcriptomic features with a repeated cross-validation scheme. Scripts to compute and collect the SHAP values associated with each unimodal predictive model (see scripts folder).
-
Plotting functions and notebooks to reproduce the figures of our study: several functions to plot and compare the performances of different multimodal combinations as well as to display feature importance with SHAP values.
We also provide another Github repository, named deep-mulitpit with a Pytorch implementation of an end-to-end integration strategy with attention weights, inspired by Vangurie et al, 2022.
Modify the configurations in .yaml config files (in config/ subfolder) then run the following command in your terminal:
python latefusion.py -c config/config_latefusion.yaml -s path/to/results/folder
python collect_shap_survival.py -c config/config_latefusion_survival.yaml -s path/to/results/folder
Warning: For Windows OS paths must be written with '\' or '\' separators (instead of '/').
Note: In order to modify more deeply the loading of the data or the predictive pipelines, please update the PredictionTask class in the file _init_scripts.py.
In the examples folder we provide a brief example on how to slightly modify the scripts and codes from our original experiments to perform multimodal learning for the prediction of Overall Survival from clinical and RNA-seq data extracted from TCGA (i.e., stage III and IV TCGA-LUAD and TCGA-LUSC samples).
We simply updated the PredictionTask class in a new file _init_scripts_tcga.py to load TGCA data and build predictive pipelines.
Note: clinical and transcriptomic data extracted for 201 stage III/IV TCGA patients (i.e., LUAD or LUSC) are available in the data folder.
This repository was created as part of the PhD project of Nicolas Captier in the Computational Systems Biologie of Cancer group and the Laboratory of Translational Imaging in Oncology (LITO) of Institut Curie.