/Omics-Imaging-Fusion

Models

We provide the corresponding resources according to the modules of our paper, including data preprocessing, feature processing, association and fusion, datasets and performance comparison.

Alt text

  • Figure 1. Flowchart of omics-imaging fusion

1. Data Peprocessing

1.1 Tools and methods for omics data preprocessing

1.2 Tools and methods for imaging data preprocessing

-MRI

-fMRI

  • SPM : Analysis of brain imaging data sequences.
  • AFNI: Analysis and display of multiple MRI modalities.
  • FSL: A comprehensive library of analysis tools for brain imaging data.
  • REST: A toolkit to calculate FC, ReHo, ALFF, fALFF, Gragner causality and perform statistical analysis.
  • DPARSF: A convenient plug-in software within DPABI.
  • DPABI: a GNU/GPL toolbox for Data Processing & Analysis of Brain Imaging Histopathological images.

-Histopathological images

2.Feature Processing

2.1 Traditional feature extraction methods for omics data in omics-imaging fusion

-Gene expression (mRNA expression)

-DNA methylation

-SNPs

2.2 Traditional feature extraction methods for imaging data in omics-imaging fusion

-MRI

-CT

-Histopathological images

3. Association and Fusion

3.1 Omics-imaging Association

-For real-valued data

-For category data

  1. X. Zhan, J. Cheng, Z. Huang, Z. Han, B. Helm, X. Liu, J. Zhang, T.-F. Wang, D. Ni, and K. Huang, “Correlation Analysis of Histopathology and Proteogenomics Data for Breast Cancer,” Molecular & Cellular Proteomics, vol. 18, 2019. code: https://github.com/xiaohuizhan/cor_image_omics_BRCA dataset: TCGA
  2. J. Cheng, J. Zhang, Y. Han, X. Wang, X. Ye, Y. Meng, A. Parwani, Z. Han, Q. Feng, and K. Huang, “Integrative Analysis of Histopathological Images and Genomic Data Predicts Clear Cell Renal Cell Carcinoma Prognosis,” Cancer Research, vol. 77, no. 21, 2017. code: https://github.com/chengjun583/image-mRNA-prognostic-model

-Network-based methods

3.2 Feature Fusion

3.2.1 Fusion through association

-CCA-based methods

  1. Y. Bai, Z. Pascal, W. Hu, V. D. Calhoun, and Y.-P. Wang, “Biomarker identification through integrating fmri and epigenetics,” IEEE Transactions on Biomedical Engineering, vol. 67, no. 4, pp. 1186–1196, 2019.
  2. L. Du, K. Liu, X. Yao, S. L. Risacher, L. Guo, A. J. Saykin, and L. Shen, “Diagnosis Status Guided Brain Imaging Genetics Via Integrated Regression And Sparse Canonical Correlation Analysis,” in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), 2019, pp. 356–359.
  1. S.-P. Deng, W. Hu, V. D. Calhoun, and Y.-P. Wang, “Integrating imaging genomic data in the quest for biomarkers of schizophrenia disease,” IEEE/ACM transactions on computational biology and bioinformatics, vol. 15, no. 5, pp. 1480–1491, 2017.
  2. W. Hu, D. Lin, V. D. Calhoun, and Y.-p. Wang, “Integration of SNPs-FMRI-methylation data with sparse multi-CCA for schizophrenia study,” in 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2016, pp. 3310–3313.

-ICA-based methods

  1. J. Liu, G. Pearlson, A. Windemuth, G. Ruano, N. I. Perrone- Bizzozero, and V. Calhoun, “Combining fmri and snp data to investigate connections between brain function and genetics using parallel ica,” Human brain mapping, vol. 30, no. 1, pp. 241–255, 2009.
  2. S. A. Meda, B. Narayanan, J. Liu, N. I. Perrone-Bizzozero, M. C. Stevens, V. D. Calhoun, D. C. Glahn, L. Shen, S. L. Risacher, A. J. Saykin et al., “A large scale multivariate parallel ica method reveals novel imaging–genetic relationships for alzheimer’s disease in the adni cohort,” Neuroimage, vol. 60, no. 3, pp. 1608–1621, 2012.

-NMF-based methods

3.2.2 Direct Fusion

-Basic methods

  • Vector concatenation
  1. X. Zhu, J. Yao, X. Luo, G. Xiao, Y. Xie, A. Gazdar, and J. Huang, “Lung cancer survival prediction from pathological images and genetic data: An integration study,” in 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), 2016, pp. 1173– 1176.
  2. K.-H. Yu, G. J. Berry, D. L. Rubin, C. RA˜ ©, R. B. Altman, and M. Snyder, “Association of Omics Features with Histopathology Patterns in Lung Adenocarcinoma,” Cell Systems, vol. 5, no. 6, pp. 620–627.e3, 2017.
  3. H. Zeng, L. Chen, M. Zhang, Y. Luo, and X. Ma, “Integration of histopathological images and multi-dimensional omics analyses predicts molecular features and prognosis in high-grade serous ovarian cancer,” Gynecologic Oncology, 2021.

-Sparse representation

  1. H. Cao, J. Duan, D. Lin, Y. Y. Shugart, V. Calhoun, and Y.-P.Wang, “Sparse representation based biomarker selection for schizophrenia with integrated analysis of fMRI and SNPs,” NeuroImage, pp. 220–228, 2014.
  2. H. Cao, J. Duan, D. Lin, V. Calhoun, and Y.-P. Wang, “Integrating fMRI and SNP data for biomarker identification for schizophrenia with a sparse representation based variable selection method,” BMC Medical Genomics, vol. 6, no. 3, p. S2, 2013.

-MKL-based methods

-DNNs-based methods

-Others

3.3 Decision Fusion

3.3.1 Ensemble learning

  1. H. Li, H. Zeng, L. Chen, Q. Liao, J. Ji, and X. Ma, “Integrative Analysis of Histopathological Images and Genomic Data in Colon Adenocarcinoma,” 2020.
  2. A. El-Nabawy, N. El-Bendary, and N. A. Belal, “A feature-fusion framework of clinical, genomics, and histopathological data for METABRIC breast cancer subtype classification,” Applied Soft Computing, vol. 91, p. 106238, 2020.
  3. L. Chen, H. Zeng, M. Zhang, Y. Luo, and X. Ma, “Histopathological image and gene expression pattern analysis for predicting molecular features and prognosis of head and neck squamous cell carcinoma,” Cancer Medicine, 2021.

4. Datasets

5. Performance comparison

5.1 Papers using ADNI dataset

  1. simpleMKL Z. Zhang, H. Huang, D. Shen, and A. D. N. Initiative, “Integrative analysis of multi-dimensional imaging genomics data for Alzheimer’s disease prediction,” Frontiers in aging neuroscience, vol. 6, p. 260, 2014. | - | ADNI
  2. MKL J. Peng, L. An, X. Zhu, Y. Jin, and D. Shen, “Structured sparse kernel learning for imaging genetics based Alzheimer’s disease diagnosis,” in International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016, pp. 70–78.
  3. Deep Model J. Venugopalan, L. Tong, H. R. Hassanzadeh, and M. D. Wang, “Multimodal deep learning models for early detection of alzheimer’s disease stage,” Scientific reports, vol. 11, no. 1, pp. 1–13, 2021.
  4. Multilevel Logistic Regression with Structured Penalties P. Lu, O. Colliot, A. D. N. Initiative et al., “Multilevel modeling with structured penalties for classification from imaging genetics data,” in Graphs in Biomedical Image Analysis, Computational Anatomy and Imaging Genetics. Springer, 2017, pp. 230–240.
  5. ss-HMFSS L. An, E. Adeli, M. Liu, J. Zhang, S.-W. Lee, and D. Shen, “A hierarchical feature and sample selection framework and its application for alzheimer’s disease diagnosis,” Scientific reports, vol. 7, no. 1, pp. 1–11, 2017.
  6. CaMCCo A. Singanamalli, H. Wang, and A. Madabhushi, “Cascaded Multi-view Canonical Correlation (CaMCCo) for Early Diagnosis of Alzheimer’s Disease via Fusion of Clinical, Imaging and Omic Features,” Scientific Reports, vol. 7, no. 1, p. 8137, 2017.
  7. DSCCA J. Yan, S. L. Risacher, K. Nho, A. J. Saykin, and L. Shen, “Identification of discriminative imaging proteomics associations in alzheimer’s disease via a novel sparse correlation model,” in Biocomputing 2017, 2016, pp. 94–104.

5.2 Papers using TCGA dataset

  1. A deep survival learning model C. Huang, A. Zhang, and G. Xiao, “Deep Integrative Analysis for Survival Prediction,” in Biocomputing 2018, Kohala Coast, Hawaii, USA, 2018, pp. 343–352. cancers: LUSC&GBM
  2. Deep learning-based A. Cheerla and O. Gevaert, “Deep learning with multimodal representation for pancancer prognosis prediction,” Bioinformatics, vol. 35, no. 14, pp. i446–i454, 2019. cancers: 20 different cancer types code: https://github.com/gevaertlab/MultimodalPrognosis
  3. SCCA+ordinal information W. Shao, K. Huang, Z. Han, J. Cheng, L. Cheng, T. Wang, L. Sun, Z. Lu, J. Zhang, and D. Zhang, “Integrative Analysis of Pathological Images and Multi-Dimensional Genomic Data for Early-Stage Cancer Prognosis,” IEEE Transactions on Medical Imaging, vol. 39, no. 1, pp. 99–110, 2020. cancers: LUSC&KIRC&KIRP
  4. Diagnosis-guided multi-modal feature selection method (DGM2FS) W. Shao, T. Wang, Z. Huang, J. Cheng, Z. Han, D. Zhang, and K. Huang, “Diagnosis-guided multi-modal feature selection for prognosis prediction of lung squamous cell carcinoma,” in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2019, pp. 113–121. cancers: lung squamous cell carcinoma(SCC)
  5. GPMKL D. Sun, A. Li, B. Tang, and M. Wang, “Integrating genomic data and pathological images to effectively predict breast cancer clinical outcome,” Computer Methods and Programs in Biomedicine, vol. 161, pp. 45–53, 2018. cancers: BRCA
  6. Deep Learning-based J. Hao, S. C. Kosaraju, N. Z. Tsaku, D. H. Song, and M. Kang, “PAGE-Net: Interpretable and Integrative Deep Learning for Survival Analysis Using Histopathological Images and Genomic Data,” in Biocomputing 2020, Kohala Coast, Hawaii, USA, 2019, pp. 355–366. cancer: GBM code: https://github.com/DataX-JieHao/PAGE-Net
  7. Kronecker product R. J. Chen, M. Y. Lu, J.Wang, D. F. K.Williamson, S. J. Rodig, N. I. Lindeman, and F. Mahmood, “Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” IEEE Transactions on Medical Imaging, pp. 1–1, 2020. cancers: LGG&GBM code: https://github.com/mahmoodlab/PathomicFusion
  8. BFPS Z. Ning, W. Pan, Y. Chen, Q. Xiao, X. Zhang, J. Luo, J. Wang, and Y. Zhang, “Integrative analysis of cross-modal features for the prognosis prediction of clear cell renal cell carcinoma,” Bioinformatics, vol. 36, no. 9, pp. 2888–2895, 2020. cancers: ccRCC code: https://github.com/zhang-de-lab/zhang-lab?%20from=singlemessage
  9. LSCDFS-MKL A. Zhang, A. Li, J. He, and M. Wang, “LSCDFS-MKL: A multiple kernel based method for lung squamous cell carcinomas diseasefree survival prediction with pathological and genomic data,” Journal of Biomedical Informatics, vol. 94, p. 103194, 2019. cancers: lung squamous cell carcinomas(SCC)
  10. Deep learning-based L. A. Vale Silva and K. Rohr, “Pan-Cancer Prognosis Prediction Using Multimodal Deep Learning,” in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), 2020, pp. 568–571. cancers: Pancancer

5.3 Papers using other datasets