# Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Diagnosis and Prognosis

# Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Diagnosis and Prognosis

<details>

<details>

<summary>

<summary>

  <b>Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis</b>, IEEE Transactions on Medical Imaging, 2020.

  <b>Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis</b>, IEEE Transactions on Medical Imaging, 2020.

  <a href="https://ieeexplore.ieee.org/document/9186053" target="blank">[HTML]</a>

  <a href="https://ieeexplore.ieee.org/document/9186053" target="blank">[HTML]</a>

  <a href="https://arxiv.org/abs/1912.08937" target="blank">[arXiv]</a>

  <a href="https://arxiv.org/abs/1912.08937" target="blank">[arXiv]</a>

  <a href="https://www.youtube.com/watch?v=TrjGEUVX5YE" target="blank">[Talk]</a>

  <a href="https://www.youtube.com/watch?v=TrjGEUVX5YE" target="blank">[Talk]</a>

  <br><em>Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, Faisal Mahmood</em></br>

  <br><em>Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Lindeman, Faisal Mahmood</em></br>

</summary>

</summary>

```bash

```bash

@article{chen2020pathomic,

@article{chen2020pathomic,

  title={Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis},

  title={Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis},

  author={Chen, Richard J and Lu, Ming Y and Wang, Jingwen and Williamson, Drew FK and Rodig, Scott J and Lindeman, Neal I and Mahmood, Faisal},

  author={Chen, Richard J and Lu, Ming Y and Wang, Jingwen and Williamson, Drew FK and Rodig, Scott J and Lindeman, Neal I and Mahmood, Faisal},

  journal={IEEE Transactions on Medical Imaging},

  journal={IEEE Transactions on Medical Imaging},

  year={2020},

  year={2020},

  publisher={IEEE}

  publisher={IEEE}

}

}

```

```

</details>

</details>

**Summary:** We propose a simple and scalable method for integrating histology images and -omic data using attention gating and tensor fusion. Histopathology images can be processed using CNNs or GCNs for parameter efficiency or a combination of the the two. The setup is adaptable for integrating multiple -omic modalities with histopathology and can be used for improved diagnostic, prognostic and therapeutic response determinations. 

**Summary:** We propose a simple and scalable method for integrating histology images and -omic data using attention gating and tensor fusion. Histopathology images can be processed using CNNs or GCNs for parameter efficiency or a combination of the the two. The setup is adaptable for integrating multiple -omic modalities with histopathology and can be used for improved diagnostic, prognostic and therapeutic response determinations. 

<img src="https://github.com/mahmoodlab/PathomicFusion/blob/master/main_fig.jpg" width="1024"/>

<img src="https://github.com/mahmoodlab/PathomicFusion/blob/master/main_fig.jpg?raw=true" width="1024"/>

## Community / Follow-Up Work :)

## Community / Follow-Up Work :)

<table>

<table>

<tr>

<tr>

<td>GitHub Repositories / Projects</td>

<td>GitHub Repositories / Projects</td>

<td>

<td>

<a href="https://github.com/Liruiqing-ustc/HFBSurv" target="_blank">★</a>

<a href="https://github.com/Liruiqing-ustc/HFBSurv" target="_blank">★</a>

<a href="https://github.com/mahmoodlab/PORPOISE" target="_blank">★</a>

<a href="https://github.com/mahmoodlab/PORPOISE" target="_blank">★</a>

<a href="https://github.com/TencentAILabHealthcare/MLA-GNN" target="_blank">★</a>

<a href="https://github.com/TencentAILabHealthcare/MLA-GNN" target="_blank">★</a>

<a href="https://github.com/zcwang0702/HGPN" target="_blank">★</a>

<a href="https://github.com/zcwang0702/HGPN" target="_blank">★</a>

<a href="https://github.com/isfj/GPDBN" target="_blank">★</a>

<a href="https://github.com/isfj/GPDBN" target="_blank">★</a>

</td>

</td>

</tr>

</tr>

</table>

</table>

## Updates

## Updates

* 05/26/2021: Updated Google Drive with all models and processed data for TCGA-GBMLGG and TCGA-KIRC. found using the [following link](https://drive.google.com/drive/u/1/folders/1swiMrz84V3iuzk8x99vGIBd5FCVncOlf). The data made available for TCGA-GBMLGG are the **same ROIs** used by [Mobadersany et al.](https://github.com/PathologyDataScience/SCNN)

* 05/26/2021: Updated Google Drive with all models and processed data for TCGA-GBMLGG and TCGA-KIRC. found using the [following link](https://drive.google.com/drive/u/1/folders/1swiMrz84V3iuzk8x99vGIBd5FCVncOlf). The data made available for TCGA-GBMLGG are the **same ROIs** used by [Mobadersany et al.](https://github.com/PathologyDataScience/SCNN)

## Setup

## Setup

### Prerequisites

### Prerequisites

- Linux (Tested on Ubuntu 18.04)

- Linux (Tested on Ubuntu 18.04)

- NVIDIA GPU (Tested on Nvidia GeForce RTX 2080 Tis on local workstations, and Nvidia V100s using Google Cloud)

- NVIDIA GPU (Tested on Nvidia GeForce RTX 2080 Tis on local workstations, and Nvidia V100s using Google Cloud)

- CUDA + cuDNN (Tested on CUDA 10.1 and cuDNN 7.5. CPU mode and CUDA without CuDNN may work with minimal modification, but untested.)

- CUDA + cuDNN (Tested on CUDA 10.1 and cuDNN 7.5. CPU mode and CUDA without CuDNN may work with minimal modification, but untested.)

- torch>=1.1.0

- torch>=1.1.0

- torch_geometric=1.3.0

- torch_geometric=1.3.0

## Code Base Structure

## Code Base Structure

The code base structure is explained below: 

The code base structure is explained below: 

- **train_cv.py**: Cross-validation script for training unimodal and multimodal networks. This script will save evaluation metrics and predictions on the train + test split for each epoch on every split in **checkpoints**.

- **train_cv.py**: Cross-validation script for training unimodal and multimodal networks. This script will save evaluation metrics and predictions on the train + test split for each epoch on every split in **checkpoints**.

- **test_cv.py**: Script for testing unimodal and unimodal networks on only the test split.

- **test_cv.py**: Script for testing unimodal and unimodal networks on only the test split.

- **train_test.py**: Contains the definitions for "train" and "test". 

- **train_test.py**: Contains the definitions for "train" and "test". 

- **networks.py**: Contains PyTorch model definitions for all unimodal and multimodal network.

- **networks.py**: Contains PyTorch model definitions for all unimodal and multimodal network.

- **fusion.py**: Contains PyTorch model definitions for fusion.

- **fusion.py**: Contains PyTorch model definitions for fusion.

- **data_loaders.py**: Contains the PyTorch DatasetLoader definition for loading multimodal data.

- **data_loaders.py**: Contains the PyTorch DatasetLoader definition for loading multimodal data.

- **options.py**: Contains all the options for the argparser.

- **options.py**: Contains all the options for the argparser.

- **make_splits.py**: Script for generating a pickle file that saves + aligns the path for multimodal data for cross-validation.

- **make_splits.py**: Script for generating a pickle file that saves + aligns the path for multimodal data for cross-validation.

- **run_cox_baselines.py**: Script for running Cox baselines.

- **run_cox_baselines.py**: Script for running Cox baselines.

- **utils.py**: Contains definitions for collating, survival loss functions, data preprocessing, evaluation, figure plotting, etc...

- **utils.py**: Contains definitions for collating, survival loss functions, data preprocessing, evaluation, figure plotting, etc...

The directory structure for your multimodal dataset should look similar to the following:

The directory structure for your multimodal dataset should look similar to the following:

```bash

```bash

./

./

├── data

├── data

      └── PROJECT

      └── PROJECT

            ├── INPUT A (e.g. Image)

            ├── INPUT A (e.g. Image)

                ├── image_001.png

                ├── image_001.png

                ├── image_002.png

                ├── image_002.png

                ├── ...

                ├── ...

            ├── INPUT B (e.g. Graph)

            ├── INPUT B (e.g. Graph)

                ├── image_001.pkl

                ├── image_001.pkl

                ├── image_002.pkl

                ├── image_002.pkl

                ├── ...

                ├── ...

            └── INPUT C (e.g. Genomic)

            └── INPUT C (e.g. Genomic)

                └── genomic_data.csv

                └── genomic_data.csv

└── checkpoints

└── checkpoints

        └── PROJECT

        └── PROJECT

            ├── TASK X (e.g. Survival Analysis)

            ├── TASK X (e.g. Survival Analysis)

                ├── path

                ├── path

                    ├── ...

                    ├── ...

                ├── ...

                ├── ...

            └── TASK Y (e.g. Grade Classification)

            └── TASK Y (e.g. Grade Classification)

                ├── path

                ├── path

                    ├── ...

                    ├── ...

                ├── ...

                ├── ...

```

```

Depending on which modalities you are interested in combining, you must: (1) write your own function for aligning multimodal data in **make_splits.py**, (2) create your DatasetLoader in **data_loaders.py**, (3) modify the **options.py** for your data and task. Models will be saved to the **checkpoints** directory, with each model for each task saved in its own directory. At the moment, the only supervised learning tasks implemented are survival outcome prediction and grade classification.

Depending on which modalities you are interested in combining, you must: (1) write your own function for aligning multimodal data in **make_splits.py**, (2) create your DatasetLoader in **data_loaders.py**, (3) modify the **options.py** for your data and task. Models will be saved to the **checkpoints** directory, with each model for each task saved in its own directory. At the moment, the only supervised learning tasks implemented are survival outcome prediction and grade classification.

## Training and Evaluation

## Training and Evaluation

Here are example commands for training unimodal + multimodal networks.

Here are example commands for training unimodal + multimodal networks.

### Survival Model for Input A

### Survival Model for Input A

Example shown below for training a survival model for mode A and saving the model checkpoints + predictions at the end of each split. In this example, we would create a folder called "CNN_A" in "./checkpoints/example/" for all the models in cross-validation. It assumes that "A" is defined as a mode in **dataset_loaders.py** for handling modality-specific data-preprocessing steps (random crop + flip + jittering for images), and that there is a network defined for input A in **networks.py**. "surv" is already defined as a task for training networks for survival analysis in **options.py, networks.py, train_test.py, train_cv.py**.

Example shown below for training a survival model for mode A and saving the model checkpoints + predictions at the end of each split. In this example, we would create a folder called "CNN_A" in "./checkpoints/example/" for all the models in cross-validation. It assumes that "A" is defined as a mode in **dataset_loaders.py** for handling modality-specific data-preprocessing steps (random crop + flip + jittering for images), and that there is a network defined for input A in **networks.py**. "surv" is already defined as a task for training networks for survival analysis in **options.py, networks.py, train_test.py, train_cv.py**.

```

```

python train_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task surv --mode A --model_name CNN_A --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

python train_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task surv --mode A --model_name CNN_A --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

```

```

To obtain test predictions on only the test splits in your cross-validation, you can replace "train_cv" with "test_cv".

To obtain test predictions on only the test splits in your cross-validation, you can replace "train_cv" with "test_cv".

```

```

python test_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task surv --mode input_A --model input_A_CNN --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

python test_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task surv --mode input_A --model input_A_CNN --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

```

```

### Grade Classification Model for Input A + B

### Grade Classification Model for Input A + B

Example shown below for training a grade classification model for fusing modes A and B. Similar to the previous example, we would create a folder called "Fusion_AB" in "./checkpoints/example/" for all the models in cross-validation. It assumes that "AB" is defined as a mode in **dataset_loaders.py** for handling multiple inputs A and B at the same time. "grad" is already defined as a task for training networks for grade classification in **options.py, networks.py, train_test.py, train_cv.py**.

Example shown below for training a grade classification model for fusing modes A and B. Similar to the previous example, we would create a folder called "Fusion_AB" in "./checkpoints/example/" for all the models in cross-validation. It assumes that "AB" is defined as a mode in **dataset_loaders.py** for handling multiple inputs A and B at the same time. "grad" is already defined as a task for training networks for grade classification in **options.py, networks.py, train_test.py, train_cv.py**.

```

```

python train_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task grad --mode AB --model_name Fusion_AB --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

python train_cv.py --exp_name surv --dataroot ./data/example/ --checkpoints_dir ./checkpoints/example/ --task grad --mode AB --model_name Fusion_AB --niter 0 --niter_decay 50 --batch_size 64 --reg_type none --init_type max --lr 0.002 --weight_decay 4e-4 --gpu_ids 0

```

```

## Reproducibility

## Reproducibility

To reporduce the results in our paper and for exact data preprocessing, implementation, and experimental details please follow the instructions here: [./data/TCGA_GBMLGG/](https://github.com/mahmoodlab/PathomicFusion/tree/master/data/TCGA_GBMLGG). Processed data and trained models can be downloaded [here](https://drive.google.com/drive/folders/1swiMrz84V3iuzk8x99vGIBd5FCVncOlf?usp=sharing).

To reporduce the results in our paper and for exact data preprocessing, implementation, and experimental details please follow the instructions here: [./data/TCGA_GBMLGG/](https://github.com/mahmoodlab/PathomicFusion/tree/master/data/TCGA_GBMLGG). Processed data and trained models can be downloaded [here](https://drive.google.com/drive/folders/1swiMrz84V3iuzk8x99vGIBd5FCVncOlf?usp=sharing).

## Issues

## Issues

- Please open new threads or report issues directly (for urgent blockers) to richardchen@g.harvard.edu.

- Please open new threads or report issues directly (for urgent blockers) to richardchen@g.harvard.edu.

- Immediate response to minor issues may not be available.

- Immediate response to minor issues may not be available.

## Licenses, Usages, and Acknowledgements

## Licenses, Usages, and Acknowledgements

- This project is licensed under the GNU GPLv3 License - see the [LICENSE.md](LICENSE.md) file for details. A provisional patent on this work has been filed by the Brigham and Women's Hospital.

- This project is licensed under the GNU GPLv3 License - see the [LICENSE.md](LICENSE.md) file for details. A provisional patent on this work has been filed by the Brigham and Women's Hospital.

- This code is inspired by [SALMON](https://github.com/huangzhii/SALMON) and [SCNN](https://github.com/CancerDataScience/SCNN). Code base structure was inspired by [pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix).

- This code is inspired by [SALMON](https://github.com/huangzhii/SALMON) and [SCNN](https://github.com/CancerDataScience/SCNN). Code base structure was inspired by [pytorch-CycleGAN-and-pix2pix](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix).

- Subsidized computing resources for this project were provided by Nvidia and Google Cloud. 

- Subsidized computing resources for this project were provided by Nvidia and Google Cloud. 

- If you find our work useful in your research, please consider citing our paper at:

- If you find our work useful in your research, please consider citing our paper at:

```bash

```bash

@article{chen2020pathomic,

@article{chen2020pathomic,

  title={Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis},

  title={Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis},

  author={Chen, Richard J and Lu, Ming Y and Wang, Jingwen and Williamson, Drew FK and Rodig, Scott J and Lindeman, Neal I and Mahmood, Faisal},

  author={Chen, Richard J and Lu, Ming Y and Wang, Jingwen and Williamson, Drew FK and Rodig, Scott J and Lindeman, Neal I and Mahmood, Faisal},

  journal={IEEE Transactions on Medical Imaging},

  journal={IEEE Transactions on Medical Imaging},

  year={2020},

  year={2020},

  publisher={IEEE}

  publisher={IEEE}

}

}

```

```

© [Mahmood Lab](http://www.mahmoodlab.org) - This code is made available under the GPLv3 License and is available for non-commercial academic purposes. 

© [Mahmood Lab](http://www.mahmoodlab.org) - This code is made available under the GPLv3 License and is available for non-commercial academic purposes.