PANTHER / Git / Diff of /README.md

Models:

DanielG/

PANTHER

Downloads: 1

Diff of /README.md [e66fb7] .. [286bfb]

Switch to unified view


# PANTHER


  <b>Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology</b>, CVPR 2024.
    <br><em>Andrew H. Song\*, Richard J. Chen\*, Tong Ding, Drew F.K. Williamson, Guillaume Jaume, Faisal Mahmood</em></br>




[Arxiv](https://arxiv.org/pdf/2405.11643) | [Cite](#cite)

**Abstract:** Representation learning of pathology whole-slide images (WSIs) has been has primarily relied on weak supervision with Multiple Instance Learning (MIL). However, the slide representations resulting from this approach are highly tailored to specific clinical tasks, which limits their expressivity and generalization, particularly in scenarios with limited data. Instead, we hypothesize that morphological redundancy in tissue can be leveraged to build a task-agnostic slide representation in an unsupervised fashion. To this end, we introduce **PANTHER**, a prototype-based approach rooted in the Gaussian mixture model that summarizes the set of WSI patches into a much smaller set of morphological prototypes. Specifically, each patch is assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. Utilizing the estimated mixture parameters, we then construct a compact slide representation that can be readily used for a wide range of downstream tasks.
By performing an extensive evaluation of **PANTHER** on subtyping and survival tasks using 13 datasets, we show that 1) **PANTHER** outperforms or is on par with supervised MIL baselines and 2) the analysis of morphological prototypes brings new qualitative and quantitative insights into model interpretability.




## Updates
- **02/17/2025**: You can also use [TRIDENT](https://github.com/mahmoodlab/Trident) to extract patch features for PANTHER.
- **07/02/2024**: Multimodal extension of PANTHER ([MMP](https://github.com/mahmoodlab/MMP)) codebase is now live.
- **05/06/2024**: The first version of PANTHER codebase is now live.

## Installation
Once you clone the repo, please run the following command to create PANTHER conda environment.
```shell
conda env create -f env.yaml
```

## PANTHER Walkthrough
There are two workflows for PANTHER, depending on the use case.
- **Workflow 1** 
    - Step 0 ⇒ Step 1 ⇒ Step 2A ⇒ (Step 3)
    - Useful for constructing *unsupervised* slide representations, without any specific downstream tasks at hand.
- **Workflow 2**
    - Step 0 ⇒ Step 1 ⇒ Step 2B ⇒ (Step 3)
    - Useful for when there is a specific downstream task at hand. The slide representations will be constructed. 

### Step 0. Dataset organization
**Data csv**: The data csv files (with appropriate splits, e.g., train, test) are placed within `src/splits` with appropriate folder structure. For example, for classification task on ebrains, we would have
```bash
splits/
    ├── ebrains
            ├── train.csv
            ├── val.csv
            └── test.csv
```

Alternatively, for 5-fold cross-validation survival task on TCGA BRCA, we would have
```bash
splits/
    ├── TCGA_BRCA_survival_k=0
            ├── train.csv
            ├── val.csv
            └── test.csv
    ├── ...

        ├── TCGA_BRCA_survival_k=4
            ├── train.csv
            ├── val.csv
            └── test.csv
```

**Patch features**: For the following steps, we assume that features for each patch have already been extracted and that each WSI is represented as a set of patch features. For examples of patch feature extraction, please refer to [CLAM](https://github.com/mahmoodlab/CLAM). 

The code assumes that the features are either in `.h5` or `.pt` formats - the feature directory path `FEAT_DIR` has to end with the directory `feats_h5/` if the features are in `.h5` format, and `feats_pt/` for `.pt` format.

While there is no de facto standard, one good practice of organizing features are as follows (used as examples in [clustering](src/scripts/prototype/clustering.sh) and [panther](src/scripts/survival/panther.sh))
```
/path_to_data_folder/tcga_brca/extracted_mag20x_patch256_fp/extracted-vit_large_patch16_224.dinov2.uni_mass100k/feats_h5
```
which specifies *magnification*, *patch size*, and *feature extractor* used to create the patch features. 

### Step 1. Prototype construction
**Tips (Updated on 02/17/2025)** 
- Based on users' opinions, it seems CPU-based K-means is yielding better clustering quality and overall better initialization prototypes. We will provide update on the FAISS-based clustering procedure.
- Users have also reported good performance when the patch features are normalized to a unit norm. However, since the feature norm itself could contain valuable morphological information, we recommend trying both ways.

For prototype construction, we use K-means clustering across all training WSIs. We can use either CPU-based K-Means or GPU-based FAISS for clustering patch features. 
For example, we can use the following command to find 16 prototypes (of 1,024 dimension each) using FAISS from WSIs corresponding to `SPLIT_DIR/train.csv`.
```shell
CUDA_VISIBLE_DEVICES=0 python -m training.main_prototype \
--mode faiss \
--data_source FEAT_DIR_1,FEAT_DIR_2 \
--split_dir SPLIT_DIR \
--split_names train \
--in_dim 1024 \
--n_proto_patches 1000000 \
--n_proto 16 \
--n_init 5 \
--seed 1 \
--num_workers 10 \
```
The list of parameters is as follows:
- `mode`: 'faiss' uses GPU-enabled K-means clustering to find the prototypes. 'kmeans' uses sklearn K-means clustering on CPU ('faiss' or 'kmeans').
- `data_source`: comma-separated list of feature directories ending with either `feats_h5` or `feats_p5`. Example of a feature dictory is provided in **Step 0**.
- `split_names`: Which data split to perform clustering/prototyping on. By default `train` is the best (Since train split has the most data.) 
- `in_dim`: Dimension of the patch features, dependent on the feature encoder.
- `n_proto`: Number of prototypes.
- `n_proto_patches`: Number of patch features to use per prototype. In total, `n_proto * n_proto_patches` features are used for finding prototypes.
- `n_init`: Number of K-means initializations to try.

The prototypes will be saved in the `SPLIT_DIR/prototypes` folder.


A concrete script example of using TCGA-BRCA patch features can be found below. 
```shell
cd src
./scripts/prototype/brca.sh 0
```
This will initiate the script `scripts/prototype/clustering.sh` for K-means clustering. Detailed explanations for clustering hyperparameters can be found in **clustering.sh**. 

### Step 2A. Unsupervised slide representation construction
Once the prototypes are constructed, we can use **PANTHER** or **OT** to construct unsupervised slide representations.
```shell
cd src
./scripts/embedding/brca.sh 0 panther
``` 
This step will create two files in the `SPLIT_DIR/embeddings` folder: 1) **(\*.pkl)** original slide-level representation and 2) **(\*_tokenized.pkl)** slide-level representation tokenized into each prototype and statistics (e.g., mixture probability, mean, cov). Note that for **OT**, mixture probability is uniform and `cov=None`.

Alternatively, if you want to construct slide representations as part of a classification or survival downstream task, you can skip Step 2A and go straight to Step 2B.


### Step 2B. Training downstream model
Once the prototypes are constructed in Step 1 (Step 2A is not required), we can run a downstream task
```shell
cd src
./scripts/survival/brca_surv.sh 0 panther
``` 

### Step 3. Visualization

To visualize GMM mixture proportions in prototypical assignment maps in PANTHER, see the accompanying [notebook](src/visualization/prototypical_assignment_map_visualization.ipynb).

<img src='docs/prototypical_assignment_map.jpg' width="1400px" align="center"/>

## Additional Findings in PANTHER
- We observe that using high-quality pretrained ROI encoders (such as UNI) leads to significant performances across all MIL and set-based learning methods (see the **Supplement**). Specifically:
- - When using Resnet-50 (ImageNet Transfer) and CTransPath features, unsupervised set representation methods such as OT and PANTHER underperform MIL methods (using the same features). With UNI features, OT and PANTHER can readily outperform MIL, and should be considered strong baselines when evaluating slide-level tasks.
- - DeepAttnMISL with UNI features becomes a strong MIL baseline. This can be attributed to DeepAttnMISL dependent on K-Means for cluster pooling (which depends on high-quality representations).
- - With unsupervised slide representations extracted per WSI (via OT or PANTHER), training survival models on WSIs is now much more stable since you can directly use Cox loss (instead of NLL). Across all of our ablation experiments, PANTHER with UNI features always achieved C-Index > 0.6.
  
## PANTHER Limitations
As unsupervised slide representations in PANTHER are created using non-parametric techniques such as K-Means Clustering and GMMs (which rely on Euclidean distance or dot product to compare embeddings), we note the following limitations:
- Dependent on the degree of dataset shift between the train and test distributions (due to variable H&E stain variability, known as image acquisition shift), prototype assignment for certain WSIs may lead to results in which all patches are assigned to a single prototype. This is exemplified in TCGA which has site-specific biases, and is thus an important consideration when considering using PANTHER (or any non-parametric approach) for histopathologic biomarker discovery.
- When clustering over a WSI dataset composed of millions to billions of patches, clustering with only `C=16` clusters will likely underfit the dataset, and also lead to collapse of all patches in a WSI falling under a single prototype. Empirically, we found `C=16` to outperform `C=32` in supervised settings. However, in settings such as biomarker discovery or unsupervised tissue segmentation, using more prototypes may improve performance.

## Acknowledgements
If you find our work useful in your research or if you use parts of this code please cite our paper:

```bibtext
@inproceedings{song2024morphological,
    title={Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology},
    author={Song, Andrew H and Chen, Richard J and Ding, Tong and Williamson, Drew FK and Jaume, Guillaume and Mahmood, Faisal},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2024},
}
```

The code for **PANTHER** was adapted and inspired by the fantastic works of [DIEM](https://openreview.net/forum?id=MXdFBmHT4C), [OTK](https://openreview.net/forum?id=ZK6vTvb84s), and [CLAM](https://github.com/mahmoodlab/CLAM). Boilerplate code for setting up supervised MIL benchmarks was developed by Ming Y. Lu.

## Issues 
- Please open new threads or report issues directly (for urgent blockers) to `asong@bwh.harvard.edu`.
- Immediate response to minor issues may not be available.



	a/README.md		b/README.md
1	# PANTHER	1	# PANTHER
2		2
3		3
4	<b>Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology</b>, CVPR 2024.	4	<b>Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology</b>, CVPR 2024.
5	<br><em>Andrew H. Song\, Richard J. Chen\, Tong Ding, Drew F.K. Williamson, Guillaume Jaume, Faisal Mahmood</em></br>	5	<br><em>Andrew H. Song\, Richard J. Chen\, Tong Ding, Drew F.K. Williamson, Guillaume Jaume, Faisal Mahmood</em></br>
6		6
7		7
8	<img src="docs/panther.png" width="300px" align="right" />	8
9
10	[Arxiv](https://arxiv.org/pdf/2405.11643) \| [Cite](#cite)	9	[Arxiv](https://arxiv.org/pdf/2405.11643) \| [Cite](#cite)
11		10
12	Abstract: Representation learning of pathology whole-slide images (WSIs) has been has primarily relied on weak supervision with Multiple Instance Learning (MIL). However, the slide representations resulting from this approach are highly tailored to specific clinical tasks, which limits their expressivity and generalization, particularly in scenarios with limited data. Instead, we hypothesize that morphological redundancy in tissue can be leveraged to build a task-agnostic slide representation in an unsupervised fashion. To this end, we introduce PANTHER, a prototype-based approach rooted in the Gaussian mixture model that summarizes the set of WSI patches into a much smaller set of morphological prototypes. Specifically, each patch is assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. Utilizing the estimated mixture parameters, we then construct a compact slide representation that can be readily used for a wide range of downstream tasks.	11	Abstract: Representation learning of pathology whole-slide images (WSIs) has been has primarily relied on weak supervision with Multiple Instance Learning (MIL). However, the slide representations resulting from this approach are highly tailored to specific clinical tasks, which limits their expressivity and generalization, particularly in scenarios with limited data. Instead, we hypothesize that morphological redundancy in tissue can be leveraged to build a task-agnostic slide representation in an unsupervised fashion. To this end, we introduce PANTHER, a prototype-based approach rooted in the Gaussian mixture model that summarizes the set of WSI patches into a much smaller set of morphological prototypes. Specifically, each patch is assumed to have been generated from a mixture distribution, where each mixture component represents a morphological exemplar. Utilizing the estimated mixture parameters, we then construct a compact slide representation that can be readily used for a wide range of downstream tasks.
13	By performing an extensive evaluation of PANTHER on subtyping and survival tasks using 13 datasets, we show that 1) PANTHER outperforms or is on par with supervised MIL baselines and 2) the analysis of morphological prototypes brings new qualitative and quantitative insights into model interpretability.	12	By performing an extensive evaluation of PANTHER on subtyping and survival tasks using 13 datasets, we show that 1) PANTHER outperforms or is on par with supervised MIL baselines and 2) the analysis of morphological prototypes brings new qualitative and quantitative insights into model interpretability.
14		13
15
16	<img src='docs/fig_main.jpg' width="1400px" align="center"/>
17
18	## Updates	14	## Updates
19	- 02/17/2025: You can also use [TRIDENT](https://github.com/mahmoodlab/Trident) to extract patch features for PANTHER.	15	- 02/17/2025: You can also use [TRIDENT](https://github.com/mahmoodlab/Trident) to extract patch features for PANTHER.
20	- 07/02/2024: Multimodal extension of PANTHER ([MMP](https://github.com/mahmoodlab/MMP)) codebase is now live.	16	- 07/02/2024: Multimodal extension of PANTHER ([MMP](https://github.com/mahmoodlab/MMP)) codebase is now live.
21	- 05/06/2024: The first version of PANTHER codebase is now live.	17	- 05/06/2024: The first version of PANTHER codebase is now live.
22		18
23	## Installation	19	## Installation
24	Once you clone the repo, please run the following command to create PANTHER conda environment.	20	Once you clone the repo, please run the following command to create PANTHER conda environment.
25	```shell	21	```shell
26	conda env create -f env.yaml	22	conda env create -f env.yaml
27	```	23	```
28		24
29	## PANTHER Walkthrough	25	## PANTHER Walkthrough
30	There are two workflows for PANTHER, depending on the use case.	26	There are two workflows for PANTHER, depending on the use case.
31	- Workflow 1	27	- Workflow 1
32	- Step 0 ⇒ Step 1 ⇒ Step 2A ⇒ (Step 3)	28	- Step 0 ⇒ Step 1 ⇒ Step 2A ⇒ (Step 3)
33	- Useful for constructing unsupervised slide representations, without any specific downstream tasks at hand.	29	- Useful for constructing unsupervised slide representations, without any specific downstream tasks at hand.
34	- Workflow 2	30	- Workflow 2
35	- Step 0 ⇒ Step 1 ⇒ Step 2B ⇒ (Step 3)	31	- Step 0 ⇒ Step 1 ⇒ Step 2B ⇒ (Step 3)
36	- Useful for when there is a specific downstream task at hand. The slide representations will be constructed.	32	- Useful for when there is a specific downstream task at hand. The slide representations will be constructed.
37		33
38	### Step 0. Dataset organization	34	### Step 0. Dataset organization
39	Data csv: The data csv files (with appropriate splits, e.g., train, test) are placed within `src/splits` with appropriate folder structure. For example, for classification task on ebrains, we would have	35	Data csv: The data csv files (with appropriate splits, e.g., train, test) are placed within `src/splits` with appropriate folder structure. For example, for classification task on ebrains, we would have
40	```bash	36	```bash
41	splits/	37	splits/
42	├── ebrains	38	├── ebrains
43	├── train.csv	39	├── train.csv
44	├── val.csv	40	├── val.csv
45	└── test.csv	41	└── test.csv
46	```	42	```
47		43
48	Alternatively, for 5-fold cross-validation survival task on TCGA BRCA, we would have	44	Alternatively, for 5-fold cross-validation survival task on TCGA BRCA, we would have
49	```bash	45	```bash
50	splits/	46	splits/
51	├── TCGA_BRCA_survival_k=0	47	├── TCGA_BRCA_survival_k=0
52	├── train.csv	48	├── train.csv
53	├── val.csv	49	├── val.csv
54	└── test.csv	50	└── test.csv
55	├── ...	51	├── ...
56		52
57	├── TCGA_BRCA_survival_k=4	53	├── TCGA_BRCA_survival_k=4
58	├── train.csv	54	├── train.csv
59	├── val.csv	55	├── val.csv
60	└── test.csv	56	└── test.csv
61	```	57	```
62		58
63	Patch features: For the following steps, we assume that features for each patch have already been extracted and that each WSI is represented as a set of patch features. For examples of patch feature extraction, please refer to [CLAM](https://github.com/mahmoodlab/CLAM).	59	Patch features: For the following steps, we assume that features for each patch have already been extracted and that each WSI is represented as a set of patch features. For examples of patch feature extraction, please refer to [CLAM](https://github.com/mahmoodlab/CLAM).
64		60
65	The code assumes that the features are either in `.h5` or `.pt` formats - the feature directory path `FEAT_DIR` has to end with the directory `feats_h5/` if the features are in `.h5` format, and `feats_pt/` for `.pt` format.	61	The code assumes that the features are either in `.h5` or `.pt` formats - the feature directory path `FEAT_DIR` has to end with the directory `feats_h5/` if the features are in `.h5` format, and `feats_pt/` for `.pt` format.
66		62
67	While there is no de facto standard, one good practice of organizing features are as follows (used as examples in [clustering](src/scripts/prototype/clustering.sh) and [panther](src/scripts/survival/panther.sh))	63	While there is no de facto standard, one good practice of organizing features are as follows (used as examples in [clustering](src/scripts/prototype/clustering.sh) and [panther](src/scripts/survival/panther.sh))
68	```	64	```
69	/path_to_data_folder/tcga_brca/extracted_mag20x_patch256_fp/extracted-vit_large_patch16_224.dinov2.uni_mass100k/feats_h5	65	/path_to_data_folder/tcga_brca/extracted_mag20x_patch256_fp/extracted-vit_large_patch16_224.dinov2.uni_mass100k/feats_h5
70	```	66	```
71	which specifies magnification, patch size, and feature extractor used to create the patch features.	67	which specifies magnification, patch size, and feature extractor used to create the patch features.
72		68
73	### Step 1. Prototype construction	69	### Step 1. Prototype construction
74	Tips (Updated on 02/17/2025)	70	Tips (Updated on 02/17/2025)
75	- Based on users' opinions, it seems CPU-based K-means is yielding better clustering quality and overall better initialization prototypes. We will provide update on the FAISS-based clustering procedure.	71	- Based on users' opinions, it seems CPU-based K-means is yielding better clustering quality and overall better initialization prototypes. We will provide update on the FAISS-based clustering procedure.
76	- Users have also reported good performance when the patch features are normalized to a unit norm. However, since the feature norm itself could contain valuable morphological information, we recommend trying both ways.	72	- Users have also reported good performance when the patch features are normalized to a unit norm. However, since the feature norm itself could contain valuable morphological information, we recommend trying both ways.
77		73
78	For prototype construction, we use K-means clustering across all training WSIs. We can use either CPU-based K-Means or GPU-based FAISS for clustering patch features.	74	For prototype construction, we use K-means clustering across all training WSIs. We can use either CPU-based K-Means or GPU-based FAISS for clustering patch features.
79	For example, we can use the following command to find 16 prototypes (of 1,024 dimension each) using FAISS from WSIs corresponding to `SPLIT_DIR/train.csv`.	75	For example, we can use the following command to find 16 prototypes (of 1,024 dimension each) using FAISS from WSIs corresponding to `SPLIT_DIR/train.csv`.
80	```shell	76	```shell
81	CUDA_VISIBLE_DEVICES=0 python -m training.main_prototype \	77	CUDA_VISIBLE_DEVICES=0 python -m training.main_prototype \
82	--mode faiss \	78	--mode faiss \
83	--data_source FEAT_DIR_1,FEAT_DIR_2 \	79	--data_source FEAT_DIR_1,FEAT_DIR_2 \
84	--split_dir SPLIT_DIR \	80	--split_dir SPLIT_DIR \
85	--split_names train \	81	--split_names train \
86	--in_dim 1024 \	82	--in_dim 1024 \
87	--n_proto_patches 1000000 \	83	--n_proto_patches 1000000 \
88	--n_proto 16 \	84	--n_proto 16 \
89	--n_init 5 \	85	--n_init 5 \
90	--seed 1 \	86	--seed 1 \
91	--num_workers 10 \	87	--num_workers 10 \
92	```	88	```
93	The list of parameters is as follows:	89	The list of parameters is as follows:
94	- `mode`: 'faiss' uses GPU-enabled K-means clustering to find the prototypes. 'kmeans' uses sklearn K-means clustering on CPU ('faiss' or 'kmeans').	90	- `mode`: 'faiss' uses GPU-enabled K-means clustering to find the prototypes. 'kmeans' uses sklearn K-means clustering on CPU ('faiss' or 'kmeans').
95	- `data_source`: comma-separated list of feature directories ending with either `feats_h5` or `feats_p5`. Example of a feature dictory is provided in Step 0.	91	- `data_source`: comma-separated list of feature directories ending with either `feats_h5` or `feats_p5`. Example of a feature dictory is provided in Step 0.
96	- `split_names`: Which data split to perform clustering/prototyping on. By default `train` is the best (Since train split has the most data.)	92	- `split_names`: Which data split to perform clustering/prototyping on. By default `train` is the best (Since train split has the most data.)
97	- `in_dim`: Dimension of the patch features, dependent on the feature encoder.	93	- `in_dim`: Dimension of the patch features, dependent on the feature encoder.
98	- `n_proto`: Number of prototypes.	94	- `n_proto`: Number of prototypes.
99	- `n_proto_patches`: Number of patch features to use per prototype. In total, `n_proto * n_proto_patches` features are used for finding prototypes.	95	- `n_proto_patches`: Number of patch features to use per prototype. In total, `n_proto * n_proto_patches` features are used for finding prototypes.
100	- `n_init`: Number of K-means initializations to try.	96	- `n_init`: Number of K-means initializations to try.
101		97
102	The prototypes will be saved in the `SPLIT_DIR/prototypes` folder.	98	The prototypes will be saved in the `SPLIT_DIR/prototypes` folder.
103		99
104		100
105	A concrete script example of using TCGA-BRCA patch features can be found below.	101	A concrete script example of using TCGA-BRCA patch features can be found below.
106	```shell	102	```shell
107	cd src	103	cd src
108	./scripts/prototype/brca.sh 0	104	./scripts/prototype/brca.sh 0
109	```	105	```
110	This will initiate the script `scripts/prototype/clustering.sh` for K-means clustering. Detailed explanations for clustering hyperparameters can be found in clustering.sh.	106	This will initiate the script `scripts/prototype/clustering.sh` for K-means clustering. Detailed explanations for clustering hyperparameters can be found in clustering.sh.
111		107
112	### Step 2A. Unsupervised slide representation construction	108	### Step 2A. Unsupervised slide representation construction
113	Once the prototypes are constructed, we can use PANTHER or OT to construct unsupervised slide representations.	109	Once the prototypes are constructed, we can use PANTHER or OT to construct unsupervised slide representations.
114	```shell	110	```shell
115	cd src	111	cd src
116	./scripts/embedding/brca.sh 0 panther	112	./scripts/embedding/brca.sh 0 panther
117	```	113	```
118	This step will create two files in the `SPLIT_DIR/embeddings` folder: 1) *(\.pkl) original slide-level representation and 2) (\_tokenized.pkl)* slide-level representation tokenized into each prototype and statistics (e.g., mixture probability, mean, cov). Note that for OT, mixture probability is uniform and `cov=None`.	114	This step will create two files in the `SPLIT_DIR/embeddings` folder: 1) *(\.pkl) original slide-level representation and 2) (\_tokenized.pkl)* slide-level representation tokenized into each prototype and statistics (e.g., mixture probability, mean, cov). Note that for OT, mixture probability is uniform and `cov=None`.
119		115
120	Alternatively, if you want to construct slide representations as part of a classification or survival downstream task, you can skip Step 2A and go straight to Step 2B.	116	Alternatively, if you want to construct slide representations as part of a classification or survival downstream task, you can skip Step 2A and go straight to Step 2B.
121		117
122		118
123	### Step 2B. Training downstream model	119	### Step 2B. Training downstream model
124	Once the prototypes are constructed in Step 1 (Step 2A is not required), we can run a downstream task	120	Once the prototypes are constructed in Step 1 (Step 2A is not required), we can run a downstream task
125	```shell	121	```shell
126	cd src	122	cd src
127	./scripts/survival/brca_surv.sh 0 panther	123	./scripts/survival/brca_surv.sh 0 panther
128	```	124	```
129		125
130	### Step 3. Visualization	126	### Step 3. Visualization
131		127
132	To visualize GMM mixture proportions in prototypical assignment maps in PANTHER, see the accompanying [notebook](src/visualization/prototypical_assignment_map_visualization.ipynb).	128	To visualize GMM mixture proportions in prototypical assignment maps in PANTHER, see the accompanying [notebook](src/visualization/prototypical_assignment_map_visualization.ipynb).
133		129
134	<img src='docs/prototypical_assignment_map.jpg' width="1400px" align="center"/>	130	<img src='docs/prototypical_assignment_map.jpg' width="1400px" align="center"/>
135		131
136	## Additional Findings in PANTHER	132	## Additional Findings in PANTHER
137	- We observe that using high-quality pretrained ROI encoders (such as UNI) leads to significant performances across all MIL and set-based learning methods (see the Supplement). Specifically:	133	- We observe that using high-quality pretrained ROI encoders (such as UNI) leads to significant performances across all MIL and set-based learning methods (see the Supplement). Specifically:
138	- - When using Resnet-50 (ImageNet Transfer) and CTransPath features, unsupervised set representation methods such as OT and PANTHER underperform MIL methods (using the same features). With UNI features, OT and PANTHER can readily outperform MIL, and should be considered strong baselines when evaluating slide-level tasks.	134	- - When using Resnet-50 (ImageNet Transfer) and CTransPath features, unsupervised set representation methods such as OT and PANTHER underperform MIL methods (using the same features). With UNI features, OT and PANTHER can readily outperform MIL, and should be considered strong baselines when evaluating slide-level tasks.
139	- - DeepAttnMISL with UNI features becomes a strong MIL baseline. This can be attributed to DeepAttnMISL dependent on K-Means for cluster pooling (which depends on high-quality representations).	135	- - DeepAttnMISL with UNI features becomes a strong MIL baseline. This can be attributed to DeepAttnMISL dependent on K-Means for cluster pooling (which depends on high-quality representations).
140	- - With unsupervised slide representations extracted per WSI (via OT or PANTHER), training survival models on WSIs is now much more stable since you can directly use Cox loss (instead of NLL). Across all of our ablation experiments, PANTHER with UNI features always achieved C-Index > 0.6.	136	- - With unsupervised slide representations extracted per WSI (via OT or PANTHER), training survival models on WSIs is now much more stable since you can directly use Cox loss (instead of NLL). Across all of our ablation experiments, PANTHER with UNI features always achieved C-Index > 0.6.
141		137
142	## PANTHER Limitations	138	## PANTHER Limitations
143	As unsupervised slide representations in PANTHER are created using non-parametric techniques such as K-Means Clustering and GMMs (which rely on Euclidean distance or dot product to compare embeddings), we note the following limitations:	139	As unsupervised slide representations in PANTHER are created using non-parametric techniques such as K-Means Clustering and GMMs (which rely on Euclidean distance or dot product to compare embeddings), we note the following limitations:
144	- Dependent on the degree of dataset shift between the train and test distributions (due to variable H&E stain variability, known as image acquisition shift), prototype assignment for certain WSIs may lead to results in which all patches are assigned to a single prototype. This is exemplified in TCGA which has site-specific biases, and is thus an important consideration when considering using PANTHER (or any non-parametric approach) for histopathologic biomarker discovery.	140	- Dependent on the degree of dataset shift between the train and test distributions (due to variable H&E stain variability, known as image acquisition shift), prototype assignment for certain WSIs may lead to results in which all patches are assigned to a single prototype. This is exemplified in TCGA which has site-specific biases, and is thus an important consideration when considering using PANTHER (or any non-parametric approach) for histopathologic biomarker discovery.
145	- When clustering over a WSI dataset composed of millions to billions of patches, clustering with only `C=16` clusters will likely underfit the dataset, and also lead to collapse of all patches in a WSI falling under a single prototype. Empirically, we found `C=16` to outperform `C=32` in supervised settings. However, in settings such as biomarker discovery or unsupervised tissue segmentation, using more prototypes may improve performance.	141	- When clustering over a WSI dataset composed of millions to billions of patches, clustering with only `C=16` clusters will likely underfit the dataset, and also lead to collapse of all patches in a WSI falling under a single prototype. Empirically, we found `C=16` to outperform `C=32` in supervised settings. However, in settings such as biomarker discovery or unsupervised tissue segmentation, using more prototypes may improve performance.
146		142
147	## Acknowledgements	143	## Acknowledgements
148	If you find our work useful in your research or if you use parts of this code please cite our paper:	144	If you find our work useful in your research or if you use parts of this code please cite our paper:
149		145
150	```bibtext	146	```bibtext
151	@inproceedings{song2024morphological,	147	@inproceedings{song2024morphological,
152	title={Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology},	148	title={Morphological Prototyping for Unsupervised Slide Representation Learning in Computational Pathology},
153	author={Song, Andrew H and Chen, Richard J and Ding, Tong and Williamson, Drew FK and Jaume, Guillaume and Mahmood, Faisal},	149	author={Song, Andrew H and Chen, Richard J and Ding, Tong and Williamson, Drew FK and Jaume, Guillaume and Mahmood, Faisal},
154	booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},	150	booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
155	year={2024},	151	year={2024},
156	}	152	}
157	```	153	```
158		154
159	The code for PANTHER was adapted and inspired by the fantastic works of [DIEM](https://openreview.net/forum?id=MXdFBmHT4C), [OTK](https://openreview.net/forum?id=ZK6vTvb84s), and [CLAM](https://github.com/mahmoodlab/CLAM). Boilerplate code for setting up supervised MIL benchmarks was developed by Ming Y. Lu.	155	The code for PANTHER was adapted and inspired by the fantastic works of [DIEM](https://openreview.net/forum?id=MXdFBmHT4C), [OTK](https://openreview.net/forum?id=ZK6vTvb84s), and [CLAM](https://github.com/mahmoodlab/CLAM). Boilerplate code for setting up supervised MIL benchmarks was developed by Ming Y. Lu.
160		156
161	## Issues	157	## Issues
162	- Please open new threads or report issues directly (for urgent blockers) to `asong@bwh.harvard.edu`.	158	- Please open new threads or report issues directly (for urgent blockers) to `asong@bwh.harvard.edu`.
163	- Immediate response to minor issues may not be available.	159	- Immediate response to minor issues may not be available.
164
165	<img src=docs/joint_logo.png>