--- a
+++ b/README.md
@@ -0,0 +1,62 @@
+# Multi-modal Longitudinal Representation Learning for Predicting Neoadjuvant Therapy Response in Breast Cancer Treatment
+In this study, we design a temporal foundation model, a multi-modal longitudinal representation learning pipeline (MLRL). 
+We developed MLRL using an in-house longitudinal multi-modal dataset comprising 3,719 breast MRI scans and paired reports. 
+We also evaluated MLRL system on an international public longitudinal dataset comprising 2,516 exams. 
+We proposed MLRL in a multi-scale self-supervision scheme, including single-time scale vision-text alignment (VTA) learning and multi-time scale visual/textual progress (TVP/TTP) learning. 
+Importantly, the TVP/TTP strategy overcomes the limitation of uniformly temporal learning across patients (i.e., the positive-free pairs problem) and enables the extraction of visual changing representations and the textual as well, 
+facilitating downstream evaluation of tumor progress. 
+We evaluated the label-efficiency ability of our method by comparing it to several state-of-the-art self-supervised longitudinal learning and multi-modal VL methods. 
+The results on two longitudinal datasets show that our approach presents excellent generalization capability and brings significant improvements, 
+with unsupervised temporal progress metrics obtained from TVP/TTP showcasing MLRL ability in distinguishing temporal trends between therapy response populations. 
+Our MLRL framework enables interpretable visual tracking of progressive areas in temporal examinations with corresponding report aligned, 
+offering insights into longitudinal VL foundation tools and potentially facilitating the temporal clinical decision-making process. 
+## Workflow of multi-modal longitudinal representation learning (MLRL)
+<img src="https://github.com/yawwG/MLRL/blob/main/src/figures/method1.png"/>
+
+[comment]: <> (## Overview of proposed temporal progress transformer and multi-scale self-supervised consistent learning)
+
+[comment]: <> (<img src="https://github.com/yawwG/MLRL/figures/method2.png"/>)
+## Longitudinal disease progress tracking and visualization of word-based attention given temporal visual progress embeddings
+<img src="https://github.com/yawwG/MLRL/blob/main/src/figures/results1.png"/>
+
+## Environment Setup
+Start by [installing PyTorch 1.8.1](https://pytorch.org/get-started/locally/) with the right CUDA version, then clone this repository and install the dependencies.  
+
+```bash
+$ conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch
+$ pip install git@github.com:yawwG/MLRL.git
+$ conda env create -f environment.yml
+```
+
+## Code Description
+This codebase has been developed with python version 3.7, PyTorch version 1.8.1, CUDA 11.1 and pytorch-lightning 1.5.9. 
+Example configurations for pretraining and classification can be found in the `./configs`. 
+All training and testing are done using the `run.py` script. For more documentation, please run: 
+
+```bash 
+python run.py --help
+```
+
+The preprocessing steps for dataset can be found in `datasets`
+The dataset using is specified in config.yaml by key("dataset").
+
+### Pre-Train MLRL
+```bash 
+python run.py -c configs/MRI_pretrain_config.yaml --train
+```
+
+### Fine-tune and Test Applications
+```bash 
+python run.py  -c configs/MRI_cls_config.yaml --train --test --train_pct 1 &
+python run.py  -c configs/MRI_cls_config.yaml --train --test --train_pct 0.1 &
+python run.py  -c configs/MRI_cls_config.yaml --train --test --train_pct 0.05
+```
+
+## Contact details
+If you have any questions please contact us. 
+
+Email: ritse.mann@radboudumc.nl (Ritse Mann); taotanjs@gmail.com (Tao Tan); y.gao@nki.nl (Yuan Gao)
+
+Links: [Netherlands Cancer Institute](https://www.nki.nl/), [Radboud University Medical Center](https://www.radboudumc.nl/en/patient-care), and [Maastricht University](https://www.maastrichtuniversity.nl/nl)
+
+<img src="https://github.com/yawwG/Visualize-what-you-learn/blob/main/figures/NKI.png" width="166.98" height="87.12"/><img src="https://github.com/yawwG/Visualize-what-you-learn/blob/main/figures/RadboudUMC.png" width="231" height="87.12"/><img src="https://github.com/yawwG/Visualize-what-you-learn/blob/main/figures/Maastricht.png" width="237.6" height="87.12"/>