## RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance

**Authors:** [Chantal Pellegrini][cp], [Ege Özsoy][eo], [Benjamin Busam][bb], [Nassir Navab][nn], [Matthias Keicher][mk]

[cp]:https://www.cs.cit.tum.de/camp/members/chantal-pellegrini/

[eo]:https://www.cs.cit.tum.de/camp/members/ege-oezsoy/

[mk]:https://www.cs.cit.tum.de/camp/members/matthias-keicher/

[nn]:https://www.cs.cit.tum.de/camp/members/cv-nassir-navab/nassir-navab/

[bb]:https://www.cs.cit.tum.de/camp/members/benjamin-busam-1/

[![](https://img.shields.io/badge/Project_Page-green)](https://chantalmp.github.io/RaDialog/) [![](https://img.shields.io/badge/OpenReview-Paper-blue)](https://openreview.net/pdf?id=trUvr1gSNI) [![](https://img.shields.io/badge/Arxiv-2307.05766-blue)](https://arxiv.org/abs/2311.18681) [![](https://img.shields.io/badge/PhysioNet-Dataset-lightgrey)](https://physionet.org/content/radialog-instruct-dataset/1.1.0/) [![](https://img.shields.io/badge/Huggingface-yellow)](https://huggingface.co/ChantalPellegrini/RaDialog-interactive-radiology-report-generation)

**✨ News ✨**

- 27 March 2025: RaDialog has been accepted to MIDL 2025: The new paper includes additional experiments and details about [RaDialog_v2](https://github.com/ChantalMP/RaDialog_LLaVA) and can be found on [OpenReview](https://openreview.net/pdf?id=trUvr1gSNI)

- 12 July 2024: We published a **new version of our Instruct Datset** including additional tasks on [PhysioNet](https://physionet.org/content/radialog-instruct-dataset/1.1.0/)

- 29 May 2024: The **new and improved version of RaDialog** is now publically available on [Huggingface](https://huggingface.co/ChantalPellegrini/RaDialog-interactive-radiology-report-generation) and [Github](https://github.com/ChantalMP/RaDialog_LLaVA). This new version is much **better in conversational assistance**, **easier to use** and allows a simple **inference setup with huggingface**!

- 26 March 2024: RaDialog Instruct Dataset now available on [PhysioNet](https://physionet.org/content/radialog-instruct-dataset/1.0.0/)!

---

<img align="right" src="figs/example.png" alt="teaser" width="50%" style="margin-left: 20px">

Conversational AI tools that can generate and discuss clinically correct radiology reports for a given medical image have the potential to transform radiology. Such a human-in-the-loop radiology assistant could facilitate a collaborative diagnostic process, thus saving time and improving the quality of reports. Towards this goal, we introduce RaDialog, the first thoroughly evaluated and publicly available large vision-language model for radiology report generation and interactive dialog. RaDialog effectively integrates visual image features and structured pathology findings with a large language model (LLM) while simultaneously adapting it to a specialized domain using parameter-efficient fine-tuning. To keep the conversational abilities of the underlying LLM, we propose a comprehensive, semi-automatically labeled, image-grounded instruct dataset for chest X-ray radiology tasks. By training with this dataset, our method achieves state-of-the-art clinical correctness in report generation and shows impressive abilities in interactive tasks such as correcting reports and answering questions, serving as a foundational step toward clinical dialog systems.

## Installation

### Environment Setup:

#### 1) RaDialog Environment

- clone this repository and move to the radialog directory with `cd RaDialog`

- Install the RaDialog environment with `conda create --name radialog python=3.7`

- Activate the environment with `conda activate radialog`

- Install the requirements with `pip install -r requirements.txt`

- Install hl-ml-multimodal with `pip install hi-ml-multimodal==0.2.0`

- Reinstall correct versions of torch and transformers with `pip install torch==1.13.0 transformers==4.28.1`

- Install java and set JAVA_HOME and PATH in local_config.py (we used jre1.8.0)

#### 2) CheXbert Environment

- Install the CheXbert environment with `conda create --name chexbert python=3.7`

- Activate the environment with `conda activate chexbert`

- Move to the chexbert directory with `cd chexbert`

- Install the requirements with `pip install -r requirements.txt`

- Set the absolute path to the chexbert env and folder in `RaDialog/local_config.py`

### Prepare the Data and Models:

#### 1) Download pretrained models

- Download the pretrained models from [here](https://github.com/ChantalMP/RaDialog/releases/tag/weights)

- place chexbert.pth in RaDialog/chexbert/src/checkpoint/

- unzip vicuna-7b-img-instruct.zip and vicuna-7b-img-report.zip and place folders into RaDialog/checkpoints/

- unzip chexpert_train and place folder into RaDialog/findings_classifier/checkpoints/

- unzip embs and place folder into RaDialog/pretraining/

- unzip checkpoint_4.pth and place it into outputs/stage1_pt_instruct_blip_origlr_img448/

#### 2) Download MIMIC-CXR

- Download the MIMIC-CXR-JPG dataset from [here](https://www.physionet.org/content/mimic-cxr-jpg/2.0.0/)

- The dataset should be saved in .../physionet.org/files/mimic-cxr-jpg

- Go to physionet.org/files/mimic-cxr-jpg/files/ and unzip mimic-cxr-2.0.0-split.csv.gz

- from [here](https://physionet.org/content/mimic-cxr/2.0.0/), dowload mimic-cxr-reports.zip

- unzip it and place the folder in the same directory as the MIMIC-CXR-JPG dataset (e.g. physionet.org/files/)

- in local_config.py set the path to the MIMIC-CXR dataset (e.g. .../physionet.org/files/)

- in model/lavis/defaults_report.yaml set the path to the MIMIC-CXR-JPG dataset (e.g. .../physionet.org/files/mimic-cxr-jpg/2.0.0 )

#### 3) Create sectioned report data

- go to the mimic-cxr folder in the code with `cd mimic-cxr`

- run `python create_section_files.py` to prepare the report data

- go back to the RaDialog directory with `cd ..`

#### 4) Prepare the instruct dataset

- As MIMIC-CXR needs a certified PhysioNet account to be accessed, we can not publish our instruct dataset directly.

- We are working on publishing the instruct dataset on PhysioNet. In the meantime, you can create an instruct dataset yourself by following the steps below or just use our pre-trained model.

- The MIMIC-NLE data has to be generated first, as it also contains protected data. Follow the instructions [here](https://github.com/maximek3/MIMIC-NLE/tree/main) to generate the MIMIC-NLE data and set the path to the MIMIC-NLE data in `local_config.py`.

- For the correction task, you can write us, then we can share the used incorrect predictions with you.

- To generate data without Correction or Reasoning (MIMIC-NLE), please comment our line 335 or 336 in "create_data.py" accordingly.

Data for RaDialog-RG:

- run `python -m data.create_data --mode "RG"` to generate the report generation dataset in the required format (no instruct data)

Data for RaDialog-INS:

- run `python -m data.create_data --mode "INS"` to generate the instruct dataset

### Run Demo:

- run `python demo.py --cfg-path pretraining/configs/blip2_pretrain_stage1_emb.yaml` to start the demo

- connect to the demo with a browser at `http://127.0.0.1:7860` and start chatting with RaDialog

### Evaluate RaDialog on MIMIC-CXR test set:

- RaDialog-RG: run `python test.py --prompt img_matching_examples_ig2_noexamples_IMG_findings --use_embs --num_workers 0 --lora_model checkpoints/vicuna-7b-img-report/checkpoint-11200`

- RaDialog-INS: run `python test.py --prompt img_matching_examples_ig2_noexamples_IMG_findings --use_embs --num_workers 0 --lora_model checkpoints/vicuna-7b-img-instruct/checkpoint-4800`

- RaDialog-INS (correction): run `python test.py --prompt img_matching_examples_ig2_noexamples_IMG_findings --use_embs --num_workers 0 --lora_model checkpoints/vicuna-7b-img-instruct/checkpoint-4800 --do_corr` 

- RaDialog-INS (findings QA): run `python test.py --prompt img_matching_examples_ig2_noexamples_IMG_findings --use_embs --num_workers 0 --lora_model checkpoints/vicuna-7b-img-instruct/checkpoint-4800 --do_cp_all_qa` (or --do_cp_bin_qa)

### Train RaDialog:

#### 1) CheXbert classifier Training

- run `python -m findings_classifier.chexpert_train --train --run_name "train_chexbert"`

- in chexpert_train.py set ckpt_path (line 152) to the path of the trained model you just trained

- then run `python -m findings_classifier.chexpert_train --run_name "save_preds"` to save the predictions of the trained model

#### 2) Alignment Module Pretraining

- run `python -m pretraining.train --cfg-path pretraining/configs/blip2_pretrain_stage1.yaml`, we used the 4th epoch checkpoint

- run `python -m pretraining.train --cfg-path pretraining/configs/blip2_pretrain_stage1_emb.yaml`, to save the embeddings of the trained model

#### 3) LLM Training

Train RaDialog-RG:

- run `python finetune.py --use_embs True --base_model 'vicuna_v7' --output_dir 'checkpoints/lora-vicuna-7b-report' --wandb_run_name lora-vicuna-7b-report --prompt_template_name vicuna_v11 --data_path "data/data_files/mimic_cxr_reports_stratified.json" --cutoff_len 600 --num_epochs 10`

- we used checkpoint-11200

Train RaDialog-INS:

- run `python finetune.py --use_embs True --base_model 'vicuna_v7' --output_dir 'checkpoints/lora-vicuna-7b-instruct' --wandb_run_name lora-vicuna-7b-instruct --prompt_template_name vicuna_v11 --data_path "data/data_files/mimic_cxr_instruct_stratified.json" --cutoff_len 800 --num_epochs 10`

- we used checkpoint-4800

To use a model from a checkpoint, you'll need to perform the following steps:

- make a copy of "pytorch_model.bin" and rename it to "adapter_model.bin"

- copy adapter_config.json to the checkpoint folder (it will be generated after the last epoch or you can copy it from the checkpoints we provide) 

## Reference

When using our model or dataset, please cite:

```

@article{pellegrini2023radialog,

  title={RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance},

  author={Pellegrini, Chantal and {\"O}zsoy, Ege and Busam, Benjamin and Navab, Nassir and Keicher, Matthias},

  journal={arXiv preprint arXiv:2311.18681},

  year={2023}

}

```