Tutorial 8: Multiple-Instance Learning

In contrast with tutorials 1-4, which focused on training and evaluating traditional tile-based models, this tutorial provides an example of training a multiple-instance learning (MIL) model. MIL models are particularly useful for heterogeneous tumors, when only parts of a whole-slide image may carry a distinctive histological signature. In this tutorial, we'll train a MIL model to predict the ER status of breast cancer patients from whole slide images. Note: MIL models require PyTorch.

We'll start the same way as :ref:`tutorial1`, loading a project and preparing a dataset.

>>> import slideflow as sf
>>> P = sf.load_project('/home/er_project')
>>> dataset = P.dataset(
...   tile_px=256,
...   tile_um=128,
...   filters={
...     'er_status_by_ihc': ['Positive', 'Negative']
... })

If tiles have not yet been :ref:`extracted <filtering>` for this dataset, do that now.

>>> dataset.extract_tiles(qc='otsu')

Once a dataset has been prepared, the next step in training an MIL model is :ref:`converting images into features <mil>`. For this example, we'll use the pretrained Virchow feature extractor, a vision transformer pretrained on 1.5M whole-slide images. Virchow has an input size of 224x224, so our images will be resized to match.

>>> virchow = sf.build_feature_extractor('virchow', center_crop=True)
>>> virchow.cite()
@misc{vorontsov2024virchowmillionslidedigitalpathology,
    title={Virchow: A Million-Slide Digital Pathology Foundation Model},
    author={Eugene Vorontsov and Alican Bozkurt and Adam Casson and George Shaikovski and Michal Zelechowski and Siqi Liu and Kristen Severson and Eric Zimmermann and James Hall and Neil Tenenholtz and Nicolo Fusi and Philippe Mathieu and Alexander van Eck and Donghun Lee and Julian Viret and Eric Robert and Yi Kan Wang and Jeremy D. Kunz and Matthew C. H. Lee and Jan Bernhard and Ran A. Godrich and Gerard Oakley and Ewan Millar and Matthew Hanna and Juan Retamero and William A. Moye and Razik Yousfi and Christopher Kanan and David Klimstra and Brandon Rothrock and Thomas J. Fuchs},
    year={2024},
    eprint={2309.07778},
    archivePrefix={arXiv},
    primaryClass={eess.IV},
    url={https://arxiv.org/abs/2309.07778},
}
>>> virchow.num_features
2560

The Virchow feature extractor produces a 2560-dimensional vector for each tile. We can generate and export :ref:`bags <bags>` of these features for all slides in our dataset using :func:`slideflow.Project.generate_feature_bags`.

>>> P.generate_feature_bags(
...     virchow,
...     dataset,
...     outdir='/bags/path'
... )

The output directory, /bags/path, should look like:

/bags/path
├── slide1.pt
├── slide1.indez.npz
├── slide2.pt
├── slide2.index.npz
├── ...
└── bags_config.json

The *.pt files contain the feature vectors for tiles in each slide, and the *.index.npz files contain the corresponding X, Y coordinates for each tile. The bags_config.json file contains the feature extractor configuration.

The next step is to create an MIL model configuration using :func:`slideflow.mil.mil_config`, specifying the architecture and relevant hyperparameters. For the architecture, we'll use :class:`slideflow.mil.models.Attention_MIL`. For the hyperparameters, we'll use a learning rate of 1e-4, a batch size of 32, 1cycle learning rate scheduling, and train for 10 epochs.

>>> from slideflow.mil import mil_config
>>> config = mil_config(
...     model='attention_mil',
...     lr=1e-4,
...     batch_size=32,
...     epochs=10,
...     fit_one_cycle=True
... )

Finally, we can train the model using :func:`slideflow.mil.train_mil`. We'll split our dataset into 70% training and 30% validation, training to the outcome "er_status_by_ihc" and saving the model to /model/path.

>>> from slideflow.mil import train_mil
>>> train, val = dataset.split(labels='er_status_by_ihc', val_fraction=0.3)
>>> train_mil(
...     config,
...     train_dataset=train,
...     val_dataset=val,
...     outcomes='er_status_by_ihc',
...     bags='/bags/path',
...     outdir='/model/path'
... )

During training, you'll see the training/validation loss and validation AUROC for each epoch. At the end of training, you'll see the validation metrics for each outcome.

After training has completed, the output directory, /model/path, should look like:

/model/path
├── attention
│   ├── slide1_att.npz
│   └── ...
├── models
│   └── best_valid.pth
├── history.csv
├── mil_params.json
├── predictions.parquet
└── slide_manifest.csv

The final model weights are saved in models/best_valid.pth. Validation dataset predictions are saved in the "predictions.parquet" file. A manifest of training/validation data is saved in the "slide_manifest.csv" file, and training history is saved in the "history.csv" file. Attention values for all tiles in each slide are saved in the attention/ directory.

The final saved model can be used for evaluation (:class:`slideflow.mil.eval_mil`) or inference (:class:`slideflow.mil.predict_slide` or :ref:`Slideflow Studio <studio_mil>`). The saved model path should be referenced by the parent directory (in this case, "/model/path") rather than the model file itself. For more information on MIL models, see :ref:`mil`.