.. _evaluation:

Evaluation

==========

Slideflow includes several tools for evaluating trained models. In the next sections, we'll review how to evaluate a model on a held-out test set, generate predictions without ground-truth labels, and visualize predictions with heatmaps.

Evaluating a test set

*********************

The :meth:`slideflow.Project.evaluate` provides an easy interface for evaluating model performance on a held-out test set. Locate the saved model to evaluate (which will be in the project ``models/`` folder). :ref:`As with training <training_with_project>`, the dataset to evaluate can be specified using either the ``filters`` or ``dataset`` arguments. If neither is provided, all slides in the project will be evaluated.

.. code-block:: python

    # Method 1: specifying filters

    P.evaluate(

      model="/path/to/trained_model_epoch1",

      outcomes="tumor_type",

      filters={"dataset": ["test"]}

    )

    # Method 2: specify a dataset

    dataset = P.dataset(tile_px=299, tile_um='10x')

    test_dataset = dataset.filter({"dataset": ["test"]})

    P.evaluate(

      model="/path/to/trained_model_epoch1",

      outcomes="tumor_type",

      dataset=test_dataset

    )

Results are returned from the ``Project.evaluate()`` function as a dictionary and saved in the project evaluation directory. Tile-, slide-, and patient- level predictions are also saved in the corresponding project evaluation folder, ``eval/``.

Generating predictions

**********************

For a dataset

-------------

:meth:`slideflow.Project.predict` provides an interface for generating model predictions on an entire dataset. As above, locate the saved model from which to generate predictions, and specify the dataset with either ``filters`` or ``dataset`` arguments.

.. code-block:: python

    dfs = P.predict(

      model="/path/to/trained_model_epoch1",

      filters={"dataset": ["test"]}

    )

    print(dfs['patient'])

.. rst-class:: sphx-glr-script-out

    .. code-block:: none

                                patient  ...  cohort-y_pred1

        0    TCGA-05-4244-01Z-00-DX1...  ...        0.032608

        1    TCGA-05-4245-01Z-00-DX1...  ...        0.216634

        2    TCGA-05-4249-01Z-00-DX1...  ...        0.000858

        3    TCGA-05-4250-01Z-00-DX1...  ...        0.015915

        4    TCGA-05-4382-01Z-00-DX1...  ...        0.020700

        ..                          ...  ...             ...

        936  TCGA-O2-A52S-01Z-00-DX1...  ...        0.983500

        937  TCGA-O2-A52V-01Z-00-DX1...  ...        0.773328

        938  TCGA-O2-A52W-01Z-00-DX1...  ...        0.858558

        939  TCGA-S2-AA1A-01Z-00-DX1...  ...        0.000212

        940  TCGA-XC-AA0X-01Z-00-DX1...  ...        0.632612

Results are returned as a dictionary of pandas DataFrames (with the keys ``'tile'``, ``'slide'``, and ``'patient'`` for each level of prediction) and saved in the project evaluation directory, ``eval/``.

For a single slide

------------------

You can also generate predictions for a single slide with either :func:`slideflow.slide.predict` or :meth:`slideflow.WSI.predict`.

.. code-block:: python

    import slideflow as sf

    slide = '/path/to/slide.svs'

    model = '/path/to/model_epoch1'

    sf.slide.predict(slide, model)

.. rst-class:: sphx-glr-script-out

    .. code-block:: none

        array([0.84378019, 0.15622007])

The returned array has the shape ``(num_classes,)``, indicating the whole-slide prediction for each outcome category. If the model was trained with uncertainty quantification, this function will return two arrays; the first with predictions, the second with estimated uncertainty.

.. _generate_heatmaps:

Heatmaps

********

For a dataset

-------------

Predictive heatmaps can be created for an entire dataset using :meth:`slideflow.Project.generate_heatmaps`. Heatmaps will be saved and exported in the project directory. See the linked API documentation for arguments and customization.

.. code-block:: python

    P.generate_heatmaps(model="/path/to/trained_model_epoch1")

For a single slide

------------------

:class:`slideflow.Heatmap` provides more granular control for calculating and displaying a heatmap for a given slide. The required arguments are:

- ``slide``: Either a path to a slide, or a :class:`slideflow.WSI` object.

- ``model``: Path to a saved Slideflow model.

Additional keyword arguments can be used to customize and optimize the heatmap. In this example, we'll increase the batch size to 64 and allow multiprocessing by setting ``num_processes`` equal to our CPU core count, 16.

.. code-block:: python

    heatmap = sf.Heatmap(

      slide='/path/to/slide.svs',

      model='/path/to/model'

      batch_size=64,

      num_processes=16

    )

If ``slide`` is a :class:`slideflow.WSI`, the heatmap will be calculated only within non-masked areas and ROIs, if applicable.

.. code-block:: python

    from slideflow.slide import qc

    # Prepare the slide

    wsi = sf.WSI('slide.svs', tile_px=299, tile_um=302, rois='/path')

    wsi.qc([qc.Otsu(), qc.Gaussian()])

    # Generate a heatmap

    heatmap = sf.Heatmap(

      slide=wsi,

      model='/path/to/model'

      batch_size=64,

      num_processes=16

    )

If ``slide`` is a path to a slide, Regions of Interest can be provided through the optional ``roi_dir`` or ``rois`` arguments.

Once generated, heatmaps can be rendered and displayed (ie. in a Jupyter notebook) with :meth:`slideflow.Heatmap.plot`.

.. code-block:: python

    heatmap.plot(class_idx=0, cmap='inferno')

Insets showing zoomed-in portions of the heatmap can be added with :meth:`slideflow.Heatmap.add_inset`:

.. code-block:: python

    heatmap.add_inset(zoom=20, x=(10000, 10500), y=(2500, 3000), loc=1, axes=False)

    heatmap.add_inset(zoom=20, x=(12000, 12500), y=(7500, 8000), loc=3, axes=False)

    heatmap.plot(class_idx=0, mpp=1)

.. image:: heatmap_inset.jpg

|

Save rendered heatmaps for each outcome category with :meth:`slideflow.Heatmap.save`. The spatial map of predictions, as calculated across the input slide, can be accessed through ``Heatmap.predictions``. You can save the numpy array with calculated predictions (and uncertainty, if applicable) as an \*.npz file using :meth:`slideflow.Heatmap.save_npz`.