Although classification metrics are good for summarizing a model’s performance on a dataset, they disconnect the user from the data itself. Similarly, a confusion matrix might tell us that performance is suffering because of false positives, but it obscures information about what patterns may have caused those misclassifications and what types of false positives there might be. 

One way to gain interpretability is to group sampled images by the category of their output (true negative, false negative, false positive, true positive), and display them in a powerpoint file for facile review. These visualizable categories make it easy to identify patterns in misclassified data that can be exploited to improve performance (e.g., hard negative mining, or image analysis based filtering).

This blog post describes and demonstrates a workflow that produces such a powerpoint slide deck automatically for review, as shown below:

The complete workflow consists of two parts:

1. Perform Image prediction
2. Powerpoint slide deck generation

For some users, Part 1 of the workflow might not be necessary. This is the case if you already have a .pytable file with the following datasets (pytable terminology for named stored arrays): “labels”, “filenames”, “imgs”, and “predictions”. If not, Part 1 provides a script for generating a predictions dataset using the other three. Users who have completed the “Digital Pathology Classification Using PyTorch + DenseNet” blog post, will find both parts to work seamlessly with their existing workflow.

Part 1: Perform Image Prediction

The generate_densenet_predictions.py script created a “predictions” dataset in the provided .pytable file. This column will eventually contain the classification results from the user-provided model when applied to the patch.

Here is a template for running the script at the command line:

python3 generate_densenet_predictions.py [pytable path] [model checkpoint path] --patch_size [patch size] --gpuid [gpu id] --batch_size [batch size]

Script Requirements

1. A .pytable file with the following keys: 
   • “labels”: returns an array of integers in the range [0,#classes – 1]
   • “filenames”: returns an array of strings corresponding to image filepaths
   • “imgs”: returns an array of images with the dimension (width, height, n_channels)
2. A pytorch model checkpoint file.
   • To use this script with a model other than Densenet, pay attention to the model initialization step of the code walkthrough.

Code Walkthrough

This script is similar to the train_densenet_albumentations.py script in the aforementioned DenseNet blog post.

We begin by defining a Dataset class. This class is the interface from a user’s data storage to PyTorch’s DataLoader class. We will initialize a Dataset object later, but know that the __init__() method takes an image transform object and a .pytable file path. 

Note that the Dataset class defined below is different from the pytable dataset that we introduced earlier, and for that reason we format the two differently.

Next we check the .pytable file for a “predictions” dataset. If such a dataset already exists, the user should be aware that it will be overwritten.

Then we set the device to the specified gpu, if available:

Next, we compose an image transform object. We have chosen to apply just one transform, the CenterCrop. The reason for using this transform is to crop each image to 224×224, which is required by Densenet.

Then we initialize a Dataset and pass it into a PyTorch DataLoader. Note that the DataLoader takes args.batch_size as an argument. A higher value for args.batch_size will speed up the inference process, and the argument’s upper bound will depend on your machine. In the code, the default batch size is set to 32.

Now we have reached the model initialization step. Here we load the model checkpoint, and pass its hyper-parameters into a DenseNet. If you are using this script for a different kind of PyTorch model, import your model and set model = YourModel(…)

We finish the model initialization by loading the weights dictionary, setting the model to evaluation mode, and sending it to the preset device:

We are ready to generate an array of predictions! We load each image batch into the model and append each respective output (a batch of predicted labels) to the predictions array.

Once all of the predictions have been generated, we check for a pre-existing “predictions” dataset in the the pytable and remove it. Finally we create a new “predictions” dataset in the pytable and fill it with the contents of our predictions array.

With this set of predictions we can use the second script.

Part 2: Powerpoint slide deck generation

The visualize_classification_groups.py script outputs a .pptx file at the path specified by the ppt_save argument. Here is a template for running the script at the command line:

python3 visualize_classification_groups.py [pytable path] [ppt_save] --str_criteria [“criteria1_name” “criteria2_name” … “criteriaN_name”] --criteria [d1d2 d3d4 …] --num_rows [a] --num_cols [b]

Importantly: the criteria item corresponds to a pptx slide you want created. It should be a pair of integer digits, e.g., the item “01” tells the script to produce a slide of images with a ground truth label of 0 and prediction of 1.

Script Requirements

1. A .pytable file with the following keys: 
   • “labels”: stores an array of integers in the range [0, #classes – 1]
   • “filenames”: stores an array of strings corresponding to image filepaths
   • “imgs”: stores an array of images with the dimension (width, height, n_channels)
   • “predictions”: stores an array of integers in the range [0,#classes – 1]
2. For parsing simplicity, we assume that class labels are single-digit integers (0 through 9).

Code Walkthrough

We begin by defining the method addimagetoslide(). This method reads an image from a bytestream object and adds it to a powerpoint slide.

Then we parse the criteria arguments and create a dict with the following structure:

criteria_dict = {
        criteria1_name: [d1, d2],
        criteria2_name: [d3, d4],
        …
}

Then we read in the ground truth labels and predictions from the provided .pytable file.

We initialize a Powerpoint object, and define a coordinate grid using np.meshgrid. The dimensions of the grid are specified by the arguments num_rows and num_cols.

Now we loop through each classification group in the criteria_dict. For each group, we find all of the images that match the criteria, and choose at most num_rows * num_cols images.

For each classification group, we also append a new slide to the powerpoint and add the slide name.

For each new slide added, we loop through the selected images. For each image we attach a file name title, load the image into an image buffer, and add it to the slide.

We’re done! The powerpoint is ready to be saved.

Let’s test this script on the DenseNet workflow. Assuming that we are using the subtyping dataset from the DenseNet blog post and that the data-storage pytable has a predictions array, use this command to generate a four-slide powerpoint:

python3 visualize_classification_groups.py [pytable path] [ppt_save] --str_criteria “gt:CLL pred:CLL” “gt:CLL pred:FL” “gt:FL pred:FL” “gt:FL pred:MCL --criteria 00 01 11 12

The script saved a powerpoint with the following slides:

Conclusion

Now that we have a slide deck that groups several combinations of ground truth labels and predictions, we can hunt for patterns in the data that may lead our model to misclassify images!

I hope that this computational method for visualizing classification groups will be helpful in your own workflows. Feel free to leave a comment if you have any questions.

The code is freely available for both generate_densenet_predictions.py and visualize_classification_groups.py.

A Note on the Author and Contributions

Jackson Jacobs is a 4th year undergraduate student at Case Western Reserve University expecting to graduate in the spring of 2023. The focus of his research is to improve Multiple Instance Learning techniques for detecting cancer in Whole Slide Images.

This blog post was conceived of and edited by Andrew Janowczyk, and was written by Jackson Jacobs.