In digital pathology, input data is often exceedingly too large for DL models to process directly, with Whole Slide Images (WSI) around 100k x 100k pixels. This post provides a quantitative and qualitative method, with code, to help optimize important digital pathology specific hyperparameters: patch size and magnification. Optimizing these variables can decrease training times, lowers hardware requirements, and reduces the amount of data required to effectively train a model.

This blog post was written by Kien Rea kien.rea@case.edu

Introduction

The training deep of learning models typically requires the ability to analyze a “batch” of images, essentially a group of small images or “patches”, stored in a Graphic Processing Unit’s (GPU) RAM. In digital pathology (DP), the input data is often exceedingly large, with Whole Slide Images (WSI) around 100k x 100k pixels and can thus be upwards of 2GB in compressed file size (30GB uncompressed).  As a result of this massive storage requirement, specialized approaches are needed to apply DL to DP images.

While recent approaches have been proposed which can operate on WSI directly, using feature extraction techniques (e.g. Streaming CNN [link] and Gigapixel NIC [link]), the most common way still to assign class labels to histologic primitives of interest remains extracting patches around these Objects of Interest (OOI) from within the WSI. Common patch sizes can range between 32×32 px, for smaller objects like cells, to larger 512×512 px typically associated with more complex regions or multicellular objects (e.g. cancer region detection, or tubule classification).

Therefore, in DP, there is a balance to be found between the apparent magnification and the associated patch size. We do not want patches too large because that can increase the training time while possessing unneeded context. On the other hand, patches which are too small may not provide enough context to produce a sufficient signal when classifying the OOI. Images can also be downscaled to decrease patch size. However, downscaling too far will result in the loss of relevant details that would be useful for classifying OOI.

It is likely that a model with larger patches will perform well but the issue is that large patches tend to unnecessarily increase model sizes. This will result in longer training times, larger GPU memory requirements, and more data required in order to train sufficiently. If similar performance could be achieved at a smaller patch size, then employing larger patches would be a waste of resources/overhead. Practically speaking, decreasing model sizes will lower hardware requirements and lower costs associated with running or training.

To help find the balance between patch size and magnification, this post will look at ways to visualize a dataset to help identify a suitable patch size and magnification for efficiently developing DL algorithms for DP image analysis.

First Steps

Understanding what features/properties of the OOI are driving its classification can be extremely helpful. In DP these concerns can be understood by asking, “what details would a pathologist look for if they were to perform this classification?” With this information in hand, next can be asked, how much “context” is required to contain this information, and what resolution retains sufficient details.

For example, when classifying glomeruli as normal versus segmentally or globally sclerotic, the OOI are multicellular and require a wider context to distinguish tissue patterns. Essentially, we would want a larger patch size so that the entire glomeruli could fit into it. On the other hand, when trying to classify individual cell-types (e.g., lymphocytes versus non-lymphocytes) a higher magnification (working on the cellular level) is likely required with less overall tissue area/context. This results in a smaller patch at higher magnification.

Context and patch size are often at odds with each other. Fixing resolution, it is easy to see lots of context at low detail, or lots of detail with minimal context. Finding a suitable balance, at least to begin with, based on domain expertise will already provide a good place to optimize from. The main goal is for most (if not all) patches to include enough task-specific signal for a classifier to pick up on.

Keep in mind that different tasks will likely require different configurations!

Adjusting Magnification 

If we need to lower the magnification, we can downscale (or downsample) the associated image. This reduction in resolution can be calculated by: (desired magnification)/(slide magnification) = (downscale factor). For example, if a slide was scanned at 40x magnification, reducing to 20x magnification requires a downsampling of half (0.5) from the resolution of the original image.

Magnification and patch size are also related. Suppose we took a 256×256 px patch at 40x magnification then took another 256×256 px patch at the same location but at 20x magnification, as shown in figure below. The 20x patch would look “zoomed out”, i.e. include more context at the expense of the clarity of the OOI.

This means, if you want to include more context, you can either increase patch size (preserve OOI resolution) or decrease magnification (preserve patch size).

Adjusting Patch Size

The goal is to reduce patch size in order to increase the model’s speed and decrease hardware requirements. Decreasing patch size is effectively cropping out a smaller section of the OOI (typically centered at the same location). This preserves detail at the expense of context and risks cutting out useful information. If decreasing patch size will result in the loss of useful context, magnification can be reduced simultaneously in order to fix viewing area at the expense of detail. An analog to the magnification equation is: (desired patch size)/(slide patch size) = (downscale factor). For example, in order for a 64×64 px patch to match the viewing area of a 256×256 px patch, the slide will need to be downsampled by a quarter (0.25). This drop in clarity from downsampling 40x to 10x can be visualized with the example below.

Overview of Magnification vs Patch Size Relationship

• Patch size needs to be increased if:
  • Additional context is needed for a relevant classification signal.
  • Higher resolution is needed to distinguish relevant features.

• Patch size needs to be decreased if:
  • GPU memory limits are preventing training.
  • Data transfer times to GPU are prohibitively long.
  • Additional context is not needed for a relevant classification signal.

• Magnification needs to be increased if a higher resolution is needed to distinguish relevant features.

• Magnification needs to be decreased if additional context is needed for a relevant classification signal.

As a proof of concept, we examined a tubule classification example. We took patches containing tubules from PAS stained kidney tissue and assigned them into proximal versus distal categories. The goal is to find a patch size and magnification combination which contains sufficient information for our DL classifier to learn and then predict the right tubule type. Each patch will thus contain a single centered tubule which has been cropped from the larger WSI.

Process

1. Load WSI and mask, then calculate morphological metrics (height, width, etc.) that will give information about image metadata.
2. Find a magnification that gives just enough detail to classify your data.
3. Find a patch size that will include enough of the OOI and context.

Part 1: Preparation

Import libraries:

import skimage
import tables
import os,sys
import glob
import math
import random
import PIL
import numpy as np
import cv2
import sklearn.feature_extraction.image
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle

%matplotlib inline
%matplotlib widget

The Tubule dataset for this example is saved in a number of PNG images in a single directory. The mask images are saved in another directory with the same filename as each corresponding slide image with “_mask” added at the end. Knowing this, the filenames for slide and mask images can be loaded in.

# Collect image and mask filenames
image_files=glob.glob('./Tubules/PAS/*.png')
files = [(x, x.replace("./Tubules/PAS/","./Tubules/mask/").replace(".png","_mask.png")) for x in image_files]

initial_magnification = 40 # Magnification of the original image

After retrieving the filenames, each mask image is scanned for OOI and metrics including area, centroid location, and the largest of either width or height is recorded. Optionally, patches too close to the edge can be filtered to fix cropping issues in the future.

areas = [] # Area of each OOI
rect_max = [] # Vertical or Horizontal dimension of each OOI (whichever is larger)
centroids = [] # Centroid of each OOI
image_index = [] # Which image each OOI is on
patch_label = [] # Track mask value for each OOI
edge_patch_thresh = 256 # Ignore OOI too close to the edge of the slide image (0 to disable)

# Gather metrics from mask images
for i,(_,fname) in enumerate(files):
    img_mask = cv2.cvtColor(cv2.imread(fname), cv2.COLOR_BGR2RGB)[:, :, 0].squeeze()
    img_mask_measure = skimage.measure.label(img_mask)
    rp = skimage.measure.regionprops(img_mask_measure)
    used_patches = 0
    for r in rp:
        if (r.centroid[0] >= edge_patch_thresh and r.centroid[1] >= edge_patch_thresh and 
            r.centroid[0] <= img_mask.shape[0] - edge_patch_thresh and r.centroid[1] <= img_mask.shape[1] - edge_patch_thresh):
            image_index.append(i)
            areas.append(r.area)
            rect_max.append(max([r.bbox[2]-r.bbox[0],r.bbox[3]-r.bbox[1]]))
            centroids.append(r.centroid)
            patch_label.append(img_mask[r.coords[0,0],r.coords[0,1]])
            used_patches += 1
    print(f"fname: {fname}, \tNumber of total patches: {len(rp)},\tNumber of usable patches:{used_patches}")

total_ooi = len(image_index)
areas = np.asarray(areas)
rect_max = np.asarray(rect_max)
centroids = np.asarray(centroids)

Part 2: Magnification Test

Next, we need to test out a few magnifications to rule out ones that result in patches that are too pixelated. From a speed standpoint, it is best to go with the lowest usable magnification because patch sizes will decrease as well. We visualize a variety of patches, some random and some at different relative sizes, and see how each magnification changes their level of detail.

First, list some magnifications you want to test in the trial_magnifications list. Optimally, add some padding and apply a mask to provide context or make OOI clearer. To choose a minimum magnification, find the lowest magnification for each patch where relevant features are still distinguishable. Notice the detail of smaller patches are most affected by changing magnification.

Second, decide the minimum magnification such that patches retain enough relevant detail to generate a good signal for classification. Note that losing too much clarity, beyond the point of reliable human classification, does not necessarily mean a model will perform badly, but this becomes more likely. If a model can perform well with lower magnification, patch size can decrease so speed improves. However, this no longer guarantees the model will have sufficient signal to perform its own classification, so proceed with caution.

# Find a magnification that works best (Clarity Test)

trial_magnifications = [30, 20, 10, 5] # What magnifications to test
padding = 10 # Number of pixels of context to each OOI
mask = False # Visualize with or without mask applied

# Create a matrix of OOI, left column is the original magnification, each column to the right is a different magnification

a = plt.figure(figsize=(15,30))
plt.tight_layout(pad = 0.5)
dim = (10, 1 + len(trial_magnifications))

def compare_plot(row, index, y_label = "patch "):
    half_patch_size = rect_max[index]//2
    dx = [int(centroids[index][0] - half_patch_size - padding),int(centroids[index][0] + half_patch_size + padding)]
    dy = [int(centroids[index][1] - half_patch_size - padding),int(centroids[index][1] + half_patch_size + padding)]
    offset = -1*min(dx[0],dx[1],dy[0],dy[1],0)
    dx[0] += offset
    dx[1] += offset
    dy[0] += offset
    dy[1] += offset
    ax = plt.subplot2grid(dim, (row, 0))
    image = cv2.cvtColor(cv2.imread(files[image_index[index]][0]),cv2.COLOR_BGR2RGB)
    image = cv2.copyMakeBorder(image, offset, offset, offset, offset, cv2.BORDER_CONSTANT)
    patch = image[dx[0]:dx[1], dy[0]:dy[1]]
    if mask:
        img_mask = cv2.cvtColor(cv2.imread(files[image_index[index]][1]), cv2.COLOR_BGR2RGB)
        img_mask = cv2.copyMakeBorder(img_mask, offset, offset, offset, offset, cv2.BORDER_CONSTANT)[dx[0]:dx[1],dy[0]:dy[1],0].squeeze()
        img_mask = img_mask == patch_label[index]
        patch = np.multiply(patch, img_mask[:,:,None])
    plt.imshow(patch)
    ax.set_ylabel(y_label + str(index))
    if row == 0:
        ax.set_title("Original | "+str(initial_magnification)+"x")
    for j, mag in enumerate(trial_magnifications):
        patch_size_resize = int((rect_max[index] + 2*padding)*mag/initial_magnification)
        patch_resize = cv2.resize(patch, (patch_size_resize, patch_size_resize), interpolation = cv2.INTER_NEAREST)
        ax = plt.subplot2grid(dim, (row, j + 1))
        plt.imshow(patch_resize)
        if row == 0:
            ax.set_title(str(mag)+"x")

# First 4 OOI are random
for i in range(0,4):
    compare_plot(i, random.randint(0, total_ooi - 1))
        
# 5th OOI is smallest 1% by area, 6th is smallest 5% by area, 7th is smallest 10% by area, 8th is smallest 15% by area
compare_plot(4, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 1))), y_label = "Smallest | 1% | patch ")
compare_plot(5, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 5))), y_label = "Smallest | 5% | patch ")
compare_plot(6, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 10))), y_label = "Smallest | 10% | patch ")
compare_plot(7, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 15))), y_label = "Smallest | 15% | patch ")

# 9th is average area, 10th is largest 5% by area 
compare_plot(8, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 50))), y_label = "Average | 50% | patch ")
compare_plot(9, np.argmin(np.abs(np.array(areas)-np.percentile(areas, 95))), y_label = "Largest | 95% | patch ")

plt.show()

This plot shows 10 patches across the rows and the magnifications we want to test across the columns. The first 4 patches are random, the last 6 patches are at different size percentiles: 1, 5, 10, 15, 50, and 95% (100% is the largest patch). In this case, most patches are clear at 10x, but a few smaller patches become questionable. An ideal magnification looks to be somewhere between 10x and 20x. Consider changing the trial_magnifications list to some values between 10 and 20x and re-run this section to home in on an ideal magnification. For the sake of this post, we will stick with 20x.

Part 3: Patch Size Test

First, we need to define our desired magnification (20x in this case) and recalculate our metrics from earlier. Afterward, we can plot histograms based on patch area and maximum horizontal/vertical dimensions to see which patch sizes fit our OOIs (based on the shape and size of mask regions). It is useful to see cutoffs for some example patch sizes. These can be added to the mark_patchsizes variable with corresponding colors.

# Choose a final magnification and update patch metrics
final_magnification = 20

areas_final = areas*(final_magnification/initial_magnification)**2
rect_final = rect_max*(final_magnification/initial_magnification)
resize = final_magnification/initial_magnification
centroids_final = centroids*(final_magnification/initial_magnification)

# Find a patch size that works the best (Size Test)

mark_patchsizes = [(512,"yellow"),(256,"green"),(128,"orange"),(64,"red")] # What patch sizes to test
bins = 40 # Number of bins in histograms

# Visualize patch areas compared to the largest circle that can fit in each patch size
areas_total = areas_final[areas_final < 220000] # Filter outliers to make graphs readable
a = plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1)
plt.hist(areas_total,bins=bins)
for ps,color in mark_patchsizes:
    plt.axvline(x=math.pi*((ps/2)**2), color=color)
plt.title("Areas")

# Visualize patch maximum vertical/horizontal dimensions compared to each patch size
rect_total = rect_final[rect_final < 600] # Filter outliers to make graphs readable
plt.subplot(1, 2, 2)
plt.hist(rect_total,bins=bins)
for ps,color in mark_patchsizes:
    plt.axvline(x=ps, color=color)
plt.title("Normal Axis Lengths")
plt.show()

# Print the number of patches that will be cut off at each patch size
#patchsizes = [x[0] for x in mark_patchsizes]
mark_patchsizes.sort()
for dim,_ in mark_patchsizes:
    i = len(rect_final[rect_final<=dim])
    print(f"{i} / {total_ooi}, {int(100*i/total_ooi)}% fit in {dim}px patch size")

The resulting graphs and statistics show that almost all patches fit in 512×512 px patch size and a good number also fit in 256×256 px. Before jumping to 512×512 px, know that the relationship between increasing patch size and area is x², which means 512×512 px patches are exponentially larger in terms of area than 256×256 px patches. Also consider the area of patches; most patches are much smaller than the largest circle that fits in a 256×256 or 512×512 px patch sizes. This means many areas in these patches might not contain useful context. We look at some patch examples next to visualize this point.

Last, we look at some examples where patches get cutoff at each test patch size.

# Create a matrix of OOI that are cutoff at each patch size in mark_patchsizes, each column is a different patch size
# A patch cutoff at a larger patch size in the mark_patchsizes will not be included in smaller patch sizes

mask = True # Visualize with or without mask applied
padding = 64 # Visualize at larger patch size to show what is cutoff
example_count = 5 # Number of cutoff examples per patch size

mark_patchsizes.sort(reverse=1)
a = plt.figure(figsize=(15,20))
plt.tight_layout(pad = 0.5)
dim = (example_count,len(mark_patchsizes))
rect_cutoff = rect_final.copy()


for i,(ps,_) in enumerate(mark_patchsizes):
    half_patch_size = ps//2 + padding
    for j in range(0, example_count):
        image = None
        while np.shape(image) != (ps + 2*padding,ps + 2*padding,3) and len(rect_cutoff[rect_cutoff >= ps]) > 0:
            index = random.choice(np.where(rect_cutoff >= ps)[0])
            image = cv2.cvtColor(cv2.imread(files[image_index[index]][0]),cv2.COLOR_BGR2RGB)
            image = cv2.resize(image, (int(image.shape[1]*resize), int(image.shape[0]*resize)),interpolation = cv2.INTER_NEAREST)
            dx = (int(centroids_final[index][0] - half_patch_size),int(centroids_final[index][0] + half_patch_size))
            dy = (int(centroids_final[index][1] - half_patch_size),int(centroids_final[index][1] + half_patch_size))
            image = image[dx[0]:dx[1], dy[0]:dy[1]]
            rect_cutoff[index] = 0
        if len(rect_cutoff[rect_cutoff >= ps]) > 0:
            if mask:
                img_mask = cv2.cvtColor(cv2.imread(files[image_index[index]][1]), cv2.COLOR_BGR2RGB)
                img_mask = cv2.resize(img_mask, (int(img_mask.shape[1]*resize), int(img_mask.shape[0]*resize)),
                                      interpolation = cv2.INTER_NEAREST)[dx[0]:dx[1], dy[0]:dy[1], 0].squeeze()
                img_mask = img_mask == patch_label[index]
                image = np.multiply(image, img_mask[:,:,None])
            ax = plt.subplot2grid(dim, (j, i))
            if j == 0:
                ax.set_title(f"Cutoff by Patch Size {ps}px")
            plt.imshow(image)
            for ps2,color in mark_patchsizes:
                if ps2  ps] = 0

plt.show()

This plot shows patch size across the columns and examples of patches that are cutoff at that magnification (but would not have been if a magnification larger was chosen instead) along the rows. This plot also outlines boxes that show where the cutoff would be for that magnification and smaller. From above, 128×128 px patch size and smaller cuts off a noticeable amount of useful data so it is unlikely anything 128×128 px or lower will be a good choice. Most OOI that are cutoff by 512×512 px patches still retain enough relevant detail to classify. This is also the case for 256×256 px patches. Furthermore, OOIs cutoff by 512px still have relevant details at 256px (outlined by the green box in the left column). Given the significant performance decrease going from 256px to 512px, it makes the most sense to go with 256px patch size out of these options. Like before, feel free to re-run this section with different patch sizes in mark_patches in order to home in on an ideal patch size.

Yellow: 512px, Green: 256px, Orange: 128px, Red: 64px

Conclusion

The real-world usefulness of a deep learning model relies on a few factors: how useful the results are from the model (accuracy/application), can the model run in a reasonable amount of time, and will the users have access to sufficient hardware to run the model. One factor that plays a role in all three of these metrics is the formatting of the dataset. In digital pathology especially, hardware begins to pose limitations. When dealing with multiple WSI we often resort to techniques like smaller patch extraction around OOI. The smaller the patch, the smaller the potential model. This results in faster training/runtime, smaller datasets, and lower hardware requirements.

However, there exists a lower bound on patch size while maintaining model accuracy. We must tailor each dataset/application individually due to the level of variation in a field like histology. So, in this post we went through a process in order to help discover the ideal patch size and magnification for a given dataset. First, we found the lowest magnification that retained relevant details. Second, we extracted patches at different sizes with this magnification to find the lowest patch size that retained a useful amount of relevant data/context. Visualizations, graphs, and statistics were used to interpret how each patch size/magnification combination fits the entire dataset.

The code used in this post might be useful for others as a jumping-off point for their own investigations, thus it is available here.