Data set creation for supervised deep learning–based analysis of microscopic images: Review of important considerations and recommendations

Images

WSI scanners are increasingly integrated into routine workflows of veterinary pathology laboratories; therefore, WSIs are readily available for DL-AIA. As WSIs are the most common image type used for data set creation in current veterinary pathology literature, this article will focus on WSIs. Other digitization devices (such as cameras mounted on light microscopes or mobile phones attached to the ocular) and nonlight microscopy imaging modalities (such as gross images or electron microscopy images) are not specifically addressed in this review due to the paucity of research in this area; however, many of the considerations addressed below are applicable to these modalities as well.

Key considerations for image sets are the following:

The selection of appropriate cases is critical, and clear inclusion and exclusion criteria should be defined that align with the intended use case of the DL-AIA. For example, inclusion of glass slides and/or WSIs from multiple laboratories may be considered to increase variability (see below).^12,13 Software for automated quality control processes, such as HistoQC, may help to identify unsuitable images with pronounced artifacts or other outlier features.^36,68 While these quality control tools are undoubtedly valuable for application data sets, caution should be exercised when removing cases with realistic artifacts and image variability from the primary data set. Image exclusion may be done when the image quality interferes with annotating the images with sufficient accuracy, and when images of poor tissue quality will also be excluded from the analysis data set. ¹⁰¹ In fact, it may be advantageous to intentionally oversample certain artifacts to improve the models’ ability to distinguish them from the pattern of interest (see below).

Determining the total number of cases to include is almost impossible in advance, as there are no established methods to predict how many images are required to achieve the target algorithmic performance. The necessary data set size depends, in the first instance, on the complexity and morphological variability of the pattern of interest (eg, a larger training data set may be required for detection of mitotic figures than for renal glomeruli), as well as other structures present in the images (eg, apoptotic cells as imposters for mitotic figures). Up to a certain point (when performance plateaus), models follow data scaling laws (ie, the more training data, the higher the performance); however, this is specific for the task, prohibiting general data size recommendations. There are generally 2 strategies—(1) maximal approach: include as many cases as can feasibly be annotated within the available time and resources, aiming for the highest possible algorithmic performance ²³ and (2) iterative approach: begin with a small training data set, then incrementally add more images as needed to reach the desired performance, typically focusing on those image features that are problematic for preliminary DL models. While concentrating on particularly difficult regions may reduce the amount of training data required (and thus the annotation budget; see below), this approach requires repeated model training and sufficient computational resources. For this second approach, it is essential to establish a sufficiently large test data set starting at the first iteration, otherwise reliable performance evaluation and an informed decision on the improvement over data set iterations are not possible. Another factor that may influence the data set size is the model architecture used. As compared with general purpose pretrained networks (eg, pretrained on ImageNet) that require end-to-end training, histopathology foundation models (ie, large-scale models pretrained using self-supervised learning on vast amounts, diverse, histopathology-specific data sets) promise to achieve high performance levels while requiring significantly less training data.^5,35 Related to their semantically rich feature vectors, foundation models can be adapted to numerous downstream image analysis tasks with minimal additional training using linear probing or low-rank adaptation. While foundation models have been shown to achieve outstanding performance and improve transferability to out-of-domain data sets, even with limited training data for the downstream task, they still follow general data scaling laws (ie, more data improves performance) and show domain-specific dependencies^{5,18,120,125,141}; thus, the relevance of an appropriate training data set (even if smaller than for other model types) should not be neglected. Most studies using foundation models focused on image classification; therefore, the performance and data scaling of foundation models for other pattern recognition tasks (eg, object detection and segmentation) are less well known. Apart from training data set quantity, foundation models require data sets with the same quality criteria of annotations, as discussed in the section below, and a large-scale test set for the same rigorous performance evaluation, as discussed in other articles.^21,64

Although it may be convenient to include multiple samples from the same patient, such images are unlikely to introduce the same degree of variability as images from different patients; thereby providing less benefit for model development and testing. It also needs to be considered that images from the same patient, even if they originate from different tissue blocks or cytologic smears, can only be part of one data subset (training, validation, or test data set), due to the high similarity. Data leakage, ie, distribution of data from the same patient across data subsets, would lead to overly optimistic performance evaluations ³² and must be avoided.

Image Variability and Domain Shift

Beyond the sheer number of cases, it is important that the images encompass the variability expected for the intended application (target domain), both for the pattern of interest (foreground) and for other background patterns present in the images. ¹⁴⁴ It is well-known that DL models can learn biological and technical features specific to the image batch(es) of the training data (hidden variables).^{11 –13,53,64,65,76,85,144} Models often fail to predict the pattern of interest if the image features differ too greatly from those presented during training (covariate domain shift). This image variability is related to the multi-step process of image creation, encompassing tissue acquisition, tissue processing, slide preparation, image creation, and image postprocessing, which often leads to a batch and laboratory specific image signature (Fig. 2).^{13,53,65,76,119,144} While there are several approaches to dealing with image variability, this section focuses on “natural augmentation” of the data set, thereby curating a diverse data set to support models intended for a broad target domain (as opposed to a narrow scope: eg, a single laboratory, a single tumor type, etc). Other approaches, not discussed here further, include (1) narrowing the target domain by image normalization to known physical features (eg, color characteristics) of a reference or color calibration across scanners, (2) diversifying the training images during model training (image augmentation/transformation using color and contrast adjustments, image distortion, rotation, or synthetic augmentation), (3) and specifically tailored training strategies. ⁶⁷ Whereas these tricks have been shown to improve model performance,^7,12,69 neither of them can fully circumvent all sources of domain shift and some may introduce new bias. Thus, it remains compulsory to include natural images of all relevant domains in the test sets.

Images from different species, including humans, often do not appear to cause major domain shifts, since tissues and cell morphologies are often similar.^8,59,92,93 However, caution is warranted for disease entities with species-specific features, for instance, the differing distribution of mammary carcinoma / breast cancer subtypes between dogs (with numerous mixed tumors) and humans, or the differences between red blood cell morphology between mammals and nonmammalian vertebrae with nucleated cells.

The included images should reflect the natural biological variability of tissue morphology.^64,85 This includes the need to represent all relevant disease entities, with potential oversampling of rare subtypes to counteract class imbalance. ¹⁴⁴ DL models usually do not generalize well across disease types (such as different tumor types) if they (or similar domains) were not included in training.^12,13,72 For example, a model trained to classify colon images as benign or malignant tumors showed a nearly 50% drop in performance when applied to the same task for breast and prostate tissue. ⁷² A mitotic figure object detection algorithm trained exclusively with soft tissue sarcoma images dropped in performance from an F1-score of 0.70 to 0.49 when applied to lymphoma cases (images from same laboratory and scanner), whereas the in-domain performance for lymphoma was F1 = 0.79. ¹³ In contrast, a DL model that was trained with multiple tumor types was able to detect mitotic figures with high performance in previously unseen tumor types, showing the higher robustness of models when trained with a diverse data set. ¹³ For example, models trained with multiple tumor types, but without soft tissue sarcomas, achieved an F1 = 0.65 in soft tissue sarcoma images, whereas other models that were trained with soft tissue sarcomas and other tumor types had an F1 between 0.67 and 0.69, ie, only 2 to 4 percentage points higher. ¹³

Even within the same tissue section/images, variability exists (intraimage variability). Consequently, it is an important consideration whether entire WSIs, individual tissue fragments, or smaller regions within an image should be annotated, balancing time available for annotation and the expected degree of variability between image regions. For example, a tumor section may contain hundreds of thousands of nuclei, and it is neither feasible nor sensible to annotate all of them. However, the selected image regions should—depending on the intended application of the algorithm—not only include ideal (from the diagnostic standpoint) tumor regions but also more challenging regions, like necrotic or inflamed tissue, adjacent normal tissue, and image artifacts, to ensure adequate model generalizability. For mitotic figures, it has been shown that a model trained only on ideal (pathologist-selected) regions of interest with a size of 10 high-power field (2.0–2.37 mm²) can effectively detect mitotic figures in similar regions, but may perform poorly in other image areas (eg, fatty tissue, thermal artifacts from cautery, or inking).^10,13 Thus, such an algorithm is not particularly suited to analyze entire WSIs and identify the tumor regions with the highest mitotic density. The size of regions of interest may be justified through the pathology task (eg, 10 high-power fields/2.0 mm² for mitotic figures object detection), ¹³ density of objects of interest, and by the time invest for annotations of each region (eg, only half a high-power field for tumor nuclei segmentation).^60,66 Given a fixed annotation time budget and, at the same time, the intention to maximize tissue variability, it is often advantageous to select relatively smaller regions of interest, but to sample several regions per image (from different tissue locations) and more cases in the data set.

Considering that there are everyday differences in slide preparation, particular attention should be given to producing samples across multiple batches (ie, at different time points) to ensure realistic variability in tissue processing steps, such as section thickness and staining composition, unless the model is intended solely for use in a highly controlled laboratory environment. It should be ensured that cases from all relevant image subgroups are distributed across the batches. As an alternative to the creation of new slides, images from a WSI archive may be used, which naturally includes the real-world variability across the retrospective sample inclusion period. ⁷⁴

Different WSI scanners (see Patel et al ¹⁰⁸ for a summary of the different scanners) have been identified as a significant source of domain shift in studies that digitized the same test slides with multiple scanners.^11,109,139 For example, a study on tumor segmentation models reported a decrease in the mean intersection over union (mIoU) of 0.38 percentage points when comparing test set performance of the in-domain scanner (mIoU = 0.82) with that of out-of-domain scanner (mIoU = 0.44). ¹³⁹ Another study, evaluating a model that segmented lymph node tissue from the surrounding fatty tissue, demonstrated a significant drop in performance when trained on one scanner and tested on another, with performance decreasing by a factor of 4 (in-domain Matthews correlation coefficient [MCC] = 0.81; out-of-domain MCC = 0.18). ⁷⁵ While WSIs from different scanners exhibit obvious differences in color distribution (Supplemental Figure S1), ¹³⁹ image postprocessing methods intended to counteract this source of variability, such as extensive color augmentation (increasing variability in the training set) or image normalization (reducing color variability), do not fully restore performance.^11,56 This indicates that WSIs from different scanners differ not only in color representation (and image resolution) but also in other features, possibly related to hardware (such as optics) or scanning methods (eg, depth of field, tile stitching, and noninclusion of nontissue areas that contain artifacts such as dust).^108,139 There are clear recommendations to use multiple scanners for data set creation, if the derived model is intended for a widespread application across multiple laboratories/studies.^11,144

The degree of algorithmic bias (performance drop related to a domain shift) of the other potential sources of domain shift listed in Fig. 2 has not, to our knowledge, been systematically evaluated. Whereas it is well-recognized that different laboratories produce different staining results, ⁴⁵ it is, to our knowledge, not well understood if default color augmentation or normalization methods during training can fully overcome this laboratory variability. In human medicine, several studies on AI models for classifying hematoxylin and eosin (HE) images based on mutational patterns have demonstrated significant performance differences among images from different demographic/ethnic groups, related to underrepresentation of patients from certain demographic groups in training data sets (demographic bias). ¹²⁷ Whether comparable differences exist among animal breeds remains unknown. Until further research reveals the impact of these unexplored variables, it may be advisable (depending on the intended application of the DL-AIA algorithm) to include these in data sets and particularly in test subsets.

Figure 3 illustrates examples of false algorithmic predictions produced by a mitotic figure detection model when applied to arbitrary tumor regions of WSIs. The detector follows a standard anchor-free object detection paradigm based on the FCOS (fully convolutional one-stage object detection) architecture ¹²⁶ and was trained on the MIDOG++ data set, ¹³ which is, to our knowledge, the largest and most diverse open resource for mitotic figure detection, spanning multiple tumor types, species, scanner vendors, and laboratory origins. Despite this diversity, the data set consists of ideal, pathologist-selected regions of interest (2.0 mm²) that are rich in viable tumor cells and contain only few artifacts. When deployed to less curated regions, the model becomes more sensitive to technical and biological artifacts that were previously unseen in training. Dust particles on the slide (Fig. 3a, b), out-of-focus areas (Fig. 3c, d), sebaceous glands and other nonneoplastic structures (Fig. 3e, f), cautery artifacts (Fig. 3i, j), and surgical ink deposits (Fig. 3k, l) can all exhibit dark, hyperchromatic profiles that closely resemble the shape of mitotic figures. Similarly, cells in underrepresented tissue types, such as chondrocytes in cartilage (Fig. 3g, h), superficial keratinocytes (Fig. 3m, n), and necrotic debris (Fig. 3o, p), mimic texture and intensity patterns learned by the detector, especially when the surrounding cellular morphology is degraded or missing. The principal causes of these false algorithmic predictions are (1) a domain shift between the “ideal” training regions and the more heterogeneous data of other region in the WSIs, (2) the presence of nontumor or low-quality tissue that was underrepresented in the training data, and (3) genuine morphological ambiguity between mitotic figures and certain artifacts or cell death phenomena. This underscores the need for training data sets and model designs that explicitly account for such technical and biological variability when the model is intended for whole-slide inference.

OPEN IN VIEWER

Figure 3.

Examples of biological and technical variability of images between training and test data sets that have caused false algorithmic predictions for detection of mitotic figures. The object detection model (based on the FCOS architecture ¹²⁶ ) was trained with the open-access MIDOG++ data set, ¹³ which only consists of hand-picked, “ideal” tumor region; thus, not completely covering the variability experienced in an entire whole-slide image. A panel shows either the broader tissue context of the hematoxylin and eosin-stained image or the model’s false prediction highlighted in the corresponding zoomed-in panel next to it. (a, b) Dust. (c, d) Out-of-focus. (e, f) Sebaceous glands. (g, h) Chondrocyte in cartilage. (h, i) Cautery artifacts. (k, l) Ink. (m, n) Keratinized epithelial cells. (o, p) Necrosis.

The final question is how to measure image variability. This knowledge can help data set creators prioritize which images to focus on, under the assumption that more diverse images provide the most meaningful information for training. For example, annotators may have access to 1000 WSIs but only enough time to label 100; the goal could be to select those images that will lead to the most diverse subset. The practicable approach is to analyze the image metadata (eg, disease subgroups, types of WSI scanners, etc) and randomly sample within each category. A more sophisticated approach is to use quality control metrics ⁴⁹ or learned embeddings of a feature extractor ¹³⁰ and subsequently encode images into a continuous space where semantically similar samples are nearby. Visualization of this space (eg, t-distributed stochastic neighbor embedding) and unsupervised clustering (eg, k-means) can reveal meaningful subgroups related to image variability.^49,130 In the example above, investigators could form 10 clusters based on the embeddings and select 10 cases from each cluster for annotations. Similar approaches can be used to support splitting of the primary data set and quantify the domain shift between the primary and secondary data sets.

Annotation Software

There is a variety of user-friendly annotation software, including both proprietary/commercial software (mostly combined with DL development frameworks; not listed in this article) and free open-source software (Table 1).^{9,16,19,57,81,82,86,89,99,134} Data set creators need to choose software that supports viewing the image, allows for the efficient creation of annotations in the intended manner (see the next section), and generates data sets with a meaningful, reusable structure for AI development. ¹³³ For proprietary software with integrated model development tools, it is important to ensure that it is possible (as it is for all open-source software) to extract the data set in a meaningful format to allow for long-term storage and reuse in other projects. Most software are designed for a broad scope of annotation applications using microscopic (or even a wider range of biomedical) images,^{9,19,57,81,86,89} while others have been developed for specific annotation tasks.^16,82 For example, SWAN enables swipe-based patch classification using a mobile device, allowing pathologists to annotate in offsite locations. ¹⁶ The annotation tool TissueWand is specialized for tissue wand polygon annotations using a mouse or pressure-sensitive stylus, ie, a linear annotation is dragged in the center of the object and the annotations automatically spreads to the object borders. ⁸² PatchSorter focuses on two-dimensional annotation maps that clusters similar images patches in close proximity (generated with DL), which speeds up patch classification. ¹³⁴

OPEN IN VIEWER

Table 1.

Comparison of relevant features of a selection of open-source annotation software.

Feature		QuPath 19	SlideRunner 8	Cytomine 86	EXACT 89	SWAN 16
Application	WSI support	Yes	Yes	Yes	Yes	—
	DICOM support	Yes	Yes	Yes	Yes	—
	Offline/online	Offline	Offline	Online	Online and offline	Online, mobile device
Collaboration	User management	—	—	Yes	Yes	Yes
	Blinded mode	—	Yes	—	Yes	—
Annotation tools	Image classification	—	—	—	Yes	Yes
	Single-click spot annotations	Yes	Yes	—	Yes	—
	Bounding box/circle (variable size)	Yes	Yes	Yes	Yes	—
	Polygon	Yes	Yes	Yes	Yes	—
	Multiuser annotations	—	Yes	Yes	Yes	—
	Guided screening	—	Yes	—	Yes	—
	Plug-ins/inference	Yes	Yes	Yes	Yes	—

Abbreviation: WSI, whole-slide image.

Whereas most annotation software can open a variety of image types (including gross photographs and cellphone pictures through the microscope ocular), WSIs and z-stack WSIs are those with the highest requirements for viewing software. WSIs are not only extremely large files, but they also contain a pyramidal structure and most WSI vendors use their own proprietary file format. Most open-source annotation software use open-source libraries, eg, OpenSlide (https://openslide.org/), SlideIO (https://github.com/Booritas/slideio), or Bio-Formats (https://www.openmicroscopy.org/bio-formats/), which allow opening most WSI formats.^9,19,89,108 However, some vendors use proprietary formats that cannot be opened directly by vendor-independent software (eg, Philips iSyntax) and therefore require conversion before use.^38,108 Based on one study, the file type is not expected to markedly impact algorithmic performance. ⁷² However, there are ongoing efforts to harmonize WSI formats, such as the adoption of the DICOM (digital imaging and communications in medicine) format, which will facilitate standardization and interoperability across different software platforms and data exchange.^38,57 While DICOM is not yet widely utilized by data set creators, ¹⁰² an advantage is that application-specific metadata is embedded within the WSI.

One of the main differences between annotation software lies in whether they are offline (mostly desktop-based)^9,19 or online (mostly web-based).^{16,57,81,86,89,99,134} Offline software is easier to set up for a standalone installation but offers limited collaboration capabilities (eg, simultaneous annotations by multiple pathologists). Online platforms, on the contrary, share images by accessing a server, allowing collaboration with partners outside the organization while maintaining data privacy and protection through user management (eg, secure user authentication, individual user rights and access), metadata encryption, and prohibition of image download. The collaborative nature of online platforms also benefits from version control to ensure traceability of changes. When multiple annotators are involved in data set creation, it is critical that each annotation is linked to an annotator ID to support auditability, quality control, and interannotator agreement analyses.

Depending on the pattern recognition task for the DL model, annotations need to be created at different levels. The software may allow different annotation shapes at the image level for image classification, at the object level (spot annotation, rectangle or circle at a fixed or variable size) for object detection, and at the pixel level (polygons or tissue wand polygons) for segmentation. Time-efficient and accurate labeling should be facilitated through standard features (eg, single-click annotation and keyboard shortcuts) and more advanced features, such as guided screening, blinded mode, and plug-ins.^9,89 The guided screening mode is particularly useful for large images (with many fields of view) to ensure complete annotations without skipping image regions.^9,89 In this mode, the large image is divided at a self-defined zoom level into patches of the size of the viewing field (with some percentage overlap), and the annotator is navigated through these patches sequentially. In the blinded mode, other annotators do not see the label class of annotations, which may be used for unbiased multi-expert majority voting. Plug-ins are essential for visualization or image analysis and are often used for computer-assisted labeling or registration of two images (eg, immunohistochemistry [IHC]-assisted labeling, see below).^89,99,134 Many of the more recent annotation software focus on algorithmic support during annotations and thereby provide innovative tools to increase label efficiency (see annotations section).^89,99,134

Annotations

When annotating, a label is assigned to the image or objects of interest within an image, and these labels are considered the ground truth (also referred to as reference standard), which serves as the desired output value during training and as the reference in the validation and test data sets for evaluating whether algorithmic predictions are correct or wrong. The ground truth represents the concept of truth generated by the most suitable method (gold standard method), balancing time investment with the following key quality criteria, the 3 “C”s:

The ground truth method can be defined as interpretation by humans (manual annotations), humans supported by algorithmic suggestions (computer-assisted annotations), fully computerized annotations (see subsections below), or a superior method. For most pathology tasks, the gold standard had been considered to be trained pathologists. However, human annotators are known to have well-recognized visual and cognitive biases, which can result in an imperfect ground truth.^1,43,113,114 For particularly challenging pathology tasks, human error of a simple single-annotator data set can be significant enough to create a catch-22 situation: while a DL model may be developed with the objective to improve the diagnostic ability (ie, to be superior than a pathologist), it is trained with a noisy ground truth, leading to instability in model convergence, and tested against noisy ground truth, leading to an unreliable performance evaluation.^21,71 Harada et al ⁶¹ examined the impact of labels errors (noise) on a tumor segmentation model. They artificially altered ground truth annotations by dilation and shrinking (contour incorrectness), omission (incompleteness) and addition, or a random combination thereof (inconsistency) and clearly showed a negative impact on model training. Furthermore, they showed that when these artificially induced label noises were also present in the test set, the true performance was significantly overestimated or underestimated (depending on the type of noise) as compared with the actual ground truth. It is important to note that a true (error-free) ground truth does not exist for most pathology tasks; however, we recommend placing strong emphasis on high quality of labels (in particular for the test data sets), which can be achieved by a well-planned annotation workflow and numerous advanced annotation methods, as discussed in this section (Table 2).

OPEN IN VIEWER

Table 2.

Comparison of annotation methods that can be used to generate labels for microscopic data sets.

	Manual	Computer-assisted		IHC-registered transfer of computerized annotations
		Previously labeled images	Unlabeled images
Purpose	Simple setup; initial data set for algorithmic-augmented labeling	Improve quality	Increase label speed	Eliminate manual annotation workload
Applied method	Pathologists’ decisions	Detection of missed candidates or annotation maps and subsequent pathologists’ decision	Expert-algorithm-collaboration	Create HE and IHC image from same section; computerized detection of IHC signal, extraction as annotations and transfer to HE image
Required pathology skills	High	High	High	Low
Required computer science skills	Low	High	High	High
Limitations	Visual and cognitive bias of humans; high time investment	Algorithmic error propagation; confirmation bias of humans	Algorithmic error propagation; confirmation bias of humans	Weak labeling and background labeling of IHC, loss of tissue during re-reprocessing
Options to improve	Clear decision criteria; multiannotator majority vote; image registration with another staining/IHC, etc	Highly sensitive predictions (include hard negatives to reduce confirmation bias), highly diligent review of predictions by expert	Highly specific predictions? Highly diligent review of predictions by expert	Improve laboratory workflow

Abbreviations: HE, hematoxylin and eosin; IHC, immunohistochemistry.

The 3 “C”s (correctness, consistency, and completeness) are meant to stimulate thoughts on possible sources of label errors and means of reducing these errors (as discussed in the annotation methods below), even though they are partially overlapping concepts (Supplemental Figures S4–S6). Correctness indicates if the annotated pattern truly represents the pattern of interest, and thus, if the derived algorithm will have a biologically meaningful application. For example, if a data set for mitotic figures includes a lot of false annotations of necrotic and apoptotic cells (look-alikes), the developed model, regardless of state-of-the-art DL methods and a seemingly high performance on the likewise biased test set, will overcount nonmitotic objects and might subsequently not be able to discriminate patients with short and long survival times. For segmentation tasks, accurate object contours (as opposed to oversegmentation or undersegmentation) need to be considered. For example, it might be difficult to accurately delineate an equine sarcoid from surrounding dermis or to delineate nuclei in neoplastic mast cells when they are partially obscured by metachromatic granules. ⁶⁰ Inconsistencies are random error in the data set that introduce “confusion” during training and uncertainty in performance evaluation. For example, annotators are inconsistent if telophase mitotic figures with beginning separation into the two daughter cells are sometimes annotated as one object and other times as two objects (the latter also being a biologically incorrect label approach). Completeness indicates if all the objects of interest were annotated or if there were unintentional omissions. For example, if a test set with 100 mitotic figures only includes annotations for 80 of these, a perfect algorithm would have a measured performance of F1 = 0.89% (due to incorrectly classifying 20 predictions as “false positives”) instead of the actual F1 = 1.0 (with all predictions being correctly classified as true positives).

Depending on the intended pattern recognition task for the DL model, different annotation shapes need to be applied (including global image/case-level labels for object detection tasks; object-level annotations like spot annotations or bounding boxes for object detection tasks; and pixel-level annotations like polygons/masks for segmentation tasks), with higher level of label details granularity (polygons > bounding box > spot annotations > image labels) typically requiring more time investment (Fig. 4). ¹³³ Many pathology tasks can be solved with different pattern recognition tasks, based on the intended output and application of the model. For example, models for predicting lymph node metastasis have either used patch classification or pixel segmentation as a pattern recognition task, each requiring different annotations types as a reference standard. ²⁹ While having higher label granularity is typically beneficial for model development,^121,122 the most suitable annotation shapes should be determined based on their time investment, balancing the richness of information most beneficial for the pattern recognition task with the number of annotations that can be created within the same time frame. ¹²¹ When time efficiency is a critical concern, some projects may opt for sparse annotations, which involve intentionally annotating only a limited percentage of the pattern within the selected image. ¹⁴⁴ This lack of completeness introduces limitations for DL model development and is therefore typically combined with fully annotated images. ¹⁴⁴ It is important to note that sparse annotations are not suitable for validation and test data sets because any unlabeled object will lead to an incorrect interpretation of the model performance (see the mitotic figure example above). Unfortunately, there are currently no general guidelines for the ideal composition of a data set regarding the trade-off between data quantity (data scaling laws) and granularity (detailed annotations for few images vs “weak” annotations for many images) or completeness (complete annotations for few images vs sparse annotations for many images).

OPEN IN VIEWER

Figure 4.

Comparison of different annotation shapes with increasing granularity. (a) Classification labels of image patches with labels for normal (red) and atypical mitotic figures (green). Hematoxylin and eosin (HE) images of different canine and human tumors from an open data set. ¹⁸ The difference in color between the patches is related to slides being stained at different laboratories and scanned with different whole-slide image (WSI) scanners. (b) Canine cutaneous mast cell tumor from an open data set. ²³ Single-click spot annotation in the center of mitotic figures (green) and hard negatives (blue). For better detectability, the annotation is displayed as a fixed size circle. HE. (c) Bounding box annotations of variable sizes labeling Helicobacter sp. in a canine gastric biopsy. HE. (d) Bronchoalveolar lavage of a horse with exercise-induced pulmonary hemorrhage from an open data set. ⁹² Bounding box annotations of variable sizes labeling 5 different classes of hemosiderophages. Perl’s Prussian blue. (e) Canine cutaneous mast cell tumor from an open data set. ⁶⁰ Polygon annotation delineating the contours of neoplastic nuclei. HE. (f) Canine mammary tumor. Region of interest annotation (red box) and polygon annotations delineating the contours of neoplastic epithelium (purple is inclusion area and yellow is exclusion area). HE.

For image classification, the labels can be done either on the tile-level (small subregions in the WSI) or on the global case/WSI level. An example of a global label is the diagnosis by pathologist, ⁵² and for these types of labels, weakly supervised learning (such as multiple instance learning) is typically used.^52,109 Some of these global labels can be regarded as superior gold standard methods, when they evaluate the pattern of interest independent from human interpretation, often even without a clear understanding of the corresponding morphological features in the WSI, such as patient outcomes (defined by survival time) or mutational patterns (determined by molecular methods).^40,41,78,109 These superior ground truth methods are not further discussed in this review, due to their paucity in veterinary data sets.^78,109 For global labels, a sophisticated annotation software is not necessarily needed, and labels may be simply collected in a spreadsheet. For certain image-level labels, such as case diagnoses, annotations can sometimes be retrieved from existing medical records,^52,87 reducing the need for additional annotators but possibly still requiring careful quality control. Ideally, a systematic ontology and standardized terminology (eg, INHAND in toxicologic pathology) is used.

For image classification, there must always be at least two label classes. For object detection and segmentation, annotations may have only a singular label class (eg, tumor tissue) ⁶ or multiple classes (eg, number of argyrophilic nuclear organizer regions per tumor nucleus with 10 label classes), ⁵⁴ whereas all nonannotated objects or pixels belong to the background class. For some pathology tasks, it may be useful to additionally annotate particularly challenging (for the DL model) background patterns as “hard negatives.”^23,85 The primary benefit of hard negatives is that they can be shown to the DL model during training at an overproportionate ratio to reinforce the model’s ability to distinguish patterns with overlapping morphology. These hard negatives include (1) objects with an overlapping morphology to the pattern of interest (eg, apoptotic cells for a mitotic figure data set), for which pathologists may also struggle to classify them correctly and (2) patterns that may be difficult for the DL model to learn due to their rarity in some data sets, but are obvious for pathologists (eg, green ink with a shape of chromosomes in metaphase could be confused with a mitotic figure, Fig. 3k, l). A secondary benefit of hard negatives is that they can be used (in addition to the candidate objects) when forming a majority vote by multiple annotators to avoid confirmation bias (see below).

The Evolution of Data set Creation for Mitotic Figures

This section aims to demonstrate the challenges of data set creation using the example of data sets for mitotic figures from both human and veterinary samples. We compare relevant annotation methods and explore how these methods impact the derived DL object detection models. Mitotic figures represent a particularly challenging pattern for both pathologists and computer vision systems, necessitating especially high standards for ensuring data set quality. ²² We believe that mitotic figure data sets are a good example since this is one of the few pathology tasks for which several data sets (open data and private) exist, with the first published in 2012. ¹¹⁶ From the 8 open-access data sets listed in Table 3 and Supplemental Table S1, it is evident that the applied methods for data set creation have become increasingly complex over time to address challenges with annotation bias, making a comparison interesting .^{8,13,23,27,115,116,128,129} Based on the number of annotators and the use of computer assistance, we have classified these annotation methods into three versions. Another version of data set creation, currently being investigated and discussed below (however, no open data set is available yet), is the use of IHC-assisted methods (version 4).

OPEN IN VIEWER

Table 3.

Comparison of creation methods for mitotic figures data sets using human and animal tumor images, ordered by annotation method version and year of publication.

Data set acronym or reference	Year	Annotation method
		Number of annotators	Computer-assisted missed candidate search	Other	Version a
MITOS 116 ,*	2012	1	—	—	1
AMIDA 13 129 ,*	2013	4 (majority vote)	—	—	2
MITOS-ATYPIA 115 ,*	2014	3 (majority vote)	—	—	2
TUPAC 128 ,*	2016	4 (majority vote)	—	—	2
MITOS_ccMCT 23 ,*	2019	2 (consensus)	Yes	—	3
TUPAC alternative 27 ,*	2020	2 (consensus)	Yes	—	3
MITOS_CMC 9 ,*	2020	3 (majority vote)	Yes	Annotation map	3
MIDOG++ 13 ,*	2023	3 (majority vote)	Yes	—	3
Tellez et al 124	2018	0	—	IHC-based pseudolabels and transfer to HE images	4
Ganz et al 55	2024	1	—	IHC-assisted annotations in HE images	4

a

Data set versions: 1, single annotator; 2, multiexpert majority vote; 3, multiexpert majority vote plus computer-assisted missed candidate screening; 4, immunohistochemistry-assisted labeling.

Open data sets are indicated by an asterisk.

The earliest mitotic figure data set (the MITOS data set) used a ground truth derived from a single annotator (version 1). ¹¹⁶ This data set was used for a computer scientist challenge, where several research teams competed to develop the best-performing algorithm. A high F1-score of 0.782 was achieved by the winning team; however, this high performance can be largely attributed to data leakage (ie, image patches from each case were included in both the training and test data sets, significantly overinflating the performance). The primary source of label bias is the correctness of the mitotic figure annotations. Previous studies have shown that pathologists apply variable thresholds when classifying mitotic figures against the background (eg, apoptotic cells), which can result in differences in the number of annotations of up to a factor of 3.5 when analyzing the same images.^22,138 While it is typically unknown whether the annotator applied sensitive or specific thresholds, the individual annotator has a great influence on the derived DL model.

Version 2 data sets attempted to address interannotator differences by employing a multiexpert majority vote for formulating the ground truth (eg, the AMIDA 13, MITOS-ATYPIA, and TUPAC data sets).^115,128,129 The AMIDA13 data set is a good example of this version and it provides insight into the agreement between their annotators. ¹²⁹ Two annotators independently screened the 23 images, resulting in 1088 annotations by annotator 1 and 1599 annotations by annotator 2. Of these, 649 objects were annotated by both annotators (31.8%), which were directly accepted for the final data set, while 1389 objects (68.2%) were labeled only by one annotator. The latter annotations were evaluated by additional pathologists who agreed that 434 of the 1389 objects (31.2%) were consistent with mitotic figures. This multiexpert majority vote aims to balance the individual pathologists’ thresholds toward an average pathologist’s interpretation. Two studies have evaluated the number of annotators needed for a consensus and both concluded that 3 expert annotators provide an ideal balance between increased label accuracy (defined as the average pathologists’ interpretation) and additional time investment.^58,138 While this mean decision threshold is considered beneficial for achieving acceptance of the DL model by the majority of pathologists, it should be acknowledged that there will always be pathologists with divergent thresholds, meaning this approach does not completely resolve the issue of label correctness. Even the same annotators will not result in the same ground truth, revealing that there are challenges to apply the decision thresholds consistently. For example, when the ground truth objects of the AMIDA13 test set were reevaluated by the same annotators, only 379/533 objects (71.1%) were considered to be true mitotic figures upon reevaluation. ¹²⁹ Another relevant source of label bias in this version 2 method is the completeness of annotations. This is also supported by the data from the AMIDA13 computer scientist challenge, for which all predictions by the best-performing DL models that were initially interpreted as “false positives” (ie, predictions not marked as ground truth labels) were reevaluated. Upon reevaluation of these objects, it was found that 61/208 (29.3%) “false positives” were indeed consistent with mitotic figures (ie, true positives) and thus should not have been interpreted as algorithmic error. Instead, these represent label errors of the initial annotation workflow (omission of actual mitotic figures) that had resulted in falsely low performance estimation. ¹²⁹

Successive data sets incorporated computer assistance for identification of missed candidates and thereby increasing label completeness (version 3; eg, the MITOS_ccMCT, TUPAC alternative, MITOS_CMC, and MIDOG++ data sets).^8,13,23,27 This approach is applied after all images have been annotated using a version 2 workflow (Fig. 5), with an emphasis on reducing annotator confirmation bias by predicting candidates at high sensitivity (ie, intentionally including many false positives). Two studies have demonstrated that the measured test set performance of the DL model (using the same images and training methods) between version 2 and 3 data sets can be improved from F1 = 0.755 to 0.820 ²³ and from F1 = 0.707 to 0.785, ⁸ showcasing the relevance of label completeness. Other researchers have applied this approach at a sensible diagnostic detection threshold (ie, without intentional sampling imposters), ¹¹² which reduces the number of patches that need to be labeled by a pathologists; however, this approach may also induce a higher confirmation bias (possibly reducing label correctness) and does not detect the mitotic figures that the model struggles with (ie, have a lower detection threshold).

OPEN IN VIEWER

Figure 5.

Annotation workflow for a version 3 data set, ie, majority voting and computer assistance for annotation. First, an initial data set is created by a pathologist screening all images for mitotic figures and hard negatives (background structures resembling mitotic figures such as apoptotic cells). The initial data set is further processed in two ways: (a) blinding for the label classes and majority voting by further pathologists and (b) defined as training data for a deep learning model that is used to screen for mitotic figure candidates that were missed by initial annotations. The model is designed to predict candidates with a high sensitivity, resulting in an intentionally high number of false positives, which removes confirmation bias in subsequent majority voting. This graph was modified from Bertram et al. ²⁷ MFs, mitotic figures; DL, deep learning.

Given that the version 3 annotation workflow is time-consuming, alternative methods have been evaluated that use destaining and immunohistochemical labeling with pHH3 of the same tissue sections (version 4).^{12,22,55,58,124} pHH3 is a marker that highlights mitotic figures throughout prophase (not consistently recognizable in HE images) to anaphase. Tellez et al ¹²⁴ have automatically detected the brown signal of mitotic figures (color deconvolution) and used a DL model (trained with only 2000 manually annotated objects) to distinguish staining artifacts from real signals (pseudolabels for a relatively easy computer vision task as compared the HE task). These pseudolabels were then transferred to the registered HE images, enabling the creation of a large-scale WSI data set with only 2 hours of annotation work. However, the derived model did not perform particularly well, despite the large amount of training data. Based on our experience, many pHH3-positive structures cannot be recognized in HE images based on morphology (eg, early mitotic figures, overstained cells, out-of-focus image regions, or marginally sectioned cells).^12,55 This may have introduced label noise in the annotation method by Tellez et al ¹²⁴ and impaired model training. A subsequent study by Ganz et al ⁵⁵ used the pHH3 slide as decision support for pathologists during annotations in HE images (Supplemental Figure S9). This approach aimed to improve label completeness (eliminating the need for computer-assisted missed candidate screening) and enhance annotator consistency by helping pathologists better distinguish mitotic figures from imposters. While data sets from single annotators showed much higher agreement, this annotation method did not result in improved model performance. This outcome was attributed to an information mismatch between the HE and pHH3 images, which led to a label shift by annotators (ie, increasingly annotating morphologically less certain mitotic figures). Further improvements of this annotation approach, such as by majority voting (blinded to pHH3) for borderline morphologies, were suggested.

Further Considerations

Before using the data set for model development, it is recommended to conduct data cleaning, which involves detecting and correcting or removing errors, incompleteness, and inconsistencies, and quality control of annotations and images. ¹³³ For example, missing metadata can be added and obvious errors, such as duplicate annotations and wrong annotation types that should not have been used, can be automatically removed. The development of an initial DL model may also provide insights into the data set’s quality and reveal potential areas for improvement through augmentation of the data set.

Effective data set management requires thoughtful storage, backup, and long-term accessibility. Storage should match access needs; fast local or network solutions during annotation and model training, and slower archival systems after project completion. To prevent data loss, at least one independent backup should be maintained on a separate medium (eg, cloud, external drive, server, or public data repository). During active data set creation, backups should be performed regularly, ideally with version control to track changes. Equally important is the choice of image and annotation formats. Proprietary or third-party formats may become unreadable if their software is discontinued and open, standardized solutions are preferred. In digital pathology, DICOM is a proposed solution for a well-documented, vendor-independent standard for WSI and annotations, which ensures interoperability and long-term accessibility. ³⁸

Proper documentation of the intended use case of the data set and all data set creation steps, as well as a detailed description of the distribution of the final data set (eg, the number of images/cases and annotations as well as their distribution across image domains and disease subgroups) is essential for ensuring transparency regarding potential data set bias and enabling informed decision in subsequent DL model development steps.^26,46,64 For example, metadata on patient subgroups will allow a stratified performance evaluation, possibly revealing sources of algorithmic error (hidden stratification). ⁶⁴ However, it has been shown that public data sets with human samples often lack relevant information, ¹²³ highlighting the need for particular attention to this matter. To support authors in complete reporting of data set characteristics, Elfer et al ⁴⁶ have published a reporting guideline for annotations.

Although splitting the primary data set is typically done after data set creation and before DL model development (and is thus not necessarily part of data set creation), it is based on data set characteristics. Splitting into 3 subsets (training, validation, and test) must be done at the patient level to avoid data leakage between subsets, for which information on the patient ID for each image is needed.^64,144 While the test data set should be representative of real-world applications, a random patient-wise splitting approach may not be ideal for small data sets. In such cases, stratified splitting, where groups are defined based on, for example, image domains or the density of annotations, is often preferred. These data set characteristics (patient ID, subgroup categorization) should be encoded in the final data set. Alternatively, unsupervised clustering of image variability has been proposed to assist data set splitting. ⁴⁹ The data set splitting scheme should also be documented so it can be used consistently across studies enabling comparison of the results.

Data set owners should also discuss sharing policies with AI developers, whether they are from the same organization or from external institutions (eg, codevelopers or independent research groups). Open-access publications under the various Creative Commons licenses allow for the broadest distribution of data sets (see the section below). To enable others to use the data set effectively, usage notes (eg, recommended software, suggested data splitting strategies) should be prepared. Before sharing data with other institutions, pseudonymization (or anonymization) of images and the removal of private information about the patient or owner, which is information that is not necessary for DL model development, should be performed.

Open Data

Considering the dependency of DL model development on available data sets, publishing data sets and making them accessible for research purposes is an extremely important contribution to science. The opportunities enabled by open data are diverse; some of the most significant are the following:

As listed in Table 4 and Supplemental Table S2, some veterinary researchers have made their data sets publicly available for research purposes.^{8,13,18,23,24,31,59,60,92,131,136,139,140,142} Some of these data sets span various domains, including several WSI scanner types,^13,139 multiple tumor types,^13,140 multiple species,^13,92 and multiple laboratories,^13,92 showcasing collaborative initiatives that will foster robust DL model development. However, despite the growing number of publications on AI models for various applications in veterinary pathology, this literature review reveals that the majority of the veterinary research community remains hesitant to publish their data sets. Instead of open data, some have indicated in their manuscripts (via data availability statements) that they are willing to share data upon reasonable request and through individual agreements,^7,53,78 which introduces uncertainties regarding long-term availability. This hesitancy may stem from a desire to maintain control over proprietary data or plans to publish the data in subsequent studies.^{88,92,112,118} While we recognize these restrictions faced by some AI developers, we hope that the opportunities provided by open data, as listed above, will positively influence the veterinary research community’s willingness to publish data sets following the FAIR (findable, accessible, interoperable, reusable) principles. ¹³⁷

OPEN IN VIEWER

Table 4.

List of open-access data sets containing microscopic images of animal tissue samples and labels for patterns of interest.

Reference	Acronym	Tissue type	Species	Pattern of interest	Annotation type/shape
Histology
Burrai et al. 31	CMTD	Mammary tumors	Dog	Benign vs malignant	Image classification
Bertram et al 23	MITOS_WSI_CCMCT	Cutaneous mast cell tumors	Dog	Mitotic figures	Spot annotations
Aubreville et al 9	MITOS_WSI_CMC	Mammary carcinoma	Dog	Mitotic figures	Spot annotations
Aubreville et al 13	MIDOG++	Several tumor types	Dog, and human	Mitotic figures	Spot annotations
Weiss et al 136	MIDOG 2025 Atypical	Several tumor types	Dog and human	Normal vs atypical mitotic figures	Patch classification
Bertram et al 24	N/A	Cutaneous mast cell tumors	Dog	Bi- and multinucleated tumor cells	Spot annotations
Haghofer et al 60	N/A	Cutaneous mast cell tumors	Dog	Neoplastic nuclei	Polygons
Haghofer et al 59	N/A	Lymphoma	Dog and cat	Neoplastic nuclei	Polygons
Wulcan et al 142	N/A	Lymphoma	Cat	Intestinal tissue compartments and lymphocytes	Polygons
Wilm et al 140	CATCH	Several cutaneous tumor types	Dog	Tissue types	Polygons
Wilm et al 139	MC-SCC	Skin with squamous cell carcinoma	Dog	Tissue types	Polygons
Cytology
Marzahl et al 92	N/A	Bronchioalveolar lavage fluid	Horse, cat, human	Hemosiderophages	Multiclass bounding boxes
Vogelbacher et al 131	N/A	Blood smear	Avian	Cell types	Polygons

These data sets were developed for supervised deep learning (DL) model development. See Supplemental Table S2 for an expanded list. The data sets were identified through an extensive literature review of primary research articles; we acknowledge that some data sets may be available in open data repositories without a corresponding journal publication and therefore may have been missed in our search.

N/A, not available.

An additional finding from this literature review (Table 4) is that there is a particular lack of data sets related to laboratory animals and toxicologic pathology applications. This gap likely reflects the legal and data privacy constraints faced by toxicologic companies. Supporting this observation, a survey of toxicologic pathologists revealed that most respondents do not consider sharing training data sets with codevelopers for commercial or public projects. ¹⁰⁶ Likewise, we acknowledge that diagnostic companies are often restrictive in data sharing due to concerns over future commercial and proprietary interests.^7,28 Initiatives of whole-slide imaging platforms through multi-institutional and multidisciplinary consortia, such as Bigpicture, are promising to overcome these legal and proprietary challenges of image availability and sharing. ¹⁰³

A variety of human data sets are also available;^13,37,123 however, also the majority of DL articles in human medicine either use existing open data sets or private data sets, and a request for more publicly accessible data sets and a higher diversity of open data set (eg, on rare tumor types) has been made.^37,132 Human data sets may serve as valuable supplements to animal data sets when developing models for veterinary applications. Previous research has demonstrated that models developed using human data sets (eg, algorithms for mitotic figure detection in tumors) can be successfully transferred to animal samples, and vice-versa, while maintaining a high performance or archiving high performance after some transfer learning.^8,13,92,93 However, while cell and tissue morphologies are often comparable between humans and animals, there may be significant differences, such as variations in the frequency of disease subtypes. These differences require the incorporation of species-specific data, particularly in test data sets.

For image classification task, another valuable resource may be WSI repositories with metadata (eg, disease diagnosis, mutational status, etc) as potential image-level labels. In human medicine, a frequently used WSI repository is The Cancer Genome Atlas (TCGA), ³⁹ which has been used by several studies for developing models that can predict mutational patterns in HE images.^40,41 A potential limitation of WSI repositories are a higher degree of impurities (as compared with data sets specifically curated for DL-AIA) of the WSI (eg, large air bubbles, marker signs, lack of relevant tissue, poor tissue quality) and metadata, possibly requiring data cleaning.^68,123 Research projects that enrich these WSI repositories by publishing object- or pixel-level annotations are valuable contributions. ¹⁵

When publishing data sets, it is essential to use data repositories that ensure long-term storage and access, such as Zenodo (https://zenodo.org/) or GitHub (https://github.com/). Clear usage notes and detailed data set descriptions (eg, included disease subtypes, digitization devices, etc) should be provided to enable other researchers to utilize the data as intended. The use of systematic nomenclature for metadata and disease diagnosis will facilitate the combination of several open (and in-house) data sets for model development. The publication of images and their associated metadata requires a well-planned strategy to address ethical, data privacy, and proprietary considerations, even more so than for internal use of images. ⁵³ For veterinary samples, ethical approval may be waived (depending on the regulatory framework of the individual country) when routine diagnostic samples are used. However, many journals require owner consent from animal owners, and diagnostic laboratories intending to use their caseload for future research should consider strategies for obtaining such consent (eg, including a statement in the submission form). While the publication of an open data set does not necessarily require training and testing of a first DL model, it is highly recommended to do so as it provides first insight into the data set quality and provides a baseline for the intended task, which can be used in subsequent studies as orientation. When working with public data sets, researchers should ideally adopt the original training, validation, and test partitions to facilitate comparison between published studies.

Conclusion

Creation of large-scale and high-quality data sets for supervised DL in microscopic image analysis is a complex and resource-intensive process. Bias in selected images and created labels will be reflected in the model’s generalization performance and one’s ability to interpret performance metrics. This article has therefore highlighted critical considerations for data set creation regarding image selection, annotation software, annotation methods, and further considerations. For image selection, not only the sheer number matters; it is also essential to capture the image variability expected in the application cases. For example, if the development data set was digitized with one WSI scanner and the model is later applied to images from another scanner, performance may drop considerably. Active learning and synthetic data can help to increase the quantity or informativeness of training data sets and reduce the overall time investment; however, these methods should not be used to construct the test subset. The selected annotation software must provide all project-required features, including viewing support for specific file formats, online or offline application, and—if applicable—specific annotation tools such as guided screening and plug-ins for computer assistance. While annotations are often created by a single annotator, for complex patterns of interest (such as mitotic figures), there may be relevant visual and cognitive bias of annotators. Key criteria for annotation quality are the 3 “C”s: correctness, consistency, and completeness. Label bias of a single annotator may be reduced through multiexpert majority votes or computer assistance (such as by screening for missed candidates). To reduce the annotation workload, computer-assisted methods (algorithmic inference on unlabeled images and real-time algorithm-annotator interactions) or fully computerized methods (such as stain-registered transfer of computerized annotations) can be applied, with particular care to ensure high label quality for the test data set. Further considerations before concluding data set creation include data cleaning, storage and backup, documentation of data set characteristics, and deciding on data sharing policies.

Supplemental Material