Automated Zebrafish Phenotype Pattern Recognition: 6 Years Ago,and Now

Abstract

The article assesses the developments in automated phenotype pattern recognition: Potential spikes in classification performance, even when facing the common small-scale biomedical data set, and as a reader, you will find out about changes in the development effort and complexity for researchers and practitioners. After reading, you will be aware of the benefits and unreasonable effectiveness and ease of use of an automated end-to-end deep learning pipeline for classification tasks of biomedical perception systems.

In 2016, we published a study on Automated phenotype pattern recognition of zebrafish for high-throughput screening.¹ The central contribution was a classifier to distinguish four different fish lines (Fig. 1) of hatched zebrafish larvae at 72 h postfertilization with respect to their phenotypical features. Back then (Spring 2015), the study was carried out within a 4-month project, and an average accuracy of 89% was achieved. Looking past the small-scale data set (1057 samples in four classes), and the handcrafted feature approach, the addressed biomedical visual classification task is just as relevant today.² Thus, inspired by Andrew Karpathy's blogpost* on reproducing the article on Backpropagation applied to handwritten zip code recognition on Modified National Institute of Standards and Technology dataset,³ we revisited our study on automated phenotype pattern recognition to find out what has changed during the past years in terms of methodology in biomedical vision tasks.

FIG. 1.

Exemplary samples of the four zebrafish classes: Wild type (top left), 1-phenyl 2-thiourea (top right), homozygous rx3 Mutant (bottom left), and Kita HRAS (bottom right).

Introduction: 6 Years Ago

Before the AlexNet⁴ revolution caught on and got widespread traction, a computer vision pipeline, so ours, followed a three-step approach of data loading, feature extraction, and finally pattern recognition. To put things into perspective, let us shortly recapitulate the specifics of the 2016 setup. Image acquisition was handled by two cameras in orthogonal pose (transversal and sagittal view), directed at a cuvette that holds the zebrafish.

Let us take a closer look on the different classes in Figure 1: The top left image represents the wild-type zebrafish, with a natural pigmentation pattern and all developmental features in place. The top right image contrasts the wild type by a reduced pigmentation due to the 1-phenyl 2-thiourea (PTU) treatment, the effects on the pigmentation strength can vary so differentiation can be ambiguous. The bottom left shows a homozygous rx3 (retinal homeobox gene 3) mutant in which the development of the eye segment is suppressed, which is easily spotted by a human. The bottom right features a transgenic kita proto-oncogene with protein product (HRAS) sample,⁵ which develops an amplified pigmentation phenotype, especially on the yolk bag. Apparently, pigmentation phenotype expression is crucial for classification; however, the individual fish lines express a broad spectrum of pigmentation intensity within themselves. Finally, the larvae's pose has an impact on how well the features can be assessed by a human or a computer vision system.

Feature extraction was carried out based on handcrafted features and heavily relying on biomedical expert domain knowledge. Usually done manually, classifying different zebrafish types is a hard and tiring undertaking. Feature extraction had to be carried out specifically for each class, following multiple processing and extraction steps, leading to features based on the level of pigmentation (Fig. 2) for PTU and kita-HRAS (kita-driven protein), or the eye segment formation (homozygous rx3 Mutant). For pattern recognition we have been using support-vector networks⁶ in a binary classification setting, separating wild type versus variant, while we had to optimize for two control variables: The fault-tolerance and the Radial Basis Function-Kernel. Everything was implemented in MATLAB and OpenCV. Performance has been 89.0% and has been reported as classification accuracy (Table 1).

FIG. 2.

Feature extraction pipeline for the pigmentation level of the zebrafish larvae. (a) Original image, (b) Edge detection, (c) Segmentation, (d) Cropping. Figure from Schutera et al.¹

Table 1.

Average Accuracy Performance of the Binary Classifiers, Learned by a Support Vector Machine

Zebrafish type	Accuracy
Class average	89.0%

Method: Now

We draw on an arsenal of matured and practice-proven deep learning methods. This revision does not intend to use the past performance as a benchmark that needs to be surpassed. We deliberately refrain from an extensive hyperparameter tuning. Quite the opposite, we want to reflect on what has changed, and is possible out of the box with the current plug-and-play approaches and methods, which we would opt for if asked to tackle the problem today. Now, neural network specification is supported by high-level scriptable Application Programming Interfaces,⁷ bringing to the table everything that is necessary for training and developing a neural network, from architecture design, optimizers, loss functions, data loaders, and monitoring tools.

There are now a multitude of pretrained neural networks suitable for image classification, which come ready for deployment.² This reduces architectural design complexity and neural network training time, which might otherwise pose a barrier to implementation. It can be expected that the neural network will be able to have its way even with a multiclassification setting, arguably even benefitting from the multitask training, thus the output layer is designed as a dense layer with four nodes. This accommodates the neural network architecture for the task of classifying the same four classes as in 2016: Wild type, PTU, kita HRAS, and the homozygous rx3 mutant (Fig. 1).

In detail our deep learning pipeline^† follows a transfer learning approach⁸ provided originally as a demo by TensorFlow,^‡ which is based on python (3.9), tensorflow (2.7.0), tensorflow-hub (0.12.0). Thus, the time for setting up the pipeline is neglectable. Wherever possible we fall back to the default parameters given by the corresponding package and version. The only thing we could not default on was the choice of the feature extraction backbone, so we revert to the first thing that comes to mind, and a common choice for that matter: Inception_v3^§^,9 with a prepended normalization layer, and the optimizer where we go for the Adam optimizer.¹⁰

For training early stopping¹¹ was used, with a patience of five, so we did not have to decide on how many epochs to train until convergence. A more difficult decision to make is on a training to validation split. Where we could utilize the whole data set for validation in 2016, as we could draw training samples online while setting up the imaging platform. To reach a fair comparison, we want enough training samples to be able to converge on a model with little prediction uncertainty. However, we judge the evaluation variance to be of even higher importance. Thus, knowingly sidelining the deep learning approach's performance we set the train-validation-ratio to 0.5. Notice that once the classes have been labeled, there is no further domain knowledge necessary to develop and operate the deep learning pipeline.

Besides the advancements in methodology and software tooling, hardware accelerators have played a major role in the success of deep learning. This is not the concern in this study, due to the transfer learning approach, thus the training was carried out on an Intel^® Core™ i7-6820HQ Central Processing Unit (CPU)@ 2.70GHz, 2712 MHz, 4 Cores, 8 Logical Processors and 32.0 GB Random Access Memory (RAM).

Results: The Unreasonable Effectiveness

While one epoch took an average of 41 s, we reach our 2016 average accuracy performance after 4 epochs (∼2.5 min), achieving a validation accuracy of 89.0%.

The console output then looks like this:

Epoch 4/1000

17/17 [ = = = = = = = = = = = = = = = = ] - 36s 2s/step - loss: 0.3808 - acc: 0.8922 - val_loss: 0.4287 - val_acc: 0.8902

As we set up early stopping, our training automatically finished after 23 epochs with a validation accuracy of 94.70%:

Epoch 23/1000

17/17 [ = = = = = = = = = = = = = = = = ] - 45s 3s/step - loss: 0.0521 - acc: 1.0000 - val_loss: 0.2363 - val_acc: 0.9470

Proceeding naively, we out of the box have been able to sweep away our 2016 approach's performance and exceed it by ∼6% in validation accuracy, or seen from the other side, the error has been reduced by ∼50% (Table 2).

Table 2.

Final Accuracy Performance of the Multiclass Classification Setting, As Achieved By Our Deep Learning Pipeline

Zebrafish type	Accuracy
Class average	94.7%

In 2022, this is not where one would end if set out to crunch a conventional approach's baseline. The generalization gap of ∼5% (see the light blue band in Fig. 3) after 23 epochs asks for further regularization such as Dropout¹² or adjusting the objective function.¹³ Taking the small scale of the available data into account, data augmentation methods¹⁴ is an obvious next step. Knowing, that we want to follow a data-driven approach to feature extraction and selection through a neural network—acquiring more data is a definite to-do-next.^15,16 If interpretability is crucial for the application, there are heat-map approaches, and others that are fully developed for classification tasks within perception.

FIG. 3.

Training accuracy (upper lines) and validation accuracy (lower lines) over the epochs—as provided by tensorboard. The light lines are the real values, whereas the strong lines reflect the smoothed learning curves.

Discussion: So Now?

What has been changed and achieved within automated phenotype pattern recognition since we last looked (Table 3)?

Table 3.

Comparison of the Conventional Handcrafted Feature Extraction Approach and the End-to-End Deep Learning Approach to Automated Phenotype Recognition

Property	2016	2022
Development effort (man-days)	∼20	∼1
Label effort (class labels)	Low	Low
Interpretability for the end user	High	Low^*
Required domain knowledge	High	Low
Effort to scale (man-days/class)	∼5	∼0
Training and inference duration	Low	Low
Class-average performance	89.0%	94.7%

An asterisk depicts evaluations that can be improved with available methods.

The key findings and what you should take away from this article are as follows:

The paradigm shift of an end-to-end data-driven pattern recognition by deep learning caught on conclusively. This is also reflected in automated phenotype pattern recognition and allows for automated adaptation and extension of phenotypical classification tasks. Whereas we would have to handcraft feature representations for each added phenotype, today we implicitly introduce further types by providing the associated training samples, in a dedicated directory, then even adding a node to the final layer of our neural network is realized seamless. The required programming skills are reduced to a bare minimum of setting up and running an integrated development environment and require no formal or academic training.

Development of phenotype pattern recognition has become more accessible, as only little domain knowledge is necessary to group the samples, whereas it is no longer necessary for a human to understand the features that need to be extracted to best describe a specific phenotype. However, we would argue that at the latest with validation and continuous improvement of the model, domain knowledge remains as crucial as it has ever been.

More difficult biomedical tasks are targeted,^17–19 as phenotypical classification tasks require less development, free up research capacities, and by now scale seamless due to transfer learning. At the same time deep learning pipelines naively reach competitive performances, compared with handcrafted feature approaches.

The notion of neural networks being compute and data hungry has been done away with, especially during training, as we were able to achieve satisfying performance on a CPU and without robbing the piggy bank to get our hands on the latest Graphics Processing Unit or even Tensor Processing Unit. Owing to transfer learning the number of required training samples is significantly reduced as we can limit ourselves to fine-tuning the last classification layer of our model. Besides label-efficient learning strategies are developed and constantly improving.¹⁵

Footnotes

Acknowledgments

We thank our colleagues at the Institute for Automation and Applied Informatics, and especially Prof. Dr. Christian Pylatiuk who supervised the study on the baseline method from 2016.

Authors' Contributions

Conceptualization, investigation, data curation, and writing—original draft by M.S. Methodology and formal analysis by M.S. and M.R. Software and validation by M.S. and L.R. Resources, supervision, project administration, and funding acquisition by M.R. Writing—review and editing and visualization by M.S., L.R., and M.R.

Disclosure Statement

No competing financial interests exist.

Funding Information

No funding was received for this article.

References

Schutera

, Dickmeis

, Mione

, et al. Automated phenotype pattern recognition of zebrafish for high-throughput screening. Bioengineered, 2016; 7(4):261–265; doi: 10.1080/21655979.2016.1197710.

Kim

, Yan

, Yang

, et al. Deep learning in biomedical image analysis. In: Biomedical Information Technology. (Feng DD. Ed.) Academic Press: Amsterdam, The Netherlands; 2020, pp. 239–263.

LeCun

, Boser

, Denker

, et al. Backpropagation applied to handwritten zip code recognition. Neural Comput, 1989; 1(4):541–551; doi: 10.1162/neco.1989.1.4.541.

Krizhevsky

, Sutskever

, Hinton

. Imagenet classification with deep convolutional neural networks. Adv Neural Inform Process Syst, 2012; 25. Available at https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf.

Santoriello

, Gennaro

, Anelli

, et al. Kita driven expression of oncogenic HRAS leads to early onset and highly penetrant melanoma in zebrafish. PLoS One, 2010; 5(12):e15170; doi: 10.1371/journal.pone.0015170.

Cortes

, Vapnik

. Support-vector networks. Mach Learn, 1995; 20(3):273–297; doi: 10.1007/BF00994018.

Abadi

, Agarwal

, Barham

, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467. 2016.

Zhuang

, Qi

, Duan

, et al. A comprehensive survey on transfer learning. Proc IEEE, 2020; 109(1):43–76; doi: 10.1109/JPROC.2020.3004555.

Szegedy

, Vanhoucke

, Ioffe

, et al. Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE: Las Vegas, NV; June 27–30, 2016; pp. 2818–2826.

10.

Kingma

, Ba

. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.

11.

Prechelt

Early stopping—But when? In: Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science. (Montavon G, Orr GB, Müller KR. eds.) Springer: Berlin, Heidelberg; 2012, pp. 53–67.

12.

Srivastava

, Hinton

, Krizhevsky

, et al. Dropout: A simple way to prevent neural networks from overfitting. J Machine Learn Res, 2014; 15(1):1929–1958.

13.

Moradi

, Berangi

, Minaei

. A survey of regularization strategies for deep models. Artif Intell Rev, 2020; 53(6):3947–3986; doi: 10.1007/s10462-019-09784-7.

14.

Shorten

, Khoshgoftaar

. A survey on image data augmentation for deep learning. J Big Data, 2019; 6(1):1–48; doi: 10.1186/s40537-019-0197-0.

15.

Schutera

, Rettenberger

, Pylatiuk

, et al. Methods for the frugal labeler: Multi-class semantic segmentation on heterogeneous labels. PLoS One, 2022; 17(2):e0263656; doi: 10.1371/journal.pone.0263656.

16.

Halevy

, Norvig

, Pereira

. The unreasonable effectiveness of data. IEEE Intell Syst, 2009; 24(2):8–12; doi: 10.1109/MIS.2009.36.

17.

Schutera

, Just

, Gierten

, et al. Machine learning methods for automated quantification of ventricular dimensions. Zebrafish, 2019; 16(6):542–545; doi: 10.1089/zeb.2019.1754.

18.

Falk

, Mai

, Bensch

, et al. U-Net: Deep learning for cell counting, detection, and morphometry. Nat Methods, 2019; 16(1):67–70; doi: 10.1038/s41592-018-0261-2.

19.

Scherr

, Löffler

, Böhland

, et al. Cell segmentation and tracking using CNN-based distance predictions and a graph-based matching strategy. PLoS One, 2020; 15(12):e0243219; doi: 10.1371/journal.pone.0243219.