The required training sample size for a particular machine learning (ML) model applied to medical imaging data is often unknown. The purpose of this study was to provide a descriptive review of current sample-size determination methodologies in ML applied to medical imaging and to propose recommendations for future work in the field.
Methods
We conducted a systematic literature search of articles using Medline and Embase with keywords including “machine learning,” “image,” and “sample size.” The search included articles published between 1946 and 2018. Data regarding the ML task, sample size, and train-test pipeline were collected.
Results
A total of 167 articles were identified, of which 22 were included for qualitative analysis. There were only 4 studies that discussed sample-size determination methodologies, and 18 that tested the effect of sample size on model performance as part of an exploratory analysis. The observed methods could be categorized as pre hoc model-based approaches, which relied on features of the algorithm, or post hoc curve-fitting approaches requiring empirical testing to model and extrapolate algorithm performance as a function of sample size. Between studies, we observed great variability in performance testing procedures used for curve-fitting, model assessment methods, and reporting of confidence in sample sizes.
Conclusions
Our study highlights the scarcity of research in training set size determination methodologies applied to ML in medical imaging, emphasizes the need to standardize current reporting practices, and guides future work in development and streamlining of pre hoc and post hoc sample size approaches.
GorbanA.N.TyukinI.Y.Blessing of dimensionality: mathematical foundations of the statistical physics of dataPhilos Trans A Math Phys Eng Sci376201820170237
2.
IthapulV.K.SinghV.OkonkwoO.JohnsonS.C.Randomized denoising autoencoders for smaller and efficient imaging based AD clinical trialsMed Image Comput Comput Assist Interv172014470–478
3.
RasmussenP.M.HansenL.K.MadsenK.H.ChurchillN.W.StrotherS.C.Model sparsity and brain pattern interpretation of classification models in neuroimagingPattern Recogn4520122085–2100
4.
ObermeyerZ.EmanuelE.Predicting the future — big data, machine learning, and clinical medicineN Engl J Med37520161216–1219
5.
RaudysS.JainA.Small sample size effects in statistical pattern recognition: recommendations for practitionersIEEE Trans Pattern Anal Mach Intell131991252–264
6.
WayT.W.SahinerB.HadjiiskiL.M.ChanH.P.Effect of finite sample size on feature selection and classification: a simulation studyMed Phys372010907–920
7.
BeidenS.V.MaloofM.A.WagnerR.F.A general model for finite-sample effects in training and testing of competing classifiersIEEE Trans Pattern Anal Mach Intell2520031561–1569
8.
ChanH.P.SahinerB.WagnerR.F.PetrickN.Classifier design for computer-aided diagnosis: effects of finite sample size on the mean performance of classical and neural network classifiersMed Phys2619992654–2668
9.
BiauD.J.KernéisS.PorcherR.Statistics in brief: the importance of sample size in the planning and interpretation of medical researchClin Orthop Relat Res46620082282–2288
10.
MoodyA.Perspective: the big pictureNature5022013S95
11.
ParkS.H.HanK.Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and predictionRadiology2862018800–809
12.
PellegriniE.BalleriniL.Hernandez M delC.V.Machine learning of neuroimaging for assisted diagnosis of cognitive impairment and dementia: a systematic reviewAlzheimers Dement102018519–535
13.
SchnackH.G.KahnR.S.Detecting neuroimaging biomarkers for psychiatric disorders: sample size mattersFront Psychiatry7201650
14.
TangA.TamR.Cadrin-ChenevertA.Canadian Association of Radiologists white paper on artificial intelligence in radiologyCan Assoc Radiol J692018120–135
15.
MoherD.LiberatiA.TetzlaffJ.AltmanD.G.Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statementPLoS Med62009e1000097
16.
CuiZ.GongG.The effect of machine learning regression algorithms and sample size on individualized behavioral prediction with functional connectivity featuresNeuroimage1782018622–637
17.
CohenO.ZhuB.RosenM.S.MR fingerprinting Deep RecOnstruction NEtwork (DRONE)Magn Reson Med802018885–894
18.
ZhengB.ChangY.-H.H.GoodW.F.Adequacy testing of training set sample sizes in the development of a computer-assisted diagnosis schemeAcad Radiol41997497–502
19.
AbdulkadirA.MortametB.B.VemuriP.JackC.R.Jr.KruegerG.KlöppelS.Alzheimer's Disease Neuroimaging Initiative Effects of hardware heterogeneity on the performance of SVM Alzheimer's disease classifierNeuroimage582011785–792
20.
SamalaR.K.ChanH.P.HadjiiskiL.HelvieM.A.RichterC.D.ChaK.H.Breast cancer diagnosis in digital breast tomosynthesis: effects of training sample size on multi-stage transfer learning using deep neural netsIEEE Trans Med Imaging382018686–696
21.
DunnmonJ.A.YiD.LanglotzC.P.ReC.RubinD.L.LungrenM.P.Assessment of convolutional neural networks for automated classification of chest radiographsRadiology2902018537–554
22.
LooneyP.StevensonG.N.NicolaidesK.H.Fully automated, real-time 3D ultrasound segmentation to estimate first trimester placental volume using deep learningJCI Insight32018e120178
23.
McKinleyR.HungF.WiestR.LiebeskindD.S.ScalzoF.A machine learning approach to perfusion imaging with dynamic susceptibility contrast MRFront Neurol92018717
24.
TourassiG.D.FloydC.E.The effect of data sampling on the performance evaluation of artificial neural networks in medical diagnosisMed Decis Making171997186–192
25.
WangJ.Y.NgoM.M.HesslD.HagermanR.J.RiveraS.M.Robust machine learning-based correction on automatic segmentation of the cerebellum and brainstemPLoS One112016e0156123
26.
ChangH.BorowskyA.SpellmanP.ParvinB.Classification of tumor histology via morphometric contextProc IEEE Comput Soc Conf Comput Vis Pattern Recognit2013201310
27.
ChangH.NayakN.SpellmanP.T.BahramP.Characterization of tissue histopathology via predictive sparse decomposition and spatial pyramid matchingMed Image Comput Comput Assist Interv16201391–98
28.
DinhC.V.SteenbergenP.GhobadiG.Multicenter validation of prostate tumor localization using multiparametric MRI and prior knowledgeMed Phys442017949–961
29.
JuntuJ.SijbersJ.De BackerS.RajanJ.Van DyckD.Machine learning study of several classifiers trained with texture analysis features to differentiate benign from malignant soft-tissue tumors in T1-MRI imagesJ Magn Reson Imaging312010680–689
30.
SahinerB.ChanH.-P.PetrickN.WagnerR.F.HadjiiskiL.Feature selection and classifier performance in computer-aided diagnosis: the effect of finite sample sizeMed Phys2720001509–1522
31.
ZhaoG.LiuF.OlerJ.A.MeyerandM.E.KalinN.H.BirnR.M.Bayesian convolutional neural network based MRI brain extraction on nonhuman primatesNeuroimage175201832–44
32.
MorraJ.H.TuZ.ApostolovaL.G.GreenA.E.TogaA.W.ThompsonP.M.Comparison of AdaBoost and support vector machines for detecting Alzheimer's disease through automated hippocampal segmentationIEEE Trans Med Imaging29201030–43
33.
ParkS.C.SukthankarR.MummertL.SatyanarayananM.ZhengB.Optimization of reference library used in content-based medical image retrieval schemeMed Phys3420074331–4339
34.
HitzlW.ReitsamerH.A.HornykewyczK.MistlbergerA.GrabnerG.Application of discriminant, classification tree and neural network analysis to differentiate between potential glaucoma suspects with and without visual field defectsJ Theor Med52003161–170
35.
BaumE.B.HausslerD.What size net gives valid generalization?Neural Comput11989151–160
36.
HaykinS.Multilayer perceptronsHaykinS.Neural Networks: A Comprehensive Foundation2nd ed.1998Prentice HallUpper Saddle River, NJ205–226
ChoJ.LeeK.ShinE.ChoyG.DoS.How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?arXiv preprint2015 arXiv:1511.06348. Available at: https://arxiv.org/pdf/1511.06348.pdf
39.
FukunagaK.HayesR.R.Effects of sample size in classifier designIEEE Trans Pattern Anal Mach Intell111989873–885
40.
WagnerR.F.ChanH.-P.SahinerB.PetrickN.MossobaJ.T.Finite-sample effects and resampling plans: applications to linear classifiers in computer-aided diagnosisMed Imaging30341997467–477
41.
ChanH.-P.SahinerB.WagnerR.F.PetrickN.Effects of sample size on classifier design for computer-aided diagnosisProc SPIE Conf Medical Imaging33381998845–858
42.
ChanH.P.SahinerB.WagnerR.F.PetrickN.MossobaJ.Effects of sample size on classifier design: quadratic and neural network classifiersImage Process Med Imaging3034Pts 1 219971102–1113 1150
43.
DobbinK.K.ZhaoY.SimonR.M.How large a training set is needed to develop a classifier for microarray data?Clin Cancer Res142008108–114
44.
DobbinK.K.SimonR.M.Sample size planning for developing classifiers using high-dimensional DNA microarray dataBiostatistics82007101–117
45.
MukherjeeS.TamayoP.RogersS.Estimating dataset size requirements for classifying DNA microarray dataJ Comput Biol102003119–142
46.
BeleitesC.NeugebauerU.BocklitzT.KrafftC.PoppJ.Sample size planning for classification modelsAnal Chim Acta760201325–33
47.
FigueroaR.L.Zeng-TreitlerQ.KandulaS.NgoL.H.Predicting sample size required for classification performanceBMC Med Inform Decis Mak1220128
48.
Samper-GonzálezJ.BurgosN.FontanellaS.Yet another ADNI machine learning paper? Paving the way towards fully-reproducible research on classification of Alzheimer's diseaseProc Machine Learning in Medical Imaging MLMI 2017, MICCAI Worskhop, Lecture Notes in Computer Science10541201753–60
49.
SanchezB.N.WuM.SongP.X.WangW.Study design in high-dimensional classification analysisBiostatistics172016722–736
50.
VapnikV.N.ChervonenkisA.Y.On the uniform convergence of relative frequencies of events to their probabilitiesTheory Probab Appl161971264–280
