Comparison of the diagnostic accuracy,sensitivity and specificity of four odontological methods for age evaluation in Italian children at the age threshold of 14 years using ROC curves

Abstract

The age threshold of 14 years is relevant in Italy as the minimum age for criminal responsibility. It is of utmost importance to evaluate the diagnostic accuracy of every odontological method for age evaluation considering the sensitivity, or the ability to estimate the true positive cases, and the specificity, or the ability to estimate the true negative cases. The research aims to compare the specificity and sensitivity of four commonly adopted methods of dental age estimation – Demirjian, Haavikko, Willems and Cameriere – in a sample of Italian children aged between 11 and 16 years, with an age threshold of 14 years, using receiver operating characteristic curves and the area under the curve (AUC). In addition, new decision criteria are developed to increase the accuracy of the methods. Among the four odontological methods for age estimation adopted in the research, the Cameriere method showed the highest AUC in both female and male cohorts. The Cameriere method shows a high degree of accuracy at the age threshold of 14 years. To adopt the Cameriere method to estimate the 14-year age threshold more accurately, however, it is suggested – according to the Youden index – that the decision criterion be set at the lower value of 12.928 for females and 13.258 years for males, obtaining a sensitivity of 85% and specificity of 88% in females, and a sensitivity of 77% and specificity of 92% in males. If a specificity level >90% is needed, the cut-off point should be set at 12.959 years (82% sensitivity) for females.

Keywords

dental age estimation sensitivity specificity legal age threshold ROC curves

Introduction

Dental age estimation (DAE), which analyses the maturation of the radicular structure of the teeth, has been a well-known and widely utilised procedure for a long time.^1–4 Several methods of DAE in subadults are based on the study of seven permanent teeth, while others, which are dedicated mostly to the DAE of individuals older than 16, rely on analysis of the third molar.^5–10 Every method, however, has different accuracy, sensitivity and specificity in the estimation when considered at different age thresholds.^11,12 To date, very little research has focused on the sensitivity and specificity indices and the connected rate of false age classifications (false attribution over/under the threshold) especially for the age threshold of 14 years.¹³ Moreover, receiver operating characteristic (ROC) curves have been seldom experimented to evaluate the performance of dental methods for age assessment.^14,15

According to Italian law, there are two different age thresholds to consider. The minimum age for criminal accountability is 14 years. Every case involving individuals between 14 and 18 years of age must then be established individually using different criteria, depending mainly on the assessment of the psychological maturity of each subject. An individual older than18 years of age is considered a fully responsible and accountable adult.

In criminal law, one of the most important issues which often arises during the analysis of the age estimation procedures is the different specificity index of the adopted method which expresses the rate of false positives.¹⁶ Even if an age misclassification over the threshold does not lead to an erroneous guilty verdict, it could imply heavy legal and ethical consequences, since it could result in a minor being treated as an accountable juvenile (age > 14 years) or as an adult (age > 18 years).¹⁷ Hence, in criminal proceedings, only methods which are characterised by very high specificity and which can provide estimations with a very high probability (at least 90%) can fulfil the legal requirements and support the criminal court in the process of age assessment.¹⁸ Otherwise, in a civil law context, the operator is allowed to take a different approach to the specificity and sensibility of the method of age estimation, since in Italy and many other countries, a probability of 51% may suffice for age assessment in civil proceedings.¹⁹

The present study is a continuation of previous research¹⁶ which explored and compared the accuracy in estimating the age of a sample of Italian subadults using four different methods: the Cameriere (C),^20,21 the Demirjian (D),^22,23 the Haavikko (H)²⁴ and the Willems (W)^25–27 methods. As in the previous research, methods were applied based on the maturation of the dentition up to the second molar and excluding the third. Therefore, Cameriere's research which considered all the right mandibular teeth is not considered here.²⁸ Starting from the specificity and sensitivity indexes evaluated for the same methods (D, W, H and C), the present research explores the performance of these methods in predicting the attainment of the age threshold of 14 years. Moreover, the statistical analysis based on ROC curves and new decision criteria are developed, thereby yielding an improvement in the performances of the methods with respect to their best values of specificity and sensitivity, which are inherently linked.^29,30

Materials and methods

A total of 501 digital ortopantomographs (OPGs) of Italian children of Caucasian origin were taken from three selected clinical radiology offices from northern, southern and central Italy. The inclusion criteria were aged between 11 years (4015 days) and 15 years and 364 days (5839 days), Italian nationality, Caucasian ethnic group, an unremarkable medical history and digital files and good quality images available. Exclusion criteria were systemic diseases; premature birth; congenital anomalies; oral pathologies such as tooth agenesia, endodontic treatments, large carious lesions involving the dental pulp, gross mandibular pathologies; X-rays of poor quality; and incomplete data regarding sex or chronological age.

One operator, blind to the chronological age of the subjects and already trained in DAE, analysed all the OPGs. The age of the subjects was calculated in days (date of OPG exposition–date of birth), and the cohorts were composed of a very similar number of subjects for both sexes (approximately 20 individuals; Table 1).

Table 1.

The composition of the sample.

Age	Females (%)	Males (%)
11	56 (22)	46 (19)
12	51 (20)	46 (19)
13	47 (18)	43 (18)
14	50 (19)	56 (23)
15	53 (21)	53 (22)

Four methods were adopted to provide dental age estimations, and each one took into consideration the maturation of dentition up to the second molar (excluding the third molar). The methods were:

The original Demirjian (D) method for seven teeth, with values taken at the 50th percentile.

The Willems (W) conversion score elaborated with the polynomial regression system for Belgian individuals. The estimated age was calculated at the 50th percentile.

The Cameriere (C) method, using the European formula, as presented in the AgEstimation Project website (http://agestimation.unimc.it).

The Haavikko (H) method, with values taken at the 50th percentile.

Statistics

The intra-rater agreement was calculated for each method using the intra-class correlation coefficient, rescoring 50 radiographs six weeks after the first assessment of dental development. Inter-rater agreement was calculated using two other examiners using the intra-class correlation coefficient for each method.

Using the age threshold of 14 years, the area under the ROC curve (AUC)^29,30 was calculated for each method and each sex. The binomial exact 95% confidence intervals (CI) for the AUC were also calculated.

A ROC curve is a graphical plot which illustrates the performance of a binary classifier system as its discrimination threshold is varied. It is created by plotting the fraction of true positives out of the total actual positives (true positive rate or sensitivity) versus the fraction of false positives out of the total actual negatives (false positive rate or 1-specificity), at various threshold settings. In the present context, the sensitivity represents the rate of subjects correctly classified as ≥14 years old. The specificity represents the rate of the subjects correctly classified as <14 years. An AUC of 1 represents a perfect test; an AUC of 0.5 represents a worthless test.

Comparisons in AUC for C, D, H and W methods were performed for females and males using the test of De Long.³¹

A criterion was chosen maximising the Youden index (sensitivity-[1-specificity]), and sensitivity, specificity, accuracy (the proportion of all corrected results) and positive predictive value (PPV) were calculated.³² Sensitivity and specificity are measures of intrinsic diagnostic accuracy because they are not affected by the prevalence of the condition. Conversely, the accuracy and PPV are influenced by the prevalence of the condition. This new criterion was chosen to improve the estimate of the age threshold of 14 years in the Italian population, and for the best method, a new criterion with a specificity of 90% was calculated.

Results

Intra-class correlation coefficients for intra-rater agreement are reported in Table 2, and intra-class correlation coefficients for inter-rater agreement are reported in Table 3. The values of the intra-class correlation coefficients were >0.80 for each method.

Table 2.

Intra-class correlation coefficients (ICC) for intra-rater agreement.

Method	ICC	Lower limit 95% CI	Upper limit 95% CI
Cameriere	0.92	0.83	0.95
Demirjian	0.87	0.76	0.93
Haavikko	0.92	0.84	0.96
Willems	0.92	0.97	0.96

Table 3.

ICC for inter-rater agreement.

Method	ICC	Lower limit 95% CI	Upper limit 95% CI
Cameriere	0.84	0.80	0.87
Demirjian	0.81	0.76	0.85
Haavikko 50	0.83	0.78	0.86
Willems 50	0.88	0.86	0.89

The sample was almost equally divided into males (244) and females (257), and nearly equally divided into age cohorts from 11 to 16 years (Table 1). One hundred and three subjects (40%) in the female group, and 109 (45%) in the male group, were 14 years old or older.

AUC, Youden criterion, sensitivity and specificity for females and males are reported respectively in Tables 4 and 5. ROC curves are shown in Figures 1 and 2.

Table 4.

AUC, Youden criterion, sensitivity, specificity, accuracy and PPV for females.

Method	AUC	AUC 95% CI	Criterion (>)	Sensitivity	Specificity	Accuracy	PPV
Willems	0.87	0.83–0.91	14.88	0.78	0.86	0.83	0.79
Demirijan	0.87	0.83–0.91	13.6	0.82	0.84	0.83	0.78
Cameriere	0.90	0.86–0.94	12.928	0.85	0.88	0.87	0.82
Haavikko	0.85	0.80–0.89	12.20	0.72	0.86	0.81	0.78

AUC: area under the curve; Accuracy: proportion of all corrected results; PPV: positive predictive value.

Table 5.

AUC, Youden criterion, sensitivity, specificity, accuracy and PPV for males.

Method	AUC	AUC 95% CI	Criterion (>)	Sensitivity	Specificity	Accuracy	PPV
Willems	0.83	0.78–0.88	14.21	0.74	0.79	0.77	0.74
Demirijan	0.84	0.79–0.89	14.4	0.80	0.76	0.78	0.73
Cameriere	0.90	0.86–0.93	13.258	0.77	0.92	0.85	0.88
Haavikko	0.87	0.82–0.91	12.53	0.88	0.70	0.78	0.70

Figure 1.

Receiver operating characteristic (ROC) curve for each method in females.

Figure 2.

ROC curve for each method in males.

Differences between methods in AUC are reported in Table 6 for females and in Table 7 for males. In females, the Cameriere method had an AUC higher than the Willems and Haavikko methods. In males, the Cameriere method had an AUC higher than the other methods, and the Willems method had an AUC lower than the other methods.

Table 6.

Difference in AUC between methods in females.

Comparison	AUC Difference	95% CI	p-Value
C vs. H	0.06	0.02–0.10	.0040
C vs. W	0.03	0.001–0.06	.0437
C vs. D	0.03	–0.001–0.06	.0581
D vs. H	0.03	–0.01–0.07	.1563
W vs. H	0.03	–0.01–0.07	.1700
D vs. W	0.001	–0.01–0.01	.8053

Table 7.

Difference in AUC between methods in males.

Comparison	AUC difference	95% CI	p-Value
C vs. W	0.07	0.04–0.09	<.0001
C vs. D	0.06	0.03–0.08	.0001
C vs. H	0.03	0.003–0.06	.0322
H vs. W	0.03	0.0002–0.07	.0489
D vs. W	0.01	0.001–0.02	.0282
H vs. D	0.02	–0.01–0.06	.1452

ROC curves are a graphical chart built to plot the fraction of true positives (and so the sensitivity) versus the fraction of false positives (1-specificity). True positives make the curve grow upward, while false positives make the curve shift to the right. Every curve creates an area below it called the area under the curve (AUC); the higher the value of the area, the higher is the predictive value of the test in terms of the combination of sensitivity and specificity.

Moreover, with the need to minimise the false positive cases, as they are the worst possible error in DAE in criminal law cases, it can be useful to change the cut-off point of the different methods to get the best results in terms of the combination of sensitivity and specificity. The cut-off point is the value that discriminates between positive and negative results.

The Youden criterion for the Cameriere method in females was 12.928 years. The sensitivity was 85% [95% CI: 77–93%] and the specificity was 88% [95% CI: 81–92%]. Setting a specificity of 90%, the criterion becomes 12.959 years with a sensitivity of 82% [95% CI: 73–88%]. The Youden criterion for the Cameriere method in males was 13.258 years. The sensitivity was 77% [95% CI: 68–85%], and the specificity was 92% [95% CI: 86–96%]. There is no need to set a specificity of 90% in this case because the Youden criterion has a specificity of 92%

Discussion

There has been great interest for many years in the evaluation of age in subadults via the analysis of dental structure calcification and maturation, with a particular focus on specific age thresholds (i.e. 14, 16 and 18 years). There is strong evidence in the literature of the accuracy of the methods for DAE based on the analysis of OPGs.^33–39 Each, however, offers a different specificity and sensitivity when applied to specific age thresholds, and most research has focused on the third molar and the higher age thresholds (16, 18 and 21 years). The issue of specificity, moreover, is of special importance – especially in a criminal law context – to avoid false positive attributions.

This study therefore compared four of the most common methods for DAE in an Italian sample distributed around the age threshold of 14 years (11–16 years). With the aid of ROC curve analysis, it aimed to find a method of evaluation that combines the highest possible specificity, which is necessary in the context of criminal case analysis, and the highest obtainable sensitivity in order to limit the percentage of false negatives. ROC curves are very helpful in determining the diagnostic capabilities of different methods. With the AUC value, it is easy to establish which of the methods gives the best combination of sensitivity and specificity.

The ROC curves examined were divided by sex and method.

The results of the present study show that the Cameriere method, which is based on the maturation of dentition up to the second molar and uses the European formula, provides the highest AUC value both for males and females. The Cameriere method shows a highly discriminative accuracy for the age threshold of 14 years, even if it was not developed for this purpose. Cameriere himself proposed an alternative approach that also included the wisdom tooth when a cut-off age of 14 years is used.¹⁵ In fact, the European formula for a subject with a complete calcification of the seven mandibular permanent teeth allows a maximum attribution of 13.689 in males and 14.064 in females.

In terms of the Haavikko and Cameriere methods, it is necessary to set the cut-off point at a lower level to have greater accuracy of estimation at the age threshold of 14 years, while in the Demirjian and Willems methods, there is no need for a cut-off point change.

Therefore, to estimate more effectively the age threshold of 14 years in the Italian population, the Cameriere method can be used with a proper adjustment. In fact, setting the cut-off point at 12.928 for females and 13.258 years for males, it is possible to obtain a sensitivity percentage of 85% and a specificity percentage of 88% in females, and a sensitivity of 77% and specificity of 92% in males. If a specificity level >90% is needed, the cut-off point can be set at 12.959 (82% sensitivity) for females.

Conclusions

After examining a sample of Italian children aged between 11 and 16 years using ROC curves, it can be stated that:

In the case of the Haavikko and Cameriere methods, it is necessary to set the cut-off point at a lower level to have greater accuracy of estimation at the age threshold of in 14 years. Considering the Youden index, in the Cameriere method, the cut-off point should be set at 12.928 years in females and 13.258 years in males; in the Haavikko method, the cut-off point should be set at 12.20 years in females and 12.53 in males.

The Cameriere method has the highest AUC value in both males and females. This method shows a high discrimination accuracy for the age threshold of 14 years.

In the Cameriere method, by setting the cut-off point at 12.928 for females and 13.258 years for males, it is possible to obtain a sensitivity of 85% and specificity of 88% in females, and a sensitivity of 77% and specificity of 92% in males. If a specificity level >90% is needed, the cut-off point can be set at 12.959 (82% sensitivity) for females.

Future research should validate the proposed approach in a larger sample and investigate possible improvements of the predictions by applying the Cameriere's method taking into consideration the apex opening of the third molars as well.

Footnotes

Funding

This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References

Merwin

Harris

. Sibling similarities in the tempo of human tooth mineralization. Arch Oral Biol 1998; 43: 205–210.

Demirjian

. Interrelationships among measures of somatic, skeletal, dental and sexual maturity. Am J Orthod 1985; 88: 433–438.

Liversidge

Molleson

. Variation in crown and root formation and eruption of human deciduous teeth. Am J Phys Anthropol 2004; 123: 172–180.

Ritz-Timme

Cattaneo

Collins

. Age estimation: the state of the art in relation to the specific demands of forensic practice. Int J Legal Med 2000; 113: 129–136.

Harris

. Mineralization of the mandibular third molar: a study of American blacks and whites. Am J Phys Anthropol 2007; 132: 98–109.

Thevissen

Algerban

Asaumi

. Human dental age estimation using third molar development stages: accuracy of age predictions not using country specific information. Forensic Sci Int 2010; 201: 106–111.

Thevissen

Fieuws

Willems

. Third molar development: measurements versus scores as age predictor. Arch Oral Biol 2011; 56: 1035–1040.

Thevissen

Kaur

Willems

. Human age estimation combining third molar and skeletal development. Int J Legal Med 2012; 126: 285–292.

Olze

Pynn

Kraul

. Studies on the chronology of third molar mineralization in First Nations people of Canada. Int J Legal Med 2010; 124: 433–437.

10.

Liversidge

Marsden

. Estimating age and the likelihood of having attained 18 years of age using mandibular third molars. Br Dent J 2010; 209: E13.

11.

Staaf

Mörnstad

Welander

. Age estimation based on tooth development: a test of reliability and validity. Scand J Dent Res 1991; 99: 281–286.

12.

Nik-Hussein

Kee

Gan

. Validity of Demirjian and Willems methods for dental age estimation for Malaysian children aged 5–15 years old. Forensic Sci Int 2011; 204: 208.e1–6.

13.

Cameriere

Cingolani

Giuliodori

. Radiographic analysis of epiphyseal fusion at knee joint to assess likelihood of having attained 18 years of age. Int J Legal Med 2012; 126: 889–899.

14.

Martin-de las Heras

Pedro Garcia-Fortea

Ortega

. Third molar development according to chronological age in populations from Spanish and Magrebian origin. Forensic Sci Int 2008; 174: 47–53.

15.

Cameriere

Brkic

Ermenc

. The measurement of open apices of teeth to test chronological age of over 14-year olds in living subjects. Forensic Sci Int 2008; 174: 217–221.

16.

Pinchi

Norelli

G-A

Pradella

. Comparison of the applicability of four odontological methods for age estimation of the 14 years legal threshold in a sample of Italian adolescents. J Forensic Odontostomatol 2012; 30: 17–25.

17.

Corradi

Pinchi

Barsanti

. Optimal age classification of young individuals based on dental evidence in civil and criminal proceedings. Int J Legal Med 2013; 127: 1157–1164.

18.

Focardi

Pinchi

De Luca

. Age estimation for forensic purposes in Italy: ethical issues. Int J Legal Med 2014; 128: 515–522.

19.

Corradi

Pinchi

Barsanti

. Probabilistic classification of age by third molar development: the use of soft evidence. J Forensic Sci 2013; 58: 51–59.

20.

Cameriere

Ferrante

Cingolani

. Age estimation in children by measurement of open apices in teeth. Int J Legal Med 2006; 120: 49–52.

21.

Cameriere

De Angelis

Ferrante

. Age estimation in children by measurement of open apices in teeth: an European formula. Int J Legal Med 2007; 121: 449–453.

22.

Demirjian

Goldstein

Tanner

. A new system of dental age assessment. Hum Biol 1973; 45: 211–227.

23.

Demirjian

Goldstein

. New systems for dental maturity based on seven and four teeth. Ann Hum Biol 1976; 3: 411–421.

24.

Haavikko

. Tooth formation age estimated on a few selected teeth. A simple method for clinical use. Proc Finn Dent Soc 1974; 70: 15–19.

25.

Chaillet

Willems

Demirjian

. Dental maturity in Belgian children using Demirjian's method and polynomial functions: new standard curves for forensic and clinical use. J Forensic Odontostomatol 2004; 22: 18–27.

26.

Willems

Van Olmen

Spiessens

. Dental age estimation in Belgian children: Demirjian's technique revisited. J Forensic Sci 2001; 46: 893–895.

27.

Willems

Thevissen

Belmans

. Willems II. Non-gender-specific dental maturity scores. Forensic Sci Int 2010; 201: 84–85.

28.

Ferrante

Cameriere

. Statistical methods to assess the reliability of measurements in the procedures for forensic age estimation. Int J Legal Med 2009; 123: 277–283.

29.

Zou

O'Malley

Mauri

. Receiver-operating characteristic analysis for evaluating diagnostic test and predictive models. Circulation 2007; 115: 654–657.

30.

Hanley

McNeil

. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 1983; 148: 839–843.

31.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 1988; 44: 837–845.

32.

Youden

. Index for rating diagnostic tests. Cancer 1950; 3: 32–35.

33.

Liversidge

Smith

Maber

. Bias and accuracy of age estimation using developing teeth in 946 children. Am J Phys Anthropol 2010; 143: 545–554.

34.

Cameriere

Ferrante

Liversidge

. Accuracy of age estimation in children using radiograph of developing teeth. Forensic Sci Int 2008; 176: 173–177.

35.

Philips

Van Wyk Kotze

. Testing standard methods of dental age estimation by Moorrees, Fanning and Hunt and Demirjian, Goldstein and Tanner on three South African children samples. J Forensic Odontostomatol 2009; 27: 20–28.

36.

Schmeling

Grundmann

Fuhrmann

. Criteria for age estimation in living individuals. Int J Legal Med 2008; 122: 457–460.

37.

Grover

Mohan Marya

Avinash

. Estimation of dental age and its comparison with chronological age: accuracy of two radiographic methods. Med Sci Law 2012; 52: 32–35.

38.

Hongwei

Jingtao

Cameron

. Age determination of the molars. Med Sci Law 1991; 31: 65–68.

39.

Ambarkova

Galić

Vodanović

. Dental age estimation using Demirjian and Willems methods: cross sectional study on children from the former Yugoslav republic of Macedonia. Forensic Sci Int 2014; 234: 187.e1–187.e7.