Improved Diabetes Screening Using an Extended Predictive Feature Search

Abstract

Background:

Screening entire populations for diabetes is not cost-effective. Hence, an efficient screening process must select those people who are at high risk for diabetes. In this study, we investigated whether screening procedures could be improved using an extended predictive feature search.

Materials and Methods:

In order to develop our model and identify persons with diabetes (prevalence) we used data from years of the National Health and Nutrition Examination Survey (2005–2010), which has not been explored for this purpose before. We calculated all combinations of predictors in order to identify the optimal subset, and we used a linear logistic classification model to predict diabetes. V-fold cross-validation was used for the process of including variables and for validating the final models. This new model was compared with two established models.

Results:

In total, 5,398 participants were included in this study. Among these, 478 participants had unidentified diabetes. The established models had a receiver operating characteristics curve for the area under the curve (AUC) of 0.74 and 0.71 compared with an AUC of 0.78 for the new model, showing a significant difference (P<0.05). A proposed cutoff point for the established models yielded respective sensitivities/specificities of 63%/72% and 40%/72% compared with the new model, which had a sensitivity/specificity of 70%/72%.

Conclusions:

Our data indicate that simple healthcare and economic information such as ratio of family income to poverty can add value in deciding who is at risk of unknown diabetes by using extended investigations of predictor combinations.

Introduction

The prevalence of type 2 diabetes is increasing, and the disease is associated with a major increase in morbidity, mortality, and healthcare costs.¹ In 2010, the prevalence of type 2 diabetes in the United Kingdom, United States, China, and Arabic Emirates ranged from 7% to as high as 34% of the population.^2,3 In the United States, an estimated 7 million people have undiagnosed diabetes,⁴ and when finally diagnosed, up to 30% show clinical manifestations of diabetes complications.⁵ Early diagnosis of type 2 diabetes is thus very important, as intensive diabetes control can significantly reduce long-term complications.^6
–8

Inasmuch as screening entire populations is not cost-effective, priority in the screening process must rely on selecting those people at high risk for diabetes.^9,10 Several studies of diabetes screening have been published during the last decade.^2,11 Risk prediction or risk stratification models have a substantial potential to be utilized in a screening context in order to identify high-risk individuals who would subsequently undergo testing for diabetes. These models often include a combination of predictors, such as anthropometrics, lifestyle, hereditary conditions, and clinical measurements.¹¹ Multivariable statistical methods (i.e., logistic regression) are used to combine these risk predictors into a model.^11,12 Despite the large number of proposed models, not many are used in daily clinical practice. Optimal model performance is crucial for all tests. Most models are based on elimination of predictors with focus on statistical significance of the model, which may not yield optimal performance.¹¹ Cohort data obtained for a different scientific purpose are often used, and this can limit the models' use in an entire population. Furthermore, attempts are often made to construct models that present risk scores that reflect the complexity of the data, but that also may be perceived as simple and applicable in clinical practice. This is commonly done by collapsing continuous variables into two or more categories or preselecting predictors based on subjective judgment.¹³ From a performance point of view, this may lead to an oversimplification and deterioration of the effectiveness of the proposed models.^14,15

In the present study, we investigated the feasibility and performance of a model based on extended predictive features and compared it with two widely accepted models.

Research Design and Methods

We used data from multiple years of the National Health and Nutrition Examination Survey (NHANES) (2005–2010),¹⁶ where results from an oral glucose tolerance test (OGTT) and a fasting plasma glucose test were available to identify persons with undiagnosed diabetes using a logistic classification model.

The NHANES is a cross-sectional study conducted by the National Center for Health Statistics of the Centers for Disease Control and Prevention. To represent the U.S. population, NHANES uses complex, multistage probability sampling of the civilian, noninstitutionalized population. To produce reliable statistics, NHANES oversamples elderly persons and some racial and ethnic minorities. Participants aere limited to those 20 years of age and above. Pregnant women and participants with diagnosed diabetes are excluded. Based on these data, we developed and tested a new model for diabetes screening and compared it with two accepted screening models.

Selection of predictors to be included in this new model as well as validation of the model was done by V-fold cross-validation. We calculated all 16,383 possible combinations of predictors in order to find the optimal subset. We derived and tested the model on all combinations of nine (of 10) partitions of training data and one (of 10) partitions of test data. The accepted statistical methods ensured valid testing of the model performance, reducing generalization bias.^17,18

End point

Our primary objective was detection of undiagnosed diabetes (prevalence) in the cohort of NHANES. OGTT results of≥11.1 mmol/L or fasting plasma glucose of >7 mmol/L was used as a threshold for the diagnosis of diabetes. The American Diabetes Association recommends that a test result diagnostic of diabetes should be repeated to rule out laboratory error, unless the diagnosis is clear on clinical grounds.¹⁹ Multiple test results were not available in our dataset, so one positive test was considered to be a diagnostic criterion.

Predictive variables

Multiple variables were selected to be investigated for their ability to distinguish between those without diabetes and those with undiagnosed diabetes. We selected several well-established risk factors, such as age, body mass index (BMI), and hereditary conditions,^20

–23 all available in the NHANES dataset. In addition, we selected several more uncertain predictors, such as self-perceived oral health,²⁴ which can be an early complication of diabetes,²⁵ and socioeconomic status in the form of income and educational level, which have been associated with diabetes-related mortality.²⁶

Possible predictor candidates were age, sex, family history of diabetes (yes/no) (questionnaire), history of hypertension (yes/no) (questionnaire), BMI, physical activity (yes/no) (questionnaire), waist circumference, educational level (questionnaire), income (questionnaire), race, self-perceived health (questionnaire), nicotine use (questionnaire), self-perceived dental condition (questionnaire), and blood pressure.

The NHANES questionnaire concerning physical activity was changed from 2006 to 2007 and forward. We considered the participants to be physically active if they answered “yes” to the following question: “Over the past 30 days, did you do any vigorous activities for at least 10 min that caused heavy sweating, or large increases in breathing or heart rate?” in the 2005–2006 cohort, or the question “Do you carry out any vigorous-intensity sports, fitness, or recreational activities that cause large increases in breathing or heart rate, like jogging or basketball, for at least 10 minutes continuously?” in the 2007–2010 cohort. Participants' family histories of diabetes were considered as binary without considering which family member had diabetes. Assessment of self-perceived oral health was formulated as “Rate the health of your teeth and gums” on a scale from 1 (excellent) to 5 (poor).

Missing data

We used multiple imputation to replace missing values. This imputation technique involves creating multiple copies of the data and replacing missing values with imputed values on the basis of a suitable random sample from their predicted distribution. Table 1 gives the distribution of missing values between variables included in the modeling. Multiple imputations allow patients with incomplete data to be included in analyses, thereby making full use of all available data and increasing precision and power without compromising validity.²⁷

Table 1.

Characteristics of the Population Studied

Variable	Missing [n (%)]	Without diabetes (n=4,920)	With diabetes (n=478)	P value
BMI (kg/m²)	459 (8.5%)	28±6.4	31±6.6	<0.05
Waist (cm)	97 (1.8%)	97±15	106±15	<0.05
Gender (women %)	0	2,465 (50%)	215 (45%)	<0.05
Age (years)	259 (4.8%)	47±16	60±14	<0.05
History of family diabetes? (yes %)	0	1,824 (37%)	201 (42%)	<0.05
Physically active? (yes %)	0	1,331 (27%)	43 (9%)	<0.05
Told high blood pressure (yes %)	0	1,429 (29%)	25 (53%)	<0.05
Told high cholesterol (yes %)	1,619 (30%)	2,070 (42%)	220 (46%)	NS
Education level (%)	0	—	—	—
Less than 9^th grade	—	490 (10%)	86 (18%)	<0.05
9–11^th grade	—	736 (15%)	82 (17%)	NS
High school graduate or equivalent	—	1,183 (24%)	143 (30%)	<0.05
Some college or associate degree	—	1,380 (28%)	105 (22%)	<0.05
College graduate or above	—	1,134 (23%)	62 (13%)	<0.05
Ratio of family income to poverty	389 (7.2%)	2.63±1.6	2.47±1.5	<0.05
Race/ethnicity (%)	0	—	—	—
Mexican American	—	884 (18%)	96 (20%)	NS
Other Hispanic	—	441 (9%)	43 (9%)	NS
Non-Hispanic white	—	2,465 (50%)	268 (56%)	<0.05
Non-Hispanic black	—	887 (18%)	268 (12%)	<0.05
Other race (including multiracial)	—	243 (5%)	14 (3%)	NS
Self-perceived health (1–10)	200 (3.7%)	2.72±0.9	2.97±1.0	NS
Nicotine use (yes %)	211 (3.9%)	1,282 (26%)	105 (22%)	<0.05
Self-perceived dental condition (1–10)	437 (8.1%)	3.23±4	3.65±7	NS

Data are mean±SD or proportion of the group. Missing data describe the percentage of participants lacking information about a particular predictor.

NS, not significant.

Comparison with conventional models

Bang et al.²⁸ used data from NHANES 1999–2004 to model and develop a “patient self-assessment score” for use in a wide variety of community settings and clinical encounters (including patient waiting rooms or on the Internet) via a simple pencil-and-paper method. In brief, the model was compared with several other scoring systems and validated on a cohort of approximately 21,000 participants, with the primary end point being diagnosis of diabetes. The final model included variables of age, sex, obesity (BMI), history of hypertension, family history of diabetes, and physical activity predictors and was simplified for user-friendliness and presentation. Not many proposed models have been designed for and validated on an entire adult population (age >20 years). This is one advantage of using the model formulated by Bang et al.²⁸ as a foundation for comparison with our new model. In addition, we also used a model by Baan et al.,²⁹ developed from a European Dutch population, for comparison. In brief, participants from the Rotterdam Study, 55–75 years old, not known to have diabetes, completed a questionnaire on diabetes-related symptoms and risk factors and underwent a clinical examination. The main finding from the evaluation of Baan et al.²⁹ was that the additional information from the questionnaire sparsely improved the prediction capability compared with using routinely collected data only.

We evaluated prediction models through sensitivity and specificity for predetermined cutoff points and receiver operating characteristics (ROC) based on logistic regression models comparing the area under the curve (AUC) of the new model with that of the sparser models proposed by Bang et al.²⁸ and Baan et al.²⁹ We used Student's t tests or z tests for proportions in order to compare the difference between the groups with and without diabetes.

Results

In total, 51 participants with diagnosed diabetes and 203 pregnant women were excluded. Thus, the study included a total of 5,398 participants. The cohort characteristics are given in Table 1. This cohort included 478 (8.8%, 478 of 5,398) patients with undiagnosed type 2 diabetes. Participants with undiagnosed diabetes had a tendency to be older, had a lower level of physical activity and a higher waist circumference and BMI, were less educated, earned less income, had less use of nicotine, and had a higher incidence of diabetes in their families than their counterparts without diabetes (P<0.05).

The final model, selected on the basis of performance from all possible combinations of predictors, was composed of BMI, waist circumference, age, history of hypertension, family history of diabetes, physical activity, education level, and race. Figure 1 presents the ROC curves for the two models. The established models had AUCs of 0.74 and 0.71, respectively, compared with an AUC of 0.78 (P<0.05 for both comparisons) for the new model. Table 2 presents sensitivity, specificity, positive predictive value, and negative predictive value at proposed cutoff points by Bang et al.²⁸ compared with the new model and the model of Baan et al.²⁹ with fixed specificity. For the cutoff point of 4 (risk score), the established models yielded sensitivities of 82% and 57%, respectively, 95% confidence intervals of 79.4%, 83.7% and 55.1, 58.5%, respectively, and a specificity of 56% for both. The new extended model yielded a sensitivity of 85% with 95% confidence interval (83.1%, 87.6%) when the specificity was maintained at the same level.

FIG. 1.

Receiver operating characteristics curve for the detection of undiagnosed type 2 diabetes: curve A, logistic model including age, sex, family history of diabetes, history of hypertension, body mass index, degree of physical activity, waist circumference, education level, income, race, health, nicotine use, dental condition, and blood pressure; curve B, Bang et al.²⁸ screening score (1–10), based on age, sex, family history of diabetes, history of hypertension, body mass index, and physical activity; and curve C, Baan et al.²⁹ screening score, using age, antihypertension medication, body mass index, and gender as predictors and the oral glucose tolerance test or fasting plasma glucose as the reference standard.

Table 2.

Model Comparison Using Data from the 2005–2010 National Health and Nutrition Examination Surveys (n=5,398)

Cutoff, model	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
Cutoff >4
New model	85 (83.1, 87.6)	56	14	98
Bang et al.²⁸ ≥4	82 (79.4, 83.7)	56	13	97
Baan et al.²⁹	57 (55.1, 58.5)	56	9	94
Cutoff >5
New model	70 (68.2, 71.9)	72	18	97
Bang et al.²⁸ ≥5	63 (61.7, 65.1)	72	16	96
Baan et al.²⁹	42 (40.7, 42.9)	72	11	94

Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for different cutoff points were calculated. Sensitivities are shown with 95% confidence interval.

For the cutoff point of 5 (risk score), the established models yielded a sensitivity of 63% (61.7%, 65.1%) and 42% (40.7%, 42.9%), respectively, and a specificity of 72%. The new model yielded a sensitivity of 70% (68.2%, 71.9%) when the specificity was held at the same level. The new model thus yields better sensitivity at both cutoff points. The increased sensitivity from the new model corresponds to discovering 33 additional patients with diabetes compared with Bang et al.²⁸ and an additional 129 patients compared with Baan et al.²⁹ when the cutoff point of 5 (risk score) is used in the population sample.

The total computation time for all predictor combinations for the new model took approximately 8 h on a standard laptop (Dell, Round Rock, TX) (Intel^® [Santa Clara, CA] Core i5-2410 CPU with 2.30 GHz, 4 GB RAM, 64-bit) using a self-written algorithm (MATLAB^® R2011b; Mathworks, Natick, MA).

Discussion

Numerous diabetes prediction models have been developed over the years,^{2,11
–13,28

–31} and most studies use modeling approaches focusing on simplicity and statistical selection of predictors rather than performance. Most studies also use specific cohorts, which often exhibit small age ranges or racial/ethnic origin groups, thus limiting their generalizability to an entire population. We explored whether an extended investigation of predictor combinations could lead to better performance in a diabetes screening model for a population of people over 20 years of age.

Our results show that information about education level, racial/ethnic origin, and waist circumference can significantly improve sensitivity and specificity of an established model when screening for diabetes by using a full investigation of possible predictor combinations. Using this kind of information can help reduce unwarranted screening for type 2 diabetes, thus lowering costs¹⁰ of performing additional test such as glycosylated hemoglobin/OGTT for high-risk persons. Waist circumference and/or BMI have been used in most models,^{12,13,28

–31} as has racial/ethnic origin, which is not surprising, as we know that racial differences are associated with the development of diabetes in some groups.³² To our knowledge, information about education level has not been used in screening models, possibly because this kind of data is related to diabetes through lifestyle. Overall, however, it seems sensible to include this information. Saydah and Lochner²⁶ have shown that having less than a high school education was related to a twofold higher mortality from diabetes, after controlling for gender, age, ethnicity, marital status, and BMI.

The performance of the established model by Bang et al.²⁸ in the present cohort was different from that originally reported when we examined it on our cohort. The cutoff point was ≥4 (risk score), and sensitivity and specificity were reported between 91% to 97% and 38% to 51%. Our validation yielded sensitivity and specificity levels of 82% and 56%, respectively. The major difference between the studies was that we used a diagnostic criteria based on fasting plasma glucose and OGTT, whereas the Bang model used only fasting plasma glucose, possibly because OGTT was only available in one of the two validation datasets. From the perspective of the ROC, the model by Baan et al.²⁹ performed equal to that of Bang et al.²⁸ for specificities above 80%. Below this threshold, however, the performance of the model of Baan et al.²⁹ model is inferior. This model was originally developed and validated in a cohort 55 years of age and older, which might explain the poor performance of the algorithm on a cohort that used an entire adult population.

These findings highlight the difficulties of comparing results across different population samples, even when identical underlying sample techniques are used. Models generally perform best on the data for which they have been developed, and they perform better on a training dataset than on a validation dataset.¹⁸ We used the accepted cross-validation approach, which is basically an average of the performance of multiple models.^17,18 Our main concern when using this approach is the representativeness of the participants included in the study. NHANES uses complex, multistage probability sampling of the civilian, noninstitutionalized population to create a realistic and representative sample of the U.S. population. Participants included in this study had an undiagnosed diabetes prevalence of 8.8%, which is significantly higher than the estimates from the International Diabetes Federation, which cites 2.6–3.4% of the U.S. population.³³ Although our cohort is believed to be representative of the U.S. population, it should be emphasized that elderly persons and some racial and ethnic minorities, with higher prevalence of diabetes, have been oversampled in the NHANES. However, the comparison of our new model with the established models tested under similar conditions is valid, without a potential bias from the sampling. Several other studies have used (earlier) versions of the NHANES for the purpose of developing a diabetes screening model.^28,32

Our end point was undiagnosed diabetes by definition, although incidence prediction of diabetes and people with impaired glucose tolerance, also known as prediabetes, was beyond the scope of this study. In many respects, however, these issues are similar to those we have investigated. Hence, we believe our approach could add to these models as well.³⁴ For a model to have generalizability over time and to other populations, the model needs to be validated precisely with respect to these factors.¹⁸ Further validation of our screening approach with other samples and for other uses is therefore important.

When developing models to predict undiagnosed diabetes, it is necessary to maintain a balance between including too many model variables and too few.³⁵ Automated variable selection methods have been used in most studies involving prediction of diabetes. Collins et al.,¹¹ in a systematic review of diabetes prediction, showed that 56% of the studies used automated variable selection to develop their model. These methods are suitable for prespecified hypothesis testing, but in a predictive/classification approach, where performance is the goal, these techniques may not always yield optimal performance and in some cases may also create unstable models.³⁶

The artificial simplicity presented in most models developed over the last decade is based on the belief that simplicity is the key in implementing a model successfully and in ensuring that it is accepted widely. Nevertheless, almost all models could easily be implemented in a standard Web page and made available for clinical use or to the public and still maintain the complexity without oversimplification. Our findings, based on a relatively large dataset of 5,398 participants encompassing 14 variables, showed that it is feasible to do a full investigation of the optimal subset of predictors to be included in the model. It should be noted, however, that in more complex datasets focusing on classification, elimination methods from machine learning can be used with success in a medical context.³⁷ In brief, machine learning concerns the development and study of systems that can learn from data.

We have shown how simple healthcare and economic data (such as ratio of family income to poverty) can help determine who is most at risk for undiagnosed diabetes. We have also shown that a calculation of all possible predictor combinations is feasible and can improve the performance of the predictive model.

Footnotes

Author Disclosure Statement

No competing financial interests exist.

References

International Diabetes Federation: Diabetes Atlas, 4 ^th ed. Brussels: International Diabetes Federation, 2009.

Noble

, Mathur

, Dent

, Meads

, Greenhalgh

: Risk models and scores for type 2 diabetes: systematic review. BMJ, 2011; 343:1–31.

Joshi

, Saboo

, Vadivale

, Dani

, Mithal

, Kaul

, Badgandi

, Iyengar

, Viswanathan

, Sivakadaksham

, Chattopadhyaya

, Biswas

, Jindal

, Khan

, Sethi

, Rao

, Dalal

: Prevalence of diagnosed and undiagnosed diabetes and hypertension in India—results from the Screening India's Twin Epidemic (SITE) study. Diabetes Technol Ther, 2012; 14:8–15.

Division of Diabetes Translation, National Center for Chronic Disease Prevention and Health Promotion: National diabetes fact sheet. Atlanta: Division of Diabetes Translation, National Center for Chronic Disease Prevention and Health Promotion, 2011.

UK Prospective Diabetes Study 6. Complications in newly diagnosed type 2 diabetic patients and their association with different clinical and biochemical risk factors. Diabetes Res, 1990; 13:1–11.

Tight blood pressure control and risk of macrovascular and microvascular complications in type 2 diabetes: UKPDS 38. UK Prospective Diabetes Study Group. BMJ, 1998; 317:703–713. Erratum in BMJ 1999;318:29.

Lifetime benefits and costs of intensive therapy as practiced in the Diabetes Control and Complications Trial. The Diabetes Control and Complications Trial Research Group. JAMA, 1996; 276:1409–1415. Erratum in JAMA 1997;278:25.

Gupta

, Misra

: Screening for and prevention of diabetes in India: need for simple and innovative strategies. Diabetes Technol Ther, 2012; 14:651–653.

Simmons

, Echouffo-Tcheugui

, Griffin

: Screening for type 2 diabetes: an update of the evidence. Diabetes Obes Metab, 2010; 12:838–844.

10.

Gillies

, Lambert

, Abrams

, Sutton

, Cooper

, Hsu

, Davies

, Khunti

: Different strategies for screening and prevention of type 2 diabetes in adults: cost effectiveness analysis. BMJ, 2008; 336:1180–1185.

11.

Collins

, Mallett

, Omar

, Yu

L-M

: Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med, 2011; 9:103.

12.

Liu

, Pan

, Jin

: A Chinese diabetes risk score for screening of undiagnosed diabetes and abnormal glucose tolerance. Diabetes Technol Ther, 2011; 13:501–507.

13.

Lee

Y-H

, Bang

, Kim

, Park

, Kim

: A simple screening score for diabetes for the Korean population: development, validation, and comparison with other scores. Diabetes Care, 2012; 35:1723–1730.

14.

Royston

, Altman

, Sauerbrei

: Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med, 2006; 25:127–141.

15.

Sun

, Shook

, Kay

: Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol, 1996; 49:907–916.

16.

Centers for Disease Control and Prevention; National Center for Health Statistics: National Health and Nutrition Examination Survey data. Hyattsville, MD: U.S. Department of Health and Human Services, 2005–2010.

17.

Schumacher

, Holländer

, Sauerbrei

: Resampling and cross-validation techniques: a tool to reduce bias caused by model building?. Stat Med, 1997; 16:2813–2827.

18.

König

, Malley

, Weimar

, Diener

H-C

, Ziegler

: Practical experiences on the necessity of external validation. Stat Med, 2007; 26:5499–5511.

19.

American Diabetes Association: Diagnosis and classification of diabetes mellitus. Diabetes Care, 2012; 35(Suppl 1):S64–S71.

20.

Sullivan

, Morrato

: Obesity, inactivity, and the prevalence of diabetes and diabetes-related cardiovascular comorbidities in the US, 2000–2002. Diabetes Care, 2005; 28:1599–1603.

21.

Wei

, Gaskill

, Haffner

, Stern

: Waist circumference as the best predictor of noninsulin dependent diabetes mellitus (NIDDM) compared to body mass index, waist/hip ratio and other anthropometric measurements in Mexican Americans—a 7-year prospective study. Obes Res, 1997; 5:16–23.

22.

Grarup

, Sparsø

, Hansen

: Physiologic characterization of type 2 diabetes-related loci. Curr Diabetes Rep, 2010; 10:485–497.

23.

Cichosz

, Fleischer

, Hoeyem

, Laugesen

, Poulsen

, Christiansen

, Ejskjaer

, Hansen

: Objective measurements of activity patterns in people with newly diagnosed Type 2 diabetes demonstrate a sedentary lifestyle. Diabet Med, 2013; 30:1063–1066.

24.

Sandberg

, Sundberg

, Wikblad

: A controlled study of oral self-care and self-perceived oral health in type 2 diabetic patients. Acta Odontol Scand, 2001; 59:28–33.

25.

Lamster

, Lalla

, Borgnakke

, Taylor

: The relationship between oral health and diabetes mellitus. J Am Dent Assoc, 2008; 139(Suppl 5):19–24.

26.

Saydah

, Lochner

: Socioeconomic status and risk of diabetes-related mortality in the U.S. Public Health Rep, 2010; 125:377–388.

27.

Clark

, Altman

: Developing a prognostic model in the presence of missing data: an ovarian cancer case study. J Clin Epidemiol, 2003; 56:28–37.

28.

Bang

, Edwards

, Bomback

, Ballantyne

, Brillon

, Callahan

, Teutsch

, Mushlin

, Kern

: Development and validation of a patient self-assessment score for diabetes risk. Ann Intern Med, 2009; 151:775–783.

29.

Baan

, Ruige

, Stolk

: Performance of a predictive model to identify undiagnosed diabetes in a health care setting. Diabetes Care, 1999; 22:213–219.

30.

Schmidt

, Duncan

, Bang

, Pankow

, Ballantyne

, Golden

, Folsom

, Chambless LE; Atherosclerosis Risk in Communities Investigators: Identifying individuals at high risk for diabetes: the Atherosclerosis Risk in Communities Study. Diabetes Care, 2005; 28:2013–2018.

31.

Glümer

, Carstensen

, Sandbaek

, Lauritzen

, Jørgensen

, Borch-Johnsen

; Inter99 Study: A Danish diabetes risk score for targeted screening: the Inter99 study. Diabetes Care, 2004; 27:727–733.

32.

Borrell

, Kunzel

, Lamster

, Lalla

: Diabetes in the dental office: using NHANES III to estimate the probability of undiagnosed disease. J Periodont Res, 2007; 42:559–565.

33.

International Diabetes Federation: Undiagnosed diabetes. 2011. www.idf.org/diabetesatlas/5e/undiagnosed-diabetes (accessed February 6, 2013 ).

34.

Cowie

, Rust

, Byrd-Holt

, Gregg

, Ford

, Geiss

, Bainbridge

, Fradkin

: Prevalence of diabetes and high risk for diabetes using A1C criteria in the U.S. population in 1988–2006. Diabetes Care, 2010; 33:562–568.

35.

Austin

, Tu

: Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol, 2004; 57:1138–1146.

36.

Steyerberg

, Eijkemans

, Harrell

, Habbema

: Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med, 2000; 19:1059–1079.

37.

Jensen

, Cichosz

, Dinesen

, Hejlesen

: Moving prediction of exacerbation in chronic obstructive pulmonary disease for patients in telecare. J Telemed Telecare, 2012; 18:1–5.