Abstract
Introduction
Dengue is a disease with a wide clinical spectrum. The early identification of dengue cases is crucial but challenging for health professionals; therefore, it is necessary to have effective diagnostic instruments to initiate timely care.
Objective
To evaluate the effectiveness of an algorithm based on an artificial neural network (ANN) to diagnose dengue in an endemic area.
Methods
A single-center case–control study was conducted in a secondary-care hospital in Ciudad Obregón, Sonora. An algorithm was built with the official operational definitions, which was called the “direct algorithm,” and for the ANN algorithm, the brain.js library was used. The data analysis was performed with the diagnostic tests of sensitivity, specificity, positive predictive value (ppv), and negative predictive value (npv), with 95% confidence intervals and Cohen's kappa index.
Results
A total of 233 cases and 233 controls from 2022 were included. The ANN presented a sensitivity of 0.90 (95% CI [0.85, 0.94]), specificity of 0.82 (95% CI [0.77, 0.87]), npv of 0.91 (95% CI [0.87, 0.94]) and ppv of 0.81 (95% CI [0.76, 0.85]) and a kappa of 0.72. The direct algorithm had a sensitivity of 0.97 (95% CI [0.94, 0.99]), specificity of 0.96 (95% CI [0.92, 0.98]), npv 0.97 (95% CI [0.94, 0.98]), ppv 0.96 (95% CI [0.93, 0.98]) and kappa 0.93.
Conclusions
The direct algorithm performed better than the ANN in the diagnosis of dengue.
Introduction
The World Health Organization (WHO) defines dengue as a “viral infection transmitted by mosquitoes that in recent years has spread rapidly to all regions. Dengue virus is transmitted through female mosquitoes mainly of the Aedes aegypti species and, to a lesser extent, of the Ae. Albopictus species.” 1
Dengue currently represents one of the greatest public health problems in the world. There are between 50 and 100 million infections per year and thousands of deaths in more than 100 countries, with approximately 2.5 billion people at risk (approximately 40% of the world's population). 2
The Official Mexican Standard NOM-032-SSA2-2014 for the epidemiological surveillance, promotion, prevention, and control of diseases transmitted by vectors establishes one of the main public health concerns in the Mexican demarcation. The country's climatic and geographical characteristics as well as socioeconomic and demographic conditions benefit the risk of transmission of all of these diseases in each federal entity. 3
Dengue is described as a disease with a clinical spectrum that ranges from asymptomatic forms to forms that cause serious illness and death. Despite the fact that death from dengue is preventable in approximately 99% of cases, 20,000 deaths occur annually in more than 100 countries. 4
To achieve effective and efficient clinical identification and classification of dengue cases, operational case definitions have been made to unify criteria for epidemiological surveillance of the disease in the National Health System. These same operational definitions are distinguished by having high sensitivity, which means that they allow us to locate most of the cases by means of the most common signs and symptoms of the disease and laboratory tests. 2
Dengue diagnosis is important in all regions where the disease is endemic; however, the clinical forms of the disease and the presence of other epidemics with a similar clinical picture (leptospirosis, malaria, chikungunya, and Zika) in the same geographic regions make it difficult to diagnose it correctly. 5
The classification conceived by the WHO in 2009 has a high sensitivity (87% to 95%) but a low specificity (6% to 20%); however, it should be noted that it was planned to quantify the clinical severity of dengue in people who met with the operational definition and not for the purposes of differential diagnostic accuracy. 6
Among the laboratory tests that detect dengue virus infection, identification of IgM and the detection of DENV through molecular methods by reverse transcription polymerase chain reaction (RT-PCR test) are sensitive and specific; however, they require specialized infrastructure and competent personnel. For these reasons, the diagnosis of dengue remains predominantly clinical in professional practice.6,7
There are several studies in which clinical algorithms for the diagnosis of dengue were proposed using different methodologies, but their usefulness has been reduced due to the lack of external validation in medical assessment and practice.6,8–10
Therefore, it is necessary to refine the diagnosis of dengue using clinical data observed in health services using alternative methods for the development of algorithms. 11
The WHO classifications of 1997 and 2009 and that of the 2010 Colombian guideline were sensitive, but their specificity was very low using the “Red Aedes” (Aedes Network) database (cohort). The classification with the highest sensitivity was that of the 2010 Colombian guideline (98%), which included diarrhea among the warning signs, while the WHO proposal had a sensitivity of 96%, compared to 94% for the WHO classification of 1997. The results provided by the WHO classification of 2009 were very similar to those reported in prospective studies carried out by Gutiérrez et al. in Nicaragua 12 and by Low et al. in Singapore. 13 However, when measuring the precision of the classifications in the database of the study carried out by Osorio et al., 14 a decrease in sensitivity was found in its entirety, being higher than that of the WHO classification of 1997, while the sensitivity of the 2010 Colombian guidelines and the 2009 WHO guidelines was 81%. Likewise, in the studies carried out in Brazil 15 and De Cavalcanti et al. 16 based on medical records, the decrease in sensitivity was similar.
This study uses artificial neural networks (ANNs), which are the units most similar to biological neurons with the processor element (PE) as an artificial and simplified model of the human brain. The PE has several inputs and outputs that make up a basic sum. This sum of inputs is transformed by a transfer function, and the value of the output of this transfer function is passed directly to the output of the PE. One of the applications of these neural networks is for medical diagnoses, but there is insufficient evidence of their benefit in the diagnosis of dengue.17–20
It is important to note that dengue disease in its initial or prodromal stage is similar to other diseases, and there is constant hesitation on the part of treating physicians. Thus, conducting this study will allow us to assess the benefit of an algorithm, thus directly helping patients and operational personnel who provide medical care.
Therefore, the objective of the study was to evaluate an algorithm based on an ANN and compare it with a direct algorithm for the diagnosis of dengue based on the official operational definitions of this disease.
Methods
General description of the study
A case‒control study was conducted at Regional General Hospital 1 in Ciudad Obregón, Sonora, belonging to the Mexican Institute of Social Security.
The source data was divided into two groups: (1) data from 2022 for the design of the case‒control study and (2) records from 2006 to 2015 for the training of the ANN. In both groups, the collection period was from 1 January 2022 to 31 December 2022. The data were extracted from hospital records, clinical records, and the dengue epidemiological surveillance system.
A case was considered to be any registration with a positive result from a laboratory in the epidemiological surveillance system, based on the standardized operational definitions in the country. 5 For cases of dengue with warning signs and severe dengue, clinical laboratory results (total leukocytes, hematocrit, hemoglobin, and liver profile) were not included. The controls were patients with a different diagnosis of dengue who went to the hospital for medical attention and were registered in the emergency censuses. Patients who did not have a complete medical record were excluded.
The information was compiled in a format specially designed to collect the variables of clinical manifestations and epidemiological antecedents. The data were taken from the first emergency admission note.
Information gathering
The study format included the clinical and epidemiological characteristics that have been identified in dengue cases, in accordance with current national regulations. 5 There were 46 items that were answered dichotomously, as absent or present. The instrument responded to the objectives set out in the protocol and did not include mandatory clauses. The researchers were only involved in the realization of the database, but there was no direct contact with the patients, so informed consent was not needed.
Construction of the algorithm
The neural network model was based on what was published by Ho T-S et al., 21 which consisted of exploiting the key variables identified to predict laboratory-confirmed dengue cases in patients with probable dengue symptoms.
Direct diagnostic algorithm
The direct diagnostic algorithm was based on the frequency of clinical manifestations, thus establishing operational definitions in Mexico. First, simple frequencies of the data were obtained; rare or null values were not included.
Major clinical-epidemiological data were as follows: The signs and symptoms that reached the highest percentage or without which a probable diagnosis was not integrated were considered.
Minor clinical-epidemiological data were defined as follows: These data represented information that was presented less frequently.
Nonsevere dengue model.
Major data included a temperature greater than 38 °C, headache, arthralgia, pain in the whole body, retroorbital pain, place of residence, place visited, and contact with mosquitoes.
Minor data included fever of less than five days, cervical lymphadenopathy, nausea, vomiting, rash, and petechiae.
S1 = Sign or symptom needed.
S2 = Major data.
Fr = Complementary risk factor.
For the specific case of nonsevere dengue, the following were included:
S1 = Temperature greater than 38 °C.
S2 = Two or more of the following clinical findings: headache, arthralgia, pain throughout the body, retroorbital pain, and rash.
Fr = Place of residence, place visited, or contact with mosquitoes.
To perform the calculation, the following formula was used:
Probable diagnosis (DX) = S1 + S2 + Fr
where S1, S2, and Fr took the values of 1 and 0 depending on the presence or absence of these symptoms, and when DX was equal to 3, the diagnosis of nonsevere dengue was fulfilled.
In the event that the major data was not complete or as a complement to the diagnosis, the proportion of clinical epidemiological data was carried out.
SD = Clinical epidemiological data found
DT = Total clinical epidemiological data available.
DX = SD/DT × 100
A proportion of 85% or higher was considered a probable case in the absence of the necessary clinical data or greater clinical data.
Dengue model with warning signs:
Major data included fever or hypothermia, headache, arthralgia, myalgia, retroorbital pain, rash, epigastric pain, hematological alterations, place of residence, place visited, and contact with mosquitoes.
Minor data included fever of less than five days, daily fever, cervical lymphadenopathy, nausea, vomiting, rash, petechiae, location of rash, date of onset of rash, odynophagia, splenomegaly, hepatomegaly, conjunctivitis, stupor, edema, ascites, ecchymosis, gingivorrhagia, and epistaxis.
S1 = Sign or symptom needed.
S2 = Major data
Fr = Complementary risk factor
For the specific case of dengue with warning signs, the following were included:
S1 = Two or more of the following clinical symptoms: headache, arthralgia, retroorbital pain, and macules.
S2 = At least one of the following clinical symptoms: epigastric pain, ecchymosis, gingivorrhagia, or epistaxis.
Fr = Place of residence, place visited, or contact with mosquitoes.
The above formulas apply: DX = S1 + S2 + Fr and DX = SD/DT × 100.
Model for severe dengue:
Major data included fever or hypothermia, headache, arthralgia, myalgia, retroorbital pain, rash, epigastric pain, dyspnea, fainting, tachycardia, hematological abnormalities, place of residence, place visited, and contact with mosquitoes.
Minor data included fever less than five days, daily fever, cervical lymphadenopathy, nausea, vomiting, rash, petechiae, location of rash, date of onset of rash, odynophagia, splenomegaly, hepatomegaly, conjunctivitis, stupor, edema, ascites, hemoptysis, hematemesis and melena.
S1 = Sign or symptom needed.
S2 = Major data.
Fr = Complementary risk factor.
For the specific case of severe dengue, the following were included:
S1 = Two or more of the following clinical symptoms: headache, arthralgia, retroorbital pain, and macules.
S2 = At least one of the following clinical symptoms: epigastric pain, dyspnea, fainting, tachycardia, hemoptysis, hematemesis, or melena.
Fr = Place of residence, place visited, or contact with mosquitoes.
The above formulas were applied: DX = S1 + S2 + Fr and DX = SD/DT × 100.
Calculation of the ANN
The ANN are systems based on layers of perceivers, which is the minimum functional unit of the network, analogous to a neuron, where there are two possible outputs, 0 and 1.
To carry out the training of the ANN models, records from the hospital's epidemiological surveillance system of dengue from 2006 to 2015 and were divided according to the dengue classification in the Manual of Standardized Procedures for the Epidemiological Surveillance of Diseases Transmitted by Vector (Manual de Procedimientos Estandarizados para la Vigilancia Epidemiológica de las Enfermedades Transmitidas por Vector). 5 There were 150 cases of nonsevere dengue, 148 dengue with warning signs, and 23 cases of severe dengue. For the training of the ANN, records were used that met the operational definition of a case of dengue or viral respiratory disease, whose result was positive by laboratory and that the clinical variables were at least 80% complete (Figure 1).
Flow diagram of subject selection.
For the calculation of the ANN, the javascript library was used: brain.js (https://unpkg-com-443.web.bisu.edu.cn/brain.js), which was reproduced in the Chrome search engine, Google, Inc.
A combination of the direct algorithm with the ANN was used. The ANN was used to calculate the complement of the final diagnosis of dengue with the same JavaScript library.
Finally, all the outputs will become 1 or 0, depending on whether the diagnosis was positive or not. In addition, the diagnosis percentage was established.
Statistical analysis
Diagnostic tests of sensitivity and specificity were used for qualitative variables, as well as positive predictive value (ppv) and negative predictive value (npv). The Youden index and 95% confidence intervals were determined. Cohen's kappa index was also obtained, with the standard error and the z-test with alpha less than .05. All analyses were performed in the R program Version 4.3.0 (2023-04-21 ucrt) “Already Tomorrow” Copyright (C) 2023 The R Foundation for Statistical Computing).
Ethical considerations
This project was approved by the Local Health Research Committee No. 2601 (registration number R-2023-2601-011). It was information collected from databases and clinical records without direct patient contact, so informed consent was not required.
Results
A total of 233 cases of dengue were identified during the year 2022 in the Regional General Hospital No. 1 of Ciudad Obregón, Sonora. Women predominated in the cases (67.24% vs. 43.29% p < .001), and their average age was less than that of the controls (34 (SD 18) vs. 65 (SD 16)).
The main variables that the cases presented were fever (100%), contact with mosquitoes (97%), headache (96%), myalgia (94%), and arthralgia (90%). Clinical data considered in minor criteria were the least frequent (Table 1).
Clinical and epidemiological manifestations of dengue cases.
When calculating the probability with the direct algorithm, according to the operational definition of dengue (independent of its classification), a sensitivity of 0.97 (95% CI [0.94, 0.99]), a specificity of 0.96 (95% CI [0.92, 0.98]), an npv of 0.97 (95% CI [0.94, 0.98]) and a ppv of 0.96 (95% CI [0.93, 0.98]) were observed.
The clinical diagnosis of dengue was calculated with ANN, identifying the sensitivity, specificity, npv and ppv at cutoff points of 0.5, 0.6, 0.7, 0.8, and 0.9, respectively (Table 2). A high sensitivity of 0.99 (95% CI [0.97, 0.99]) was observed at the point of 0.5, but a low specificity of 0.53 (95% CI [0.47, 0.60]) was observed; however, at the point of 0.9, the opposite was observed, with a low sensitivity of 0.43 (95% CI [0.37, 0.50]) and a high specificity of 0.97 (95% CI [0.93, 0.99]). Subsequently, the optimal cutoff point was calculated with the receiver operating characteristic (ROC) curve, which was 0.76 for the ANN, with an area under the curve (AUC) of 0.899 (95% CI [0.87, 0.92]). With this premise, the estimators of the diagnostic tests were calculated, reporting a sensitivity of 0.90 (95% CI [0.85, 0.94]), specificity 0.82 (95% CI [0.77, 0.87]), npv 0.91 (95% CI [0.87, 0.94]) and ppv 0.81 (95% CI [0.76, 0.85]). Finally, a combination was made between the direct algorithm and the ANN with the cutoff point of the ROC, which resulted in a sensitivity of 0.89 (95% CI [0.84, 0.92]), specificity 0.99 (95% CI [0.97, 0.99]), npv 0.87 (95% CI [0.83, 0.90]) and ppv 0.99 (95% CI [0.97, 0.99]) (Figure 2).
Receiver operating characteristics curve.
Sensitivity, specificity, and predictive values by model for dengue cases.
Cohen's kappa index was calculated to obtain the congruence of the confirmatory diagnosis with the diagnostic algorithms, which reported k = 0.93, z = 53.37, p < .001 for the direct algorithm and k = 0.87, z = 37.5, p < .001 for the combination with ANN. The ANN per ROC cutoff point was k = 0.72 z = 22.31, p < .001. At the other cutoff points, there was a barely moderate kappa index (Table 2).
Cohen's kappa index was also calculated between the direct algorithm and the classification given by the physicians who cared for dengue patients with a poor concordance result. The results were as follows: Nonsevere dengue k = 0.14, z = 3.504, p < .001; dengue with warning signs k = 0.34, z = 9.605, p < .001 and nonsevere dengue k = 0.24, z = 3.907, p < .001 (Table 3).
Results of the direct algorithm by clinical classification of dengue.
Discussion
A single-center study was carried out with a social security population to evaluate the effectiveness of an algorithm based on an ANN to make a laboratory diagnosis of dengue in an endemic area of Mexico based on clinical manifestations.
In the past three decades, the diagnosis of diseases has been made with greater certainty and has not depended only on the experience of the doctor. One of the fruits of these efforts is evidence-based medicine, which has become established today. Technology has advanced in information systems, with terms such as data mining, machine learning, artificial intelligence or ANNs being in common use. In the analysis carried out, the direct algorithm is positioned with the best results versus the neural network, which can be explained by the use of standardized case definitions for the identification of cases in the epidemiological surveillance of dengue. This process revealed the usefulness of carrying out operational definitions of “case” such as those used in outbreak epidemiology. The results of sensitivity and specificity were better compared to the work carried out by Caicedo et al. in Colombia, although clinical laboratory data, such as hematic biometry, were not included. 22
The ANN performed better when the ROC cutoff point was obtained, which had an AUC = 89.9% higher than that demonstrated in the study by Ho et al. (AUC = 83.5% to 85.87%), despite studying a larger population. However, the differences are explained by the chosen population. In this study, records were classified as dengue according to the International Classification of Diseases or with a positive laboratory diagnosis in the acute or convalescent phase. 21 In this study, they were from the epidemiological surveillance system. In addition, the combination of the direct algorithm with the ANN-ROC reached a specificity of 99%. These results are probably explained by the fact that the controls in the China study were patients with a negative result for dengue, while in the present study, they had a pathology other than dengue. The same situation is found in the study by Gambhir et al., 23 given that their results for the diagnosis of dengue by means of an ANN presented 79.09% precision, 55.5% sensitivity and 88.5% specificity, although this was omitted if the optimization of hyperparameters or the selection of characteristics was used in the dataset. Given that the behavior of the ANN was evaluated, it was based on the sectoral operational definition. For this reason, the cases with a negative result were not considered in the controls.
It is necessary to consider that the ANN is more efficient because it has more training and presents more layers of perceptrons, which was not taken into account in this study because it was a basic ANN for learning purposes. An attempt has been made to approach various techniques of artificial intelligence and machine learning; however, because several variables have been included, the results have been very heterogeneous. 24 Despite this, ANNs have many advantages, including a high capacity to learn and generalize and the ability to handle imprecise, confusing, noisy, and probabilistic information. 25 The published values are heterogeneous due to the type of technology used and the selection of cases and means of confirmation. Despite this variety, it shows the possible scenarios and is able to decide the best options for learning the selected technology. For this reason, the diversity of this type of publication with its own particularities will facilitate the most appropriate methodology in the future.
In our study, Cohen's kappa index was used, and the best congruence was between confirmed cases and the direct algorithm. However, the congruence was very poor with the clinical classifications assigned by the physicians who attended the patients. This test has not been performed in other studies, which reveals that the operational definition of a case endorsed by the National Epidemiological Surveillance Committee is sensitive. Likewise, another strength apart from having a standardized criterion for the study of people with suspected dengue is that the cases were included directly from the epidemiological surveillance system after validation of the operational personnel responsible for epidemiological surveillance. Although there are other algorithms that have been used in medical diagnosis with a good level of accuracy when compared with traditional machine learning and volumetric techniques 26 ; all of them have focused on images (radiological 27 or histopathological 28 ); based on Convolutional Neural Network 26 or dual-path network. 28 In the present study, data from various sources of information (clinical records and epidemiological surveillance systems) were used, which is why an ANN was used in the study. The epidemiological arbovirosis surveillance (included dengue, zika, and chikungunya) in Mexico is based on the reporting of probable and/or laboratory-confirmed cases to the National Epidemiological Surveillance System, with a subsample of 30% sent for laboratory confirmation, 29 that although it operates throughout the country in a homogeneous manner, in the present study it was contemplated in a medical unit; Therefore, the limitations of the study were (1) the number of records involved from a secondary-care hospital, (2) not including or stratifying by adjacent diseases, (3) the entire year was included without delineating the epidemic periods, (4) dengue fever was not differentiated by severity and (5) participant were not matched by sex.
Despite being a single-center study, the included cases did not present selection bias due to clinical severity. Because half of the cases were non-severe dengue and the rest of the classifications of the disease were included according to the WHO guidelines, the algorithm can be used in patients treated on an outpatient basis and those that are hospitalized even when there are no laboratory diagnostic aids. This is the first study carried out in the IMSS that involves the ANN for the diagnosis of dengue.
Among the new lines of research is the application of the ANN for a priori guidance of dengue virus based on clinical manifestations and viral circulation, which has been documented only in Ecuador with good results (sensitivity of 100% and specificity of 88.2% for hospitalization). 30 The use of SVM for dengue has been documented; Although they have presented beneficial results, they have been with operational case definitions that are not compatible with our study, which has used the definitions with the classification proposed by the WHO.31,32
The present study exclusively involves clinical manifestations. It is feasible to use it in medical units with fewer resources and even with mobile devices connected to health personnel or places where there are doctors in training (interns or specialty doctors). Thus, its use contributes to the decision-efficient clinic for dengue and can improve the quality of health care, beginning with the medical process at the time of diagnosis to improve the use of resources and the patient's quality of life. However, being a technology based on the artificial reconstruction of reality from previously collected data, the criterion of clinical behavior is essential, as well as the registration of data in the information sources when performing the semiology.
In conclusion, the direct algorithm had the best performance, and the specificity was reinforced when it was combined with the cutoff point of the ROC of the ANN. However, the dengue classification did not have adequate concordance when comparing the direct algorithm with the diagnosis established by the doctor.
Footnotes
Contributorship
CARV—conceptualization and project administration; OMAM—conceptualization, data collection, and data curation; BLRR—conceptualization, data collection, and data curation; PFHB—conceptualization, formal analysis, investigation, methodology, and software; DACG—visualization, writing original draft preparation, and review and editing; AVP—validation and writing original draft preparation; LAN—validation, visualization, writing original draft preparation, and review and editing; LJB—validation and writing original draft preparation; AMP—conceptualization, investigation, supervision, and methodology; MGRM—investigation and methodology; GVA—validation and writing original draft preparation; BIVG—investigation and review and editing.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Ethical approval
Local Committee for Scientific Research of the IMSS was approved with the registration number R-2023-2601-011.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Guarantor
DACG.
