Effectiveness of an Automated Error Checking and Feedback System to Improve Text Message Reporting for Disease Surveillance in Viet Nam

Abstract

Purpose:

Viet Nam currently relies on a manual paper-based system to track and monitor 28 major infectious diseases. This inefficient system takes 2 or more months to complete.

Method:

We designed and pilot tested the use of text messaging to report certain infectious disease symptoms in rural areas of northern Viet Nam. The project was divided into three 6-month phases carried out in two provinces. The current analysis focuses on the implementation of a two-way feedback system between Phases II and III, which aims at (1) evaluating whether this system improves efficiency by determining the number of correctly (vs. incorrectly) formatted text messages; (2) assessing this system's influence on accuracy by comparing text messages with their respective official paper-based documented forms; and (3) determining whether the amount of information required to report through text messages influences the efficiency and accuracy of the text messages.

Results:

Between Phases II and III, results revealed a significant improvement in correctly formatted texts in comparison to incorrectly formatted texts. As the number of fields required to report increased, the number of correctly formatted texts (efficiency) as well as the number of matched text messages (accuracy) decreased.

Conclusion:

Our research demonstrates that an automated error bidirectional feedback system can significantly improve both the efficiency and accuracy of a Short Message Service-based method for disease surveillance. Also, our data may suggest that two-way communication has better engaged health care staff to follow reporting protocols as well as to maintain accuracy from their clinic's own data.

Introduction

Currently, infectious disease data in Viet Nam are collected by a paper-based system being transmitted through four government tiers. Originating at a commune health center (CHC), targeted disease data are aggregated from the patient logbook.¹ After aggregation from the commune level, information is sent to the District Health Center (DHC) followed by to the Provincial Health Center (PHC), and ultimately arriving at the central level for analysis by the Ministry of Health (MOH).

This system takes 2 months or longer to produce a final report to the MOH. This extensive period is due to the inefficient method of paper reporting and the requirement that each commune to report on a total of 28 infectious diseases.² The current long-winded process of infectious disease reporting is potentially catastrophic in emergency situations, such as an outbreak of a serious infectious disease (e.g., coronavirus) or a bioterrorist attack. A more efficient system of syndromic surveillance might enhance the monitoring of certain chronic diseases.

Use of Mobile Technology for Infectious Disease Surveillance

Recently, mobile technology has increasingly becoming a common tool for global infectious disease and syndromic surveillance, especially in developing countries.³ For example, Madagascar administered a national surveillance system for the detection of influenza-like illness through the use of mobile phones. Despite the country's limited resources, clinicians are willingly adopting Short Message Service (SMS) reporting of infectious disease symptoms because of its practical and economic feasibility.⁴ Given this context, similar efforts could be employed for disease surveillance by mobile technology in Viet Nam.^5,6 In addition to the high rate of mobile phone subscription (131 subscriptions per 100 people) in Viet Nam,⁷ other factors that might contribute to the potential value of utilizing mobile phones in disease surveillance, including the relatively inexpensive costs of SMS and many community clinics, do not have access to computers or the Internet.

The Remote Interaction, Consultation and Epidemiology (RICE2) Project

In 2012, the Institute of Population, Health and Development (PHAD), in collaboration with the National Institute of Hygiene and Epidemiology (NIHE), Dartmouth College, University of California at Los Angeles, and Columbia University, designed and piloted the use of text messaging to report certain infectious disease symptoms in rural areas of northern Viet Nam. The project was divided into three 6-month phases carried out in two provinces. Phase I and II piloted text message reporting to determine the feasibility of this system.^5,7 Phase III improved the pilot reporting system by implementing a two-way automated format error checking communication system between project staff and CHC workers.

The current analysis focuses on the implementation of a two-way feedback system between Phases II and III, which aims at (1) evaluating whether this system improves efficiency by determining the number of correctly (vs. incorrectly) formatted text messages; (2) assessing this system's influence on accuracy by comparing text messages to their respective official paper-based documented forms; and (3) determining whether the amount of information required to report through text messages influences the efficiency and accuracy of the text messages. Further, these specific aims may suggest that a two-way communication system can improve behaviors, such as engagement and competency, of the health staff.

Methods

Study Settings

Two provinces (Hung Yen and Hoa Binh) in northern Viet Nam were chosen to participate in the study. The current analysis used data from Phase II and III. Phase II occurred from June to December 2013, whereas Phase III occurred from March to August 2016. Both phases were carried out in a total of 40 communes, located as follows:

Seventeen communes in Yen My district, Hung Yen province (delta area)

Thirteen communes in Cao Phong district, Hoa Binh province (mountainous area)

Ten communes in Ky Son district, Hoa Binh province (mountainous area)

In both phases, two CHC staff from each of the participating communes, selected and appointed by the CHC's head, participated in the study.

Study Design

For this project, a central data repository server was monitored and managed by NIHE and the RICE2 research team. The server was equipped with an SMS gateway to collect text messages through a designated cell phone number. During each phase, CHC staff were asked to perform two tasks: (1) a regular record log into a patient log book and (2) an extra text reporting to our data repository server.

Patients diagnosed with two common symptoms in northern Viet Nam, diarrhea and influenza-like illness, were chosen to be reported. In the case where patients demonstrated both symptoms, CHC staff reported the patient's chief complaint. Please see “Structure of SMS report” for further details on reporting.

Differences Between Phase II and Phase III

During Phase II, CHC staff would send a message to the server to be analyzed by project staff. Regardless of whether the message was sent in the correct format, the message would be collected without any feedback on the message format.

In Phase III, a two-way feedback communication system with an automated error screening component for SMS reports were implemented. If the system detected an incorrectly formatted incoming message, a feedback message would be sent to the CHC staff detailing the errors, for example which fields in the structure of SMS report were wrong. CHC staff were allowed up to four times to send an incorrectly formatted SMS. After the fourth time, a warning message would be sent to the CHS to ask them to call the project staff for support, and another message would be sent to the project staff to report the situation. In the other hand, if CHC staff sent a correctly formatted message, the server would send a confirmation message.

Structure of SMS Report

The SMS structure contained a certain number of encoded disease information separated by a space character (Fig. 1). During Phases II and III, three different districts (two in Hoa Binh province and one in Hung Yen province) were randomly assigned to report a structure of differing composition: 3, 6, or 11 fields.

Fig. 1.

Example of correctly formatted SMS report for districts required to report six fields. SMS, Short Message Service. Color images are available online.

Three fields: commune number (assigned by research team), disease diagnosis, date of diagnosis.

Six fields: commune number, disease diagnosis, patient's age, patient's gender, date of diagnosis, ethnicity.

Eleven fields: commune ID, disease diagnosis, patient's age, patient's gender, date of diagnosis, village, patient's occupation, patient's ethnic group, disease symptoms, provided treatment, name of physician.

Training and Monitoring

Training

Face-to-face training classes, focused on sending SMS techniques and possible common mistakes, were provided to all participating CHC staff before implementation. Two participating CHC staff appointed by the respective CHC's head attended the training. Handouts and leaflets about the project were also provided to the CHC staff at the training.

Monitoring

For monitoring purposes, data were also made available live on a data management website built on the server. The website was securely accessible through the Internet, and only authorized personnel were able to access these data.

To minimize mistyping, typos, or duplicate SMS reports, data were monitored closely through the data management website by a project officer. Daily SMS reports were randomly picked up and checked. Any mistyping or doubtful report was reported by phone to the corresponding CHC staff for double checking, fixing, and resending (if necessary).

Onsite monitoring trips were also organized every 3 months during the implementation of the project to randomly chosen CHCs in each of the three districts. The research team checked with the CHC staff to ensure the project's protocol was followed. Project staff also collected data from a logbook, which is a paper book, that is, an official record for patient visits.

Data entry was conducted by using EpiData software⁸, and data analysis was performed by using Stata (Stata 12.0, StataCorp LP). Quantitative descriptive analysis was conducted by using odds ratios and logistic regression. p-value of <0.05 was considered statistically significant.

Two logistic regression models were created:

Comparing the percentage of wrong vs. correctly formatted texts by phase, geography, and number of fields required to report. Each message sent was checked twice by project staff to determine whether it was correctly formatted.

The CHCs from each district were randomly selected to use their logbook data. Correctly formatted messages were compared with their respective patient data documented in the CHC's logbook to determine the accuracy rate of reported text messages.

Ethical Consideration

The activities in this project, which received MOH approval before implementation, were considered to be part of the daily duty of participating CHC staff. The project was designed to collect a patient's general information only for statistical purposes through SMS reports. Access to the repository server and the SMS reports/databases are only permitted for authorized personnel.

Results

Comparing Wrong Versus Correctly Formatted Texts

In Phase II (2013), 36% of messages were wrongly formatted whereas 64% of messages were correctly formatted. In Phase III (2016), 11% of messages were incorrectly formatted whereas 89% of messages were correctly formatted.

Each district was randomly assigned a certain number of fields to report (e.g., age, sex, ethnicity). The percentage of incorrectly formatted text messages increased as the number of fields increased: 3 fields (21%), 6 fields (20%), and 11 fields (48%). The percentage of correctly formatted text messages decreased as the number of fields increased: 3 fields (79%), 6 fields (80%), and 11 fields (52%) (Table 1).

Table 1.

Logistic Regression Model: Right/Wrong Formatted Text by Phase, No. of Fields, District, and Province

PREDICTOR VARIABLES	WRONG	RIGHT	TOTAL	95% CI	P
Phase
2013	896 (36%)	1,572 (64%)	2,468
2016	217 (11%)	1,760 (89%)	1,977	9.85 (7.83–12.38)	<0.001
No. of fields^a
3	400 (21%)	1,482 (79%)	1,882
6	360 (21%)	1,461 (79%)	1,821
11	353 (48%)	389 (52%)	742	0.35 (0.29–0.43)	<0.001
District
Ky Son (ref)	265 (49%)	281 (51%)	546
Cao Phong	370 (37%)	628 (63%)	998
Yen My	478 (16%)	2,423 (84%)	2,901	2.04 (1.51–2.75)	<0.001
Province^b
Hoa Binh (ref) Mountainous Region	635 (41%)	909 (59%)	1,544
Hung Yen Delta Region	478 (16%)	2,423 (84%)	2,901	0.57 (0.37–0.88)	<0.05

Percentage based on total. Right/Wrong coded as 0 = wrong, 1 = right.

No. of fields = number of fields required to report, coded as 0 = 3 fields, 1 = 6 fields, 2 = 11 fields.

Hoa Binh province includes Ky Son/Cao Phong districts. Hung Yen province includes Yen My district.

Districts and provinces were also analyzed for their ability to adhere to correct reporting protocols. Yen My district (part of Hung Yen province) had the highest rate of correctly formatted messages (84%) and the lowest rate of incorrectly formatted messages (16%). Of the messages sent by the Cao Phong district (Hoa Binh province), 63% were correctly formatted whereas 37% were incorrectly formatted. Ky Son district (Hoa Binh province) had the lowest rate of correctly formatted messages (51%) and the highest rate of incorrectly formatted messages (49%). Of the messages reported by Hoa Binh province (mountainous region), 59% were correctly formatted whereas 41% were incorrectly formatted. Of the messages reported by Hung Yen province (delta region), 84% were correctly formatted whereas 16% were incorrectly formatted (Table 1).

A logistic regression model was used to analyze the influence of phase, number of fields reported, and geography on the rate of correctly formatted texts (Table 1). After adjusting for other covariates, the 2016 phase was 9.85 times significantly more likely to report correctly formatted messages than the 2013 phase (9.85, 95% CI: 7.83–12.38, p < 0.001). A higher number of fields required to report (3 fields vs. 11 fields) was 0.35 times significantly less likely to report correctly formatted texts than a decreased number of fields (0.35, 95% CI: 0.29–0.43, p < 0.001). There were significant differences between districts (2.04, 95% CI: 1.51–2.75, p < 0.001) and provinces (0.57, 95% CI: 0.37–0.88, p < 0.05) in reporting correctly formatted text messages.

Matching Correctly Formatted Texts to Logbook Data

Our analysis also evaluated whether correctly formatted text messages were consistent with data in the logbook (Fig. 2, Table 2). In Phase II (2013), 56% of messages were consistent with logbook data whereas 44% were not consistent. In Phase III, 73% of messages were consistent with logbook data whereas 27% were not.

Fig. 2.

Percentage of correctly formatted text messages consistent with logbook data. Color images are available online.

Table 2.

Logistic Regression Model on Correctly Formatted Messages Matched with Logbook Data by Phase, Number of Fields, District, and Province

	MATCHED WITH LOGBOOK DATA		TOTAL	95% CI	P
	NO	YES	TOTAL	95% CI	P
Phase
2 (ref)	329 (44%)	425 (56%)	754
3	237 (27%)	633 (73%)	870	6.35 (4.32–9.33)	<0.001
No. of fields^a
3 (ref)	139 (22%)	506 (78%)	645
6	222 (30%)	516 (70%)	738
11	205 (85%)	36 (15%)	241	0.16 (0.11–0.23)	<0.001
District
Ky Son (ref)	164 (61%)	103 (39%)	267
Cao Phong	97 (71%)	40 (29%)	137
Yen My	305 (25%)	915 (75%)	1,220	1.68 (0.84–3.36)	0.15
Province^b
Hoa Binh (ref) Mountainous Region	261 (65%)	143 (35%)	404
Hung Yen Delta Region	305 (25%)	915 (75%)	1,220	0.69 (0.23–2.12)	0.52

Percentages are row percentages based on Total. Coded 0 = not matched, 1 = matched.

No. of fields = number of fields required to report, coded as 0 = 3 fields, 1 = 6 fields, 2 = 11 fields.

Hoa Binh province includes Ky Son/Cao Phong districts. Hung Yen province includes Yen My district.

Analysis was also conducted on the number of fields required to report. The percentage of unmatched text messages increased as the number of fields increased: 3 fields (22%), 6 fields (30%), and 11 fields (85%). The percentage of matched text messages decreased as the number of fields increased: 3 fields (78%), 6 fields (70%), and 11 fields (15%).

We also evaluated geography. Yen My district (part of Hung Yen province) had the highest rate of matched messages (75%) and the lowest rate of unmatched messages (25%). Of the messages sent by the Cao Phong district (Hoa Binh province), 29% were correctly matched whereas 71% were unmatched. Of the messages sent by the Ky Son district (Hoa Binh province), 75% were correctly matched whereas 25% were incorrectly matched. Of the messages reported by the Hoa Binh province (mountainous region), 35% were matched whereas 65% were unmatched. Of the messages reported by the Hung Yen province (delta region), 75% were matched whereas 25% were unmatched.

A logistic regression model was used to analyze the influence of phase, number of fields reported, and geography on the rate of correctly formatted texts that matched with logbook data (Table 2). After adjusting for other covariates, Phase III (2016) was 6.35 times significantly more likely to have messages consistent with logbook data than the 2013 phase (6.35, 95% CI: 4.32–9.33, p < 0.001). After adjusting for other covariates, a higher number of fields required to report (3 fields vs. 11 fields) was 0.16 times significantly less likely to have text messages consistent with logbook data (0.16, 95% CI: 0.11–0.23, p < 0.001). After adjusting for other variables, there were no significant differences between districts (1.16, p = 0.15) and provinces (0.69, p = 0.52) in reporting correctly formatted text messages.

Discussion

Key Findings

The implementation of an automated error checking and a two-way communication system reveals several key findings. First, implementing a bidirectional system improves the efficiency of text messages. Between Phases II and III, results revealed a significant improvement in correctly formatted texts in comparison to incorrectly formatted texts. Second, there were improvements in accuracy of the text messages, as there was a significant increase in the amount of correctly formatted text messages that were consistent with the logbook data even after adjusting for notable confounders such as geography and number of fields. Third, as the number of fields required to report increased, the number of correctly formatted texts (efficiency) as well as the number of matched text messages (accuracy) decreased. This finding has major implications for future programs that intend to use SMS for disease surveillance, in that when more information is required for reporting it may negatively affect both the efficiency and accuracy of the reporting system.

Equally interesting were the geographic variations in efficiency and accuracy of text messages, and previous literature may help explain this disparity. The Yen My district had the highest percentage of properly formatted text messages as well as the highest percentage of messages matched with logbook data. On the other hand, the Ky Son district had the lowest percentage of properly formatted text messages whereas Cao Phong had the lowest percentage of messages matched with logbook data. A recent publication of the RICE2 project indicates that Yen My CHS are only of Kinh ethnicity (the major ethnic group in Viet Nam) whereas staff working at Ky Son and Cao Phong districts are mostly ethnic minorities.⁷ In fact, of the three districts, the Ky Son district consists of the largest amount of ethnic minority.⁷ Research on ethnicities in Viet Nam have documented substantial disparities between ethnic minorities and Kinh (the major ethnic group). Ethnic minorities have poorer living conditions, lower enrollment in higher education, and higher difficulty gaining access to educational opportunities and health care.^9,10 In addition, there is documented discrimination against ethnic minorities in health care work settings.^8,9 Although many factors certainly contribute to the disparity in performance between Yen My and Ky Son, ethnic composition may be an influential one. Future disease surveillance programs should take ethnicity and associated socioeconomic issues into consideration when training health staff from different geographical regions.

Previous literature and the current findings support the notion that a two-way communication system during disease surveillance can improve the quality of data. The U.S. Center for Disease Control and Prevention (CDC) implemented a feedback system for HIV surveillance in 2010, resulting in an improvement in the quality of data collected.¹¹ Our study's significant increase in the efficiency and accuracy of text messages after the implementation of a two-way feedback communication system may be due to improved behaviors, such as engagement and competency, of health care staff. Moreover, the improved results between Phases II and III illustrate the need for continuous education and feedback to CHC staff in the adoption of mobile technology. A two-way feedback system provides an efficient and effective response to those needs.

Challenges and Opportunities

Findings from this project offers both challenges and opportunities. This article demonstrates that an automated error bidirectional feedback system can significantly improve both the efficiency and accuracy of an SMS-based method for disease surveillance. Also, our data suggest that a two-way communication has better prepared health care staff to follow reporting protocols as well as to maintain accuracy by using their clinic's own data. Further, requiring staff to report more fields in a text message would negatively affect their ability to send a properly formatted message as well as to maintain accuracy.

Our project adds to existing literature by illustrating the strong potential for the adoption of text messaging for infectious disease surveillance in Viet Nam, and it expands on mobile technology disease surveillance through the implementation and evaluation of a two-way communication system. The current paper-based system in Viet Nam is an inefficient method that could be potentially catastrophic, given the 2-month timeline required for the MOH to compile the data. In events such as an infectious disease outbreak, it is critical to efficiently collect large amounts of data with a high degree of accuracy. As mobile technology has become more readily available in Viet Nam,¹² our mobile-based disease surveillance with a bidirectional communication system could prevent such a catastrophic disaster because of its practicality, efficiency, and accuracy. Further, the progression of our project illustrated that health care staff need educational support and feedback to produce increased levels of accuracy and efficiency during mobile reporting. As such, our project demonstrates that mobile disease surveillance is improved when there is a two-way communication system between administration and health care staff.

As the current paper-based system is still the predominating system for reporting infectious disease systems in Viet Nam, the RICE2 project team is motivated to evolve beyond two provinces to provide a detailed recommendation to the Vietnamese government for the adoption of mobile-based disease surveillance.

Footnotes

Acknowledgments

The authors would like to sincerely thank the Hung Yen and Hoa Binh Centers for Disease Control and Prevention for their support for the work described in this article.

Disclosure Statement

No competing financial interests exist.

Funding Information

The work described in this article received support from the Institute of Population, Health, and Development (PHAD). The 2 trials in Hung Yen and Hoa Binh province were supported by the University of California, Los Angeles, The Center of Medical Multimedia Education and Technology, Dartmouth College, and PHAD.

References

Ministry of Health, Viet Nam, Guiding functions and roles of community health center, Circular No. 33/2015/TT-BYT. 2015.

Ministry of Health, Viet Nam, Guiding the informing and reporting regime, declaration of infectious diseases, Circular No. 48/2010/TT-BYT. 2010.

World Health Organization. mHealth: New Horizons for health through mobile technologies: Based on the findings of the Second Global Survey on eHealth (Global Observatory for eHealth Series, Volume 3). Healthc Inform Res, 2012; 18:231–233.

Rajatonirina

, Heraud

, Randrianasolo

, Orelle

, Razanajatovo

, Raoelina

, et al. Short message service sentinel surveillance of influenza-like illness in Madagascar, 2008–2012. Bull World Health Organ, 2012; 90:385–389.

Katona

, Rosen

, Vu

, Nguyen

, Dang

, Thiem

, Nguyen

, Ratner

, Gan

, Katona

. A new paradigm for disease surveillance in Vietnam. Telemed J E Health, 2014; 20:493–495.

Lam

, Dang

, Phan

, Trinh

, Vu

, Nguyen

. Mobile health initiatives in Vietnam: Scoping study. JMIR Mhealth Uhealth, 2018; 6:e106.

Dang

, Vu

, James

, Katona

, et al. Perceptions of the feasibility and practicalities of text messaging-based infectious disease surveillance: A questionnaire survey. JMIR Mhealth Uhealth, 2016; 4:e65.

Lauritsen

, Bruss

EpiData (version 3.1). A comprehensive tool for validated entry and documentation of data 2008. Avaiable at www.epidata.dk (last accessed on July 9, 2020 ).

Malqvist

, Hoa

, Liem

, Thorson

, Thomsen

. Ethnic minority health in Vietnam: A review exposing horizontal inequity. Glob Health Action, 2013; 6:1–19.

10.

Dang

, Hai-Anh. 2010. Growth in Vietnam is strong but not shared equitably across ethnic. Indigenous Peoples country brief; no. 1. Washington, DC: World Bank. Available at http://documents.worldbank.org/curated/en/572571468133547315/Growth-in-Vietnam-is-strong-but-not-shared-equitably-across-ethnic (last accessed October 20, 2018 ).

11.

Beltrami

, Wang

, Usman

, Lin

. Quality of HIV testing data before and after the implementation of a national data quality assessment and feedback system. J Public Health Manag Pract, 2017; 23:269–275.

12.

The World Bank. Mobile cellular subscriptions. Available at http://data.worldbank.org/indicator/IT.CEL.SETS.P2 (last accessed October 20, 2018 ).