Development of a web-based epidemiological surveillance system with health system response for improving maternal and newborn health: Field-testing in Thailand

Abstract

Surveillance systems are yet to be integrated with health information systems for improving the health of pregnant mothers and their newborns, particularly in developing countries. This study aimed to develop a web-based epidemiological surveillance system for maternal and newborn health with integration of action-oriented responses and automatic data analysis with results presentations and to assess the system acceptance by nurses and doctors involved in various hospitals in southern Thailand. Freeware software and scripting languages were used. The system can be run on different platforms, and it is accessible via various electronic devices. Automatic data analysis with results presentations in the forms of graphs, tables and maps was part of the system. A multi-level security system was incorporated into the program. Most doctors and nurses involved in the study felt the system was easy to use and useful. This system can be integrated into country routine reporting system for monitoring maternal and newborn health and survival.

Keywords

electronic and mobile health electronic health records maternal and newborn health surveillance system web-based information service

Introduction

According to the World Health Organization, 15 percent of all pregnant women are at risk of developing major obstetric complications.¹ Common major obstetric complications attributed to maternal and newborn death are pregnancy-induced hypertension, particularly preeclampsia/eclampsia, antepartum/postpartum hemorrhage, obstructed labor/uterine rupture, septic abortion, preterm birth and newborn with low birth weight.² The incidence of these complications is highest during the first two days after birth when the majority of women are still in hospital.³ There are many studies highlighting the limitations of evidence-based practices and substandard care in hospital, especially in developing countries.^4–6 Moreover, major obstetric complications and their managements are often improperly and inadequately recorded. This may result in providing national health services with incorrect data, and thus less than ideal health policy planning.⁶ A well-designed surveillance system for monitoring these complications would reduce these problems.

Various considerations must be addressed in the establishment of a good surveillance system, of which we believed the following the most important: (a) integration of the surveillance system with existing health information systems, (b) establishment of high standards for data recording, (c) health-related data recorded in electronic form and (d) relevant information in the system that facilitates accurate and timely public health decision-making.⁷ The main problems of existing surveillance systems are irrelevant or inadequate quality of data collected, delayed reporting, low data utilization and a lack of feedback to health providers.⁸ A system that can be used to gather information of maternal and newborn events on a daily basis and provide evidence for immediate corrective actions may be helpful for monitoring and improving maternal and newborn health and survival. One way to address these problems would be a web-based epidemiological surveillance system with integration of action-oriented responses using evidence-based policies and decision-making processes.^9,10

Whenever health information technology is designed and implemented, the acceptance of its use and the intention to use it, which predicts actual use, should also be assessed.^11,12 Therefore, the objectives of this study were to develop a web-based epidemiological surveillance system for maternal and newborn health with integration of action-oriented responses and automatic data analysis with results presentations and to assess the system acceptance by the nurses and doctors involved. To our knowledge, this is the first time that an epidemiological surveillance system with immediate action responses has been developed for maternal and newborn health using web-based technology.

Materials and methods

The proposal for this study was approved by the Institutional Ethics Committee of the Faculty of Medicine, Prince of Songkla University. Permission to conduct the study was obtained from the directors of all study hospitals.

System design and development

Development tool

A web-based application was designed for this development tool due to ease of access and flexibility of operating systems. The system can be accessed through all web browsers that support the hypertext markup language version 5 (HTML5), including Internet Explorer, Chrome, Firefox and Safari. HTML5 allows generation of cross-platform applications. Information in the central database can be added, deleted or updated using jQuery and Hypertext Preprocessor (PHP).

Database design

Three levels of database access were executed. Level 1 is for use by the “data entry operator,” a nurse or clerk who enters the individual baseline data of pregnant women and newborns in a participating hospital. Level 2 is for the “actor,” a responsible doctor or nurse who enters important clinical details and patient management for action responses and reviews the individual data in a participating hospital. Level 3 is for the “administrator,” an upper-managerial staff or policy maker who has access to all individual data, action responses and aggregated summaries of the data from all hospitals including result presentations in terms of tables, graphs and maps.

During the trial period, the individual data of women during their pregnancy, labor and delivery, and the immediate postpartum period, as well as that of the newborn were recorded in the database on a daily basis. Data were stored into various database tables which were linked using the women’s unique identification number and hospital number. As soon as the data of individual women were entered and confirmed in a particular hospital by a nurse or clerk, all the data from that hospital were saved and the maternal and newborn complications which were assigned to be monitored as the indicators would be summarized and updated on the server for the actor to view and perform the action-oriented responses. All data were recorded in a web server at the data center, located at the Epidemiology Unit, Faculty of Medicine, Prince of Songkla University. Administrators at policy levels could view the complications and individual data of women presenting the complications with their managements in the forms of summary tables, graphs and maps by selecting various options from the menu. Remarkable incidence of complications and improper managements in any hospitals can be timely observed and discussed. The system allowed data entry from multiple users as shown in Figure 1.

Figure 1.

Multi-user system design for linking data between hospitals, the web server and the data center.

Functional design

Nurses and doctors—both general practitioners from the district hospitals or obstetricians in the tertiary hospitals—were trained on how to use the system, which consisted of data entry techniques and following up cases to check the accuracy and the functionality of the system. Data entry for individual women and newborns was the responsibility of trained nurses or health personnel in the labor and delivery rooms or emergency room in each hospital. Trained nurses acted as the data entry operator and trained doctors as the actors. During the development and testing period, T.L. and M.I. acted as the administrators to check the database and design the presentation of results.

All data were entered into the system using a smartphone, pocket PC or desktop computer. The confidentiality and security of data were protected by the use of username and password for determining the scope of functions and access to the data. Daily analysis and aggregation of data were performed using automatic data analysis techniques. To check the accuracy of data entered, the data were first reviewed before being confirmed in the system. If the data were not confirmed, a notification appeared in the database and remained until the data were confirmed. Once a record was confirmed, no further changes to the data were allowed. Complications occurring within 24 h after delivery were allowed to be added to a confirmed record.

Figure 2 shows the main screen after a successful login to the system for data entry. Five options are available, namely aggregate indicators, statistics, entry individual data, confirmed data and sign out. The aggregate indicators list 10 complications which were monitored in this testing database. The 10 indicators were severe preeclampsia/eclampsia, postpartum hemorrhage, sepsis, obstructed labor, cesarean section, maternal death, preterm birth, low birth weight, stillbirth and neonatal death. After data related to these indicators were confirmed, the system would automatically update the summary page of aggregated indicators and totals for viewing by the actor as shown in Figure 3. Cases with complications that had not been acted upon by the actor would be highlighted in red, indicating that these cases should be reviewed, evaluated, and confirmed. This process involved evidence-based healthcare and review of attributed factors and actions done.

Figure 2.

Sections in the level of data entry after login page.

Figure 3.

Sections in the level of actor after login page.

At the administrative level, the summary of data from each individual hospital could be reviewed in the form of a table, graph or map (Figure 4). The administrator could evaluate the percentage of action-oriented responses and individual data of women through the system (Figure 5). In addition, the data entered could be extracted for further analysis or printed.

Figure 4.

Summary of data at administrative level.

Figure 5.

Percentage of action-oriented responses and individual data assessment.

Data security techniques

Three levels of data security were implemented, namely client level, data transmission level and server level. Client level applied the user authentication credentials with session timeout every 10 min to protect attacks from session hijacking and escaping questionable characters in queries were used to prevent SQL injection.^13–15 Moreover, encoding HTML entities and input validation with sanitization to prevent cross-site scripting were also used.¹⁶ Data transmission level used encryption of the data between client and server through open-source software (NoSSL).¹⁷ A multi-level firewall protection system was used for server security.¹⁸

Testing system

Study design and setting

The system was tested by health personnel from eight hospitals in Songkhla province, southern Thailand. Songkhla province was selected because it contains district, general and regional hospitals and a university hospital, all providing maternity services. District hospitals are the lowest-level facilities which provide delivery services for pregnant women. In order to have sufficient cases for field-testing of the surveillance system and to facilitate timely actions, the criteria for selecting the hospitals were as follows: at least 600 deliveries per year on average for district hospitals and at least 2500 deliveries per year on average for tertiary hospitals. From the 19 public hospitals in the province, six district and two tertiary hospitals met this criteria and all were included in the study.

Study samples

One or two labor and delivery nurses and one obstetrician from each of the selected hospitals were invited to attend a one-day training workshop on 16 June 2013 aiming to orientate them on the objectives of the study and train them on how to use the system. They were informed about the study and asked to sign a consent form to participate in the study. During the workshop, the principles of the system were explained and they were given an opportunity to practice data entry with module instruction and an interactive demonstration. An evaluation of the online web-based system, including mobile device application, important variables related to pregnancy, delivery and complications, the system for checking the accuracy of the data, real-time data presentation and daily and monthly demonstration of results, was performed at the end of the workshop. More than 80 percent of the participants were satisfied with the system and provided feedback for improvements.

Data collection

One smartphone (Samsung Tab2, 7.0), including 3G for web-based functions with the instruction of system use was made available in the delivery rooms of all participating hospitals for the express purpose of entering data into the trial system. Data of all pregnant women who were admitted to the study hospitals for delivery during the study period were entered within 2 h after delivery. If any of the 10 complications monitored occurred within 24 h after delivery, then they were recorded as well.

Another smartphone of the same model was given to the trained doctors who worked as the actors of the study hospitals for entering action-oriented responses. The actors were responsible for reviewing the indicator events and recording the essential information of the indicator cases, and important treatments and actions performed. Continuous monitoring of data entry for the surveillance system and action responses was performed by the administrator at the data center. After the period of field-testing the system ended, an anonymous evaluation form was sent to all doctors and nurses who participated in the study to get their opinions on how well the system performed, and if it would be a useful thing to implement in their hospitals.

Outcome measures and data analysis

The main outcome measures in this study were the completeness of data entry in terms of the number of women who were admitted and delivered with or without complications, those who had complications which were entered into the system and acted on, and the acceptance of the system as evaluated by the health personnel at the end of study. The percentages of number of data entered completely were analyzed. The results in the system were updated automatically and presented in the form of tables, graphs and maps. The tabular results were created automatically using a PHP script. The graphs were displayed using JpGraph, a freeware PHP library.¹⁹ The map showed the location of events using a geographic information system via the Google Maps application. Different options, such as daily or monthly frequency, by hospital or by indicator were made available for data analysis and data visualizations including individual data. Number of deliveries and complications by age and other variables were summarized and shown in the tables.

The acceptance of the system was measured by evaluating the perceived ease of use and perceived usefulness of the system from the users’ perspective. Perceived ease of use was measured through three items, overall functionality of the system, the data entry process and time consumed for data entry. Overall functionality and data entry process were rated using a 5-point Likert scale ranging from 1 (very difficult) to 5 (very easy) while time consumed was rated using a 5-point Likert scale ranging from 1 (too long) to 5 (very reasonable). Perceived usefulness of the system was measured on three items in terms of the benefit of variables recorded in the system, improvement of services provided and a perceivable effect on improving women health using a 5-point Likert scale range from 1 (not beneficial at all) to 5 (strongly beneficial). The percentage of each rating scale for the items was descriptively presented. The internal consistency of the three items for measuring the perceived ease of use and usefulness of the system was tested by Cronbach’s alpha before summing into scores of perceived ease of use and usefulness of the system. The Cronbach’s alphas of perceived ease of use and usefulness were 0.78 and 0.80, respectively.

Intention to use the system for data entry operators and actors was measured by asking the following question: “How likely would it be for you to use the system in the future if implemented into your hospital in terms of individual data entry and action-oriented responses?.” The responses were rated using a 5-point Likert scale ranging from 1 (very difficult) to 5 (very easy). Responses of 1, 2 and 3 were categorized into “no intention” and responses of 4 and 5 into “intention.” Scores of perceived ease of use and perceived usefulness of the system were tested for their association with the intention to use by univariate and multivariate analysis. A p-value of less than 0.05 was considered as significant.

Results

During the field-testing period, 2459 deliveries were recorded. Of these, 920 women had at least 1 of the 10 complications monitored in the surveillance system. Table 1 presents a summary of deliveries and complications. The individual data of all admitted and delivered women in the hospitals during the study periods were entered in the system as planned. The proportion of women with monitored complications which required a response from the actor varied by complication and by hospital.

Table 1.

Number of deliveries and complications recorded by hospitals in the surveillance system.

	Hospital								Total
	A	B	C	D	E	F	G	H	Total
Deliveries	304	1138	117	296	196	148	74	185	2459
Total births	313	1160	116	298	192	143	77	186	2485
Live births	309	1149	113	296	190	142	75	185	2460
Complications, n (%)
Severe preeclampsia/eclampsia^a	8 (2.6)	19 (1.7)	0	3 (1.0)	0	2 (1.4)	0	8 (4.3)	43
Postpartum hemorrhage^a	10 (3.3)	17 (1.5)	0	9 (3.0)	2 (1.0)	5 (3.4)	2 (2.7)	2 (1.1)	50
Sepsis^a	1 (0.3)	5 (0.4)	0	0	1 (0.5)	0	2 (2.7)	1 (0.5)	11
Obstructed labor^a	71 (23.4)	41 (3.6)	0	13 (4.4)	0	13 (8.8)	9 (12.2)	2 (1.1)	157
Cesarean section^a	175 (57.6)	300 (26.4)	0	58 (19.6)	0	1 (0.7)	0	1 (0.5)	555
Maternal death	0	0	0	0	0	0	0	0	0
Preterm^c	40 (12.9)	71 (6.2)	4 (3.5)	6 (2.0)	7 (3.7)	15 (10.6)	3 (4.0)	3 (1.6)	151
Low birth weight^c	38 (12.3)	95 (8.3)	6 (5.3)	25 (8.4)	10 (5.3)	12 (8.4)	2 (2.7)	10 (5.4)	208
Stillbirths^b	4 (13)	2 (2)	0	2 (7)	1 (5)	0	0	2 (11)	12
Neonatal deaths^c	2 (6)	2 (2)	0	0	0	0	1 (13)	0	6

Per deliveries.

Per 1000 births.

Per 1000 live births.

A total of 2 obstetricians, 3 general practitioners and 40 nurses who were involved in the data entry process assessed the system. Their mean age and working experience were 34 and 11 years, respectively. The assessments of perceived ease of use and perceived usefulness of the system are shown in Tables 2 and 3, respectively. Approximately half of them assessed the system as easy or very easy. Two-thirds said that the system was beneficial or strongly beneficial. Less than 10 percent thought that the system was not beneficial.

Table 2.

Perceived ease of use.

Items	Assessment scale, N = 45
	Very difficult, n (%)	Difficult, n (%)	Somewhat easy, n (%)	Easy, n (%)	Very easy, n (%)
	Too long, n (%)	Long, n (%)	Fair, n (%)	Reasonable, n (%)	Very reasonable, n (%)
Overall functionality of system	0	2 (4.4)	18 (40)	22 (48.9)	3 (6.7)
Data entry process	0	3 (6.7)	17 (37.8)	20 (44.4)	5 (11.1)
Time consuming for data entry	0	4 (8.7)	20 (43.5)	18 (39.1)	4 (8.7)

Table 3.

Perceived usefulness of the system.

Items	Assessment scale, N = 45
Items	Not beneficial at all, n (%)	Not beneficial, n (%)	Undetermined, n (%)	Beneficial, n (%)	Strongly beneficial, n (%)
Variables in the system	0	4 (8.9)	10 (22.2)	24 (53.3)	7 (15.6)
Improvement of services provided	0	3 (6.7)	14 (31.1)	23 (51.1)	5 (11.1)
Effect on improving women health	1 (2.2)	1 (2.2)	14 (13.1)	22 (48.9)	7 (15.6)

Table 4 presents the responses of the study personnel regarding their intention to use the system in future routine use. Approximately 40 percent thought that the data entry was somewhat easy and the actor response was easy. The overall rate of intention to use the system for individual data entry was 39.1 percent and for action-oriented responses 41.3 percent. The main reasons given by those who were reluctant to commit to the system was they were too busy to enter the data on a daily basis within 2 h of delivery and monitoring complications and 24 h after delivery. Both perceived ease of use and usefulness of the system were significantly associated with the intention to use for data entry and actor response in univariate analysis. Only perceived ease of use was shown to be significant for data entry (adjusted OR = 4.15, 95% confidence interval 1.08–15.98) and actor response (adjusted OR = 8.17, 95% confidence interval 1.50–44.58), when adjusting for the perceived usefulness of system.

Table 4.

Intention to use the system in routine use.

Items	Assessment Scale, N = 45
	Very difficult, n (%)	Difficult, n (%)	Somewhat easy, n (%)	Easy, n (%)	Very easy, n (%)
Entering data for data entry	0	8 (17.8)	19 (42.2)	14 (31.1)	4 (8.9)
Entering data for actor response	0	10 (22.2)	16 (35.6)	17 (37.8)	2 (4.4)

In addition, the system was informally presented to two policy makers in the Ministry of Public Health, Thailand. Both of them showed interest in the timely reports of aggregated data for complication indicators, automatic updates and ability to view the results instantly.

Discussion

The consensus by all stakeholders was that this electronic web-based database system, including both surveillance and action-oriented responses for common complications, was acceptable and useful. Epidemiological surveillance systems are mostly used for monitoring diseases or disease outbreaks to provide important information for future actions.⁷ However, low accuracy or poor functionality of surveillance and health information systems in most developing countries is common. Maternal and newborn health was chosen for this trial study because it is a global public concern, and there is currently no surveillance system to monitor common morbidities with the individual data of pregnant women. Timely analysis and use of information for program monitoring and taking corrective actions are virtually absent.⁸ A warning system was found to be useful in surveillance systems.^20,21 In addition, the dissemination of results and actions should be provided in a timely manner.⁹ According to these concepts of lessons learned, a facility-based electronic epidemiological surveillance system for maternal and newborn health with timely action-oriented responses for important obstetric complications was successfully developed and tested.

Recently, surveillance systems of maternal and perinatal/neonatal morbidity have been used in Canada, Brazil and China.^22–30 However, the maternal morbidities were measured variously in terms of obstetric or maternal complications, severe obstetric complications or life threatening complications.³¹ Within the system of our study, both common maternal and newborn morbidities and mortality were recorded, and automatically measured with the linkage of all important holistic information to represent representativeness and quality of data and the flexibility, timeliness and stability of the system.¹⁰

However, the goals of developing a surveillance system will not be achieved if the system is not used properly or if the data are not entered completely and in a timely fashion. Nurses and doctors in the field-testing period accepted the system, stating it to have high perceived ease of use and usefulness which was partially consistent with the concept of the technology acceptance model.¹¹ Perceived ease of use and usefulness of the technology system were commonly applied in previous studies to measure users’ reactions to health information technology.^11,12,32 A literature review by Holden and Karsh¹¹ showed various theories to measure the technology acceptance. Attitude as the determinant component as in the original Technology Acceptance Model (TAM), a “subjective norm” component as in the extended Technology Acceptance Model (TAM2) and social influence as in the Unified Theory of Acceptance and Use of Technology (UTAUT) were not measured in our study.^33–35

The prediction of actual use by acceptance and intention to use was also emphasized in the model of technology acceptance; however, the determinants of intention to use were explained differently in related theories.¹¹ The perceived ease of use and usefulness of the system as determinants of attitude measurement were used to predict the intention to use directly. A previous study found that “subjective norm” or social influence showed no direct effect on adoption of healthcare information technology among Thai healthcare personnel.³⁶ Our web-based system and use of mobile devices were accepted in our settings which have also been noted in previous studies.^12,32 Attitudes of medical doctors have been shown to influence the intention to use a smartphone at the hospital more than social influences. Cost of the smartphone, quality of information, ease of use and support were associated with doctors’ attitudes.³⁷ In our study, perceived ease of use, not usefulness, was significantly associated with intention to use the system, which was opposite to a previous study that showed a direct effect of usefulness.³² This may be explained by the positive agreement of health personnel on the requirement of a surveillance database system via electronic health records. Apart from evaluating the perceived ease of use and usefulness of the system, the acceptance of the system based on its image, job relevance, output quality and result demonstration as in the TAM2 were also assessed in our study.^11,38

When a web-based system is developed, the field-testing is crucial in both non-functional and functional requirements, aiming to check both the process of data input and any failures to link with the various tables in the database.³⁹ This system would provide two main added values which can improve existing systems: real-time, online electronic health records with a surveillance system for major obstetric complications and a warning system for timely action response for case review to prevent avoidable maternal and newborn morbidities and deaths. In addition, stakeholders at higher levels, such as policy makers or program managers, can monitor the aggregate workloads, fluctuations and performances of the system on a daily basis. Having a good infrastructure for entering the data in a database system is important to ensure that the correct number of complications is recorded. Information from women who visit more than one health facility, due to referral or otherwise, can be linked by their unique identification number.

There were some limitations in our study. First, the main purpose of the field-testing was to test the user-friendliness of the system in terms of data entry, data analysis and visualizations, thus the outcome of action responses on health and survival could not be assessed. Second, only complications that occurred within 24 h after delivery were recorded. Third, stratification of deliveries and complications was possible only for a limited number of variables such as age group, gravidity and some antenatal care parameters. However, more variable options can easily be added in future versions if required. Fourth, quick and uninterrupted internet access is essential for data entry and processing. Finally, the social influences or subjective norms and perceived control were not tested.

In conclusion, our study found that a web-based surveillance system with action-oriented responses accessible from various electronic devices for monitoring and evaluating maternal and newborn health and survival at hospital and policy levels is feasible. Appropriate functional designs and administration of this system should be considered for its implementation in different health systems. This system can be modified appropriately and tested in other settings. Active participation of all healthcare workers and administrative officials is important, therefore introduction of the system must be done with appropriate orientation and training of the various levels of providers.

Footnotes

Acknowledgements

T.L. was the principal investigator responsible for the project, database design guidance and management and article preparation. T.P. was involved in database and system development and P.P. participated in database analysis and presentation of results into tables and graphs. N.O. was responsible for coordination and data collection. M.I. supervised the project and commented on the article. All authors approved the final article for submission. We would like to thank the directors of the hospitals in Songkhla province who allowed data collection in their hospitals and the responsible doctors and nurses who participated and facilitated the development and evaluation of this system, namely Dr Jitti Lawantrakul, Mrs Wilaiporn Pengsri, Mrs Peansri Sriint, Hat Yai Regional Hospital, Hat Yai District; Dr Krantarat Peeyananjarassri, Dr Nungruthai Saeaib, Mrs Salita In-Keaw, Songklanagarind Hospital, Hat Yai Disctrict; Dr Napat Pheapol, Mrs Somjit Aupmon, Somdejprabaromrachineenart Nathawee District; Dr Kanjana Pimsupoth, Mrs Mingkuan, Chana Hospital, Chana District; Dr Thanyaporn Kanjanasuwan, Ms Pojana In-Jareon, Thepa Hospital, Thepa District; Dr Worawat Kanjanasawitri, Mrs Onanong Thosakul, Sabayoi Hospital, Sabayoi District; Dr Yuthamas Wandao, Mrs Faizah Hleesen, Sadao Hospital, Sadao District; Dr Wiset Sirinsopon, Mrs Wassana Buasom, Padangbesar Hospital, Sadao District. We would also like to thank Mr Edward McNeil and Mr Dave Patterson of Faculty of Medicine, Prince of Songkla University for their help with English article editing.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was financially supported by the WHO Thailand Country Programme.

References

World Health Organization (WHO), United Nations Population Fund (UNFPA), United Nations Children’s Fund (UNICEF). Monitoring emergency obstetric care: a handbook. Geneva: WHO, 2009.

United Nations Children’s Fund (UNICEF), World Health Organization (WHO) and United Nations Population Fund (UNFPA). Guidelines for monitoring the availability and use of obstetric services. New York: UNICEF, 1997.

Ronsmans

Graham

WJ.

Maternal mortality: who, when, where, and why. Lancet 2006; 368: 1189–1200.

Laopaiboon

Lumbiganon

McDonald

. Use of evidence-based practices in pregnancy and childbirth: South East Asia optimising reproductive and child health in developing countries project. PLoS ONE 2008; 3: e2646.

Schutte

Schuitemaker

van Roosmalen

. Substandard care in maternal mortality due to hypertensive disease in pregnancy in the Netherlands. BJOG 2008; 115: 732–736.

Liabsuetrakul

Promvijit

Pattanapisalsak

. A criterion-based obstetric morbidity audit in southern Thailand. Int J Gynaecol Obstet 2008; 103: 166–171.

Berg

Danel

Mora

Guidelines for maternal mortality epidemiological surveillance. Washington, DC: Office of Protection Services (OPS), 1996.

World Health Organization (WHO). Monitoring and evaluation of maternal and newborn health and services at the district level. Geneva: WHO, 2007.

Alexandor

Survey, surveillance, monitoring and recording. In: Alexandor

(ed.) Management planning for nature conservation. Gwynedd: Springer, 2008, pp. 49–62.

10.

Centers for Disease Control and Prevention (CDC). Maternal health epidemiology. Atlanta, GA: U.S. Department of Health and Human Services, 2003.

11.

Holden

Karsh

BT.

The technology acceptance model: its past and its future in health care. J Biomed Inform 2010; 43: 159–172.

12.

Sun

Wang

Guo

. Understanding the acceptance of mobile health services: a comparison and integration of alternative models. J Electron Commerce Res 2013; 14: 183–199.

13.

Johns

Lekies

Braun

. Web authentication revisited. In: Proceedings of the 28th annual computer security applications conference (ACSAC), Orlando, FL, 3–7 December 2012, pp. 169–178. New York: ACM.

14.

Snyder

Myer

Southwell

. Pro PHP security. 2nd ed. New York: Apress, 2010.

15.

Liang

JavaScript security. 1st ed. Birmingham: Packt Publishing Ltd, 2014.

16.

Malviya VK Saurav

Gupta

. On security issues in web applications through cross site scripting (XSS). In: Proceedings of the 20th Asia-Pacific software engineering conference (APSEC), Bangkok, Thailand, 2–5 December 2013, pp. 583–588. New York: IEEE.

17.

Sweat

Kent

Slenc

PHP graphics handbook. London: Wrox Press, 2003.

18.

Joshi

JBD

Aref

Ghafoor

. Security models for web-based applications. Commun ACM 2001; 44: 38–44.

19.

Fahl

Harbach

Muders

. Why Eve and Mallory love android: an analysis of android SSL (in)security. In: Proceedings of the 2012 ACM conference on computer and communications security (CCS), Raleigh, NC, 16–18 October 2012, pp. 50–61. New York: ACM.

20.

Singh

McGlennan

England

. A validation study of the CEMACH recommended modified early obstetric warning system (MEOWS). Anaesthesia 2012; 67: 12–18.

21.

Souza

Cecatti

Haddad

. The WHO maternal near-miss approach and the maternal severity index model (MSI): tools for assessing the management of severe maternal morbidity. PLoS ONE 2012; 7: e44129.

22.

Wen

Huang

Liston

.; Maternal Health Study Group, Canadian Perinatal Surveillance System. Severe maternal morbidity in Canada, 1991–2001. CMAJ 2005; 173: 759–764.

23.

Allen

Campbell

Carson

.; Canadian Maternal Morbidity Working Group. Maternal mortality and severe maternal morbidity surveillance in Canada. J Obstet Gynaecol Can 2010; 32: 1140–1146.

24.

Dzakpasu

Joseph

Huang

.; Fetal and Infant Health Study Group of the Canadian Perinatal Surveillance System. Decreasing diagnoses of birth asphyxia in Canada: fact or artifact. Pediatrics 2009; 123: e668–e672.

25.

Joseph

Huang

Liu

.; Fetal and Infant Health Study Group of the Canadian Perinatal Surveillance System. Reconciling the high rates of preterm and postterm birth in the United States. Obstet Gynecol 2007; 109: 813–822.

26.

Joseph

Liu

Rouleau

.; Maternal Health Study Group of the Canadian Perinatal Surveillance System. Severe maternal morbidity in Canada, 2003 to 2007: surveillance using routine hospitalization data and ICD-10CA codes. J Obstet Gynaecol Can 2010; 32: 837–846.

27.

Liu

Joseph

Bartholomew

.; Maternal Health Study Group of the Canadian Perinatal Surveillance System. Temporal trends and regional variations in severe maternal morbidity in Canada, 2003 to 2007. J Obstet Gynaecol Can 2010; 32: 847–855.

28.

Liu

Joseph

Liston

.; Maternal Health Study Group of Canadian Perinatal Surveillance System (Public Health Agency of Canada). Incidence, risk factors, and associated complications of eclampsia. Obstet Gynecol 2011; 118: 987–994.

29.

Amaral

Souza

Surita

. A population-based surveillance study on severe acute maternal morbidity (near-miss) and adverse perinatal outcomes in Campinas, Brazil: the Vigimoma Project. BMC Pregnancy Childbirth 2011; 11: 9.

30.

Liang

Dai

Zhu

. Preventable maternal mortality: geographic/rural-urban differences and associated factors from the population-based maternal mortality surveillance system in China. BMC Public Health 2011; 11: 243.

31.

Koblinsky

Chowdhury

Moran

. Maternal morbidity and disability and their consequences: neglected agenda in maternal health. J Health Popul Nutr 2012; 30: 124–130.

32.

Chen

Yang

Tang

. Applying the technology acceptance model to explore public health nurses’ intentions towards web-based learning: a cross-sectional questionnaire survey. Int J Nurs Stud 2008; 45: 869–878.

33.

Davis

Bagozzi

Warshaw

PR.

User acceptance of computer technology: a comparison of 2 theoretical models. Manag Sci 1989; 35: 982–1003.

34.

Sjana

Empirical evaluation of the revised technology acceptance model. Manag Sci 1996; 42: 85–92.

35.

Venkatesh

Morris

Davis

. User acceptance of information technology: toward a unified view. MIS Quart 2003; 27: 425–478.

36.

Phichitchaisopa

Naenna

Factors affecting the adoption of healthcare information technology. EXCLI J 2013; 12: 413–436.

37.

Moon

Chang

Technology acceptance and adoption of innovative smartphone uses among hospital employees. Healthc Inform Res 2014; 20: 304–312.

38.

Hayman

Panniers

Theory development in nursing and healthcare informatics: a model explaining and predicting information and communication technology acceptance by healthcare consumers. ANS Adv Nurs Sci 2007; 30: E37–E49.

39.

Di Lucca

Fasolino

. Testing web-based applications: the state of the art and future trends. Inform Software Tech 2006; 48: 1172–1186.