Understanding the challenges associated with the use of data from routine health information systems in low- and middle-income countries: A systematic review

Abstract

Background:

Routine health information systems (RHISs) are crucial to informing decision-making at all levels of the health system. However, the use of RHIS data in low- and middle-income countries (LMICs) is limited due to concerns regarding quality, accuracy, timeliness, completeness and representativeness.

Objective:

This study systematically reviewed technical, behavioural and organisational/environmental challenges that hinder the use of RHIS data in LMICs and strategies implemented to overcome these challenges.

Method:

Four electronic databases were searched for studies describing challenges associated with the use of RHIS data and/or strategies implemented to circumvent these challenges in LMICs. Identified articles were screened against inclusion and exclusion criteria by two independent reviewers.

Results:

Sixty studies met the inclusion criteria and were included in this review, 55 of which described challenges in using RHIS data and 20 of which focused on strategies to address these challenges. Identified challenges and strategies were organised by their technical, behavioural and organisational/environmental determinants and by the core steps of the data process. Organisational/environmental challenges were the most commonly reported barriers to data use, while technical challenges were the most commonly addressed with strategies.

Conclusion:

Despite the known benefits of RHIS data for health system strengthening, numerous challenges continue to impede their use in practice.

Implications:

Additional research is needed to identify effective strategies for addressing the determinants of RHIS use, particularly given the disconnect identified between the type of challenge most commonly described in the literature and the type of challenge most commonly targeted for interventions.

Keywords

health information systems developing countries public health informatics patient generated health data public reporting of healthcare data data science systematic review

Introduction

Routine health information systems (RHISs) are increasingly being recognised as an essential tool for health system strengthening (Hotchkiss et al., 2012; Wickremasinghe et al., 2016), and the aim of their implementation is to improve health service management through the ongoing collection of information on the supply and use of health services in health facilities. Defined as “any system of data collection, distribution, and use that provides information at regular intervals” (Hotchkiss et al., 2012), RHISs capture data on health status, health services and health resources within a population and represent rich information sources, vital for informing decision-making at all levels of the health system, including resource allocation, day-to-day management, strategy development and policymaking (AbouZahr and Boerma, 2005; Leon et al., 2015; Wagenaar et al., 2016; World Health Organisation, 2007).

However, in many low- and middle-income countries (LMICs), RHISs remain fragmented and disorganised, and concerns regarding the quality, accuracy, timeliness, completeness and representativeness of RHIS data are widespread (Dehnavieh et al., 2018; Mutemwa, 2006; Odhiambo-otieno, 2005). Indeed, a number of studies have assessed the quality of RHISs in LMIC settings and have identified several technical, behavioural and organisational challenges to the introduction, implementation and use of these systems (Aqil et al., 2009; Dehnavieh et al., 2018; Hotchkiss et al., 2012; Kumar et al., 2018). At the technical level, for instance, a lack of knowledge, skills and specialised technical infrastructure can hinder the collection and use of high-quality data, while at the behavioural level, poor demand for RHIS data, as well as poor motivation and competency among health workers, can impede its use (Leon et al., 2015; MEASURE Evaluation, 2019). Organisationally, inadequate governance and management, a lack of training, supervision and resources and the failure to promote a culture of data use have been identified as issues (Leon et al., 2015; MEASURE Evaluation, 2019). These challenges often render RHIS data unreliable or irrelevant, impede their usefulness in practice and contribute to the continued preference for intermittent cross-sectional population-based surveys as the primary source of data for tracking population health, risk factors and health service coverage (Aqil et al., 2009; Ashton et al., 2017; Wagenaar et al., 2016).

Given that strong RHISs that capture, store, manage and transmit health information are necessary for improving the quality of healthcare in LMICs, as well as for tracking progress towards achieving targets such as those outlined in the Sustainable Development Goals (Thomas et al., 2016), a greater understanding of the factors that inhibit the effective use of RHIS data and potential strategies to address these factors is required. Although health information technology in general has been previously explored in a number of systematic and scoping reviews, for example that by Ndabarora et al. (2014), who assessed data quality issues associated with health information at the community and district levels in LMICs, the majority of these articles focused on the effect of such technologies on specific health outcomes, with very few looking solely at RHISs, or at LMIC populations. In fact, only one systematic review by Hotchkiss et al. (2012) has investigated the challenges associated with RHISs in LMICs to date. However, Hotchkiss et al. described the effectiveness of interventions to improve RHIS performance rather than challenges in implementation and use more broadly and were primarily concerned with the role of RHISs at the local level. While a protocol for an additional similar review by Leon et al. (2015) was identified, that study also intended to focus on interventions to improve RHISs and (at time of publication) the results have yet to be released. Given that RHISs can improve health system functioning at all levels and that challenges associated with RHISs have been widely described in studies focusing on the implementation of such systems in addition to those evaluating interventions to improve them, a broader review of the determinants of RHIS data use and strategies to improve RHISs is warranted.

This review aimed to systematically identify common challenges associated with the use of RHISs in LMICs, as well as strategies implemented to mitigate these challenges. Specifically, we aimed to identify the technical, behavioural and organisational/environmental determinants of RHIS data use in LMICs and evaluate the effectiveness of interventions that have targeted them (Aqil et al., 2009; MEASURE Evaluation, 2019). The findings of this study will provide new insights into the challenges that impede the use of RHIS data and may inform the development of future strategies to reduce these challenges and improve RHIS functioning and use in LMICs.

Method

Key definitions

For the purposes of this review, RHISs were defined as systems designed to collect, process and use health information at regular intervals of a year or less to meet predictable information needs within healthcare systems (Dagnew et al., 2018; Hotchkiss et al., 2012). This includes information on clinical service delivery, medicine or diagnostic service provision, as well as routine administrative systems and management information systems, which aim to improve healthcare management decisions. Conversely, individual-level data collected for clinical decision-making purposes and pilot projects testing the implementation of a new data collection method were not considered RHISs. RHIS data were defined as information retrieved from the RHIS by stakeholders with the intent to use the data for decision-making within health systems.

Search strategy

A comprehensive literature search strategy was developed and was tested and refined for each of four electronic databases: PubMed, Scopus, EMBASE and EconLit, current as of 20 February 2019, inclusive. These databases were selected based both on previous reviews of similar topics, for example those by Hotchkiss et al. (2012), who searched PubMed, and Leon et al. (2015), who searched Embase, and on what was available at the authors’ institutions. After searching the four databases listed, the authors identified substantial saturation in results and thus opted to end the search. Within each database, varied phrases to describe RHISs such as “routine health information system”, “health information system” or “health management information system” were included, along with a list of LMICs as defined by the World Bank’s (2019) classification system (see Appendix 1 for search strategy). Database filters were used to eliminate all non-English language studies. A review protocol was not registered prior to undertaking the literature search.

Eligibility criteria

Articles were selected for inclusion in this review if they met the following criteria: (i) were published in English, (ii) were an original research study and (iii) described or investigated challenges associated with RHIS data use or strategies implemented to improve RHIS data use in at least one LMIC. Studies were excluded if (i) their description of RHIS data use challenges was not presented as part of the main findings but was included as background information or discussion only or (ii) they focused only on the challenges of RHIS implementation, without any link to data use. Studies for which the full text could not be located, even after attempts were made to contact the authors, were also excluded. All retrieved studies were classified as challenge articles or strategy articles. Challenge articles were defined as studies that described any factor that was perceived to impede the use of RHIS data. Strategy articles included studies that discussed an intended solution to improve or correct a perceived challenge.

Selection of studies

This study followed the preferred reporting items for systematic reviews and meta-analyses guidelines (Moher et al., 2009). The screening process, including the number of records excluded at each stage, is outlined in Figure 1. In total, 2310 records were retrieved from the database search and were imported into Covidence systematic review software (2013). After the removal of duplicates, 1459 articles were independently screened against the inclusion and exclusion criteria by two reviewers, KH and YH, in two stages: (i) title and abstract and (ii) full text. Disagreements at each stage were resolved through discussion and a third reviewer, BI, was consulted when required. For convenience sake, the authors served as the reviewers. Where there was insufficient information to determine the eligibility at the title and abstract screening stage, the article was included for full-text screening. Title and abstract screening resulted 396 records progressing to full-text assessment. Of these, 60 met the inclusion criteria and were extracted. No risk of bias assessment was conducted, due to the fact that our systematic review was not concerned with the results of the individual included studies and did not seek to undertake any statistical analyses.

Figure 1.

PRISMA diagram summarising study selection process. PRISMA: preferred reporting items for systematic reviews and meta-analyses.

Data extraction

Data from all included studies were extracted by three authors, KH, YH and BI, using a modified extraction form. Background information including the article’s first author, year of publication, journal and location of study was collected first, along with study information, including study design, objective, participants (if applicable), description of RHIS data used (if applicable), description of challenges that impeded the use of RHIS data (if applicable), description of strategies implemented to improve RHIS use or circumvent data quality issues (if applicable) and study recommendations and conclusions.

The extracted data were synthesised narratively. Challenges and strategies described in each study were categorised according to their technical, behavioural and organisational/environmental determinants as introduced in the performance of routine health information system management (PRISM) framework (Aqil et al., 2009; MEASURE Evaluation, 2019), and multiple determinants within a single study were permitted. Technical determinants refer to factors related to the specialised knowledge and technology that are required to develop, improve and manage RHISs, such as software development and data collection form design. Behavioural determinants include user-focused factors such as users’ demand, motivation, confidence and competence to perform RHIS tasks. Organisational/environmental determinants encompass organisational processes and structures such as human and financial resources, management, leadership and supervision. Within each determinant, challenges were organised by the specific stage of the RHIS data process they occurred at: the information needs stage, the data collection stage, the data transmission stage, the data processing/analysis stage and the dissemination of processed information stage (Aqil et al., 2009; MEASURE Evaluation, 2019).

Results

Study characteristics

The general characteristics of the 60 studies included in this review are presented in Table 1. Overall, the majority of studies described RHIS data use in countries in sub-Saharan Africa (67%). Most of the studies were recent, with more than half published after 2014 (60%), and mixed-methods studies were the most common study design (27%). Of the included studies, 92% described challenges that hindered the use of RHIS data at different levels of the health system, while a third presented or described strategies implemented to improve the use of RHIS data. A quarter of the articles described both challenges associated with the use of RHIS data and strategies that were implemented to overcome these challenges. The majority of the studies (n = 35, 58%) were undertaken at the subnational health system level, while only five (8.3%) occurred at the district level. A third of the included studies identified both national and subnational levels.

Table 1.

General characteristics of included studies.

	n	%
Geographical region
East Asia and Pacific	5	8.3
Latin America and the Caribbean	3	5.0
Middle East and North Africa	2	3.3
South Asia	9	15.0
Sub-Saharan Africa	40	66.7
Multiple regions	1	1.7
Year of publication
<2000	3	5.0
2000–2009	8	13.3
2010–2014	13	21.7
2015–2019	36	60.0
Types of study design
Action research study	1	1.7
Case study	10	16.7
Case study and scoping review	1	1.7
Cross-sectional study	5	8.3
Descriptive study	6	10.0
Ecological study – longitudinal	2	3.3
Field experiment study	1	1.7
Intervention study with pre- to post-assessment	2	3.3
Mixed-methods study	16	26.7
Multi-case study	4	6.7
Pre- to post-study	1	1.7
Qualitative study	11	18.3
Description of challenges/strategies
Challenges in using data from routine health information systems	55	91.7
Strategies to improve data use	20	33.3
Level of health system
National	5	8.3
Subnational	35	58.3
Provincial	4	6.7
District	26	43.3
Facility level	5	8.3
Mixed	20	33.3
National and subnational (provincial)	5	8.3
National and subnational (district)	11	18.3
National and subnational (facility level)	4	6.7

Challenges associated with the use of RHISs

The most common challenges associated with RHIS data use are summarised in Table 2. Challenges presented in the articles were grouped based on similarities, then categorised according to the definitions of determinants and the core components of health information system data processes, as described in the PRISM framework (Aqil et al., 2009; MEASURE Evaluation, 2019).

Table 2.

Common challenges identified in included studies.

Data process	Technical determinants	Behavioural determinants	Environment/organisational determinants
Information needs	Forms lack explicit information on disaggregation (e.g. sex) or definition Different definitions of disaggregation in paper vs. electronic form Data collection lack information on emerging diseases	Misclassification of conditions by health workers Incomplete data collection by health workers due to lack of recognition of data needs and purpose Incorrect input from health workers’ recollection	Lack of diagnostic confirmation Shortage of human resources Lack of training for staff on recording data Local needs for decision-making not included in study design
Data collection	Fragmentation of data due to use of different tools and reporting systems Lack of integration with community-level data	Lack of supervisory feedback to prevent falsification of data	Lack of system for data storage and dedicated staff for data management Lack of or limited standardisation of monitoring and supervision process on data
Data transmission	Inconsistency/low Internet connectivity for data transmission and use of web-based tools Data submitted in format difficult to access	Delay in data submission; some occasions due to delay in salary Resistance to electronic communication	Lack of resources for data transmission from facility Lack of transportation of manual data transmission and verification of data from district Lack of or limited standardisation of monitoring and supervision process on data
Data processing/analysis	Inaccurate population estimates for calculation of percentages and rates	Common errors on simple mathematical computation Infrequent data analysis	Lack of training on data management analysis and interpretation Lack of equipment to facilitate data analysis (e.g. computer) Lack of or limited standardisation of monitoring and supervision process on data
Dissemination of processed information	Conflicting figures for same service across different reporting platform and survey	Difficulty understanding feedback from central level	Limited feedback from ministry to district level Limited dissemination of data Lack of inter- and intradepartmental coordination on data sharing

Technical determinants

Of the 55 articles that presented challenges in RHIS data use, 21 (38%) described technical issues, which occurred across the information needs, data collection, data transmission, data processing/analysis and dissemination of processed information stages. At the information needs stage, the majority of technical issues were related to limitations with the data collection forms, which prevented researchers from accurately assessing the current state of health in a population (Soeung et al., 2007; Qazi, Ali and Kuroiwa, 2008; Scott et al., 2017; Rasmussen, 2018). In Peru, for example, Iguiñiz-Romero and Palomino (2012) identified the poor structure of forms as a major challenge to collecting high-quality data, finding an absence of standardised case definitions, as well as information on patient characteristics such as sex. This made it impossible to accurately estimate maternal mortality because the sex of the deceased patients was unknown (Iguiñiz-Romero and Palomino, 2012). Similarly, Abajebel et al. (2011) observed that in Ethiopia, the forms used to collect RHIS data were not regularly updated to include data on emerging diseases and that this hindered use (Abajebel et al., 2011).

At the data collection stage, the method of collection was identified as an important factor, and discrepancies in the information recorded using multiple different reporting systems and tools were reported in Uganda and Botswana, resulting in discordant estimations and missing data (Seitio-Kgokgwe et al., 2015; Ward et al., 2017). In line with this, Mutale et al. (2013)’s evaluation of a series of interventions aimed at enhancing health information systems for decision-making in sub-Saharan Africa, known as the Population Health Implementation and Training (PHIT) Partnerships, concluded that community-level data were poorly integrated with RHISs in Tanzania. These technical issues in data collection yielded discrepancies and missing information, discouraging the use of the resulting data (Mutale et al., 2013; Qazi and Ali, 2009).

Technical determinants also presented challenges at the data transmission and processing stages, and Njuguna et al. (2017), Qazi and Ali (2009), Mpofu et al. (2014) and Chikumba and Ramussen (2016) all described issues with inconsistent and poor quality Internet, which limited the use of web-based tools, hindering data entry and transmission (Chikumba and Rasmussen, 2016; Mpofu et al., 2014; Njuguna et al., 2017; Qazi and Ali, 2009). Moreover, Qazi and Ali (2009) observed the submission of RHIS data in formats that could not be accessed by other data users in Pakistan, resulting in delays in processing the data and publishing reports (Qazi and Ali, 2009). Additionally, unreliable reporting by health facilities and inaccuracies in the population estimates in Papua New Guinea prevented proper data processing and analysis (Field et al., 2018). The use of multiple reporting platforms also delayed information dissemination due to conflicting information (Mpofu et al., 2014; Tshimanga et al., 1997).

Behavioural determinants

Over a third (38%, n = 21) of the included studies investigated behavioural challenges associated with RHIS data use, which generally resulted from inadequately trained health workers (Otieno, Kaseje and Kaseje, 2017; Bernardi, 2017). At the information needs and data collection stages, for instance, studies in Zimbabwe (Omar et al., 1994) and India (Dehury and Chatterjee, 2018) identified a widespread lack of understanding among health workers of the importance of the data being collected, which contributed to incomplete or low-quality data. Dehury and Chatterjee (2018) suggested that improved education on the purpose and uses of the data, particularly among lower level workers, may improve completeness (Dehury and Chatterjee, 2018). Frequent health worker error in classifying health conditions during documentation as well as post hoc recording was also described, resulting in inaccurate estimates of the prevalence of the condition in the population, and thus discouraging the use of the data (Cibulskis and Hiawalyer, 2002; Mosse and Sahay, 2003). Furthermore, an other study reported falsification of data when there was a lack of supervision or feedback provided to lower level healthcare workers (Asah et al., 2017).

Behavioural factors also played an important role in data transmission issues, primarily due to poor staff motivation. Asah et al. (2017), for instance, found that data submission was commonly delayed when health workers were not paid on time (Asah et al., 2017), while Vidor et al. (2011) observed that, despite widespread computer and Internet availability, many of the healthcare facilities in southern Brazil continued to prefer traditional communication methods and this led to delays in information feedback from the central levels to the municipalities (Vidor et al., 2011). In addition, the Brazil study found that over a sixth of the municipalities did not analyse their data, while a fifth analysed their data but did not use it for planning (Vidor et al., 2011). The authors attributed the lack of data analysis to the limited training and qualifications of health workers, combined with the belief that data should be selectively analysed at the central level. Such issue was also reflected by common mathematical errors, which hindered data processing and analysis (Omar et al., 1994). Lastly, at the dissemination stage, a study in Tanzania by Kimaro and Twaakyondo (2005) observed that health workers often had difficulties understanding the feedback provided to them from the central level on reports, leading to misunderstandings and delays for correcting mistakes (Kimaro and Twaakyondo, 2005).

Organisational determinants

Organisational and environmental challenges were the most common issues found to impede RHIS data use in the included studies, described by over half (62%, n = 34) of articles. Of these challenges, the majority were related to resource constraints, such as human and capital shortages, which hindered all steps of the data process (Koivu et al., 2016; Tiwari et al., 2016; Ahsan et al., 2017; Nicol et al., 2017; Askar, Ardakani and Majdzade, 2017; Radin et al., 2017; Nemser et al., 2018; Pinto et al., 2018). For instance, at the information needs stage, Pinchoff et al. (2016) found that it was not always possible to confirm diagnoses such as malaria, due to limited laboratory capacity in Zambia (Pinchoff et al., 2016). This resulted in delayed or inaccurate reporting of data on the burden of diseases from health facilities, making it difficult to apply the information to inform decision-making (Pinchoff et al., 2016). Studies from Pakistan (Qazi et al., 2008; Qazi and Ali, 2011), South Africa (Nicol et al., 2017) and Mozambique (Jani et al., 2006) also noted substantial human resource scarcities within health facilities, and clinicians in particular were severely overworked. This limited staff capacity to participate in the data collection, reporting and analysis process, impeding the usefulness of the RHIS. Furthermore, Yazdi-Feyzabadi et al. (2015) revealed that there was a general assumption among staff that RHIS data would only be used at the ministry level in Iran, where decision-making occurred (Yazdi-Feyzabadi et al., 2015). Additional resource constraints, namely the lack of an adequate data storage system was another frequent challenge faced by health facilities at the data collection stage (Mboera et al., 2017; Ranasinghe et al., 2012). These studies also described the lack of dedicated and motivated staff for data management as a barrier to the data collection process.

Limited resources also emerged as a challenge to the transmission of RHIS data in many studies, with multiple authors reporting that a lack of transportation impeded the entry of manually collected data, particularly in settings in which poor computer and Internet availability prevented electronic reporting (Abajebel et al., 2011; Mpofu et al., 2014; Nutley et al., 2014; Scott and Gilson, 2017; Yazdi-Feyzabadi et al., 2015). Due to a lack of web-based technologies for recording data, for example, statistical forms completed at the point of service delivery in Iran must often be physically transferred to higher levels using vehicles, particularly in disadvantaged districts (Yazdi-Feyzabadi et al., 2015). One district health centre worker noted that this process can postpone data transfer, and thus use, and that in some cases, the data were simply lost, representing not only increased work for the healthcare workers but also a potential security issue (Yazdi-Feyzabadi et al., 2015).

At the data processing and analysis stage, equipment shortages and a lack of training provided to health personnel were identified as major barrier to RHIS data use by many authors. Low knowledge of data management, analysis and interpretation among staff was described in several studies (Ledikwe et al., 2013; Qazi and Ali, 2009; Shiferaw et al., 2017; Vidor et al., 2011). In Ethiopia, Shiferaw et al. (2017) found that 95% of RHIS health workers surveyed had not received training on data analysis or management, while more than half (53%) hadn’t been trained on the use of the RHIS system itself (Shiferaw et al., 2017). The study identified a higher odds of good RHIS utilisation among health workers that had strong data analysis skills, suggesting that a lack of training inhibits RHIS use (Shiferaw et al., 2017). Indeed, the authors noted that inadequate monitoring and supervision procedures resulted in poor skills development and the misuse of resources by health workers, which in turn led to reduced availability and poor data quality, ultimately contributing to a weak management system (Qazi and Ali, 2009).

In addition to resource constraints, the dissemination of processed data was inhibited by unsystematic and non-institutionalised feedback from the ministry to the district-level and a lack of inter- and intradepartmental coordination. As described by Seitio-Kgokgwe et al. (2015, 2016) in Botswana, for example, district managers frequently felt that the information generated at the health facility and the district levels should be sent back from the ministry with feedback to facilitate corrective measures; however, no such mechanisms were in place to ensure that the information was shared. Specifically, lack of coordination between various health systems level further limited use of RHIS data. In Pakistan, federal perception was that the RHIS was a provincial process, whereas at the district level, the managers felt indifferent due to lack of feedback (Qazi and Ali, 2009).

Strategies to mitigate challenges

In addition to challenges, 20 (33.3%) of the studies included in this review described strategies to improve the use of RHIS data, which are summarised in Table 3. As with challenges, strategies were organised according to the specific determinant they addressed: technical, behavioural or organisational/environmental, and multiple determinants could be targeted within a single strategy.

Table 3.

Strategies described to improve data use in included studies.

Strategy	Study description	Evidence of improvement	Reference
Technical determinant
Development of indicators	Analysis and dissemination of data limited to annual booklet on family health indicators, of which data are available and considered important	Publication prepared in advance, allowing for feedback. Feedback helped to increase reporting rates from 73% in 1994 to 85% in 1995 and to 93% in 2000	Cibulskis and Hiawalyer (2002)
Tool creation /development	Dashboard development for regular review to identify problematic HF	Led to increased improvement in completeness and accuracy of reports across country	Etamesor et al. (2018)
	Implementation of mobile health management information system	Data were collected, then stored in the cloud for immediate transmission to HMIS. Helped with complete and accurate data (with build-in data validation)	Mengesha et al. (2018)
	DHP tool development: integrate data and enable staff	Led to improved review and use of data, also increasing demand for additional data	Nutley et al. (2013)
	Connect project to integrate data using data capture tools	Increase in review of community and HF data for feedback	Mutale et al. (2013)
	HRHIS staff training tool	Generation of a report took the HRD department less than 10 min after implementation, with the added feature of being able to sort the data by cadre, facility or health district	Driessen et al. (2015)
Register development	Simplification of registers to ensure consistent supply	Improved flow of information and supports integration of health service operations, streamlined reporting	Awoonor-Williams et al. (2013); Mutale et al. (2013)
Data quality assessment/audit tool	Data quality improvement strategy implemented in Zambia, Rwanda and Mozambique	Improvement in data availability and consistency led to targeted resource allocation (Mozambique). Improved linkages between facility and community health programmes (Rwanda)	Gimbel et al. (2017)
	Data quality assessment system included data quality audits at national level and RDGAs at the regional level	Accuracy: increase from 43% to 60% and 40% to 81% at the facility and district levels. Completeness: increase from 43% to 65% and 80% to 98% at the facility and district levels	Nutley et al. (2014)
Behavioural determinant
Training/skill building of staff	Health workers (HW) provided with tools and training for data analysis, given a “health centre record” – a booklet designed to assess trends over time	Not mentioned	Cibulskis and Hiawalyer (2002)
	Data improvement teams visited health facilities to train staff in improving data	Staff felt that visit by the teams helped them develop systems to enhance completeness and accuracy of data reported on the monthly HMIS report	Ward et al. (2017)
	The trainings involved basic computer use, the DHIS and data handling skills	Not mentioned.	Kimaro and Twaakyondo (2005)
	Staff training to combat weaknesses in terms of data use, analysing, interpretation and presentation	Discussion of RHIS analyses increased from 40% to 82% at the district level and 34% to 42% at facility level	Nutley et al. (2014)
	Focus-based training on revised data tools and on daily responsibilities	Since inception, there has been substantial improvement in completeness and accuracy of reports from health facilities	Etamesor et al. (2018)
	Intervention covered overview of HMIS, data quality, HF HMIS recording and reporting system, analysis, presentation, information use and feedback	Improvements found in completeness, accuracy, timeliness, feedback and use	Nwankwo and Sambo (2018)
	Training of staff on the use of DHIS software	None	Mosse and Sahay (2003)
Data use workshops	Each district presented and assessed their own data using standardized analysis templates from the Millennium Development Goals. Peers discussed and critiqued the presentations	Increased feeling of competence by workers, increased problem-solving skill. Improvement in computer skills. Ultimately improve health worker skills; 15% and 38% of facilities showed improvements in data completeness and timeliness, respectively	Braa and Sahay (2012)
Organisational/environmental determinant
Management support/changes	Facilities that had poor data quality received additional supportive supervision	Concordance improved significantly by an average of 1.04% per month from 2010 to 2011 and by 1.56% per month from 2011 to 2012	Wagenaar et al. (2015)
	Introduction of structural changes in district health management: dividing responsibilities	Improvements found in indicator duplicates, internet affordability, managerial control, capacity of staff	Latifov and Sahay (2013)
Monitoring and evaluation cadre development	Recent graduates recruited and provided with on-the-job training/mentoring to develop new cadre of health worker: District Monitoring and evaluation (M&E) officer	Cadre improved health worker capacity to monitor and evaluate programmes within the districts; improved data quality, management and reporting; increased use of health data for surveillance and planning purposes and increased availability of time for other health workers to concentrate on clinical duties	Ledikwe et al. (2013); Mpofu et al. (2014)
	Creation of “director of monitoring and evaluation” position to ensure M&E and data-based decision-making would be priority	Information feedback after supervisory visits increased from 7% to 29% at the facility level	Nutley et al. (2014)
Health management programme implementation	SBM-R approach implemented in over 30 LMIC; model designed to confront realities in health systems in resource-constrained settings	Improved provider compliance and performance standards	Necochea et al. (2015)
	To develop leadership of M&E systems; ministries participated in virtual leadership development programme	Not mentioned	Nutley et al. (2014)
	League table development: aimed to assist health management	Compared HF within districts and exposed missing data and data gaps. Improved communication and sharing of data between DHMT	Moyo et al. (2016)
Ministry strengthening	Strengthen HIS through innovated approaches to improve quality and use for resource allocation, monitoring and service delivery	Improved feedback; discovery of missing data and outliers	Mutale et al. (2013)

SBM-R: Standards-Based Management and Recognition; LMICs: low- and middle-income countries; DHP: district health profile; DHIS: District Health Information System; HRHIS: Human Resources for Health Information System; RDGA: routine data quality assessment; RHIS: routine health information system; HF: health facilities; HMIS: health management information system; HRD: human resources development; HIS: health information system; DHMT: district health management team.

Technical determinants

Technical challenges were targeted by two-fifths (n = 8) of the RHIS interventions described in the included studies, the majority of which focused on indicator or register development or the creation of tools. For instance, the introduction of simplified registers in Ghana to standardise data sources led to the consolidation of data collection from 27 register books to 5 (Awoonor-Williams et al., 2013; Mutale et al., 2013). It streamlined data collection and aggregation operations, as well as simplified feedback, allowing health workers to review their performance (Mutale et al., 2013). Additionally, to improve the analysis and dissemination of data at all levels of the health system in Papua New Guinea, Cibulskis et al. (1995) described their strategy of limiting indicators to an annual booklet containing only those that were considered highly important and were readily available for analysis (Cibulskis and Hiawalyer, 2002). This enabled provincial managers to prepare the annual report in advance and allow for feedback, which in turn helped increase reporting rates over a 6-year period (Cibulskis and Hiawalyer, 2002).

The development of various tools as a strategy to combat poor RHIS data use was described in five studies (25%), all of which yielded good results (Etamesor et al., 2018; Mengesha et al., 2018; Mutale et al., 2013; Nutley et al., 2013). In Nigeria, Etamesor et al. (2018) developed the DHIS2 Routine Immunization Dashboard Project, which aimed to improve routine immunisation reports through identifying the health facilities that were falling short (Etamesor et al., 2018). The program was successful at enhancing the completeness, timeliness and availability of coverage data in health facilities across the country, allowing these data to be used to improve the immunisation program (Etamesor et al., 2018). Another study in Ethiopia implemented a mobile health management system to collect and store data in the cloud for immediate transmission to the RHIS, preventing information loss at the data transmission stage (Mengesha et al., 2018). Several strategies focused on improving RHIS data integration with community-level data, which also encouraged community coordinators to have ownership over the reporting and use of RHIS data (Mutale et al., 2013). Similarly, Nutley et al. (2013) described the development of a District Health Profile tool in Kenya to integrate data from multiple sources, allowing staff to comprehensively review the data with auto-generated complex graphs by district to make informed decisions (Nutley et al., 2013). In addition to the tools described above, data quality assessment (DQA) strategies were implemented in Rwanda (Gimbel et al., 2017), Mozambique (Gimbel et al., 2017) and Cote d’Ivoire (Nutley et al., 2013). Gimbel et al. (2017) found that in Mozambique, the addition of a mini data quality audit tool in health facilities allowed for district supervisors to assess the concordance between registries and facility forms, which led to a significant improvement in data availability (84% to 99%) and consistency (54% to 87%) (Gimbel et al., 2017). In Côte d’Ivoire, the introduction of a routine DQA tool – a self-administered tool intended to verify the quality of reported data and improve data management and reporting systems – led to an almost 40% increase in data accuracy and a 20% increase in data completeness (Nutley et al., 2013).

Behavioural determinants

Strategies to address the behavioural challenges associated with RHIS data use were also common (35%, n = 7) in the included studies, with most incorporating a staff education or training component to enhance skills among health workers (Cibulskis and Hiawalyer, 2002; Etamesor et al., 2018; Kimaro and Twaakyondo, 2005; Mosse and Sahay, 2003; Nutley et al., 2014; Nwankwo and Sambo, 2018; Ward et al., 2017). Several of these education interventions resulted in significant improvements, and Ward et al. (2017), for example, observed a marked improvement in staff capacity in Uganda following visits to health facilities by data improvement teams (Ward et al., 2017). The training took place over a 5-day period and involved practical classroom workshops, deployments with rapid data quality and organisational assessments and on-the-job learning (Ward et al., 2017). The study found improved knowledge among majority of district staff and better preparedness. Similarly, Nutley et al. (2014) described courses in data monitoring and evaluation, quality and use, offered at national training institutions and universities, that enhanced staff capacity to use data (Nutley et al., 2014). Improved staff training contributed to an overall increase of discussion of RHIS analyses and decisions taken at the district and facility levels (Nutley et al., 2014).

On the other hand, a few studies (10%, n = 2) did not find evidence of improved RHIS use following training. In Nigeria, health workers participated in a series of modules that covered key RHIS topics such as data quality, health facility recording and reporting, data analysis, presentation, information use and feedback (Nwankwo and Sambo, 2018). In general, the authors found no significant improvements in the accuracy, completeness and timeliness of data or in analysis and local use of information, following the training (Nwankwo and Sambo, 2018). Additionally, although Etamesor et al. (2018) observed marked improvements in rates of RHIS data completion and accuracy following training programs that were tailored to health workers’ experiences and responsibilities, the strategies implemented were multifaceted, and thus, these positive changes could not be attributed directly to the training (Etamesor et al., 2018). A further three studies (15%, n = 3) (Cibulskis and Hiawalyer, 2002; Kimaro and Twaakyondo, 2005; Mosse and Sahay, 2003) mentioned training staff on District Health Information System use, computer use, data handling or data analysis as part of their interventions to improve RHIS use but did not provide details on the outcomes of these interventions.

Along with training, data use workshops were implemented in health facilities in an attempt to improve health workers’ competence and motivation in Tanzania (Braa and Sahay, 2012). Braa et al. (2012) described quarterly week-long workshops, in which each district presented and assessed their own data using standardised analysis templates based on the Millennium Development Goals and local strategic plans (Braa and Sahay, 2012). Following each presentation, participants discussed and critiqued the data to identify common issues of data quality and health service performance, which promoted local involvement and improved quality. The workshops also provided opportunities for direct feedback on the revision of indicators and datasets and increased problem-solving and computer skills among workers (Braa and Sahay, 2012).

Organisational determinants

Eight articles (40%) evaluated strategies to address the organisational and environmental challenges hindering RHIS use, including management support/changes, monitoring and evaluating cadre development, health management program implementation and ministry strengthening, all of which were largely successful. In Mozambique, for instance, Wagenaar et al. (2015) described the implementation of the Mozambique PHIT Partnership, which ranked health facilities by summary data concordance measures and provided supportive supervision and data training to those who placed at the bottom (Wagenaar et al., 2015). In Tajikistan, the Ministry of Health introduced infrastructural changes to divide RHIS responsibilities within district health management (Latifov and Sahay, 2013). Both programs were found to foster improvements in data use processes, with the former resulting in a significant increase in data concordance (Wagenaar et al., 2015) and the latter yielding greater local RHIS data use, due to increased autonomy and budgetary control among local health workers and district managers (Latifov and Sahay, 2013).

Rather than management changes, several studies (15%, n = 3) reported the development of special monitoring and evaluation positions as a strategy to improve data use, largely in response to the shortage of skilled health information personnel in some settings (Ledikwe et al., 2013; Mpofu et al., 2014; Nutley et al., 2014). In Botswana, for example, a new cadre of health workers known as Monitoring and Evaluation District Officers was introduced, providing on-the-job training and mentoring to recent university graduates, the majority of whom had no monitoring or evaluation experience (Ledikwe et al., 2013; Mpofu et al., 2014). The creation of this specialised cadre freed up other health workers to concentrate on clinical duties, while simultaneously enhancing the capacity to monitor and evaluate programs within the districts through improved data quality, management and reporting and increased use of health data for surveillance and planning purposes (Mpofu et al., 2014). Similarly, in Côte d’Ivoire, a Director of Monitoring and Evaluation position was created to ensure that decision-making would be prioritised (Nutley et al., 2014). Following the introduction of the role, the use of data for decision-making at the district level rose from 40% to 70%, evidenced by PRISM assessments completed pre- (2007) and post-intervention (2012) (Nutley et al., 2014).

In addition to the creation of a monitoring and evaluation director position, Nutley et al. (2014) developed a virtual leadership development program to build a culture of data use (Nutley et al., 2014). Although the effectiveness of the program was unclear (Nutley et al., 2014), a similar Standards-Based Management and Recognition (SBM-R) approach designed specifically for health systems in resource-constrained settings was found to significantly improve compliance and performance standards in over 30 LMICs (Necochea et al., 2015). Furthermore, to facilitate comparisons between health facilities and identify data quality issues, Moyo et al. (2016) introduced league tables, wherein facilities were ranked according to a set of characteristics (Moyo et al., 2016). Such a process fostered transparency through visible and accessible information, which led to improved information communication and sharing of data, ultimately contributing to an enhanced understanding of the definition and interpretation of indicators among data users and management of the district health teams (Moyo et al., 2016). Similar strategies also enhanced the capacity for feedback, leading to the discovery of missing data and outliers (Mutale et al., 2013).

Discussion

This systematic review explored the technical, behavioural and organisational challenges that affect the use of RHIS data in LMICs, as well as the strategies that have been implemented to address them. Overall, we found that while there has been an increase in the number of studies describing RHIS data use and its challenges in recent years, evidenced by the finding that over half of the articles included in this review were published after 2014, the challenges described in prior reviews by Hotchkiss et al. (2012) and Lippeveld et al. (2000) remain relevant and continue to affect the use of RHIS data. This suggests that little progress has been made in overcoming common challenges impeding greater use of RHIS data in LMIC health systems over the past decade. In line with this, of the studies identified in our review, there were twice the number of articles that described challenges to the use of RHIS compared to studies that described strategies to overcome them, indicating that the majority of research to date has focused on the problems rather than the solutions.

Among the three types of determinants of RHIS data use described in the PRISM framework: technical, behavioural and organisational/environmental (Aqil et al., 2009; MEASURE Evaluation, 2019), we also observed a disconnect between the frequency of types of challenges described and of strategies developed to address them. For example, our review found that technical challenges were the least commonly discussed in the literature, with less than a third of the articles presenting technical challenges such as factors related to tools and technology to manage RHIS performance. Despite this, strategies that incorporated technical components were the most prevalent, many of which involved a focus on developing indicators, registers and tools for data use improvement. Conversely, while more than half of studies highlighted the organisational or environmental challenges associated with RHIS use, including resource shortages, lack of training and management and poor feedback between administrative levels, just 13% of RHIS strategies targeted organisational determinants. Importantly, this number represents an increase in studies targeting organisational determinants in recent years, indicating an expanded recognition of such issues (Hotchkiss et al., 2012).

Among studies that focused on strategies to address the challenges of RHIS data use, those that targeted behavioural determinants were less likely to report success than those that targeted the other determinants, despite aligning closely with the behavioural challenges described. This suggests that addressing the behavioural determinants of RHIS data use may not be an effective means of improvement, particularly when targeted in isolation. Indeed, as Aqil et al. (2009) emphasise, the determinants of RHIS use do not operate independently, but rather are all interconnected, and these relationships must be considered when developing strategies to address them (Aqil et al., 2009). The lack of staff motivation to use RHIS, for example, may be related to insufficient supervision and feedback from the administration (Aqil et al., 2009). Tools to assess the performance of RHISs, such as PRISM, also emphasises a system-based approach to sustain improvement of RHIS data use (Thomas and Barry, 2018). As such, the connections between the behavioural, technical and organisational/environmental determinants of RHIS data use should be investigated further, and comprehensive strategies to address the underlying challenges in combination should be developed if strategies to mitigate these challenges are to be successful.

This review has several limitations. First, we included English language articles only, and our literature search was limited to peer-reviewed articles in just four databases. As such, potentially relevant studies located in books, non-peer reviewed articles, reports or grey literature were not considered. We also searched the literature using phrases that described health information systems in title and abstracts, thus limiting our reach to articles in which the phrases appeared selectively in the full text. Additionally, we searched RHISs and their variations, possibly missing relevant studies that may have used other key words to describe these data. Finally, given that two-thirds of the included studies were conducted in countries located in sub-Saharan Africa, the results of this systematic review may not be generalisable to other contexts. That said, many of the challenges described in these studies are common to all resource-poor settings, including those outside of Africa, and thus the findings are still likely to be valuable.

Conclusion

In this study, we identified common challenges that limit the use of RHIS data in LMICs at various stages of the health information system process, as well as summarised the effectiveness of strategies that have been implemented to improve data use and health service management. The findings of this review suggest that, despite substantial investments in RHISs in LMICs in recent years, a number of technical, behavioural and organisation/environmental challenges continue to impede the use of these data in practice and that strategies to address these challenges remain under-researched. Thus, there is a need for additional studies, including those evaluating the effectiveness of broader, more comprehensive strategies targeting multiple determinants of RHIS data use in combination, to better understand how to overcome the identified challenges. Such knowledge is vital if the full potential of RHIS data is to be achieved in LMICs.

Supplemental Material

sj-pdf-1-him-10.1177_1833358320928729 – Supplemental material for Understanding the challenges associated with the use of data from routine health information systems in low- and middle-income countries: A systematic review

Supplemental material, sj-pdf-1-him-10.1177_1833358320928729 for Understanding the challenges associated with the use of data from routine health information systems in low- and middle-income countries: A systematic review by Klesta Hoxha, Yuen W Hung, Bridget R Irwin and Karen A Grépin in Health Information Management Journal

Footnotes

Acknowledgements

We acknowledge that this work was conducted on the Haldimand Tract, traditional territory of the Neutral, Anishinaabe and Haudenosaunee peoples. We are grateful to the AXA Research Foundation for financial support for this project.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Yuen W Hung, PhD

Karen A Grépin, PhD

Supplemental material

Supplemental material for this article is available online.

References

Abajebel

Jira

Beyene

(2011) Utilization of health information system at district level in Jimma Zone Oromia Regional State, South West Ethiopia. Ethiopian Journal of Health Science 21: 65–76.

AbouZahr

Boerma

(2005) Health information systems: the foundations of public health. Bulletin of the World Health Organisation 83(8): 578–583.

Ahsan

Tahsina

Iqbal

, et al. (2017) Production and use of estimates for monitoring progress in the health sector: the case of Bangladesh. Global Health Action 10(1): 29–38.

Aqil

Lippeveld

Hozumi

(2009) PRISM framework : a paradigm shift for designing, strengthening and evaluating routine health information systems. Health Policy and Planning 24: 217–228.

Asah

Nielsen

Sæbø

(2017) Challenges for health indicators in developing countries: misconceptions and lack of population data. In: Choudrie

, et al. (eds) Information and Communication Technologies for Development. Cham: Springer International Publishing, pp. 593–604.

Ashton

Bennett

Yukich

, et al. (2017) Methodological considerations for use of routine health information system data to evaluate malaria program impact in an era of declining malaria transmission. American Journal of Tropical Medicine and Hygiene 97(Suppl 3): 46–57.

Askar

Ardakani

Majdzade

(2017) Bridging gaps in health information systems: a case study from Somaliland, Somalia. Eastern Mediterranean Health Journal 23(11): 764–773.

Awoonor-Williams

Bawah

Nyonator

, et al. (2013) The Ghana essential health interventions program: a plausibility trial of the impact of health systems strengthening on maternal & child survival. BMC Health Services Research 13(Suppl 2): S3.

Bernardi

(2017) Health information systems and accountability in Kenya: a structuration theory perspective. Journal of the Association for Information Systems 18(12): 931–958.

10.

Braa

Sahay

(2012) Improving quality and use of data through data-use workshops: Zanzibar, United Republic of Tanzania. Bulletin of the World Health Organisation 90: 379–384.

11.

Chikumba

Rasmussen

(2016) Management and use of health information in Malawi and Burkina Faso: the role of technology. In: Cunningham

Cunningham

(eds) IST-Africa Week Conference, Durban, South Africa, 11–13 May 2016, pp. 1–9.

12.

Cibulskis

Hiawalyer

(2002) Information systems for health sector monitoring in Papua New Guinea. Bulletin of the World Health Organisation 80(9): 752–758.

13.

Cibulskis

Posonai

Karel

(1995) Initial experience of using a knowledge based system for monitoring immunization services in Papua New Guinea. Journal of Tropical Medicine and Hygiene 98: 107–113.

14.

Covidence systematic review software (2013) Veritas Health Innovation, Melbourne, Australia. Available at: www.covidence.org (accessed 20 February 2019).

15.

Dagnew

Woreta

Shiferaw

(2018) Routine health information utilization and associated factors among health care professionals working at public health institution in North Gondar, Northwest Ethiopia. BMC Health Service Research 18: 685.

16.

Dehnavieh

Haghdoost

Khosravi

, et al. (2018) The district health information system (DHIS2): a literature review and meta-synthesis of its strengths and operational challenges based on the experiences of 11 countries. Health Information Management Journal 48(2): 62–75.

17.

Dehury

Chatterjee

(2018) Assessment of health management information system for monitoring of maternal health in Jaleswar Block of Balasore district, Odisha, India. Indian Journal of Public Health 62(4): 259–264.

18.

Driessen

Settle

Potenziani

, et al. (2015) Understanding and valuing the broader health system benefits of Uganda’s national human resources for health information system investment. Human Resources for Health 13(49): 1–9.

19.

Etamesor

Ottih

Ndalami Salihu

, et al. (2018) Data for decision making: using a dashboard to strengthen routine immunisation in Nigeria. BMJ Global Health 3: 1–7.

20.

Field

Usurup

Nathan

, et al. (2018) Contextual factors and health service performance from the perspective of the provincial health administrators in Papua New Guinea. Rural and Remote Health 18(4): 4484.

21.

Gimbel

Mwanza

Nisingizwe

, et al. (2017) Improving data quality across 3 sub-Saharan African countries using the consolidated framework for implementation research (CFIR): results from the African health initiative. BMC Health Services Research 17(Suppl 3): 828.

22.

Hotchkiss

Diana

Foreit

(2012) How Can Routine Health Information Systems Improve Health Systems Functioning in Low-Resource Settings? Assessing the Evidence Base. MEASURE Evaluation Special Report. Available at: https://www.measureevaluation.org/resources/publications/sr-11-65 (accessed 20 February 2019).

23.

Iguiñiz-Romero

Palomino

(2012) Data do count! collection and use of maternal mortality data in Peru, 1990–2005, and improvements since 2005. Reproductive Health Matters 20(39): 174–184.

24.

Jani

Araújo

, et al. (2006) Assessment of routine surveillance data as a tool to investigate measles outbreaks in Mozambique. BMC Infectious Diseases 6(1): 29.

25.

Kimaro

Twaakyondo

(2005) Analysing the hindrance to the use of information and technology for improving efficiency of health care delivery system in Tanzania. Tanzania Health Research Bulletin 7: 189–197.

26.

Koivu

Mavengere

Ruohonen

, et al. (2016) Exploring the information and ICT skills of health professionals in low- and middle-income countries. In: Brinda

et al. (eds) Stakeholders and Information Technology in Education. SaITE 2016. IFIP Advances in Information and Communication Technology, Cham: Springer, 493, pp. 152–162.

27.

Kumar

Gotz

Nutley

, et al. (2018) Research gaps in routine health information system design barriers to data quality and use in low- and middle-income countries: a literature review. International Journal of Health Planning Management 33: 1–9.

28.

Latifov

Sahay

(2013) Challenges in moving to “health information for action”: an infrastructural perspective from a case study in Tajikistan. Information Technology for Development 19(3): 215–229.

29.

Ledikwe

Reason

Burnett

, et al. (2013) Establishing a health information workforce: innovation for low- and middle-income countries. Human Resources for Health 11: 35.

30.

Leon

Brady

Kwamie

, et al. (2015) Routine health information system (RHIS) interventions to improve health systems management. Cochrane Database of Systematic Reviews 12: 1–18.

31.

Lippeveld

Sauerborn

Bodart

(2000) Design and Implementation of Health Information Systems. Geneva: World Health Organization.

32.

Mboera

LEG

Rumisha

Mayala

, et al. (2017) Malaria surveillance and use of evidence in planning and decision making in Kilosa district, Tanzania. Tanzania Journal of Health Research 19(3): 1–10.

33.

MEASURE Evaluation (2019) PRISM: performance of routine health information system management series. Available at: https://www.measureevaluation.org/resources/tools/health-information-systems/prism (accessed 20 February 2019).

34.

Mengesha

Steege

Zerihun Kea

, et al. (2018) Can mHealth improve timeliness and quality of health data collected and used by health extension workers in rural Southern Ethiopia? Journal of Public Health 40(2): 74–86.

35.

Moher

Liberati

Tetzlaff

, et al. (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Medicine 6(7): e1000097.

36.

Mosse

Sahay

(2003) Counter networks, communication and health information systems: a case study from Mozambique. IFIP Advances in Information and Communication Technology 126: 35–51.

37.

Moyo

Kaasbøll

Nielsen

, et al. (2016) The information transparency effects of introducing league tables in the health system in Malawi. The Electronic Journal of Information Systems in Developing Countries 75(2): 1–16.

38.

Mpofu

Semo

B-W

Grignon

, et al. (2014) Strengthening monitoring and evaluation (M&E) and building sustainable health information systems in resource limited countries: lessons learned from an M&E task-shifting initiative in Botswana. BMC Public Health 14: 1032.

39.

Mutale

Chintu

Amoroso

, et al. (2013) Improving health information systems for decision making across five sub-Saharan African countries: implementation strategies from the African health initiative. BMC Health Services Research 13(Suppl 2): S9.

40.

Mutemwa

(2006) HMIS and decision-making in Zambia: re-thinking information solutions for district health management in decentralized health systems. Health Policy and Planning 21(1): 40–52.

41.

Ndabarora

Chipps

Uys

2014 Systematic review of health data quality management and best practices at community and district levels in LMIC. Information Development 30(2): 103–120.

42.

Necochea

Tripathi

Kim

Y-M

, et al. (2015) Implementation of the standards-based management and recognition approach to quality improvement in maternal, newborn, and child health programs in low-resource countries. International Journal of Gynecology and Obstetrics 130: S17–S24.

43.

Nemser

Aung

Mushamba

, et al. (2018) Data-informed decision-making for life-saving commodities investments in Malawi: a qualitative case study. Malaria Medical Journal 30: 111–119.

44.

Nicol

Bradshaw

Uwimana-Nicol

, et al. (2017) Perceptions about data-informed decisions: an assessment of information-use in high HIV-prevalence settings in South Africa. BMC Health Services Research 17(Suppl 2): 765.

45.

Njuguna

Kamau

Muruka

(2017) Impact of free delivery policy on utilization of maternal health services in county referral hospitals in Kenya. BMC Health Services Research 17: 1–6.

46.

Nutley

Gnassou

Traore

, et al. (2014) Moving data off the shelf and into action: an intervention to improve data-informed decision making in Cote d’Ivoire. Global Health Action 7: 1–10.

47.

Nutley

Mcnabb

Salentine

(2013) Impact of a decision-support tool on decision making at the district level in Kenya. Health Research Policy and Systems 11(34): 1–8.

48.

Nwankwo

Sambo

(2018) Can training of health care workers improve data management practice in health management information systems: a case study of primary health care facilities in Kaduna state, Nigeria. Pan African Medical Journal 30(289): 1–8.

49.

Odhiambo-otieno

(2005) Evaluation of existing district health management information systems: a case study of the district health systems in Kenya. International Journal of Medical Informatics 74: 733–744.

50.

Omar

Pinto

Charimari

(1994) The district health information system and its potential in the management of district and rural hospitals. World Hospital 30(3): 15–20.

51.

Otieno

Kaseje

(2017) Perspectives on utilization of community based health information systems in Western Kenya. Pan African Medical Journal 27: 1–9.

52.

Pinchoff

Larsen

Renn

, et al. (2016) Targeting indoor residual spraying for malaria using epidemiological data: a case study of the Zambia experience. Malaria Journal 15(11): 1–6.

53.

Pinto

Soares de Freitas

Sant’Anna de Figueiredo

(2018) National information and population survey systems: selected contributions from the ministry of health and the IBGE for analysis of Brazilian state capitals over the past 30 years. Ciencia & Saude Coletiva 23(6): 1859–1870.

54.

Qazi

Ali

(2009) Pakistan’s health management information system: health managers’ perspectives. Journal of Pakistan Medical Association 59(1): 10–14.

55.

Qazi

Ali

(2011) Health management information system utilization in Pakistan: challenges, pitfalls and the way forward. BioScience Trends 5(6): 245–254.

56.

Qazi

Ali

Kuroiwa

(2008) The health management information system of Pakistan under devolution: health managers’ perceptions. BioScience Trends 2(2): 75–80.

57.

Radin

Abutu

Okwero

, et al. (2017) Confronting challenges in monitoring & evaluation: innovation in the context of the global plan towards the elimination of new HIV infections among children by 2015 and keeping their mothers alive. HHS Public Access 75(Suppl 1): 1–18.

58.

Ranasinghe

Chan

Yaralagadda

(2012) Information support for health management in regional Sri Lanka: health managers’ perspectives. Health Information Management Journal 41(3): 20–26.

59.

Rasmussen

(2018) Plans and “off-plan activities”: exploring the roles of data and situated action in health planning in Burkina Faso. Electronic Journal of Information Systems in Developing Countries 84(5): 1–9.

60.

Scott

Clarke

Grevendonk

, et al. (2017) Country immunization information system assessments (IISAs), in Kenya (2015) and Ghana (2016). Weekly Epidemiological Record 66(44): 1226–1229.

61.

Scott

Gilson

(2017) Exploring how different modes of governance act across health system levels to influence primary healthcare facility managers’ use of information in decision-making: experience from cape town, South Africa. International Journal for Equity in Health 16(159): 1–15.

62.

Seitio-Kgokgwe

Mashalla

Seloilwe

, et al. (2016) Utilization of the district health information software (DHIS) in Botswana: from paper to electronic based system. In: IST-Africa Week Conference, Durban, South Africa, 11–13 May 2016, pp. 1–10. IIMC.

63.

Seitio-Kgokgwe

Gauld

RDC

Hill

, et al. (2015) Development of the national health information systems in Botswana: pitfalls, prospects and lessons. Online Journal of Public Health Informatics 7(2): e210.

64.

Shiferaw

Zegeye

Assefa

, et al. (2017) Routine health information system utilization and factors associated thereof among health workers at government health institutions in East Gojjam Zone, Northwest Ethiopia. BMC Medical Informatics and Decision Making 17(1): 116.

65.

Soeung

Grundy

Kamara

, et al. (2007) Developments in immunization planning in Cambodia – rethinking the culture and organisation of national program planning. Rural and Remote Health 7(2): 630.

66.

Thomas

Barry

(2018) PRISM: performance of routine information system management series. Chapel Hill: MEASURE Evaluation.

67.

Thomas

Silvestre

Salentine

, et al. (2016) What systems are essential to achieving the sustainable development goals and what will it take to marshal them? Health Policy and Planning 31(10): 1445–1447.

68.

Tiwari

Kumar

Sherin

, et al. (2016) Standards, frameworks and practices in health management information and evaluation systems (HMIES) in Australia and India: lessons for future transition in India? Journal of Health Management 18(1): 70–83.

69.

Tshimanga

Peterson

Dlodlo

(1997) The health information system in the city of Bulawayo, Zimbabwe: how good is it? Central African Journal of Medicine 43(7): 195–199.

70.

Vidor

Fisher

Bordin

(2011) Use of health information systems in small municipalities in Southern Brazil. Reve Saude Publica 45(1): 1–7.

71.

Wagenaar

Gimbel

Hoek

, et al. (2015) Effects of a health information system data quality intervention on concordance in Mozambique: time-series analyses from 2009–2012. Population Health Metrics 13: 9.

72.

Wagenaar

Sherr

Fernandes

, et al. (2016) Using routine health information systems for well-designed health evaluations in low- and middle-income countries. Health Policy and Planning 31(1): 129–135.

73.

Ward

Mugenyi

Benke

, et al. (2017) Enhancing workforce capacity to improve vaccination data quality, Uganda. Emerging Infectious Diseases 23(13): 85–93.

74.

Wickremasinghe

Hashmi

Schellenberg

, et al. (2016) District decision-making for health in low-income settings: a systematic literature review. Health Policy and Planning 31 (Suppl 2): 12–24.

75.

World Bank (2019) World Bank Country and Lending Groups. Available at: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups (accessed 3 June 2020).

76.

World Health Organisation (2007) Strengthening Health Systems to Improve Health Outcomes: Who’s Framework for Action. Available at: https://www.who.int/healthsystems/strategy/everybodys_business.pdf (accessed 20 February 2019).

77.

Yazdi-Feyzabadi

Emami

Mehrolhassani

(2015) Health information system in primary health care: the challenges and barriers from local providers’ perspective of an area in Iran. International Journal of Preventive Medicine 6: 57.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.06 MB