REDCap mobile data collection: Using implementation science to explore the potential and pitfalls of a digital health tool in routine voluntary medical male circumcision outreach settings in Zimbabwe

Abstract

Background

Digital data collection tools improve data quality but are limited by connectivity. ZAZIC, a Zimbabwean consortium focused on scaling up male circumcision (MC) services, provides MC in outreach settings where both data quality and connectivity is poor. ZAZIC implemented REDCap Mobile app for data collection among roving ZAZIC MC nurses. To inform continued scale-up or discontinuation, this paper details if, how, and for whom REDCap improved data quality using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework.

Methods

Data were collected for this retrospective, cross-sectional study for nine months, from July 2019 to March 2020, before COVID-19 paused MC services. Data completeness was compared between paper- and REDCap-based tools and between two ZAZIC partners using two sample, one-tailed t-tests.

Results

REDCap reached all roving nurses who reported 26,904 MCs from 1773 submissions. REDCap effectiveness, as measured by data completeness, decreased from 89.2% in paper to 76.6% in REDCap app for Partner 1 (p < 0.001, 95% CI: −0.24, −0.12) but increased modestly from 86.2% to 90.3% in REDCap for Partner 2 (p = 0.05, 95% CI: -.007, 0.12). Adoption of REDCap was 100%; paper-based reporting concluded in October 2019. Implementation varied by partner and user. Maintenance appeared high.

Conclusion

Although initial transition from paper to REDCap showed mixed effectiveness, post-hoc analysis from service resumption found increased REDCap data completeness across partners, suggesting locally-led momentum for REDCap-based data collection. Staff training, consistent mentoring, and continued technical support appear critical for continued use of digital health tools for quality data collection in rural Zimbabwe and similar low connectivity settings.

Keywords

Mobile data collection REDCap implementation science Zimbabwe male circumcision digital health in routine settings

Introduction

Zimbabwe is among the highest HIV/AIDS prevalence countries in the world, with a prevalence of 12.9%.¹ As voluntary medical male circumcision (MC) is a safe, cost-effective method to reduce the spread of HIV,^2–4 the World Health Organization (WHO) set a target to circumcise 80% of men in 14 priority countries with high HIV burden,⁵ including Zimbabwe. Efforts by the Zimbabwe Ministry of Health and Child Care (MoHCC) to increase MC services are largely succeeding, as nearly 2 million men were circumcised between 2008 and 2020.^6,7 The ZAZIC consortium, comprised of the Zimbabwe Technical Assistance Training and Education Center for Health (Zim-TTECH), the International Training and Education Center for Health (I-TECH, Seattle WA, USA), the Zimbabwe Association of Church related Hospitals (ZACH) and the Zimbabwe Community Health Intervention Research Project (ZICHIRE), collaborates with Zimbabwe’s MoHCC in MC service delivery across the country, focusing on outreach sites and rural health center facilities in hard to reach regions. By 2021, ZAZIC performed over 500,000 MCs in 36 sites across 13 districts. ZAZIC sites are predominantly rural, with some peri-urban and a few urban settings, as described previously.^8,9

Large-scale healthcare programs, such as ZAZIC’s MC program, require efficient, high-quality data to maintain patient safety and to accurately track progress towards targets. However, in low- and middle-income countries (LMIC) or low-resource settings, timely, high-quality data collection, delivery, aggregation, and, ultimately, data use for program improvement is a challenge, especially with paper-based systems.^10–14 ZAZIC’s quality assurance efforts address several data quality concerns.¹⁵ Yet, MC teams continue to note persistent challenges to data quality, including the workload burden and data delays, that may compromise program quality.^9,16

REDCap is an open-source platform providing secure, web-based data capturing, management, and analytics.^17,18 Previous studies show that REDCap can improve data management in both research and routine healthcare contexts in LMIC settings, including those with limited connectivity.^19–21 An extension of the original REDCap, REDCap Mobile app, was launched in 2015 to allow for offline data capture; the features in the mobile application support its widespread use by over 800 global partners, including nearly 100 in sub-Saharan Africa.²² In Zimbabwe, where internet connectivity can be unpredictable, RED Mobile was selected for its expected performance in ZAZIC’s routine MC settings, many of which have connectivity challenges.

To provide more timely monitoring and evaluation (M&E) data for program improvement and target achievement, ZAZIC initiated a pilot program to use RED Mobile in settings where roving MC teams from larger health facilities performed MCs in surrounding schools, MC-focused camps, or rural health centers. Roving MC nurses entered daily MC program data from these outreach settings into REDCap Mobile, providing more timely access to M&E data. MC nurses also completed paper-based routine MoHCC MC forms at the facility in accordance with MoHCC policy.

In this retrospective, cross-sectional study, we applied the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework²³ to assess the impact of REDCap Mobile app data capture on the quality of data collected by roving MC teams during the first nine months of deployment, from July 2019 to March 2020 when MC service delivery was paused due to COVID-19 service delivery restrictions. The RE-AIM framework has been applied to assess and improve myriad, diverse global health programs;²⁴ however, to the knowledge, RE-AIM has not been specifically applied to assess data quality improvement initiatives in routine settings. In this paper, therefore, we employ RE-AIM in a novel context to frame an evaluation of REDCap Mobile for routine data quality improvement in rural MC outreach settings. Use of RE-AIM provides a more nuanced understanding of the successes and failures of REDCap Mobile and provides insights for continued MC data quality improvement. The findings from this assessment may be applicable to other LMIC settings where routine data collection is challenged by inconsistent connectivity and poor cellular networks.

Materials and methods

REDCap mobile app development and deployment

ZAZIC paper-based, roving M&E tools were used to create the electronic REDCap Mobile M&E form, aiming to collect the same data in the same order as the paper form to ease training, data entry, and data management. I-TECH led initial REDCap Mobile form development; ZAZIC-wide testing and feedback led to early improvements to the REDCap Mobile app. Like the paper-based M&E form, REDCap Mobile data collection included a series of close-ended questions on productivity outputs, staff scheduling, adverse events, follow-ups, and MC productivity. The partially open-ended questions were also included in the app to solicit more detailed comments regarding service delivery, follow-ups, and demand creation. Site and ZAZIC MC nurses and M&E teams were trained in app use via a half-day, in-person training that included practice-based activities on how to input REDCap Mobile app offline data, troubleshooting, and data syncing when connected to internet. REDCap Mobile app was piloted in 4 sites from July-September 2019 before expanding to all ZAZIC sites in October 2019; there was no further paper-based data collection among roving teams. Roving nurses were provided follow-up mentoring by ZAZIC M&E leads during routine site-visits and WhatsApp-based support. ZAZIC M&E leads were trained on data management, report generation, and data downloading using both REDCap and the offline REDCap Mobile app. Data from the REDCap Mobile app was stored within a REDCap database and extracted quarterly by a ZAZIC program administrator for routine analysis.

Study setting

REDCap Mobile was implemented in all 36 ZAZIC sites from July 2019 to March 2020 using the REDCap Mobile app in offline mode, syncing daily or weekly when internet access was available.

Study population

The study population was comprised of ZACH and ZICHIRE roving nurses implementing MCs alongside MoHCC MC teams in outreach settings.

Data collection and analysis using the RE-AIM framework (Table 1)

Reach

Reach, or the number of those who participated in the intervention, was defined as the total number of REDCap Mobile roving reports submitted per month, per partner, and presented in descriptive statistics.

Effectiveness

Effectiveness, or the impact of REDCap Mobile on data completeness, was defined as completeness of roving team data in REDCap Mobile compared to previous paper-based data forms. Completeness, a marker for data quality,¹⁴ was calculated as a proportion: the number of non-missing variable fields divided by all variable fields per observation (roving event). Two sample tests of differences in proportions of data completeness, stratified by ZAZIC partner, were reported using RStudio.²⁵

Adoption

Adoption, or the absolute number and proportion of those willing to initiate REDCap Mobile app, was measured at the ZAZIC program level by determining the number and proportion of ZAZIC nurses, by partner, submitting data via REDCap Mobile from July 2019 to March 2020.

Implementation

Implementation, including both intervention fidelity and adaptation, was measured by the average number of reports nurses submitted each month throughout the observation period as well as by ZAZIC-led REDCap Mobile form adaptations to improve data collection.

Maintenance

COVID-19 restrictions resulted in a complete pause in MC service delivery in March 2020. Although maintenance would typically measure whether and how an intervention becomes institutionalized or part of routine practice, we assess ZAZIC’s intention to continue REDCap Mobile via internal partner communication, user feedback, and ZAZIC ownership of REDCap Mobile form optimization as proxy measures for maintenance.

Results

Reach

ZAZIC partners submitted 1773 reports via REDCap Mobile from July 2019 to March 2020 (Figure 1). Of these reports, 958 (54%) reports were submitted by Partner 1 and 815 (46%) reports were submitted by Partner 2. Thirty-five nurses submitted forms and recorded 26,904 male circumcisions: 19,090 from Partner 1 and 10,814 from Partner 2. The highest REDCap Mobile report yield was from November 2019 with 406 reports submitted: 247 from Partner 1 and 159 from Partner 2. Notable decreases in REDCap Mobile report submission by both partners occurred in December 2019 (festive season) and March 2020 (beginning of COVID-19 related restrictions).

Figure 1.

REDCap mobile reports submitted per month by program partner, July 2019 to March 2020.

Effectiveness

Using REDCap Mobile app, average data completeness rate among Partner 1 forms (Figure 2A) decreased significantly from 89.2% in paper forms (N = 161) to 76.6% completeness in REDCap Mobile app (N = 960), (p < 0.001, 95% CI: −0.24, −0.11). Average form completion for Partner 2 (Figure 2B) increased modestly from 86.2% in paper (N = 170) to 90.3% in REDCap Mobile (N = 813), (p = 0.05, 95% CI: -.007, 0.12).

Figure 2.

Average completion rate of paper forms and REDCap mobile forms by partner 1 and partner 2. Error bars indicate 95% confidence interval.

Adoption

Once implemented in July 2019, ZAZIC required all roving nurses to use REDCap Mobile for data collection, resulting in 100% adoption by ZAZIC program staff and discontinuation of paper-based reporting by October 2019.

Implementation

REDCap Mobile implementation was measured by the average number of reports submitted by nurses each month (Figure 3). During this period, 17 nurses from Partner 1 (Figure 3A) submitted a total of 958 reports from July 2019 to March 2020, an average of 11 REDCap Mobile reports per month. From Partner 2, 18 nurses submitted a total of 815 reports from July 2019 to March 2020, an average of 12 reports per month (Figure 3B). There was noticeable variation in report submission over time.

Figure 3.

REDCap mobile form submission among MC roving team nurses, July 2019 to March 2020.

Maintenance

Several programmatic activities suggest intention to maintain REDCap Mobile data collection beyond these initial nine months. First, in response to REDCap Mobile data quality weaknesses, ZAZIC led informal user interviews and gathered program team feedback on the app, providing support and refresher training tips via WhatsApp. Second, during the COVID-19 related pause, independent ZAZIC partner investigation into sub-optimal REDCap Mobile performance and form completion rates identified one nurse who submitted 50% of all Partner 1 REDCap Mobile using a previous REDCap Mobile app version with outdated form fields, decreasing the Partner 1 average data completion to 60.3%. In response, ZAZIC partner M&E teams completed another set of REDCap Mobile app improvements, including: 1) making all fields within the app form mandatory; 2) offsetting critical fields with attention-grabbing header fields; and 3) closely monitoring to ensure all users were submitting data using the most recent REDCap Mobile form version. These changes were rolled out by ZAZIC teams without help from I-TECH from October 2020 when service delivery resumed.

Discussion

Use of the REDCap Mobile app to improve data quality among roving nurses in a routine MC setting in Zimbabwe did not initially find improvements. However, application of the RE-AIM framework helped explain reasons for REDCap Mobile successes and failures and, therefore, contributed to additional app changes that ultimately resulted in improved data completeness over paper-based forms. These results support the utility of the RE-AIM approach and suggest that mobile apps may improve data quality in routine LMICs healthcare settings with limited internet access. Several key lessons learned merit discussion.

First, this assessment supports application of the RE-AIM framework as a practical tool for routine program evaluation,²⁶ including for interventions in sub-Saharan Africa.^27–28 In this evaluation of a data quality improvement intervention, RE-AIM helped demonstrate reach, effectiveness, and adoption while identifying facilitators to REDCap Mobile implementation that, in turn, led to improvements in the REDCap Mobile app. As the teams noted subsequent increases in data quality, they affirmed their continued intention to maintain REDCap Mobile data collection. During the COVID-19 restriction-induced pause, ZAZIC reviewed data, iterated app improvements, and addressed issues in REDCap Mobile usability by further streamlining data collection fields. Once service delivery resumed, ZAZIC Partner teams led quarterly REDCap Mobile analysis and used the results to more closely supervise roving teams’ use of the REDCap Mobile app. Post-hoc analysis (excluding submissions from one Partner 1 nurse using the previous REDCap mobile form) from October 2020 to February 2021 (Figure 4), suggests that these locally-led changes resulted in additional improvement of REDCap Mobile data completeness for both partners. Assumption of local ownership and management, in accordance with the final component of the RE-AIM framework, suggests that ZAZIC partners may maintain REDCap Mobile as a data collection tool.

Figure 4.

Post-hoc analysis of average data completeness before (July 2019 to March 2020) and after additional improvements (October 2020-February 2021).

Second, several issues challenged the initial roll-out of REDCap Mobile app and may provide insights for other potential users of REDCap Mobile. First, data fields were not marked mandatory in REDCap Mobile, leading to nurses leaving blanks among fields required for quality improvement. Second, even with required fields, ZAZIC roving staff still required training and consistent mentoring to complete fields with a “no” rather than leave fields blank as was practiced routinely with paper forms. Lastly, consistent with other reviews of digital health challenges,^29–32 factors such as fluctuations in electricity, network access, tablet functionality, digital literacy and staff attrition affected REDCap Mobile report submission. These persistent challenges will likely extend beyond this roll-out period.

Table 1.

Summary of RE-AIM dimensions aligned to REDCap mobile use for ZAZIC program M&E data.

RE-AIM Dimension	REDCap Mobile Outcome Measure	Level
Reach	Number of REDCap Mobile roving reports submitted per month, by partner	Individual
Effectiveness	Data completeness of nurse forms in REDCap Mobile compared to completeness in traditional paper-based forms	Individual
Adoption	Number and proportion of nurses submitting data using REDCap Mobile	Institutional
Implementation	Average proportion of monthly reports by nurse	Institutional
Maintenance	ZAZIC’s intention to continue REDCap Mobile use once MC procedures continued	Institutional

As MC roving services continue, several recommendations should be considered to improve REDCap Mobile replication, scale-up, and sustainability that are applicable to other settings. First, although REDCap Mobile data collection will likely improve M&E of MC outreach activities, consistent practice-based training on REDCap Mobile platform will be needed to increase user competence and confidence, including refreshers on data collection, form submission, and troubleshooting syncing issues. Second, ZAZIC Partner teams should regularly review REDCap Mobile data for quality to quickly identify opportunities for increased supervision to help raise the likelihood of sustained success. Third, ZAZIC REDCap Mobile managers need to archive previous versions of the app when updated versions are available and verify that all users adopt the most recent app version. Lastly, and perhaps most importantly, while REDCap Mobile is locally managed, external technical support is still required to identify and resolve some issues in system optimization. A more robust, virtual REDCap Mobile forum with heightened monitoring from LMIC experts is needed to both document challenges and leverage LMIC user crowdsourcing to identify viable solutions. In turn, this would help strengthen local REDCap Mobile technical capacity and oversight, promoting sustainability of open-source tools such as REDCap Mobile.

Limitations

This study has several limitations. First, the observation period for REDCap Mobile was interrupted by COVID-19, limiting the study period to nine months of significantly decreased MC productivity due to reductions in service delivery. COVID-19 also shifted HIV prevention resources away from MC to HIV treatment and care, reducing MC data quality efforts as well. Additionally, COVID-19 reduced access to critical REDCap Mobile training and mentoring, highlighting a potential limitation for digital tools. Furthermore, as the team transitioned to an electronic system, data entry for paper forms became more inconsistent, creating a potential bias in data completeness assessment for paper forms. Moreover, one individual’s submissions heavily skewed REDCap Mobile data and were omitted from post-hoc analysis. This exclusion reduces the bias from one individual but should not diminish the need for vigilance in ensuring all users update and refresh the app for optimization. Lastly, REDCap Mobile was chosen as the platform due to I-TECH experience and expertise; however, it is possible that other digital data collection tools may be better suited or more sustainable for the Zimbabwean rural context.^33–34

Conclusion

Overall, use of the RE-AIM framework helped understand how, where, and why REDCap Mobile was beneficial to improve routine data quality in ZAZIC’s MC outreach settings. REDCap Mobile’s secure platform and ability to collect data offline also increased the tool’s utility, reducing connectivity and confidentiality concerns that may be similar in other healthcare contexts. These findings suggest that use of REDCap Mobile app for routine data collection should be maintained in the ZAZIC MC program and considered for replication in other LMIC rural or outreach settings.

Declarations

Footnotes

Acknowledgements

We would like to thank the ZAZIC consortium partners for their dedication to quality data collection for VMMC program monitoring. We would like to thank the roving nurses for their consistent efforts to provide quality care to VMMC patients.

Author contributions

VT led study literature review and data analysis. CF, FG, VM, LM and PM conceived the study and supported REDCap training. JH and TM led data collection. OR and PC led REDCap system adaptation. ZR developed the initial REDCap forms. TM, VSC, BMC, SB, and MH provided management support. SX and GN provided MoHCC oversight. VT, CF, and BM wrote the first draft of the manuscript. All authors reviewed and edited the manuscript and approved the final version of the manuscript

Data availability

The individual-level routine data aggregated for the ZAZIC study is owned by the Ministry of Health of Child Care (MoHCC) and are not publicly available due to MoHCC data restrictions and confidentiality protections. However, data may be made available for researchers who sign a data sharing agreement and abide by strict confidentiality protocols. Interested researchers may contact Jane Edelson jedelson@uw.edu, Regulatory Specialist at University of Washington, for access requests

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Centers for Disease Control and Prevention, (grant number NU2GGH002116).

Ethical approval

The ethics committee of the University of Washington (UW) internal review board (IRB) instructs UW researchers to complete the Human Subjects Research Worksheet to self-determine if the proposed project meets the definition of research and therefore requires IRB review. This study using secondary data, “RedCap for improved M&E of voluntary medical male circumcision outreach” did not meet the definition of research, and therefore did not require UW IRB review.

Funding

REDCap Mobile data collection, about which this manuscript was written, is part of routine ZAZIC MC program delivery, funded by the US Centers for Disease Control and Prevention under award, NU2GGH002116, Barnhart (PI).

Guarantor

ORCID iD

Caryl Feldacker

References

Zimbabwe Ministry of Health and Child Care. Zimbabwe Population-Based HIV Impact Assessment (ZIMPHIA) 2020. 2020.

Auvert

Taljaard

Lagarde

, et al. Randomized, controlled intervention trial of male circumcision for reduction of HIV infection risk: the ANRS 1265 trial. PLoS Med 2005; 2: e298.

Bailey

Moses

Parker

, et al. Male circumcision for HIV prevention in young men in Kisumu, Kenya: a randomised controlled trial. Lancet 2007; 369: 643–656.

World Health Organization. Considerations for implementing models for optimizing the volume and efficiency of male circumcision services2010 [cited 2014 June 1]; Field testing edition: February 2010 Available from: http://www.malecircumcision.org/programs/documents/mc_MOVE_2010_web.pdf.

WHO/UNAIDS. UNAIDS And WHO progress brief: voluntary medical male circumcision, February 2021. Geneva: WHO, 2021.

Zimbabwe Ministry of Health and Child Care. Sustainability Transition Implementation Plan 2019–2021: VOLUNTARY MEDICAL MALE CIRCUMCISION. 2018.

President's Emergency Plan for AIDS Relief. ZIMBABWE Country Operational Plan (COP) 2020; Strategic Direction Summary March 20202020. Available from: https://www.state.gov/wp-content/uploads/2020/07/COP-2020-Zimbabwe-SDS-FINAL-with-Appendix-C.pdf.

Feldacker

Makunike-Chikwinya

Holec

, et al. Implementing voluntary medical male circumcision using an innovative, integrated, health systems approach: experiences from 21 districts in Zimbabwe. Glob Health Action 2018; 11: 1414997.

Feldacker

Murenje

Makunike-Chikwinya

, et al. Balancing competing priorities: quantity versus quality within a routine, voluntary medical male circumcision program operating at scale in Zimbabwe. PloS one 2020; 15: e0240425.

10.

Rumisha

Lyimo

Mremi

, et al. Data quality of the routine health management information system at the primary healthcare facility and district levels in Tanzania. BMC Med Inform Decis Mak 2020; 20: 1–22.

11.

Lemma

Janson

Persson

L-Å

, et al. Improving quality and use of routine health information system data in low-and middle-income countries: a scoping review. PloS one 2020; 15: e0239683.

12.

Were

Kariuki

Chepng'Eno

, et al. Leapfrogging paper-based records using handheld technology: experience from Western Kenya. Stud Health Technol Inform 2010; 160: 525–529.

13.

Abiy

Gashu

Asemaw

, et al.

A comparison of electronic medical record data to paper records in antiretroviral therapy clinic in Ethiopia: what is affecting the quality of the data?

Online J Public Health Inform 2018; 10(2): e212.

14.

Zeleke

Worku

Demissie

, et al. Evaluation of electronic and paper-pen data capturing tools for data quality in a public health survey in a health and demographic surveillance site, Ethiopia: randomized controlled crossover health care information technology evaluation. JMIR Mhealth Uhealth 2019; 7: e10995.

15.

Xiao

Bochner

Makunike

, et al. Challenges in data quality: the influence of data quality assessments on data availability and completeness in a voluntary medical male circumcision programme in Zimbabwe. BMJ Open 2017; 7: e013562.

16.

Marongwe

Gonouya

Madoda

, et al.

Trust but verify: is there a role for active surveillance in monitoring adverse events in Zimbabwe’s large-scale male circumcision program?

PloS one 2019; 14: e0218137.

17.

Harris

Taylor

Thielke

, et al. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009; 42: 377–381.

18.

Harvey LA. REDCap: web-based software for all types of data storage and collection. Spinal Cord. 2018; 56(7): 625.

19.

Ndlovu

Maucob

Groverc

, et al. Evaluation of REDCap as an alternative data management and quality improvement tool for cutaneous lymphoma care in Botswana. 2019.

20.

Rivera

Rosas-Romero

Castro

, et al. Implementing a REDCap-based research data collection system for mental health. Revista Colombiana de Psiquiatría (English ed). 2021.

21.

Odukoya

Nenrot

Adelabu

, et al. Application of the research electronic data capture (REDCap) system in a low-and middle income country–experiences, lessons, and challenges. Health Technol 2021; 11(6): 1297–1304.

22.

Harris

Delacqua

Taylor

, et al. The REDCap mobile application: a data collection platform for research in regions or situations with internet scarcity. JAMIA Open 2021 Sep 13; 4: ooab078. PMID: 34527889; PMCID: PMC8435658.

23.

Glasgow

Vogt

Boles

. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999; 89: 1322–1327.

24.

Glasgow

Harden

Gaglio

, et al. RE-AIM Planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health 2019; 7: 64.

25.

R Core Team R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2020. URL https://www.R-project.org/.

26.

D'Lima

Soukup

Hull

. Evaluating the application of the RE-AIM planning and evaluation framework: an updated systematic review and exploration of pragmatic application. Front Public Health 2022; 9: 755738.

27.

Iwelunmor

Nwaozuru

Obiezu-umeh

, et al. Is it time to RE-AIM: A systematic-review of economic empowerment HIV prevention programs for adolescent girls and young-women in sub-Saharan Africa using the RE-AIM framework. 2019. Paper presented at the APHA's 2019 Annual Meeting and Expo (Nov. 2-Nov. 6).

28.

Onono

Abdi

Opondo

, et al. Using the RE–AIM framework to evaluate the implementation of integrated community case management in Kenya. Acta Paediatr 2018; 107: 53–62.

29.

Aranda-Jan

, et al. Systematic review on what works, what does not work and why of implementation of mobile health (mHealth) projects in Africa. BMC Public Health 2014; 14: 188.

30.

Hall

Fottrell

Wilkinson

, et al.

Assessing the impact of mHealth interventions in low-and middle-income countries–what has been shown to work?

Glob Health Action 2014; 7: 25606.

31.

Marcolino

Oliveira

D'Agostino

, et al. The impact of mHealth interventions: systematic review of systematic reviews. JMIR Mhealth Uhealth 2018; 6: e23.

32.

McCool

Dobson

Whittaker

, et al. Mobile health (mHealth) in low-and middle-income countries. Annu Rev Public Health 2022; 43: 525-539.

33.

Sezgin

. Can we use commercial mobile apps instead of research mobile apps in healthcare research? Front Public Health 2021; 9: 685439.

34.

Grundy

. A review of the quality and impact of mobile health apps. Annu Rev Public Health 2022; 43: 117–134.