Characteristics and efficacy of digital health education: An overview of systematic reviews

Abstract

Objective:

The primary aim of this overview was to synthesise results from studies including digital education and its effect on knowledge or learning outcomes, student satisfaction, student enrolment, attendance rate, course completion rate, clinical practice, health outcomes for patients and cost-effectiveness in health-care education. A secondary aim was to report on successful instructional design strategies, and barriers or contextual factors influencing the effectiveness of online learning course delivery in healthcare education.

Method:

We conducted an overview of systematic reviews (SRs) for digital education interventions delivered to health-care students and practitioners.

Results:

We scanned 848 titles, reviewed 247 abstracts and assessed 49 full-text articles against pre-determined inclusion and exclusion criteria. This overview includes data collected from 31,730 participants across 16 SRs. The quality of evidence included in the SRs ranged from very low (n = 2), low (n = 6) to moderate (n = 8). The best available SRs were of moderate quality (7.4 of 11 AMSTAR). SR authors did not report other teaching methods as being superior to digital learning. In most cases (n = 9), digital education when used in addition to traditional methods augmented knowledge acquisition. Other SRs (n = 7) did not show statistically significant differences across interventions including digital education as a replacement, or additive resource to traditional intervention.

Conclusion:

Student enrolment, attendance rates, course completion rates, cost-effectiveness and changes in clinical outcomes for patients are underreported in the existing evidence. Although the quality and quantity of data are limited, evidence-based instructional design for digital education is becoming more possible, especially as educators establish learning activities that track to learning objectives for knowledge acquisition in health care.

Keywords

Courses digital health education health care online review systematic

Background

Technological advancement has improved access to information and increased the speed and efficiency of communication in education. Beyond this basic communication and information flow, educational institutions have developed a mix of online and in-person (blended) learning environments, digitalised lectures with in-person workshop sessions (flipped classrooms), webinars, chat forums and live digital discussions, alongside new automated learning applications.

Educational offerings in the current digital era comprise at one extreme, independent, automated, self-guided digital courses (downloadable or available online) and at the other end individual attention through private tutorial, face-to-face sessions. Educational media and approaches vary considerably in terms of accessibility, individualised attention, scalability, flexibility, responsiveness, replicability and cost. Often, their relative impact on health education in terms of learning outcomes remains unclear (Consorti et al., 2012; Curran and Fleet, 2005).

Continuing medical education (CME) among health-care professionals (HCPs) in general leads to the acquisition and retention of knowledge, attitudes, skills, behaviours and clinical outcomes (Marinopoulos et al., 2007). Medical conferences, workshops or traditional in-person classes often show an increase in physician knowledge and in some instances, health outcomes. However, the degree to which didactic learning approaches, or non-experiential learning translates into better health-care practice is more uncertain (Davis et al., 1999). Digital media for health-care education, especially for working professionals, provide a potentially convenient and cost-effective method to deliver training to improve knowledge and patient outcomes. This systematic overview, therefore, seeks to understand the type of interventions and outcomes that may be expected to improve not only HCP student knowledge, but also health-care practice.

The primary aim of the overview was to synthesise results from studies of digital education in terms of knowledge or learning outcomes, student satisfaction, student enrolment, attendance rate, course completion rate, clinical practice, health outcomes for patients and cost-effectiveness. A secondary aim of this overview was to report effective instructional design strategies and list barriers or contextual factors that influence the effectiveness of online learning course delivery in health-care education. In addition, we sought to explore different types of student- or teacher-specific characteristics that might improve either the quality, satisfaction or performance in digital learning in health care.

Methods

We conducted an overview of systematic reviews (SRs) for digital education interventions delivered to students and practitioners of health care.

Inclusion criteria

SRs synthesising data from at least one randomised controlled trial (RCT) of digital health education intervention were included. To qualify as an SR, each article needed to satisfy five Oxman Criteria (Oxman, 1994), including a statement of a replicable search method, whether the authors adequately attempted to retrieve all relevant data, whether data were collected in a systematic way, whether the authors analysed and presented results appropriately, and whether the authors considered sources of bias and quality of evidence.

Included participants were adult (18 years or older) students of health-care subjects, or HCPs who received education or training through a digital medium for the medical treatment of humans or animals. Only SRs conducted with at least one 100% online, digital intervention to improve knowledge or services in health care of humans or animals were included. Part-time, short units and full courses were included provided the descriptions of the intervention included subject details and a description of the medium. Multimedia classes including online video, text, or audio files via computers or personal devices were included. Studies using comparison groups that did not include 100% digital-based education–including waitlists, in-person, or blended learning styles–were included as adequate comparisons.

Exclusion criteria

Case series, business reports, market analyses or opinion articles, or SRs that sought to address health behaviour or patient education (as opposed to students of a health-related field or HCPs) were excluded from the overview. Interventions that did not include a digital medium (such as paper-based or using textbooks) were not included. Also, specialised surgical simulations of procedures were excluded.

Search strategy

An information specialist (N.R.) designed and conducted a search of the following online databases: CINAHL, Cochrane Database of Systematic Reviews, Database of Abstract of Reviews of Effects, Embase, Epub Ahead of Print, In-Process & Other Non-Indexed Citations, Ovid MEDLINE(R) Daily and Ovid MEDLINE, ERIC and British Education Index, Science Citation Index and Social Science Citation Index and PROSPERO for SR protocols. Additional online databases were searched for relative studies including The Campbell Collaboration Online Library, Best Evidence Medical and Health Professional Education, the JISC e-learning programme and Google Scholar. For a comprehensive list of free-text terms and search dates, please see online Appendix A. The search for evidence was originally performed in December 2016, and an updated search was performed in June 2017.

Outcome measures

This overview collected data from research that sought to measure change or differences in the aspects of health education in the context of Kirkpatrick’s (1998) four levels of evaluation: learning outcomes (knowledge or skills), student satisfaction, student enrolment, attendance rate, course completion rate, clinical practice (of HCPs only), health outcomes for patients (of HCPs) and cost-effectiveness (including administration, logistics for teacher or student).

In addition, quantitative or qualitative data that described any of the following outcomes were collected: strategies of course design or delivery that increase student learning, satisfaction, or retention rates in health-care education, barriers or contextual factors influence effectiveness of online learning course delivery in health-care education and type of student or teacher-specific characteristics (i.e. personality, learning style, socio-demographics) thought to improve either the quality, satisfaction or performance of online learning for health-care education.

Data management and analysis

The first reviewer (M.M.) scanned all the titles and abstracts for exclusion based on the pre-determined inclusion and exclusion criteria. See Figure 1 for the article flow diagram. Full-text articles (n = 49) were assessed by two reviewers (E.S. and H.O.) with consultation by an additional reviewer (M.M.) to resolve any differences. The quality of each SR was assessed by two reviewers (E.S. and H.O.) using the 11-point AMSTAR tool (Shea et al., 2009). The quality of evidence contained within each SR was assessed by two reviewers (E.S. and H.O.) using the GRADE tool as a guide (Guyatt et al., 2011).

Figure 1.

Article flow diagram.

Subgroup analyses

We planned for subgroup analyses in the population, intervention and comparators of SRs. Lack of consistent reporting on interventions prevented a subgroup analysis for the difference in short versus long online courses. Excessive heterogeneity in data reporting for comparisons across SRs prevented a sensible meta-analysis of effect size differences between different comparators.

Qualitative analysis

One reviewer (M.M.) analysed the qualitative data for keywords, themes and characteristics according to our primary and secondary research questions. Two reviewers (E.S. and H.O.) verified the selection of supporting qualitative data and its analysis. NVivo 10 for Mac (QSR International) software was used to code qualitative data from which we were able to extrapolate themes.

Results

Search results

We scanned 848 titles and 247 abstracts based on the pre-determined inclusion and exclusion criteria, and assessed 49 full-text articles (see Figure 1). Of these, we excluded 33 with reasons (see Table 1). The primary reason for exclusion was that studies did not satisfy the criteria for a SR (n = 15), or the intervention did not match our set criteria (n = 8). One article focused on Internet-based interventions only and did not provide an adequate comparison to non-digital based interventions (Cook et al., 2010b); however, it includes relevant information on instructional design for health-care education. Therefore, we included findings from this SR for discussion purposes only. In the full analysis of this overview, we consider data from 16 SRs.

Table 1.

Summary of included reviews.

First author, year	Population	Taught subject	Type of intervention	Primary outcomes	Comparisons	Meta-analysis	Direction of results
Al-Jewair et al. (2009)	Undergraduate or post-graduate, pre-clinical, clinical students, and educators of orthodontics	Orthodontics	Computer-assisted learning*	Efficacy of computer-assisted learning (knowledge acquisition, learning efficiency, learner and education attitudes, cost and labour)	Traditional methods	x	Favours digital education (+)
Consorti et al. (2012)	Undergraduate and graduate medical education	General Medicine	Virtual patients and blended learning	Clinical reasoning (including history taking and differential diagnosis)	Traditional methods	x	Favours digital education (+)
Cook et al. (2010a)	Health professionals in training or practice	Specialised and General Medicine (i.e. cardiovascular, ambulatory care, examination of eye and ear, back pain, psychotherapy, women’s health, health screening, hygiene)	Internet-based learning	Learning efficiency (time on task to acquire knowledge)	Traditional methods and other computer-assisted learning intervention	x	No difference (o)
Du et al. (2013)	Nursing students and practitioners	Nursing	Internet-based and blended learning	Knowledge acquisition and skill performance	Traditional methods and no intervention		Favours digital education (+)
Feng et al. (2013)	Undergraduate, graduate students and professionals in nursing and medicine	General Medicine and Nursing	Computer-assisted learning	Knowledge acquisition and skill performance	Traditional methods and no intervention	x	No difference in knowledge of students and clinicians (o); slightly favours digital education for student performance only (+)
George et al. (2014)	Undergraduate and graduate students of medicine dentistry, nursing, physical therapy, pharmacy	General Medicine and Allied Health (including dentistry, midwifery, physical therapy, diagnostic technology, pharmacy)	Internet-based and blended learning	Knowledge acquisition, skill performance, learner satisfaction and attitude	Traditional methods and no intervention		No difference (o)
Härkänen et al. (2016)	Nurses	Medication administration skills in hospital	Internet-based learning, computer-based learning and traditional methods	Increasing medical administration and safety skills	Traditional methods and no intervention	x	Favours digital education (+)
Jayakumar et al. (2015)	Medical, dental students, residents and practising surgeons	Surgical education	Internet-based and blended learning	Identify current modalities of e-learning~	Traditional methods and no intervention		Favours digital education (+)
Lahti et al. (2014)	Nurses and nursing students	Nursing skills, hygiene, ergonomics	Computer-assisted and Internet-based learning	Knowledge acquisition and skill performance	Traditional methods, blended learning and no intervention	x	No difference (o)
Lam-Antoniades et al. (2009)	Health professionals in practice	Continuing education (clinical guidelines, ergonomics, patient management, nursing skills, supplementation)	Computer-assisted and Internet-based learning	Knowledge acquisition and skill performance	Traditional methods and no intervention		Favours digital education (+)
Maertens et al. (2016)	Medical students or surgeons in training or practice	Surgical training (including general, trauma, orthopaedic, paediatrics, obstetrics, nursing, plastics, cardiovascular)	Computer-assisted learning	Knowledge acquisition and skill performance	Traditional methods and no intervention		Favours digital education (+)
McCutcheon et al. (2015)	Undergraduate nurses	Nursing skill, communication and epidemiology	Internet-based and blended learning	Knowledge acquisition and skill performance	Traditional methods, blended learning and no intervention		No difference (o)
Rasmussen et al. (2014)	Undergraduate students	Medicine, Dentistry, Nursing, Psychology, Physical Therapy	Computer-assisted, Internet-based and blended learning	Knowledge acquisition, skill performance, learner satisfaction and attitude	Traditional methods and blended learning		Favours digital education (+)
Santos et al. (2016)	Predoctoral dental students	Oral radiology	Computer-assisted learning	Knowledge acquisition, skill performance and learner satisfaction	Traditional methods and no intervention		No difference (o)
Sinclair et al. (2016)	Health professionals in practice	Patient screening, medical assessments, emergency preparedness	Computer-assisted learning	Effect on health-care professional behaviour and patient outcomes	Traditional methods		No difference (o)
Webb et al. (2017)	Pre-registration nurses	Nursing	Computer-assisted, Internet-based learning and blended learning	Deliver and efficacy of pre-registration nurse education	Traditional methods		No difference (o)

Limitations and comments.
Review ID	SR Author comments of results favouring digital education	SR author comments with no difference/no significant results	SR Author comments of results favouring traditional methods
Al-Jewair et al. (2009)	Confounders including level of motivation and prior knowledge, uncertain about knowledge retention
Consorti et al. (2012)	Unable to discern best practice in digital formats due to poor description of studies, uncertain about knowledge retention
Cook et al. (2010a)		Poor description of interventions and outcomes
Du et al. (2013)	Learner preferences for flexibility, independence and time efficiency and some used interactive web-based mediums
Feng et al. (2013)		Lack of computer skills for some participants, learning needs of a clinician are different than a student (i.e. problem solving of their patient, application of knowledge for clinician)
George et al. (2014)		High risk of bias, authors suggest that online learning is possibly superior to traditional learning, as studies show mixed results (i.e. 12 of 50 studies favoured online learning; 27 found no significant differences; 11 studies did not test for differences), subjective outcome measures primarily based on survey recall questionnaires
Härkänen et al. (2016)		Five of 14 studies used digital learning as an additive (n = 1) or alternative (n = 4) to traditional classroom learning; the most effective of these studies was the blended learning model see Fig 2 of SR
Jayakumar et al. (2015)	Small samples in 18 studies including data from six randomised-controlled trials, unclear about pre-existing knowledge, slow Internet connection and technical issues were listed as the largest impediments, one study at a 54% completion rate (Cochran et al., p. 1,150 of SR)
Lahti et al. (2014)		All studies were randomised-controlled trials, only 4 of 11 used in meta-analysis due to poor reporting of numerical data
Lam-Antoniades et al. (2009)	Effects of multi-component (blended learning) are longlasting (up to 12 months), flat-text information on screen is of limited value and to be limited, risk of publication bias
Maertens et al. (2016)	Limited evidence to support digital education over traditional methods, stronger evidence for comparison with no interventions, serves as an adjunct and can assist in cognitive phase of skill acquisition
McCutcheon et al. (2015)			Mixed results, where digital education is ‘no less effective than traditional means’ in all but three studies (where traditional method was favoured), students felt ‘disadvantaged’ having only online learning, those in traditional method reported wanting more online learning
Rasmussen et al. (2014)	Findings suggest that ‘offline eLearning is at least equivalent, possibly superior to traditional learning, favourable’ because it offers a ‘more convenient and more cost-effective, alternative to facilitate competency development and the training of health-care professionals around the globe’
Santos et al. (2016)		Digital education may be more properly used as a support to traditional education and not as a substitute for it, students are positive about digital learning material, may be cost and time-effective for instructors
Sinclair et al. (2016)		No papers reported the effectiveness of digital education
Webb et al. (2017)		Many studies highlight concerns for staff training, support and perceived external pressures, lack of technical competence among teachers and students, personal motivation is key to engagement

Quality of SRs

The quality of the 16 included SRs assessed using the AMSTAR tool ranged from low (5.5) to satisfactory (9), average 7.4 points of a possible maximum 11 (see online Table 2). Review quality was consistently downgraded due to the lack of provision of a list of included and excluded studies, poor listing of the characteristics of included studies and lack of sufficient assessment of publication bias.

The quality of evidence using the GRADE parameters ranged from very low (n = 2), low (n = 6) to moderate (n = 8) (see online Table 3). The quality of evidence was downgraded most frequently due to perceived limitations in the authors’ consideration of risk for potential bias, including lack of allocation concealment and lack of blinding. SR authors note the considerable inconsistency of reporting, lack of confidence in quantitative analyses, excessive heterogeneity in terms of study design and methods, while existing reports often included a small number of RCTs. Our analysis, therefore, was limited to descriptive reporting and qualitative analyses due to this inconsistent reporting across reviews.

Duplicate data

This overview identified 278 RCT references included across SRs (n = 16), of which 44 RCT were duplicates across SRs. We did not reject any SR with an RCT from a duplicate source. Instead, we noted this duplication as a limitation of an overview, reported where relevant in quantitative and qualitative syntheses and analysed for differences in the SR qualitative reporting of results.

Characteristics of participants

This overview includes data collected from 31,730 participants across 16 SRs (see online Table 4). The SRs included HCPs (n = 12) and students in undergraduate (n = 9) or graduate training in health care (n = 5). Research findings were most prevalent for nurse training (n = 10), followed by education of SRs including general physicians (n = 6). Other listed participants were dentists, orthodontists, paramedics, physical therapists, surgeons and educators in health care.

In the reporting SRs, digital health education studies have been conducted primarily in the USA and the UK, with other contributions from Brazil, Taiwan and China. We are unable to provide a subgroup description or analysis of the population characteristics such as age, gender, language, geographical region, socioeconomic characteristics, education status or perception of technology due to lack of data.

Characteristics of interventions

Digital education in this review includes the use of technology to communicate information for the purposes of knowledge or skill transfer. Of the included SRs, all reviews included digital education that included the use of a computer and a multimedia component. For a summary of the types of digital education identified in this overview, (adapted from Maertens et al., 2016) see Box 1.

Box 1.

Types of digital education.

● Multimedia components (videos, images, animations, audio files, text and hypertext)● Interactive sessions (conference, chats, forums, messaging)● Virtual reality (simulations, case-based learning)● Automated assessments (multiple choice, short answer)● Gaming (learning through structured play)● Applications (downloaded to personal device for decision tools, reminders, information)

The term e-learning was cited (n = 5) as equivalent to Internet-based learning, or website supported education. Virtual patients (VPs) and case-based scenarios were used in four SRs (Consorti et al., 2012; Cook et al., 2010a; Härkänen et al., 2016; Lam-Antoniades et al., 2009).

Computer-assisted learning (CAL) was described in instances (n = 5) where the computer was used to enhance, or blend in, with traditional in-person methods. CAL included web-based material, as well as multimedia components such as CD-ROMs, videos and text files.

Digital interventions varied substantially in the amount of delivery time: from a minimum of 15 minutes (Consorti et al., 2012) to a maximum duration of 8,760 hours over 12 months (Maertens et al., 2016). Due to lack of consistent reporting, we were unable to determine an average or median time of duration or intervention for digital education. Insufficient data reporting for intervention characteristics by SR authors was identified in 13 of 15 reviews.

Characteristics of comparisons

From the available data, studies included within the SRs used in-person teaching (traditional method) comparison groups (n = 127), no intervention or waitlisted participants (n = 88), or a mix of either in-person, digital, or another unspecified intervention (n = 61). The detailed characteristics of comparison groups, including description of teaching styles, session duration and physical location were not sufficiently reported in several SRs.

Outcomes

Six SRs conducted quantitative data-analysis to compare digital learning with other education methods. Excessive heterogeneity, including differences in outcome measures and the type of interventions across these reviews prohibited further meta-analysis. Instead, the SR quantitative summaries are reported according to result direction of primary outcome (as first stated by SR authors). Three of the SRs favour digital education over various comparison groups.

Favours digital education

One review (Al-Jewair et al., 2009) included undergraduate, post-graduate, pre-clinical, clinical and educators of orthodontics to examine differences in knowledge acquisition, learning efficiency, learner, education attitudes, cost and labour of delivery. In terms of knowledge gain, results favoured CAL with a weighted mean difference (WMD) 9.78% confidence interval (CI) 95% 2.89 to 16.67; p = .005). Three sources were included in this synthesis (Aly et al., 2003; Clark et al., 1997; Rosenberg et al. 2008). Results were heavily weighted from one study (Rosenberg, 2008) that accounted for 67.47% of the measured outcome. The other two studies did not show statistically significant results and indicated no mean difference between computer use and traditional learning.

In the Consorti et al. (2012) review, undergraduate and graduate medical education students of general medicine were evaluated for changes in clinical reasoning following training using VPs in blended learning. Clinical reasoning skills improved using VPs as an additive resource, odds ratio (OR): 2.39 (95% CI: 1.364 to 4.791; p = .003) and as an alternative to traditional teaching OR: 2.190 (95% CI: 1.059 to 4.527; p = .034). Five RCTs (Kerfoot et al., 2006; Schittek Janda et al., 2004; Triola et al., 2006; Vash et al., 2007; Wahlgren et al., 2006) were synthesised in this review to analyse the effects of VPs as an additive resource and 7 RCTs (Botezatu et al., 2010; Deladisma et al., 2007; Fleetwood et al., 2000; Fleming et al., 2009; Kandasamy and Fung, 2009; Kumta et al., 2003; Youngblood et al., 2008) were synthesised to analyse effects of VPs as a replacement to traditional methods.

Härkänen et al’.s (2016) review compared Internet-based and CAL with traditional methods and no intervention for effects on medical administration and safety skills of practising nurses. The pooled effect size (Hedges’ g) of 1.06 (95% CI: 0.44–1.69; p = .001) favoured digital learning from four studies whose quality was evaluated as moderate or strong (Lu et al., 2013; Simonsen et al., 2014; Sung et al., 2008; Tsai et al., 2008).

No significant difference for digital education

In Feng et al’.s (2013) meta-analysis, the digital learning group demonstrated a statistically significant positive effect on knowledge with a Cohen’s standard mean difference (SMD) of 1.66 (95% CI: 0.97 to 2.42; p < .0001) when compared to no other learning programme. However, when combined with studies using traditional methods as the comparator, the overall effect was not significant on knowledge acquisition SMD of 0.24 (95% CI: −0.15 to 0.62; p < .0001). Eight RCTs were included in Feng’s knowledge data analyses (Elgie et al., 2010; Gega et al., 2007; Hugenholtz et al., 2008; McDonough and Marks, 2002; Smits et al., 2012; Truncali et al., 2011; Vash et al., 2007; Wenk et al., 2009 [n = 638]).

Lahti et al.’s (2014) review included practising nurses as well as nursing students. The review included CAL and Internet-based learning, and compared this intervention with traditional methods in its meta-analysis for differences in knowledge level. Results indicated no statistically significant differences in knowledge acquisition for digital learning: mean difference (MD) 0.44 (95% CI: −0.57 to 1.46; p = .39) across four studies (Gega et al., 2007; Horiuchi et al., 2009; Paladino and Peres, 2007; Tsai et al., 2004).

One SR (Cook et al., 2010a) investigated learning efficiency of 100% Internet-based learning compared to traditional and other CAL interventions of health professionals in training or practice. In their random effects meta-analysis of eight studies (Bell et al., 2000; Cook et al., 2005; Dennis, 2003; Friedl et al., 2006a, 2006b; Grundman et al., 2000; Leong et al., 2003), the pooled effect size was −0.10 (95% CI: −0.51 to 0.31; p = .63) for Internet-based learning when compared to non-Internet instruction. These data suggest no significant difference in time spent learning on Internet courses and non-Internet courses.

Subgroup analyses

In the population subset, we sought to examine any differences in primary outcomes of health-care students versus working professionals. One study provided quantitative results (Feng et al., 2013) and found significant improvements in skill performance following digital education in medical and nursing students, not in practising clinicians, (Cohen’s) SMD = 0.30 (95% CI: 0.02 to 0.57; p = .038).

Cost information for digital education

Aspects of cost for digital education were reported in five SRs (Al-Jewair et al., 2009; Du et al., 2013; George et al., 2014; Jayakumar et al., 2015; Rasmussen et al., 2014). In general, the literature suggests upfront investment in software, hardware and faculty time is the most expensive aspect of digital education implementation. Al-Jewair et al. (2009) indicated 300 hours of administration time is needed to create a 60-minute lecture on a computer-assisted medium.

Two SRs provided estimate costs of developing a digital course, including initial hardware and software investments, as well as faculty time. In Al-Jewair et al. (2009), the direct design and set-up costs of a computer-assisted website costs range from US$1,850. George et al. estimated an initial investment requirement of US$10,000 for a digital course, with an additional US$2,200 investments being required for video cases.

For cost-effectiveness of digital education compared to traditional teaching methods, one SR reported digital learning required 50% less faculty time and incurred less expense by both students and faculty in its implementation (Jayakumar et al., 2015). Three additional SRs sought to analyse data on cost-effectiveness, but their included RCTs did not provide sufficient data (Al-Jewair et al., 2009; George et al., 2014; Rasmussen et al., 2014).

Learner preferences in digital education

Fifteen of the included SRs presented data on some aspects of learner preferences. Many of the reviews reported high satisfaction with digital education, especially when used as an additional resource to in-person teaching. Students of health-care education may prefer digital education for its ‘flexibility (asynchronous design), learner independence, and time efficiency’ (Du et al., 2013). In addition, the multiple components of digital education provide for an extensive variety of content delivery (text, video, animation and games) to arouse and retain the attention of the learner. Highly interactive courses are desirable for students, especially in the absence of in-person teaching. Ease of accessibility, affordability, self-pacing and repetition of material are important advantages of the digital education interventions.

Evidence-based instructional design

All included SRs (n = 16) reported aspects of effective instructional design for health-care students or professionals. Three central themes regarding instructional design were highlighted in these reviews, including the nature and complexity of learning requirements, efficiency trade-offs with interactive learning and general design preferences.

Matching learning requirements

The qualitative analysis across SRs suggests that capable learners benefit more from online (asynchronous learning). The independent, self-paced nature of digital learning may be more suitable for students who are experienced in knowledge acquisition. Virtual or case-based scenarios delivered appear more applicable to produce and monitor learning outcomes in contextual knowledge:

Situated e-learning platform provides learners with photographs, videos, or multimedia on an individual basis … (it) provides opportunities for students to interact with a virtual client within the programme as in a real-life situation, and emphasise perception and action in various contexts rather than memory of knowledge. (Feng et al., 2013: 181)

To ensure a positive learning outcome, the learning environment and learning objectives must match the instructional method (Sinclair et al., 2016). SR authors were not more explicit about which learning objectives are best matched with digital learning activities.

Facets of interactive design

Highly interactive designs, where by the learner engages more in-depth with the course content (its platform, components and/or peers and faculty), allow for a potential increase in communication and feedback during the learning process. An increase in interaction is associated with an increased time spent on learning activities. Learner satisfaction can be increased, or decreased with a highly interactive design, depending on the expectations of the student. In comparison, students enjoy a greater amount of privacy, independent learning and time-efficiency in digital courses with a low-interactive design.

General design preferences

Instructional activities do not facilitate learning with equal effectiveness or efficiency (Cook et al., 2010a: 766), or elicit the same amount of satisfaction across subjects or students. In general, however, the following elements of general design appear in existing literature to support a positive digital learning experience. First, to encourage participation and prevent student drop-out, SRs ‘suggest a goal-directed curriculum that adheres to principles of educational psychology, including focused and immediate feedback’ (Maertens et al. 2016: 1435). Intermittent evaluations of student progress can assist in arousing the participant to learn. Click and drag evaluation tools benefit the learner, and have shown benefits in knowledge and skill acquisition (Rasmussen et al., 2014).

Second, courses split into small units of material have higher satisfaction, and are easier for students to track and maintain progress. In addition, short video clips have led to students spending more time on learning, and are associated with higher knowledge test scores (Cook et al., 2010a). Furthermore, providing content so that a student may repeat specific material, and replaying videos at their own pace also shows greater knowledge gains (George et al., 2014).

Third, course components should be based on student-centred principles, and include a variety of activities that tend towards visual or auditory stimulation. Interventions consisting of ‘flat text are of limited value and should be avoided if possible’ (Lam-Antoniades et al., 2009: 50). In terms of lecture style, poor-quality videos should be avoided, ‘students prefer online lectures of power point slides with audio narration’ (George et al., 2014: 11). Clarity, motivating exercises and real-world examples were especially appreciated by students (Santos et al., 2016).

Barriers for digital education

All included SRs (n = 16) reported on contextual barriers or limitations to digital education. Negative themes related primarily to technical issues, including ‘lack of equipment and support, poor Internet connection, poor infrastructure and anxieties about fairness, security and cheating’ (Webb et al., 2017: 167). In addition, lack of faculty interest or training in learning tools for online education. Finally, this emergent theme points towards a misconception of the need or lack of resources to develop high-quality multimedia components (administration time, money, professional expertise).

Discussion

Summary of main findings

Of the included reviews in this synthesis, no SR authors reported other teaching methods as being superior to digital learning. In most cases (n = 9), digital education when used in addition to traditional methods augmented the primary outcomes of knowledge acquisition. Other SRs (n = 7) did not show statistically significant differences across interventions including digital education as a replacement, or additive resource to traditional intervention.

Does digital education change clinical practice?

Three SRs (Feng et al., 2013; Lahti et al., 2014; Sinclair et al., 2016) reported changes in clinical practice of nurses and physicians following an e-learning intervention. Their findings suggest digital education methods are at least as equivalent to traditional learning approaches, or superior to no instruction at all. Durmaz et al’.s (2012) study reported that e-learning was more effective (p = .04) than skill laboratories alone for second year undergraduate nursing students in teaching pre-operative patient admission skills. However, in the same cohort’s post-intervention deep breathing and coughing exercises, e-learning was not found to be more effective than clinical laboratory instruction (p = .867). The effectiveness of e-learning compared to no training at all was demonstrated in three studies (Elgie et al., 2010; Gordon et al., 2011; Smeekens et al. 2011). Gordon et al. (2011) was the only study to include a longitudinal element in its design and reported that e-learning was superior to no intervention at all (p < .0001), and that paediatric prescribing skills outcomes were maintained 3 months post intervention (p < .0001). As expected, the strength of results favouring digital education increase when compared to a waitlist or no comparison intervention.

What is evidence-based instructional design for digital health-care education?

Instructional material delivered in clear, small, sub-sections of multiple components using audio, video, hypertext, graphics or case-based simulations can improve learning outcomes and student satisfaction. Regular evaluation, feedback and opportunities for student interaction tend to motivate learner participation. The cost-effectiveness of digital education is an area for further exploration, especially as it relates to behavioural outcomes and skill performance of HCPs trained using traditional in-person methods. Adequate faculty training, equipment and resources (time, money) to design the course are significant barriers that need to be addressed. Although the quality and quantity of data are limited, evidence-based instructional design for digital education is becoming more possible, especially as educators establish learning activities that track learning objectives for knowledge acquisition in health care.

Overall completeness and applicability of evidence

Our overview includes quantitative and qualitative data gathered from 16 SRs (278 RCTs and 31,730 participants). The quality of SRs was moderately good. In general, insufficient reporting of interventions and comparison groups prevented further meta-analysis. Nursing students and practitioners are well-represented in existing literature (included in 10 of 16 reviews). Data from the included SRs did not sufficiently report on cost-effectiveness, learner characteristics or effective digital design elements. Questions of behavioural outcomes of HCPs and downstream patient outcomes were raised but have not been adequately answered in existing research.

Potential biases and discussion of heterogeneity

The quality of evidence from the RCTs was graded as low, the body of evidence risking high exposure to bias from lack of adequate randomisation, lack of blinding and lack of adequate control groups. SR reported concerns about evidence from other study designs that did not provide a baseline measure, and lack of clear methods for detecting outcomes, subjecting the results to confounding and measurement bias (Al-Jewair et al., 2009; Sinclair et al., 2016). Evidence of publication bias was considered but not found in several SRs (n = 5) (Al-Jewair et al., 2009; Härkänen et al., 2016; Maertens et al., 2016; Sinclair et al., 2016; Webb et al., 2017), although an overrepresentation of positive studies from other SRs may have affected results.

Weak quality ratings of RCTs were consistent across all SRs. Poor descriptions of interventions and comparison groups may have led to misinterpretation or wrong classification of some results (Consorti et al., 2012). An additional limitation is the overlap in RCTs for the synthesis, although we have disclosed two instances where the individual RCTs impacted the quantitative description of the SR results. This body of literature has an additional bias towards English-language studies.

The effect sizes of digital education are affected by a significantly heterogeneous set of interventions and comparison groups. In several SRs (n = 4), the comparison groups also included an aspect of digital intervention (blended learning). The interventions often included blended learning and 100% digital interventions to compare with traditional methods. The information was not provided as a clear subset and therefore limited subgroup analyses or extraction of 100% online digital interventions. A future SR may consider a more limited definition of what constitutes digital learning, and seek to eliminate hybrid or other digital learning components from control groups.

Implications for practice and future research

Given the growth of online education services for health care students and professionals, it is clear that the evidence needs to be strengthened to understand how, and what type of digital education is most suitable for specific learning objectives. In terms of SR methodology, we recommend future reviewers follow the Cochrane guidelines (Green et al., 2008) to ensure data can be used in an overview synthesis. Future randomised controlled studies should report detailed intervention characteristics (duration, features, prototypes) and comparison characteristics. It is also important for future research to explore cost-efficacy, learning efficiency, and the impact of HCP digital education on clinical outcomes for patients. Future studies should also seek to evaluate specific instructional design components, assessing for relative effects or cost-benefits of video versus audio with slides or animated graphics and other customised interactive games or platforms on learning.

Conclusion and recommendations

The digital education interventions analysed are at least equal to traditional methods in terms of knowledge outcomes and learner satisfaction. Student enrolment, attendance rates, course completion rates, cost-effectiveness and changes in clinical outcomes for patients are underreported in the existing evidence. Barriers to implementing digital education include skills training for faculty, technological glitches and lack of resources (time, money) to invest in the course. At this time, evidence suggests effective instructional design strategies: (1) communicate in clear, short segments of varied student-centred learning activities (audio, video, graphics, text and hypertext, animation, simulations, evaluations) that are tracked to learning objectives, (2) use adequate real-world examples and (3) include an appropriate level of interactivity and feedback throughout course delivery.

Supplemental Material

HEJ-17-0175_supplementary_material – Supplemental material for Characteristics and efficacy of digital health education: An overview of systematic reviews

Supplemental material, HEJ-17-0175_supplementary_material for Characteristics and efficacy of digital health education: An overview of systematic reviews by Marcy McCall, Elizabeth Spencer, Helen Owen, Nia Roberts and Carl Heneghan in Health Education Journal

Footnotes

Funding

The author(s) received no financial support for the research, authorship and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

ORCID iD

Marcy McCall

References

Al-Jewair

Azarpazhooh

Suri

et al . (2009) Computer-assisted learning in orthodontic education: A systematic review and meta-analysis. Journal of Dental Education 73(6): 730–739.

Aly

Willems

Carels

et al . (2003) Instructional multimedia programs for self-directed learning in undergraduate and postgraduate training in orthodontics. European Journal of Dental Education 7(1): 20–26.

Bell

Fonarow

Hays

et al . (2000) Self-study from web-based and printed guideline materials: A randomized, controlled trial among resident physicians. Annals of Internal Medicine 132(12): 938–946.

Botezatu

Hult

Fors

(2010) Virtual patient simulation: What do students make of it? A focus group study. BMC Medical Education 10: 91.

Clark

Weekrakone

Rock

(1997) A hypertext tutorial for teaching cephalometrics. British Journal of Orthodontics 24(4): 325–328.

Consorti

Mancuso

Nocioni

et al . (2012) Efficacy of virtual patients in medical education: A meta-analysis of randomized studies. Computers & Education 59(3): 1001–1008.

Cook

Dupras

Thompson

et al . (2005) Web-based learning in residents’ continuity clinics: A randomized, controlled trial. Academic Medicine 80(1): 90–97.

Cook

Levinson

Garside

(2010a) Time and learning efficiency in internet-based learning: A systematic review and meta-analysis. Advances in Health Sciences Education: Theory and Practice 15(5): 755–770.

Cook

Levinson

Garside

et al . (2010b) Instructional design variations in internet-based learning for health professions education: A systematic review and meta-analysis. Academic Medicine 85(5): 909–922.

10.

Curran

Fleet

(2005) A review of evaluation outcomes of web-based continuing medical education. Medical Education 39(6): 561–567.

11.

Davis

O’Brien

Freemantle

et al . (1999) Impact of formal continuing medical education: Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes? Journal of the American Medical Association 282(9): 867–874.

12.

Deladisma

Cohen

Stevens

et al . (2007) Do medical students respond empathetically to a virtual patient? American Journal of Surgery 193(6): 756–760.

13.

Dennis

(2003) Problem-based learning in online vs. face-to-face environments. Education for Health 16(2): 198–209.

14.

Liu

et al . (2013) Web-based distance learning for nurse education: A systematic review. International Nursing Review 60(2): 167–177.

15.

Durmaz

Dicle

Cakan

et al . (2012) Effect of screen-based computer simulation on knowledge and skill in nursing students’ learning of preoperative and postoperative care management: A randomized controlled study. Computers Informatics Nursing 30(4): 196–203.

16.

Elgie

Sapien

Fullerton

et al . (2010) School nurse online emergency preparedness training: An analysis of knowledge, skills, and confidence. The Journal of School Nursing 26(5): 368–376.

17.

Feng

Chang

et al . (2013) Systematic review of effectiveness of situated e-learning on medical and nursing education. Worldviews on Evidence-Based Nursing 10(3): 174–183.

18.

Fleetwood

Vaught

Feldman

et al . (2000) MedEthEx online: A computer-based learning program in medical ethics and communication skills. Teaching and Learning in Medicine 12(2): 96–104.

19.

Fleming

Olsen

Stathes

et al . (2009) Virtual reality skills training for health care professionals in alcohol screening and brief intervention. Journal of the American Board of Family Medicine 22(4): 387–398.

20.

Friedl

Höppler

Ecard

et al . (2006a) Comparative evaluation of multimedia driven, interactive, and case-based teaching in heart surgery. The Annals of Thoracic Surgery 82(5): 1790–1795.

21.

Friedl

Höppler

Ecard

et al . (2006b) Multimedia-driven teaching significantly improves students’ performance when compared with a print medium. The Annals of Thoracic Surgery 81(5): 1760–1766.

22.

Gega

Norman

Marks

(2007) Computer-aided vs. tutor-delivered teaching of exposure therapy for phobia/panic: Randomized controlled trial with pre-registration nursing students. International Journal of Nursing Studies 44(3): 397–405.

23.

George

Papachristou

Belisario

et al . (2014) Online e-learning for undergraduates in health professions: A systematic review of the impact on knowledge, skills, attitudes and satisfaction. Journal of Global Health 4(1): 010406.

24.

Gordon

Chandratilake

Baker

(2011) Improved junior paediatric prescribing skills after a short e-learning intervention: A randomised controlled trial. Archives of Disease in Childhood 96(12): 1191–1194.

25.

Green

Higgins

Alderson

et al . (2008) Cochrane Handbook for Systematic Reviews of Interventions. Chichester: John Wiley & Sons.

26.

Grundman

Wigton

Nickol

(2000) A controlled trial of an interactive, web-based virtual reality program for teaching physical diagnosis skills to medical students. Academic Medicine 75(10 suppl): S47–S49.

27.

Guyatt

Oxman

Schünemann

et al . (2011) GRADE guidelines. Journal of Clinical Epidemiology 64(4): 380–382.

28.

Härkänen

Voutilainen

Turunen

et al . (2016) Systematic review and meta-analysis of educational interventions designed to improve medication administration skills and safety of registered nurses. Nurse Education Today 41: 36–43.

29.

Horiuchi

Yaju

Koyo

et al . (2009) Evaluation of a web-based graduate continuing nursing education program in Japan: A randomized controlled trial. Nurse Education Today 29(2): 140–149.

30.

Hugenholtz

de Croon

Smits

et al . (2008) Effectiveness of e-learning in continuing medical education for occupational physicians. Occupational Medicine 58(5): 370–372.

31.

Jayakumar

Brunckhorst

Dasgupta

et al . (2015) e-Learning in surgical education: A systematic review. Journal of Surgical Education 72(6): 1145–1157.

32.

Kandasamy

Fung

(2009) Interactive Internet-based cases for undergraduate otolaryngology education. Otolaryngology-Head and Neck Surgery 140(3): 398–402.

33.

Kerfoot

Baker

Jackson

et al . (2006) A multi-institutional randomized controlled trial of adjuvant Web-based teaching to medical students. Academic Medicine 81(3): 224–230.

34.

Kirkpatrick

(1998) Techniques for Evaluating Training Programs. 2nd ed. San Francisco, CA: Berrett-Koehler Publishers, Inc.

35.

Kumta

Tsang

Hung

et al . (2003) Fostering critical thinking skills through a web-based tutorial programme for final year medical students: A randomized controlled study. Journal of Educational Multimedia and Hypermedia 12(3): 267–273.

36.

Lahti

Hätönen

Välimäki

(2014) Impact of e-learning on nurses’ and student nurses knowledge, skills, and satisfaction: A systematic review and meta-analysis. International Journal of Nursing Studies 51(1): 136–149.

37.

Lam-Antoniades

Ratnapalan

Tait

(2009) Electronic continuing education in the health professions: An update on evidence from RCTs. The Journal of Continuing Education in the Health Professions 29(1): 44–51.

38.

Leong

Baldwin

Adelman

(2003) Integrating web-based computer cases into a required clerkship: Development and evaluation. Academic Medicine 78(3): 295–301.

39.

Chen

et al . (2013) Nurses’ knowledge of high-alert medications: A randomized controlled trial. Nurse Education Today 33(1): 24–30.

40.

McCutcheon

Lohan

Trayno

et al . (2015) A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. Journal of Advanced Nursing 71(2): 55–70.

41.

McDonough

Marks

(2002) Teaching medical students exposure therapy for phobia/panic: Randomized, controlled comparison of face-to-face tutorial in small groups vs. solo computer instruction. Medical Education 36(5): 412–417.

42.

Maertens

Madani

Landry

et al . (2016) Systematic review of e-learning for surgical training. The British Journal of Surgery 103(11): 1428–1437.

43.

Marinopoulos

Dorman

Ratanawongsa

et al . (2007) Effectiveness of continuing medical education. Evidence Report Technology Assessment 149(149): 1–69.

44.

Oxman

(1994) Checklists for review articles. British Medical Journal 309(6955): 648–651.

45.

Paladino

Peres

(2007) E-learning: A comparative study for knowledge apprehension among nurses. The Revista Latino-American de Enfermagem 15(3): 397–403.

46.

Rasmussen

Belisario

Wark

et al . (2014) Offline e-learning for undergraduates in health professions: A systematic review of the impact on knowledge, skills, attitudes and satisfaction. Journal of Global Health 4(1): 010405.

47.

Rosenberg

(2008) An evaluation of computer-aided learning in orthodontics. Master’s Thesis, University of Toronto, Toronto, ON, Canada.

48.

Santos

Leite

Figueiredo

et al . (2016) Effectiveness of e-learning in oral radiology education: A systematic review. Journal of Dental Education 80(9): 1126–1139.

49.

Schittek Janda

Mattheos

Nattestad

et al . (2004) Simulation of patient encounters using a virtual patient in periodontology instruction of dental students: Design, usability, and learning effect in history-taking skills. European Journal of Dental Education 8(3): 111–119.

50.

Shea

Hamel

Wells

et al . (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology 62(10): 1013–1020.

51.

Simonsen

Daehlin

Johansson

et al . (2014) Improvement of drug dose calculations by classroom teaching or e-learning: A randomised controlled trial in nurses. BMJ Open 4(10): e006025.

52.

Sinclair

Kable

Levett-Jones

et al . (2016) The effectiveness of Internet-based e-learning on clinician behaviour and patient outcomes: A systematic review. International Journal of Nursing Studies 57: 70–81.

53.

Smeekens

Broekhuijsen-van Henten

Sittig

et al . (2011) Successful e-learning programme on the detection of child abuse in emergency departments: a randomised controlled trial. Archives of Disease in Childhoood 96(4): 330–334.

54.

Smits

de Graaf

Radon

et al . (2012) Case-based e-learning to improve the attitude of medical students towards occupational health, a randomised controlled trial. Occupational and Environmental Medicine 69(4): 280–283.

55.

Sung

Kwon

Ryu

(2008) Blended learning on medication administration for new nurses: Integration of e-learning and face-to-face instruction in the classroom. Nurse Education Today 28(8): 943–952.

56.

Triola

Feldman

Kalet

et al . (2006) A randomized trial of teaching clinical skills using virtual and live standardized patients. Journal of General Internal Medicine 21(5): 424–429.

57.

Truncali

Lee

Ark

et al . (2011) Teaching physicians to address unhealthy alcohol use: A randomized controlled trial assessing the effect of a Web-based module on medical student performance. Journal of Substance Abuse Treatment 40(2): 203–213.

58.

Tsai

Chai

Hsieh

et al . (2008) The use of virtual reality computer simulation in learning Port-A cath injection. Advances in Health Sciences Education: Theory and Practice 13(1): 71–87.

59.

Tsai

Chai

et al . (2004) Evaluation of computer-assisted multimedia instruction in intravenous injection. International Journal of Nursing Studies 41(2): 191–198.

60.

Vash

Yunesian

Shariati

et al . (2007) Virtual patients in undergraduate surgery education: A randomized controlled study. ANZ Journal of Surgery 77(1–2): 54–59.

61.

Wahlgren

Edelbring

Fors

et al . (2006) Evaluation of an interactive case simulation system in dermatology and venereology for medical students. BMC Medical Education 6: 40.

62.

Webb

Clough

O’Reilly

et al . (2017) The utility and impact of information communication technology (ICT) for pre-registration nurse education: A narrative synthesis systematic review. Nurse Education Today 48: 160–171.

63.

Wenk

Waurick

Schotes

et al . (2009) Simulation-based medical education is no better than problem-based discussions and induces misjudgment in self-assessment. Advances in Health Sciences Education: Theory and Practice 14(2): 159–171.

64.

Youngblood

Harter

Srivastava

et al . (2008) Design, development, and evaluation of an online virtual emergency department for training trauma teams. Simulation in Healthcare 3(3): 146–153.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.59 MB