Updated Meta-Analysis of Learner Control Within Educational Technology

Abstract

Giving a student control over their learning has theoretical and intuitive appeal, but its effects are neither powerful nor consistent in the empirical literature base. This meta-analysis updated previous meta-analytic research by Niemiec, Sikorski, and Walberg by studying the overall effectiveness of providing learner control within educational technology, the characteristics of instruction along the continuum of learner control, and elements of the instructional environments that may play a role in the effectiveness of educational technology. The search terms identified 85 distinct articles, 18 of which met the inclusion criteria (29 effects were computed). The overall effect of including learner control within educational technology was almost zero (g = 0.05), and were also near zero when examining most characteristics of control and classroom contextual factors. Moderate effects were reported for providing learner control within social studies/history courses and for comprehensive technology instructional programs. The effects were larger for behavioral outcomes than academic outcomes, but both were small.

Keywords

learner control educational technology

The rapid growth of availability and the ever-changing evolution of educational technologies (Cheung & Slavin, 2012) make it increasingly difficult to determine how to best incorporate technology into the classroom. Moreover, the increasing diversity of instructional needs make it difficult for teachers to reach every learner without the aid of technology (Ysseldyke & McLeod, 2007; Ysseldyke et al., 2003). Thus, our education system is challenged to utilize modern technology in order to create engaging, relevant, and personalized learning experiences (U.S. Department of Education, 2010). Many schools have integrated laptops and other digital tools into daily practice, but it is unclear if those devices are being used in ways that best maximize their potential (Greaves, Hayes, Wilson, Gielniak, & Peterson, 2010).

Despite increased prevalence and use of computers in schools, research on the effectiveness has yielded mixed results (Machin, McNally, & Silva, 2007). Early research found that educational technology produced positive but small to moderate effect sizes (mean ES = 0.30) on student achievement (Kulik & Kulik, 1991). However, interventions based on practice and strategies were equally effective without technology (e.g., Ehri, Dreyer, Flugman, & Gross, 2007; Torgesen & Barker, 1995; Torgesen, Wagner, Rashotte, Herron, & Lindamood, 2010; Wentink, Van Bon, & Schreuder, 1997). Dynarski et al. (2007) conducted a fairly recent large-scale randomized trial on the effectiveness of computer-based instruction and found only small effect sizes on standardized achievement tests. Cohen’s d, an effect size measured by the difference between two means divided by standard deviation, was used to measure size of effect in the following studies. Second-order meta-analytic research found moderate effects that were consistent with earlier research in magnitude (d = 0.35, Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011; d = 0.31, Hattie, 2009). Moreover, meta-analytic research that compared use of technology to lecture-based instruction when teaching statistics also resulted in a small to moderate effect of d = 0.33 favoring technology (Sosa, Berger, Saw, & Mary, 2011). However, research investigating the use of technology beyond a simple comparison of the same task completed in a traditional format versus aided by a computer is lacking.

In addition to finding small to moderate effects, the quality of previous research regarding instructional technology was criticized for inconsistent or low rigor (Tamim et al., 2011), and has decreased in experimental rigor over the past 20 years (Ross, Morrison, & Lowther, 2010). The quality of the research design can affect the outcomes in meta-analytic research of intervention effectiveness (Jitendra, Burgess, & Gajria, 2011; Methe, Kilgus, Neiman, & Riley-Tillman, 2012). Moreover, the companies and institutions that have created and promoted the technology often funded previous research, which could raise questions of the research line’s objectivity. Rather, the aim of educators should be to understand the underlying processes in order to exploit the full capabilities of technology to advance student learning (Duncan, 2010).

Learner Control

There are multiple interpretations of previous research that showed inconsistent or small to moderate effects for educational technology (Ross et al., 2010), and additional research is needed to understand the underlying mechanisms of educational technology (Duncan, 2010). One explanation for the inconsistent results in previous research could be the degree of learner control within the specific educational technology. In contrast to traditional classroom instruction, the use of technology potentially allows educators to empower students to take control of their own learning (U.S. Department of Education, 2010). Learner control is the degree to which students can direct their own learning experiences (Shyu & Brown, 1992), including path, pace, and instructional approach (Hannafin, 1984). Such control could include choices at the curriculum level (sequence of instructional materials), the opportunity to choose how long to focus on a learning objective (pacing), or the ability to select and sequence a variety of review strategies (choice of practice items or amount of review material; Merrill, 1980; Niemiec, Sikorski, & Walberg, 1996). Although traditional materials of instruction, such as textbooks, follow a linear sequence, computer-based instruction is capable of more flexibility in how the student accesses and follows the available information. Computer-assisted or adapted instruction provides control to the learner along a continuum. At one end of the continuum is program control, where the student follows a specified path. At the other end of the continuum is learner control, where the learner freely interacts with and directs their learning.

Providing learner control within computer-assisted instruction is potentially consistent with the theory of intercontextuality, in which learners are responsible for transferring what they know from one context to others (Engle, 2006; Engle, Nguyen, & Mendelson, 2011), because it gives the learner responsibility for important aspects of the learning activity. However, the sense of responsibility for transfer is usually facilitated through social interactions (Hammer, Elby, Scherr, & Redish, 2005), or by explicitly introducing a lesson as an opportunity for students to have knowledgeable responsibilities within social communities (Engle et al., 2011). Thus, the link between learner control within computer-assisted instruction and the theory of intercontextuality is only hypothesized and appears to be an area in need of future research.

Although giving a student control over their learning has theoretical and intuitive appeal, its effects seem neither powerful nor consistent in the empirical literature base. When the concept of learner control in computer-adapted instruction was first introduced, it was assumed that the more control the learner had, the better the student would perform (Mager, 1964; Merrill, 1975). The hypothesis was that students would feel competent and be more interested in those activities in which they were allowed to make choices and affect their own outcomes (Lepper, 1985). As a result of making their own instructional decisions while learning, students would be more likely to explore tactics for different situations. In other words, students would learn how to better learn in the future (Merrill, 1975). This belief is still prevalent today. For example, many educators agree that personalizing learning is a key priority, and more than two thirds of teachers agree that technology helps them accomplish this differentiation (Marshall et al., 2009).

Although the positive effect of learner control seems to appeal intuitively to most students and teachers, empirical research has yielded varied findings. Some studies indicate there are no differences between learner and program control groups (Canino & Cicchelli, 1988), whereas others found nuanced differences across the varied levels of control. For example, Hannafin (1984) found that learner control is more likely to be successful over program control when the learner is older and more able, the objective is higher order skills rather than factual information, the content is familiar, and feedback is provided to assist the learner in making decisions. Because students differ in their ability to determine appropriate learning goals and instruction, some learners may view less material and skip important instructional components when they need additional review (Lee & Wong, 1989; Lepper, 1985), whereas others may practice unnecessary items they have already mastered.

A review and meta-analysis by Niemiec et al. (1996) synthesized research regarding learner control within educational technology. These authors coded learner control over pacing, time allocations for mastery, sequencing of instructional materials, choice of practice items, and amount of review material. The results of the meta-analysis yielded results that indicated an overall comparative effect of learner control that was slightly negative but near zero on average. The type of technology and the amount of technology available in the schools has changed drastically since 1996. Although Niemiec and colleagues’ frequently cited meta-analysis concluded that the average student would not receive positive benefits, and might even be slightly better off without control over their learning, it can be hypothesized that effects for computer programming in general as well as nuances across the components of learner control (pacing, time, sequencing, practice, review) have changed with the ever-advancing technological climate. An update to this original meta-analysis may help identify current trends in learner control within educational research.

Rationale for Current Study

Although previous research has discussed and compared learning effectiveness between instruction delivered on a computer versus traditional instruction, few studies have asked about what caused effective learning on a computer to occur. Students access information on technology across several mediums. For example, a high level of interactivity characterizes hypermedia systems, which are common among most recent programming (Scheiter & Gerjets, 2007). It is also becoming common for educators to supplement or replace traditional print curriculum with other trending media including podcasts, VoiceThreads, e-books, and online websites and communication tools (O’Brien & Scharber, 2010). With the changing landscape in how students are able to access information, technology offers schools a way to differentiate learning and provides students varied ways of approaching and understanding content (Sutherland, Shin, & McCall, 2010).

There is a need for more and better empirical evidence studying causal mechanisms by which technology affects learning. By identifying causal mechanisms and their effects, such as learner control, researchers and educators are able to make a continued effort to design and adapt to ever-changing technologies. The current meta-analysis was conducted to update the Niemiec et al. (1996) study and to extend it by directly examining the effects of various components of learner control (pacing, time, sequencing, practice, review) on student learning. The following research questions guided the study:

Research Question 1: To what extent does incorporating learner control within educational technology impact student outcomes?

Research Question 2: What characteristics of instruction (pacing, time allocations for mastery, sequencing of instructional materials, choice of practice items, and amount of review material) lead to the highest effects when controlled by the learner?

Research Question 3: To what extent do study features such as age of learner, setting of instruction, subject of instruction, type of outcome measured (behavioral or academic achievement), and format of program instruction affect the size of the effect?

Method

Data Collection

The Academic Search Premier and ERIC databases were searched in September of 2012, limited to abstracts and using the terms learner control AND reading, learner control AND math, learner control AND literacy, learner control AND science, learner control AND social studies, learner control AND history, learner control AND language, learner control AND English language arts, learner control AND writing, learner control AND English language learners, learner control AND computer assisted learning, learner control AND classroom, learner control AND computers in education, learner control AND educational technology, learner control AND school. Learner control was also searched individually to ensure an exhaustive exploration of potential articles. The search terms identified above were chosen because we were interested in examining learner control and its impact on a broad range of student outcomes.

The current meta-analysis included only published articles, as the threat presented by excluding nonpublished studies is minimal (Rosenthal, 1979). The following criteria were used to select articles to include in the current meta-analysis:

The study was published in a peer-reviewed journal between 1996 and 2012; this date range was selected because a previous study synthesized and aggregated learner control research conducted prior to 1996 (Niemiec et al., 1996).

The study used computers or technology for instructional (learning) purposes.

The study investigated aspects of learner control within technology and identified with sufficient detail the degree of control exhibited by the program versus learner.

The outcome variable(s) measured in the study was related to student outcomes (behavioral or academic achievement).

The study reported quantitative data in sufficient detail to calculate an effect size.

The study utilized a group design. Single-case experimental designs were not included because there is not clear consensus on how to best meta-analyze those data or how to incorporate them into meta-analyses with other studies that use group designs (Burns, 2012).

The study was written in English.

The study followed appropriate methodology. The methodology rubric initially developed by Gersten et al. (2005), and later adapted by Jitendra et al. (2011), was used to screen the methodology of the studies. The rubric provides a score of 1 (low) to 3 (high) across four categories including participants and setting, description of treatment and comparison conditions, outcome measures, and data analysis procedures. A study’s methodology was considered appropriate and included in this analysis if it received a score of at least eight points on the rubric.

The search terms identified 85 distinct articles, 18 of which met the above criteria (Figure 1). Sixty-seven studies were excluded, mostly due to lack of quantitative data reported, lack of detail explaining learner control, and not being published in a peer-reviewed journal.

Figure 1.

Flow chart of the study selection process.

Coding of Articles

Many studies reported multiple measures of behavior or academic achievement, resulting in 29 total outcomes. A total of 3,618 students participated in the studies within this meta-analysis. Table 1 further delineates study characteristics for each article. Studies that met inclusion criteria were systematically reviewed and coded according to the degree of control (program vs. learner) across five aspects of program design: pacing, sequencing, time allotment, choice of practice items, and choice of review items (see Niemiec et al., 1996).

Table 1

Study characteristics of articles included in analysis

Title	Primary author	Publication year	Grade level	Subject	Setting	Format of instruction	Outcome measure
Variability in reading ability gains as a function of computer-assisted instruction method of presentation	Johnson	2010	Elementary	Reading	Other	Comprehensive	Achievement
Learning effectiveness in a web-based virtual learning environment: a learner control perspective	Chou	2005	Middle school	Other	School	Comprehensive	Behavioral
Learners’ behaviors in computer-assisted input activities elicited through tracking technologies	Hwu	2003	College	Language	Home	Tutorial	Academic & behavioral
Web-based virtual learning environments: A research framework and a preliminary assessment of effectiveness in basic IT skills	Piccoli	2001	College	Technology	Home	Tutorial	Achievement & behavioral
Effects of game technology on elementary student learning in mathematics	Shin	2012	Elementary	Math	School	Comprehensive	Achievement
Effectiveness of cognitive-load based adaptive instruction in genetics education	Mihalca	2010	Multiple grades	Science	School	Tutorial	Achievement
Learner-control vs. program-control instructional multimedia: a comparison of two interactions when teaching principles of orthodontic appliances	Aly	2005	Adult	Other	Other	Tutorial	Achievement
Effects of multimedia redundancy in history learning among “deep and surface” students	Hii	2010	Middle school	Social studies	School	Tutorial	Achievement
Effects of locus of control and learner-control on web-based language learning	Mei Mei Chang	2012	College	Language	School	Comprehensive	Achievement
Effects of learner control with advisement on English as a second language students in a web-based learning environment	Wang	2002	Multiple ages	Social studies	School	Comprehensive	Achievement
Learner preferences and prior knowledge in learner-controlled computer-based instruction	Kopcha	2008	Middle school	Math	School	Tutorial	Achievement
Can instruction as such affect learning? The case of learner control	Vandewaetere	2011	College	Language	School	Tutorial	Achievement
On the effectiveness of self-paced learning	Tullis	2010	College	Language	School	Drill & practice	Behavioral
Dynamic task selection: effects of feedback and learner control on efficiency and motivation	Corbalan	2009	College	Science	School	Tutorial	Achievement
Effects of practice in a linear and nonlinear web-based learning environment	Martin	2008	College	Technology	School	Tutorial	Achievement
Virtual reality and brain anatomy: a randomized trial of e-learning instructional designs	Levinson	2007	College	Science	School	Tutorial	Achievement
Dynamic task selection in flight management system training	Salden	2006	College	Other	School	Tutorial	Achievement
The effects of self-regulation and instructional control on performance and motivation in computer-based instruction	Eom	2000	Middle school	Other	Lab	Tutorial	Achievement

Pacing indicated how quickly the content was presented to the learner. The sample was split into two subsamples for this variable, where k refers to the number in a subsample. Studies were coded as Learner Control if the participant controlled the pace with which the content was delivered (k = 21), and Program Control if the program dictated the pace (k = 4). Sequencing denoted how the information was ordered, such as when particular objectives or tasks were presented in relation to other objectives or tasks. A study was coded as Learner Control if the participant controlled the sequence of materials (k = 15), and was coded as Program Control if the program determined how the material was sequenced (k = 10).

Time allotment referred to the amount of time the learner was given to complete the content in its entirety for that session. There were 4 studies in which the learner determined the length of the session, 5 in which the computer program determined the length of each session, and 20 studies did not report on aspects of the session length or how it was determined. Practice items indicated the type and amount of practice on a particular objective, whereas review items were typically presented at the end of a lesson as a check for understanding. Studies were coded as Learner Control if the participant chose what and how to practice the material (k = 13), whereas those studies coded as Program Control presented practice material as determined by the educational technology (k = 12). Review of material was coded as Learner Control if the participant decided the extent of review provided (k = 14) and coded as Program Control if the program dictated the review process (k = 7).

The age level of participants (Pre K–Grade 6, Grade 7–12, College, and Across Multiple Ages) and subject matter taught by program (Math, English/Language, Reading, Science, Social Studies/History, Computers/Technology) were also coded. The setting of instruction was coded across School, Home, and Lab. Those studies coded as Lab were within a research setting rather than the typical environment for those particular students. The format of instruction was operationally defined across three categories, Drill and Practice Tool, Tutorial, and Comprehensive. A program was coded as drill and practice (k = 1) if the skill had been previously taught by an instructor and the computer provided an opportunity for the learner to practice the material. A tutorial program (k = 16) was coded as such if the computer provided instruction on one skill and consequently the learner practiced and reviewed the skill following instruction. A program was coded as comprehensive (k = 6) if it provided instruction, practice, and review on more than one discrete skill. Studies were coded as using a behavioral outcome if the measure related to time on task, motivation, self-efficacy, and so on (k = 7), and were coded as using an achievement measure (k = 16) if the objectives focused on improving reading or math.

Interobserver Agreement

The primary author, a third-year graduate student, coded all studies included in the analysis. A second-year graduate student provided interrater agreement on 40% of the outcomes reported in the studies. The percentage agreement was calculated as agreements divided by agreements plus disagreements multiplied by 100% for coding variables. Percentage agreement between the two raters was 95% for coding degree of control, 91% for study characteristics beyond program control components, and 87% for methodology quality indicators.

Effect Size Calculation

Hedge’s (1981) g was computed for each of the 34 outcomes measured across the 18 studies included in this meta-analysis. Hedge’s g is a standardized mean difference statistic, calculating the difference between the posttest treatment and control means divided by the standard pooled deviation. Although Hedge’s g index has a slight upward bias when estimating population effects, an adjustment is made for each outcome measure to correct for this bias (Hedges, 1983). With this adjustment, Hedge’s g has sound statistical properties in small sample sizes. If reported in standardized terms, the results are scale-free and are comparable across studies (Hedges & Olkin, 1985). For those studies in which means, SDs, or sample sizes were not reported, the F statistic was converted to Hedge’s g (Rosenthal, 1991). Some studies included multiple measures of the same construct (e.g., two reading measures, two behavioral outcomes), and the resulting effect sizes were averaged so that one estimate of effect was included in the analysis. According to Cohen (1988), an effect size of 0.2 is small, 0.5 is medium, and 0.8 is large. Slavin (1990) argued that in educational contexts, an effect size of 0.25 or greater had practical significance. Hattie (2009) suggested that an effect size of 0.40 had practical significance, but that was for second-order meta-analyses. Thus, the Slavin (1990) recommendation of 0.25 was used to judge the practical significance.

When outliers exist within data, estimates of mean and standard deviation are distorted. To ensure the current results were not affected by extreme values, outliers were identified and removed from the data set following Tukey’s statistical approach (Hoaglin, Mostellar, & Tukey, 1983). If Y < [Q1 − 1.5 (IQR)] OR Y > [Q3 + 1.5 (IQR)], where Q1 denotes the lower quartile, Q3 denotes the upper quartile, and IQR = (Q3 − Q1) denotes the interquartile range, then effect sizes were identified as outliers and not included in the analyses. A total of four effects, two positive and two negative, were removed from the data leaving 25 total effects.

Because only published studies were used in the meta-analysis, a failsafe N was computed (Orwin, 1983). The statistical procedure provides information on the stability of a meta-analysis by identifying how many null articles (studies with a zero effect) would have to be found to change a medium or large effect to a small effect. The criterion of 0.20 is used to indicate a small effect (Cohen, 1988), although this criterion is somewhat arbitrary. However, if no other criterion is available to guide a particular line of research, it is commonly accepted.

Results

The 18 included studies yielded 25 effect sizes (outcomes) in total, after removing outliers. Table 2 includes a median effect size for overall use of computer programming and results across aspects of instruction. The first research question focused on the effect of learner control within computer program instruction on student outcomes. All the studies included some aspect of learner control and resulted in a median effect size of g = 0.05 (95% confidence interval [CI] = −0.09 to 0.19), which was negligible and included zero within the range.

Table 2

Median effect sizes for computer program instruction and learner control characteristics

Learner control variable	K	Median g	SD	Failsafe N	95% Confidence interval
Some Aspect of Learner Control	25	.05	.35	NA	−.09 to .19
Program Control
Pacing	4	.17	.23	NA	−.07 to .41
Time	5	.00	.05	NA	−.13 to .12
Sequence	10	.02	.36	NA	−.20 to .24
Practice	13	.05	.30	NA	−.11 to .21
Review	7	.00	.23	NA	−.18 to .18
Learner Control
Pacing	21	.05	.37	NA	−.11 to .21
Time	4	−.05	.52	NA	−.56 to .46
Sequence	15	.16	.35	NA	−.02 to .34
Practice	12	.15	.41	NA	−.09 to .39
Review	14	.17	.36	NA	−.03 to .37

Note. Failsafe N based on criterion effect of 0.20 and rounded to the nearest whole number. NA = not applicable.

The second research question examined characteristics of instruction (pacing, time allocations for mastery, sequencing of instructional materials, choice of practice items, and amount of review material) that led to the highest effects when controlled by the learners. Table 2 shows median effect sizes across instructional characteristics. The median effect sizes for the studies for each category of instructional characteristics was small (g < 0.20; Cohen, 1988), lacked practical significance (g < 0.25; Slavin, 1990), and included zero within the range. Moreover, there was considerable overlap between the confidence intervals of median effect sizes computed from studies that used learner controlled features and program controlled features.

The third research question focused on variability of effect sizes and study features including age of learner, setting of instruction, subject of instruction, type of outcome measures, and format of program instruction. As shown in Table 3, effect sizes varied across grade levels but were largest for Grades 7 through 12 with a median effect of 0.13 (95% CI = −0.18 to 0.44), and studies that included students from across grade levels (g = 0.27, 95% CI = −0.23 to 0.80). The remaining median effect sizes were negligible.

Table 3

Median effect sizes for variables within learner control studies

Study feature	k	Median g	SD	Failsafe N	95% CI
Age of Learner
Grades K through 12	6	.13	.39	NA	−.18 to .44
College	14	.07	.36	NA	−.13 to .27
Adult education	1	.00	.37	NA	−.20 to .20
Across grades	2	.27	.39	1	−.23 to .80
Setting of Instruction
School	17	.05	.31	NA	−.09 to .19
Home	3	.23	.48	1	−.32 to .78
Other	3	.00	.43	NA	−.49 to .49
Subject of Instruction
Math	2	−.03	.12	NA	−.12 to .05
English/language	9	.19	.39	NA	−.07 to .45
Science	3	.00	.01	NA	−.01 to .01
Social studies/history	2	.42	.17	2	.31 to .54
Computers/technology	4	−.08	.19	NA	−.28 to .12
Other	3	.00	.55	NA	−.63 to .63
Format of Instruction
Drill & practice	1	.05	NA	NA	NA
Tutorial	16	.00	.31	NA	−.16 to .16
Comprehensive	6	.46	.36	8	.17 to .75
Outcome Measure
Behavioral	7	.19	.41	NA	−.12 to .50
Academic achievement	16	.00	.28	NA	−.14 to .14

Note. Failsafe N based on criterion effect of 0.20 and rounded to the nearest whole number. NA = not applicable.

Home was the setting for which the largest effect was found, but it was a small effect of g = 0.23 (95% CI = −0.32 to 0.78). Again, the other effects for settings were negligible. Social studies/history had the largest effect (g = 0.42, 95% CI = 0.31 to 0.54) of any of the subject areas and was one of only two effects in the entire study that did not include zero in its 95% CI. However, this effect size was based on only two studies. The remaining effects were either small (English/language arts g = 0.19) or negligible.

The format of instruction was examined by coding programs as a drill and practice tool, tutorial to teach one skill, or a comprehensive program covering several skills. Programs that were comprehensive in nature yielded the highest gains with a median effect of 0.46 (95% CI = 0.17 to 0.75), which was the largest effect size in the study. The other two median effect sizes were small. Finally, learner control within educational technology led to a larger effect for behavioral outcomes (g = 0.19, 95% CI = −0.12 to 0.50) than academic outcomes (g = 0.00, 95% CI = −0.14 to 0.14). However, the effects were both small and there were considerable overlap between the ranges.

Discussion

The current meta-analysis examined the effects of learner control within technology across several characteristics of instruction. The first research question focused on the overall effectiveness of learner control within educational technology. Consistent with previous research (Niemiec et al., 1996), effects for learner control within educational technology led to a small effect of g = .05. The findings suggest that the use of learner control within educational technology did not directly lead to increased outcomes for students.

The current study, in keeping with the previous meta-analysis (Niemiec et al., 1996), found near zero effects for all components of instruction (pacing, time, sequence, practice, review). Thus, there does not seem to be an advantage to giving the learner control over any particular instructional component. Previous research found that effects were greatest when the learner was older (Hannafin, 1984), but the current data suggested a somewhat stronger effect for students in K–12 as compared with college students or adult learners. Thus, the effect that age has on the effectiveness of learner control appears to be an area in need of additional research. Moreover, the relatively larger median effect for home-based educational technology with learner control suggests a potential interesting finding that might also warrant additional research given that the effect size was computed from only three studies.

As indicated in Table 2, programs that were comprehensive in nature led to the highest gains for students, which was consistent with previous research in that traditional interventions, practices, and strategies were found to be effective without the use of technology (e.g., Ehri et al., 2007; Torgesen & Barker, 1995; Torgesen et al., 2010; Wentink et al., 1997). Perhaps the most interesting finding when examining the effects of contextual factors was in relation to the outcome variable. Studies that measured effectiveness with an academic variable produced near zero effects when aggregated, but studies that used a behavioral variable as their outcome measure showed the higher effectiveness. The relatively higher effect for behavioral variables could suggest that the use of computers and technology may help improve motivation and engagement (e.g., time on task), but the effect on academic outcomes is less clear (Dynarski et al., 2007). Further research is warranted to examine not only how students learn on computers but also how we are measuring such learning.

Implications for Practice

Programs that offered a comprehensive approach had larger effects than practice-based applications, suggesting that educators should consider more comprehensive programs that provide the learner with a unique experience beyond what is commonly received in their classroom. For example, one study included in the current meta-analysis employed the structure of a video game to increase arithmetic skills (Shin, Sutherland, Norris, & Soloway, 2011). Participants created their own characters and identities while they solved various math problems on a game boy delivering instruction and practice across several skills. A separate study delivered instruction via a virtual learning environment, allowing the student to cycle through controlled hyperlinks to gain experience using information-technology skills (Chou & Liu, 2005). Educational technology within Social Studies and History had larger effects compared to other areas (e.g., math, reading, science). Thus, educators could consider adopting technologies designed to enhance Social Studies and History instruction. However, the effect size for Social Studies and History was based on only two studies, which is too small for confident changes in practice and suggested an area for future research.

Potential Theoretical Implications

More comprehensive meta-analyses of educational technology in general led to larger effects of approximately 0.30 to 0.35 (Hattie, 2009; Tamim et al., 2011). However, the consistently negligible findings from the current and previous meta-analysis on learner control within educational technology (Niemiec et al., 1996) suggest that any positive effects noted from the use of educational technology are likely not attributable to learner control. Research is needed to better understand the potential causal mechanisms explaining how educational technology affects student learning (Duncan, 2010), and technology allows educators to empower students to take control of their own learning (U.S. Department of Education, 2010). However, learner control can likely be ruled out as a potential causal mechanism for the positive effects of educational technology.

In a fairly recent review of online instruction within higher education (Larreamendy-Joerns & Leinhardt, 2006), the presentational view, performance-tutoring view, and epistemic-engagement view to stand-alone instruction were proposed to help better explain the higher effects found here for comprehensive programming. The presentational view refers to presenting information in a variety of ways by incorporating multimedia resources, which may be more attractive to learners but is not necessarily better. The performance-tutoring view emphasizes the importance of individualized instruction in providing guidance, structured tasks, and feedback. As an example, intelligent tutoring systems are discussed because they streamline data and allow instructors to analyze student learning on a level that enables them to identify and provide additional instruction for those students that are struggling. Within the traditional classroom, this level of response and feedback is usually only found with one-to-one tutoring although some argue that the benefits of unexpected interaction and feedback may not take place through online learning. The epistemic-engagement view relates to interaction and how providing experiences on the computer allows the instruction to become more student-centered.

The findings within the current meta-analysis suggest that it is possible to provide instruction that is student-centered by providing learner control within the task. Moreover, studies with behavioral variables had stronger effects than studies with academic variables, which suggested that the instructional technology was more attractive to the user and supported the presentational view. However, the small overall effects, especially for academic variables, question the importance of these two approaches to instruction. The performance-tutoring view was not directly studied in this meta-analysis and could provide a potential framework to hypothesize causal mechanisms, other than learner control, for the positive outcomes associated with educational technology.

Limitations

Some limitations were discussed above in reference to specific effects found for each of the three research questions. Other aspects of the current meta-analysis may limit the conclusions we can make about the effects reported. First, the effect sizes may be inflated due to the exclusion of studies that are unpublished. However, some caution was taken here by reporting the failsafe N for each effect where applicable. Second, there were no data reported for the differences between learners that were progressing normally and learners that struggled. Although there are several studies that explore this research question, no studies that differentiated the types of learners were included in the current meta-analysis. Exploring the effects of computer programming across the instructional components for students that struggle with content may prove beneficial in the future. Third, the duration of the study and duration of the session within each study were not examined within the current meta-analysis. Such information on exposure to the instruction may provide some insight into explaining varied effects across the studies and should be included in future research. Finally, single-case experimental designs were not included in the data. Future researchers may wish to replicate the methods used here with studies that use different designs.

Conclusion

There continues to be an increased use of educational technology in our schools with mixed results. The current meta-analysis examined components of learner control within educational technology and found mostly negligible effects on student outcome measures. Although overall effects of including learner control within educational technology produced near zero effects, some variables contributed to higher student outcomes. Factors that yielded effects with practical significance (Slavin, 1990) included using technology for English/Language and Social Studies/History as opposed to other subjects, and using a comprehensive approach rather than drill and practice. Moreover, studies with behavioral variables had larger effects than measures of academic achievement, which suggests that providing learner control within educational technology may enhance engagement, but may not increase student skills.

There is utility in reexamining technology because of its ever-changing nature (Cheung & Slavin, 2012). Thus, additional research regarding learner control is warranted. Future researchers should also continue to examine other underlying mechanisms related to the effectiveness of educational technology beyond learner control that more directly relate to positive student outcomes. Given the increasing use of educational technology, additional research on its effectiveness seems warranted.

Footnotes

Authors

ABBEY C. KARICH is a doctoral student in educational psychology at the University of Minnesota, 341 Education Science Building, 56 E. River Road, Minneapolis, MN 55455; e-mail: kari0060@umn.edu . Her research interests include technology enhanced instruction and adolescent literacy.

MATTHEW K. BURNS is a professor of educational psychology, coordinator of the School Psychology program, and co-director of the Minnesota Center for Reading Research at the University of Minnesota, 341 Education Science Building, 56 E. River Road, Minneapolis, MN 55455; e-mail: burns258@umn.edu . He has published over 150 articles and book chapters in national publications and has co-authored or co-edited 10 books. He is also the editor of School Psychology Review and past editor of Assessment for Effective Intervention. Specific areas in which he has conducted research include response to intervention, assessing the instructional level, academic interventions, and facilitating problem-solving teams.

KATHRIN E. MAKI is a doctoral student in educational psychology at the University of Minnesota, 341 Education Science Building, 56 E. River Road, Minneapolis, MN 55455; e-mail: makix312@umn.edu . She is a former special education teacher. Her current research interests include response-to-intervention frameworks, reading interventions, and identification of learning disabilities.

References

*Aly

Elen

Willems

(2005). Learne-control vs. program-control instructional multimedia: A comparison of two interactions when teaching principles of orthodontic appliances. European Journal of Dental Education, 9, 157–163. doi:10.1111/j.1600-0579.2005.00385.x

Burns

M. K.

(2012). Meta-analyses of single-case design research: Introduction to the special series. Journal of Behavioral Education, 21, 175–184. doi:10.1007/s10864-012-9158-9

Canino

Cicchelli

(1988). Cognitive styles, computerized treatments on mathematics achievement and reaction to treatments. Journal of Educational Computing Research, 4, 253–264. doi:10.2190/HX30-5354-JFUW-JK6T

*Chang

M. M.

C. M.

(2009). Effects of locus of control and learner-control on web-based language learning. Computer Assisted Language Learning, 22, 189–206. doi:10.1080/09588220902920094

Cheung

A. C.

Slavin

R. E.

(2012). How features of educational technology applications affect student reading outcomes: A meta-analysis. Educational Research Review, 7, 198–215. doi:10.1016/j.edurev.2012.05.002

*Chou

S. W.

Liu

C. H.

(2005). Learning effectiveness in a web-based virtual learning environment: A learner control perspective. Journal of Computer Assisted Learning, 21(1), 65–76. doi:10.1111/j.1365-2729.2005.00114.x

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.

*Corbalan

Kester

van Merriënboer

(2009). Dynamic task selection: Effects of feedback and learner control on efficiency and motivation. Learning and Instruction, 19, 455–465. doi:10.1016/j.learninstruc.2008.07.002

Duncan

(2010). Bang for the buck in schooling. The new normal: Doing more with less remarks of U.S. Secretary of Education Arne Duncan at the American Enterprise Institute Panel. Retrieved from http://www.aei.org/files/2010/11/17/20101117-Arne-Duncan-Remarks.pdf

10.

Dynarski

Agodini

Heaviside

Novak

Carey

Campuzano

(2007). Effectiveness of reading and mathematics software products: Findings from the first student cohort (Report to Congress; Publication No. NCEE 2007-4005). Washington, DC: U.S. Department of Education.

11.

Ehri

L. C.

Dreyer

L. G.

Flugman

Gross

(2007). Reading rescue: An effective tutoring intervention model for language-minority students who are struggling readers in first grade. American Educational Research Journal, 44, 414–448. doi:10.3102/0002831207302175

12.

Engle

R. A.

(2006). Framing interactions to foster generative learning: A situative account of transfer in a community of learners classroom. Journal of the Learning Sciences, 15, 451–498. doi:10.1207/s15327809jls1504_2

13.

Engle

R. A.

Nguyen

P. D.

Mendelson

(2011). The influence of framing on transfer: Initial evidence from a tutoring experience. Instructional Science, 39, 603–628. doi:10.1007/s11251-010-9145-2

14.

*Eom

Reiser

R. A.

(2000). The effects of self-regulation and instructional control on performance and motivation in computer-based instruction. International Journal of Instructional Media, 27, 247–260.

15.

Gersten

Fuchs

L. S.

Compton

Coyne

Greenwood

Innocenti

M. S.

(2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional children, 71, 149–164.

16.

Greaves

Hayes

Wilson

Gielniak

Peterson

(2010). The technology factor: Nine keys to student achievement and cost-effectiveness. Chicago, IL: Market Data Retrieval.

17.

Hammer

Elby

Scherr

R. E.

Redish

E. F.

(2005). Resources, framing, and transfer. In Mestre

(Ed.), Transfer of learning: Research and perspectives (pp. 89–119). Greenwich, CT: Information Age.

18.

Hannafin

M. J.

(1984). Guidelines for using locus of instructional control in the design of computer-assisted instruction. Journal of Instructional Development, 7(3), 6–10. doi:10.1007/BF02905753

19.

Hattie

(2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.

20.

Hedges

L. V.

(1983). A random effects model for effect sizes. Psychological Bulletin, 93, 388–395. doi:10.1037/0033-2909.93.2.388

21.

Hedges

L. V.

Olkin

(1985). Statistical methods for meta-analysis. London, England: Academic Press.

22.

*Hii

S. C.

Fong

S. F.

(2010). Effects of multimedia redundancy in history learning among deep and surface’ students. Asian Social Science, 6(6), 119.

23.

Hoaglin

D. C.

Mostellar

Tukey

J. W.

(Eds.). (1983). Understanding robust and exploratory data analysis (Vol. 3). New York, NY: Wiley.

24.

*Hwu

(2003). Learners’ behaviors in computer-based input activities elicited through tracking technologies. Computer Assisted Language Learning, 16, 5–29. doi:10.1076/call.16.1.5.15529

25.

Jitendra

A. K.

Burgess

Gajria

(2011). Cognitive strategy instruction for improving expository text comprehension of students with learning disabilities: The quality of evidence. Exceptional Children, 77, 135–159.

26.

*Johnson

E. P.

Perry

Shamir

(2010). Variability in reading ability gains as a function of computer-assisted instruction method of presentation. Computers & Education, 55, 209–217. doi:10.1016/j.compedu.2010.01.006

27.

*Kopcha

T. J.

Sullivan

(2008). Learner preferences and prior knowledge in learner-controlled computer-based instruction. Educational Technology Research and Development, 56, 265–286. doi:10.1007/s11423-007-9058-1

28.

Kulik

C. L. C.

Kulik

J. A.

(1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7, 75–94. doi:10.1016/0747-5632(91)90030-5

29.

Larreamendy-Joerns

Leinhardt

(2006). Going the distance with online education. Review of Educational Research, 76, 567–605. doi:10.3102/00346543076004567

30.

Lee

Wong

S. C.

(1989). Adaptive program vs. learner control strategy on computer-aided learning of gravimetric stoichiometry problems. Journal of Research on Computing in Education, 21, 367–379.

31.

Lepper

M. R.

(1985). Microcomputers in education: Motivational and social issues. American Psychologist, 40(1), 1–18. doi:10.1037/0003-066X.40.1.1

32.

*Levinson

A. J.

Weaver

Garside

McGinn

Norman

G. R.

(2007). Virtual reality and brain anatomy: A randomized trial of e-learning instructional designs. Medical Education, 41, 495–501. doi:10.1111/j.1365-2929.2006.02694.x

33.

Machin

McNally

Silva

(2007). New technology in schools: Is there a payoff? Economic Journal, 117, 1145–1167. doi:10.1111/j.1468-0297.2007.02070.x

34.

Mager

R. F.

(1964). Learner-controlled instruction: 1958–1964. Programmed Instruction, 4(2), 1, 8, 10–12.

35.

*Martin

(2008). Effects of practice in a linear and non-linear web-based learning environment. Educational Technology & Society, 11, 81–93.

36.

Marshall

Teeman

Mundy

Lin

Morrison

Yeshanew

Rudd

(2009). Harnessing technology schools survey 2009: Data report—Part 1, Descriptive analysis. Coventry, England: Becta.

37.

Merrill

M. D.

(1975). Learner control: Beyond aptitude-treatment interactions. AV Communications Review, 23, 217–226. doi:10.1007/BF02768379

38.

Merrill

M. D.

(1980). Learner control in computer-based learning. Computer & Education, 4, 143–150.

39.

Methe

S. A.

Kilgus

S. P.

Neiman

Riley-Tillman

(2012). Meta-analysis of interventions for basic mathematics computation in single-case research. Journal of Behavioral Education, 21, 230–253. doi:10.1007/s10864-012-9161-1

40.

*Mihalca

Salden

R. J.

Corbalan

Paas

Miclea

(2011). Effectiveness of cognitive-load based adaptive instruction in genetics education. Computers in Human Behavior, 27, 82–88. doi:10.1016/j.chb.2010.05.027

41.

Niemiec

R. P.

Sikorski

Walberg

H. J.

(1996). Learner-control effects: A review of reviews and a meta-analysis. Journal of Educational Computing Research, 15, 157–174. doi:10.2190/JV1U-EQ5P-X2PB-PDBA

42.

O’Brien

Scharber

(2010). Teaching old dogs new tricks: The luxury of digital abundance. Journal of Adolescent & Adult Literacy, 53, 600–603. doi:10.1598/JAAL.53.7.7

43.

Orwin

R. G.

(1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8, 157–159.

44.

*Piccoli

Ahmad

Ives

(2001). Web-based virtual learning environments: A research framework and a preliminary assessment of effectiveness in basic IT skills training. MIS Quarterly, 25, 401–426.

45.

Rosenthal

(1979). The “file drawer problem” and tolerance for null results. Psychological Bulletin, 86, 638–641. doi:10.1037/0033-2909.86.3.638

46.

Rosenthal

(1991). Meta-analytic procedures for social research (Revised ed.). Newbury Park, CA: Sage.

47.

Ross

S. M.

Morrison

G. R.

Lowther

D. L.

(2010). Educational technology research past and present: Balancing rigor and relevance to impact school learning. Comparative Educational Technology, 1, 17–35.

48.

*Salden

R. J.

Paas

Van der Pal

van Merriënboer

J. J.

(2006). Dynamic task selection in flight management system training. International Journal of Aviation Psychology, 16, 157–174. doi:10.1207/s15327108ijap1602_3

49.

Scheiter

Gerjets

(2007). Learner control in hypermedia environments. Educational Psychology Review, 19, 285–307. doi:10.1007/s10648-007-9046-3

50.

*Shin

Sutherland

L. M.

Norris

C. A.

Soloway

(2012). Effects of game technology on elementary student learning in mathematics. British Journal of Educational Technology, 43, 540–560. doi:10.1111/j.1467-8535.2011.01197.x

51.

Shyu

H. Y.

Brown

S. W.

(1992). Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning? International Journal of Instructional Media, 19, 85–96.

52.

Slavin

R. E.

(1990). IBM’s writing to read: Is it right for reading? Phi Delta Kappan, 72, 214–216.

53.

Sosa

Berger

D. E.

Saw

A. T.

Mary

J. C.

(2011). Effectiveness of computer-assisted instruction in statistics: A meta-analysis. Review of Educational Research, 81, 97–128. doi:10.3102/0034654310378174

54.

Sutherland

L. M.

Shin

McCall

K. L.

(2010). Integrating science inquiry, literacy, technology, and universal design for learning to enhance middle school students’ opportunities to learn science. Paper presented at the annual meeting of the National Association for Research in Science Teaching. Philadelphia, PA.

55.

Tamim

R. M.

Bernard

R. M.

Borokhovski

Abrami

P. C.

Schmid

R. F.

(2011). What forty years of research says about the impact of technology on learning: A second-order meta analysis and validation study. Review of Educational Research, 81, 4–28. doi:10.3102/0034654310393361

56.

Torgesen

J. K.

Barker

T. A.

(1995). Computers as aids in the prevention and remediation of reading disabilities. Learning Disability Quarterly, 18, 76–87.

57.

Torgesen

J. K.

Wagner

R. K.

Rashotte

C. A.

Herron

Lindamood

(2010). Computer-assisted instruction to prevent early reading difficulties in students at risk for dyslexia: Outcomes from two instructional approaches. Annals of Dyslexia, 60, 40–56. doi:10.1007/s11881-009-0032-y

58.

*Tullis

J. G.

Benjamin

A. S.

(2011). On the effectiveness of self-paced learning. Journal of Memory and Language, 64, 109–118. doi:10.1016/j.jml.2010.11.002

59.

U.S. Department of Education, Office of Educational Technology. (2010). Transforming American education: Learning powered by technology. Washington, DC: Education Publications Center.

60.

*Vandewaetere

Clarebout

(2011). Can instruction as such affect learning? The case of learner control. Computers & Education, 57, 2322–2332. doi:10.1016/j.compedu.2011.05.020

61.

*Wang

L. C. C.

Sutton

R. E.

(2002). Effects of learner control with advisement on English-as-a-second-language students in a web-based learning environment. International Journal of Instructional Media, 29, 317–324.

62.

Wentink

H. W.

Van Bon

W. H.

Schreuder

(1997). Training of poor readers’ phonological decoding skills: Evidence for syllable-bound processing. Reading and Writing, 9, 163–192. doi:10.1023/A:1007921805360

63.

Ysseldyke

J. E.

McLeod

(2007). Using technology tools to monitor response to intervention. In Jimerson

Burns

M. K.

VanDerHeyden

A. M.

(Eds.), Handbook of response to intervention (pp. 396–407). New York, NY: Springer. doi:10.1007/978-0-387-49053-3_29

64.

Ysseldyke

Spicuzza

Kosciolek

Teelucksingh

Boys

Lemkuil

(2003). Using a curriculum-based instructional management system to enhance math achievement in urban schools. Journal of Education for Students Placed at Risk, 8, 247–265. doi:10.1207/S15327671ESPR0802_4