Impact Results of the eMINTS Professional Development Validation Study

Abstract

This article presents the findings of an evaluation of the eMINTS (enhancing Missouri’s Instructional Networked Teaching Strategies) professional development program. eMINTS is an intensive teacher professional development program designed to promote inquiry-based learning, support high-quality lesson design, build community among students and teachers, and create technology-rich learning environments. This evaluation included 60 high-poverty rural schools across Missouri that were randomly assigned to two treatment conditions and a control condition, with approximately 200 teachers and 3,000 students in the 2011–2012 baseline academic year. The researchers conclude that after 3 years, the eMINTS treatment group and an eMINTS treatment group with an additional year of Intel support resulted in changed teacher instructional behaviors and increased student achievement in mathematics.

Keywords

educational policy educational reform leadership organization theory/change principals

Despite years of implementing standards-based accountability systems, many teachers today lack the necessary preparation to develop standards-based instructional strategies and inform decisions based on student assessments (Drake, 2007). Standards-based instruction necessitates the application of a new skill set for most teachers, and Drake (2007) contended that “often the challenge is not in the standards themselves but in our assumptions about how to work with them” (p. 3). The need to advance effective ways to train teachers, especially in high-need regions, to develop and use standards-based instruction and assessments is acute. To respond to this need, the eMINTS (enhancing Missouri’s Instructional Networked Teaching Strategies) National Center provides intensive professional development to teams of teachers, especially those serving high-need students. eMINTS is designed to create building-wide reform by helping teachers master the translation of Common Core or any state standards and information from assessments into engaging classroom practices that use technology.

The eMINTS professional development model emphasizes a number of features identified by research on teacher professional development (Penuel, Fishman, Yamaguchi, & Gallagher, 2007). First, the program provides intensive (240 hours) professional development sustained over a long period of time (2 years), which increases the likelihood of having impact (Garet, Porter, Desimone, Birman, & Yoon, 2001). Second, the program provides teachers with opportunities for “hands-on” work that connects to their daily instructional practice, an additional salient feature of effective professional development noted by Garet and colleagues (2001). Third, collective participation in this program promotes teacher communication and collaboration to support instructional changes (Darling-Hammond, Wei, Andree, Richardson, & Orphanos, 2009; Garet et al., 2001). eMINTS embeds these features by focusing on specific strategies—inquiry-based learning, high-quality lesson design, community of learners, and technology integration—to address issues they have identified as barriers to the consistent use of standards-based instruction. Table 1 describes how these four eMINTS strategies can be effectively implemented by teachers.

Table 1

Core Strategies of eMINTS Program

eMINTS strategy	To do this effectively, the teacher . . .
Inquiry-based learning	Constructs authentic, engaging, and student-centered units of study based on standards; promotes discovery, creativity, inventiveness, and innovation; creates a classroom environment where questioning promotes critical thinking and inquiry; and cultivates complex thinking and knowledge construction.
High-quality lesson design	Equips students to become self-directed learners, uses varied assessments aligned to standards, makes instructional decisions based on data, provides multiple opportunities for students to demonstrate proficiency, and addresses diverse learner needs.
Community of learners	Fosters a schoolwide culture of commitment to lifelong learning, establishes a classroom where students work effectively in teams, fosters an atmosphere of safety, and promotes the development of students as leaders.
Technology integration	Cultivates a global collaboration and creates opportunities for safe interactions with others outside the classroom; seamlessly integrates technology into the classroom as a fundamental tool for learning; models appropriate, responsible technology use; and prepares students to become information and media literate.

Note. eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies.

Inquiry-based learning activities require students to construct knowledge through meaningful investigations that require higher-order thinking skills such as reasoning analysis, judgment, and decision making (National Research Council, 2000). Instruction is typically framed around open-ended questions that are authentic and relevant to life outside the classroom (Hmelo-Silver, Duncan, & Chinn, 2007). Hmelo-Silver and colleagues (2007) noted that teachers play “a key role in facilitating the learning process and may provide content knowledge on a just-in-time basis” (p. 100). Students may learn how to complete tasks and why the tasks should be completed in a certain way through active thinking and drawing conclusions from data (Hmelo-Silver, 2006; Minner, Levy, & Century, 2010). There is a growing body of research suggesting that inquiry-based instructional strategies have resulted in student learning gains in both content areas and process skills (constructing and reflecting; Geier et al., 2008; Guthrie et al., 2004; Hickey, Wolfe, & Kindfeld, 2000; Langer, 2001; Wu & Tsai, 2005).

High-quality lesson design emphasizes that effective facilitation is based on the teacher’s ability to select instructional strategies to best meet the needs of learners (Tomlinson et al., 2003) and to help students build personal understanding of lesson content through processes such as reflection and metacognition (National Research Council, 2003). Wiggins and McTighe (1998) suggested that the use of multiple data sources enables teachers to improve student achievement through prioritization of instructional time, targeted instruction for struggling students, identification of student strengths, gauging instructional effectiveness of classroom lessons, and refinement of instructional methods (Hamilton et al., 2009). In addition, feedback from assessments can help students focus their own efforts on areas for growth (Heritage, 2007) and reasoning ability (Belland, Glazewski, & Richardson, 2011) and increase their achievement levels (Simons & Klein, 2007). Professional development focusing on lesson design enables teachers to plan standards-based instruction and inform instruction with formative and summative assessments.

Communities of learners support standards- and inquiry-based instruction in classrooms. Kohn (1994) cited multiple studies showing higher levels of achievement when students are part of a learning community. Such a community supports learning through respectful communication between teachers and students and among students and by exhibiting a positive regard for diversity (Pulakos, Arad, Donovan, & Plamondon, 2000). The open-minded atmosphere in classrooms, in which students feel comfortable taking risks and sharing ideas and experiences, allows for responsible social interactions that underpin the development of 21st-century life and career skills (Slavin, 2010). In addition, prior research suggests that students who learn in classrooms where decisions are made collaboratively display more creativity and higher-order thinking (Kohn, 2006).

Technology integration can support change in teacher instructional practices (Lawless & Pelligrino, 2007), most notably by influencing teachers’ pedagogical approaches and the content or context of learning (Mitchem, Wells, & Wells, 2003). Research suggests that technology be integrated into instructional design and practices, and the integration should align with the desired learning and teaching goals (Clark, 1994; Ross, Morrison, & Lowther, 2010; Tamim, Bernard, Borokhovski, Abrami, & Schmid, 2011). In particular, researchers advocate that the strength of technology lies in engaging student analytical thinking and problem solving rather than acting as a tool for content delivery (Hart, Allensworth, Lauen, & Gladden, 2002; Schmid et al., 2009).

The eMINTS program provides intensive and differentiated professional development grounded in constructivist pedagogy, technology resources and support to engage teachers and students, and accountability from colleagues and school leadership. These components are designed to support teachers in improving the extent to which their lesson plans are aligned to standards and grounded in assessment, incorporate more inquiry-based learning strategies, strengthen relationships with students by establishing a community of learners, and increase their own technology literacy.

Overview of the eMINTS Comprehensive Program

The overall goal of eMINTS is to help teachers develop student-centered, purposeful instruction fostered by technology utilization. The program addresses issues identified as barriers among teachers in the consistent use of standards-based instruction, student assessment information, and technology. The program includes a specific set of school and classroom technology equipment, intensive on-site training, online and face-to-face professional learning communities, and job-embedded coaching to enhance teachers’ classroom practices (Joyce & Showers, 1995).

Equipment Requirements

eMINTS classrooms must meet minimum hardware, software, Internet, and equipment connectivity requirements. An official eMINTS classroom includes an interactive whiteboard (e.g., SMART Board), LCD projector, teacher laptop, camera, printer/scanner, and, for this study, at least a 1:1 ratio of students to computers. Specific Internet and equipment connectivity requirements must be met in an eMINTS classroom to ensure proper instructional functionality. Connectivity requirements may be met using either wired or wireless connectivity configurations or may be a combination of the two.

Professional Development

eMINTS Comprehensive Program consists of training for school principals, district and school technology coordinators, and classroom teachers. Before the start of the school year, a certified eMINTS instructional specialist is assigned to a collection of schools according to the instructional specialist’s geographic location to provide formal and individualized training to principals and teachers. Along with these training responsibilities, the instructional specialist facilitates the development of a school-based leadership team within each school to support implementation and ensure that the required technology infrastructure and equipment functionality is maintained in eMINTS classrooms. The instructional specialists train principals and technology specialists to monitor and support eMINTS teachers’ instructional improvements. Specifically, early in the first training year, the instructional specialist introduces principals to constructivist theories underlying the eMINTS model and their implications for classroom teaching and learning. They introduce principals to an observation rubric for rating eMINTS teachers’ instructional performance according to eMINTS standards.

School technology coordinators are trained to understand eMINTS pedagogy and how technology is intended to support instruction. A certified eMINTS consultant conducts two 2-hour online sessions with coordinators (4 hours total) during each school year to introduce technology coordinators to eMINTS pedagogy and train them on equipment maintenance.

The bulk of the program, however, is the development of teachers through the provision of approximately 240 hours of face-to-face professional development over the course of 2 academic years to design high-quality inquiry-based lesson plans, implement inquiry-based learning strategies, build community among teachers and students, and integrate technology into classroom instruction. During the first year of training, teachers receive about 125 hours of formal training in 26 sessions that are held throughout the school year. Training sessions typically take place in a computer lab in a central location and last between 4 and 6.5 hours each. The first-year curriculum focuses on understanding constructivist pedagogy, community-building strategies (including interaction and interdependence), inquiry-based learning strategies, technology integration, and introducing authentic learning experiences into the classroom. At the end of the first training year, teachers spend up to 12 additional hours developing a classroom website with the help of the instructional specialist.

During the second year of training, the specialist conducts 88 hours of training in 20 sessions. The Year 2 curriculum focuses on classroom management, website enhancement, assessment, interdisciplinary teaching and learning, and development of multimedia and online projects. One session is reserved for teachers to travel to an eMINTS school and observe a certified eMINTS teacher. During both training years, the specialist supplements these formal training sessions with 9 to 10 on-site and individualized coaching sessions (about 14 hours total) and within-building communities of practice where teaching staff meet to share ideas, collaborate on online project development, and deepen their existing understanding of concepts embedded in the eMINTS training. Finally, eMINTS provides teachers with written curricula and just-in-time learning opportunities via online courses to help teachers improve their practice over time.

School Leadership

School leadership plays a critical role in facilitating instructional improvement through teacher professional development, which ultimately leads to gains in student learning (Ertmer & Ottenbreit-Leftwich, 2010; Hallinger & Heck, 1996; Lindstrom & Speck, 2004). Bredeson and Johansson (2000) identified two areas where principals may have a substantial impact on teacher professional development: creating a learning environment and assessing professional development outcomes. Guided by the literature, eMINTS embeds school leadership support into its model. First, the program provides a professional development session to school leaders on the topic of constructivist theories and technology integration underlying eMINTS model. Second, school leaders meet with eMINTS consultants to discuss how to support implementation and integrate eMINTS with other school initiatives. This activity is designed to increase principals’ ability to view the school and instructional practices through an eMINTS lens and identify areas for program improvement. Third, school leaders are introduced to an observation rubric for supervising and supporting teachers’ instructional performance according to eMINTS standards. These efforts promote principal support for eMINTS implementation and maintain a schoolwide learning environment for teachers.

Prior Research on eMINTS

Since its inception in 1999, annual external evaluations of eMINTS have been on teacher and student outcomes. Qualitative research and formative evaluations also contributed to a better understanding of the facilitating factors and challenges associated with school and classroom implementation of eMINTS. None of the research prior to this evaluation, however, meets What Works Clearinghouse (WWC) standards without or with reservations.

eMINTS external program evaluations conducted from 2002 through 2005 used a quasi-experimental design that compared the performance of students in eMINTS classrooms with performance of students in non-eMINTS classrooms. These evaluations consistently found that intermediate elementary (Grades 3–6) students enrolled in eMINTS classrooms significantly outperformed students enrolled in non-eMINTS classrooms on Missouri’s state standardized performance assessment, the Missouri Assessment Program (MAP), in communication arts (i.e., English/language arts), mathematics, science, and social studies. These results primarily pertained to students in Grade 3 communication arts and science and Grade 4 mathematics and social studies, with small sample sizes suggesting similar results may exist at Grades 5 and 6 (Office of Social and Economic Data Analysis [OSEDA], 2005). OSEDA analyses were conducted using student achievement data from MAP to compare the percentage of students attaining proficient and advanced levels of achievement in eMINTS classrooms with the percentage of students reaching those levels in non-eMINTS classrooms. A larger percentage of eMINTS students attained proficiency or advanced levels of achievement than did non-eMINTS students in communication arts from 2002 to 2005, the difference being statistically significant at the .05 level from 2003 to 2005. Mathematics results are similar, with the only exception being 2004, when non-eMINTS students had a slightly (0.4%) higher rate of proficiency. The other 3 years of mathematics assessment data indicate statistically significant differences in favor of eMINTS students.

More recent evaluations focused on schools that received competitive Title II.D Enhancing Education Through Technology (EETT) grant awards in Missouri. These reports of a more established eMINTS program extended to Grades 5 and 6, where students in eMINTS classrooms consistently attained higher rates of proficiency or advanced levels in all grades (3–6) in communication arts and mathematics, with significant results at the .01 level in most comparisons, including Grades 5 and 6 (Strother, Martin, & Dechaume, 2006).

Results of perhaps the strongest previous evaluation of eMINTS utilized a matched-schools design that analyzed student outcomes in math and communication arts for 2 years. Martin, Strother, and Reitzes (2009) found that students assigned to eMINTS classrooms during both years significantly outperformed students assigned to non-eMINTS classrooms for both years at Grade 5 in communication arts (p < .05) and Grade 6 in communication arts (p < .05) and mathematics (p < .001). In addition, scores of students having 2 years with eMINTS teachers were significantly greater than those of students having an eMINTS teacher for only 1 year in Grade 6 communication arts (p < .01) and Grade 6 mathematics (p < .001). Moreover, the variance explained by having two eMINTS teachers was sizeable, especially for mathematics (23.8%).

Collectively, the studies of eMINTS on teacher and student outcomes suggest that the program makes a difference. Teachers appear to change practices, and students seem to achieve at higher levels. But none of the studies is a rigorous quasi-experimental design or randomized controlled trial. In addition, the studies overall lack a focus beyond elementary schools; that is, the studies include elementary students, ranging from Grades 3 to 6, and the participating schools represent multiple states and populations (e.g., rural and urban settings).

A Third Year: eMINTS + Intel® Teach

Prior to the study, one third of the study schools were randomly assigned to receive eMINTS plus an additional year of professional development that used the Intel® Teach courses and online tools to further enhance teachers’ technology integration skills. The specific set of online resources for those teachers was designed to support the four eMINTS components. The third year of professional development was expected to accelerate outcomes for teachers and students by providing teachers with additional professional development and Web-based tools to build on what they learned during the first 2 years of the program (for more details, see Brandt, Meyers, & Molefe, 2012; Brandt, Meyers, Molefe, Dhillon, & Zhu, 2013).

Study Context

Missouri’s rural K–12 student population is nearly one quarter of a million students, the 18th largest in the United States. Poverty is high, with 44% of Missouri rural students qualifying for free or reduced-price lunch (Strange, Johnson, Showalter, & Klein, 2012). The Institute of Education Sciences (IES; 2010) reported that only 19% of Missouri students eligible for free or reduced-price lunch attained proficiency or better on the 2009 National Assessment of Education Progress (NAEP) mathematics test. Student mobility in Missouri rural schools ranks 14th highest in the United States, with 12.4% of families reporting changing residences in the 15 months prior to being surveyed (Strange et al., 2012). Per-pupil expenditure for Missouri rural schools is the 13th lowest in the United States (Strange et al., 2012). The attributes of many rural communities contribute to the scarcity of qualified teachers for rural schools nationally and in Missouri (Monk, 2007).

Middle schools (typically Grades 6–8) provide additional challenges. Middle school represents a critical time for students to develop the knowledge and skills they will need to achieve college and career readiness. Unfortunately, middle school also is the period in which students may begin to lag in academic performance. Lembke and Gonzales (2006) reported that the performance of United States middle school students is lower than that of their peers in other countries, particularly when tested on tasks embedded in 21st-century skills. Tasks of this nature commonly require skills cited by groups such as the Partnership for 21st-Century Skills (Bellanca & Brandt, 2010) as being associated with problem solving, communication, collaboration, creativity and innovation, and use of information technology.

Study Overview

A decade of evaluation of eMINTS has shown promise in changing teachers’ practice and raising student achievement. But these evaluation studies typically focused on intermediate elementary students (Grades 3–6) with either nonequivalent groups or pre–post designs without comparison groups. This randomized controlled trial evaluates the efficacy of eMINTS professional development in increasing seventh- and eighth-grade students’ communication arts and mathematics achievement.¹ This study includes approximately 200 teachers and 3,000 students in 60 high-poverty rural schools across Missouri. This study departs from prior studies on eMINTS both in terms of its target population and rigorous design. This article focuses on the impact of the eMINTS program after 3 years—that is impact 1 year after program implementation—on teacher instructional practices, student engagement, and student achievement in communication arts, mathematics, and 21st-century skills. Specifically, this article addresses the following research questions:

Research Question 1: What is the impact of the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach on seventh- and eighth-grade teachers’ instructional practices?

Research Question 2: What is the impact of the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach on school mean seventh- and eighth-grade performance in mathematics, communication arts, and 21st-century skills?

Research Question 3: What is the impact of the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach on school mean engagement of seventh- and eighth-grade students?

Sixty schools were randomly assigned to one of three groups: Treatment 1 (eMINTS), Treatment 2 (eMINTS + Intel), or control. To determine school assignments, schools were blocked on grade configuration: 30 PK–8 schools; 8 schools enrolling students in Grades 5 to 8, 6 to 8, or 7 to 8; and 22 schools enrolling students in Grades 6 to 12 or 7 to 12. Table 2 illustrates the school assignment and treatment timeline.

Table 2

School Assignment and Treatment Timeline

Summer 2010	Fall 2010 (Year 0)		2011–2012 (Year 1)	2012–2013 (Year 2)	2013–2014(Year 3)
Recruit 60 schools for participation	Random assignment of schools	eMINTS (n = 20)	eMINTS	eMINTS	No additional professional development
		eMINTS + Intel (n = 20)	eMINTS	eMINTS	Intel Teach
		Control (n = 20)	Business as usual	Business as usual	Business as usual

Note. eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies.

Baseline Equivalence

Random assignment, particularly when the sample sizes in each group are large, is expected to result in statistically equivalent groups on all observable and unobservable variables (Bloom, 2006). Group equivalence strengthens our capability to attribute any observable differences in outcomes between the groups in the intervention. Although it is not possible to test equivalence for unobservable variables, baseline equivalence of the groups can be assessed for variables on which data are available. In this section, we examine the initial equivalence of the three groups. For student analyses, differences in baseline student characteristics are estimated using a two-level model on the pooled sample of seventh- and eighth-grade students, adjusting for student’s grade in Year 3. Differences were estimated separately by randomization block and then pooled into an overall weighted average, using the number of study schools in each block as the weight. Baseline comparisons of teacher characteristics were conducted in the same way except a single-level regression model was used. Baseline school characteristics also used a single-level regression model but did not include a grade indicator as a predictor. The chi-square test of independence was used to test group equivalence for categorical variables, and the two-sided t test for equality of means was used for continuous variables.

School Characteristics

Of the 60 schools in the study at the time of random assignment, 30 served Grades PK–8 or K–8, eight schools served Grades 6–8 or 7–8, and the remaining 22 schools served Grades 6–12 or 7–12. Across all the school demographic characteristics examined, the results showed that treatment and control schools were relatively similar (Table 3). In previous study years, however, when the two treatment groups were considered as one treatment group based on their progress in the program, school enrollment was significantly larger (approximately 50 students) in treatment schools. Across groups, about 4% to 7% of students were minorities, approximately 60% qualified for free or reduced-price lunch, 1% to 2.5% were English language learners, and between 12% and 14% had an identified learning disability. Teacher credentials and experience were also similar. Average state test results among seventh- and eighth-grade students on the MAP mathematics and communication arts assessments were not significant although nearing the .05 level of difference. It should also be noted that in the previous year when the treatment schools were considered collectively, eMINTS schools had significantly lower communication arts scores than did control schools, with a difference of about 7 test points.

Table 3

Characteristics of Study Schools: 2010–2011 (Before Year 1 Implementation)

Characteristics^a	eMINTS + Intel	eMINTS	Control	p value
Number of schools	20	20	20
Average total school enrollment	183.2	180.2	128.8	.116
Average percentage of students eligible for free or reduced-price lunch	59.7	60.8	59.6	.938
Average percentage of non-White students	7.3	3.7	5.8	.227
Average percentage of male students	51.5	49.6	49.9	.262
Average percentage of LEP students	2.5	1.2	1.2	.648
Average percentage of students with a disability	13.7	12.8	12.3	.547
Average years of teaching experience	11.7	12.5	10.9	.216
Average percentage of teachers with master’s degree	43.1	42.9	42.5	.992
Mean scale score in spring 2011 MAP communication arts
Grade 7	675.2(SD = 11.1; n^b = 19)	684.0(SD = 15.1; n = 18)	678.1(SD = 10.3; n = 20)	.064
Grade 8	691.7(SD = 10.6; n = 19)	692.9(SD = 15.4; n = 17)	699.4(SD = 9.4; n = 19)	.083
Mean scale score in spring 2011 MAP mathematics
Grade 7	685.5(SD = 13.8; n = 19)	691.2(SD = 16.5; n = 18)	682.3(SD = 13.8; n = 20)	.132
Grade 8	706.1(SD = 10.0; n = 19)	704.4(SD = 16.0; n = 17)	708.3(SD = 12.6; n = 19)	.633

Note. From authors’ analysis based on data from the National Center for Education Statistics Common Core of Data 2010–2011 and DESE. MAP = Missouri Assessment Program. eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies; LEP = limited English proficient; DESE = Missouri Department of Elementary and Secondary Education.

Means were regression adjusted to account for block effects using ordinary least squares regression. For each characteristic, the p value is from an overall chi-square test of whether the means of the three treatment groups (eMINTS + Intel, eMINTS, and control) differed more than what might be expected by chance.

When data are missing, n is the actual number of teachers used to calculate the average characteristic in each treatment group. Missing data on the spring 2011 school mean MAP scores in communication arts and mathematics result from the fact that DESE does not report grade-level school means when fewer than five students in a grade took the test. As the table shows, fewer than five seventh-grade students took the test in two eMINTS schools and one control school, and fewer than five eighth-grade students took the test in one eMINTS + Intel schools, three eMINTS schools, and one control school.

Teacher Characteristics

The demographic characteristics of participating Grades 7 and 8 mathematics teachers are summarized in Table 4 (results for communication arts teachers are similar). In mathematics, the percentage of male teachers in the eMINTS + Intel group was the highest at 29.4% versus eMINTS at 11.6% and control group at 24.8%. In addition, the percentage of eMINTS + Intel mathematics teachers with a master’s degree was highest at 40.9% versus 23.8 for eMINTS teachers and 27.5% for control teachers. eMINTS + Intel teachers had the least experience, however, averaging 7.7 years, whereas eMINTS teachers averaged 9.3 years and control teachers averaged 9.1 years, respectively. None of these group differences was significant, however.

Table 4

Characteristics of Teachers in the MAP Mathematics Analytic Sample: 2010–2011 (Before Year 1 Implementation)

Characteristics	eMINTS + Intel	eMINTS	Control	p value
Number of teachers	37	34	29
Percentage male	29.4 (n = 37)	11.6 (n = 34)	24.8 (n = 29)	.194
Percentage with graduate degree	40.9 (n = 27)	23.8 (n = 29)	27.5 (n = 22)	.368
Years of teaching experience	7.7 (n = 27)	9.3 (n = 29)	9.1 (n = 22)	.701

Note. MAP = Missouri Assessment Program; eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies.

Student Characteristics

Table 5 shows the characteristics of the seventh- and eighth-grade MAP mathematics analytic student samples in the study groups (results for the communication arts student analytic group are similar). A joint test of overall differences suggests that the student characteristics of eMINTS, eMINTS + Intel, and control group students are equivalent on observed variables. Joint tests of overall differences on student engagement and 21st-century skills analytic samples suggest the same, although those samples are smaller because they required student assent and parent consent.

Table 5

Characteristics of Students in the MAP Mathematics Analytic Sample: 2010–2011 (Before Year 1 Implementation)

Characteristics	eMINTS + Intel	eMINTS	Control	p value
Number of students	1,141	1,108	823
2011 MAP mathematics z scores	−0.037 (n = 942)	0.068 (n = 947)	−0.006 (n = 669)	.562
2011 MAP communication arts z scores	−0.017 (n = 941)	0.033 (n = 947)	−0.023 (n = 669)	.775
Percentage eligible for free or reduced-price lunch	0.563 (n = 946)	0.610 (n = 949)	0.571 (n = 669)	.505
Percentage White	0.929 (n = 946)	0.953 (n = 949)	0.941 (n = 669)	.634
Percentage with disability	0.126 (n = 946)	0.141 (n = 949)	0.120 (n = 669)	.642
Percentage LEP	0.027 (n = 946)	0.000 (n = 949)	0.019 (n = 669)	.287
Percentage male	0.482 (n = 1,118)	0.539 (n = 1,081)	0.511 (n = 807)	.106

Note. From authors’ analysis based on student baseline data collected from study districts in spring 2011, when Grade 7 students were in Grade 4 and Grade 8 students were in Grade 5. MAP = Missouri Assessment Program; eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies; LEP = limited English proficient. Means were regression adjusted to account for block effects, grade, and clustering of students within schools, and weighted by the number of schools in each block. For each characteristic, the p value is from an overall chi-square test of whether the (weighted) means of the three treatment groups (eMINTS + Intel, eMINTS, and control) differed more than what might be expected by chance. Because of zero counts for LEP in some cells, the percentage LEP was estimated on the whole sample (instead of separately by blocks). When data are missing, n is the actual number of students used to calculate the average characteristic in each treatment group.

Differential Attrition

Differential attrition, or differences in the number of participants lost from the treatment and control groups, can introduce violations to the critical assumption of baseline equivalence in experimental designs (McKnight, McKnight, Sidani, & Figueredo, 2007). If severe enough, it can result in seriously biased impact estimates. When data can be assumed missing completely at random or missing at random, then differential attrition is less problematic because the participants who dropped out can be assumed to be representative of the original sample population.

During the study’s first year, two of the original 60 eMINTS schools dropped out. The first school dropped out about 4 months after random assignment was announced and shortly after baseline data collection began. A second school dropped out during the summer before Year 1 implementation. A third school was closed after the 2011–2012 school year (Year 1), and students were reassigned to one of several districts in the area. Moreover, the following number of schools could not be included in the analyses because they had no data on a particular outcome: two schools for the student engagement survey analysis, and one school each for the teacher survey and the classroom observation analyses. Because we were able to access MAP scores from the two schools that dropped out, our intent-to-treat analysis of MAP outcomes includes 59 schools; however, all other student and teacher analyses include 56 schools at most.²

Data Sources

Teacher Survey

The majority of the items measuring classroom-level community of learners and inquiry-based learning practices came from SRI’s Teaching and Learning Research Survey (Shear et al., 2010) and the National Survey of High School Reform and Project Based Learning (Buck Institute for Education, 2007). Items measuring technology integration primarily came from the eMINTS Technology Literacy Survey (eMINTS National Center, University of Missouri, 2009) and the Intel Teach Program Essentials Course Impact Survey (Martin & Shulman, 2006). Items to measure high-quality lesson planning were developed by the research team or pulled from SRI’s survey. We also used a minimum number of items from other publicly available surveys. Items are designed to measure teachers’ pedagogical beliefs (e.g., beliefs and practices aligned with an inquiry-based, constructivist approach), approaches to lesson planning, and instructional practices. Content validity was established for this survey in spring 2011. Baseline survey results were used to assess item reliabilities using the Rasch dichotomous model (Rasch, 1980; Wright & Masters, 1982). Internal reliability estimates for each domain ranged between .83 and .94.

Classroom Observations

Classroom observations were conducted each spring in 2011, 2012, 2013, and 2014 to measure teaching outcomes across the four eMINTS model components. Each Grade 7 and Grade 8 communication arts and mathematics class was observed once per year,³ where each observation consisted of at least one 10-minute segment, with majority of the classes observed in two 10-minute segments.

The observation protocol measures teaching practices, including inquiry-based learning, collaborative teaching, student and teacher use of technology, use of community resources, classroom organization, instructional support, and emotional and relational support. To measure changes in a community of learners and inquiry-based learning constructs, we used the Classroom Assessment Scoring System–Secondary (CLASS-S; Pianta, La Paro, & Hamre, 2008). CLASS-S is a theoretically driven and empirically supported conceptualization of classroom interactions organized into three major domains: emotional supports, classroom organization, and instructional supports (Allen, Pianta, Gregory, Mikami, & Lun, 2011). Certified scorers rated classroom observations within the domains and subdomains on a scale from 1 to 7, as advanced by the creators of the CLASS-S protocol. Because the CLASS-S is conducted in classrooms as teachers lead instruction of students, the observation measure for community of learners primarily captures teacher-to-student and student-to-student interactions.

In addition to CLASS-S, we conducted a literature review of technology use and technology integration literature to establish a conceptual framework. After discussion with eMINTS personnel and a review of eMINTS material, we created a brief observation protocol to measure technology use in the classroom by the teacher and students and used preexisting observation tools to measure technology integration in the classroom (Northwest Regional Educational Laboratory, 2005; West Virginia Department of Education, n.d.). In spring 2011, prior to baseline data collection, instructional content experts and eMINTS staff members reviewed the complete observation protocol, which included CLASS-S and these additional technology items, to establish content validity.

Student Engagement Survey

The student engagement survey was used to measure student perspectives on engagement in school, academic efficacy, the relevance of school for future success, educational aspirations, self-directed learning, and perceptions of emotional and relationship support from teachers. Items were selected primarily from the University of Chicago’s Consortium on Chicago School Research student survey (2007a, 2007b). We also reviewed student surveys used in the New Hope Study (Huston et al., 2003), the Research Assessment Package for Schools (Midgley et al., 2000), and other scales of student efficacy and engagement with strong psychometric properties, including Cook et al. (1996), Middleton and Midgley (2002), and Zimmerman, Bandura, and Martinez-Pons (1992). The reported reliability estimates for the scale are .80 or higher.

MAP

Primary student outcome data include annual MAP results for Grade 7 and Grade 8 students in communication arts and mathematics. MAP assessments are norm referenced and administered annually in the spring of each school year to students in Grades 3 to 8. Designed to measure student acquisition of skills and knowledge as described in Missouri’s Grade-Level Expectations, the assessment provides information on academic achievement at the student, class, school, and district levels. Student data are provided both as scale scores and performance level (e.g., Below Basic, Basic, Proficient, and Advanced; CTB/McGraw-Hill, 2012).

21st-Century Skills Assessment

To measure students’ skills in areas identified as 21st-century skills, we used the 21st-Century Skills Assessment, which is a 72-item criterion-referenced assessment used to measure the six International Society for Technology in Education (ISTE; 2012) National Educational Technology Standards for Students. A Rasch analysis indicated that the instrument demonstrated high levels of reliability and validity (consistently greater than .90 for pretest/posttest analyses) and only one construct (i.e., technology literacy) (Condon, Dawson, Molefe, & Swanlund, 2009).

Analytic Methods

This study used a cluster randomized design that randomly assigned 60 high-poverty rural Missouri middle schools to one of three groups in fall 2010 (Year 0). Schools assigned to Group 1 received the eMINTS 2-year professional development program (eMINTS) in 2011–2012 and 2012–2013 (Years 1 and 2); Group 2 schools received eMINTS 2-year professional development program in Years 1 and 2 and additional professional development in the form of Intel® Teach (eMINTS + Intel Teach) in 2013–2014 (Year 3); and Group 3 schools conducted business as usual (BAU), with no exposure to eMINTS or Intel Teach in Years 1 through 3, and then receive eMINTS 2-year professional development program in 2014–2015 and 2015–2016 (Years 4 and 5). To remove extraneous variation due to differences in grade configuration, randomization was blocked by the grade range in each school (Block 1: PK–8 schools; Block 2: 5–8, 6–8, 7–8 schools; Block 3: 6–12, 7–12 schools). The randomization of schools to treatment conditions provides a strong framework with which to accurately estimate program impact. This is because the randomization of a sufficient number of participants to one of three conditions should, on average, equalize any measured and unmeasured baseline differences among the three groups that may confound impact estimates. Next, we explain how we analyzed student and teacher data. For both student and teacher outcomes, analyses were conducted to examine the difference between Year 3 results and pretest measures collected at baseline. Prior years’ results can be found in previous reports (Brandt, Meyers, & Molefe, 2012; Brandt, Meyers, Molefe, Dhillon, & Zhu, 2013).

Analysis of Student Outcomes

Data were not imputed for missing outcomes on any of the student outcomes examined. There were no missing values for student characteristics or school means for any of the analytic samples. For missing teacher characteristics and missing pretest data, the dummy variable approach advanced by Puma, Olsen, Bell, and Price (2009) was used to impute these missing covariate values. Student analyses used two-level hierarchical linear models with students nested within schools. Means and differences were regression adjusted to account for blocked effects, grade, and clustering of students within schools and weighted by the number of schools in each block. Effect sizes were calculated separately by block and then pooled into an overall effect size weighted by the number of schools in each block. Impact results for each domain are calculated separately as pooled averages of scores collected from seventh- and eighth-grade students.

Level 1 (students):

Y_{i j} = π_{0 j} + π_{1 j} Grad e_{i j} + X_{i j} π_{2 j} + C_{i j} π_{3 j} + ε_{i j} .

Level 2 (schools):

π_{0 j} = \sum_{k} β_{00 k} D_{k} + \sum_{k} β_{01 k} T_{j} \times D_{k} + r_{0 j},

π_{1 j} = β_{10}, π_{2 j} = β_{20}, π_{3 j} = β_{30},

where $Y_{i j}$ is the outcome (i.e., communication arts or mathematics test score or measures of academic orientation) of student i in school j; $D_{k}$ = 1, if school j is in randomization block k, and 0 otherwise, $k = 1, \dots, 3$ ; $T_{j}$ = 1, if school j belongs to the eMINTS or eMINTS + Intel group, and 0 otherwise; ${Grade}_{ij}$ is a grade-level indicator that is equal to 1 for eighth-grade students and 0 for seventh-grade students (centered around the grand mean); $X_{i j}$ is a row vector of background characteristics (e.g., prior achievement) of student i in school j; $C_{i j}$ is a row vector of teacher characteristics (e.g., teaching experience) for student i in school j; $Z_{i j}$ is a row vector of school characteristics (e.g., percentage of minority students, school size) for school j; $ε_{i j}$ and $r_{0 j}$ are random residuals at the student and school levels, respectively. ${Grade}_{i j}$ was centered around the grand mean (i.e., the overall proportion of eighth graders in the sample) to control for differences in the proportions of seventh and eighth graders across schools.

With the above equations, $β_{00}$ is the overall average achievement level of seventh and eighth graders, and $β_{10}$ is the average achievement-level gap between seventh and eighth graders. The mean of the block-specific impact estimates $β_{01 k}$ ( $k = 1, \dots, 3$ ), weighted by the number of schools in each block, is an estimate of the overall average impact of eMINTS on the performance of seventh and eighth graders in a particular subject area relative to the control group. A test of the null hypothesis that $β_{01} = 0$ is a test of whether eMINTS has a statistically significant impact.

Analysis of Teacher Outcomes

Single-level rather than multilevel models were used to measure teacher outcomes because several schools in the analytic samples had only one teacher, precluding estimation of either the between-school variance or the block-specific impact estimates. Means and impacts reported on domains from the teacher survey and teacher observations were regression adjusted using ordinary least squares to account for block effects and baseline teacher and classroom characteristics and weighted by the number of schools in each block. Effect sizes were calculated separately by block and then pooled into an overall effect size weighted by the number of schools in each block. Block-specific effect sizes were computed using standardized mean differences (Hedges’s g). The p values are from a two-tailed test of the null hypothesis of equality of eMINTS and control means. Teacher survey and classroom observation data for inquiry-based learning strategies, a community of learners, and technology integration were analyzed separately. High-quality lesson design (e.g., teachers’ use of data and standards to guide lesson planning) was examined using results from the teacher survey only.

Teacher Survey Analysis

The teacher survey was administered in spring 2011, 2012, and 2013. Teacher outcomes on each construct—a community of learners, inquiry-based learning, high-quality lesson design, and technology integration—are the logit ability measures from a Rasch analysis of the survey items belonging to each construct. The Rasch analysis was conducted separately for mathematics and communication arts teachers. Teachers who taught both subjects were included in both the mathematics and communication arts analyses, and their scores from the two subjects were averaged. Analysis of impact of eMINTS on teacher survey logit scores was then conducted on the pooled sample of mathematics and communication arts teachers.

Classroom Observation Analysis

Classroom observations of teachers were conducted with CLASS-S certified observers in spring 2011, 2012, 2013, and 2014. As in the teacher survey outcomes, classroom observation outcomes on each of three constructs⁴—a community of learners, inquiry-based learning, and technology integration—were placed on a Rasch scale for comparison. Each classroom observation consisted of one to four 10-minute observation segments, with the majority (about 90%) of the teachers observed in two 10-minute segments. The Rasch analysis was conducted separately by segment but limited to the first two segments per observation for consistency, and then the resulting logit scores were averaged across the two segments. Analyses of impacts on classroom observation scores were conducted on the pooled sample of mathematics and communication arts teachers.

Study Findings

Teacher Survey Results

According to our analyses of teacher self-report data (from the spring 2014 teacher survey) in Table 6, no statistically significant differences arose between treatment and control teachers for a community of learners after Year 3. The eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach teachers reported significantly higher levels of inquiry-based learning (effect sizes of 0.73 and 0.96, respectively)⁵ and technology integration (effect sizes of 1.43 and 1.56, respectively) relative to the control group. The eMINTS Comprehensive Program + Intel Teach group also reported significantly higher levels of high-quality lesson design than their control group counterparts. However, no significant differences for any domain were found between the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach.

Table 6

Overall Impact of eMINTS on Four Teacher Survey Outcomes in Implementation Year 3 (2013–2014)

Outcome^a	Mean^b			Estimated impact^b
Outcome^a	eMINTS + Intel	eMINTS	Control	Impact	SE	p value	Effect size^c
Community of learners
eMINTS vs. control	—	2.115 (n = 24)^d	1.410 (n = 38)	0.705	0.860	.412	0.237
eMINTS + Intel vs. control	2.561 (n = 35)	—	1.410 (n = 38)	1.151	0.727	.113	0.346
eMINTS + Intel vs. eMINTS	2.561 (n = 35)	2.115 (n = 24)	—	0.445	0.837	.595	0.126
Inquiry-based learning
eMINTS vs. control	—	−0.042 (n = 24)	−0.632 (n = 37)	0.591	0.135	.000*	0.730
eMINTS + Intel vs. control	0.046 (n = 35)	—	−0.632 (n = 37)	0.679	0.114	.000*	0.959
eMINTS + Intel vs. eMINTS	0.046 (n = 35)	−0.042 (n = 24)	—	0.088	0.131	.501	0.122
High-quality lesson design
eMINTS vs. control	—	0.565 (n = 24)	0.211 (n = 37)	0.355	0.237	.135	0.371
eMINTS + Intel vs. control	0.702 (n = 35)	—	0.211 (n = 37)	0.491	0.201	.015*	0.503
eMINTS + Intel vs. eMINTS	0.702 (n = 35)	0.565 (n = 24)	—	0.136	0.230	.554	0.119
Technology integration
eMINTS vs. control	—	0.568 (n = 24)	−0.097 (n = 37)	0.665	0.100	.000*	1.431
eMINTS + Intel vs. control	0.573 (n = 35)	—	−0.097 (n = 37)	0.671	0.084	.000*	1.551
eMINTS + Intel vs. eMINTS	0.573 (n = 35)	0.568 (n = 24)	—	0.006	0.097	.952	0.012

Note. From authors’ analysis based on Year 3 implementation (2013–2014) data from the study districts and Missouri Department of Elementary and Secondary Education. eMINTS = enhancing Missouri’s Instructional Networked Teaching Strategies.

Teacher outcomes are Rasch logit ability measures from a Rasch analysis of teachers’ responses to the spring 2014 teacher survey items that fall under the four constructs: community of learners (12 items), inquiry-based learning (43 items for mathematics and 49 items for communication arts), high-quality lesson design (nine items), and technology integration (96 items).

Means and impacts were regression adjusted using ordinary least squares to account for block effects and baseline teacher and classroom characteristics. Because of the relatively small number of teachers within each block, a constant treatment effect was assumed across blocks.

Effect sizes were calculated using standardized mean differences (Hedges’s g). The p values are from a two-tailed test of the null hypothesis of equality of means.

n gives the sample sizes used in the analysis. It includes all eligible mathematics and communication arts teachers who provided consent to participate and have nonmissing outcomes.

Statistically significantly different from zero at the .05 level.

Classroom Observation Results

As highlighted in Table 7, we found significant positive results for the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach teachers versus control teachers for each domain observed. Each domain had medium effect sizes—community of learners (0.50 and 0.52, respectively), inquiry-based learning (0.68 and 0.56, respectively), and technology integration (0.58 and 0.78, respectively). There were no significant differences between the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach, however. The observation instrument did not include a high-quality lesson design domain.

Table 7

Overall Impact of eMINTS on Three Classroom Observation Outcomes in Implementation Year 3 (2013–2014)

Outcome^a	Mean^b			Estimated Impact^b
Outcome^a	eMINTS + Intel	eMINTS	Control	Impact	SE	p value	Effect size^c
Community of learners
eMINTS vs. control	—	2.404 (n = 33)^d	1.692 (n = 41)	0.713	0.340	.036*	0.498
eMINTS + Intel vs. control	2.506 (n = 43)	—	1.692 (n = 41)	0.815	0.301	.007*	0.521
eMINTS + Intel vs. eMINTS	2.506 (n = 43)	2.404 (n = 33)	—	0.102	0.329	.757	0.069
Inquiry-based learning
eMINTS vs. control	—	−0.425 (n = 33)	−1.776 (n = 41)	1.351	0.526	.010*	0.684
eMINTS + Intel vs. control	−0.534 (n = 43)	—	−1.776 (n = 41)	1.242	0.465	0.008*	0.555
eMINTS + Intel vs. eMINTS	−0.534 (n = 43)	−0.425 (n = 33)	—	−0.109	0.506	.0829	−0.045
Technology integration
eMINTS vs. control	—	0.387 (n = 33)	−0.186 (n = 41)	0.573	0.210	.006*	0.579
eMINTS + Intel vs. control	0.592 (n = 41)	—	−0.186 (n = 41)	0.779	0.182	.000*	0.777
eMINTS + Intel vs. eMINTS	0.592 (n = 41)	0.387 (n = 33)	—	0.206	0.203	.311	0.234

Teacher outcomes are Rasch logit ability measures from a Rasch analysis of items in the spring 2014 classroom observation protocol that belong to three constructs: a community of learners (six items), inquiry-based learning (five items), and technology integration (15 items). There were no items in the classroom observation protocol measuring the high-quality lesson design construct, so this construct was measured through only the teacher survey.

Effect sizes were calculated using standardized mean differences (Hedges’s g). The p values are from a two-tailed test of the null hypothesis of equality of eMINTS and control means.

n gives the sample sizes used in the analysis. It includes all eligible mathematics and communication arts teachers who provided consent to participate and have nonmissing outcomes.

Statistically significantly different from zero at the .05 level.

Impacts on Student Achievement and 21st-Century Skills

Table 8 summarizes the overall impact estimates across student outcomes. Impact estimates on mathematics achievement for the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach are positive and significant, with effect sizes of 0.128 and 0.178, respectively. Neither treatment group had significant impacts on 21st-century skills or communication arts when compared with the control group. Similarly, impacts on student engagement were not statistically significant. There also were no significant differences in mean student outcomes between the eMINTS Comprehensive Program and the eMINTS Comprehensive Program + Intel Teach.

Table 8

Year 3 Impact of eMINTS on Grades 7 and 8 School Means of Student Outcomes (2013–2014)

Outcome	M ^a			Estimated impact^a
Outcome	eMINTS + Intel	eMINTS	Control	Impact	SE	p value	Effect size^b
MAP 2013 mathematics z scores^c
eMINTS vs. control	—	0.083 (n = 1,108)^d	−0.047 (n = 823)	0.129	0.063	.040*	0.128
eMINTS + Intel vs. control	0.122 (n = 1,141)	—	−0.047 (n = 823)	0.168	0.065	.009*	0.178
eMINTS + Intel vs. eMINTS	0.122 (n = 1,141)	0.083 (n = 1,108)	—	0.039	0.063	.532	0.037
MAP 2013 communication arts z scores^c
eMINTS vs. control	—	0.052 (n = 1,208)	0.087 (n = 871)	−0.035	0.050	.484	−0.035
eMINTS + Intel vs. control	0.011 (n = 1,216)	—	0.087 (n = 871)	−0.076	0.051	.137	−0.077
eMINTS + Intel vs. eMINTS	0.011 (n = 1,216)	0.052 (n = 1,208)	—	−0.041	−0.041	.384	−0.041
21st-century skills scale scores^e
eMINTS vs. control	—	272.1 (n = 827)	266.0 (n = 683)	6.123	6.700	.361	0.103
eMINTS + Intel vs. control	270.4 (n = 866)	—	266.0 (n = 683)	4.374	6.527	.503	0.059
eMINTS + Intel vs. eMINTS	270.4 (n = 866)	272.1 (n = 827)	—	−1.749	−0.038	.798	−0.038
Student engagement scores^e
eMINTS vs. control	—	0.629 (n = 868)^d	0.788 (n = 659)	−0.160	0.115	.166	−0.176
eMINTS + Intel vs. control	0.667 (n = 751)	—	0.788 (n = 659)	−0.121	0.112	.280	−0.149
eMINTS + Intel vs. eMINTS	0.667 (n = 751)	0.629 (n = 868)	—	0.038	0.037	.737	0.038

Note. From authors’ analysis based on Year 3 implementation (2013–2014) data from the study districts and Missouri Department of Elementary and Secondary Education.

Means and impacts were regression adjusted to account for clustering of students within schools, block effects, and baseline student, teacher, and school characteristics. They were weighted by the number of schools in each block.

Effect sizes were calculated separately by block and then pooled into an overall effect size weighted by the number of schools in each block. Block-specific effect sizes were computed using standardized mean differences (Hedges’s g). The p values are from a two-tailed test of the null hypothesis of equality means.

Because the student analytic samples are pooled samples of seventh and eighth graders, MAP scale scores were converted into a common metric by standardizing the scores separately by year (i.e., 2011 for the pretest and 2014 for the posttest), grade, subject (mathematics and communication arts), and randomization block. Specifically, each test score was converted into a z score by subtracting from the test score the average score for a particular year, grade, subject, and block and then dividing by the standard deviation for a particular year, grade, subject, and block.

n gives the sample sizes used in the analysis. For the MAP mathematics and communication arts analyses, n includes all eligible students with nonmissing outcomes. For the 21st-century skills and student engagement analyses, n includes all eligible students who provided consent to participate and have nonmissing outcomes.

Student engagement scores are Rasch logit ability measures from a Rasch analysis of students’ responses to the spring 2014 student engagement survey.

Discussion and Limitations

eMINTS professional development is an intensive 2-year professional development package designed to promote inquiry-based learning, support high-quality lesson design, build community among students and teachers, and create technology-rich learning environments. Although the program has been studied continuously for more than a decade, no study prior to this one could be characterized as a rigorous evaluation. We randomly assigned 60 rural schools in Missouri to two treatment groups and a control group. Both treatment groups received the 2 years of eMINTS. Three years after the implementation of eMINTS began, we demonstrate significant self-reported and observed changes in teacher instructional practices. Teachers from both treatment groups reported statistically significant (with large effect sizes) increases in inquiry-based instruction and technology integration. Survey research suggests that respondents often inaccurately self-report (Tourangeau, Rips, & Rasinski, 2000). Indeed, classroom observation results suggest that teachers might have overstated the amount of change, but it is important to emphasize that the change was real (statistically significant with medium effect sizes). Observation data also indicate that treatment teachers were increasingly establishing a classroom-level community of learners. Survey data suggest that eMINTS+ teachers believed that their instructional planning had improved.

These changes are perhaps realized with statistically significant but small effect sizes in mathematics achievement. We know, however, that effecting change on state assessments is typically difficult to do. There is a considerable body of research on professional development that emphasizes the need for sustained professional development (e.g., Darling-Hammond et al., 2009; Garet et al., 2001) as well as some patience—perhaps 3 to 5 years—before the school-level programs can be fully realized (Berends, Kirby, Naftel, & McKelvey, 2001). Indeed, prior research literature on eMINTS suggests that impacts occur after the completion of the program.

Given the funding timeline of the i3 award, technology requirements were not met in many treatment schools until the spring semester of the first study year. It is conceivable that more time is needed before the program will impact 21st-century skills. The results are consistently positive but not significant. However, as technology advances and student access, even in rural settings, increases, instruments assessing such skills should be adapted. The instrument used for this study was constant, which was good for measurement purposes but could have been outdated before the study’s completion. We should also note here that qualitative aspects of our classroom observations suggest that a number of treatment teachers incorporated laptops and interactive whiteboards into their lessons, but many of the associated activities were drill or other elementary activities. For example, increased use of an interactive whiteboard could demonstrate initial growth in technology use but not necessarily in an instructionally meaningful or unique way.

Although we find statistically significant mathematics achievement gains as well as significant increases in technology use and classroom-level communities of learners, there are relatively few randomized controlled trial evaluations that suggest a causal relationship between specific professional development strategies and achievement. Since Yoon, Duncan, Lee, Scarloss, and Shapley’s (2007) review of teacher professional development interventions that meet evidence standards of the U.S. Department of Education’s WWC, we found only one published study that meets WWC standards of evidence and examines a teacher professional development program’s impacts on student achievement in middle school. Garet et al. (2011) conducted a randomized trial that examined the impact of providing a professional development program in rational number topics to seventh-grade mathematics teachers after 2 years of implementation.

In turn, we briefly juxtapose that evaluation and ours because the program in Garet et al. (2011) and eMINTS share common features of effective professional development found in Yoon et al.’s (2007) review. For instance, both programs included more than 100 direct contact hours with teachers, ongoing workshops and supports, and teacher training embedded in daily instruction. In addition, both studies were conducted in middle schools and focused on changes in teacher knowledge and student achievement. So why did the eMINTS program find positive effects on teachers’ knowledge, instructional practices, and students’ mathematics achievement while the program in Garet et al.’s study did not?

Several key differences worth consideration emerge. First, although Garet et al.’s (2011) study included a substantial amount of direct professional development, the eMINTS program included over twice this amount (110 contact hours vs. 240 contact hours over 2 years). Professional development content also differed. The professional development in Garet et al.’s (2011) study focused on common and specialized knowledge of mathematical content (specifically, teaching positive rational numbers) while eMINTS focused more comprehensively on understanding and teaching grade-level content standards. In addition, although both programs encouraged instruction to elicit higher-order thinking and reasoning skills, eMINTS focused on technology integration and inquiry-based learning strategies as the critical vehicles to elicit such skills.

The eMINTS program’s focus on the principal’s role in promoting effective professional development may have also influenced results. Prior research prioritizes the role of the principal in successful professional development implementation (Desimone, 2002). Principal training, albeit much less intensive than training provided to teachers, included a summer workshop and ongoing training. eMINTS training focuses on helping principals understand eMINTS requirements, identify and align existing instructional priorities with eMINTS’s strategies, and demonstrate a commitment to sustaining the eMINTS instructional model over time. Through biannual school walk-throughs and resources to support principals’ evaluation process—including observation rating forms, surveys, and other measures—eMINTS training helps principals support eMINTS model implementation and eliminate competing programs and initiatives. Implementation results and feedback on principal surveys suggested that the majority of principals bought into and supported implementation of eMINTS practices.

Context and school background characteristics might have also contributed to the high levels of buy-in and support among eMINTS principals. For instance, competing initiatives and policies may have been easier to remove given the small size of most districts and schools in the sample. In some cases, the principal was also the superintendent, and all districts in the sample had only one middle school. Most principals in the sample had the ability to make decisions and changes when needed without a lot of red tape. eMINTS is also a well-known and well-respected program that is in high demand across Missouri. eMINTS’s reputation as an effective program may have reduced teacher and principal reluctance and increased motivation to incorporate innovative instructional approaches and new technology into their practice.

Although this evaluation concludes with positive outcomes, there are a number of considerations for future research and decisions for the providers. This evaluation lends itself to further analytic work to identify potential mediators or moderators. The findings also do not generalize beyond rural schools enrolling seventh- and eighth-grade students, and perhaps then only rural Missouri schools. Prior research on eMINTS implies effectiveness in elementary and urban settings, as well, but this evaluation does not have the capacity to confirm it. This evaluation also lacks the ability to determine whether certain eMINTS focuses (e.g., lesson planning versus technology integration) have more or more lasting impact than others. We would learn a great deal to better understand the nuance of the program, both for the field and for the programmers themselves. eMINTS is an intensive, 2-year program. Any ability to pare it while still effecting instructional change and student achievement would be very important in terms of time and cost.

Footnotes

Acknowledgements

The authors would like to thank Jim Lindsay, Shazia Miller, Cris Price, and two anonymous reviewers for feedback at various evaluative and reporting stages.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a grant from the Office of Innovation and Improvement (OII), U.S. Department of Education (U396B100038). The findings and opinions expressed are those of the authors and do not represent the position or policies of the U.S. Department of Education.

Notes

Authors

COBY V. MEYERS is the chief of research of the Darden/Curry Partnership for Leaders in Education (PLE) and associate professor of education in the Curry School of Education at the University of Virginia; e-mail: cvm2x@virginia.edu. His research focuses on understanding the role of school system leadership, especially in the context of school turnaround.

AYRIN MOLEFE is a senior statistician/methodologist at American Institutes for Research with considerable experience in statistical analysis. She has expertise in formulating scientifically based research designs and in analyzing complex educational data sets. She collaborates with various teams as a statistician, developing and/or providing advice on the study design and statistical methodology and conducting and/or guiding statistical analysis.

W. CHRISTOPHER BRANDT is vice president of education at IMPAQ International, LLC, a public policy research firm with headquarters in Columbia, Maryland. His recent work focuses on rigorous research and evaluation studies in educator evaluation, district and school accountability, and school reform.

BO ZHU is a researcher with the American Institutes for Research. Her research focuses on professional development in education, teacher and leader effectiveness, and program and policy evaluation in education.

SONICA DHILLON is a researcher at the American Institutes for Research. She is dedicated to furthering local and state efforts to mitigate student dropout through effective use of early warning system (EWS) data. Measuring fidelity of implementation is one of her primary research interests.

References

Allen

J. P.

Pianta

R. C.

Gregory

Mikami

A. Y.

Lun

(2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333, 1034–1037.

Bellanca

Brandt

(2010). 21st century skills: Rethinking how students learn. Bloomington, IN: Solution Tree.

Belland

Glazewski

Richardson

(2011). Problem-based learning and argumentation: Testing a scaffolding framework to support middle school students’ creation of evidence-based arguments. Instructional Science, 39, 667–694.

Berends

Kirby

S. N.

Naftel

McKelvey

(2001). Implementation and performance in New American Schools: Three years into scale-up. Santa Monica, CA: RAND.

Bloom

H. S.

(2006). The core analytics of randomized experiments for social research (MDRC working papers on Research Methodology). New York, NY: MDRC. Retrieved from http://www.mdrc.org/sites/default/files/full_533.pdf

Brandt

W. C.

Meyers

Molefe

(2012). The impact of eMINTS professional development on teacher instruction and student achievement: Year 1 report. Washington, DC: American Institutes for Research.

Brandt

W. C.

Meyers

Molefe

Dhillon

Zhu

(2013). The impact of eMINTS professional development on teacher instruction and student achievement: Year 2 report. Washington, DC: American Institutes for Research.

Bredeson

Johansson

(2000). The school principal’s role in teacher professional development. Journal of In-Service Education, 26, 385–401.

Buck Institute for Education. (2007). National survey of high school reform and project based learning. Novato, CA: Author.

10.

Clark

R. E.

(1994). Media will never influence learning. Educational Technology Research & Development, 42(2), 21–29. doi:10.1007/BF02299088

11.

Condon

Dawson

Molefe

Swanlund

(2009). Results of the Rasch analysis of items based on the ISTE-NETS-S technology literacy standards for 5th and 8th grades (pretest). Chicago, IL: Learning Point Associates.

12.

Consortium on Chicago School Research at the University of Chicago. (2007a). Elementary student survey form. Chicago, IL: University of Chicago. Retrieved from http://ccsr.uchicago.edu/downloads/42592007_elem_student_codebook.pdf

13.

Consortium on Chicago School Research at the University of Chicago. (2007b). High school teacher survey form. Chicago, IL: University of Chicago. Retrieved from http://ccsr.uchicago.edu/downloads/44132007_hs_teacher_codebook.pdf

14.

Cook

T. D.

Church

M. B.

Ajanaku

Shadish

W. R.

Jr. Jeong-Ran

Cohen

(1996). The development of occupational aspirations and expectations among inner-city boys. Child Development, 67, 3368–3385.

15.

CTB/McGraw-Hill. (2012). Missouri Assessment Program: Grade-level assessments (Technical report 2012: Final). Monterey, CA: Author.

16.

Darling-Hammond

Wei

Andree

Richardson

Orphanos

(2009). Professional learning in the learning profession: A status report on teacher development in the United States and abroad. Dallas, TX: National Staff Development Council.

17.

Desimone

(2002). How can comprehensive school reform models be successfully implemented? Review of Educational Research, 72, 433–479.

18.

Drake

S. M.

(2007). Creating standards-based integrated curriculum: Aligning curriculum, content, assessment, and instruction. Thousand Oaks, CA: Corwin Press.

19.

eMINTS National Center, University of Missouri. (2009). eMINTS teacher technology literacy skill survey (Survey instrument). Columbia: Author.

20.

Ertmer

P. A.

Ottenbreit-Leftwich

A. T.

(2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 42, 255–284.

21.

Garet

Porter

Desimone

Birman

Yoon

(2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915–945.

22.

Garet

Wayne

Stancavage

Taylor

Eaton

Walters

. . . Doolittle

(2011). Middle school mathematics professional development impact study: Findings after the second year of implementation (NCEE 2011-4024). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

23.

Geier

Blumenfeld

Marx

Krajcik

Fishman

Soloway

Clay-Chambers

(2008). Standardized test outcomes for students engaged in inquiry-based science curricula in the context of urban reform. Journal of Research in Science Teaching, 45, 922–939.

24.

Guthrie

J. T.

Wigfield

Barbosa

Perencevich

K. C.

Taboada

Davis

M. H.

. . . Tonks

(2004). Increasing reading comprehension and engagement through concept-oriented reading instruction. Journal of Educational Psychology, 96, 403–423.

25.

Hallinger

Heck

R. H.

(1996). Reassessing the principal’s role in school effectiveness: A review of empirical research, 1980-1995. Educational Administration Quarterly, 32, 5–44.

26.

Hamilton

Halverson

Jackson

S. S.

Mandinach

Supovitz

J. A.

Wayman

J. C.

(2009). Using student achievement data to support instructional decision making (NCEE 2009-4067). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/practice_guides/dddm_pg_092909.pdf

27.

Hart

H. M.

Allensworth

Lauen

D. L.

Gladden

R. M.

(2002). Educational technology: Its availability and use in Chicago’s public schools. Chicago, IL: Consortium on Chicago School Research.

28.

Heritage

(2007). Formative assessment: What do teachers need to know and do? Phi Delta Kappan, 89, 140–145.

29.

Hickey

D. T.

Wolfe

E. W.

Kindfield

A. C. H.

(2000). Assessing learning in a technology-supported genetics environment: Evidential and consequential validity issues. Educational Assessment, 6, 155–196.

30.

Hmelo-Silver

C. E.

(2006). Design principles for scaffolding technology-based inquiry. In O’Donnell

A. M.

Hmelo-Silver

C. E.

Erkens

(Eds.), Collaborative reasoning, learning and technology (pp. 147–170). Mahwah, NJ: Lawrence Erlbaum.

31.

Hmelo-Silver

C. E.

Duncan

Chinn

(2007). Scaffolding and achievement in problem-based and inquiry learning: A response to Kirschner, Sweller, and Clark (2006). Educational Psychologist, 42, 99–107.

32.

Huston

Miller

Richburg-Hayes

Duncan

Eldred

Weisner

T. S.

. . . Redcross

(2003). New hope for families and children: Five-year results of a program to reduce poverty and reform welfare. New York, NY: MDRC.

33.

Institute of Education Sciences. (2010). The nation’s report card: Mathematics 2009: National assessment of educational progress. Washington, DC: U.S. Department of Education. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2009/2010451.pdf

34.

Institute of Education Sciences. (2011). What Works Clearinghouse reviewer training manual. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

35.

International Society for Technology in Education. (2012). NETS for students. Retrieved from http://www.iste.org/standards/ISTE-standards/standards-for-students

36.

Joyce

Showers

(1995). Student achievement through staff development: Fundamentals of school renewal (2nd ed.). White Plains, NY: Longman.

37.

Kohn

(1994). Choices for children: Why and how to let students decide. Phi Delta Kappan. Retrieved from http://www.alfiekohn.org/article/choices-children/

38.

Kohn

(2006). The homework myth: Why our kids get too much of a bad thing. Cambridge, MA: Da Capo Press.

39.

Langer

J. A.

(2001). Beating the odds: Teaching middle and high school students to read and write well. American Educational Research Journal, 38, 837–880.

40.

Lawless

K. A.

Pelligrino

J. W.

(2007). Professional development in integrating technology into teaching and learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of Educational Research, 77, 575–614.

41.

Lembke

Gonzales

(2006). U.S. student and adult performance on international assessments of educational achievement (NCES 2006-073). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubs2006/2006073.pdf

42.

Lindstrom

P. H.

Speck

(2004). The principal as professional development leader. Thousand Oaks, CA: Corwin Press.

43.

Martin

Shulman

(2006). Intel teach essentials instructional practices and classroom use of technology survey report. New York, NY: Education Development Center, Center for Children and Technology.

44.

Martin

Strother

Reitzes

(2009). eMINTS 2009 program evaluation report: An analysis of the persistence of program impact on student achievement. New York, NY: Education Development Center, Center for Children and Technology.

45.

McKnight

P. E.

McKnight

K. M.

Sidani

Figueredo

A. J.

(2007). Missing data: A gentle introduction. New York, NY: Guilford Press.

46.

Middleton

M. J.

Midgley

(2002). Beyond motivation: Middle school students’ perceptions of press for understanding in math. Contemporary Educational Psychology, 27, 373–391.

47.

Midgley

Maehr

M. L.

Hruda

L. Z.

Anderman

Freeman

K. E.

. . . Urdan

(2000). Manual for the Patterns of Adaptive Learning Scales (PALS). Ann Arbor: University of Michigan.

48.

Minner

Levy

Century

(2010). Inquiry-based science instruction—What is it and does it matter? Results from a research synthesis years 1984 to 2002. Journal of Research in Science Teaching, 47, 474–496.

49.

Mitchem

Wells

D. L.

Wells

J. G.

(2003). Effective integration of instructional technologies (IT): Evaluating professional development and instructional change. Journal of Technology and Teacher Education, 11, 397–414.

50.

Monk

(2007). Recruiting and retaining high-quality teachers in rural areas. Future of Children, 17, 155–174.

51.

National Research Council. (2000). Inquiry and the national science education standards. Washington, DC: National Academies Press.

52.

National Research Council. (2003). How people learn: Brain, mind, experience, and school. Washington, DC: National Academies Press.

53.

Northwest Regional Educational Laboratory. (2005). Observation protocol for technology integration in the classroom. Portland, OR: Author.

54.

Office of Social and Economic Data Analysis. (2005). Analysis of 2004 MAP results for eMINTS students. Columbia, MO: Author.

55.

Penuel

W. R.

Fishman

Yamaguchi

Gallagher

(2007). What makes professional development effective? Strategies that Foster curriculum implementation. American Educational Research Journal, 44, 921–958.

56.

Pianta

R. C.

La Paro

K. M.

Hamre

B. K.

(2008). Classroom assessment scoring system (CLASS). Baltimore, MD: Brookes.

57.

Pulakos

E. D.

Arad

Donovan

M. A.

Plamondon

K. E.

(2000). Adaptability in the workplace: Development of a taxonomy of adaptive performance. Journal of Applied Psychology, 85, 612–624.

58.

Puma

M. J.

Olsen

R. B.

Bell

S. H.

Price

(2009). What to do when data are missing in group randomized controlled trials (NCEE 2009-0049). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.

59.

Rasch

(1980). Probabilistic models for some intelligence and attainment tests (Exp. ed.). Chicago, IL: University of Chicago Press.

60.

Ross

S. M.

Morrison

G. R.

Lowther

D. L.

(2010). Educational technology research past and present: Balancing rigor and relevance to impact school learning. Contemporary Educational Technology, 1, 17–35.

61.

Schmid

R. F.

Bernard

R. M.

Borokhovski

Tamim

Abrami

P. C.

Wade

C. A.

. . . Lowerison

(2009). Technology’s effect on achievement in higher education: A Stage I meta-analysis of classroom applications. Journal of Computing in Higher Education, 21, 95–109.

62.

Shear

Gorges

Means

Singleton

Novais

Gallagher

Lundh

(2010). The Microsoft Innovative Schools Program year 2 evaluation report. Redmond, WA: Microsoft.

63.

Simons

K. D.

Klein

J. D.

(2007). The impact of scaffolding and student achievement levels in a problem-based learning environment. Instructional Science, 35, 41–72.

64.

Slavin

(2010). Co-operative learning: What makes groupwork work? In Centre for Educational Research and Innovation (Ed.), The nature of learning: Using research to inspire practice (pp. 161–178). Paris, France: Organization for Economic Cooperation and Development.

65.

Strange

Johnson

Showalter

Klein

(2012). Why rural matters 2011–2012: The condition of rural education in the 50 states. Washington, DC: Rural School and Community Trust.

66.

Strother

Martin

Dechaume

(2006). Analysis of 2006 MAP results for eMINTS and non-eMINTS students. New York, NY: Education Development Center, Center for Children and Technology.

67.

Tamim

R. M.

Bernard

R. M.

Borokhovski

Abrami

P. C.

Schmid

R. F.

(2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81, 4–28.

68.

Tomlinson

Brighton

Hertbery

Callahan

Moon

Brimijjoin

. . . Reynolds

(2003). Differentiating instruction in response to student readiness, interest, and learning profile in academically diverse classrooms: A review of literature. Journal for the Education of the Gifted, 27, 119–145.

69.

Tourangeau

Rips

L. J.

Rasinski

(2000). The psychology of survey response. New York, NY: Cambridge University Press.

70.

West Virginia Department of Education. (n.d.). Classroom observation instrument scoring rubric. Charleston: Author.

71.

Wiggins

McTighe

(1998). Understanding by design. Upper Saddle River, NJ: Prentice Hall.

72.

Wright

B. D.

Masters

G. N.

(1982). Rating scale analysis: Rasch measurement. Chicago, IL: MESA Press.

73.

Y.-T.

Tsai

C.-C.

(2005). Effects of constructivist-oriented instruction on elementary school students’ cognitive structures. Journal of Biological Education, 39, 113–119.

74.

Yoon

K. S.

Duncan

Lee

S. W.-Y.

Scarloss

Shapley

(2007). Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007-No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Laboratory Southwest.

75.

Zimmerman

B. J.

Bandura

Martinez-Pons

(1992). Self-motivation for academic attainment: The role of self-efficacy beliefs and personal goal setting. American Educational Research Journal, 29, 663–676.