Abstract
Within their teacher preparation courses and field experiences, preservice teachers are introduced to numerous instructional practices, not all of which are considered research-based. For this reason, instruction in how to evaluate the effectiveness of one’s practices is essential, but it is often a lacking component of initial certification programs. In this article, a flexible, problem-solving model for collecting and reflecting on practice-based evidence (PBE) is described. The model, utilized in a graduate program in Special Education, was designed to assist teacher candidates in evaluating the effectiveness of the practices they implement to optimize students’ learning outcomes. Implications for practice in the K-12 environment are also provided.
In teacher education programs, the terms evidence-based practices (EBPs), research-based practices, and promising practices are often used when describing interventions to preservice teachers. The definition of the terms, however, varies based on, among other things, the rigor of the research design, review process employed for validation, and impact on student outcomes (Cook & Odom, 2013). Although these terms are distinguished between by educational researchers and teacher educators, practitioners often use these terms interchangeably, without understanding the subtleties that differentiate one category of practices from another (Santangelo, Novosel, Cook, & Gapsis, 2015; Smith, Schmidt, Edelen-Smith, & Cook, 2013). As Santangelo et al. (2015) noted, information about effective practices can be overwhelming, and even intimidating, to practitioners who lack the requisite research literacy needed to make sense of the findings.
Educators also frequently rely on “experientially-based” practices, or those practices which stem from one’s professional judgment, wisdom, and experiences (Mazzotti, Rowe, & Test, 2013; Turnbull et al., 2010) instead of implementing established, research-based practices to assist students in achieving targeted educational outcomes. These experientially-based practices are often grounded in educational theory, but may not have been investigated under the scrutiny of rigorous data-based inquiry nor widely disseminated in peer-reviewed scholarly journals. Thus, educators continue implementing pedagogical and behavioral practices that “feel” appropriate and effective in their impact on identified student outcomes without utilizing a process of systematic inquiry to confirm (or refute) their intuition. The research-to-practice gap is further exacerbated when preservice teachers experience a lack of exposure to, and consequently a lack of opportunities for practice of, research-based practices in their fieldwork settings.
One way to bridge this gap is to encourage educators to gather data about the extent to which their educational practices effectively and efficiently influence the educational outcomes of their students under typical teaching conditions. This approach, which has been referred to as practice-based evidence (PBE), originated in the fields of medicine and psychology and involves educators using their classrooms as the site of inquiry to gather evidence concerning “where and under what conditions a practice works, with whom the practice works, how a practice can be adapted and maintained successfully, and how practitioners feel about a practice” (Cook, 2011, p. 1). When collecting PBE, in-service and preservice teachers utilize a data-based decision making process to (a) investigate the degree to which a behavioral or pedagogical practice identified in the literature as an EBP produces desirable outcomes given the particular circumstances in the local educational context, or (b) collect data about an experientially- based practice that has little to no established record of validated evidence in the research literature. That is, when employing a PBE approach, data are collected and evaluated by practitioners in real-world settings to not only validate an EBP given the particular circumstances in an educator’s classroom but also provide inquiry-based evidence related to an intervention grounded in their professional experience, intuition, or judgment.
Learning systematic methods of collecting reliable and valid data about one’s professional practice is an indispensable skill for beginning teachers. Working through a systematic inquiry model, like PBE, challenges novice teachers to consider ways to document more explicitly their matriculation through the cycle of planning, instruction, assessment, and future planning. Furthermore, the application of PBE to experientially-based practices or EBPs within the local context can empower educators to follow their professional judgment while documenting the results with sound evidence. As a result of this inquiry-based process, the likelihood of replicating changes in students’ academic, social, and/or behavioral performance is increased (Salend, Baker, & Gardner, 2012) and practitioners are better positioned to make evidence-based decisions about their subsequent professional practices (Cook, 2011; Cook & Cook, 2016).
The PBE approach has faced criticism from some due to the popularity of the EBP movement in education. An EBP, however, provides no guarantee of results in all situations. Literature about the use and effectiveness of EBPs notes that “EBPs should be the first option for responsible teachers, administrators, teacher-educators, and policy makers” while also including caveats that caution that “teachers should not, then, expect every student to respond favorably to EBPs even when appropriately applied” (Cook, Tankersley, Cook, & Landrum, 2008, p. 72). Other scholars have advocated for more “bottom-up, field demonstrated approaches” for understanding and assessing school-based interventions (Shannon, 2015, p. 3). That is, these scholars have rightly recognized the valuable role practitioners play in collecting, evaluating, and disseminating the data based on their classroom experiences (Kratochwill et al., 2012; Shannon, 2015). Scruggs, Mastropieri, Berkeley, and Marshak (2010) underscored the importance of practitioner-led inquiry by not only describing the research behind the EBP of mnemonic strategies but also including examples of PBE. Cook and Cook (2016) noted the reciprocal relationship between PBE and EBP, observing the dynamics that govern the ways in which each informs the other. Furthermore, Cook and Cook (2016) recognized the value of teacher training activities which include PBE by noting that “practice-based evidence can result in a student teaching experience that reinforces, rather than negates, coursework in evidence-based practice” (p. 153).
In this article, we present a flexible, problem-solving model (see Figure 1) for collecting and reflecting on PBE. This model for inquiry provides a framework for educators to document and analyze the extent to which identified educational interventions effectively and efficiently produce desired student outcomes. This PBE model was initially designed as a means of developing the pedagogical knowledge and skills of teacher candidates matriculating through a graduate special education program at a mid-Atlantic university. The application of this model by teacher candidates aligns with the call for teacher preparation programs to incorporate “assignments and training that allow preservice SETs [special education teachers] to learn how to locate research- and evidence-based practices, to design interventions and implement them within a classroom, and to understand the importance of fidelity to treatment” (Mason-Williams, Frederick, & Mulcahy, 2015, p. 208).
Problem-solving model for collecting and reflecting on PBE.
Constructing the PBE Model
When designing the PBE model as a means of assisting our teacher candidates in making evidence-based decisions about the practices they employ in their field experiences, several elements were deemed essential. First, the model features the principles of “backward design.” Backward design involves careful consideration of the desired outcomes or results before the planning process begins (Wiggins & McTighe, 2005). The stages of backward design, as identified in McTighe and Wiggins’s (2012) framework, include (a) identifying desired results, (b) determining assessment evidence, and (c) planning learning experiences and instruction. The emphasis on clearly understanding the desired goal at the outset helps the investigator to make more focused methodological, instructional, and/or behavioral decisions throughout the inquiry process. The influence of backward design principles is evidenced in the first several steps of the PBE model.
Next, our PBE model includes components of curriculum-based assessment (CBA). CBA, a type of formative assessment closely associated with progress monitoring, involves the continuous administration of brief measures of students’ progress and proficiency that are connected to typical classroom-based content and academic activities (Foegen & Morrison, 2010). A primary purpose of CBA is to gather targeted data about student performance and then use the findings to inform subsequent classroom decisions (The IRIS Center, 2006). The effectiveness of curriculum-based measurement (CBM), one type of CBA, on student outcomes is documented in the research literature (cf. Stecker, Fuchs, & Fuchs, 2005) and was therefore embedded into the PBE model.
Finally, critical self-reflection was included as the third element essential to the model. Educator preparation programs are increasingly recognizing the value of including curricular and experiential activities featuring critical reflection (Hatton & Smith, 1995; Jaeger, 2013). Schön’s (1983) book, The Reflective Practitioner: How Professionals Think in Action, provided a framework for understanding the ways in which professional practice can be optimized by engaging in “reflection-in-action” and “reflection-on-action” (p. 49). Smith and Glenn’s (2016) research explored the role of critical reflection in creating self-awareness among teacher candidates and advocates for “teacher educators to create course mechanisms that structure reflection on practice” (p. 317). In the PBE model, evidence of this statement can be seen in several steps as teacher candidates work to implement practices with fidelity, reflect on the data, and consider a range of possibilities based on their formative assessments. In the following section, we describe the various components of the PBE model and how it is implemented in a graduate program for beginning special educators.
Implementing the PBE Model
The PBE model is used within a graduate program for beginning special educators who already hold their initial licensure in early childhood, childhood, or adolescent education and is embedded at three points in the program. First, the model is introduced within an initial course on educational assessment. In this course, teacher candidates learn how to collect and evaluate progress monitoring data (including CBA), and then they practice these skills with one or two students with a disability within their field experience. Next, the PBE model is scaffolded for and then implemented by teacher candidates in a course focusing on literacy instruction for students with special needs. This course provides the opportunity for the teacher candidates to make evidence-based decisions when selecting and implementing a practice specifically designed to develop a small group of students’ reading and/or writing skills. Finally, in the culminating practicum experience, teacher candidates utilize the model within a supervised context to identify, implement, and evaluate an intervention aimed at developing their students’ academic, social, or behavioral skills.
The PBE model described next is an approach to inquiry that provides teacher candidates with the tools to explore the efficacy of established EBPs within their local classroom contexts and experientially-based practices that have not yet met the rigorous industry standard required to be classified among “evidence-based practices.” Although we discuss the use of the PBE model in a graduate school teacher education context, the PBE model is flexible enough to be utilized by any preservice or in-service teachers who are interested in discovering a framework to guide them in the process of evidence-based decision making.
Identify Educational Outcomes
Teacher candidates begin the PBE process by identifying the desired educational outcomes they want one or more targeted students to achieve. Educational outcomes relate to mastery of meaningful academic content and skills related to benchmarks within the curriculum (e.g., Common Core State Standards [CCSS], statewide and district-wide learning standards) or areas of need identified by universal screening measures (e.g., CBA; Dynamic Indicators of Basic Literacy Skills [DIBELS], Good & Kaminski, 2007). Educational outcomes related to the acquisition of social and behavioral skills that support learning and foster positive interactions with peers and teachers could also be identified. This includes outcomes related to the Individualized Education Programs (IEPs) and 504 accommodation plans of students with disabilities.
Determine Appropriate Assessment Strategies
Considering the principles of backward design previously noted, teacher candidates determine appropriate classroom assessment strategies before planning and instruction begins. Initially, these assessment strategies are used to collect baseline data about the targeted students’ performance prior to identifying and implementing an intervention. These baseline data then provide a standard from which the teacher candidates can judge the impact of the practice on the students’ progress. If support is needed to effectively implement progress monitoring, online resources can assist educators such as those available through the National Center on Response to Intervention (www.rti4success.org), the National Center on Intensive Intervention (www.intensiveintervention.org), and Intervention Central (www.interventioncentral.org).
Consider a Range of Possibilities and Select the Most Appropriate Practice
Teacher candidates utilize the identified educational outcomes and assessment strategies to consider a range of potential practices for use with the targeted students. To identify potential practices, teacher candidates are encouraged to review relevant coursework in which direct instruction of EBPs was provided, as well as explore established websites that provide additional information on content-specific EBPs (cf. Santangelo et al., 2015; Torres, Farley, & Cook, 2012). For example, What Works Clearinghouse (https://ies.ed.gov/ncee/WWC/) provides research summaries and guides on educational practices, products, and programs across the curriculum, and the IRIS Center (www.iris.peabody.vanderbilt.edu) offers online instructional resources that guide educators in learning how to identify and implement EBPs in their classrooms. Frameworks to guide teachers in considering a range of practices based on the best available research evidence located in the research literature (e.g., Santangelo et al., 2015) are also made available to the teacher candidates. In addition, teacher candidates are encouraged to consider the high-leverage practices (HLP) distributed by the Council for Exceptional Children (CEC) and the CEEDAR Center, which were designed to support teacher candidates in using effective practices (McLeskey et al., 2017). For example, teacher candidates are asked to consider which practices, such as teaching social behaviors (HLP9) or cognitive and metacognitive strategies (HLP14), can be utilized in a systematic way to assist students in achieving the specific learning outcome (HLP12).
Teacher candidates are also prompted to consider a range of experientially-based practices they believe might effectively address the identified need. The literature supports teachers exploring the pedagogical possibilities that exist beyond EBPs because many educational practices “may not be recognized as EBPs because (a) a sufficient number of high-quality, experimental studies have not been conducted regarding their effectiveness or (b) researchers have not systematically reviewed the existing body of literature regarding the practice” (Cook et al., 2008, p. 73). As teacher candidates explore possible experientially informed practices, they are encouraged to explore the pedagogical theories that undergird the strategy and the ways in which that might formalize the process to ensure later replicability.
After exploring possible interventions that might address the identified student outcome, teacher candidates select a targeted intervention. Teacher candidates are asked to examine the degree to which the research supporting each practice, if it is an EBP, aligns with their students’ demographic characteristics and learning environments (Cook et al., 2008; Mellard, McKnight, & Jordan, 2010). They also need to consider the extent to which each practice is effective in fostering student mastery of important educational outcomes. To scaffold the critical reflection process, teacher candidates are provided with guided reflection questions (see Figure 2, for examples). Regardless of the type of practice selected (i.e., evidence-based or experientially-based), PBE for the approach is established through collecting classroom-based data under typical teaching conditions.
Reflection questions for selecting the most appropriate practice.
Implement Practices With Fidelity and Flexibility
Once selected, a practice is implemented with fidelity by the teacher candidates, which means they are well prepared to implement the practice and comply with the appropriate components, timelines, group sizes, instructional sequences and procedures, frequency and length of sessions, and teaching and student behaviors (McMaster et al., 2010). If the practice is not conducted as originally intended, the findings are vulnerable to many confounding factors. The failure to implement the practice as intended, or inadvertently adding extraneous elements to the practice, can compromise the practice’s effect with students (Kretlow & Blatz, 2011). As a result, teacher candidates will not know if the practice is not effective as designed or if the practice is simply not effective as implemented. In the case of an experientially-based design that lacks an explicit implementation protocol, the teacher candidate must be able to articulate the steps necessary to faithfully perform the intervention across multiple iterations of planning, implementation, and assessment.
To promote fidelity of implementation, teacher candidates are also encouraged to follow several steps. First, they are encouraged to rehearse implementation. Second, recognizing the impact of in-classroom coaching on the fidelity of implementation (Kretlow & Blatz, 2011), it is suggested that teacher candidates work in collaborative teams with their cooperating teachers to monitor implementation and foster fidelity. Because our special education programs are at the graduate level, some teacher candidates implement the practice in their own classrooms for their field experience. In this case, they are encouraged to work with a colleague or mentor teacher as part of a collaborative team. Collaborative teams can monitor the implementation fidelity if a valid instrument for assessing fidelity is associated with the practice. For experientially-based practices that lack a psychometrically validated instrument, a checklist containing the essential features of the practice can be created. During lessons, observers can use the instrument or checklist to collect implementation fidelity data. After implementing lessons, teacher candidates can use the data as well as lesson artifacts such as lesson plans, student work samples, interviews, and self-report questionnaires to examine the extent to which the practice was implemented in the desired manner, sequence, and time. This information is then used to make appropriate adjustments, if needed, to ensure implementation fidelity (The IRIS Center, 2010). Teacher candidates also reflect on deviations in the implementation of the practice, assess the impact of their deviations, and consider the future implications in subsequent trials.
Because there is contextual variation in classrooms, teacher candidates are encouraged to exercise a level of flexibility in the implementation of practices to accommodate the unique characteristics of the students and classrooms without compromising aspects of the practice which make it effective (Cook et al., 2008; McMaster et al., 2010), if known. While ensuring fidelity to the practice’s core principles and components, teacher candidates are told that they may need to make minor adjustments to certain aspects of the practice based on individual students’ needs (Cook, Shepherd, Cook, & Cook, 2012). They are also, however, cautioned that, depending on the selected intervention, some EBPs may be very prescriptive and do not allow for variations in implementation (Peck & Scarpati, 2010).
Seek Students Feedback
Although practices may be effective in promoting student learning, they also may have otherwise (un)intended positive and negative consequences (Salend et al., 2012). For example, some interventions for students with special needs may be effective in supporting their academic performance; some students, however, experience the support as exacerbating their differences from their peers (Salend, 2009). Thus, teacher candidates are instructed to use interviews, surveys, and feedback forms to identify students’ perceptions of practices (Chorzempa, Salend, & Maheady, 2012). For example, students can provide feedback regarding the practice by (a) responding to questions (e.g., What did you like [or not] like about using the practice?), (b) rating their agreement with statements using a Likert-type scale, and (c) completing sentences (e.g., The practice helped me by —.).
Analyze and Reflect on the Data
After PBE is collected and summarized, teacher candidates analyze and reflect on the data to assess the extent to which the practice worked effectively and was acceptable to both the students and themselves. In this step, teacher candidates are prompted to ascertain if the desired outcome was achieved through the implementation of the practice (see Figure 1). If so, a new learning objective might be targeted, and the PBE process could begin again. If the practice was deemed effective but the desired outcome was not yet achieved, teacher candidates consider additional iterations of implementation. Teacher candidates also examine the data to identify ineffective, problematic, and/or unacceptable aspects of the practice. If the practice was not deemed as effective or appropriate for the desired outcome, a new practice might be considered.
Opportunities to practice interpreting data and using the evidence to make instructional decisions are an essential component of teacher preparation programs (van den Bosch, Espin, Chung, & Saab, 2017). In their study of teachers’ comprehension of progress monitoring graphs, van den Bosch and her colleagues (2017) found that, although teachers were relatively proficient at reading the data, they lack the skills to interpret and then use the data for instructional decision making. Through the implementation of the PBE model, teacher candidates are guided through the data analysis process and are then positioned to make evidence-based decisions to continue, revise, or eliminate a particular practice.
Share the Evidence With Others
Once collected and analyzed, teacher candidates can share their PBE findings with relevant stakeholders such as their cooperating teachers, administrators, teacher educators, fellow teacher candidates, and the students and their families. The use of graphs for progress monitoring data facilitates evidence sharing by providing a visual representation of students’ progress and demonstrating the intervention’s effectiveness (Darling-Hammond, 2012). They also can disseminate information about their strategies and PBE to help others (e.g., fellow teacher candidates, cooperating teachers) enhance their own classroom practices. When sharing the evidence with the students and their families, teacher candidates are reminded to avoid educational jargon and use culturally sensitive language (Cook et al., 2012).
Implications for Teacher Preparation and Future Directions
In this article, we provided a description of an inquiry model that we have used to help our teacher candidates understand the process of collecting data about their classroom practices, analyzing those data, and, subsequently, making sound data-based decisions. Implementing the PBE model within a teacher education program aligns with the current focus of providing practice-based opportunities for teacher candidates (Benedict, Holdheide, Brownell, & Foley, 2016). Practice-based opportunities allow for the application of skills and knowledge learned in teacher education courses to be practiced repeatedly throughout the program and across varied settings (Benedict et al., 2016). Many of the features of practice-based opportunities identified by Benedict et al. (2016; for example, modeling, spaced learning, analyzing and reflecting) are utilized in the structured and systematic implementation of the PBE model within our program.
When employed by teacher candidates who are conducting field-based assignments for their teacher education classes, the PBE model reinforces coursework in EBPs through application. This integration of content and pedagogy (i.e., a practice-based opportunity)—practice and theory—may then help to combat a common criticism of teacher education programs: that focus is placed on teaching about effective instructional practices instead of providing opportunities for teacher candidates to use those practices (Grossman, Hammerness, & McDonald, 2009; McLeskey & Brownell, 2015). Encouraging teacher candidates to utilize many of the 22 HLPs (McLeskey et al., 2017) when implementing the PBE model supports CEC’s goal for beginning special educators “to develop highly responsive, explicit, systematic instructional and behavioral interventions that support the success of students with disabilities and responds to their diverse and complex needs” (“High-Leverage Practices in Special Education,” 2017, p. 355). The flexibility of the PBE model to allow for systematic exploration of experientially based practices also adds value to the teacher candidate’s training experience. Cook et al. (2008) noted that “it seems neither possible nor desirable to mandate that special educators use only a limited array of practices, even if those practices are evidence based”; instead, high-performing special educators should be able to “remain flexible in their instructional approaches and seek out innovative practices for some students who do not respond to EBPs” (pp. 72-73). That is, in addition to recognizing the value of EBPs, the PBE model empowers students to interrogate and support their professional intuition through an explicit process of systematic inquiry.
Data from assignments using the PBE model have provided important program-level data for our accreditation efforts. Furthermore, program review and revisions supported by a U.S. Department of Education 325T grant revealed the effectiveness of exposing our candidates to the PBE model in multiple places along their course sequence. That said, future research for wider dissemination using the PBE model might explore numerous targeted questions about teacher candidates’ engagement with components of the model (e.g., How do candidates reflect on practice and make future instructional decisions based on the data they collected?) or the model as a whole (e.g., How does the use of the PBE model facilitate evidence-based decision making among teacher candidates?). To assist in establishing the role of PBE and its importance to the field, further research should include validation of this conceptual model.
Each school year, educators are exposed to a range of evidence-based and experientially based practices, and challenged to make evidence-based decisions about which ones to use. Although there are practices that have numerous research studies documenting their effectiveness, the PBE model presented here helps educators determine if a particular EBP works with their students in their learning environments. Similarly, the model also can be used to validate promising and experientially based practices. Educators and teacher candidates who utilize this PBE model can then have an impact on future research, thus demonstrating the cyclical relationship of EBPs and PBE and the importance of partnerships between practitioners and researchers as noted by Cook and Cook (2016).
Footnotes
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The contents do not necessarily represent the policy of the U.S. Department of Education, and you should not assume endorsement by the federal government. Project Officer, Sarah Allen.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The contents of this article were developed under a grant from the U.S. Department of Education, H325T100002.