Stepping-Up Technology Implementation

Abstract

While technology-based interventions enhance instruction and improve outcomes for students with disabilities, implementing and integrating technology in authentic learning environments continues to be a challenge. Based on the experiences of a variety of Stepping-Up Technology Implementation projects funded by the U.S. Department of Education, Office of Special Education Programs, this mixed-methods study explored the essential factors for the successful implementation of technology-based interventions in K-12 schools and early childhood programs. Based on the qualitative analysis of projects’ implementation reports and responses to the follow-up questionnaire, four major themes emerged. The barriers and facilitators to technology implementation were reported across such areas as (a) developing and sustaining buy-in, (b) ensuring implementation fidelity to support the intervention, (c) research-to-practice dilemmas, and (d) data serving multiple purposes. The discussion and practical implications for supporting technology implementation are provided.

Technological innovations have revolutionized special education and created new possibilities for students with disabilities by supporting their unique needs (Edyburn, 2013; Kennedy & Boyle, 2017). When implemented with fidelity, educational technology enhance instruction and improve outcomes for students with disabilities (Israel, 2019; McLeskey et al., 2017). The teachers of these students should be equipped to select, implement, and monitor use of the technology tools, although teacher education programs in the U.S. vary widely in terms of preparing savvy consumers of educational technology (Rock et al., 2017). While there has been an explosion in the development of evidence-based technological innovations in recent years with the majority of K-12 classrooms in the U.S. using digital age technologies every day, the challenge researchers continue to face is translating their research into everyday practice (Kennedy & Boyle, 2017). Even with laws requiring the provision of assistive technologies to students with disabilities in the U.S. (Individuals with Disabilities Education Act, 2004) and efforts to decrease disparities in access to educational technology across contexts (Every Student Succeeds Act of 2015, 2015), differences in the availability and accessibility of educational technology tools in U.S. schools and elsewhere around the world persist (Gallup, 2019; Ong, 2020). Further hindering implementation and sustained use of evidence-based educational technology in the classroom is the complexity of technology itself including the technology’s relevance to the target environment, knowledge and skills of the user implementing the technology, and available resources and organizational structure to support training and coaching (Cook & Odom, 2013). One approach to address the challenges of implementing and integrating technology in authentic learning environments is implementation science, which is the study of methods to promote the uptake of EBPs into real-world contexts (Eccles & Mittman, 2006).

Literature Review

Many models of implementation science exist to facilitate the effective adoption of EBPs. Fixsen and colleagues from the National Implementation Research Network (NIRN) developed a series of Active Implementation Frameworks that apply to implementing and integrating EBPs, including educational technologies, in authentic learning environments and then scaling-up and sustaining these practices and technologies in multiple sites (Fixsen et al., 2009). The model consists of four implementation stages and multiple implementation drivers organized into three categories (Bertram et al., 2015; Blasé & Fixsen, 2013). The four stages of implementation describe a comprehensive non-linear process that requires a few years of trial and error before finding a sustainable fit for new innovations (Fixsen et al., 2009). The stages include:

1. Exploration—investigating a fit between innovation and setting; identify resources needed to implement it.

2. Installation—preparing to start the innovation (develop protocols, confirm resources).

3. Initial implementation—launching innovation. This is a period of learning and growth as it requires rapid problem solving and reassessment of tools and strategies. The procedures are refined, and the iterative process of implementation, feedback, and change happens.

4. Full implementation—fully integrating innovation into classroom and school district settings. In this stage, fidelity and outcomes are measured to continue improving the impact of the innovation in practice (Bertram et al., 2015).

Embedded within the NIRN Active Implementation Frameworks are implementation drivers used to integrate and sustain innovations over time. A discussion of the implementation drivers occurs throughout the implementation stages, where they are selected, planned for, installed, refined, and sustained along with the innovation itself (Fixsen et al., 2009). The three main categories of implementation drivers include (a) competency, (b) organization, and (c) leadership (Blasé & Fixsen, 2013). Specifically, competency drivers include selecting, training, and coaching staff who deliver the technology-based program. First, the team must be selected with the required skills and abilities to serve the program’s needs. Next, the team receives training to learn when, how, and with whom to use these new skills. Finally, coaching strategies are put into place to increase mastery, while fidelity assessments ensure consistent and appropriate implementation of the innovative practices (Fixsen et al., 2013).

The organizational drivers include decision support data systems, facilitative administration supports, and systems interventions. As described by Fixsen et al. (2009), decision support data systems include collecting accurate and reliable data to inform decisions at the student, teacher, and staff level to implement the innovations. Facilitative administrative supports seek to identify challenges, develop clear communication, and reduce administrative barriers for the technology program. Systems interventions are strategies used by the organization to ensure alignment with external systems not under the administration’s control (e.g., local, district, state priorities, and regulations; Fixsen et al., 2009). Lastly, leadership drivers include technical and adaptive strategies. The goal of leadership drivers is to establish reliable management for productive output of intervention strategies. The leadership driver refers to a core group of people in the organization to continue running effective programs while adapting to new circumstances. These leaders utilize “adaptive drivers” when they support an organization through changing procedures and philosophies. They are the “champions” that create and maintain buy-in throughout the implementation stages. Leaders apply technical strategies to focus on the necessary details and specific activities for adopting a new technology-based practice or program (Fixsen et al., 2013).

Technology Implementation and Integration—Scaling-Up

Scaling-up innovations, especially technology ones, is generally difficult in educational settings (Dede et al., 2005). While technology innovations have the potential to improve learning outcomes for students across ages and abilities, simply providing access to technology does not ensure its use or the realization of potential outcomes (Grunwald & Associates, 2010). Instead, technology must be aligned with the goals and practices of the systems where it is being implemented (ISTE, 2021; Perlman & Redding, 2011). Two significant challenges of this successful technology implementation and integration are that technologies are often difficult to scale in contexts where they were not developed, and researchers and developers often have little incentive to scale their solutions prior to broad dissemination (Looi & Teh, 2015b). Despite these challenges, educational and technology innovations have been scaled successfully (Looi & Teh, 2015a); however, no single approach or formula has emerged as the impetus for these successes (Pang et al., 2015). Instead, general practices that support successful scaling vary a great deal and typically include attention to various dimensions including depth, sustainability, spread, shift (Coburn, 2003) and evolution (Clarke & Dede, 2009) as the technologies and interventions are implemented locally (Means & Penuel, 2005).

Stepping-Up Technology Implementation Funding Program

This study explores the experiences of a variety of Stepping-Up Technology Implementation projects funded by the U.S. Department of Education in the Office of Special Education Programs (OSEP). This project first started in 2012, supporting 35 projects by the time of the study. The overarching purpose of all projects is to implement and integrate educational technology, with a strong evidence base, in early childhood programs and K-12 classroom settings. The innovative technology is meant to engage, motivate, and benefit learners with disabilities as well as to improve teaching pedagogy. All projects work directly with educators, students, and families by providing coaching and professional development (PD) to understand and use the technologies. The target users are expected to be ready to integrate the technology tools in their practices. Projects are required to test the implementation of their technology as well as provide supporting materials and products in three development, four pilot, and 10 dissemination sites. This enables the technologies to be scaled-up and sustained across multiple settings, while documenting the impact on educators’ use and learners’ outcomes.

Across the 35 individual projects located at Institutes of Higher Education (IHE) and research institutes, the technologies are being implemented to support learners with disabilities in a wide array of areas: literacy (n = 11), science/math (n = 8), behavior management and social skills (n = 5), college/career readiness (n = 3), PD for in-service trainers (n = 3), teacher coaching (n = 2), early childhood (n = 1), progress monitoring (n = 1), and student engagement/dropout prevention (n = 1). B, At the time of the study, the projects have disseminated technology across 350 sites in 37 States and the District of Columbia. They supported over 457 K-12 school sites and early childhood programs (e.g., public, private, charter, alternative, and vocational); improved outcomes for over 36,301 students, including more than 10,150 students with disabilities, 26,145 students in general education and 130 English Language Learners; as well as supported over 3745 educators, 540 parents, and 40 early childhood providers.

Study Purpose and Research Questions

The purpose of this study was to identify factors essential for the successful implementation of technology-supported interventions in K-12 schools and early childhood programs serving children with or at-risk for disabilities. It also aimed to characterize common challenges and solutions to conducting implementation research in authentic education settings.

Method

A sequential exploratory mixed-methods design was used in this study (Creswell & Plank Clark, 2017). Both quantitative and qualitative data were rigorously collected and analyzed to explore the implementation of technologies in authentic settings. The initial qualitative data collection and analysis was followed by the quantitative phase. Then, the data from each phase were mixed to corroborate the overall findings.

Participating Projects

Stepping-Up Technology Implementation projects that had been funded by OSEP at least 1 year prior to the study were invited to participate. Twenty-six projects had received at least 1 year of funding and submitted at least one annual report by the beginning of this study. All 26 project Principal Investigators (PIs) consented to participate. They represented 15 IHEs and 2 research institutes. Of those projects, 70% implemented technology with students in K-12 settings (n = 4 in elementary grades; n = 8 in secondary school; and n = 6 across grades); 27% focused on educators (n = 7); and 3% was based in the early childhood setting (n = 1). The innovative technology-based interventions implemented with learners supported literacy (28%), science/math (22%), college/career readiness (16%); behavior management and social skills (11%), literacy and assessment for students with complex needs (11%), progress monitoring (6%), and student engagement/dropout prevention (6%). Learners supported by technology included students with learning disabilities, emotional disturbance and behavior disorders, other health impairments, autism, deaf/hard of hearing, visual impairments, students with significant disabilities, complex communication needs, and speech impairments, English language learners as well as students at risk for school failure and infants/toddlers with or at-risk for disabilities.

The majority of the projects (57%) have used the National Implementation Research Network (Blasé & Fixsen, 2013) to guide their technology implementation project. Among other implementation models, the following were reported:

• Practical, Robust Implementation and Sustainability Model (PRISM) (Feldstein & Glasgow, 2008)

• RE-AIM Framework (Glasgow et al., 1999)

• Dynamic Sustainability Framework (Chambers et al., 2013)

• The Consolidated Framework for Implementation Research (Damschroder et al., 2009)

• Others; for example, Knowledge-to-Action Framework (Graham et al., 2006); adaptation of the Knoster Model (Knoster, 1991)

Based on the most frequently used Blasé and Fixsen (2013) implementation model, 30% of projects (n = 7) were in the exploration stage, 19% (n = 5) were in the installation stage, 12% (n = 3) were in the initial implementation, and 42% (n = 11) were in the full implementation.

Qualitative Phase

This study began with a phase of collecting and analyzing qualitative data.

Data Sources

The primary data sources for this study were the implementation narrative reports submitted by the project research teams. At the end of each implementation year, all Stepping-Up projects prepared and submitted a descriptive “story” of the technology implementation process. Projects were given a template including guided questions around how the evidence and experiences were used as part of the iterative process to develop and scale-up the successful implementation of the technology tools and products. Qualitative and rich descriptions were requested. These reports were used as a running record for both documenting and reflecting on the key decision points, critical events, and important influences guiding the technology implementation. The templates were tailored to align with different implementation stages: exploration (submitted after year 1), exploration and installation (submitted after year 2), installation and initial implementation (submitted after year 3), and implementation (submitted after year 4). Year 1 templates focused primarily on working with sites (e.g., selecting sites, gathering information about sites, preparation work with sites, profes professional development for the implementation team, developing products to support technology delivery, influencing factors). In addition to all the similar guiding questions around working with sites, Years 2–4 templates had additional sections focused on the supports provided to the users (e.g., coaching, PD, materials, and products), implementation of technology, refinement plans, and plans for measuring outcomes.

Data Organization

After obtaining all the necessary permissions from the Institutional Review Board and receiving consent from the 26 eligible projects, the annual implementation “stories” were collected and analyzed. These reports were on average 35.6 pages long (SD = 32.02), ranging between 6 and 26 pages for Year 1: Exploration reports; 13–49 pages for Year 2: Exploration and Installation reports; 12–27 pages for Year 3: Installation and Initial Implementation reports; and 25–122 pages for Year 4: Implementation reports.

In order to minimize bias, the “stories” were first disaggregated according to the implementation drivers or key components necessary for the successful implementation and sustainability of the innovation (Blasé & Fixsen, 2013). The first two authors read each report and assigned a tag to each text block including: (1) selection and recruitment; (2) training (prior to the implementation); (3) coaching (during the implementation); (4) fidelity of implementation; (5) decision support data systems (usability data); (6) facilitative administration (internal policies and regulations on sites that can be controlled); (7) systems intervention (external policies and structures outside of control); (8) leadership; (9) description of the tool (including accessibility); and (10) sustainability plans. The two researchers reviewed all tagging and discussed those cases where tagging was unclear. After that, a graduate research assistant copied and pasted all text blocks into separate documents representing the aforementioned drivers. While all project identifying information was removed, the text within each driver was organized according to the implementation stage in an effort to explore possible patterns based on the stage.

Data Analysis and Credibility

After the “stories” were disaggregated and de-identified, 10 documents representing implementation drivers were shared with the full research team. The content within each driver was then analyzed in researcher dyads using thematic analysis (Braun & Clark, 2006; Guest et al., 2012). First, each document was read and re-read while looking for patterns. Text, organized by driver, was analyzed using open coding and compared across implementation stages to develop categories. The categories were color coded throughout the document and were later merged into three to four core themes within each driver. These thematic descriptions from each of the data sets were then provided to the second author for axial coding in order to link the themes across all of the data. This included an initial compilation of 38 themes. Observing some redundancies, connections, and subcategories across these themes, 31 of these themes were then sorted under six overarching themes. The second author conducted a virtual peer review with the researcher dyads, as needed. Finally, the matrix of themes was virtually shared with the research team for feedback and to ensure that the themes captured the meaning of the data accurately and that the themes were integrated in a systematic way. Following researcher triangulation, the team established four overarching themes. In order to establish credibility and trustworthiness of findings, peer reviews were used to examine all open and axial coding. Specifically, the researchers reviewed and commented on each other’s emerging codes using a shared platform (i.e., Google Docs). During research meetings, the debriefing and general discussions about categories and themes occurred. In addition, the data were triangulated across different projects, implementation stages, and data sources. Specifically, a matrix included themes, subthemes, and examples from multiple reports and questionnaires combining multiple perspectives on the same phenomenon. Finally, the follow-up questionnaire was used for member checking.

Quantitative Phase: Data Collection and Analysis

In order to triangulate the qualitative findings in this study, a follow-up questionnaire was developed by the research team. The first several questions asked projects to rate how much emphasis each project placed on innovation and sustainability of their technology. Then the four overarching themes from the qualitative analysis of implementation reports were presented. Projects were asked to agree/disagree using a five-point Likert-scale on how much they could relate to each theme as well as provide additional examples. Specifically, projects were asked to rate the following statements:

• Our project experienced many competing initiatives in our implementation settings and had to be creative in “selling” our technology-based intervention.

• Our project effectively monitored and worked to improve users’ knowledge of the intervention after training (e.g., through coaching and/or other ways).

• There were instances when the fidelity of implementation was low at our sites.

• In our project, we differentiated between usability testing studies (e.g., observing the technology being used by target user groups) and outcome studies (e.g., evaluating the impact of the technology intervention package on target skills and understandings).

If any of these statements were rated as 4 or 5 (agree or strongly agree), the projects were prompted to provide descriptive examples for (a) ways to better “sell” the technology-based intervention in order to improve users’ outcomes; (b) strategies to monitor and/or improve users’ knowledge after training; (c) action steps (if any) taken during the study with low fidelity of implementation as well as changes incorporated before subsequent stages of the project to improve fidelity, and (d) examples of how projects tested the usability and quality of technology by itself versus the impact of the technology intervention on user outcomes.

The questionnaire was developed by the first author and reviewed by the research team representing the experts in the field of technology implementation. The questionnaire was designed to take about 15–20 minutes to complete. One questionnaire per project was completed either by the project lead or by the person preparing the annual implementation report. Overall, 23 out of 26 projects provided complete responses resulting in the 88% response rate. The responses to the Likert-scale questions were analyzed using frequencies and descriptive statistics. The responses to the open-ended questions were analyzed using a similar thematic analysis described earlier, where all the similar responses were grouped into categories and themes (Braun & Clark, 2006; Guest et al., 2012). The researchers were asked to review and comment on all the responses and emerging themes to ensure interrater reliability.

Results

Qualitative Phase

The qualitative analysis across all project reports was conducted to identify how the implementation and integration of technology happens in authentic learning environments. The data revealed common patterns across the reports regardless of each projects’ stage of implementation. Especially pronounced were the barriers and facilitators to the implementation of technology integration. Four themes regarding the implementation and integration of technology are reported: Developing and Sustaining Buy-In, Ensuring Implementation Fidelity to Support the Intervention, Research-to-Practice Dilemma, and Data Serving Multiple Purposes (see Figure 1).

Figure 1.

Important considerations for implementation and integration of technology.

Developing and Sustaining Buy-In

The first theme characterizing the 26 project reports focuses on how projects navigate introductions of an innovative technology at the school and/or district level as well as how to maintain use of the technology.

Developing Buy-In. Data collectively identified buy-in as the greatest challenge for recruiting sites to use the technology. Project implementation reports described the initial step of the buy-in process as promoting or disseminating information about the intervention and sharing the unique focus of each technology-based intervention. During this step the targeted sites also shared their infrastructure and technology capacity (e.g., hardware, Internet bandwidth, and IT support). Typically, this sharing process began between project personnel and the individuals directly involved with the implementation of the technology such as school leadership and/or district leaders who may serve as the gatekeepers in the district’s decision-making process. Buy-in of the site was also reported to be influenced by whether or not the technology facilitated the use of state mandated curricula and/or testing policies at the time of implementation. For example, one project’s social skills intervention was implemented in one state where educators and policy makers had recently responded to a need of improving social skills for students with disabilities with the adoption of Social and Emotional State Standards.

When the intervention was not closely aligned with the targeted curricula, the competing initiatives were a challenge to implement. For example, several projects reported that a consistent barrier to buy-in was a lack of time to commit to the use of the intervention and/or the PD required for successful implementation of the intervention as intended. Sometimes the school/district partners had the foresight to this dilemma prior to the onset of training, whereas others discovered that scheduling sufficient time was a substantial barrier after they began training and initial implementation. Projects also stated that school leadership largely influenced teacher behavior and, in turn, student outcomes. For example, as one project noted, “when there was authentic buy-in from a school director, there seemed to be greater teacher interest and commitment to following through on a PD and coaching program.” A school’s culture and organizational “health seems to be an important factor in the implementation and sustainability.”

In order to increase buy-in more directly, project personnel provided testimonials, narratives of positive case studies, and offered incentives. Examples of incentives included stipends and gifts for participants, substitute teacher costs, purchase of resources related to the intervention, family engagement events, as well as “offering wait-listed comparison options” for control sites. Another part of the buy-in process was for the projects to clearly understand the users’ needs and allow flexibility in addressing those needs. Some projects directly observed implementation to better understand current practice, while most project personnel utilized a readiness assessment (e.g., surveys and/or interviews of school staff) to efficiently “match” the intervention with the needs of the implementation site. Continuous improvement of the validity of the readiness assessment was reported to be essential to the process. Readiness instruments allowed projects to determine the users’ philosophical beliefs, pre-requisite skills, technical capacities, school(s) or district(s) needs, and to ensure that the process of selection was mutual between the project and the participants.

A lack of compatibility between the district’s technology infrastructure and the intervention’s platform or operating system was reported across projects as a key barrier. For example, an Android-based intervention could not be implemented in a school district that used iOS devices exclusively. Also, some projects reported that sites had non-negotiable technology policies that conflicted with the delivery of the intervention. For example, sites may have had limited capacity of the platforms and/or filters which blocked access to the intervention.

Sustaining Buy-In. Once initial buy-in was established, selected users reportedly learned the skills and information needed to begin technology implementation. Providing a space for the users to try on the new skills before use with students reportedly facilitated users’ confidence and assured the users that the technology-based intervention was both worthwhile and aligned with their organizational structure and priorities. Delivering high quality PD was also a consistent element that reportedly contributed to sustained buy-in. Furthermore, buy-in was reported to be most successful when a prior relationship was established between project personnel and schools/districts. Cultivating community buy-in and collaboration among community partners and commercial technology companies supported the recruitment process and adoption of the technology products into school structures. For example, one project intentionally and proactively highlighted the alignment of the intervention with commercially available devices. All projects held implementation team meetings involving project personnel and the technology users with face-to-face meetings potentially contributing to the grounding of the partnership and sustaining buy-in across projects.

In the absence of having a prior relationship with the implementation sites, project reports indicated that an efficient, consistent, and flexible research implementation team was needed to establish positive relationships within the school culture and with the teachers. For example, a research team supporting implementation of a writing intervention reported, “At the building level, a needs assessment was necessary to determine how teachers perceived writing, how they went about facilitating writing instruction, how they provided feedback in the writing process, and how they assisted struggling learners with more direct and specific instruction”. Another team reported that meeting teachers and administrators at the school rather than over the phone or at their campus offices “helped them feel comfortable to ask any and all questions they might have”. Project reports also described ways they fostered the beneficial partnership. Foremost, the reports highlighted the collaborative efforts between the user and project personnel to integrate the technology effectively and support the fidelity of implementation as well as the sustainability of implementation after the project “ends”. For example, data from the reports described professional learning communities (PLCs) or groups of users and project personnel having regular meetings in order to support implementation and engage in a cycle of feedback. In addition, project reports recognized leaders of and members of the instructional technology departments at school sites as key facilitators to implementation.

Finally, reports identified that sustainability was largely conceptualized by school and district partners as having ongoing access and use of the technology-based intervention when project personnel would no longer be available. Creating open-access materials under creative commons license for non-commercial use, relying on “barebones” ready-to-use solutions, and making all or part of the technology available at no cost are just some examples of how projects ensured ongoing access to their interventions.

Ensuring Implementation Fidelity to Support the Intervention

The second theme characterizing the data addresses how implementation fidelity was achieved across the projects. Before use and during the integration of the technology products at sites, project personnel provided ways to ensure fidelity of implementation. Those included the ongoing refinement of materials, use of coaching models, and an emphasis on structural fidelity.

Refinement of Materials. Materials and resources refer to the project’s website, manuals and tutorials, marketing and recruitment materials, tips and examples, checklists and self-assessments, observational forms, and user logs. Regardless of the project’s stage of implementation, delivery of the intervention and its accompanying supporting products were highly responsive to the users’ needs. Project reports detailed how personnel revised and/or modified the developed materials in an iterative manner in response to user feedback and/or performance outcomes. For example, project personnel provided redundant resources and materials in varying formats to provide not only convenience, but to ensure clarity and comprehension of the technology products among the varied users. These multiple means of representation suggest that projects collectively embraced principles of the universal design for learning (UDL) framework to guide the implementation process. UDL is a framework to improve and optimize teaching and learning for all by setting clear, rigorous goals; anticipating barriers; and proactively designing to minimize those barriers. Teachers use UDL for all learners based on scientific insights into how humans learn. It is a core lever of change to help make learning inclusive and transformative for everyone. The three principles of UDL call for multiple means in how we represent materials and information - representation; multiple means for learners to demonstrate their understanding - action and expression; and equally important, multiple means to gain and maintain learner attention throughout the learning process - engagement. The three principles are further broken into 9 guidelines and 31 checkpoints to facilitate UDL implementation (https://udlguideliens.cast.org).

As an illustration of UDL application, one project reported relying on “videos with captions and transcripts, a glossary, visual supports such as tables and images, review sheets, checklists, and hands on explorations.” These represent all three guidelines under multiple means of representation: provide options for perception (e.g., captioned videos), for language and symbols (e.g., glossary), for comprehension (e.g., visual supports). Several projects also used a combination of formats to deliver those materials: facilitated, self-paced, individualized, face-to-face, online, and blended (representation—options for comprehension). Sometimes, the users needed the materials delivered with a higher frequency or duration (engagement—sustaining effort and persistence), while other projects created different content to “fit” the users’ needs and ability to respond (action/expression—options for expression and communication). The malleability of the materials and the flexibility in the delivery and use of the materials was a consistent thread across all project reports at every stage of implementation.

Coaching Models. OSEP clarified the term “coaching” to all grantees as on-the-job or embedded training to help teachers practice and implement new skills after having initial exposure through training or PD of the technology product. Across all 26 project reports, on-the-job training was used to support the fidelity of implementation, however implementation varied across projects. These variations are summarized in Table 1.

Table 1.

Coaching Features Reported Across Projects.

Features	Examples
Frequency	Weekly, bi-weekly, monthly
Source	School-based/onsite coaches, offsite/external coaches, teachers
Strategies	Performance-based feedback, goal setting, reflection, modeling, role playing
Format	On-demand, prescheduled, in-person, remote, on-the-ground, blended
In-person	Consultative meetings, informal check-ins
Remote	Zoom, Email, text messages, phone, video-based feedback, classroom videos submitted for review and feedback
Supports	How-to videos, vignettes, guidelines, implementation manuals, printable step-by-step guides

As a result of coaching, project personnel reported that they were able to identify barriers and facilitators to implementation in order to make adjustments and improve student outcomes. What was not largely represented in the data was how projects determined which coaching strategy to use and/or the dose of coaching needed. Reports included in the analysis suggest that measures of structural fidelity of implementation were used to determine coaching needs.

Measuring Fidelity of Implementation. The reports submitted by projects included the descriptions of how fidelity of implementation (FOI) was measured and the subsequent fidelity outcomes. Although the outcome data of the FOI were not the focus of this analysis, projects did report fairly high rates of FOI (with associated interobserver agreement reported) and a process of coaching when the FOI waned among users. Among the multiple dimensions of fidelity (Harn et al., 2013), structural FOI was predominantly represented. That is, assurances were made that the key components of the technology product were taught and the allotted time for implementation was followed. Projects used reliable measures of adherence regularly throughout implementation in the form of a checklist of the critical elements of the technology-based intervention, implementation logs (either within the technology or outside) of time spent, rating scales, and/or self-assessments (e.g., interviews, surveys). The degree to which the users “covered” the technology-based intervention was measured by (a) direct observation in real time or via video recordings, (b) permanent product produced, or (c) self-report. For example, one project measured FOI by reviewing usability data collected by the tool with “logs being used to measure time spent at each step and to track the success of implementation”. The assessment and reporting of implementation fidelity across projects were consistent with the standards for evidence-based practices (Cook et al., 2013). However, less evident in the data were the measures for how well the technology intervention was taught and/or how or if the intervention was differentiated (e.g., time, content, behaviors) across different contexts and individuals.

Research-to-Practice Dilemma

The third theme that emerged from the data reinforces a persistent and familiar barrier of translating research and development to practice. The research-to-practice barrier was apparent across all stages of implementation. Dilemmas on the research into practice paths that were identified by the projects included the balance of meeting research requirements while still being responsive to the practical needs of the site, time, and environmental challenges.

Meeting Research Requirements. In order to provide context for the reader, the educational research process used by a majority of the OSEP projects was design-based, since it was a requirement to work collaboratively with school-based personnel to refine a tool or solution that improves student learning, develop products and materials that would support its implementation, have participants use the tool with fidelity, and then make revisions as needed based on the data. The design-based research is an iterative method allowing continuous analysis, design, development, and implementation of an intervention (such as technology) in effort to improve educational practices (Wang & Hannafin, 2005). Project reports provided outcomes of the iterative research cycles as they replicated the design process with new users. However, this process provides more flexibility than other research methods, researchers still follow a coherent and logical progression of activities in order to effectively address the research problem. For example, prior to any implementation involving human subjects, parental consents and permissions for the administration of pre-assessments are required. These types of approvals specific to research procedures reportedly delayed projects’ onset of implementation.

Time. However, research personnel communicated general timeframes for establishing buy-in with the implementation sites, for recruitment, and for implementation, these were only estimations and in practice, there were ongoing adjustments made to the schedule of events. For example, despite the best intentions and promising starts, project personnel reported limited time: (a) for potential sites to determine fit of the intervention, (b) for initial training or PD to use the intervention, (c) for the administration of pre-assessments, as needed, (d) for intervention implementation while establishing fidelity, and (e) for ongoing coaching to maintain FOI. Projects reported that finding time to meet with teachers about the intervention or for coaching was challenging when teachers had obligations before, during, and after school hours.

Environmental Challenges. Other events reported in the data that interfere with bringing research to practice include environmental constraints such as time differences and inclement weather causing school sites to close. Time differences were a challenge for delivering real-time feedback and scheduling coaching sessions, especially since some of the projects’ researcher-practitioner partnerships were across the U.S. Finally, project personnel at the initial stages of implementation acknowledged adjusting to specific needs and assets of project site communities; for example, navigating family schedules, transportation or childcare needs, and community building strengths.

Data Serving Multiple Purposes

Finally, the types of data and the varying utility of data were prominent across all project reports. There were several sources of qualitative and quantitative data that research projects gathered in their work. Generally, these included process data (e.g., usage log data, user clicks, number of downloads), student outcome data (pre-post test data, progress monitoring data), teacher outcome data (e.g., teacher beliefs, self-efficacy), user perceptions, and fidelity data. These data served multiple purposes related to (a) guiding the iterative development of the tool, (b) determining the impact on student learning, and (c) establishing FOI.

To Guide the Iterative Development of the Tool. The Stepping-Up Technology Implementation reports included data that were important to the iterative development and evaluation of the technology used to scale particular EBPs. Specifically, projects described systematic ways they collected and used data to make changes to the technology-based tool, components of the PD, and/or coaching strategies after each testing cycle. For example, a majority of OSEP projects used surveys and questionnaires referenced in the reports as “evaluations” to test student and teacher usability and satisfaction with the projects’ tool. These evaluation type data were then, per OSEP’s expectations, used to measure the quality, relevance, and usefulness of the tool and subsequently guide the iterative development and refinement of the technology. The grantees regularly used the terms “quality”, “usefulness”, and “relevance”. Usefulness was consistently measured through surveys, questionnaires, and interviews that asked if different experiences (e.g., training), tools (e.g., teacher toolkit), and technologies were “useful”. In addition, project personnel reported using the feedback provided from advisory board members to inform usefulness. Projects also used qualitative methods such as interviews and direct observations to explore the interactions users had with the technology throughout all aspects of implementation. The observational data were then used to inform further revisions of the tool or products. The grantees reported many ways they sought feedback regarding the features of the technology and the kinds of updates they made as a result, but they did not directly test or evaluate the impact of those particular features per se. Specifically, the reports were absent of evaluating the impact of different features on student learning outcomes versus other features.

To Determine Impact on Student Learning. Data were gathered and analyzed across projects to evaluate the impact of the evidence-based practice as delivered by the technology on student learning. Examples of student outcome measures represented in the project reports include academic achievement and content knowledge (in reading, writing, math, science, research); transition and career awareness; engagement and motivation; behavior management and social skills; communication abilities; growth in developmental outcomes such as early language, cognitive problem solving, and motor skills. Whereas reports included student outcome data to demonstrate the promise of the evidence-based practice as delivered by the technology on student learning, some project tools included a process so that multiple users at different levels such as teachers and school leaders and district teams could access the data relevant to them. Such reports included detailed descriptions of how they used technology to develop a database for stakeholders to use the student-level or class-level data to support instruction. The participants could then actively use the data to make instructional decisions. For example, one project developed a progress monitoring tool that collected student writing samples and provided automated scoring for the teacher. The data were then displayed in a useful and meaningful way for teachers to assess student progress and determine appropriate strategies.

To Establish Fidelity of Implementation. The project reports included data to establish fidelity of the technology-based intervention’s implementation. As mentioned previously, structural fidelity data was represented in the reports with the use of checklists, logs, and rating scales. The project reports described that fidelity data were gathered by observation and/or via self-assessments and were reported to be at adequate or above average levels. However, there was one project that included a related measure, referenced as a “proof of learning” prior to intervention implementation. Specifically, there was a criterion on a given knowledge and summative performance test to “pass” before gaining access to subsequent implementation materials. The project reported that this type of gatekeeper was used to solidify that learning objectives from the training component were achieved.

The documentation and subsequent analysis of 26 implementation reports helped us deepen our understanding of what happens throughout the implementation of varied technologies for students with disabilities. These findings were corroborated and further explained by the quantitative results described next.

Quantitative Phase

Based on the responses from 23 out of 26 projects, there were the following averaged agreements with the overarching themes, based on a five-point Likert-scale:

• Our project experienced many competing initiatives in our implementation settings and had to be creative in “selling” our technology-based intervention—3.5 (SD = 1.24).

• Our project effectively monitored and worked to improve users’ knowledge of the intervention after training (e.g., through coaching and/or other ways)—4.52 (SD = 0.73).

• There were instances when the fidelity of implementation was low—3.96 (SD = 1.02).

• In our project, we differentiated between usability testing studies (e.g., observing the technology being used by target users) and outcome studies (e.g., evaluating the impact of the technology intervention package on target skills)—4.35 (SD = 0.83).

In addition to these ratings, a qualitative analysis of open-ended follow-up questions expanded our understanding of each theme. Based on the responses from 16 projects (62%), the ways to better “sell” the technology intervention focused on (a) aligning the tool with federal, state, district, and school-based education policies, standards and initiatives, (b) demonstrating how the technology can improve everyday practices and result in superior outcomes, and (c) relying on user testimonials. Flexibility of technology as well as ensuring that the target users have the sufficient background knowledge to implement the tool were reported as important elements for initial and sustained buy-in. Often “selling” had to go beyond the tool. As one project described, “We needed to help schools understand the commitments needed when adopting our technology, as it also meant adopting a practice, which in turn could be seen as something to compete with existing or other initiatives.”

Based on the responses from 22 projects (85%), some strategies to monitor and/or improve users’ knowledge after training included (a) virtual coaching and regular site visits and/or meetings, (b) building a community of practice, (c) remotely monitoring technology’s use and sending reminders, and (d) developing checklists and forms to accompany tutorials and guide implementation. These continuous check-ins were very important. As one project summarized, “Of course, “all” the content had been addressed in some way during the initial training, but we found teachers did not really “hear” it until they “needed to”...until they saw something work (or not work) in THEIR classroom.”

In an effort to improve the FOI, suggestions from 17 projects (65%) have been organized in the following action steps: (a) collect data on the FOI with immediate feedback and coaching, (b) develop comprehensive instruments with clear explanations and ample examples or teacher anecdotes, (c) disseminate regular emails and/or newsletters with tips, tricks, and direct links to project resources, (d) demonstrate the results of previous studies showing improved learning outcomes when implementing intervention with fidelity, (e) assess readiness and communicate clear expectations prior to providing access to technology, as well as (f) individualize technical assistance when needed.

Finally, based on the responses from 21 projects (81%), many have purposefully tested the usability and quality of technology by itself separate from testing the impact of the technology intervention package on user outcomes. Different types of data collected to inform the successful technology implementation included: (a) usability data (e.g., ease of use, interface) through focus groups, observations, and review of user logs, (b) pre-post outcome data (e.g., specific to the target skill), (c) implementation data (e.g., dosage, fidelity), and (d) sustainability data (e.g., surveys). While some projects tested technology usability prior to the implementation across dissemination sites, others continued to test usability throughout the duration of the project. In some cases, the satisfaction rating scale was built-into the technology to continuously receive feedback from users. All projects’ suggestions were organized into a compilation of practical implications for scaling-up technology implementation. They are presented below.

Discussion

This study represents experiences of 26 Stepping-Up Technology Implementation projects funded by OSEP that first began in 2012. All projects were charged with scaling-up the implementation of innovative, evidence-based technology-driven education interventions for special education students and teachers, from early childhood through high school. They were guided by implementation science models in order to translate these EBPs into an authentic classroom environment. The goal of this study was to identify factors that facilitated and challenged the implementation of these large-scale technology-based education interventions among projects with a similar mission but differing audiences and implementation needs. A mixed-methods research design was utilized in the collection and analysis of this study data.

Various implementation science models guided the work of the participating projects; however, for this study, project teams shared qualitative and quantitative data related to how their project could be viewed from the lens of the National Implementation Research Network (NIRN). This model consists of four main implementation stages and implementation drivers, yet the themes that emerged in this study were equally represented across all stages of implementation (Bertram et al., 2015; Fixsen et al., 2009). The four most salient thematic factors that impacted the implementation efforts across 26 projects included a) developing and sustaining partner buy-in; b) ensuring implementation fidelity to support the intervention; c) navigating the research-to-practice continuum; and 4) utilizing data to guide the iterative development, the fidelity, and the impact of the education technology.

First and foremost, finding the alignment or fit between the education technology and the implementation setting to develop buy-in was vital. Similar to previous research, many projects commented on how initial and ongoing buy-in from partner sites was influenced by how well their tool fit with the goals and values of the implementation site (Perlman & Redding, 2011). This was often driven by state initiatives, internal organization or district priorities already set in place at the school locations. Consistent with the NIRN framework, during the exploration stage, the intervention delivery team and school team must determine if the proposed intervention is of most need to the teacher or student population at the current given time (Bertram et al., 2015). Findings from this study revealed that if intervention delivery teams could make the case for and cultivate, a mutually beneficial partnership, then a strong foundation could be set for sustained adoption of the new technology.

Findings also revealed that teams used implementation drivers such as continuous data collection to refine materials and technology, strong leadership in the schools, in-service coaching and communities of practice for teachers, and fidelity processes in order to sustain buy-in. These essential implementation drivers were useful to support teams' continued integration of the education technology at the school sites. Indeed, previous research that found that strategies to ensure sustainability are critical to bring technology innovations to scale (Clarke & Dede, 2009; Pang et al., 2015). In practice, the use of these implementation drivers can be used to help sustain forward momentum, even when energy wanes during the middle of the academic year, or implementation plan, and various priorities are vying for attention.

Integrating technology-based education interventions from an implementation science perspective takes particular navigation skills and processes to ensure fidelity of implementation. Findings from this study found that beyond cultivating buy-in for the technology intervention itself, project teams encountered challenges with ensuring proper implementation of EBPs. The intention of implementation science allows for the adaptation to site and population specific needs; yet, the ongoing question is how much adaptation is allowed before the integrity of the evidence-based practice is misaligned (Ogden & Fixsen, 2014). To ensure effective delivery of technology innovations, fidelity of implementation should move beyond adherence and dosage, and include measures to assess participant responsiveness, program differentiation, and quality of delivery (Vroom et al., 2020).

Research-to-practice challenges (i.e., needs of the site, time, and environmental barriers) in technology adoption were apparent across all stages of implementation. These research-to-practice dilemmas are consistent with other studies that found that implementing evidence-based programs in authentic settings comes with distinct challenges (Cook & Odom, 2013; Looi & Teh, 2015b). Many factors have an impact on this research-to-practice process - population and organizational culture, socioeconomic factors, leadership and staff capacity, competency and turn-over are some (Ogden & Fixsen, 2014). To bridge the gap from research to practice, implementation teams should consider lessons learned from implementation science and identify change agents to adopt, implement, and maintain EBPs (Cook & Odom, 2013).

One of the most salient themes among projects in this study was the use of data for implementation, specifically for measuring fidelity of the program itself, the use of the technology, and educator or student outcomes. The findings from this study make the case for designing and delivering evidence-based interventions with fidelity within an implementation science framework so that program and participant outcomes show greater improvements (Durlak, 2015; Vroom et al., 2020). These components (fidelity, technology use, and student outcomes) must be measured in order to make a difference during implementation (Vroom et al., 2020). Coming back to the NIRN implementation science framework, decision support data systems is an implementation driver to strengthen the implementation of evidence-based programs; however, it requires having structures in place to collect, analyze and use the data for such purposes, in a timely and functional manner. If data are collected, yet not available to inform decisions in crucial moments of program implementation, then, the value of measuring fidelity becomes less applicable (Bertram et al., 2015; Harn et al., 2013).

Practical Implications

As noted, across the 26 projects who participated in this study, themes from the implementation drivers were derived as well as provided recommendations for researchers and practitioners. Findings have definite implications for future research and practice.

A frequently identified challenge for Stepping-Up Technology Implementation projects was the recruitment of sites and technology users. Researchers should do their “homework” in advance of finalizing a commitment with potential sites. Obtaining information about school structures including competing or coordinating initiatives, policies, school organization, outside influences (e.g., state, district, and parents) before and throughout implementation is crucial in order to avoid potential barriers. Additionally, it is important to clearly provide sites with information about both time commitments and tasks necessary for technology implementation. Suggestions to set the stage for success and implementation with fidelity included both obtaining and communicating clear information within each building prior to startup. Do users have time to implement the intervention and participate in other activities (e.g., PD)? Are there any competing initiatives? Does the intervention closely align with goals, initiatives, as well as curriculum at the site? Do users meet all the necessary pre-requisite requirements? These questions can be answered by collecting data via surveys, readiness assessments, or interviews from administrators, educators, support personnel, students and parents, if applicable. Over-recruitment and early recruitment should be considered. More in-depth discussions, often face-to-face, are useful following the initial recruitment and may include technology demonstrations, testimonials, and clear communication of goals and commitments.

The balance between the interest from the users around the innovation and the readiness requirements that make up the core components required to implement the technology is important. If sites who desire to use the intervention cannot sustain a required technology (e.g., hardware and supporting infrastructure such as internet reliability, bandwidth, and in-school IT support), the implementation becomes incredibly challenging for all, practitioners and researchers alike. Sustainability of a technology-based intervention should be at the core of making a decision about implementation. Developing the technology compatible with the school infrastructure or selecting an intervention that is adaptable/compatible across platforms may be a solution here. This will allow for sustainability of technology from research to practice.

The implementation of technology-based intervention with fidelity requires additional supports such as training, coaching, protocols, instruments, videos, and other “just-in-time” resources. Both researchers and practitioners should consider opportunities for peer-to-peer support and/or PLCs. The findings in this study presented above offer numerous recommendations on how to improve technology integration, which all corroborate that it is not enough to just provide access to technology (Israel, 2019). Continuous measurements of both FOI and impact on learner outcomes using various data sources are essential for successful technology integration in both research and practice.

Limitations

The breadth of interventions and target populations represented in this analysis is a strength of this report, but it can also be a limitation because it may be difficult to apply the lessons learned to a specific intervention and/or student population. For example, the implementation challenges encountered by developers of a web application designed to provide direct intervention supports to high school students’ transition to post-secondary education are likely much different than the challenges faced by developers of an application to support infant-toddler service providers’ screening and intervention decision-making for young children with disabilities. The relatively small sample size and the heterogeneity of the target populations and content areas represented by these projects prevented us from exploring relationships between implementation barriers and facilitators, and target populations or content areas.

As described in the Research-to-Practice Dilemma theme, research personnel often provided implementation support that would not be available at the conclusion of the project, which can limit sustainability. However, research activities unrelated to technology integration might and did reportedly interfere with implementation, preventing exploration of implementation science under truly authentic conditions. These activities included obtaining consents or assents, conducting classroom observations, administering standardized and other assessments with students to measure intervention effects, collecting data from educators, etc. In addition, the prospect of being involved in a research project might also interfere with recruitment. While schools/districts are interested in using EBPs, few are willing to take on the added burden of engaging in research activities simply for the sake of advancing science. Financial burden was unlikely an implementation barrier, since all projects made their technology tools and interventions available at no charge to schools or districts and often provided financial incentives in an attempt to offset the burden of engaging in research activities. However, without data on the number of schools that had the opportunity to participate versus those that agreed and achieved full implementation, it is impossible to make valid claims about the effect of research activities and incentives on implementation of EBPs. Finally, the reliability of the implementation information reported for each project is unknown because it is self-reported by project PIs and it was provided in a narrative format rather than any observable data.

Future Research

This study provides a unique look at the challenges and facilitators for developing and evaluating implementation support for evidence-based technology interventions. Because these findings were primarily based on self-reported data and qualitative analyses, additional work is needed to validate the themes that emerged from the project reports. For example, using common measures of implementation based on student and educator data (e.g., collected using common metrics or technology logs) across projects would provide a more precise and reliable method of measuring implementation and identifying barriers.

Additionally, due to the limited timeframe of these projects, we are unable to report data on the sustainability of these technology interventions beyond funding timeline. This is a particularly important area of future research given the additional implementation supports, technical assistance, and financial incentives often provided through the projects. Although sustainability is a consideration in nearly all implementation frameworks, including the NIRN model (Fixsen et al., 2009), it is often difficult to measure without sustained research support. Factors reported to be critical to exploration, installation, initial and full implementation of the technology-based interventions in this study should be further explored during sustainability.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support forthe research, authorship, and/or publication of this article: This study was conducted in collaboration between several projects funded under U. S. Department of Education, Office of Special Education programs.

ORCID iDs

Anya S. Evmenova

Michelle Schladant

Terry Jackson

Author Biographies

Anya Evmenova is a Professor in the Division of Special Education and disAbility Research, College of Education and Human Development at George Mason University. Her research interests focus on the use of assistive and instructional technology for cognitive development and academic instruction for students with various abilities and needs, Universal Design for Learning, as well as effective online teaching and learning.

Kelley Regan is a Professor in the College of Education and Human Development at George Mason University. Her research has involved improving the knowledge and skills of preservice and inservice special education teachers. Additionally, Dr. Regan conducts applied intervention research to support the writing skills of students with disabilities.

Michelle Schladant is an assistant professor at the University of Miami. For the past 20 years, she has provided oversight to the center’s training initiatives and assistive technology program. She conducts research, teaches graduate students, and extensively presents at national conferences on assistive technology for early learning.

Tracey Hall conducts research on instructional interventions. Her work includes the design, development, and evaluation of instructional strategies and assessment tools. In this work, she implements the framework of UDL in, tools, collaborative partnerships, and professional presentations. A current focus involves application of UDL and effective teaching strategies for coaches.

Jay Bazhardt is an Associate Research Professor at the University of Kansas where his research interests focus on developing and testing technology solutions to support implementation of evidence-based practices by educators and parents, with an emphasis on data-driven intervention decision making in early childhood education.

Karen Erickson is the Director of the Center for Literacy and Disability Studies, a Professor in the Division of Speech and Hearing Sciences, and the Distinguished Professor in the Department of Allied Health Sciences in the School of Medicine at the University of North Carolina at Chapel Hill.

Jun Ai (Ph.D., BCBA-D) is an Assistant Professor of Unified Early Childhood Education at the University of Northern Iowa. She received her doctoral and postdoctoral training from the University of Kansas. Dr. Ai’s research focuses on the implementation and sustainability of early childhood positive behavior support.

Christina Sudduth has served for over a decade at the intersection of higher education and civic engagement, with a particular focus on child well-being. At the University of Miami, she was program manager for Step Up AT, promoting assistive technology strategies for the learning and development of children with disabilities.

Terry Jackson is a Project Officer at the U.S. Department of Education in the Office of Special Education Programs (OSEP). Dr. Jackson has worked at the Department for over 14 years and is the Lead for OSEPs Technology program that support grant projects that use current and emerging technologies to improve access to accessible educational materials and outcomes for students with disabilities.

References

Bertram

R. M.

Blase

K. A.

Fixsen

D. L.

(2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice, 25(4), 477-487. https://doi.org/10.1177/1049731514537687.

Blasé

Fixsen

(2013). Core intervention components: Identifying and operationalizing what makes programs work. Office of the assistant secretary for planning and evaluation. U.S. Department of Health and Human Services.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa.

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implementation Science, 8, 117. https://doi.org/10.1186/1748-5908-8-117.

Clarke

Dede

(2009). Design for scalability: A case study of the River City curriculum. Journal of Science Education and Technology, 18(4), 353-365. https://doi.org/10.1007/s10956-009-9156-4.

Coburn

C. E.

(2003). Rethinking scale: Moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3-12. https://doi.org/10.3102/0013189X032006003.

Cook

B. G.

Odom

S. L.

(2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79(2), 135-144. https://doi.org/10.1177/001440291307900201.

Creswell

J. S.

Plank Clark

V. L.

(2017). Designing and conducting mixed method research (3rd ed.). SAGE Publications.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. https://doi.org/10.1186/1748-5908-4-50.

10.

Dede

Honan

J. P.

Peters

L. C.

(2005). Scaling up success: Lessons learned from technology-based education improvement. Jossey-Bass.

11.

Durlak

J. A.

(2015). What everyone should know about implementation. In Durlak

J. A.

Domitrovich

C. E.

Weissberg

R. P.

Gullotta

T. P.

(Eds.), Handbook of social and emotional learning: Research and practice (pp. 395-405). New York: The Guilford Press.

12.

Eccles

M. P.

Mittman

B. S.

(2006). Welcome to implementation science. Implementation Science, 1(1), 1-3. https://doi.org/10.1186/1748-5908-1-1.

13.

Edyburn

D. L.

(2013). Critical issues in advancing the special education technology evidence base. Exceptional Children, 80(1), 7-24. https://doi.org/10.1177/001440291308000107.

14.

Every Student Succeeds Act of 2015 . (2015). Pub. L. No. 114-95 § 114 Stat. 1177.

15.

Feldstein

Glasgow

(2008). A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. The Joint Commission Journal on Quality and Patient Safety, 34(4), 228-243. https://doi.org/10.1016/s1553-7250(08)34030-6.

16.

Fixsen

Blase

Naoom

Duda

(2013). Implementation drivers: Assessing best practices. National Implementation Research Network (NIRN). https://doi.org/10.13140/RG.2.2.20415.30885.

17.

Fixsen

D. L.

Blase

K. A.

Naoom

S. F.

Wallace

(2009). Core implementation components. Research on Social Work Practice, 19(5), 531-540. https://doi.org/10.1177/1049731509335549.

18.

Gallup (2019). Education technology use in schools: Student and educator perspectivesNewSchools Venture Fund. https://www.newschools.org/wp-content/uploads/2020/03/Gallup-Ed-Tech-Use-in-Schools-2.pdf.

19.

Glasgow

R. E.

Vogt

T. M.

Boles

S. M.

(1999). Evaluating the public health impact of health promotion interventions: the RE-AIM framework. American Journal of Public Health, 89(9), 1322-1327. https://doi.org/10.2105/ajph.89.9.1322.

20.

Graham

I. D.

Logan

Harrison

M. B.

Straus

S. E.

Tetroe

Caswell

Robinson

(2006). Lost in knowledge translation: time for a map? Journal of Continuing Education in the Health Professions, 26(1), 13-24. https://doi.org/10.1002/chp.47.

21.

Grunwald & Associates (2010). Educators, technology, and 21st century skills: Dispelling five myths. Richard W: Walden UniversityRiley College of Education.

22.

Guest

MacQueen

K. M.

Namey

E. E.

(2012). Applied thematic analysis. SAGE Publications, Inc. https://dx-doi-org-s.web.bisu.edu.cn/10.4135/9781483384436.

23.

Harn

Parisi

Stoolmiller

(2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools. Exceptional Children, 79(3), 181-193. https://doi.org/10.1177/001440291307900204.

24.

Individuals with Disabilities Education Act . (2004). 20 U.S.C. § 1400.

25.

International Society for Technology in Education . (2021). https://www.iste.org.

26.

Israel

(2019). Using assistive and instructional technologies. In McLeskey

Maheady

Billingsley

Brownell

Lewis

(Eds.), High Leverage Practices for Inclusive Classrooms (pp. 264-278). Routledge Publishing.

27.

Kennedy

M. J.

Boyle

J. R.

(2017). The promise and problem with technology in special education. In Kauffman

J. M.

Hallahan

D. P.

Pullen

P. C.

(Eds.), Handbook of special education (2nd ed.), (pp. 606-614). Taylor & Francis. https://doi.org/10.4324/9781315517698-46.

28.

Knoster

(1991). Presentation at TASH Conference. Washington DC.

29.

Looi

C.-K.

Teh

L. W.

(2015a). Scaling educational innovations. Springer.

30.

Looi

C.-K.

Teh

L. W.

(2015b). Towards Critical Discussions of Scaling Up Educational Innovations. In Looi

C.-K.

Teh

L. W.

(Eds.), Scaling educational innovations (pp. 1-10). Springer. https://doi.org/10.1007/978-981-287-537-2_1.

31.

McLeskey

Barringer

M. D.

Billingsley

Brownell

Jackson

Kennedy

Winn

(2017). High-leverage practices in special education. Council for Exceptional Children & CEEDAR Center.

32.

Means

Penuel

W. R.

(2005). Research to support scaling up technology-based educational innovations. In Dede

Honan

J. P.

Peters

L. C.

(Eds.), Scaling up success: Lessons from technology-based educational improvement (pp. 176-197). Jossey-Bass.

33.

Ogden

Fixsen

D. L.

(2014). Implementation science: a brief overview and a look ahead. Zeitschrift für Psychologie, 222(1), 4-11. https://doi.org/10.1027/2151-2604/a000160.

34.

Ong

P. M.

(2020). COVID-19 and the digital divide in virtual learning. fall 2020. UCLA Center for Neighborhood Knowledge.

35.

Pang

E. S.

Lim

F. V.

Choe

K. C.

Peters

Chua

L. C.

(2015). System scaling in Singapore: The STELLAR story. In Looi

C.-K.

Teh

L. W.

(Eds.), Scaling educational innovations (pp. 105-122). Springer. https://doi.org/10.1007/978-981-287-537-2.

36.

Perlman

C. L.

Redding

(Eds.). (2011). Choosing and implementing technology wisely handbook on effective implementation of school improvement grants. Academic Development Institute. www.centerii.org/handbook.

37.

Rock

M. L.

Smith

Thomas

C. N.

Regan

Vasquez

III Kennedy

Bausch

(2017). Commentary: A response to an interview with joseph south by the teacher education and technology and media divisions of the council for exceptional children. Retrieved from http://www.citejournal.org/volume-17/issue-3-17/editorial/commentary-a-response-to-an-interview-with-joseph-smith-by-the-council-on-exceptional-children.()

38.

Vroom

E. B.

Massey

O. T.

Yampolskaya

Levin

B. L.

(2020). The Impact of Implementation Fidelity on Student Outcomes in the Life Skills Training Program. School Mental Health, 12(1), 113-123. https://doi.org/10.1007/s12310-019-09333-1.

39.

Wang

Hannafin

M. J.

(2005). Design-based research and technology-enhanced learning environments. Educational Technology Research & Development, 53(1), 5-23. https://doi.org/10.1007/bf02504682.

Stepping-Up Technology Implementation—How Does it Happen?

Abstract

Literature Review

Technology Implementation and Integration—Scaling-Up

Stepping-Up Technology Implementation Funding Program

Study Purpose and Research Questions

Method

Participating Projects

Qualitative Phase

Data Sources

Data Organization

Data Analysis and Credibility

Quantitative Phase: Data Collection and Analysis

Results

Qualitative Phase

Developing and Sustaining Buy-In

Ensuring Implementation Fidelity to Support the Intervention

Research-to-Practice Dilemma

Data Serving Multiple Purposes

Quantitative Phase

Discussion

Practical Implications

Limitations

Future Research

Footnotes

Declaration of conflicting interests

Funding

ORCID iDs

Author Biographies

References