Use of Self-Monitoring to Maintain Program Fidelity of Multi-Tiered Interventions

Abstract

Multi-tiered system of supports represents one of the most significant advancements in improving the outcomes of students for whom typical instruction is not effective. While many practices need to be in place to make multi-tiered systems of support effective, accurate implementation of evidence-based practices by individuals at all tiers is critical to obtain student outcomes. Effective strategies to achieve program fidelity are available; however, maintaining program fidelity at the individual level remains elusive. Lessons drawn from medicine indicate strategies to maintain program fidelity should address the implementer. Medical practitioners have used self-monitoring checklists to maintain fidelity with striking results. Research evaluating strategies to maintain program fidelity at the individual level represents an important next step in the field of education. Recommendations for a systematic research agenda focused on self-monitoring checklists are presented.

Keywords

academic achievement behavior evidence-based practice

A Significant Advancement: Multi-Tiered System of Supports

We believe that multi-tiered system of supports represents one of the most significant advancements in improving the outcomes of students for whom typical instruction is not effective. This widely used evidence-based model of schooling relies on data-based problem solving to integrate and deliver efficacious academic instruction and behavior supports in varying levels of intensity (multiple tiers) based on student need (Martella, Nelson, Marchand-Martella, & O’Rielly, 2012). The use of multi-tiered systems of supports is borrowed from the field of prevention science whereby primary, also known as universal prevention procedures (e.g., immunizations) are effective for approximately 80% of the population. Secondary prevention procedures (e.g., targeted education) are necessary for approximately 5% to 15% of the population that does not respond to primary prevention. Finally, tertiary procedures are needed for approximately 1% to 5% of the population that does not respond to primary or secondary procedures (e.g., direct care). The same multi-tiered system of supports model applied to schools suggests most students will achieve state- and district-defined outcomes based on core instruction (e.g., evidence-based curriculum) and behavior supports (e.g., school-wide positive behavior supports). Some students will need additional supplemental instruction (e.g., small group reading) and/or behavior supports (e.g., check-in-check out) in addition to that provided by the core to achieve these outcomes. Still a smaller number of students will need intensive instruction (e.g., 1:1 instruction) and/or behavior supports (e.g., functional behavioral assessment-guided behavioral support plans) to achieve successful outcomes.

We have chosen multi-tiered system of supports as a significant advancement because it provides the supportive context for promoting the integrative use of evidence-based academic and behavior-related practices to improve the outcomes of the full range of students who do not fully benefit from the typical instruction provided by schools. Prior to multi-tiered system of supports, few educators within schools were trained to use evidence-based academic and/or behavior practices in an integrative fashion to improve the outcomes of all students, including those who for whom typical instruction is not effective. Furthermore, many evidence-based practices (e.g., problem-solving method, curriculum-based general outcome measurement) were primarily used for students experiencing significant difficulties rather than the full range of students for whom typical instruction is not effective. Although there is certainly much more work needed to fully understand how to use multi-tiered system of supports more effectively, they have provided the supportive structure necessary for the wider use of evidence-based academic and behavior practices to improve the outcomes of all students, including those for whom typical instruction is not effective.

Central to the effective use of multi-tiered system of supports by schools is not only achieving initial high levels of program fidelity but also maintaining it over time. Schools implementing multi-tiered system of supports tend to focus on systems level factors (e.g., capacity-building) or pre-implementation factors (e.g., action-planning) to enhance the adoption of practices and maintaining program fidelity (Hagermoser Sanetti & Kratochwill, 2009; Han & Weiss, 2008). Although these factors are important, lessons drawn from medicine indicate that more central to the issue of maintenance of program fidelity are factors related to the implementer because maintaining program fidelity is ultimately based on these individuals. Indeed, program fidelity declines or is low within 1 to 10 days after teachers begin implementation (Hagermoser Sanetti & Kratochwill, 2009; Hagermoser Sanetti, Luiselli, & Handler, 2007; Noell, Witt, Gilbertson, Ranier, & Freeland, 1997; Witt, Noell, LaFleur, & Mortenson, 1997). Current efforts in the field of education have appeared to ignore the need for such practices and procedures (Han & Weiss, 2008). Thus, we believe research focused on maintenance of program fidelity at the individual level represents an important next step.

Next Step: Maintenance of Program Fidelity at the Individual Level

Importance

While many practices need to be in place to make multi-tiered systems of support effective (e.g., universal screening, progress monitoring), accurate implementation of evidence-based practices by individuals at all tiers is critical to obtain student outcomes (The Evidence-Based Intervention Work Group, 2005). Research demonstrates that when program or treatment fidelity is high, larger effects are obtained (Durlak & DuPre, 2008; Dusenbury, Brannigan, Falco, & Hansen, 2003; Gottfredson, Gottfredson, & Hybl, 1993; Telzrow, McNamara, & Hollinger, 2000) and lower levels of treatment fidelity make evidence-based practices less effective (Dusenbury et al., 2003). Lower accuracy of program fidelity can happen because the implementer only uses portions of the intervention, the quality of the implementation is low, or the intervention has been abandoned completely. Without support, high levels of program fidelity once achieved, tend to drop precipitously within a few days (Hagermoser Sanetti & Kratochwill, 2009; Hagermoser Sanetti et al., 2007; Noell et al., 1997; Witt et al., 1997). Regardless of the reasons for inadequate program fidelity, evidence-based practices are only as effective as the accuracy and quality with which they are implemented (DiGennaro, Martens, & Kleinmann, 2007). For this reason, it is necessary to target the implementer when developing strategies to maintain program fidelity to achieve student outcomes (Sanetti, Kratochwill, & Long, 2013).

Beyond the significance of program fidelity as it relates to student outcomes, program fidelity is also highly integral to the use of multi-tiered Response-to-Intervention (RTI) systems of support. School personnel using such a system of support base decisions on the success or failure of an intervention (Noell & Gansle, 2006). For a student to be identified as non-responsive to support, it is necessary for school personnel to ensure that the student not only has first been exposed to the evidence-based practice for an adequate length of time, but that the practice or intervention has been implemented with fidelity by the implementer (Noell & Gansle, 2006). Otherwise, school personnel cannot determine whether the student was not responsive to instruction or whether instruction was implemented poorly. In addition, often overlooked in multi-tiered RTI systems of support is the fidelity with which universal practices and programs have been implemented. When students fail to progress in the core instruction and behavior support they are referred to a problem-solving team to determine the appropriate intervention matched to student need (Gresham, 2002). The assumption is that universal practices and programs were implemented with fidelity. Achieving fidelity of universal programs and practices requires that a majority if not all individuals in the school implement them with fidelity. Therefore, achieving program fidelity at the individual level is necessary.

Fortunately, the technology to achieve initial high levels of program fidelity at the individual level is available, albeit typically underused by school personnel. Perhaps the most well-documented strategy to achieve high initial program fidelity is performance feedback, typically used within a coaching platform (Alvero, Bucklin, & Austin, 2001; Han & Weiss, 2008; Scheeler, Ruhl, & McAfee, 2004; Sheridan, Welch, & Orme, 1996). A systematic line of research examining program fidelity has been conducted using various forms of performance feedback to achieve fidelity of practices at the individual level. Research indicates performance feedback can be used successfully to achieve initial high levels of program fidelity (Codding, Livanis, Pace, & Vaca, 2008; Jones, Wickstrom, & Friman, 1997; Noell et al., 1997), a critical requisite for the efficacy of evidence-based practices (Durlak & DuPre, 2008; Dusenbury et al., 2003; Gottfredson et al., 1993; Telzrow et al., 2000). Unfortunately, as indicated earlier, implementation drift and degradation of fidelity occurs once performance feedback is withdrawn. Achieving initial high levels of fidelity at the individual level is not enough—maintaining fidelity is needed to sustain practices and student outcomes (Han & Weiss, 2008).

Little Research on Maintaining Individual-Level Program Fidelity

Much of the research examining maintaining fidelity within multi-tiered systems of supports to date has focused on systems level factors (e.g., capacity building) related to preparing the organization to support sustainability (Han & Weiss, 2008). For example, in the area of scaling-up of mental health practices in schools, Adelman and Taylor have developed a model of systems change and diffusion of innovations. Within the model, Adelman and Taylor articulate factors to enabling systems change such as (a) creating readiness for change, (b) developing the infrastructure, (c) creating organizational facilitators, and (d) establishing a change team (Adelman & Taylor, 2007). Similarly, the work of the National Implementation Resource Network focuses on “big picture” factors such as readiness, stages of implementation, and implementation drivers (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). Although the complexities of systems change and need for models described above cannot be denied, more proximal to the issue of maintaining fidelity is the individual level.

As noted above, research on program maintenance at the individual level tends to focus on achieving initial high levels of fidelity with performance feedback (Han & Weiss, 2008). The theory of maintenance behind this research posits that once the individual implements with high fidelity and observes positive outcomes, maintenance will occur. Although the belief that what one is doing is producing positive effects may be an important factor to maintenance, this notion seems a bit optimistic. Some researchers have begun examining the individual as the mediator of the intervention and maintenance of fidelity (i.e., Sanetti et al., 2013). Within this research, maintenance of program fidelity is addressed by using strategies to increase implementation intention (i.e., intention to implement and maintain an intervention) and sustainability self-efficacy (i.e., belief that one can sustain implementation). Again, little research has been conducted addressing the specific need for strategies to maintain fidelity at the individual level.

A small study using a self-monitoring checklist to maintain fidelity of the Good Behavior Game (Barrish, Saunders, & Wolf, 1969) indicates that the checklist was effective with four teachers for maintaining treatment fidelity after training and performance feedback ended (Oliver, Wehby, & Nelson, 2014). The checklist contained discrete steps for implementing the game rather than more complex items as is more typical of school-based practices. These results suggest that self-monitoring maintenance checklists may be a practical and economical approach to maintain evidence-based practices in schools. However, more research is required to evaluate the use of a self-monitoring checklist for more complex practices and larger numbers of teachers.

Lessons From Medicine to Guide Research on Individual-Level Program Fidelity

The field of medicine can provide a viable model to guide research on maintenance of program fidelity at the individual level in education. The field of medicine recognized the need for a focus on identifying procedures for maintaining fidelity at the individual level (World Health Organization, 2009). According to the World Health Organization (2009), limited treatment fidelity at the individual level (i.e., human error) accounts for at least 50% of post-surgery complications. Given the high stakes consequences for lack of program fidelity, it is not surprising that researchers in the field of medicine have been examining strategies to maintain program fidelity at the individual level for more than 25 years (World Health Organization, 2009).

Researchers have developed and tested the use of self-monitoring checklists to maintain fidelity at the individual level with striking results—47% reduction in patient deaths and 36% reduction in inpatient complications (Haynes et al., 2009; Zamir, Beresova-Creese, & Miln, 2012). In addition, patients use self-monitoring checklists to maintain their use of treatment protocols (Coster, Gulliford, Seed, Powriet, & Swaminathan, 2008; Mahoney & Ellison, 2007). Moreover, researchers have also used self-monitoring checklists with nursing assistants to maintain 80% fidelity of job skill performance over time (Stevens et al., 1998). Over 25 years of empirical support in the field of medicine is instructive to education—self-monitoring maintenance checklists may be a practical and economical approach to maintain evidence-based practices in schools.

The positive effects of self-monitoring checklists are not surprising given that their development and use is based on the theory of self-regulation (Fox & Riconscente, 2008). Briefly, the theory of self-regulation suggests that individuals engage in three functions as part of self-regulation: (a) self-observation, (b) self-judgment, and (c) self-reaction (Bandura, 1991). The self-monitoring checklist acts as a prompt to track behavior against a standard (i.e., self-observation) and begins a process of comparison of current performance to internal and external standards (i.e., self-judgment). During self-judgment, a discrepancy reduction mechanism functions to reduce the discrepancy between current performance and desired performance (Bandura, 1991). Goals are set internally, and the self-reactive influences are deployed. These processes lead to self-directed behavior change and behavior improvement or maintenance. The self-regulation process acts as a mechanism to form the habit of high fidelity of implementation, which maintains behaviors over time (Fox & Riconscente, 2008).

Research Needed on Self-Monitoring Checklists in Education

A research agenda on self-monitoring checklists in education will have to address several key things. For example, procedures and processes are needed to identify and specify fidelity criteria (Mowbray, Holter, Teague, & Bybee, 2003). Fidelity can be measured based on the structure or what is implemented as well as the process or way in which the program is implemented. Indicators or critical components of the evidence-based practice or program including anchor point for quality ratings need to be specific, objective, and measurable. Fidelity criteria can be developed from (a) published program materials or observations of programs that have proven successful; (b) expert opinions or literature reviews; or (c) qualitative research through opinions of users (Mowbray et al., 2003). Decisions regarding which method to use in developing fidelity criteria should be based on the availability of published program materials, credible experts, and the validity of user opinions (e.g., experience vs. inexperienced users).

Second, a valid index is needed to determine fidelity scores falling in the range of acceptable versus unacceptable and what adaptations, if any, implementers can make. This is accomplished through collecting measures of fidelity and then quantifying fidelity by (a) asking experts to rate fidelity based on permanent products, observations, interviews, or videotaped recordings; and (b) surveying implementers or those receiving the intervention or program (Mowbray et al., 2003). Consideration should be given to the number of items and anchors used in self-monitoring checklists to ensure the feasibility of completion by users. Some fidelity measures can be onerous to complete and therefore are not feasible for use as a self-monitoring checklist.

Third, research is needed to examine the psychometric characteristics of fidelity checklists (i.e., reliability, internal consistency, validity). Reliability across respondents and inter-rater agreement should be calculated either by kappa coefficients, intra-class correlations, percentage agreement, or Pearson correlations (Mowbray et al., 2003). Internal consistency should be calculated through methods such as cluster analysis, confirmatory factor analysis, or Cronbach’s coefficient alpha (Mowbray et al., 2003). Finally, convergent validity could be developed by measuring fidelity using a self-monitoring checklist compared with direct observations of implementation within the delivery context (Mowbray et al., 2003). Psychometrically sound self-monitoring checklists are necessary to establish that items being measured are meaningful and correlated to program outcomes.

In addition, research is needed on the use of self-monitoring checklists with the range of programs and evidence-based practices from the simple to more complex as well as those that are dynamic and change over time. Research is also needed to test self-monitoring checklists among various implementers and across implementation contexts or schools. Finally, research is needed on the pragmatic issues of self-monitoring checklists in education. For example, (a) determining the frequency with which the checklist must be completed to maintain fidelity, (b) identifying the number of items needed to measure fidelity while still being feasible for the implementer to complete, and (c) examining efficient methods for checklist completion.

Final Thoughts

Although we believe multi-tiered systems of supports is a significant advancement in the field of education to improve student outcomes, the critical next step needed is to develop procedures to maintain program fidelity of evidence-based practices. Unprecedented resources have been allocated for professional development of evidence-based practices and other school reform efforts. For example, the federal government has invested US$4.35 billion in federal funds for the Race to the Top Program to improve education and student outcomes (U.S. Department of Education, 2009). Even without the use of federal dollars, some estimate districts spend on average 3.6% of the districts’ annual budgets for professional development activities (Miles, Odden, Fermanich, Archibald, & Gallagher, 2004). The staggering amount of resources for professional development efforts to implement evidence-based practices and policies that likely will not be maintained is a significant financial issue to be addressed by schools nationwide. This will require a sustained and focused line of research on approaches and procedures for maintaining program fidelity at the individual level. We propose the use of self-monitoring checklists to maintain program fidelity as a viable approach necessary in education. Our charge to researchers is to adopt the suggestions provided herein and to systematically evaluate procedures to maintain program fidelity at the individual level. Once we in the field of education have learned to master maintenance of effective practices, the cycle of identification and implementation of “new” practices can be ended and more time and resources can be allocated to improving education for all students.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Adelman

H. S.

Taylor

(2007). Systemic change for school improvement. Journal of Educational and Psychological Consultation, 17, 55–77. doi:10.1080/10474410709336590

Alvero

Bucklin

Austin

(2001). An objective review of the effectiveness and essential characteristics of performance feedback in organizational settings. Journal of Organizational Behavior Management, 21, 3–29. doi:10.1300/J075v21n01_02

Bandura

(1991). Social cognitive theory of self-regulation. Organizational Behavior and Human Decision Processes, 50, 248–287. doi:10.1016/0749-5978(91)90022-L

Barrish

Saunders

Wolf

(1969). Good behavior game: Effects of individual contingencies for group consequences on disruptive behavior in a classroom. Journal of Applied Behavior Analysis, 2, 119–124. doi:10.1901/jaba.1969.2-119

Codding

Livanis

Pace

Vaca

(2008). Using performance feedback to improve treatment integrity of classwide behavior plans: An investigation of observer reactivity. Journal of Applied Behavior Analysis, 41, 417–422. doi:10.1901/jaba.2008.41-417

Coster

Gulliford

Seed

Powriet

Swaminathan

(2008). Self-monitoring in type 2 diabetes mellitus: A meta-analysis. Diabetic Medicine, 17, 755–761. doi:10.1046/j.1464-5491.2000.00390.x

DiGennaro

F. D.

Martens

B. K.

Kleinmann

A. E.

(2007). A comparison of performance feedback procedures on teachers’ treatment implementation integrity and students’ inappropriate behavior in special education classrooms. Journal of Applied Behavior Analysis, 40, 447–461. doi:10.1901/jaba.2007.40-447

Durlak

DuPre

(2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0

Dusenbury

Brannigan

Falco

Hansen

(2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237–256. doi:10.1093/her/18.2.237

10.

Evidence-Based Intervention Work Group. (2005). Theories of change and adoption of innovations: The evolving evidence-based intervention and practice movement in school psychology. Psychology in the School, 42, 475–494. doi:10.1002/pits.20086

11.

Fixsen

D. L.

Naoom

S. F.

Blase

K. A.

Friedman

R. M.

Wallace

(2005). Implementation research: A synthesis of the literature (FMHI Publication #231). Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, National Implementation Research Network.

12.

Fox

Riconscente

(2008). Metacognition and self-regulation in James, Piaget, and Vygotsky. Educational Psychology Review, 20, 373–389. doi:10.1007/s10648-008-9079-2

13.

Gottfredson

Hybl

(1993). Managing adolescent behavior: A multi-year, multi-school study. American Educational Research Journal, 30, 179–215. doi:10.3102/00028312030001179

14.

Gresham

F. M.

(2002). Responsiveness to intervention: An alternative approach to the identification of learning disabilities. In Bradley

Danielson

Hallahan

D. P.

(Eds.), Identification of learning disabilities: Research to practice (pp. 467–519). Mahwah, NJ: Lawrence Erlbaum.

15.

Hagermoser Sanetti

Kratochwill

(2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38, 445–459.

16.

Hagermoser Sanetti

Luiselli

Handler

(2007). Effects of verbal and graphic performance feedback on behavior support plan implementation in a public elementary school. Behavior Modification, 31, 454–465. doi:10.1177/0145445506297583

17.

Han

Weiss

(2008). Maintaining program fidelity after the thrill—and external support—is gone. In Evans

Weist

Serpell

(Eds.), Advances in school-based mental health interventions: Best practices and program models (Vol. 2, pp. 1–17). Kingston, NJ: Civic Research Institute.

18.

Haynes

A. B.

Weiser

T. G.

Berry

Lipsitz

Breizat

Dellinger

. . . Gawande

(2009). A Surgical Safety Checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360, 1356–1370. doi:10.1056/NEJMsa0810119

19.

Jones

Wickstrom

Friman

(1997). The effects of observational feedback on treatment integrity in school-based behavioral consultation. School Psychology Quarterly, 12, 316–326.

20.

Mahoney

Ellison

(2007). Assessing glucose monitor performance—A standardized approach. Diabetes Technology & Therapeutics, 9, 545–552. doi:10.1089/dia.2007.0245

21.

Martella

R. C.

Nelson

J. R.

Marchand-Martella

N. E.

O’Rielly

(2012). Comprehensive behavior management: Individualized, classroom, and school-wide approaches. Newbury Park, CA: SAGE.

22.

Miles

Odden

Fermanich

Archibald

Gallagher

(2004). Inside the black box of school district spending on professional development: Lessons from comparing five urban districts. Journal of Education Finance, 30(1), 1–26.

23.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Developmental, measurement, and validation. American Journal of Evaluation, 24, 315–340. doi:10.1177/109821400302400303

24.

Noell

Gansle

(2006). Assuring the form has substance: Treatment plan implementation as the foundation of assessing response to intervention. Assessment for Effective Intervention, 32, 32–39. doi:10.1177/15345084060320010501

25.

Noell

Witt

Gilbertson

Ranier

Freeland

(1997). Increasing teacher intervention implementation in general education settings through consultation and performance feedback. School Psychology Quarterly, 12, 77–88.

26.

Oliver

R. M.

Wehby

J. H.

Nelson

J. R.

(2014). Use of a self-monitoring checklist to maintain treatment fidelity: A pilot study. Manuscript submitted for publication.

27.

Sanetti

L. M. H.

Kratochwill

T. R.

Long

A. C. J.

(2013). Applying adult behavior change theory to support mediator-based intervention implementation. School Psychology Quarterly, 28, 47–62.

28.

Scheeler

Ruhl

McAfee

(2004). Providing performance feedback to teachers: A review. Teacher Education and Special Education, 27, 396–407. doi:10.1177/088840640402700407

29.

Sheridan

S. M.

Welch

Orme

S. F.

(1996). Is consultation effective? A review of outcome research. Remedial and Special Education, 17, 341–354. doi:10.1177/074193259601700605

30.

Stevens

A. B.

Burgio

L. D.

Bailey

Burgio

K. L.

Paul

Capilouto

. . . Hale

(1998). Teaching and maintaining behavior management skills with nursing assistants in a nursing home. The Gerontologist, 38, 379–384. doi:10.1093/geront/38.3.379

31.

Telzrow

McNamara

Hollinger

(2000). Fidelity of problem-solving implementation and relationships to student performance. School Psychology Review, 29, 443–461.

32.

U.S. Department of Education. (2009). Race to the Top: Setting the pace for gains across the education system [White House Report]. Retrieved from http://www.whitehouse.gov/sites/default/files/docs/settingthepacerttreport_3-2414_b.pdf

33.

Witt

Noell

LaFleur

Mortenson

(1997). Teacher use of interventions in general education settings: Measurement and analysis of the independent variable. Journal of Applied Behavior Analysis, 30, 693–696. doi:10.1901/jaba.1997.30-693

34.

World Health Organization. (2009). World health statistics. Retrieved from http://www.who.int/gho/publications/world_health_statistics/2009/en/

35.

Zamir

Beresova-Creese

Miln

(2012). Intraocular lens confusions: A preventable “never event”—The Royal Victorian Eye and Ear Hospital protocol. Survey of Ophthalmology, 57, 430–447. doi:10.1016/j.survophthal.2011.12.003