Reducing Low-Value Practices a Functional-Contextual Consideration to Aid in De-Implementation Efforts

Abstract

Through innovation in research and self-correction, it is inevitable that some practices will be replaced or be discredited for one reason or another. De-implementation of discredited and low-value practices is a necessary step for school psychologists’ maintenance of evidence-based practices and to reduce unnecessary costs and risk. However, efforts to clarify de-implementation frameworks and strategies are ongoing. The scope of this paper follows McKay et al. in considering the potential for de-implementation strategies to be informed by applied behavior analysis and operant learning theory. We conceptualize low-value practice as sets of behaviors evoked by their context and maintained by their consequences, and thus de-implementation as behavior reduction. We discuss the need for future research given this perspective.

Keywords

de-implementation implementation science evidence-based practices applied behavior analysis functional-contextual

Introduction

Tasked with solving big problems for students and educators (Shapiro, 2006), the field of school psychology has embraced empirically supported assessment and treatment (generally, evidence-based practice [EBP]) as necessary for ethical and efficacious practice (Forman et al., 2012; Kratochwill & Shernoff, 2004; National Association of School Psychologists [NASP], 2020; Sanetti et al., 2019). In recent years, implementation science researchers have explored various strategies and frameworks for increasing the use and fidelity of EBP (Sanetti et al., 2019). Despite the field’s affirmation of EBP, there is evidence that practitioners working with children and adolescents continue to use unproven and discredited practices (low-value practices; Koocher et al., 2015). The overlapping fields of psychology, education, and special education have their share of low-value practices such as abstinence only education (see Stanger-Hall & Hall, 2011); facilitated communication (see Jacobson et al., 1995); whole language reading (see Stahl et al., 1994); zero tolerance policies and suspension (see Ryan & Peterson, 2004; Skiba & Rausch, 2006); cognitive profile analysis (see McGill et al., 2018); the use of projective testing (see Lilienfeld et al., 2000); and so on. School psychologists are not immune from using low-value practices (Lilienfeld et al., 2012) and there’s reason to believe that ongoing use of such practices may interfere with implementation efforts for competing strategies. For instance, a low-value practice may compete for resources with EBP or even lead to contradiction and (potentially) decreased efficacy of the desired practice. Consistent with this notion, Chambers et al. (2013) found that the removal of a practice may expedite adoption of a replacement practice. Thus, for implementation science efforts to be maximally effective, there is a need to abandon (or decrease) use of low-value practices. The process for abandoning (or decreasing) the use of a clinical practice is generally referred to as de-implementation, though other terms such as de-adoption are common in the literature (McKay et al., 2018; Niven et al., 2015).

De-Implementation

De-implementation can be conceptualized either as a process or an outcome (e.g., McKay et al., 2018), though it is most readily thought of as a process, similar to implementation, in which low-value practices are reduced or eliminated from standard practice (Johns et al., 2016; McKay et al., 2017). The first step in de-implementation is to identify a low-value practice to be reduced or eliminated. Low-value practices are those practices (a) that have not been shown to be effective and efficacious, (b) that are less effective or efficacious than another available practice, (c) that cause harm, or (d) that are no longer necessary (McKay et al., 2018; Niven et al., 2015; Prasad & Ioannidis, 2014). In school psychology, we might consider interpreting cognitive profiles to be a low-value practice as profile based techniques have not been shown to lead to improved diagnostic decision-making or treatment selection (for review, see McGill et al., 2018); that is, this interpretive practice lacks evidence demonstrating its effectiveness and efficacy (a). Another key practice related to school psychology is restraints to prevent low-risk behaviors (e.g., elopement), which lacks adequate evidence (a), has clear alternatives (e.g., positive behavioral intervention supports; b), and may cause harm (Ryan & Peterson, 2004; Westling et al., 2010; c).

Once a low-value practice has been identified, the context is evaluated for variables that may affect the de-implementation process. While de-implementation mirrors implementation frameworks (McKay et al., 2018; Patey et al., 2018), there is reason to believe that the underlying processes at each step may function differently than implementation in a number of important ways (Montini & Graham, 2015; Patey et al., 2018). For instance, economic, historical, professional, political, and social factors may delay or curtail de-implementation efforts (Bodegom-Vos et al., 2017; Johns et al., 2016). Practice factors may also play a role, such as the age, size, and function(s) of the practice may increase the difficulty of de-implementation (McKay et al., 2018). A thorough assessment of these variables will directly impact the de-implementation strategy selection and, as we will discuss at length, a functional-contextual lens may be useful in identifying de-implementation strategies at the individual and group level.

Following assessment of the context, active de-implementation entails the use of strategies that should reduce or eliminate the use of the selected practice. Of note, the de-implementation strategies used during this stage may occur at a variety of levels (e.g., individual, system, policy). Common strategies include policy change at the organizational level, such as the trend in school-based policy that suggests restraint should be a last-resort procedure (Freeman & Sugai, 2013), or ‘unlearning’ (Rushmer & Davies, 2004) at the individual level. Unlearning often includes didactic information about the evidence for an intervention and may also include guidance toward an alternative practice. In school psychology, continuing education on the problems associated with the discrepancy model for specific learning disability in consort with legislation and training efforts to guide practitioners to alternative strategies (e.g., response to intervention) is a prime example of how these two types of strategies may be combined to impact practice use.

Finally, an evaluation of the effects of the de-implementation process is completed, including the extent to which the identified practice has been abandoned (McKay et al., 2018; Niven et al., 2015). It may be helpful for policy makers and administrators seeking to abandon a practice to clearly identify what would be an acceptable goal for their efforts. For instance, rather than seek to eliminate restraint procedures entirely, many policy makers have sought to reduce their use overall and to limit their use to situations where a child or staff member could be injured (Freeman & Sugai, 2013).

The assessment and active de-implementation components of the de-implementation process draw on research from the social sciences to affect the behavior of individuals. Patey et al. (2018) completed a conceptual review of published papers and identified a number of potential theories that may be useful to consider in implementation and de-implementation. However, given findings that suggest the de-implementation process is unique from implementation (McKay et al., 2018; Montini & Graham, 2015; Patey et al., 2018), it follows that a theory capable of addressing differences between how a practice is introduced versus how a practice is reduced or eliminated would be parsimonious. Only one theory identified by Patey et al. (2018), operant learning theory (OLT), posits different strategies are necessary to decrease behavior rather than to increase behavior. However, we extend this perspective to also include rule-governed behavior (O’Hora et al., 2014; Skinner, 1966). More broadly, the scientific field of applied behavior analysis (ABA), which has OLT at its roots, has much to offer the development of de-implementation. This manuscript will apply a functional-contextual lens, rooted in OLT, to the assessment of the maintaining variables of low-value practices, and suggest how such an analysis can lead to selection of de-implementation strategies informed by operant learning theory, rule-governed behavior and so forth.

Applied Behavior Analysis: A Brief Overview

Applied behavior analysis¹ is the application of the laws that govern the interaction between an individual and their environment. These laws emphasize that behaviors are shaped by their consequences and can be evoked by events that precede them. This functional-contextual lens emphasizes the functional relationship between (a) historical and immediate context and (b) future behavior; specifically, the focus is on how context functions to shape, evoke, and maintain the behavior of individuals (see Gifford & Hayes, 1999; see Skinner, 1974). These interactions are often described within the conceptualization of the three-term contingency between antecedents, behaviors, and consequences (A-B-Cs). Antecedents and Consequences (i.e., the context) interact with behavior in many different ways which results in either increasing or decreasing the likelihood of that behavior. Consequences are considered reinforcing when they increase the probability of a behavior, and are considered punishing when they reduce the probability of a behavior. Additionally, antecedents can become associated with the consequences and serve to prompt behavior when associated with a reinforcer and suppress behavior when associated with a punisher.

The science of behavior analysis employs these principles to increase or decrease behavior as desired by influencing the antecedents and consequences associated with the behavior of interest. Using these principles, the likelihood of a behavior can be increased by delivering or increasing reinforcement, increasing the salience of an associated antecedent, or reducing the effort required to engage in the behavior. Conversely, the likelihood of a behavior can be decreased by eliminating or reducing the reinforcer, reducing or eliminating the salience of the antecedent, or increasing the effort required to engage in the behavior. In addition to these direct approaches to altering behavior, indirect approaches can also increase or decrease the occurrence of a target behavior. These indirect methods consist of differentially reinforcing non-target behaviors (i.e., reinforcing behaviors other than the behavior to be reduced; e.g., Vollmer & Iwata, 1992) and altering the effort required to engage in a behavior (for review, see Friman & Poling, 1995). Differentially reinforcing non-target behaviors can serve to increase the likelihood of other behaviors and thus may interfere with the occurrence of the target behavior due to an individual’s limited behavioral capacity. There are several differential reinforcement procedures, but for each one, reinforcement is delivered contingent upon the occurrence of a competing behavior or behaviors. Alternatively, if the effort required to engage in a target behavior is increased, the probability of the behavior will decrease. Conversely, if the effort required to engage in a target behavior is decreased, the probability of the behavior will increase. Taken together these approaches to altering the probability of a target behavior provide powerful tools capable of being employed by practitioners on a number of behaviors.

Low-Value Practice as a Functional Behavior

De-implementation from a functional-contextual perspective conceptualizes low-value practices as behaviors (or sets of behaviors) to be reduced or eliminated through the application of behavioral principles. A substantial portion of psychological and educational practice, low-value or otherwise, is rule-governed behavior. A rule is a stimulus or set of stimuli that specify a reinforcement contingency, thus permitting a verbal organism to engage in behavior without contacting direct consequences (Skinner, 1966). For instance, the rule “eat your vegetables and you will get ice cream” specifies the target behavior and a reinforcement contingency; as such, the recipient of this rule need not guess at which behavior will earn them ice cream. We might also conceptualize the rule as specifying a derived relation between some of the key words in the rule (e.g., “ice cream”) with stimuli in the real world (e.g., an ice cream cone), effectively reducing the need for environmental consequences by positioning verbal stimuli and environmental stimuli as equivalent (Harte et al., 2020; O’Hora et al., 2014). This is especially important given that many consequences of psychological and educational practice lack consequences in the immediate environment. Let us take the example of psychoeducational testing. Despite our best efforts as a field, there is no verification of a student’s classification beyond how well assessment data map onto classification elements from state, federal, and clinical guidelines, and as such there is no verification of classification available in the immediate environment. It follows that assessment practices and decision criteria are sets of behaviors partially maintained by rules rather than consequences in the immediate environment.² Furthermore, rule-governed behavior can be insensitive to environmental contingencies (see Harte et al., 2020 for review). Taken together—that rule-governed behaviors are less sensitive to consequences and that consequences may be absent—we begin to understand how and why low-value psychological practices are slow to change and help to clarify how low-value practices often become entrenched from one generation to the next (Meehl, 1978).

Given that low-value practices are often entrenched (e.g., Meehl, 1978; Montini & Graham, 2015; VanDerHeyden, 2018), selecting de-implementation strategies should be informed by a systems-level functional-contextual analysis (cf. implementation science; Forman, 2019) of the variables maintaining those practices. Montini and Graham (2015) discuss how historical, economic, political, and social factors may contribute to de-implementation resistance. How these variables manifest and influence resistance may vary based on the practice to be de-implemented, the target setting, and the professionals within that system. For instance, cognitive profiles from IQ tests have no clear utility or incremental validity (see McGill et al., 2018). However, interpretation of cognitive profiles is multiply maintained by a combination of historical (i.e., a long history of individual and professional use), economic (e.g., vested interest of proponents, test companies, and practitioners), and socio-political (e.g., efforts to address IQ test misuse are seen by some as an attack on school psychology itself; distrust of discrepancy models for the identification of specific learning disabilities created a vacuum for an alternative interpretation approach) variables. However, this perspective is likely too simple and too broad (at the national level); specific de-implementation efforts will need to consider political factors, such as state and local guidelines, the influence of proponents within the region, and the position of administrators and practitioners within the system to truly understand de-implementation resistance. In redesigning system contingencies to directly reduce low-value practice behaviors among practitioners, considering what practitioners get or get away from by engaging in the behavior is important. For instance, it could be hypothesized that engaging in cognitive profile analysis gets social approval and leads to task completion. We will consider several behavior reduction strategies in the context of de-implementation.

De-Implementation as Behavior Reduction Strategies

Several behavior reduction strategies such as extinction, differential reinforcement of alternative or incompatible behaviors, change in response effort, and punishment have been used in various clinical and applied contexts to reduce behavior (e.g., Cooper et al., 2020; Martin & Pear, 2019) and have potential as de-implementation strategies. Adding these behavior reduction strategies to de-implementation efforts provides two unique advantages. One, it increases the overall number of strategies available for de-implementation efforts that can offer more flexibility and greater breadth of application. Second, behavioral reduction efforts provide solutions for situations where an increase in implementation of evidence-based practices is not accompanied by decrease of low-value practices. The socio-political effects of the discrepancy model debate in school psychology is a good example of such a scenario, where room for science-based alternatives was made in the form of a response to intervention’ (RTI) framework for eligibility. Unfortunately, despite the increase in RTI implementation, merely allowing RTI through law, policy and technical support has not greatly reduced or eliminated the discrepancy model or alternative cognitive-based approaches, which still remains the dominant model used in schools and among school psychologists. As a result, behavioral reduction strategies may play a key role in helping de-implementation efforts by directly targeting the behaviors associated with low-value practices.

However, using positive strategies to encourage implementation of evidence-based practices is also strongly encouraged. Aligned with the functional-contextual approach for behavior change that OLT embodies, behavioral reduction strategies should rarely if ever be used without attempts to increase functional alternatives. That is, behavior reduction strategies used in isolation are unlikely to have long-term sustained reduction unless viable evidence-based alternatives are available or the variables that maintain the low-value practices are completely removed (i.e., the issue has dissipated (McKay et al., 2018); moreover, de-implementation efforts in isolation targeting a specific practice may lead to other low-value practices once the original one has been reduced. With this caution in place, we will review some of the key behavior reduction strategies which may be useful in de-implementation efforts.

Brief Review of Behavior Reduction Strategies

Extinction

One of the earliest strategies for decreasing behavior used was extinction, which involves the removal of reinforcement for a previously reinforced behavior. This often involves a careful analysis of what is currently the reinforcement contingency in place, followed by blocking or fully removing reinforcement in order to reduce the behavior. Extinction is a well-documented and very effective method for behavior reduction (Martin & Pear, 2019). For de-implementation, extinction can be considered a strategy for reducing low-value practices where the source of reinforcement of the practices can be clearly identified and altered. For example, a low value or discredited practice may be reinforced and maintained by profit incentives, independent of evidence of positive clinical outcomes, and as a result, removal of the profit incentives can lead to decrease in the practice itself. For example, in health care settings where insurance policies can stipulate that only practices with sufficient evidence of their efficacy will be reimbursed, they are successfully able to limit what practices are used. However, extinction procedures can be difficult to manage as a lot of reinforcement is not always under direct control or can be inadvertently reinforced despite our best efforts. In addition, for the school context, identifying a clear source of reinforcement equivalent to profits in health care can be difficult as school professionals are not subject to similar profit driven mechanisms. Moreover, extinction alone as a procedure has some limitations and challenges. One of the most well-known challenges is the extinction burst, which is an initial spike or increase in the behavior targeted for reduction. Extinction bursts can be particularly challenging for situations where any increase, even temporally, cannot be tolerated within the environment. Despite these challenges, extinction might be in limited cases, a strategy worth consideration for de-implementation. In the context of policy and federal funding, it could be used in parallel to funding in healthcare settings where some sources of funding is not available for schools looking to use a set of identified low-value or discredited practices.

Differential Reinforcement

A closely related strategy to extinction is differential reinforcement of alternative or incompatible behavior (DRA/DRI), where reinforcement is removed (extinction) but is now specifically targeting a behavior that is an alternative to or incompatible with the behavior that is targeted for reduction. As a result, engaging in these alternative/incompatible behaviors requires not engaging in the behavior being targeted for reduction. In classrooms for example, teachers may target specific actions such as “raise your hand wait quietly” to be called on while simultaneously not reinforcing any verbal behavior such as calling out or raised hands while talking. As a result, overtime the calling out behavior is likely to be reduced while the behavior of raising your hand and waiting quietly is likely to increase. For low-value practices, DRA/DRI strategies would require identifying what practices we want to remove reinforcement for and what are alternative or incompatible (evidence-based) practices that we can simultaneously reinforce. An example mentioned earlier of restraints as low-value practices could be a good target for DRA/DRI, as there are clearly evidence-based alternatives (PBIS, Functional Behavioral Assessments and Behavioral Support Plans) that are often incompatible with any type of physical contact or force applied toward the child with restraints. Similar to extinction procedures, using DRA/DRI effectively involves the challenge of having control over reinforcement contingencies with the addition of also making sure we have the ability to shift reinforcement to the preferred practices (i.e., at the policy level, funding evidence-based practices and not low-value practices simultaneously).

Response Effort

Another strategy to reduce behaviors is based on managing the amount of effort it takes to engage in the behavior to access reinforcement, known as response effort. By increasing response effort, even powerful reinforcers can fail to maintain the target behaviors and as a result, provide another effective strategy for de-implementation. For example, for dietary health programs in schools, this could involve increasing the cost of unhealthy food products to increase the amount of overall effort required to purchase the product, making it less likely the students will purchase and consume the product. For de-implementation of low-value practices in schools, this could increase effort for low-value or unproven strategies by requiring more intensive documentation, prior approval justifying the use of the low-value practice, and on-going data collection procedures to evaluate the practice. As all of these activities are likely to considerably increase the effort required to use the low-value practice, the extra effort is also likely to lead to decrease in the overall use of low-value practices. Like the preivous stratetgies, challenges inherent with using response effort to discourage low-value practices are related to the extent that we can have control over factors that will increase or decrease response efforts, either at the school or policy level.

Punishment

In addition to reinforcement focused approaches to behavior reduction strategies, punishment-based strategies can also be used to target reduction in use of low-value practices. To clarify, we use the technical definition of punishment found in OLT as any addition or removal of a stimulus that leads to a future decrease in the probability of a behavior. This does not necessarily include aversive or otherwise cruel or demeaning procedures. Like reinforcement, punishment can also be either positive, where something is added, or negative, where something is removed, that leads to a decrease in probability of future behavior. A school-based example may include removal (negative punishment) of access to downtime during homeroom if there is unfinished work (e.g., homework, in-school work). Like reinforcement procedures, the effects the contingencies have on behavior is what will determine whether punishment was in place. In our example above, if we see a reduction of incomplete work due to loss of access to free time, we can be confident that a punishment contingency was in place. For positive punishment, the most common and everyday example schools could include the introduction of disciplinary actions for rule violations if the behavior decreases as a result of the disciplinary action.

For low-value practices, both versions of punishment can be used and have been in other contexts where the intent is to discourage harmful practices. For example, positive punishment can and has been used by adding sanction for those using discredited strategies such as distributing contaminated food products or selling products with defects that impact safety of the users. This could be sanctions deployed by government agencies or through legal action against the individuals promoting the low-value practices. Similarly, for negative punishment, examples may include removing sponsorship or other support from agencies or individual practitioner’s contingent on their use of identified discredited, harmful or low-value practices. For schools, this could involve broad law and policies from governing agencies such as the department of education that sanction schools or professionals for use of low-value practices.

Punishment like the other strategies has some limitations and contraindications that are worth consideration. Like reinforcement, they do require a clear identification of what will serve as a punisher, which depending on the situation, may be hard to identify a priori. More salient concerns include well known side effects of punishment procedures, which range from emotional backlash of those targeted as part of punishment contingencies as well as efforts to avoid or escape punishment contingencies in place, leading to cheating or avoiding surveillance efforts. Given these limitations, punishment procedures, even when not considering any type of aversives, should be used with extreme caution and alongside positive efforts to build and support implementation of evidence-based practices. However, punishment procedures may also provide a unique and sometimes only avenue for change. For example, it might be worth considering punishment in situations where the practice is not only of low value (i.e., cognitive profiles) but also has documented harmful effects (i.e., facilitated communication) and/or pose a direct safety concern.

Conclusion

As highlighted above, behavioral reduction strategies found in OLT have the potential for assisting in de-implementation efforts alongside efforts to increase implementation of EBPs. Within the OLT framework, whether we are increasing or decreasing behavior, it is essential to take a functional approach that allows us to have a clear picture of the behavior targeted for change, what our goals are, and what contingencies initiated and maintain the behaviors in question. For behaviors themselves, we have to make sure—similar to implementation efforts—de-implementation efforts operationally define the low-value practice to have a clear idea of what behavior is targeted for behavior reduction and what behaviors are not part of the practice. This is particularly important in multi-component (e.g., packaged) programs where many different practices may be involved, some of which may have evidential support and others may not. As a result, a clear operational definition of the behavior along with measurable and observable goals of the outcomes will be essential part of de-implementation efforts from an OLT perspective. Moreover, once a very clear definition and goal is identified of the behavior, a functional analysis of contingencies that are maintaining or increasing the low-value practices needs to be conducted, as well as determining replacement or alternative behaviors that will need to be actively supported in place of the unwanted practices. As many of these low-value practices are not only maintained by direct contingencies but are also rule-governed, a functional-contextual analysis to understand related behaviors should also extend to analyzing how and where these rules are generated in order to reduce them. Furthermore, understanding how these rules are generated and maintained may inform how we can create more flexible rules, and rules that would promote tracking practices that have evidence to support their use. Strengthening direct contingencies with the behavior reduction strategies detailed above can also help impact rule-governed behavior by creating more immediate and salient consequences contingent on low-value practices.

Overall, de-implementation efforts in school psychology are an essential part of making our practices evidence-based alongside with the equally important task of effective implementation of such practices. We agree with McKay et al. (2018) and extend their argument to posit that a functional-contextual approach rooted in applied behavior analysis and OLT has great potential to inform de-implementation efforts in the field of school psychology. This includes a functional-analysis of why low-value practices persist and the unique strategies of behavior reduction that can be applied at the individual, system, and policy level. Future research should investigate (a) functional matching of contextual variables to specific behavior reduction strategies, (b) the effectiveness of behavior reduction strategies for de-implementation of low-value practices, (c) the acceptability and manageability of such strategies, and (d) how identification of specific contextual variables may lead to differential selection of behavior reduction strategies.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Ryan L. Farmer

Notes

Author Biographies

Ryan L. Farmer, PhD, is an assistant professor with the school psychology program at Oklahoma State University. His areas of professional interest include evidence-based practices, assessment processes, and meta-science in school psychology.

Imad Zaheer, PhD, is an associate professor with the school psychology program at St. John’s University. His professional interests include implementation science, school mental health, and creating integrated multi-tiered systems of support in schools.

Gary J. Duhon, PhD, is a professor of school psychology at Oklahoma State University. His teaching and research interests include intervention development, behavioral consultation, pre-referral intervention models, and applied behavior analysis.

Stephanie Ghazal is a graduate student in the school psychology program at Oklahoma State University. She is interested in assessment processes, academic and behavioral interventions, working with English language learners, and the use and history of evidence-based practices.

References

Bodegom-Vos

L. V.

Davidoff

Marang-van de Mheen

(2017). Implementation and de-implementation: Two sides of the same coin? BMJ Quality and Safety, 26(6), 495–501. https://doi.org/10.1136/bmjqs-2016-005473

Chambers

D. A.

Glasgow

R. E.

Stange

K. C.

(2013). The dynamic sustainability framework: Addressing the paradox of sustainment amid ongoing change. Implementation Science, 8(1), 117. https://doi.org/10.1186/1748-5908-8-117

Cooper

J. O.

Heron

T. E.

Heward

W. L.

(2020). Applied behavior analysis (3rd ed.). Pearson.

Forman

S. G.

(2019). Implementation science and school psychology: Future needs for research and practice. Journal of School Psychology, 76, 62–65. https://doi.org/10.1016/j.jsp.2019.07.018

Forman

S. G.

Fagley

N. S.

Chu

B. C.

Walkup

J. T.

(2012). Factors influencing school psychologists’ “willingness to implement” evidence-based interventions. School Mental Health: A Multidisciplinary Research and Practice Journal, 4(4), 207–218. https://doi.org/10.1007/s12310-012-9083-z

Freeman

Sugai

(2013). Recent changes in state policies and legislation regarding restraint or seclusion. Exceptional Children, 79(4), 427–438. https://doi.org/10.1177/001440291307900403

Friman

P. C.

Poling

(1995). Making life easier with effort: Basic findings and applied research on response effort. Journal of Applied Behavior Analysis, 28(4), 583–590.

Gifford

E. V.

Hayes

S. C.

(1999). Functional contextualism: A pragmatic philosophy for behavioral science. In O’Donohue

Kitchener

(Eds.), Handbook of behaviorism (pp. 285–327). Academic Press.

Harte

Barnes-Homes

Barnes-Holmes

Kissi

(2020). The study of rule-governed behavior and derived stimulus relations: Bridging the Gap. Perspectives on Behavior Science, 43, 361–385. https://doi.org/10.1007/s40614-020-00256-w

10.

Jacobson

J. W.

Mulick

J. A.

Schwartz

A. A.

(1995). A history of facilitated communication: Science, pseudoscience, and antiscience science working group on facilitated communication. American Psychologist, 50(9), 750–765. https://doi.org/10.1037/0003-066X.50.9.750

11.

Johns

D. M.

Bayer

Fairchild

A. L.

(2016). Evidence and the politics of de-implementation: The rise and decline of the “counseling and testing” paradigm for HIV prevention at the US Centers for Disease Control and Prevention. Milbank Quarterly, 94(1), 126–162. https://doi.org/10.1111/1468-0009.12183

12.

Koocher

G. P.

McMann

M. R.

Stout

A. O.

Norcross

J. C.

(2015). Discredited assessment and treatment methods used with children and adolescents: A Delphi Poll. Journal of Clinical Child & Adolescent Psychology, 44(5), 722–729. https://doi.org/10.1080/15374416.2014.895941

13.

Kratochwill

T. R.

Shernoff

E. S.

(2004). Evidence-based practice: Promoting evidence-based interventions in School Psychology. School Psychology Review, 33(1), 34–48.

14.

Lilienfeld

S. O.

Ammirati

David

(2012). Distinguishing science from pseudoscience in school psychology: Science and scientific thinking as safeguards against human error. Journal of School Psychology, 50(1), 7–36. https://doi.org/10.1016/j.jsp.2011.09.006

15.

Lilienfeld

S. O.

Wood

J. M.

Garb

H. N.

(2000). The scientific status of projective techniques. Psychological Science in Public Interest, 1(2), 27–66. https://10.1111/1529-1006.002

16.

Martin

Pear

(2019). Behavior modification: What it is and how to do it (6th ed.). Prentice-Hall, Inc.

17.

McGill

R. J.

Dombrowski

S. C.

Canivez

G. L.

(2018). Cognitive profile analysis in school psychology: History, issues, and continued concerns. Journal of School Psychology, 71, 108–121. https://doi.org/10.1016/j.jsp.2018.10.007

18.

Mckay

V. R.

Margaret Dolcini

Hoffer

L. D.

(2017). The dynamics of de-adoption: A case study of policy change, de-adoption, and replacement of an evidence-based HIV intervention. Translational Behavioral Medicine, 7(4), 821–831. https://doi.org/10.1007/s13142-017-0493-1

19.

Mckay

V. R.

Morshed

A. B.

Brownson

R. C.

Proctor

E. K.

Prusaczyk

(2018). Letting go: Conceptualizing intervention de-implementation in public health and social service settings. American Journal of Community Psychology, 62(1–2), 189–202. https://doi.org/10.1002/ajcp.12258

20.

Meehl

P. E.

(1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. https://doi.org/10.1037/0022-006X.46.4.806

21.

Montini

Graham

I. D.

(2015). Entrenched practices and other biases”: Unpacking the historical, economical, professional, and social resistance to de-implementation. Implementation Science, 10(1), 24. https://doi.org/10.1186/s13012-015-0211-7

22.

National Association of School Psychologists. (2020). The professional standards of the National Association of School Psychologists. Author. https://www.nasponline.org/standards-and-certification/nasp-practice-model

23.

Niven

D. J.

Mrklas

K. J.

Holodinsky

J. K.

Straus

S. E.

Hemmelgarn

B. R.

Jeff

L. P.

Stelfox

H. T.

(2015). Towards understanding the de-adoption of low-value clinical practices: A scoping review. BMC Medicine, 13(1), 55. https://doi.org/10.1186/s12916-015-0488-z

24.

O’Hora

Barnes-Homes

Stewart

(2014). Antecedent and consequential control of derived instruction-following. Journal of the Experimental Analysis of Behavior, 102(1), 66–85. https://doi.org/10.1002/jeab.95

25.

Patey

A. M.

Hurt

C. S.

Grimshaw

J. M.

Francis

J. J.

(2018). Changing behavior ‘more or less”—do theories of behavior inform strategies for implementation and de-implementation? A critical interpretive synthesis. Implementation Science, 13(1), 134. https://doi.org/10.1186/s13012-018-0826-6

26.

Prasad

Ioannidis

J. P.

(2014). Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implementation Science, 9(1). https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-9-1

27.

Ryan

J. B.

Peterson

R. L.

(2004). Physical restraint in school. Behavioral Disorders, 29(2), 154–168.

28.

Rushmer

Davies

(2004). Unlearning in health care. Quality and Safety in Health Care, 13(2), ii10-ii15. https://dx-doi-org-s.web.bisu.edu.cn/10.1136%2Fqshc.2003.009506

29.

Sanetti

L. M. H.

Collier-Meek

M. A.

(2019). Increasing implementation science literacy to address the research-to practitioner gap in school Psychology. Journal of School Psychology, 76, 33–47. https://doi.org/10.1016/j.jsp.2019.07.008

30.

Shapiro

E. S.

(2006). Are we solving the big problems? School Psychology Review, 35(2), 260–265. https://doi.org/10.1080/02796015.2006.12087990

31.

Skiba

R. J.

Rausch

M. K.

(2006). Zero tolerance, suspension, and expulsion: Questions of equity and effectiveness. In Handbook of classroom management: Research, practice, and contemporary issues (pp. 1063–1089). Erlbaum Associates Publishers.

32.

Skinner

B. F.

(1966). The phylogeny and ontogeny of behavior. Science, 153(3741), 1205–1213.

33.

Skinner

B. F.

(1974). About behaviorism. Knopf.

34.

Stahl

S. A.

McKenna

M. C.

Pagnucco

J. R.

(1994). The effects of whole-language instruction: An update and a reappraisal. Educatonal Psychologist, 29(4), 175–185. https://doi.org/10.1016/j.jsp.2018.10.007

35.

Stanger-Hall

K. F.

Hall

D. W.

(2011). Abstinence-only education and teen pregnancy rates: Why we need comprehensive sex education in the U.S. PLos One, 6(10), e24658. https://doi.org/10.1371/journal.pone.0024658

36.

VanDerHeyden

A. M.

(2018). Why do school psychologists cling to ineffective practices? Let’s do what works. School Psychology Forum, 12(1), 44–52.

37.

Vollmer

T. R.

Iwata

B. A.

(1992). Differential reinforcement as treatment for behavior disorders: Procedural and functional variations. Research in Developmental Disabilities, 13(4), 393–417.

38.

Westling

D. L.

Trader

B. R.

Smith

C. A.

Marshall

D. S.

(2010). Use of restraints, seclusion, and aversive procedures on students with disabilities. Research and Practice for Persons with Severe Disabilities, 35(3–4), 116–127. https://doi.org/10.2511/rpsd.35.3-4.116