Treatment acceptability of interventions published in six school psychology journals

Abstract

Treatment acceptability (TA) is critical when selecting and implementing an intervention, as TA is associated with treatment outcomes. The significance of TA is reflected in school psychology models for services that state that school psychologists should address TA during development, implementation, and evaluation of interventions. However, the understanding of TA as it pertains to issues relevant to school psychologists is limited by the lack of analyses of the reporting of TA in intervention research. This study addresses this topic, as it represents a content analysis of intervention research published in six peer-reviewed journals in the field of school psychology from 2005–2014; the search yielded 2,343 articles, of which 243 were intervention research articles that included children as target samples. Overall results suggest low assessment or monitoring of TA in school psychology intervention research. Implications for practice and future research are discussed.

Keywords

Consumer satisfaction intervention research methods school psychology treatment acceptability

Treatment acceptability (TA) refers to the degree to which intervention strategies are seen as fair, reasonable, feasible, and appropriate by consumers for addressing behavioral or academic concerns (Kazdin, 1980; Kazdin, 1981). Factors critical to the concept of TA include efficacy considerations, secondary effects, social/legal implications, and practical considerations (Lennox and Miltenberger, 1990). Other conceptualizations of TA include factors related to immediacy of treatment effects, influence that practitioners may have on consumers, and judgments regarding freedom of choice in selecting treatments (Calvert and Johnson, 1990; Carter, 2008). Research–primarily analogue research–has supported some of these conceptualizations by indicating that different factors contribute to TA, including consultee and client knowledge and understanding of the intervention, time and effort required for implementation, complexity of the intervention, severity of the problem, and intervention effectiveness (Eckert & Hintze, 2000; Elliott, 1988).

Models of practice that incorporate TA suggest that it is a critical factor to consider in selecting and implementing interventions. For example, one of the first models of TA emphasized the sequential and reciprocal relationships between TA, treatment use, treatment integrity, and treatment effectiveness (i.e. outcomes) (Witt & Elliott, 1985). This early model has been expanded upon (e.g. Reimers, Wacker, & Koeppl, 1987), but it continues to serve as a useful heuristic for understanding the significance of TA. Namely, TA and treatment integrity are related constructs (Eckert & Hintze, 2000), and a higher degree of TA leads to improved treatment integrity and higher levels of treatment efficacy (Lane, Bocian, MacMillan, & Gresham, 2004; Perepletchikova & Kazdin, 2005).

This conceptualization of TA is of particular significance to school psychologists. As stated by the International School Psychology Association (ISPA), school psychologists should collaborate with other school personnel, as well as with students, to develop appropriate interventions to help enhance learning and development in children and youth (ISPA, n.d.). Thus, it is suggested that school psychologists should be aware of TA and of ways to gauge acceptability in order to enhance the likelihood of an intervention being implemented as intended by collaboration partners (Albers, Elliott, Kettler, & Roach, 2013). This idea is further reflected in the National Association of School Psychologists (NASP)–the national school psychology organization in the United State of America–model for services that states that school psychologists should address TA during development, implementation, and evaluation of academic, behavioral, and mental health interventions (NASP, 2010). Moreover, educational policy and legislation, such as the Individuals with Disabilities Education Act (2004) of the United States of America, have mandated the incorporation of parents, teachers, principals, and others in the development and implementation of treatments for students receiving special education services (Carter, 2008). It is important to gauge TA from these multiple persons, as knowing acceptability ratings of certain procedures may help to maintain more positive work relationships and support better outcomes (Easton & Erchul, 2011).

Despite the apparent importance of TA, research in this area is limited by the dearth of studies that incorporate TA in a meaningful manner into study methods. This may account for the inconsistent link between TA and study outcomes. For example, a few researchers have found no significant relationship between TA and treatment integrity (e.g. Noell et al., 2005; Sterling-Turner & Wason, 2002; Wickstrom, Jones, LaFleur, & Witt, 1998). Others have found a relationship between TA and adherence to treatment procedures and implementation (Allinder, 1997; Mautone et al., 2009; Reimers & Wacker, 1992; Truscott & Cosgrove, 2000). In order to establish more definitive associations and to continue to define the concept of TA, it is important to continue to examine studies that incorporate TA.

As previously noted, there is limited research about the availability of TA data, how it is collected, and who reports it. In one study, Sheridan, Welch, and Orme (1996) provided a review of consultation research conducted between 1985–1995 and found that 37% of studies used TA measures. Otherwise, journal content analyses that focus on this issue have not been conducted. Conversely, treatment integrity–a related construct–has been the subject of multiple reviews (e.g. Sanetti, Gritter, & Dobey, 2011; Sanetti & Kratochwill, 2013). Systematic reviews of both constructs are important to school psychologists, as TA has come to be viewed as an important consideration in intervention research (Gresham & Lopez, 1996). Thus, the purpose of the current study was to code the TA data reported in school psychology intervention journal articles. TA characteristics examined included assessment of TA, person(s) reporting TA, and method(s) of assessing TA. Another purpose was to analyse the assessment of TA across other study characteristics (e.g. publication year and research design) to better understand the context of the reporting of TA.

Methods

Sample

Six journals of relevance to the field of school psychology were selected for inclusion in the analysis: School Psychology Review, School Psychology Quarterly, Journal of School Psychology, Psychology in the Schools, School Psychology International, and Journal of Applied School Psychology. Electronic copies of the journals were obtained through the official journal websites, as well as the university’s journal access and library resources. All volumes and issues of the selected journals published from 2005 through 2014–representing a 10-year period–were included in this analysis.

Materials

A coding instrument and two-stage coding protocol were created to assist in the analysis of the journal articles. In the initial stage of coding, article types were identified and intervention articles were categorized. Specifically, the types of journal articles included: research articles, literature review papers, editorials/commentaries, articles examining the psychometric properties of instruments, and test/book reviews. Research articles were further categorized as: intervention research, descriptive or survey studies, or qualitative research studies. Intervention research articles (i.e. articles that included an independent variable and associated outcome data) with preschool through grade 12 samples were classified as: (a) case studies; (b) experimental/quasi-experimental studies; or (c) single-case studies; see Villarreal and colleagues (2013) for a more thorough discussion of this initial stage of coding.

The purpose of the second stage of coding was to specifically examine the extent to which treatment acceptability was assessed and reported in the intervention research studies. Thus, additional variables coded at this stage included: (a) assessment of TA; (b) person(s) reporting TA; and (c) method(s) of assessing TA. Coding procedures for each of these variables are described below.

TA assessment

TA assessment was based on whether studies included TA assessment data. For this variable, categories included: (a) study authors assessed TA and reported quantitative TA data; (b) authors mentioned that TA was assessed or monitored, but they did not report any quantitative TA data; and (c) authors did not state that TA was monitored or assessed in any way, and they did not report TA data.

TA reporters

The person(s) reporting TA were also coded. For this variable, categories included: (a) general or special education teacher or other staff (e.g. school psychologist, school counselor); (b) student’s parent(s); and (c) student or participant.

TA assessment methods

The TA assessment method was also coded. For this variable, categories included: (a) formal TA instrument (see Carter [2007] for a review of TA instruments); (b) informal instrument (e.g. researcher developed rating scale) or adapted TA instrument; (c) individual or group interview; and (d) open-ended questionnaire or comments.

Coding procedures and inter-rater agreement

For the first coding stage, six coders were trained to locate, retrieve, and code all articles in the selected journals. Inter-rater reliability, based on inter-rater percent agreement, was continually monitored; 20% of all journal articles were randomly selected and assigned for additional coding. The average agreement between coders was 92.2%. Any questions or concerns were resolved through consensus after discussion between each coder and the first author.

For the second coding stage, the first author trained two graduate students to code those variables associated with TA (i.e. assessment of TA, persons reporting TA, and methods of assessing TA). Each graduate student was responsible for coding all of the articles identified as intervention studies with preschool and school-age samples; thus, each of the articles was coded twice. The inter-rater percent agreement between the two coders was 90.2%. All instances of disagreements were resolved through additional coding completed by the first author.

Data analysis

As the primary purpose of this study was to highlight the representation of characteristics of treatment acceptability reported in intervention studies, data analysis was primarily based on descriptive statistics for each of the variables coded. However, to analyse the association between the assessment of TA (i.e. yes or no) with other study variables–including publication year and research design–Chi-square tests of independence analyses were conducted. For these analyses, TA was coded as a categorical variable that included two values: (1) study assessed or monitored TA; and (2) study did not assess or monitor TA.

Results

Types of articles

Over the ten-year period, 2,343 articles were published in the six identified journals. Results of the initial coding stage indicated that 243 articles (10.4%) were identified as intervention studies with preschool through grade 12 samples.

TA characteristics

TA assessment

Study authors assessed TA and reported quantitative TA data in approximately 30.5% of cases. In 5.8% of cases, authors mentioned that TA was assessed or monitored, but they did not report any quantitative TA data. In a majority (63.8%) of the articles coded, authors did not state that TA was monitored or assessed in any way.

TA reporters

In approximately 27.6% of the articles coded, teachers or other staff (e.g. school psychologists) provided a report of treatment acceptability. In 6.2% of cases, parents provided a report of treatment acceptability; in 23.0% of cases, students provided a report of treatment acceptability. Notably, in 18.9% of studies TA was reported by multiple groups of persons (e.g. teacher and student).

TA assessment methods

In 14.4% of the coded studies, TA was assed using a formal TA instrument. A researcher-developed rating scale or adapted version of a formal TA instrument was used in 18.1% of cases. In 2.1% of studies, TA data were collected through individual or group interviews; in 5.8% of cases, data were collected through open-ended comment forms or surveys.

Treatment acceptability by study characteristics

Year

Across the ten-year period, the percentage of coded studies that assessed or monitored TA was highly variable. Approximately 60.0% of intervention studies published in 2006 assessed or monitored TA. Conversely, only 16.7% of studies published in 2005 assessed TA; see Figure 1 for a graphic representation of these results. However, χ² analyses indicated no significant differences in the percentage of articles assessing or monitoring TA across year of publication (χ² [9, n = 243] = 9.97, p = 0.35).

Figure 1.

Percentage of intervention articles assessing or monitoring treatment acceptability by year. In total, 36.2% of articles included information indicating that TA was assessed or monitored.

Research design

TA was assessed or monitored in 27.8% of case studies, 21.2% of experimental or quasi-experimental studies, and 57.4% of single case studies. The χ² analyses indicated statistically significant differences (χ² [2, n = 243] = 30.43, p < 0.001) between the study designs.

Discussion

The purpose of this study was to provide initial insights into the assessment and reporting of TA in intervention articles published in journals of school psychology. Overall findings of this study highlight low TA assessment and monitoring in school psychology intervention research. Although there are no other studies in the field of school psychology to which to directly compare the results of the current study, it is helpful to compare the results of the current study with those examining related concepts, such as treatment integrity. In a review of school psychology literature published from 1995–2008, Sanetti et al. (2011) found that approximately one-half of the studies evaluated included quantitative treatment integrity data. The percentage of researchers monitoring treatment integrity is higher than that monitoring TA, even though both constructs are typically included in practice models (e.g. Lane et al., 2004; Perepletchikova & Kazdin, 2005). Moreover, best practices dictate that TA data be used in conjunction with treatment integrity data (Gresham & Lopez, 1996).

Sheridan et al. (1996) reported that TA was assessed in approximately 48% of consultation research studies published from 1985–1995. In that article, the authors mentioned that this represented an increase in acceptability measurement in consultation research, and they predicted that this would continue to be an area of improvement and growth. In contrast, the results of the current study suggest that only 36.2% of school psychology intervention articles measured TA. Results of the current study that suggest that the assessment of TA varied significantly based on research method may explain these differences. In the current study, articles presenting results of single case intervention were more likely to assess TA; conducting additional assessments and collecting additional data (e.g. TA data) is easier to do when working with small numbers of participants. Consultation studies are also more likely to be conducted using single case methods rather than methods employed in large scale studies (i.e. experimental methods). Notably, factors related to TA are listed as critical quality indicators within single-subject research designs to identify evidence-based practice in special education (Horner et al., 2005), but they are not listed for group experimental and quasi-experimental research designs (Gersten et al., 2005)

In the minority of studies that did assess TA, students and teachers were most likely to report TA. Moreover, in 52.9% of studies assessing or monitoring TA, TA was collected from multiple persons (e.g. teacher and student). These results are encouraging in that they indicate that researchers interested in assessing TA are taking a multi-consumer perspective to this approach. This reflects important factors considered critical to the construct of TA. Namely, they relate to the assumption that it is necessary to consider the acceptability or satisfaction to the target of an intervention (i.e. students), as well as the staff person (i.e. teacher) implementing the intervention (Carter, 2008; Lennox & Miltenberger, 1990). Research-developed or adapted rating scales and formal TA instruments are the most common methods of measuring TA. Although collecting any sort of TA data is preferable to not collecting this data, the utility of TA data is improved when a professional is able to use it to compare different interventions. Thus, it would seem that using non-standardized TA instruments would be of less value than utilizing a formal TA instrument for assessing different interventions. Researchers may look to using formal TA instruments rather than creating their own, especially if results are expected to be useful to practitioners.

Implications for practice and research

TA is critical when selecting and implementing an intervention, as research suggests that TA is associated with treatment outcomes (Mautone et al., 2009). Lennox and Miltenberger (1990) recommended that professionals collect representative acceptability ratings from the school in which they work for a number of different treatments and problem behaviors to guide treatment selection; the NASP model for services reflects this idea (2010). Despite the potential for TA to inform treatment selection and implementation, the results of this study suggest that this information is missing from current intervention research. In a majority of cases, practitioners looking to intervention research in an attempt to utilize evidence-based practices will not have access to TA data. To address this issue, researchers should be encouraged to incorporate TA into their research methods in order to provide an intervention indicator that may enhance the utility of intervention strategies and allow consumers of intervention research (e.g. teachers, school psychologists working in applied settings) to select those most likely to be perceived as favorable. Publishers should be encouraged to incorporate reporting of TA in publication guidelines. Currently, treatment fidelity is listed as an important topic in the American Psychological Association (APA) Working Group on Journal Article Reporting Standards (2008); practice models suggest that treatment acceptability should also be considered and added to reporting standards.

Limitations and future research

Although this study yielded significant findings, it has limitations. First, only journals in the field of school psychology were coded. This may contribute to a failure to grasp the full scope of intervention research, as relevant studies are presented in journals of related fields (e.g. educational psychology, special education). Additionally, the implications of these findings are limited by the low number of studies that have examined TA and the effect that differing levels of TA have on treatment outcomes and treatment selection by practitioners. Future studies may continue to focus on understanding the effects of TA on other areas (i.e. treatment integrity) posited to be influenced in models of TA and intervention outcomes.

Additionally, although this study included the examination of multiple journals that regularly publish articles of authors from around the world, most of the journals that were coded were established by associations in the United States of America (e.g. NASP, APA, American Academy of School Psychology). Moreover, a vast majority of the intervention research articles published in the coded journals represent studies that were conducted in the United States of America (see Villarreal, 2014). As models of school psychology practice may vary by nation, the lack of coding of more international journals could preclude the identification of differences in research and TA reporting standards between professionals in different nations. Additionally, although there were no significant trends in TA reporting over the ten-year period examined in this article, it may be the case that differences may be found across time in international journals. It has been suggested that there has recently been significant changes in the roles of school psychologists internationally, from a focus on assessment to problem solving, the delivery of interventions, and consulting (Jimerson et al., 2006); therefore, there may be differences in TA reporting in international journals or differences in reporting in international journals across time. This implication calls for an extension of research in this area to take a more comprehensive examination in school psychology research.

Footnotes

Author biographies

Victor Villarreal, PhD, is an Assistant Professor in the Department of Educational Psychology at the University of Texas at San Antonio in the United States of America. He is also a licensed psychologist and nationally certified school psychologist. His research interests include professional issues and the quality and utilization of evidence-based practices in the schools.

Christopher Ponce, BA, is a second-year student in the Master of Arts School Psychology program at the University of Texas at San Antonio. Christopher received his BA in Anthropology from the University of Texas at San Antonio.

Heveli Gutierrez, BS, is a second-year student in the Master of Arts School Psychology program at the University of Texas at San Antonio. Heveli received her BS in Psychology from the University of Texas–Pan American.

References

Albers

C. A.

Elliott

S. N.

Kettler

R. J.

Roach

A. T.

(2013) Evaluating intervention outcomes within problem-solving-based Assessment. In: Brown-Chidsey

Andren

K. J.

(eds) Assessment for intervention: A problem-solving approach, 2nd ed. New York, NY: Guilford Press, pp. 344–360.

Allinder

R. M.

(1997) Effects of acceptability on teachers’ implementation of curriculum-based measurement and student achievement in mathematics computation. Remedial and Special Education 18: 113–120. doi: 10.1177/074193259701800205.

APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) Reporting standards for research in psychology: Why do we need them? What might they be. American Psychologist 63: 839–851. doi: 10.1037/0003-066X.63.9.839.

Calvert

S. C.

Johnson

(1990) Acceptability of treatments for child behavior problems: Issues and implications for future research. Journal of Clinical Psychology 19: 61–74.

Carter

S. L.

(2007) Review of recent treatment acceptability research. Education and Training in Developmental Disabilities 42: 301–316.

Carter

S. L.

(2008) Further conceptualization of treatment acceptability. Education and Training in Developmental Disabilities 43: 135–143.

Easton

J. E.

Erchul

W. P.

(2011) An exploration of teacher acceptability of treatment plan implementation: Monitoring and feedback methods. Journal of Educational and Psychological Consultation 21: 56–77. doi: 10.1080/10474412.2011.544949.

Eckert

T. L.

Hintze

J. M.

(2000) Behavioral conceptualizations and applications of treatment acceptability: Issues related to service delivery and research methodology. School Psychology Quarterly 15: 123–148. doi: 10.1037/h0088782.

Elliott

S. N.

(1988) Acceptability of behavioral treatments: Review of variables that influence treatment selection. Professional Psychology: Research and Practice 19: 68–80. doi: 10.1037/0735-7028.19.1.68.

10.

Gersten

Fuchs

L. S.

Compton

Coyne

Greenwood

Innocenti

M. S.

(2005) Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children 71: 149–164. doi: 10.1177/001440290507100202.

11.

Gresham

F. M.

Lopez

M. F.

(1996) Social validation: A unifying concept for school-base consultation research and practice. School Psychology Quarterly 11: 204–227. doi: 10.1037/h0088930.

12.

Horner

R. H.

Carr

E. G.

Halle

McGee

Odom

Wolery

(2005) The use of single-subject research to identify evidence-based practice in special education. Exceptional Children 71: 165–179. doi: 10.1177/001440290507100203.

13.

Individuals With Disabilities Education Act, 20 U.S.C. § (1400) (2004).

14.

International School Psychology Association (n.d.). The accreditation of professional training programs in school psychology.2004. Retrieved from http://www.ispaweb.org/wp-content/uploads/2012/10/Accreditation-Standards-revised-2010.pdf.

15.

Jimerson, S. R., Graydon, K., Mantak, Y., Lam, S., Thurm, J., Kleuva, N., et al. & The ISPA Research Committee. (2006). The international school psychology survey. School Psychology International, 27, 532. doi: 10.1177/0143034306062813.

16.

Kazdin

A. E.

(1980) Acceptability of alternative treatment for deviant child behavior. Journal of Applied Behavior Analysis 13: 259–273. doi: 10.1901/jaba.1980.13-259.

17.

Kazdin

A. E.

(1981) Acceptability of child treatment techniques: The influence of treatment efficacy and adverse side effects. Behavior Therapy 11: 493–506. doi: 10.1016/S0005-7894(81)80087-1.

18.

Lane

L. L.

Bocian

K. M.

MacMillan

D. L.

Gresham

F. M.

(2004) Treatment integrity: An essential–but often forgotten component of school-based interventions. Preventing School Failure 48: 36–43. doi: 10.3200/PSFL.48.3.36-43.

19.

Lennox

D. B.

Miltenberger

R. G.

(1990) On the conceptualization of treatment acceptability. Education and Training in Mental Retardation 25: 211–224.

20.

Mautone

J. A.

DuPaul

G. J.

Jitendra

A. K.

Tresco

K. E.

Vile Junod

Volpe

R. J.

(2009) The relationship between treatment integrity and acceptability of reading interventions for children with ADHD. Psychology in the Schools 46: 919–931. doi: 10.1002/pits.20434.

21.

National Association of School Psychologists (2010) National Association of School Psychologists model for comprehensive and integrated school psychological services. School Psychology Review 39: 320–333.

22.

Noell, G. H., Witt, J. C., Slider, N. J., Connell, J. E., Gatti, S. L., Williams, K. L., et al (2005) Treatment implementation following behavioral consulation in schools: A comparison of three follow-up strategies. School Psychology Review 34: 87–106.

23.

Perepletchikova

Kazdin

(2005) Treatment integrity and therapeutic change: Issues and research recommendations. Clinical Psychology Science and Practice 12: 365–383. doi: 10.1093/clipsy/bpi045.

24.

Reimers

T. M.

Wacker

D. P.

Koeppl

(1987) Acceptability of behavioral interventions: A review of the literature. School Psychology Review 16: 212–229.

25.

Reimers

T. M.

Wacker

D. P.

Cooper

L. J.

De Raad

A. O.

(1992) Acceptability of behavioral treatments for children: Analog and naturalistic evaluation by parents. School Psychology Review 21: 628–643.

26.

Sanetti

L. M. H.

Gritter

K. L.

Dobey

L. M.

(2011) Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008. School Psychology Review 40: 72–84.

27.

Sanetti

L. M. H.

Kratochwill

T. R.

(2013) Treatment integrity assessment within a problem-solving model. In: Brown-Chidsey

Andren

K. J.

(eds) Assessment for intervention: A problem-solving approach, New York, NY: Guildford Press, pp. 297–318.

28.

Sheridan

S. M.

Welch

Orme

S. F.

(1996) Is consultation effective: A review of outcome research. Remedial and Special Education 17: 341–354. doi: 10.1177/074193259601700605.

29.

Sterling-Turner

H. E.

Watson

T. S.

(2002) An analog investigation of the relationship between treatment acceptability and treatment integrity. Journal of Behavioral Education 11: 39–50. doi: 10.1023/A:1014333305011.

30.

Truscott

S. D.

Cosgrove

(2000) The acceptability of organizational consulation with prereferral intervention teams. School Psychology Quarterly 15: 172–206. doi: 10.1037/h0088784.

31.

Villarreal

(2014) Cultural and linguistic diversity representation in school psychology intervention research. Contemporary School Psychology 18: 159–167. doi: 10.1007/s40688-014-0027-5.

32.

Villarreal

Gonzalez

J. E.

McCormick

A. S.

Simek

Yoon

(2013) Articles published in six school psychology journals from 2005–2009: Where’s the intervention research. Psychology in the Schools 50: 500–519. doi: 10.1002/pits.21687.

33.

Wickstrom

K. F.

Jones

K. M.

LaFleur

L. H.

Witt

J. C.

(1998) An analysis of treatment integrity in school-based behavioral consulation. School Psychology Quarterly 13: 141–154. doi: 10.1037/h0088978.

34.

Witt

J. C.

Elliott

S. N.

(1985) Accetpability of classroom intervention strategies. In: Kratochwill

T. R.

(ed.) Advances in school psychology Vol 4.Mawwah, NJ: Erlbaum, pp. 251–288.