Abstract
This study sought to validate a new measure, the Classroom Cohesion Survey (CCS), designed to examine the relationship between teachers and classroom assistants in autism support classrooms. Teachers, classroom assistants, and external observers showed good inter-rater agreement on the CCS and good internal consistency for all scales. Simple factor structures were found for both teacher- and classroom assistant–rated scales, with one-factor solutions for both scales. Paired t tests revealed that on average, classroom assistants rated classroom cohesion stronger than teachers. The CCS may be an effective tool for measuring cohesion between classroom staff and may have an important impact on various clinical and implementation outcomes in school settings.
Since the Individuals with Disabilities Education Act was passed in 1990, the number of children with autism spectrum disorders (ASD) served in the public school system has increased by 13% to 28% each year and by more than 1700% from 1992 to 2008 (Maenner & Durkin, 2010). Schools are under increasing pressure to effectively incorporate evidence-based interventions for children with ASD into special education classrooms (National Research Council, 2001). These interventions are complex and involve using multiple instructional strategies concurrently and generally have not been effectively implemented in community settings (Dingfelder & Mandell, 2011; Stahmer, Collings, & Palinkas, 2005).
Implementing these complex intervention strategies with fidelity in under-resourced urban schools requires a collaborative effort among the multiple staff members who provide behavioral and educational support in special education classrooms (Dingfelder & Mandell, 2011). More than 75% of special educators report that they supervise assistants and one-to-one aides (French, 2001). Studies find that classroom assistants perform their duties most effectively when (a) they are appropriately supervised, (b) their roles are clearly defined, (c) they are trained for assigned tasks, and (d) they participate in regularly scheduled planning meetings (French, 2001). Teachers typically receive little or no training in how to interact with their aides, and little is known about the quality and impact of staff relationships in these classrooms (French, 2001; Scheuermann, Webber, Boutot, & Goodwin, 2003).
In other fields, conceptual models from organizational psychology have been used to improve our understanding of team functioning (Wang, Ying, Jiang, & Klein, 2006). Much like in other settings, teachers and classroom assistants must function as a team, defined as “a collection of individuals who are interdependent in their tasks” (Cohen & Bailey, 1997, p. 241). Team functioning, which can include a number of dimensions such as collaboration, conflict resolution, leadership, and cohesion, has been shown to affect both performance and outcomes across a broad range of settings (Lemieux-Charles & McGuire, 2006). Cohesion, the tendency of a group to “stick together and remain united in the pursuit of its goals and objectives” (Wang et al., 2006), is most closely associated with team performance and outcomes (Lemieux-Charles & McGuire, 2006).
Although studies have examined the impact of staff cohesion in general on job performance and intervention implementation, no published study has addressed cohesion among staff in special education classrooms, where many children with autism spend the majority of their time (National Research Council, 2001). This dearth of research focused on special education settings may be due in part to the fact that no measures have been specifically developed for special education settings. The few studies of cohesion in academic settings—and measures developed for these settings—have focused on cohesion among students (e.g., Rovai, 2002). The majority of other cohesion measures have been developed for use in business or health care settings. Although these existing cohesion questionnaires, such as the Perceived Cohesion Scale (Bollen & Hoyle, 1990), may capture general aspects of the classroom dynamic, such as a sense of belonging and morale, they do not capture task-specific performance issues that are relevant to the special education classroom, such as the teachers’ role in providing effective supervision of paraprofessionals. This is important because performance is predicted most strongly when there is a match between the dimensions of cohesion and performance (Chang & Bordia, 2001). Existing measures that tap into more task-specific performance tend to focus on individuals at the same level within a group (e.g., Group Environment Questionnaire; Carron, Widmeyer, & Brawley, 1985; Chang & Bordia, 2001), rather than a supervisor–supervisee relationship with different levels of training and different roles. Developing tools that capture this hierarchical relationship is an important first step in advancing our study of how the special education classroom staff cohesion affects intervention implementation.
We recently completed a randomized-controlled field trial of an evidence-based behavioral intervention for children with ASD, which required classroom staff to deliver a proven-efficacious comprehensive intervention package, Strategies for Teaching based on Autism Research (STAR; Arick, Loos, Falco, & Krug, 2004), to children in autism support classrooms in a large, urban school district (Mandell et al., 2013). During the study, we became aware of significant variability in the degree to which teacher and classroom assistants considered themselves part of a team working toward a common goal. We observed that some classrooms that functioned as a cohesive unit were better able to implement STAR, whereas other classrooms with more fragmented teams had more difficulty. This observation that the relationship among classroom staff may be associated with the implementation of evidence-based practices in classrooms led us to develop a tool that measures classroom staff cohesion. We hypothesize that classroom cohesion will be associated with two factors: (a) teacher burnout or stress, with teachers who experience higher levels of burnout having poorer working relationships with their classroom assistants and lower classroom cohesion, and (b) STAR program fidelity, in which higher classroom cohesion will be associated with higher program fidelity. In this article, we present this measure and explore its psychometric properties.
Method
Participants
Data were drawn from the 3-year field trial described above (Mandell et al., 2013) and from a 2-year District and University fee-for-service contract for the 2011-2013 school years. The current study includes data from teachers (n = 149) and classroom assistants (n = 154) who participated in either Year 3 of the study or Year 4 or 5 as a fee-for-service contract. Table 1 provides a description of the sample. A sub-sample of teacher (n = 53) and classroom assistant (n = 53) data from Year 3 were used for additional analyses.
Teacher and Classroom Assistant Baseline Characteristics: Descriptive Statistics.
Measures
Classroom Cohesion Survey (CCS; teacher and classroom assistant versions)
The CCS was developed and revised in multiple stages. In the first stage, the literature on cohesion was reviewed. From this literature review, a preliminary survey addressing gaps in the cohesion literature pertinent to our classroom observations from the randomized trial was constructed. A total of 20 items were initially generated for review. In the second stage, the preliminary survey was distributed to teachers from a nonparticipating district to ensure its straightforwardness and relevance. In the final stage, two items that were confusing or deemed irrelevant were revised or eliminated. These included questions related to additional support staff (e.g., one-to-one instructional assistants from outside behavioral agencies) and negative feelings toward classroom staff. The CCS–Teacher version is an 18-item self-report of the working relationship between the teacher and classroom assistant. The CCS–Classroom Assistant version is also an 18-item self-report with items that parallel the teacher version. Responses from both versions are structured on a 5-point Likert-type scale (1 = not true at all, 2 = hardly ever true, 3 = true sometimes, 4 = true most of the time, 5 = always true).
Maslach Burnout Inventory (MBI)–Education Form (Maslach, Jackson, & Leiter, 1996)
To examine convergent validity with cohesion, teacher burnout was measured using the MBI, a 22-item self-report inventory designed specifically for diagnosing burnout and job stress in teachers. It is composed of three subscales: Emotional Exhaustion (EE), Depersonalization (DP), and Personal Accomplishment (PA). Responses are structured on a 7-point Likert-type scale ranging from 0 (feeling has never been experienced) to 6 (feeling is experienced daily). Scores were derived using item-level means for each subscale (ranging from 0 to 6). The MBI was available for the 53 teachers in Year 3 of the randomized controlled trial.
Program fidelity
For teachers included in Year 3 of the study (n = 53), video observations were coded to assess the teachers’ fidelity of implementation of each component of STAR. The STAR program combines three instructional approaches based on the principles of applied behavior analysis: discrete trial training (DTT), pivotal response training (PRT), and functional routines into a comprehensive curriculum for children with ASD. DTT is a highly structured, one-on-one teaching strategy that breaks complex behaviors into smaller teachable units (Arick et al., 2004), whereas PRT uses a more naturalistic, play-based approach to target crucial skills, including motivation and responsivity to the environment that are pivotal for many other skills (Koegel et al., 1989). Functional routines are predictable events that involve a chain of behaviors and are implemented in the context of daily classroom activities. Each teacher was filmed for 30 min (10 min of each of the three core intervention strategies) once per month. We selected the latest fidelity observation period (occurring between March and May of 2011) to examine the predictive validity of the CCS. Research assistants, who were blind to the research hypotheses and teachers’ training group or experience level, were trained to code the video samples using a set of behavioral definitions for each component of STAR. Measures for each component of the STAR program were designed based on the manuals and in consultation with the program developers to ensure teachers implemented the program in the way that it was designed (Mandell et al., 2013). Coders rated the use of each component of STAR on a 1 to 5 Likert-type scale after viewing the entire video clip. The coding criteria differed for each STAR intervention strategy. A low score of 1 indicated the teacher did not use the strategy during the session or never implemented it correctly. A high score of 5 indicated the teacher implemented the component competently throughout the segment. To create a global rating of fidelity, the percentage of trials in a single videotaped session that earned a passing score (in which 75% of the steps were implemented competently and coded with a 4 or 5) was calculated. All scores for each program component were averaged across all months of the academic year to create a cumulative measure of STAR program fidelity throughout the academic year.
Procedure
Teachers and classroom staff were recruited through the District’s Office of Specialized Services. The District sent each staff member a letter describing the project and asking him or her to participate. In Year 3, the classroom consultants were graduate and postdoctoral-level fellows in school and educational psychology with training in applied behavior analysis recruited from a local university. Some teachers and classroom staff were new to the STAR program in Year 3, whereas others had received previous training in Years 1 and 2, but all received the same training in Year 3. This included didactic instruction (intense professional development workshops throughout the school year) and in vivo coaching (45-60 min sessions that included direct observation, modeling, and performance feedback) that occurred every 2 weeks. Teachers were asked to facilitate DTT and PRT sessions as well as functional routines throughout the day with their students to enhance academic, language, and social outcomes. Teachers were asked to complete a battery of assessments that included demographic measures, the CCS, and MBI. As an incentive, teachers received a small stipend for completing these measures. Classroom assistants attended professional development workshops with the teachers, and all classroom assistants were invited to participate during in vivo coaching. However, classroom assistants varied in their participation during in vivo coaching. Some classroom assistants were highly involved in delivering instruction to students and were asked to facilitate DTT and PRT sessions and functional routines. Other classroom assistants did not actively participate in consultation and were not involved in delivering instruction to students, but instead took the role of monitoring students while the teacher provided instruction. Classroom assistants also were asked to complete demographic measures and the CCS.
During Years 4 and 5 as a fee-for-service contract, classroom consultants were full-time staff members with a minimum of a master’s degree in school psychology, applied behavior analysis, or education. Training in Year 4 included didactic instruction and in vivo coaching. Data were included for classrooms that were new to STAR. Didactic instruction (professional development workshops) and in vivo coaching (60-90 min sessions that included direct observation and modeling) occurred twice per month. In Year 5, all teachers receiving consultation were new to the STAR program. Didactic instruction occurred once for teachers and twice for classroom assistants, and in vivo coaching (60-90 min sessions that included direct observation and modeling) occurred once every 2 to 3 weeks.
Statistical Analyses
All analyses were conducted using PASW software version 20.0. First, exploratory principal axis factor analyses were conducted to establish the best factor solution for the teacher- and classroom assistant–rated CCS. We sought factor solutions that would meet the following criteria: (a) satisfy Cattell’s (1966) scree test, (b) retain at least three items per factor with salient loadings (≥ .40), (c) yield high internal consistency for the scale with salient items included (≥ .70), (d) achieve a simple structure with a maximum number of items retained, (e) produce a Kaiser–Meyer–Olkin (KMO) measure of sampling adequacy greater than 0.5, and (f) yield a statistically significant (p < .001) Bartlett’s test of sphericity. We then calculated a total score for the CCS by averaging the items of the measure for each type of respondent (i.e., teachers and classrooms assistants) and each factor. Next, we examined whether there were systematic within-classroom differences between teachers and classroom assistants in these scores by comparing the teacher and assistant scores using paired-sample t tests. We then examined inter-rater agreement by calculating Pearson correlations among the raters (i.e., teachers and classroom assistants). Finally, we examined the convergent validity for the CCS by examining its correlations with program fidelity and burnout.
Results
Factor Analysis
Separate exploratory principal axis factor analyses were conducted to determine the latent structure of the teacher- and classroom assistant–rated CCS. The KMO measure of sampling adequacy (0.96 and 0.94, respectively) and Bartlett’s tests of sphericity (2778.76 and 2467.28, p < .001, respectively) indicated that the tests were significant for both the teacher- and classroom assistant rated CCS. The teacher- and classroom assistant–rated CCS yielded unitary factors. Overall, 68% and approximately 60% of the total variance were explained on the teacher- and classroom assistant–rated CCS, respectively. The factor loadings of each item on the teacher and classroom assistant version of the CCS are presented in Tables 2 and 3.
Item-Level Factor Analysis of Cohesion—Teacher Form (n = 149).
Item-Level Factor Analysis of Cohesion—Classroom Assistant Form (n = 153).
Descriptive Data
Teachers had an item-level mean of 4.16 (SD = 0.82) on the CCS, whereas classroom assistants had an item-level mean of 4.41 (SD = 0.72). Teachers from Year 3 generally reported low-to-average levels of burnout on the MBI, with item-level means of 2.09 (SD = 1.33) on emotional exhaustion, 0.72 (SD = 0.99) on depersonalization, and 5.24 (SD = 0.79) on personal accomplishment (where higher scores indicate more burnout for the emotional exhaustion and depersonalization scales, but less burnout for the personal accomplishment scale). Teachers from Year 3 had high fidelity to DTT and functional routine strategies with means of 0.75 (SD = 0.30) and 0.76 (SD = 0.23), respectively (indicating an average 80% pass rate of these program components across months), and moderate ratings of program fidelity for PRT with means of 0.58 (SD = 0.23; indicating an average 60% pass rate of PRT program components across months). Overall, the mean program fidelity was 0.72 (SD = 0.16; indicating an overall pass rate of 70%).
Within-Classroom Concordance of Multiple Informants’ Reports
Separate paired samples t tests were conducted to examine the within-classroom differences among teachers’ and classroom assistants’ scores of cohesion. Overall, classroom assistants assigned more positive ratings of cohesion as compared to teachers. There were statistically significant within-classroom differences between teachers’ and classroom assistants’ cohesion scores, t(143) = −2.84, p = .01.
Inter-Rater Agreement
Pearson correlations were conducted to examine agreement among teachers’ and classroom assistants’ reports of classroom cohesion (Table 4). Overall, there was strong agreement among raters, r = .45, p < .001.
Correlations of Teacher and Classroom Assistant Classroom Cohesion Scores, Fidelity, and Teacher-Rated MBI Scores.
Note. MBI = Maslach Burnout Inventory; CA = classroom assistant.
p < .05. **p < .01.
Convergent Validity
We examined convergent validity by examining associations among teachers’ and classroom assistants’ ratings of cohesion and teachers’ ratings of burnout and fidelity of program implementation for each component of STAR. Teacher- and classroom assistant–rated cohesion were not statistically significant with measures of fidelity or burnout (see Table 4).
Discussion
This study presents preliminary evidence for the factor structure, internal consistency, and construct validity of the CCS scales. Analyses revealed good internal consistency for both scales (.97 and .95 for the teacher and classroom assistant scales, respectively). We also found simple factor structures for both the teacher- and classroom assistant–rated scales, with one-factor solutions for both scales. These measures also demonstrated good variability, with a standard deviation of 0.82 for the teacher scale and 0.72 for the classroom assistant scale (both on a 5-point scale). Our findings regarding convergent validity were less conclusive. There were no associations between cohesion ratings and ratings of burnout and fidelity, suggesting that cohesion, burnout, and fidelity were three distinct constructs.
Findings regarding rater concordance suggested overall high within-classroom reliability. In general, teachers and classroom assistants’ reports of classroom cohesion were highly correlated; however, classroom assistants systematically assigned higher ratings of classroom cohesion than teachers. Whereas classroom assistants may have had the most positive impressions of classroom cohesion, an alternative explanation is that, given their role in the classroom as supervised by the teacher, these ratings were more vulnerable to positive respondent bias.
Our study had some limitations that highlight the unique challenges associated with studying organizational factors such as cohesion in special education classrooms. First, we were limited in our sample size. Our sample of 149 teachers and 154 classroom assistants limits the reliability of our findings. For example, our factor analyses violated the general rule of thumb of having 10 respondents per item (Kerlinger, 1986; although see Sapnas & Zeller, 2002 for a counterargument). We also may have missed associations that would have been statistically significant in a larger sample. Finally, because there are no known measures of classroom cohesion in the special education literature, we had no clear measure of classroom cohesion against which to test our measures for convergent validity. We relied on teacher reports of burnout and fidelity of program implementation, but because these are different constructs, it left the findings of the lack of associations between cohesion and these other constructs somewhat difficult to interpret. We also only examined program fidelity via taped classroom observations, which focused primarily on the quality of intervention implementation (i.e., whether teachers/staff could implement each intervention strategy) rather than the quantity (i.e., whether teachers/staff did implement the intervention strategies consistently and how often they were used). It is possible that we would have seen a stronger, more consistent cohesion–fidelity relationship had we looked more at the quantity of intervention strategies delivered, which may require more consistent, coordinated effort.
Despite these limitations, the results suggest the promise of a new tool to measure classroom cohesion. Based on our preliminary observations, this tool is relatively easy to administer with minimal burden to teachers and classroom assistants and fills an important need in improving our understanding of the working relationship among staff in special education settings. Studying cohesion and related areas of team functioning has led to interventions to improve team functioning and subsequent outcomes across a broad range of settings, including business and sports (Weinberg & McDermott, 2010) and health care (Buljac-Samardzic, Dekker-van Doorn, van Wijngaarden, & van Wijk, 2010) settings. Similarly, understanding classroom cohesion may have an important impact on various clinical and implementation outcomes in school settings and warrants further exploration.
Future studies should continue to explore and refine the CCS and examine important constructs from the organizational psychology literature (e.g., team-based measures of cohesion) that may be useful in providing convergent validity. Our study focused on classroom perceptions specific to task performance; however, cohesion is multidimensional. Broadly, cohesion typically is defined in two ways: (a) as an objective attitude of the group as a whole, and (b) as a function of each member’s subjective perceptions of his or her own standing in the group (Bollen & Hoyle, 1990). Although the objective and subjective domains of cohesion are expected to be somewhat related, we expect that different dimensions of cohesion may predict different things. Examining novel ways to measure cohesion and their relative contributions in predicting both clinical and implementation outcomes is an important area of study.
While considerable empirical work remains to be done to achieve a more complete understanding of classroom cohesion and its potential utility, this study offers a unique tool to examine classroom cohesion in special education settings. The development of the CCS is an important first step in identifying and understanding variables that may be necessary and sufficient for successful implementation of classroom-wide interventions. It is our hope that the tools adapted in this study can be used to improve program effectiveness and sustainability in school settings, ultimately leading to better outcomes for children with autism.
Footnotes
Acknowledgements
We thank the children, parents, schools, teachers, and classroom staff who participated in this study, and the research associates who contributed countless hours of assessments, intervention, supervision, and data collection: Erica Reisinger, Colleen Harker, Rukiya Wideman, Margaret Downey, Megan Carolan Tomkinson, Emily Cross, Laura Macmullen, Jessica Dodge, and Shawn Gilroy.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was drawn from the Philadelphia Autism Instructional Methods Study, which was supported by NIMH Grant 1R01MH083717-01A1 and IES Grant R324A08195. This study was also funded by the NIMH F31-MH088172 grant awarded to the first author and the FARFund and the Autism Science Foundation Early Career Award granted to the second author.
