Development of the School Mental Health Self-Efficacy Teacher Survey Using Rasch Analysis

Abstract

Given the important role that teachers play in supporting student mental health, it is critical teachers feel confident in their ability to fill such roles. To inform strategies intended to improve teacher confidence in supporting student mental health, a psychometrically sound tool assessing teacher school mental health self-efficacy is needed. The current article details the initial development and psychometric functioning of the school mental health self-efficacy teacher survey (SMH-SETS). A component of the development included Rasch analysis of pilot data to provide a psychometric appraisal of the SMH-SETS functioning and initial psychometric evidence. Results suggest the SMH-SETS exhibits strong psychometric properties and can be used to measure school mental health self-efficacy, track self-efficacy over time, and inform training and professional development.

Keywords

school mental health teacher self-efficacy Rasch analysis teacher training

There is a growing emphasis on addressing students’ mental health in schools as positive mental health has been associated with greater academic success, positive life outcomes, and reduced maladaptive behavior (Jaycox et al., 2009; National Academies of Sciences, Engineering, & Medicine, 2019). Recent research suggests that the mental health needs of youth are increasing (Weinberger et al., 2018) and, simultaneously, that students’ mental well-being improves when their teachers are involved in promoting and intervening in their social-emotional development (Rothì, Leavey, & Best, 2008; Taylor, Oberle, Durlak, & Weissberg, 2017). As a result, it has been suggested that schools, and teachers specifically, provide an ideal source of mental health support to address the increasing challenges facing students and their families (Reinke, Stormont, Herman, Puri, & Goel, 2011).

Teachers’ Role in School Mental Health

Teachers play a critical role in school mental health as they are the primary adult consistently interacting with students in a structured environment (Reinke et al., 2011). Thus, teachers are in a prime position to observe, support, and refer with regard to student mental health (Substance Abuse and Mental Health Services Administration, 2004; U.S. Department of Health and Human Services, 2019). In addition to the benefits students experience from teachers who promote social-emotional development and skills within the classroom (Taylor et al., 2017), teachers are also observers of student behavior and mental health and the primary professionals who refer students to school mental health services (Rothì et al., 2008). When a multitiered model of school mental health is used, teachers are expected to support a continuum from promoting positive mental health for all students to supporting students with more intensive mental health needs in their classrooms (Brown, Phillippo, Weston, & Rodger, 2017). In sum, teachers are fundamentally connected to school mental health and expected to not only teach students with mental health concerns, but also recognize student mental health need, respond to student needs, and promote positive mental health for all students in their classroom.

Given the robust role that teachers are expected to serve for school mental health, it is crucial to evaluate their confidence in their ability to serve in these positions. Many teachers often feel inadequately prepared to support student mental health and serve in these wide-ranging positions (Reinke et al., 2011). To complete these roles successfully, teachers must feel confident in their ability to meet students’ mental health needs and a measure of such self-efficacy is needed to inform training efforts and future research. To this end, the present study details the initial development of the school mental health self-efficacy teacher survey (SMH-SETS). A component of the SMH-SETS development included use of Rasch analysis to evaluate the instrument’s psychometric properties. In order to describe the rationale for the SMH-SETS, we will first introduce the concept of self-efficacy. Then, we will review past research on teacher mental health self-efficacy.

Self-Efficacy

The concept of teacher self-efficacy has been explored considerably. Self-efficacy was initially introduced through Albert Bandura’s social cognitive theory. Bandura proposed that the concept of self-efficacy could be defined as an individual’s belief in their abilities to successfully achieve goals or carry out actions (Bandura, 1997; 2006). Therefore, the higher the self-efficacy, the more likely a person will feel confident in their ability to perform specific tasks. This confidence in turn would influence actions and behavior (Bandura, 1997).

Teacher self-efficacy has been defined as “the extent to which the teacher believes he or she has the capacity to affect student performance” (Berman, McLaughlin, Bass, Pauly, & Zellman, 1977, p. 137). Multiple studies have examined teacher self-efficacy using the short (12 items) and long (24 items) forms of the teachers’ sense of self-efficacy scale (TSES; Chang & Engelhard, 2016; Tschannen-Moran & Hoy, 2001), which includes three efficacy factors: instructional strategies, classroom management, and student engagement. The TSES developers recommend that teacher self-efficacy measures should assess a teacher’s perceived competence to perform the range of expected teaching tasks (e.g., instruction, classroom management; Tschannen-Moran, Hoy, & Hoy, 1998). Teacher self-efficacy has been linked to teacher stress, job satisfaction, and burnout (Betoret, 2006; Brouwers & Tomic, 1999). Bandura (2006) and Klassen, Tze, Betts, and Gordon (2011) recommend attention be paid to domain specificity when measuring teacher self-efficacy. They suggest self-efficacy measures are most predictive of future behaviors when domains are targeted and narrow. In their review of teacher self-efficacy measures, Klassen et al. (2011) found a lack of self-efficacy measures specific to the domain of teacher mental health self-efficacy.

Teacher mental health self-efficacy

It is important to examine teacher confidence in their abilities to carry out expected school mental health roles to better inform training and professional development needs. For example, schools measuring teacher mental health self-efficacy could specify which topics teachers feel unprepared to handle and identify groups of teachers with particularly low self-efficacy for targeted coaching and professional development opportunities (Fortier, Lalonde, Venesoen, Legwegoh, & Short, 2017). For the purpose of this study and guided by Bandura’s definition, teacher mental health self-efficacy is defined as a teacher’s belief in their capacity to successfully support students’ mental health needs. Based on national guidance regarding the roles of teachers in supporting students’ mental health needs (Substance Abuse and Mental Health Services Administration, 2004; U.S. Department of Health and Human Services, 2019), we operationalized teacher mental health self-efficacy as teacher endorsement of feeling confident to: (a) teach students with mental health needs, (b) respond to mental health concerns, (c) recognize mental health concerns, and (d) promote positive mental health in the classroom.

Few studies have evaluated teacher mental health self-efficacy. Fortier et al. (2017) examined a Canadian public school district’s evaluation of a program to develop teachers’ mental health literacy with the 13-item Mental Health Literacy and Capacity survey. This instrument collected pre- and post-program implementation data on the educators’ awareness (five items), comfort (four items), and knowledge (four items) of school mental health. Although the survey developed by the school district documented teacher growth in all three target areas as a result of the program (Fortier et al., 2017), its psychometric properties have not yet been evaluated and it does not specifically measure teacher mental health self-efficacy as defined here and in the extant literature. Another study examined teacher self-efficacy before and after 126 teachers participated in a two-hour workshop on supporting student mental health (Woods & Rodger, 2014). The authors used the TSES short form to examine teachers’ knowledge of school mental health and teaching self-efficacy (Tschannen-Moran & Hoy, 2001). However, teachers’ self-efficacy in the specific domain of mental health as recommended by Klassen et al. (2011) was unexamined.

Previous teacher mental health training studies are limited due to the lack of a standard instrument that specifically measures teacher mental health self-efficacy. In order to assess teacher mental health self-efficacy, schools need published, standardized tools that can adequately measure the domain. Given previous research on the importance of teaching self-efficacy in predicting teacher and student behavior (Betoret, 2006; Brouwers & Tomic, 1999; Klassen et al., 2011), teachers are more likely to fulfill their robust role in school mental health if they perceive they can do so successfully.

As previously described, a teacher mental health self-efficacy measure would assist schools in identifying training and coaching support needs. However, we found limited assessments for measuring teacher mental health self-efficacy. As a result of these shortcomings, it was necessary to create a targeted instrument to assess teachers’ perceived capacity to successfully carry out their expected roles in supporting students’ mental health. The current study fills a gap in the literature by providing a needed instrument to assess teacher mental health self-efficacy.

Rasch Approaches to Measure Evaluation

To develop a psychometrically sound tool, we used Rasch theory and Rasch analysis techniques to test the preliminary psychometric functioning of the SMH-SETS (Bond & Fox, 2007; Boone, Staver, & Yale, 2014; Rasch, 1960; Wright & Masters, 1982). The Rasch model was conceptualized and developed by the mathematician Georg Rasch who noted particular problems with the use of raw test scores. For example, he noted that when a total score correct is computed for a student taking a test, there is no consideration that different test items have differing difficulty. Some of the problems with the analysis of raw rating scale data is that rating scales cannot be assumed to be linear, and survey items do not all have the same level of agreeability. Within mental health and educational research, the Rasch model has been used frequently to evaluate rating scale survey data (Areepattamannil & Khine, 2018; Chang & Engelhard, 2016).

The Current Study

Given the important role that teachers play in supporting student mental health, it is critical to evaluate teacher self-efficacy in successfully achieving this goal. However, there is a lack of existing published and psychometrically evaluated measures that assess teacher mental health self-efficacy. A comprehensive teacher mental health self-efficacy measure is needed to assess teacher preparedness and confidence to successfully engage in recommended roles for supporting students’ mental health. The purpose of the current study was to develop a psychometrically and theoretically sound instrument to measure teacher mental health self-efficacy.

Method

Sample

Invited participants included all preschool through 12th grade general and special education teachers (n = 204) in one rural Midwestern school district. The school district serves over 2000 students with 50% qualifying for free and reduced lunch and the majority of students in the district are Caucasian (National Center for Education Statistics, 2018). The district was concluding a three-year school mental health partnership with a local university when the survey was administered.

Measure: School Mental Health Self-Efficacy Teacher Survey Development

To guide the development of the SMH-SETs, we first reviewed self-efficacy theory and existing teacher self-efficacy scales (Bandura, 2006; Tschannen-Moran & Hoy, 2001), guidance on mental health literacy measures (Jorm, 2012; Wei, McGrath, Hayden, & Kutcher, 2015), and best practice guidance on teachers’ role in school mental health and social-emotional development (Rodger et al., 2014; Taylor et al., 2017; U.S. Department of Health and Human Services, 2019; Weston, Anderson-Butcher, & Burke, 2008). In their review of teacher self-efficacy measures, Tschannen-Moran et al. (1998) recommend developing teacher self-efficacy measures to assess teacher-perceived competence to perform the wide range of expected teaching tasks. Therefore, we considered the range of mental health activities that teachers are expected to perform. Based on this review, teacher mental health self-efficacy was operationally defined as a teacher’s belief in their capacity to successfully support students’ mental health needs (e.g., teach students with mental health needs, respond to mental health concerns, recognize mental health concerns, and promote positive mental health in the classroom). The operational definition of teacher mental health self-efficacy guided the development of the items to ensure each activity identified in the operational definition was assessed. In constructing items, we also consulted previous research and evaluations of teacher school mental health competence (Reinke et al., 2011; Rodger et al., 2014), as well as topics from school mental health trainings, such as Youth Mental Health First Aid (National Council for Behavioral Health, 2019) and Substance Abuse and Mental Health Services Administration (2004) training on Eliminating Barriers for Learning: Social and Emotional Factors that Enhance Secondary Education. As recommended by Bandura (2006), we developed items with graduations of difficulty to avoid ceiling effects.

Items were iteratively reviewed multiple times by a measurement expert for clarity and consistency in wording. Several steps were taken to ensure that the measure was theoretically sound. First, a focus group was informally conducted with nine school mental health experts to review 19 survey items in the initial item pool and provide preliminary feedback on item clarity and content validity. Items were removed or revised if unnecessary, unrelated, or unclear. For instance, redundant items were removed. The item development phase concluded with a total of 15 final items assessed on a 6-point Likert scale that addressed teachers’ confidence in their ability to successfully support students’ mental health.

Next, a panel of seven experts was selected to formally evaluate the face validity (the extent to which the measure appears to assess what is intended) and construct validity (the extent to which the measure adequately assesses the construct; Anastasi & Urbina, 1997) of the measure in addition to providing feedback on the readability and clarity of each item (Rubio, Berg-Weger, Tebb, Lee, & Rauch, 2003). The panel included two experts in school mental health theory, two in scale development, and three in school mental health practice and teacher training. The experts received a survey with an overview of the instrument and instructions to rate the usefulness of each item for assessing teacher school mental health confidence. Experts also rated item clarity and provided open-ended comments to improve readability.

Based on expert feedback, all items were rated as essential and item wording was revised to improve clarity. For instance, example actions were provided to items 14 and 15 (e.g., for the item “Promote the emotional skills of students in my classroom,” we added the example actions to “label emotions, model coping strategies, praise for managing emotions”). The revised survey was then sent to the same experts for a final review on the appropriateness of items. Final revisions were made based on this round of reviewer feedback. Lastly, the instrument was piloted with a group of seven preservice and 16 in-service teachers. At the conclusion of the measure development and evaluation phase, the SMH-SETS survey included 15 items, each with the stem “I feel confident in my ability to....” Each item could be rated using a 6-point Likert scale that ranged from strongly disagree (SD) to strongly agree (SA). A 6-point Likert scale was selected based on psychometric properties of scores from an existing self-efficacy instrument (Boone et al., 2014; Enochs & Riggs, 1990) and the ability to increase precision and variability in responses. The items are available in electronic supplemental materials. In the current sample, internal consistency was strong (Cronbach’s alpha = .91).

Procedure

First, university Institutional Review Board approval was received. Next, each building principal sent the survey link to all members of their school teaching staff. The survey first included definitions of mental health and mental health concern and then requested participants rate their confidence in their ability to successfully complete each mental health action with each survey item corresponding to one mental health action.

To prepare for analysis, the data collected were entered into an Excel spreadsheet. Each row was a respondent and each column was an item. Coding of 1 (SD), 2 (Disagree), 3 (Somewhat Disagree), 4 (Somewhat Agree), 5 (Agree), and 6 (SA) was used to indicate the answer of each respondent. For all items, an answer of more agreement meant the respondent indicated more self-efficacy.

Analyses

For the analysis of the SMH-SETS data, the Rasch rating scale model (Andrich, 1978) was used with the Winsteps Rasch analysis program (Linacre, 2019). Below, we present the Rasch rating scale model as described by Planinic, Boone, Susac, and Ivanjek (2019):

In a rating scale model, each item will have several rating scale categories. The probability of a person n endorsing category j over previous category (j−1) or being observed in category j of item i can be expressed in a Rasch–Andrich rating scale model as

ln (\frac{P_{nij}}{P_{ni (j - 1)}}) = B_{n} - D_{i} - F_{j}

(1)

where F_j is the Rasch–Andrich threshold (step calibration) or the point on the latent variable where the probability of person n being observed in category j of item i equals the probability of the same person being observed in category (j–1). F_j is estimated from the category frequency, and the difficulty of the item is now located at the point where the highest and the lowest categories are equally probable. (p. 4)

First, we examined unidimensionality of items with Outfit Mean Squares (MNSQ) and Infit MNSQ statistics and point measure correlations. MNSQ fit ranges between .5 to 1.5 provide evidence of unidimensionality (O’Connor et al., 2016). We also evaluated unidimensionality using point measure correlations. Li et al. (2018) suggest that point-measure correlations larger than .3 indicate items are measuring the same construct. We then examined unidimensionality with principle component analysis of residuals (PCAR; Linacre, 2009). The Winsteps Rasch program allows one to conduct a PCAR analysis to evaluate whether the level of noise predicted in the data is that which would be present if one construct is being measured with a set of instrument items. If there is more noise than expected, then that can be evidence of the presence of more than one trait. In such cases, follow-up analyses are conducted.

The initial step for a PCAR analysis is to evaluate the unexplained variance in the 1st contrast, which is reported in eigenvalue units. Linacre (2019) recommends that when the value of the unexplained variance in the 1st contrast is above 2.0, then there might be more than one trait present in the data. Linacre also recommends evaluating the correlation of person measures computed by using clusters of items identified through the PCAR analysis and then examining the level of the dissatenuated correlation between the person measures using each cluster of items. If there is a high correlation of person measures between clusters, this suggests that the items of each cluster are measuring the same trait.

We also evaluated the SMH-SETS using rating scale guidelines suggested by Linacre (1999, 2004) which include

10 observations in each rating scale category.

Regular observation distribution.

Average measures advance monotonically with category.

The outfit MNSQ is less than 2.0.

Orderly series of step calibrations that advance in monotonic way.

The distance or gap between response categories should be larger than 1.4 logits and less than 5.0 logits (O’Connor et al., 2016).

Our analysis also considered an evaluation of item difficulty, presented on a Wright map, to evaluate construct validity (e.g., Lunz, 2010). Last, we examined the person and item reliability and person and item separation. Item reliability is analogous to a Cronbach alpha with a minimum value of 0 and a maximum value of 1, a higher number being a better item reliability. Item and person separation statistics are analogous to the Fisher discriminant ratio in that they measure separation in root-mean-square standard error units (Linacre, 2019). Malec et al. (2007) have suggested the following critical values: item reliability value of .90, person reliability of .80, person separation of 2.0, and item separation of 4.0.

Results

A total of 204 teachers were sent the survey, and 171 teachers answered the majority of the survey items (84% response rate). Seven of 171 (4%) respondents did not provide all requested demographic data. The sample was majority Caucasian (98.2%), females (83.4%), with 1–5 years’ experience (59.8%). The supplemental materials provide additional detail on the sample.

Unidimensionality

The Infit and Outfit MNSQ item statistics range from .72 to 1.5 (Table 1), which is within the acceptable range. Additionally, all point measure correlation values are above the suggested .3 cutoff (range from .56 to .90). These provide evidence of unidimensionality (i.e., evidence of the 15 SMH-SETS items measuring one trait).

Table 1.

Rasch Item Measures and Statistics for the SMH-SETS.

Item	Total score	Total count	Item measure	Model SE	Infit MNSQ	Outfit MNSQ	PT. corr.
SE 1	750	169	.22	.11	.97	.98	.63
SE 2	841	169	−1.03	.12	1.23	1.20	.51
SE 3	769	169	−.01	.11	.84	.85	.69
SE 4	709	167	.62	.11	.79	.82	.69
SE 5	738	169	.37	.11	.89	.82	.66
SE 6	741	168	.28	.11	.72	.74	.71
SE 7	742	166	.14	.11	.80	.81	.68
SE 8	777	169	−.12	.11	1.11	1.06	.66
SE 9	774	167	−.20	.12	.85	.82	.67
SE 10	716	169	.62	.11	1.12	1.16	.69
SE 11	839	169	−1.00	.12	1.50	1.42	.54
SE 12	763	169	.06	.11	.85	.84	.69
SE 13	661	168	1.19	.10	1.24	1.27	.65
SE 14	822	169	−.74	.12	1.05	1.03	.59
SE 15	799	169	−.41	.12	1.12	1.09	.58

Note. SE = standard error, MNSQ = mean squares, PT. corr. = point measure correlation, and SMH-SETS = school mental health self-efficacy teacher survey. Item measures and model standard errors are presented on the Rasch logit scale. Higher item measures indicate less confidence of respondents.

The PCAR analysis of unidimensionality indicated an unexplained variance in the 1st contrast of 2.30 (eigenvalue units) was observed. This is above the cutoff that has been proposed, but not substantially higher than 2.0. Follow-up analyses evaluated the “clusters” of items which were identified in the PCAR. The authors reviewed the text of items in each cluster (Cluster 1 SE4, SE6; Cluster 2 SE1, SE2, SE3, SE5, SE7; and Cluster 3 SE8, SE9, SE10, SE11, SE12, SE13, SE14, SE15) and concluded that although the items marked different parts of the trait, the items all involved one trait.

We also examined the level of disattenuated correlation between the person measures using each cluster of items. The correlations of the person measures computed with each cluster of items were as follows: Clusters 1 and 2: r = 1.00, Clusters 1 and 3: r = .92, and Cluster 2 and 3: r = .91. The high correlations observed between person measures computed (a) only with cluster 1 items, (b) only with cluster 2 items, and (c) only with cluster 3 items suggest that the items (regardless of clusters) defined one single trait. Given the fit, PCAR, and point biserial statistics from the analysis, we feel that there is evidence that the set of SMH-SETS items measure one trait.

Evaluation of Linacre’s Six Rating Scale Guidelines

As presented in Table 2, each rating scale category had a frequency count greater than 10 (guideline 1). We found a regular observation distribution as illustrated by increase in rating scale categories selected and decrease in category counts (guideline 2). For guideline 3, we observed that the average measure increases from the rating scale step of D to that of SA. The average of SD is .01 higher than the average for D. As such, a small percentage (only 10 answers, 6%) of the entire data set was used to compute the SD average. This small deviation from what is suggested to be observed is insignificant given the nearly negligible difference between the average of SD and average of D. Thus, we do find that the SMH-SETS abides by the criteria that average measures advance monotonically with response option (guideline 3). Regarding guideline 4, the average Outfit MNSQ is below 2.0 for all categories. We also observe an orderly series of step calibrations that advance in monotonic way (guideline 5).

Table 2

Rasch Item Measures and Statistics for the SMH-SETS.

Category label	Observed count	Observed %	Observed average (Logits)	Infit MNSQ	Outfit MNSQ	Andrich threshold
Strongly disagree	10	0	−.52	1.49	1.67	None
Disagree	70	3	−.53	1.11	1.21	−2.83
Somewhat disagree	242	10	−.02	.91	.92	−1.53
Somewhat agree	816	32	.96	.91	.92	−.69
Agree	1027	41	2.03	.86	.88	1.25
Strongly agree	361	14	3.24	1.09	1.09	3.80

Note. MNSQ = mean squares and SMH-SETS = school mental health self-efficacy teacher survey.

Last, from Figure 1, we can compute the gaps between categories to evaluate the distance between categories using the Andrich thresholds (guideline 6). From left to right of Figure 1, the gaps are 1.3 (−2.83 to −1.53), .84 (−1.53 to −.69), 1.94 (−.69 to 1.25), and 2.55 (1.25–3.80). In all cases, the gaps between response options are less than 5.0 logits. In two cases, our gaps are less than 1.4 logits, with one gap being near 1.4 logits. Figure 1 also shows that each rating scale category has an individual peak, which is expected for well-functioning rating scale.

Figure 1.

Category probability curve. Note. Curves showing the operation of each rating scale step. The location at which each top curve intersects the next top curve is the Andrich threshold (the threshold measure and the step calibration). A vertical peak is observed for each rating scale step. SD = strongly disagree, D = disagree, SWD = somewhat disagree, SWA = somewhat agree, A = agree, and SA = strongly agree.

Wright Map

Review of the item entry table in Table 1 suggests that the 15 items of the instrument cover a range of item difficulty from −1.03 logits to 1.19 logits. In Figure 2, we present a Wright Map in which the items of the survey are plotted on a vertical line representing the logit scale of the survey instrument. As can be seen, there is a good distribution of items along the trait. Items plotted at the base of the Wright Map are items which are easier to agree with than items plotted toward the top of the Wright Map.

Figure 2.

Wright map. Note. Items are organized by item measure. Ordering and spacing of items describes the confidence profile of respondents.

Person Reliability, Item Reliability

In this study, all values exceeded target values for person reliability (.91) and item reliability (.96). Thus, from an item and person reliability perspective, the SMH-SETS is functioning well. In our survey, a person separation of 2.92 and item separation is 4.90 was observed.

Discussion

The current study describes the development and initial analysis of the SMH-SETS. Increasingly, teachers are expected to serve an expanded role for a continuum of mental health supports (Substance Abuse and Mental Health Services Administration, 2004). The SMH-SETS was developed in alignment with federal guidelines and recommendations for the roles and competencies of teachers in supporting students’ mental health (National Academies of Sciences, Engineering, & Medicine, 2019; U.S. Department of Health and Human Services, 2019; Weston et al., 2008). That is, the SMH-SETS was designed to help schools evaluate teachers’ perceived confidence to successfully perform the recommended roles in expanded models of school mental health.

The SMH-SETS instrument was designed following a systematic process informed by experts and now examined psychometrically using Rasch analysis to evaluate the scale’s functioning. Unidimensionality was evaluated utilizing Rasch fit statistics, as well as PCAR and point measure correlations. Notably, these tests of the measure’s dimensionality all suggested the items lie on one trait as intended during survey development. Therefore, we recommend using the SMH-SETS as a unidimensional scale.

In this study, we also evaluated the functioning of the rating scale utilizing guidance provided by Linacre (2004). The current study’s data set abided by the guidelines with only two exceptions. Guideline 3 (i.e., average measures advance monotonically with response option), was not observed for every response option. This deviation was most likely due to sample size. As additional data are collected, we will continue to monitor the manner in which average measures advance with category. Also, the upper limit of guideline 6 (i.e., less than 5.0 logits) was met, but two advances in step difficulties were below the lower limit cutoff of 1.4 logits and another was near the cutoff. In summary, the guidelines proposed by Linacre for a Rasch analysis of rating scale data were generally met with a few exceptions.

Perhaps one of the most useful aspects of conducting a Rasch analysis is the creation and analysis of a Wright Map. The items of the Wright Map do indeed mark a range of the trait with few gaps, which is what one would like to see for a measurement instrument. The construct validity of an instrument can also be evaluated by comparing the ordering of items of the Wright Map to that which would be predicted from previous research. Based on the Wright map, teachers’ confidence varies based on the mental health role and indicated greater confidence for referring students to mental health providers (Q11) and promoting positive social-emotional skills (Q14 and Q15) compared to meeting students’ mental health needs in the classroom (Q4, Q5, and Q10). Therefore, teachers were more likely to feel confident referring students with mental health concerns than providing academic instruction to students with externalizing and internalizing mental health concerns. This is consistent with past research that teachers may not view supporting students with mental health needs in their classroom as their role but more likely to see it as the role of a school mental health professional (Phillippo & Blosser, 2017; Reinke et al., 2011).

The Wright map indicates that teachers were most confident in recognizing externalizing mental health concerns (Q2) but reported less confidence in recognizing internalizing mental health concerns (Q1), which is affirmed in previous research that teachers are more likely to recognize externalizing “squeaky wheel” concerns than recognizing internalizing concerns (Eklund & Dowdy, 2014; Reinke et al., 2011). Finally, relative to the other survey items, teachers indicated less confidence discussing mental health concerns with parents (Q13). This corresponds with previous findings that teachers do not feel comfortable talking with parents about mental health (Fortier et al., 2017); yet, it is expected that teachers partner with school mental health professionals to share concerns and collaborate with parents (Substance Abuse and Mental Health Services Administration, 2004). Given the varied responses, teachers in this sample did not express confidence in their ability to serve all mental health roles.

This work represents the initial development and analysis of an instrument of use to researchers and practitioners in the field of school mental health. Psychometric analysis suggests the SMH-SETS exhibits characteristics of a well-functioning scale and the Wright map provides initial construct validity evidence.

Implications for Practice

The SMH-SETS fills a gap in the literature and provides a practical tool for schools. As teacher roles broaden to include a wide-range of responsibilities in school mental health, tools are needed to evaluate teacher confidence and gaps in self-efficacy. The National Academies of Sciences, Engineering, and Medicine (2019) recommend developing targeted trainings for teachers to increase capacity and skills for mental health prevention. The SMH-SETS provides a method for schools to determine if, when, and for whom such training may be needed based on results of a psychometrically sound tool. The SMH-SETS can then also be used to monitor improvement and training needs over time. The current study provides a tool to meet these needs and sets the stage for future research to continue examining its psychometric properties.

Based on the finding that the items lie on one trait, we recommend using a unidimensional scale (i.e., all items should be pooled for one overall measure). For scoring, Rasch theory and analysis suggests using a “raw score to Rasch measure conversion table” (Boone et al., 2014), which would provide an overall indication of mental health self-efficacy on a linear scale. Given this was a preliminary evaluation of SMH-SETS with a pilot sample, we do not consider the “raw score to Rasch measure table” computed with these data to be definitive. In the coming year, we plan to assemble additional data from more respondents and then compute the conversion table. Those interested in computing the Rasch measures can freely contact the first author for guidance on scoring and interpreting results prior to our completion of the conversion table.

Analysis of the Wright map results suggests teachers may need additional support and training on supporting students with mental health needs such as responding to students with suicidal thoughts, discussing mental health concerns with parents/guardians, and providing academic instruction to students with internalizing and externalizing concerns. School mental health professional development should follow best practices of explicit training in evidence-based practices and intensive and ongoing professional development supports (State, Simonsen, Hirn, & Wills, 2019). Teacher preparation programs should also consider implementing preservice training opportunities and courses to proactively improve preservice teacher confidence for successfully serving expanded school mental health roles (Phillippo & Blosser, 2017; Weston et al., 2008).

Limitations and Directions for Future Research

The current study should be considered in the context of limitations. First, the study used a pilot sample (n = 171) from one school district, which limits generalizability. In the future, the SMH-SETS should be administered with a large, nationally representative sample of teachers. This would allow one to generalize with confidence. Second, criterion validity of the SMH-SETS should be evaluated. Future studies should continue to examine the psychometric properties, such as criterion validity, of this and other school mental health self-efficacy measures.

Future research is needed with a more diverse and larger sample to continue testing the measure’s psychometric properties. For example, we will need to evaluate the appropriateness of all items, such as those on suicidality, for preschool teachers and compare differential item functioning (Finger et al., 2012) by teaching setting (e.g., preschool, elementary, middle school, and high school) to examine the manner in which an instrument functions for different samples of respondents. Furthermore, as additional data are collected, we plan to create a score conversion table that converts a raw score to a Rasch person measure score. A conversion table can increase interpretability for researchers and school administrators interested in using the measure. Last, although results support the 6-point rating scale and retaining all items, there is some overlap on the Wright Map and continued testing should consider if all 6-points and all items are needed.

Conclusion

It is critical that teachers feel confident and prepared to serve crucial roles in school mental health. Teachers are most likely to fulfill such roles if they perceive they can do so successfully. However, without a psychometrically sound measure, teachers’ level of mental health self-efficacy and domain-specific training needs are unknown. To best identify training needs, a brief measure of teacher mental health self-efficacy was developed and the current study tested its psychometric functioning in a pilot sample using Rasch analyses. Results suggest that the SMH-SETS is a psychometrically sound tool that can evaluate teachers’ confidence to successfully serve in recommended school mental health roles. The SMH-SETS can be used to inform decisions about school mental health policy, teacher professional development, and related initiatives. Based on the important role that teachers play for students’ mental health, schools should consider using this tool.

Supplemental Material

Supplemental_Tables – Supplemental Material for Development of the School Mental Health Self-Efficacy Teacher Survey Using Rasch Analysis

Supplemental Material, Supplemental_Tables for Development of the School Mental Health Self-Efficacy Teacher Survey Using Rasch Analysis by Kristy L. Brann, William J. Boone, Joni W. Splett, Courtney Clemons and Sarah L. Bidwell in Journal of Psychoeducational Assessment

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Kristy L. Brann

Supplemental Material

Supplemental material for this article is available online.

References

Anastasi

Urbina

(1997). Psychological testing (7th ed.). Upper Saddle River, NJ: Prentice Hall/Pearson Education.

Andrich

(1978). A rating scale formulation for ordered response categories. Psychometrika, 43, 561-573. doi:10.1007/BF02293814

Areepattamannil

Khine

M. S.

(2018). Evaluating the psychometric properties of the original grit scale using Rasch analysis in an Arab adolescent sample. Journal of Psychoeducational Assessment, 36(8), 856-862.

Bandura

(1997). Self-efficacy: The exercise of control. New York: Freeman.

Bandura

(2006). Guide for constructing self-efficacy scales. In Pajares

Urdan

(Eds.), Self-efficacy beliefs of adolescents (pp. 307-337). Greenwich, CT: Information Age.

Berman

McLaughlin

Bass

Pauly

Zellman

(1977). Federal programs supporting educational change. In Factors affecting implementation and continuation (Vol. VII). Santa Monica, CA: The Rand Corp.

Betoret

F. D.

(2006). Stressors, self-efficacy, coping resources, and burnout among secondary school teachers in Spain. Educational Psychology, 26, 519-539. doi:10.1080/01443410500342492

Bond

T. G.

Fox

C. M.

(2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Boone

W. J.

Staver

J. R.

Yale

M. S.

(2014). Rasch analysis in the human sciences. New York: Springer.

10.

Brouwers

Tomic

(1999). Teacher burnout, perceived self-efficacy in classroom management, and student disruptive behaviour in secondary education. Curriculum and Teaching, 14, 7-26. doi:10.7459/ct/14.2.02

11.

Brown

E. L.

Phillippo

K. L.

Weston

Rodger

(2017). Preparing the teaching workforce for school mental health promotion. Advances in School Mental Health, 10, 1-4. doi:10.1080/1754730X.2016.1257035

12.

Chang

M.-L.

Engelhard

Jr. (2016). Examining the teachers’ sense of efficacy scale at the item level with Rasch measurement model. Journal of Psychoeducational Assessment, 34, 177-191. doi:10.1177/0734282915593835

13.

Eklund

Dowdy

(2014). Screening for behavioral and emotional risk versus traditional school identification methods. School Mental Health, 6, 40-49. doi:10.1007/s12310-013-9109-1

14.

Enochs

L. G.

Riggs

I. M.

(1990). Further development of an elementary science teaching efficacy belief instrument: A preservice elementary scale. School Science and Mathematics, 90, 694-706.

15.

Finger

R. P.

Fenwick

Pesudovs

Marella

Lamoureux

E. L.

Holz

F. G.

(2012). Rasch analysis reveals problems with multiplicative scoring in the macular disease quality of life questionnaire. Ophthalmology, 119, 2351-2357. doi:10.1016/j.ophtha.2012.05.031

16.

Fortier

Lalonde

Venesoen

Legwegoh

A. F.

Short

K. H.

(2017). Educator mental health literacy to scale: From theory to practice. Advances in School Mental Health Promotion, 10, 65-84. doi:10.1080/1754730X.2016.1252276

17.

Jaycox

L. H.

Stein

B. D.

Paddock

Miles

J. N. V.

Chandra

Meredith

L. S.

, … Burnam

M. A.

(2009). Impact of teen depression on academic, social, and physical functioning. Pediatrics, 124(4), E596-E605. doi:10.1542/peds.2008-3348

18.

Jorm

A. F.

(2012). Mental health literacy: Empowering the community to take action for better mental health. American Psychologist, 67, 231-243. doi:10.1037/a0025957

19.

Klassen

R. M.

Tze

V. M. C.

Betts

S. M.

Gordon

K. A.

(2011). Teacher efficacy research 1998-2009: Signs of progress or unfulfilled promise? Educational Psychology Review, 23(1), 21-43. doi:10.1007/s10648-010-9141-8

20.

C.-Y.

Romero

Bonilha

H. S.

Simpson

K. N.

Simpson

A. N.

Hong

Velozo

C. A.

(2018). Linking existing instruments to develop an activity of daily living item bank. Evaluation & the Health Professions, 41, 25-43. doi:10.1177/0163278716676873

21.

Linacre

J. M.

(1999). Investigating rating scale category utility. Journal of Outcome Measurement, 3(2), 103-122.

22.

Linacre

J. M.

(2004). Optimizing rating scale category effectiveness. In Smith

E. V.

Smith

R. M.

(Eds.), Introduction to Rasch measurement: Theory, model and application (pp. 258-278). Maple Grove, MN: JAM Press.

23.

Linacre

J. M.

(2009). Unidimensional models in a multidimensional world. Rasch Measurement Transactions, 23(2), 1209.

24.

Linacre

J. M.

(2019). Winsteps® Rasch measurement computer program. Retrieved from https://www.Winsteps.com

25.

Lunz

M. E.

(2010). Using the very useful Wright map. Retrieved from http://www.rasch.org/mra/mra-01-10.htm

26.

Malec

Torsher

Dunn

Wiegmann

Arnold

Brown

Phatak

(2007). The mayo high performance teamwork scale: Reliability and validity for evaluating key crew resource management skills. Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare, 2, 4-10. doi:10.1097/SIH.0b013e31802b68ee

27.

National Academies of Sciences, Engineering, & Medicine . (2019). Fostering healthy mental, emotional, and behavioral development in children and youth: A national agenda. Washington, DC: The National Academies Press. doi:10.17226/25201

28.

National Center for Education Statistics . (2018). Search for public schools. Retrieved March 24, 2018 from https://nces.ed.gov/ccd/schoolsearch/

29.

National Council for Behavioral Health . (2019). Youth mental health first aid. Retrieved on September 10, 2019 from https://www.mentalhealthfirstaid.org/take-a-course/course-types/youth/

30.

O’Connor

J. P.

Penney

Alfrey

Phillipson

S. N.

Jeanes

(2016). The development of the stereotypical attitudes in HPE scale. Australian Journal of Teacher Education, 41(7), 70-87. Retrieved from http://ro.ecu.edu.au/ajte/vol41/iss7/5

31.

Phillippo

Blosser

(2017). Stable roles, changed skills: Teacher candidate responses to instruction about adolescent psychosocial support practices. Advances in School Mental Health Promotion, 10, 5-25. doi:10.1080/1754730X.2016.1252684

32.

Planinic

Boone

W. J.

Susac

Ivanjek

(2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15, 1-14. doi:10.1103/PhysRevPhysEducRes.15.020111

33.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press.

34.

Reinke

W. M.

Stormont

Herman

K. C.

Puri

Goel

(2011). Supporting children’s mental health in schools: Teacher perceptions of needs, roles, and barriers. School Psychology Quarterly, 26(1), 1-13. doi:10.1037/a0022714

35.

Rodger

Hibbert

Leschied

Pickel

Stepien

Atkins

, … Vandermeer

(2014). Shaping a mental health curriculum for canada’s teacher education program: Rationale and brief overview. Physical & Health Education Journal, 80(3), 28-29.

36.

Rothì

D. M.

Leavey

Best

(2008). On the front-line: Teachers as active observers of pupils’ mental health. Teaching and Teacher Education, 24(5), 1217-1231. doi:10.1016/j.tate.2007.09.011

37.

Rubio

D. M.

Berg-Weger

Tebb

S. S.

Lee

E. S.

Rauch

(2003). Objectifying content validity: Conducting a content validity study in social work research. Social Work Research, 27(2), 94-104. doi:10.1093/swr/27.2.94

38.

State

T. M.

Simonsen

Hirn

R. G.

Wills

(2019). Bridging the research-to-practice gap through effective professional development for teachers working with students with emotional and behavioral disorders. Behavioral Disorders, 44(2), 107-116. doi:10.1177/0198742918816447

39.

Substance Abuse and Mental Health Services Administration . (2004). Eliminating barriers for learning: Social and emotional factors that enhance secondary education. Rockville, MD, U.S.: Author.

40.

Taylor

R. D.

Oberle

Durlak

J. A.

Weissberg

R. P.

(2017). Promoting positive youth development through school-based social and emotional learning interventions: A meta-analysis of follow-up effects. Child Development, 88, 1156-1171. doi:10.1111/cdev.12864

41.

Tschannen-Moran

Hoy

A. W.

(2001). Teacher efficacy: Capturing an elusive construct. Teaching and Teacher Education, 17(7), 783-805. doi:10.1016/S0742-051X(01)00036-1

42.

Tschannen-Moran

Hoy

A. W.

Hoy

W. K.

(1998). Teacher efficacy: Its meaning and measure. Review of Educational Research, 68, 202-248.

43.

U.S. Department of Health and Human Services . (2019). How adults can support adolescent mental health. Retrieved on September 3, 2019 from https://www.hhs.gov/ash/oah/adolescent-development/mental-health/how-adults-can-support-adolescent-mental-health/index.html

44.

Wei

McGrath

P. J.

Hayden

Kutcher

(2015). Mental health literacy measures evaluating knowledge, attitudes and help-seeking: A scoping review. BMC Psychiatry, 15, 291-310. doi:10.1186/s12888-015-0681-9

45.

Weinberger

A. H.

Gbedemah

Martinez

A. M.

Nash

Galea

Goodwin

R. D.

(2018). Trends in depression prevalence in the USA from 2005 to 2015: Widening disparities in vulnerable groups. Psychological Medicine, 48(8), 1308-1315. doi:10.1017/S0033291717002781

46.

Weston

K. J.

Anderson-Butcher

Burke

R. W.

(2008). Developing a comprehensive curriculum framework for teacher preparation in expanded school mental health. Advances in School Mental Health Promotion. 4(1), 25-41. doi:10.1080/1754730X.2008.9715737

47.

Woods

Rodger

(2014). Evaluating mental health training for teachers: Identifying and supporting students with mental health challenges (Electronic Thesis and Dissertation Repository). Retrieved from https://ir.lib.uwo.ca/etd/2022/?utm_source=ir.lib.uwo.ca%2Fetd%2F2022&utm_medium=PDF&utm_campaign=PDFCoverPages

48.

Wright

B. D.

Masters

(1982). Rating scale analysis. Chicago, IL: MESA Press.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB