Multi-Tiered System of Supports—Interventionist Beliefs Scale: Development and Initial Validation With School Counselors

Abstract

This article describes the development of the Multi-Tiered System of Supports-Interventionist Beliefs Scale (MTSS-IBS). The introduction reviews the importance between practitioner beliefs and MTSS success. Because there is no measure to assess the MTSS beliefs of school interventionists for the success of all students, we used a principal component analysis and a parallel analysis on MTSS-IBS responses from 379 school counselors who worked as MTSS interventionists. These analyses indicated a four-component structure, and subsequent internal consistency and construct validity analyses provided initial evidence of MTSS-IBS psychometric adequacy. We discuss applications of the MTSS-IBS in research and practice for enhancing MTSS implementation.

Keywords

MTSS interventionist beliefs school counselors attitudinal measurement

Introduction

School counselors often need to use tiered service delivery to effectively meet their students' increasingly diverse academic and behavior needs. The reality of school counselors experiencing high student-to-counselor ratios means that tiered supports are one of the most efficient ways for counselors to effectively meet the needs of all students (Goodman-Scott & Ockerman, 2019). A multi-tiered system of supports (MTSS) represents a data-driven, prevention-based framework for effectively delivering and evaluating academic, social, emotional, and behavioral interventions (McIntosh & Goodman, 2016). MTSS is typically a three-tiered, school-wide model of prevention and intervention. Tier 1 services focus on using empirically-based, class-wide instruction designed to meet the needs of 80% of all learners. Tier 2 services focus on small-group interventions for 15% of students whose needs are better addressed with supplemental instruction. And Tier 3, intervention is individually-tailored for approximately 5% of students requiring highly intensive instruction or remediation of academic and behavioral needs. Overall, the goal of MTSS is to create a system of prevention, promoting academic success and prosocial behavior for all students (Burns et al., 2016).

The U.S. Department of Education (2014) endorsed tiered systems of support, such as MTSS, as effective programs for fostering positive school climates. Additionally, in a brief authored by several professional educational organizations representing school counselors, school psychologists, social workers, resource officers, and principals, MTSS was endorsed as a best practice for creating safe, successful schools (Cowan et al., 2013). MTSS research demonstrates positive student outcomes in areas like state-wide assessments (King et al., 2016), literacy (Neitzel et al., 2021), mathematics (Weisenburgh-Snyder et al., 2015), and English-language learning (Gonzales & Hughes, 2021).

MTSS and School Counseling

Although many programs fall under the MTSS umbrella, two of the most widely used and recognized versions are Response to Intervention (RTI) and Positive Behavioral Interventions and Support (PBIS). Educators and interventionists such as school counselors and school psychologists use RTI to reduce referrals for special education services. RTI enables educators and interventionists to provide students with necessary support while allowing them to remain in general education (Gersten & Dimino, 2006). PBIS has been described as a function of RTI as it applies RTI principles to improving students’ behavior outcomes (McIntosh & Goodman, 2016). PBIS uses student data to monitor student referrals and PBIS implementation and ensures fidelity with the MTSS tenets (Simonsen & Sugai, 2019). PBIS has been shown to positively impact a variety of educational outcomes, including improving academic outcomes (Horner et al., 2009), decreasing student discipline referrals (Bradshaw et al., 2010), and improving school climate outcomes (Bradshaw et al., 2021).

In a 2019 position statement, the American School Counselor Association (ASCA) stated that school counselors are stakeholders in the development and implementation of MTSS and should align their work with MTSS through the delivery of a comprehensive school counseling program (CSCP). Research has shown that MTSS implementation is improved when school counselors take leadership roles in implementing MTSS. Indeed, Cressey et al. (2014) reported increased use of student data to set school goals with school counselor involvement. Additionally, improved systems for teaching and monitoring student behavior (Donohue et al., 2015) and reduced disproportionality in data review and intervention implementation (Cressey et al., 2014) were found with school counselor leadership in MTSS.

CSCPs and MTSS are data-driven, proactive, and multi-tiered approaches to meeting students' needs efficiently. Researchers have found improved student outcomes when school counselors aligned their CSCP with MTSS (Betters-Bubon & Donohue, 2016). For example, Betters-Bubon and Donohue (2016) reported that school counselors collaborating with teachers and other stakeholders in implementing and integrating MTSS into their CSCP reported improved student academic achievement, reduced disciplinary problems, and enhanced student social-emotional development. Furthermore, alignment between MTSS and CSCPs leads to more efficient use of school counselors’ time and increased leadership capacity (Betters-Bubon & Donohue, 2016; Goodman-Scott & Ockerman, 2019). Another study by Scarborough et al. (2015) found that students who received counseling services as part of a comprehensive school counseling program aligned with MTSS had higher attendance rates and fewer behavioral incidents than those who did not receive such services.

Moreover, Sink and colleagues (2015) conducted a literature review examining how CSCPs can align with MTSS—they found that when CSCPs are integrated with MTSS, students' academic and behavioral outcomes are positively impacted. Indeed, Cusumano and Decker (2019) presented a model of collaborative consultation for school counselors to integrate into MTSS. The model emphasized school counselors working collaboratively with other school stakeholders when implementing MTSS and highlighted the potential benefits of using this model, such as improved communication, increased efficiency, and better student outcomes.

Importance of Beliefs for MTSS Fidelity and Implementation

Research has established a link between educator beliefs and improved intervention knowledge and practice (Brownell et al., 2010; Lauterbach et al., 2018), and it is clear that interventionist beliefs about interventions inform their willingness to implement new practices (Castillo et al., 2015; 2018). Educators will implement new practices (such as MTSS) when they understand the need for the new approach and believe they have the skills and/or support to implement the new practices (Castillo et al., 2010). Additionally, there is a link between the theory of planned behavior and an individual’s beliefs, intentions, and behavior; for example, Sharrat and Fullan (2009) reported that educational leaders’ beliefs played a critical role in implementing RTI. Multiple studies show that educators' beliefs about students and learning styles affect the extent to which educators adopt new practices (e.g., see Brownell et al., 2010). In the school counseling literature, school counselors’ beliefs have been linked to their use of data (Hatch & Chen-Hayes, 2008), implementation of effective counseling (Larson & Daniels, 1998), accountability practices (Topdemir, 2013), and implementation of components of a CSCP (Young & Kaffenberger, 2011). The importance of counselor beliefs for effective intervention implementation makes understanding school counselors’ beliefs about MTSS a critical endeavor.

Existing Scales on MTSS Beliefs

RTI is a type of MTSS with a strong focus on academic remediation and intervention (Burns et al., 2016). The Beliefs on RTI Survey was initially developed by Castillo et al. (2010) in a pilot program. Items for the original 27-item survey were developed from a literature review, an RTI implementation conceptual framework, and specific RTI practices in the literature. The authors provided evidence for validity using data from an expert panel. Psychometric properties were investigated by administering the instrument to 2430 and 2443 educators across 62 and 68 elementary schools, respectively, over two semesters. Educators included teachers, school counselors, school psychologists, instructional coaches, and other staff members. In a follow-up study, Castillo et al. (2015) developed the 16-item RTI Beliefs Scale to measure the extent to which interventionists’ beliefs were consistent with the core tenets of RTI. Single-level factor analyses and confirmatory factor analyses resulted in the retention of 16 items across three latent factors—Academic Abilities and Performance of Students with Disabilities, Data-Based Decision-Making, and Functions of Core and Supplemental Instruction—that explained belief statements at the educator and school level. Subsequent research found that positive beliefs about data-based decision-making for academic problems and problem-solving skills were associated with stronger RTI implementation (Castillo et al., 2018).

The North Carolina MTSS Beliefs Survey ([NC MTSS Beliefs Survey]; N.C. Department of Public Instruction, 2015) was adapted from the RTI Beliefs Scale to measure educator beliefs about problem-solving, student learning, and expectations for instructional effectiveness. The RTI Beliefs Scale was modified to include language aligned with North Carolina’s MTSS Model, and one item was added to the NC MTSS Beliefs Survey. The NC MTSS Beliefs Survey is intended for completion by an entire school staff implementing MTSS. The instrument was adapted to help school leadership ensure alignment between professional development opportunities and staff professional needs. Both the Beliefs on RTI Survey and the NC MTSS Beliefs Survey focus on beliefs around the needs of students with learning disabilities or emotional/behavioral disorders instead of all students' needs.

Current Study: Examination of MTSS Beliefs with Interventionists

Interventionists from various disciplines increasingly assume leadership in designing, implementing, and evaluating MTSS interventions. ASCA (2019) stated that school counselors are stakeholders in the development and implementation of MTSS and should align their work with MTSS through the delivery of a CSCP. Similarly, the National Association of School Psychologists (NASP, 2020), outlines prerequisite skills for establishing a successful MTSS. These are operationalized through ten practice domains addressing areas such as data-based decision-making, consultation and collaboration, academic/instructional interventions, mental health interventions, and school-wide practices promoting learning. Relatedly, the policy statement of the School Social Work Association of America ([SSWAA]; Lucio et al., 2021) defines the role of school social workers as including the implementation of multi-tiered programs, monitoring of student progress, and evaluation of intervention service effectiveness.

Given the expanding role of interventionists, this study developed and validated a new quantitative measure of interventionist beliefs about MTSS implementation, the MTSS-Interventionist Beliefs Scale (MTSS-IBS). Existing measures on beliefs—RTI Beliefs Scale and NC MTSS Beliefs Survey—both focus on MTSS and the needs of students with learning disabilities or emotional/behavioral disorders as opposed to the needs of all students. The MTSS-IBS aims to help interventionists (e.g., school counselors, school psychologists, school social workers) and school and district leaders assess beliefs about MTSS implementation for adequately meeting the needs of all students. The research questions guiding the study are: What is the factor structure of the MTSS-IBS? Do interventionist beliefs about MTSS predict MTSS implementation acceptability? Do MTSS beliefs predict MTSS implementation sustainability?

Method

Participants

The study participants were 379 school counselors working as interventionists within an MTSS framework from across the United States. The sample was predominantly female (91%) and White (76.80%), and most (83.90%) held a master’s as their highest degree, which is representative of the school counseling population in the United States (ASCA, 2021). Participants were experienced with an average of 12.89 years working in schools, although the modal response (63.90%) regarding the number of graduate courses participants had on MTSS was 0. Regarding school characteristics, a third of all participants (34.30%) worked in elementary schools. Almost half of the interventionists (45.90%) worked in suburban schools. Table 1 presents sample descriptive statistics.

Table 1.

Demographic Characteristics of School Counselors (n = 379) and Their Schools.

Demographic	n	%	M	SD
Participant characteristics
Gender
Female	345	91.00
Male	32	8.40
Other	1	.30
Ethnicity
Asian	10	2.60
Black	40	10.60
White	291	76.80
Hispanic-Latino	25	6.60
Native American/Alaska Native	3	.80
Native Hawaiian/Other Pacific Islander	1	.30
Multiracial	3	.80
Other	4	1.10
Age
Under 30 years	48	12.70
30-39	114	30.20
40-49	108	28.50
50-59	74	19.50
60+	34	9.00
Highest degree earned
Bachelors	10	2.60
Masters	308	83.90
Educational specialist	32	8.40
Doctorate	18	4.70
Years’ experience in schools			12.89	9.75
Number of graduate courses with training on MTSS
0	242	63.90
1	67	17.70
2	32	8.40
3	14	3.70
4	5	1.30
5	4	1.10
6	1	.30
School characteristics
School setting
Elementary	130	34.30
Middle	54	14.20
Combined elementary/Middle	29	7.70
Combined middle/High school	34	9.00
High school	106	28.00
Other	26	6.90
Geographic descriptor
Rural	110	29.00
Urban	93	24.50
Suburban	174	45.90
Regional descriptor
Northeast	92	24.30
Midwest	95	25.10
South	97	25.60
West	94	24.80
Stage of MTSS implementation
Exploration and adoption	99	26.10
Initial implementation (1–2 yrs.)	145	38.30
Full operation (3+ years)	132	34.80
Amount of professional development (PD) on MTSS
No PD	116	30.60
One PD session per year	161	42.50
Two or more PD sessions per year	101	26.60

Note. Some variables have fewer than 379 responses due to missing data.

Measures

Item Development for the MTSS-IBS

After reviewing the RTI Beliefs Scale (Castillo et al., 2015, 2018) and the North Carolina MTSS Beliefs Scale (2015), we made two modifications to the MTSS-IBS. First, whereas the Florida and NC scales contained items related to students with learning disabilities or emotional/behavioral disorders, the MTSS-IBS focused on all students, irrespective of ability. Although these scales had some items applicable to all students, the MTSS-IBS used a broader, inclusive focus, using terms such as “most students” or “all students.” Second, we needed a measure applicable to both academic achievement and behavior. Because both reviewed measures primarily emphasized academic problems, such as performance on math and reading benchmarks, our edits included academic and behavior terminology. For example, the NC scale contained the item: The majority of students with Specific Learning Disabilities can achieve grade-level benchmarks in math. We created two separate items “Most students with learning problems can achieve grade level academic standards” and “Most students with learning problems can achieve behavioral expectations.” Based on these modifications, the MTSS-IBS initially had 24 items.

Abbreviated Acceptability Rating Profile (AARP)

We used two measures to assess the construct validity of the MTSS-IBS. We used a modified version of the Abbreviated Acceptability Rating Profile (Tarnowski & Simonian, 1992), a common measure for treatment acceptability, as an assessment of concurrent validity. Using a 6-point, Likert scale (1 = Disagree Very Strongly to 6 = Agree Very Strongly), the AARP contains eight items designed to measure consumer satisfaction with an intervention. Tarnowski and Simonian used a principal component analysis (PCA) to verify that the measure had a unitary component structure, accounting for nearly 85% of the variance. All items loaded at .89 or higher, and internal consistency estimates (split-half = .95, Cronbach α = .97) were very high. Carter (2010) recently confirmed the unitary factor structure and high reliability. Like Donnell and Gettinger (2015), who modified the original AARP items for their study on RTI acceptability, we substituted the term “MTSS” for “treatment” for each of the eight items—for example, “MTSS is an acceptable intervention program for student problems.” To ensure reliability, we conducted Cronbach alpha and Guttman split-half analyses, which were .93 and .89, respectively, on the current study sample.

MTSS Sustainability Scale (MTSS-SS)

Like the AARP, we used the MTSS-SS (Bahr et al., 2022) for a concurrent validity assessment. The MTSS-SS is a 10-item, Likert-type scale (1 = Strongly Disagree, 5 = Strongly Agree) assessing MTSS sustainability, defined as ongoing implementation with attention to core program principles and implementation fidelity (Han & Weiss, 2005). The MTSS-SS demonstrates content validity, and factor analysis indicated a unitary factor named MTSS Sustainability. Cronbach alpha and Guttman split-half reliability analyses were satisfactory at .93 and .86, respectively, and the measure demonstrated construct validity via analyses differentiating known groups.

Procedures

After receiving university IRB approval, we contacted ASCA to request access to a sample of their members. ASCA approved the study and sent an email message with our study description and an invitation to complete the MTSS Beliefs Scale anonymously to a randomly selected group of 1000 members. A statement at the beginning of the survey oriented participants to our definition of MTSS: “MTSS refers to schools using a tiered system typically focusing on class-wide instruction and interventions designed for all students (Tier 1), interventions used with small groups of students whose skills need additional remediation (Tier 2), and targeted interventions tailored for individual students who need intensive support (Tier 3).” Participants had to option to receive a $25.00 Amazon gift card for participating. ASCA sent three emails: the first during the final six weeks of the school year and follow-up ones two weeks apart thereafter. We received 379 usable responses (37.90% response rate).

Results

Data Suitability and PCA

We conducted preliminary steps to ensure the data were suitable for a PCA. We examined inter-item correlations, and neither singularity nor collinearity was present. Most correlations were low or moderate, ranging from .03-.45 with only two items above .50. The Kaiser-Meyer-Olkin test of sampling adequacy was .761, and Bartlett’s Test of Sphericity was significant, χ² (276) = 3842.948 (p < .001). Collectively, these analyses supported the suitability for further analysis (Muliak, 2009).

A PCA examined the initial 24 items with an oblique rotation because some items were correlated. After 14 iterations, the PCA identified six components with eigenvalues exceeding 1.00. We then used multiple methods to determine viable components (Muliak, 2009). We first conducted a parallel analysis with O’Connor’s (2000) procedure, and five of the six components contained eigenvalues greater than the 95th percentile of the randomly generated values. Examination of the scree plot supported five components. Using a minimum criterion of .40 for item loading, 23 of the 24 items loaded, and one item, #21, loaded on two components. However, only two items loaded on the fourth and sixth components. Despite the satisfactory item-loadings (i.e., .80+), we concluded that support for the fourth and sixth components was untenable with only two item-loadings each and the results of the parallel analysis (i.e., non-support for a sixth component). Given multiple considerations (i.e., O’Connor analysis, scree plot, item-loadings, theoretical fit with a component), we determined the MTSS-IBS best fit a four-component solution with 19 items. Table 2 displays item-loadings.

Table 2.

Results of the Principal Component Analysis with School Counselors (n = 379) on the MTSS-Interventionist Beliefs Scale.


Item	Components
Item	1	2	3	4	5	6
1. Using ongoing student performance data to determine intervention effectiveness is the most accurate method (i.e., is more reliable and valid than educator judgment alone)	.729	.076	−.011	.058	−.163	−.049
2. Measuring intervention integrity (i.e., intervention implemented as planned) is important for making accurate intervention decisions	.682	.076	−.032	−.002	−.153	.070
3. The primary goal of assessment is to inform effectiveness of intervention	.678	.003	.051	.064	.006	−.047
4. Graphing student data makes it easier to make decisions about student performance and needed interventions	.645	−.040	−.037	.018	−.068	−.004
5. Interventions should be provided with increasing intensity (time, group size, focus) based on student need	.564	.026	−.059	.001	−.045	.158
6. The severity of a behavior problem is determined by how quickly the student responds to intervention, not by how inappropriate the behavioral performance is	.539	−.059	−.030	−.055	.296	.037
7. Prevention and early intervention strategies in schools would result in fewer referrals to problem-solving teams and placements in special education	.307	−.237	−.104	.085	.103	.230
8. I have a responsibility to ensure that all students meet behavioral expectations	−.074	−.791	−.042	−.032	−.038	.043
9. I have a responsibility to ensure that all students meet grade-level academic standards	−.010	−.787	−.057	−.045	.113	−.047
10. The primary function of tier 2 (i.e., Supplemental) and tier 3 (i.e., Intensive), is to ensure that students meet grade-level academic standards	−.013	−.714	.003	.123	−.064	.066
11. The primary function of tier 2 (i.e., Supplemental) and tier 3 (i.e., Intensive), is to ensure that students meet behavioral expectations	−.028	−.713	−.014	.095	−.145	.048
12. Most students with behavioral problems can achieve behavioral expectations	−.053	.049	−.872	.064	−.009	−.004
13. Most students with behavioral problems can achieve grade-level academic standards	−.025	.118	−.840	.089	−.099	.000
14. Most students with learning problems can achieve grade-level academic standards	.069	−.098	−.778	−.071	.062	−.119
15. Most students with learning problems can achieve behavioral expectations	−.046	−.103	−.760	−.061	−.160	.017
16. All students are capable of learning at high levels	.229	−.135	−.480	−.010	.193	.091
17. Additional resources should be allocated to students who are not reaching grade-level standards before resources are directed to students who are at or above grade-level academic standards	.024	−.024	.012	.948	.034	−.030
18. Additional resources should be allocated to students who are not reaching grade-level standards before resources are directed to students who are at or above behavioral expectations	.022	−.040	−.028	.930	.009	.008
19. Multi-tiered systems of support (MTSS), when effectively implemented, is a framework that allows educators to meet the needs of all students for behavior	.025	−.054	−.173	−.003	−.794	.070
20. Multi-tiered systems of support (MTSS), when effectively implemented, is a framework that allows educators to meet the needs of all students for academics	.094	−.045	−.069	.004	−.777	.088
21. Tier 1 interventions (i.e., those provided to all students) should be effective enough to result in at least 80% of students achieving behavioral expectations	.142	−.406	.060	−.097	−.505	−.099
22. Tier 1 interventions (i.e., those provided to all students) should be effective enough to result in at least 80% of students achieving grade-level academic standards	.235	−.371	.075	−.008	−.469	−.170
23. A student’s parent (guardian) should be involved in the problem-solving process as soon as a teacher has a concern about the student	.011	−.002	.045	−.072	−.071	.851
24. Students respond better to interventions when their parent (guardian) is involved in the development and implementation of those interventions	.008	−.028	.033	.078	−.004	.817

Note. The rotation method was oblique with the Kaiser normalization, and 14 iterations were required.

The first component encompassed six items (1-6—see Table 2) and focused on various assessment, intervention implementation, and evaluation techniques. Sample items included use of technical procedures such as ongoing data monitoring of student performance, measurement of intervention integrity, and the purpose of assessment as informing intervention development and effectiveness. We titled this component Use of Data and Intervention Fidelity, which accounted for 24.97% of the variance. Five items (8-11, 21) loaded on the second component. One of these items addressed interventionist responsibility to ensure students met behavioral expectations, and a parallel item addressed responsibility for students meeting grade-level academic standards. Two other items focused on the function of tier 2 and 3 interventions, ensuring students meet either academic or behavioral standards. Item 21 loaded on this component and also on the fourth viable component (see component five in Table 2). Because of its relatively low loading on component 2, and because of its greater theoretical fit, discussed below, with component 4, we excluded it from this component. The second component, explaining 10.28% of the variance, was titled Academic and Behavioral Expectations for Students.

The third component contained five items (12-16) addressing student ability to learn and meet behavioral and grade-level academic standards. This component, explaining 8.78% of the variance, was named Student Learning Capability. Four items (19-22) also loaded on the last viable component (component 5 in Table 2). Each item focused on more general beliefs about MTSS, such as effective tier 1 systems impacting students positively regarding their academic achievement or behavior. This included item 21, which focused on tier 1 interventions. Two items stated a belief that MTSS promotes student success when it is implemented effectively. We named this component General Beliefs about MTSS, accounting for 5.36% of the variance. Collectively, the four components of the MTSS-IBS explained 50.87% of the variance.

Reliability and Validity

Cronbach’s alpha assessed the internal consistency of each subscale, and the score reliability was adequate, with alphas ranging from .72-.81. Inspection of individual items on each subscale revealed reliability would not appreciably improve with an item’s deletion. Table 3 contains subscale descriptive statistics and Cronbach values. We used three analyses to assess construct validity: an examination of subscale correlations and two regression models. The assumption with subscale correlations was that a high degree of association would indicate subscale similarity. However, this was not the case because the subscales correlated in the low to moderate range (i.e., .26-.53), thus indicating a sufficient degree of construct differentiation (see Table 3) and contributing to convergent validity (Lyons-Thomas, 2014).

Table 3.

Descriptive Statistics, Cronbach’s Alpha, and Scale Correlations for the MTSS Beliefs Subscales (n = 379).

Factor	Number of Items	M	SD	$α$	SLC	US-IF	ABES	SB
Student learning capability	5	19.93	3.30	.81	1.00
Use of data and intervention fidelity	6	24.60	3.02	.72	.35***	1.00
Academic and behavioral expectations for students	4	16.60	2.50	.79	.34***	.28***	1.00
Specific beliefs about MTSS	4	17.00	2.67	.80	.26***	.32***	.53***	1.00

Note. Each item was rated on a scale from 1 = Strongly Disagree to 5 = Strongly agree. Abbreviations denote subscales: SLC = Student Learning Capability, US-IF = Use of Data and Intervention Fidelity, ABES = Academic and Behavioral Expectations for Students, SB = Specific Beliefs about MTSS. Note. Subtest correlations are Pearson Product-Moment Coefficients. ***p < .001.

Two regression analyses demonstrated the convergent validity of the MTSS-IBS to predict outcomes on measures expected to correlate with it (Lekwa et al., 2019). The first analysis regressed the four MTSS-IBS subscales on the modified AARP. The subscales served as independent variables. Three of them—Use of Data/Intervention Fidelity, Student Learning Capacity, and Specific Beliefs about MTSS—significantly predicted outcomes on the modified AARP (M = 37.94, SD = 6.23), which served as the dependent measure. The Academic and Behavioral Expectations for Students subscale failed to predict the AARP outcome. Table 4 shows the results of this regression model. A second regression model examined the extent to which subscales predicted the sustainability of MTSS. Using the MTSS-SS (M = 40.66, SD = 7.74) as the dependent variable, this model revealed two MTSS-IBS subscales—Use of Data/Intervention Fidelity and Specific Beliefs about MTSS—were significant predictors of sustainability. Table 5 displays the results of the model.

Table 4.

MTSS Interventionist Beliefs Subscales Regressed on the MTSS Acceptability Scale (n = 379).

	Unstandardized Coefficients		Standardized Coefficients			95.0% Confidence Interval for B
Model	B	SE	B	t	p	Lower bound	Upper bound
Constant	1.561	2.474		.63	.529	−3.304	6.425
Student learning capability	.257	.085	.136	3.01	.003	.089	.425
Use of data and intervention fidelity	.915	.093	.443	9.84	.000	.732	1.097
Academic and behavioral expectations for students	.006	.123	.002	.04	.963	−.237	.248
Specific beliefs about MTSS	.510	.114	.219	4.46	.000	.285	.735
R Statistics^a	R = .621 R² = .386 adj. R² = .380

^aF (4,374) = 58.80, p < .001.

Table 5.

MTSS Interventionist Beliefs Subscales Regressed on the MTSS-Sustainability Scale (n = 379).

	Unstandardized Coefficients		Standardized Coefficients			95.0% Confidence Interval for B
Model	B	SE	B	t	p	Lower bound	Upper bound
Constant	15.206	3.665		4.15	.000	8.000	22.412
Student learning capability	.230	.126	.098	1.81	.070	−.019	.478
Use of data and intervention fidelity	.451	.138	.176	3.27	.001	.180	.721
Academic and behavioral expectations for students	−.036	.183	−.012	−0.19	.846	−.395	.324
Specific beliefs about MTSS	.611	.169	.211	3.61	.000	.278	.944
R Statistics^a	R = .358 R² = .128 adj. R² = .119

^aF (4,374) = 13.70, p < .001.

Discussion

Despite the recognized importance of understanding MTSS beliefs and their relationship for effectively meeting students’ needs, no psychometrically supported assessment has been available to measure interventionist beliefs about MTSS. This study aimed to develop and validate a new measure of interventionist beliefs about MTSS implementation. The MTSS-IBS is the first assessment developed to measure interventionist beliefs about MTSS implementation as it relates to all students' academic and behavior expectations. To achieve this, we modified items from existing scales on RTI and MTSS beliefs and evaluated the scale’s internal structure using PCA. Findings of this study support the MTSS-IBS as an assessment of beliefs that can be helpful for determining interventionist’s attitudes toward MTSS, and it may also serve as a psychometrically-sound instrument for measuring interventionist beliefs about MTSS for researchers.

Previous measures of MTSS beliefs, including RTI, have focused on the academic needs of students with learning disabilities or emotional/behavioral disorders or were normed on special educators (Castillo et al., 2015; N.C. Department of Public Instruction, 2015). This limits those instruments for assessing interventionist beliefs because interventionists serve all students and not just students with disabilities. Moreover, interventionists are charged with supporting students’ academic and behavioral development (ASCA, 2019; Lucio et al., 2021; NASP, 2022). As such, a measure that adequately assesses interventionist beliefs regarding the full scope of their use of MTSS to support all students' academic and behavior needs is necessary.

The findings of this study suggest that the MTSS-IBS is a multifactor measure of interventionist beliefs about MTSS. The four components best fit the data obtained from the sample of school counselors. The four components are generally comparable to those on the RTI Beliefs Scale (Castillo et al., 2015), which was adapted to develop the MTSS-IBS. The RTI Beliefs Scale has three factors—Academic Abilities and Performance of Students with Disabilities, Data-Based Decision-Making, and Functions of Core and Supplemental Instruction. Given the structural similarities of the MTSS-IBS to the RTI Beliefs Scale and the reliability and validity of the MTSS-IBS in this study, we maintain the MTSS-IBS demonstrates initial psychometric adequacy and is a potentially effective measure for assessing school counselor beliefs about MTSS. The current study showed that 24 items revealed a four-component solution with 19 items explaining 49.39% of the variance; moreover, scores on each subscale had adequate reliability with Cronbach alpha values ranging from .72-.81. Correlation analysis revealed a sufficient degree of construct differentiation as none of the subscales had a high association with one another.

The results of regression analyses provided preliminary evidence for convergent validity. First, we regressed the MTSS-IBS scales on the MTSS Acceptability Scale. Acceptability is a part of social validity focusing on how fair, feasible, and appropriate a treatment is for interventionists who implement it and for students who receive it (Kazdin, 1980). Acceptability is correlated with MTSS intervention efficacy, such as the perceived degree of intervention intrusiveness, the feasibility of implementation, and intervention integrity (Silva et al., 2019). Findings showed that three MTSS-IBS subscales significantly predicted MTSS acceptability as measured by the modified AARP. These results support previous research showing that educator beliefs about MTSS and RTI predict willingness to implement new practices (Castillo et al., 2010; Martens & Andreen, 2013) and lay the groundwork for future investigation into the relationship between MTSS beliefs and acceptability, an association between constructs that has not been studied.

Second, we regressed the MTSS-IBS scales on the MTSS-Sustainability Scale. Previous research has found that MTSS sustainability is linked to successful MTSS implementation (McIntosh et al., 2016). The results of this study showed a positive relationship between MTSS-IBS and MTSS-SS. Specifically, interventionist beliefs about the Use of Data/Intervention Fidelity and Specific Beliefs about MTSS significantly predicted the sustainability of MTSS. Interventionist beliefs predicted the likelihood of continued implementation of MTSS with ongoing fidelity to the core principles of MTSS. This is consistent with previous studies showing school counselor beliefs impact implementation of a comprehensive school counseling program and accountability practices (Topdemir, 2013).

Implications for Practice and Research

The MTSS-IBS has the potential for use by school leadership to assess school counselor beliefs about implementing MTSS interventions. ASCA (2019) maintains that school counselors should use tiered service delivery models such as MTSS to meet the needs of all students. Because school counselor beliefs can prevent them from implementing MTSS at all or with fidelity (Castillo et al., 2010), such hindering beliefs can limit the efficacy of MTSS interventions and diminish the support of student academic and career outcomes. The MTSS-IBS serves as a tool that leaders, researchers, and counselors themselves can utilize to regularly assess their beliefs about MTSS and the impact of those beliefs on the delivery of MTSS. Moreover, the MTSS-IBS can be used in addition to other measures of MTSS intervention evaluation and research. Furthermore, counselor educators can use the MTSS-IBS to evaluate school counselors-in-training’s (SCIT) beliefs about MTSS at the end of courses that teach SCIT how to implement MTSS. The MTSS-IBS can help counselor educators identify components of MTSS implementation that SCIT are not bought into and create content or course materials to help support SCIT’s beliefs about MTSS.

Limitations and Future Research

This study has several limitations. First, data in this study are self-reported, and both response and self-reporting bias may be present. Participants may have responded to the questions about their MTSS beliefs with socially acceptable answers as opposed to being truthful and may have been impacted by their responses to earlier questions in the survey as the MTSS-IBS was part of a larger study. Second, a convenience sample of ASCA members was used for this study, which may impact the generalizability of the results to school counselors nationally and globally. Related to the preceding limitation, further tests of dimensionality among different interventionists (e.g., school psychologists, social workers) are warranted. Finally, since this is the first study to validate the MTSS-IBS, readers should consider these validation data as preliminary. Future researchers should consider evaluating the psychometric properties of the MTSS-IBS with diverse samples, exploring the MTSS-IBS in conjunction with qualitative methods, and exploring the impact of interventionist MTSS beliefs on other aspects of MTSS implementation, such as culturally competent MTSS implementation and MTSS knowledge and skills development.

Supplemental Material

Supplemental Material - Multi-Tiered System of Supports—Interventionist Beliefs Scale: Development and Initial Validation With School Counselors

Supplemental Material for Multi-Tiered System of Supports—Interventionist Beliefs Scale: Development and Initial Validation With School Counselors by Mary Edwin, and Michael W. Bahr in Journal of Psychoeducational Assessment

Footnotes

Author’s Note

A supplemental file contains a comparison of the Abbreviated Acceptability Rating Profile with the MTSS Acceptability Scale. It also includes a listing of the items on the MTSS-Sustainability Scale.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Michael W. Bahr

Supplemental Material

Supplemental material for this article is available online.

References

American School Counselor Association . (2019). The ASCA national model: A framework for school counseling programs (4th ed.). ASCA.

American School Counselor Association . (2021). ASCA research report: State of the profession 2020. https://www.schoolcounselor.org/getmedia/bb23299b678d-4bce-8863-cfcb55f7df87/2020-State-of-theProfession.pdf

Bahr

M. W.

Edwin

Long

(2022). Development of a brief measure for MTSS sustainability. Assessment for Effective Intervention, 48(2), 1–10. https://doi.org/10.1177/15345084221119418

Betters-Bubon

Donohue

(2016). Professional capacity building for school counselors through school-wide positive behavior interventions and supports implementation. Journal of School Counseling, 14(3), 35. https://jsc.montana.edu/articles/v14n3.pdf

Bradshaw

C. P.

Cohen

Espelage

D. L.

Nation

(2021). Addressing school safety through comprehensive school climate approaches. School Psychology Review, 50, 221–236. https://doi.org/10.1080/2372966X.2021.1926321

Bradshaw

C. P.

Mitchell

M. M.

Leaf

P. J.

(2010). Examining the effects of schoolwide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133–148. https://doi.org/10.1177/1098300709334798

Brownell

M. T.

Sindelar

P. T.

Kiely

M. T.

Danielson

L. C.

(2010). Special education teacher quality and preparation: Exposing foundations, constructing a new model. Exceptional Children, 76(3), 357–377. https://doi.org/10.1177/001440291007600307

Burns

M. K.

Jimerson

S. R.

VanDerHeyden

A. M.

Deno

S. L.

(2016). Toward a unified response-to-intervention model: Multi-tiered systems of support. In Jimerson

S. R.

Burns

M. K.

VanDerHeyden

A. M.

(Eds.), Handbook of response to intervention: The science and practice of multi-tiered systems of support (2nd ed., pp. 719–732). Springer. https://doi.org/10.1007/978-1-4899-7568-3_41

Carter

S. L.

(2010). A distributive model of treatment acceptability. Education and Training in Developmental Disabilities, 43(4), 411–420. https://www-jstor-org-443.web.bisu.edu.cn/stable/23879672

10.

Castillo

J. M.

Batsche

G. M.

Curtis

M. J.

Stockslager

March

Minch

(2010). Problem solving/response to intervention project: Evaluation tool technical assistance manual. Florida Problem Solving/Response to Intervention Project, University of South Florida.

11.

Castillo

J. M.

Dedrick

R. F.

Stockslager

K. M.

March

A. L.

Hines

C. V.

Tan

S. Y.

(2015). Development and initial validation of a scale measuring the beliefs of educators regarding response to intervention. Journal of Applied School Psychology, 31(1), 1–30. https://doi.org/10.1080/15377903.2014.938282

12.

Castillo

J. M.

Wang

J. H.

Daye

J. G.

Shum

K. Z.

AmandaMarch

L. A. L.

(2018). A longitudinal analysis of the relations among professional development, educators’ beliefs and perceived skills, and response-to-intervention implementation. Journal of Educational and Psychological Consultation, 28(4), 413–444. https://doi.org/10.1080/10474412.2017.1394864

13.

Cowan

K. C.

Vaillancourt

Rossen

Pollitt

(2013). A framework for safe and successful schools [Brief]. National Association of School Psychologists.

14.

Cressey

J. M.

Whitcomb

S. A.

McGilvray-Rivet

S. J.

Morrison

R. J.

Shander-Reynolds

K. J.

(2014-2015). Handling PBIS with care: Scaling up to school-wide implementation. Professional School Counseling, 18(1), 90–99. https://www-jstor-org-443.web.bisu.edu.cn/stable/profschocoun.18.1.90

15.

Cusumano

D. L.

Decker

C. J.

(2019). Collaborative consultation: A model for school counselors' integration into MTSS. Journal of School Counseling, 17(4), 1–32. https://doi.org/10.15690/jscp.v17i4.2227

16.

Donnell

L. A.

Gettinger

(2015). Elementary school teachers’ acceptability of school reform: Contribution of belief congruence, self-efficacy, and professional development. Teaching and Teacher Education, 51, 47–57. https://dx.doi.org/10.1016/j.tate.2015.06.003

17.

Donohue

Goodman-Scott

Jennifer

B.-B.

(2015). Using universal screening for early identification of students at risk: A case example from the field. Professional School Counseling, 19(1), 133. https://doi.org/10.5330/1096-2409-19.1.133

18.

Gersten

Dimino

J. A.

(2006). RTI (Response to Intervention): Rethinking special education for students with reading difficulties (yet again). Reading Research Quarterly, 41(1), 99–108. https://doi.org/10.1598/RRQ.41.1.5

19.

Gonzales

Hughes

M. T.

(2021). Leveraging a Spanish literacy intervention to support outcomes of English learners. Reading Psychology, 42(4), 411–434. https://doi.org/10.1080/02702711.2021.1888352

20.

Goodman-Scott

Ockerman

M. S.

(2019). Integrating school counseling and MTSS. In Goodman-Scott

Betters-Bubon

Donohue

(Eds.), The school counselor’s guide to multi-tiered systems of support (pp. 29–61). Routledge. https://dx-doi-org.web.bisu.edu.cn/10.4324/9781315144467-2

21.

Han

S. S.

Weiss

(2005). Sustainability of teacher implementation of school-based mental health programs. Journal of Abnormal Child Psychology, 33(6), 665–679. https://link-springer-com-443.web.bisu.edu.cn/article/10.1007/s10802-005-7646-2

22.

Hatch

Chen-Hayes

S. F.

(2008). School counselor beliefs about ASCA National Model school counseling program components using the SCPCS. Professional School Counseling, 12(1), 34–42. https://doi.org/10.1177/2156759X0801200104

23.

Horner

R. H.

Sugai

Smolkowski

Eber

Nakasato

Todd

A. W.

Esperanza

(2009). A randomized, wait-list controlled effectiveness trail assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133–144. https://doi.org/10.1177/1098300709332067

24.

Kazdin

A. E.

(1980). Acceptability of alternative treatments for deviant child behavior. Journal of Applied Behavior Analysis, 13(2), 259–273. https://doi.org/10.1901/jaba.1980.13-259

25.

King

K. R.

Lembke

E. S.

Reinke

W. M.

(2016). Using latent class analysis to identify academic and behavioral risk status in elementary students. School Psychology Quarterly, 31(1), 43–57. https://psycnet.apa.org/doi/10.1037/spq0000111

26.

Larson

L. M.

Daniels

J. A.

(1998). Review of the counseling self-efficacy literature. Counseling Psychologist, 26(2), 179–218. https://doi.org/10.1177/0011000098262001

27.

Lauterbach

A. A.

Harrington

Yakut

A. D.

Krezmien

(2018). School counselor knowledge, beliefs, and practices related to the implementation of standards-based comprehensive school counseling in the United States. Journal of School-Based Counseling Policy and Evaluation (JSCPE), 1(1), 195. https://doi.org/10.25774/7hhc-x195

28.

Lekwa

A. J.

Reddy

L. A.

Shernoff

E. S.

(2019). Measuring teacher practices and student academic engagement: A convergent validity study. School Psychology, 34(1), 109–118. https://doi.org/10.1037/spq0000268

29.

Lucio

Souhrada

Nelson

Shayman

Gilmore

Scott

(2021). Definition of a highly qualified school social worker. School Social Work Association of America. https://www.sswaa.org/_files/ugd/426a18_ef7f88de23f541879d2cbee219f2c0ec.pdf

30.

Lyons-Thomas

(2014). Interscale correlations. In Michalos

A. C.

(Ed.), Encyclopedia of quality of life and well-being research (pp. 3352–3353). Springer. https://doi.org/10.1007/978-94-007-0753-5_1519

31.

Martens

Andreen

(2013). School counselors’ involvement with a school-wide positive behavior support intervention: Addressing student behavior issues in a proactive and positive manner. Professional School Counseling, 16(5), 313–322. https://doi.org/10.1177/2156759X1201600504

32.

McIntosh

Goodman

(2016). Integrated multi-tiered systems of support: Blending RTI and PBIS. Guilford.

33.

McIntosh

Mercer

S. H.

Nese

R. N. T.

Strickland-Cohen

M. K.

Hoselton

(2016). Predictors of sustained implementation of school-wide positive behavioral interventions and supports. Journal of Positive Behavior Interventions, 18(4), 209–218. https://doi.org/10.1177/1098300715599737

34.

Mulaik

S. A.

(2009). Foundations of factor analysis (2nd ed.). Routledge.

35.

National Association of School Psychologists (2020). The professional standards of the national association of school psychologists. NASP.

36.

Neitzel

A. J.

Lake

Pellegrini

Slavin

R. E.

(2021). A synthesis of quantitative research on programs for struggling readers in elementary schools. Reading Research Quarterly, 57(1), 149–179. https://doi.org/10.1002/rrq.379

37.

North Carolina Department of Public Instruction (2015). North Carolina MTSS beliefs survey. N.C. Department of Public Instruction.

38.

O'Connor

B. P.

(2000). SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behavior Research Methods, Instrumentation, and Computers, 32(3), 396–402. https://doi.org/10.3758/bf03200807

39.

Scarborough

J. L.

Faircloth

B. S.

Brock

A. M.

(2015). Impact of a comprehensive school counseling program aligned with multi-tiered systems of support. Professional School Counseling, 19(1), 91–101. https://doi.org/10.5330/prsc.19.1.q07778

40.

Sharrat

Fullan

(2009). Putting FACES on the data: What great leaders do! Corwin. https://eric.ed.gov/?id=ED530431#:∼:text=%22PuttingFacesontheData,ongoingassessmenttoinforminstruction

41.

Silva

M. R.

Collier-Meek

M. A.

Codding

R. S.

DeFouw

E. R.

(2019). Acceptability assessment of school psychology interventions from 2005 to 2017. Psychology in the Schools, 57(1), 62–77. https://dx-doi-org.web.bisu.edu.cn/10.1002/pits.22306

42.

Simonsen

Sugai

(2019). School-wide positive behavioral interventions and supports: A systems-level application of behavioral principles. In Little

S. G.

Akin-Little

(Eds.), Behavioral interventions in schools: Evidence-based positive strategies (pp. 35–60). American Psychological Association. https://doi.org/10.1037/0000126-003

43.

Sink

C. A.

Akos

Turnbull

R. J.

Mvududu

Lee

S. W.

(2015). Aligning comprehensive school counseling programs and multi-tiered systems of support: A review of the literature. Journal of School Counseling, 13(5), 1–34. https://files.eric.ed.gov/fulltext/EJ1088556.pdf

44.

Tarnowski

K. J.

Simonian

S. J.

(1992). Assessing treatment acceptance: The Abbreviated acceptability rating profile. Journal of Behavior Therapy and Experimental Psychiatry, 23(2), 101–106. https://dx.doi.org/10.1016/0005-7916(92)90007-6

45.

Topdemir

(2013). School counselors' beliefs about accountability. The Practitioner Scholar: Journal of Counseling & Professional Psychology, 2(1), 12–22. https://www.thepractitionerscholar.com/article/view/10899

46.

U.S. Department of Education . (2014). Guiding principles: A resource guide for implementing school climate and discipline. Retrieved from https://files.eric.ed.gov/fulltext/ED544743.pdf

47.

Weisenburgh-Snyder

A. B.

Malmquist

S. K.

Robbins

J. K.

Lipshin

A. M.

(2015). A model of MTSS: Integrating precision teaching of mathematics and a multi-level assessment system in a generative classroom. Learning Disabilities: A Contemporary Journal, 13(1), 21–41. https://eric.ed.gov/?id=EJ1080598

48.

Young

Kaffenberger

(2011). The beliefs and practices of school counselors who use data to implement Comprehensive School Counseling Programs. Professional School Counseling, 15(2). https://doi.org/10.1177/2156759X1101500204

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB