Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Abstract

Despite a call for evidence-based practice in school psychology, limited research on the topic of evidence-based assessment exists. To begin to address this gap in the research, a modified version of Jenson–Doss and Hawley’s (2010) Attitudes Toward Standardized Assessment (ASA) scale was administered to 371 U.S. school psychologists. Examination of the modified ASA’s factor structure suggested that a bifactor model with a single overall domain and three sub-domains was the most parsimonious. Indices of dimensionality indicated that the overall score may be the best indicator of school psychologist’s perceptions of standardized assessment. Additionally, school psychologists’ reported favorable attitudes of standardized assessment compared to clinical judgment alone. Limitations and implications for future research are discussed.

Keywords

school psychology evidence-based assessment standardized assessment perceptions

The focus on evidence-based practice (EBP) in behavioral health has increased substantially over the past two decades. Multiple mental health organizations, including the American Psychological Association’s Division 16 (Kratochwill & Stoiber, 2002), the National Institute for Mental Health (NIMH, 2020), and the Substance Abuse and Mental Health Service Administration (SAMSA, 2011), established guidelines for the use of EBP to improve service provision. Perhaps the biggest proponent of this movement has been the American Psychological Association (APA) with their creation of a Task Force on Evidence-Based Practice (2006). The APA task force (2008) defined EBP as the application of “empirically supported principles of psychological assessment, case formulation, therapeutic relationship, and intervention” (p. 271) that are crucial for effective practice and public health.

Notable for school psychologists, The APA task force issued a 71-page guide for implementing EBP for children and adolescence, which provided guidance in evidence-based assessment, intervention, and prevention. Additionally, while the field of school psychology was slower to respond to this trend, the National Association for School Psychology (NASP, 2020) explicitly advocated for EBP in their revision of Domain 9 of their professional standards. This is demonstrated in the domain name change from Research and Program Evaluation in 2010 to Research and Evidence-Based Practice in the 2020 update.

Evidence-Based Assessment

The APA (2002) and NASP (2020) place a specific emphasis on EBP in assessment, also known as evidence-based assessment (EBA). In a 2009 position statement, NASP specifically called for “science based assessment practices” (p. 1), which may suggest practices based in theory and found to be clinical effective (Lilienfeld et al., 2018). The role of evidence-based practice regarding assessment was reiterated in an updated version of this position paper in 2016 (NASP, 2016). There are multiple reasons why assessment is so heavily emphasized by our professional organizations. Assessment is vital for classification and diagnosis, which create the shared language that we use to consult with other practitioners and functions as a prerequisite for special education services in most states. Therefore, accurate assessment allows practitioners to effectively communicate about the children they serve. Assessment is also an important component of evidence-based intervention; effective assessment practices allow practitioners to effectively select and modify interventions and to determine the efficacy of interventions and to measure treatment outcomes (Jensen–Doss & Hawley, 2010; Norcross et al., 2006; Youngstrom & Van Meter, 2016).

Equally important is the use of assessment practices to aid in high-stakes decision making including eligibility for special education, and the allocation of services provided to students’ with disabilities (NASP, 2009). Mislabeling based on erroneous assessment practices can lead to self-fulfilling prophecies that can harm children¹ (Higgins, et al., 2002; NASP, 1986). Despite the importance of assessment, research in EBP has largely overlooked assessment focusing, instead, on intervention (e.g., Gross et al., 2019; Jensen–Doss & Hawley, 2010). Furthermore, there is little consensus regarding what differentiates EBA from typical assessment (Youngstrom & Van Meter, 2016) and no operational definition of EBA is provided by APA, NASP, or other mental health organizations. However, like Hunsley and Mash (2007), the APA (2008) and NASP (2009, 2016) stipulate that EBA includes the use of culturally responsive and psychometrically sound assessment practices.

While assessment is important to the practice of psychology as a whole, it is of special importance to school psychologists due to the central role of assessment in the field. School psychologists frequently engage in assessment practices (e.g., Farmer et al., 2021; McNamara et al., 2019). As part of the NASP Research Committee, Farmer et al. (2021) summarized data from the 2020 NASP Membership Survey and found that assessment was the most frequently endorsed activity amongst full-time, school-based school psychologists sampled from across the United States (n = 1006). On average, respondents to the NASP Membership Survey reported completing 24 initial eligibility evaluations and 32 re-evaluations during the 2019–2020 school year. Assessment has historically been a major role of the school psychologist and has consumed roughly half of practitioners’ time for the past 30 years (Benson et al., 2019). However, while EBA has begun to receive attention in the field of clinical psychology (see Jensen–Doss & Hawley, 2010; Youngstrom & Van Meter, 2016), few studies in school psychology exist as most research on EBP has focused on evidenced-based intervention with a limited focus on assessment (McGill et al., 2018).

This dearth of research is problematic as school psychologists often engage in problematic assessment practices and underuse lower inference methods such as behavioral screenings and functional assessment (e.g., Gross et al., 2019). Additionally, the use of projective measures continues even though they lack adequate empirical support (Hunsley et al., 2015; Lilienfeld et al., 2012). Benson et al. (2019) reported the results of a 2017 survey of 1317 practicing school psychologists whose demographics were comparable to the 2015 NASP survey membership. In their results, they identified that projective tasks, such as sentence completion (26%), draw-a-person (16%), and house-tree-person tasks (16%), were still used by a non-trivial proportion of school psychologists, despite their questionable utility (Hunsley et al., 2015; Lilienfeld et al., 2012). Additionally, low-value assessment practices include variations of profile analysis. In a second analysis stemming from the 2017 survey conducted by Benson and colleagues, Kranzler et al. (2020) found that profile analytic interpretive strategies are regularly used by over half of school psychologists despite at least 30 years of research indicating significant psychometric and practical concerns with these approaches (see McGill, 2019). Furthermore, school psychologists often use tests for purposes for which they were not intended (e.g., using projective tests for rapport building; Hojnoski et al., 2006) and in situations for which these tests would seem to lack practical utility. In a national survey of members of NASP and state school psychology associations (n = 323), Sotelo-Dynega and Dixon (2014) found that school psychologists reported using cognitive tests was necessary for evaluations of emotional disturbance (31%), speech/language impairment (19%), hearing impairment (7%), and visual impairment (7%). What maintains questionable assessment practices is not well understood, but training is one likely factor (Dombrowski et al., 2021).

It is both reasonable to assume and empirically documented (Cook et al., 2009; Sotelo-Dynega & Dixon, 2014; Youngstrom, 2013) that graduate level training has meaningful influence over psychologist’s practice behaviors following program completion. In school psychology, there is evidence that assessment training related to cognitive assessment includes instruction on low-value practices such as profile analysis, subtest analysis, and ipsative comparison (Lockwood & Farmer, 2020; Miller et al., 2020). Perhaps most troubling, a survey of trainers from 105 school psychology programs by Lockwood et al. (2022) found that approximately 38% of programs require students to administer one or more projective test.

Perceptions of Assessment

One likely reason why school psychologists use questionable assessment practices is because they believe them to be evidence-based. School psychologist’s beliefs impact their practice (Lockwood et al., 2019) and therefore understanding practitioner’ attitudes towards evidence-based assessment may also be useful in helping to train psychologists and encourage EBP (Jenson et al., 2010). One prominent model of examining beliefs that has be used to predict and change behavior is the Theory of Planned Behavior (TPB; Ajzen, 2005; Cooper et al., 2019). According to TPB, intentions are antecedents of behavior that predict the amount of effort that an individual will exert in order to perform a behavior (Ajzen, 1991). TPB studies in education date back over 30 years and indicate that teachers’ beliefs predict a number of behaviors such as teaching methods used (Crawley, 1990; Zint, 2002; Lee et al., 2010) and inclusion practices (MacFarlane & Woolfson, 2013).

TPB has also been used as a model for examining behavioral intentions in clinical psychology. Most germane to this study is the work of Jensen–Doss and Hawley (2010) who examined the attitudes of child-serving practitioners towards EBA. Using a sample of over 1400 mental health practitioners (MHP) including counselors, marriage and family therapists, social workers, psychologists, and psychiatrists, Jensen-Doss and Hawley examined attitudes with the Attitudes Toward Standardized Assessment (ASA) scales, a measure created by the authors and grounded in TPB. Findings from this study suggested that while practitioners expressed positive attitudes towards the psychometric properties of standardized measures, they doubted the practicality of these measures compared to clinical judgment. Jensen–Doss and Hawley found that psychologists reported more positive attitudes towards standardized assessments than did MPH from the other fields (e.g., social workers) that were examined as did practitioners who worked in clinical as opposed to private practice settings. Perhaps most interestingly, women and MHP from ethnic minority groups rated the practical use of standardized measures significantly lower than did White, male practitioners. Additionally (and perhaps unsurprisingly), positive views toward standardized assessment predicted use (Jensen-Doss & Hawley, 2010).

Current Study

As previously noted, little research examining school psychologists’ perceptions of EBA exists even though perceptions impact practice. This is likely due to the lack of a psychometrically sound measure of school psychologists’ perceptions of EBA. The purpose of the current study was to fill this need by examining the factor structure of the ASA (Jensen–Doss & Hawley, 2010) with a nationally representative sample of school psychologists. While a great deal of disagreement exists regarding what constitutes EBA, we agree with Jensen–Doss and Hawley that, at a minimum, EBA involves the use of psychometrically sound, standardized assessment measures over clinical judgment, alone. Therefore, measuring perceptions of standardized assessment provides a first step towards assessing EBA. Specifically, we asked the following research questions:

Attitudes towards Standardized Assessment scales for School Psychologists (ASA-SP) Psychometrics

Does the Attitudes Toward Standardized Assessment (ASA; Jensen–Doss & Hawley, 2010) retain its psychometric properties (i.e., reliability and factor structure) when modified and used with school psychologists (as opposed to other mental health professionals) in the ASA-SP?

Findings from ASA-SP

What are the findings from the ASA-SP regarding school psychologists’ attitudes on the Total score as well as scores on the three subscales: Benefits Over Clinical Judgment, Psychometric Quality, and Practicality? Do these attitudes differ significantly from a neutral position?

Methods

Data used in the current study were gathered as part of a larger survey examining school psychologists’ assessment and progress monitoring practices. The current authors report how the sample was collected, the response rate, all data exclusions, all manipulations, and all measures in this study. The research was approved by the lead author’s university Institutional Review Board. Eysenbach’s (2004) survey reporting methodology was used in the results section. In the interest of transparency and replicability, the authors have made available the (a) survey, (b) data, and (c) survey modifications at https://osf.io/u2k48/.

Recruitment & Participants

The addresses of 1000 currently practicing, randomly selected NASP members were provided by NASP’s research committee after approving the researchers’ proposal. The Tailored Design Method (Dillman et al., 2014) was used to maximize response rate. Specifically, all 1000 prospective respondents were mailed an initial postal letter that requested their participation, provided an url to the survey, a unique access code, and included a noncontingent $1 bill. Two follow-up request postal letters were mailed to non-respondents with survey links, QR codes and participation numbers. Letters were mailed in October and November of 2018 and data collected from November 2, 2018 to January 24, 2019.

Of the 1000 individuals selected for participation, 7 (<1%) had undeliverable addresses and 2 (<1%) informed the authors of their retirement and were therefore ineligible to participate. Of the 991 eligible individuals, 386 (39%) visited the survey URL whereas 605 [61%] did not respond and 2 (<1%) declined consent. Of those who consented, five (<1%) provided their access code but did not answer any questions. Additionally, 10 (2%) did not disclose their highest degree and so were excluded from analyses; this decision is consistent with Jensen–Doss and Hawley (2010) and addressed those responses that did not complete the instruments; the cleaned data, the dictionary, and log are available at the link provided above.

Final participants consisted of 371 school psychologist who practice in the United States. This represents a response rate of roughly 37%, which is higher than the response rate for many online surveys which average around 33% (Lindemann, 2021; Nulty, 2008). Additionally, for reference, the latest NASP member survey obtained a 33% response rate (Goforth et al., 2021). The majority of those who completed the survey identified as white (84%), female (83%), and specialist level practitioners (60%). Most participants reported working in suburban (36%) or urban (25%) traditional school districts (85%) as compared to regional, state, charter, or other types of districts. Additional details are available in Supplemental Table S1. Representation in the sample is similar to that of practicing school psychologists in the United States based on a 2020 national survey of NASP members (Goforth et al., 2021). Specifically, the majority of school psychologists who responded to that study were white (86%), female (87%), specialist level practitioners (68%) in suburban (52%), and urban (25%) settings. Despite demographic consistency between the current study and the NASP membership survey, a review of Supplemental Table S1 indicates that the current sample may underrepresent minoritized school psychologists, school psychologists working in low-income districts, and school psychologists working in districts with a large number of minoritized students. This is because individuals from these groups may be less likely to be members of NASP or respond to NASP surveys (Lewis et al., 2008).

Instrument

The original Attitudes Toward Standardized Assessment (ASA; Jensen–Doss & Hawley, 2010) was intended to measure clinicians’ attitudes towards standardized assessment. The ASA contains 23, 5-point Likert scale items that range from 1 = strongly disagree to 5 = strongly agree, which comprise three subscales including Benefit Over Clinical Judgment (5 items), Psychometric Quality (7 items), and Practicality (10 items). Benefit Over Clinical Judgment is intended to assess the degree to which standardized measures improve upon the information that a practitioner can obtain using clinical judgment only. Psychometric Quality is intended to assess the degree to which practitioners perceive of standardized measures as being reliable and valid as well as the extent to which the value psychometric properties. Practicality is intended to assess a practitioner’s perceptions of the feasibility of using standardized measures. Of note, negatively worded items are reverse coded so that higher scores indicated positive perceptions of standardized assessment (these items are identified in Supplemental Table 2). Psychometrics for the original ASA were obtained from a sample of 1442 MHP from various fields (i.e., counseling, marriage and family therapy, social work, psychology, and psychiatry). Jensen–Doss and Hawley (2010) reported internal consistency reliability for each subscale that ranged from .72 to .75. Additionally, results of a confirmatory factor analysis (CFA) suggested a “good” fit as indicated by multiple indices (see Jensen–Doss & Hawley, 2010 for more regarding psychometrics of the ASA).

Survey Design

A modified version of the ASA (Jensen–Doss & Hawley, 2010), the Attitudes towards Standardized Assessment scales for School Psychologists (ASA-SP) was created with permission of the lead author of the original instrument. The current authors modified items to reflect the slightly different roles filled, and language used, by school psychologists compared to clinical practitioners, but maintained the constructs represented by the original instrument, and therefore its grounding in TPB. Changes include altering terms such as “diagnosis” were changed to “classify,” and “children” was changed to “students.” Additionally, references to the “DSM diagnostic criteria” were changed to “IDEA and/or state education law/regulation criteria” to reflect the different nomenclatures uses by clinical practitioners and school psychologists. For example, the item—“Clinical problems are too complex to be captured by a standardized measure,” from the original ASA was altered to read—“Educational/behavioral problems are too complex to be captured by a standardized measure.” Another item—“Clinicians should use assessments with demonstrated reliability and validity” was modified to—“School psychologists should use assessments with demonstrated reliability and validity.” A full account of the changes made to the original ASA resulting in the ASA-SP can be found in an Excel file at https://osf.io/u2k48/.

Prior to data collection, the ASA-SP was reviewed and piloted by five content experts who work as current trainers of school psychologists, each of whom have experience teaching assessment and have publications in the area of EBA. Items were added and revised based on their feedback. Additionally, a demographic questionnaire was included as well as the ASA-SP survey.

As previously noted, this study was part of a larger survey examining school psychologists’ assessment and progress monitoring practices. The full survey consisted of 57 items. Of these, seven were demographic items, 14 were items regarding the current practices of participants and 36 were items that were adopted from two published measures. These measures included the ASA-SP and a modified version of Jensen–Doss et al.’s, (2018) monitoring and feedback attitudes scale. The 22 modified ASA-SP items were administered directly following consent. Questions regarding current practices (e.g., how often participants engage in standardized testing) and demographic questions were at the end of the survey. Participants completed the entire surveys using the Qualtrics online experience management software platform. The results of monitoring and feedback attitudes scale are not discussed as they are beyond the scope of this current research endeavor.

Analyses

All analyses were completed in R via RStudio (version 1.25; RStudio Team, 2020); syntax and data are available via OSF. Prior to addressing primary or exploratory research questions, we reviewed descriptive data for all relevant variables using the TableOne package (version 0.12.0; Yoshida et al., 2020). As part of this process, the ASA-SP Total Score and subscales were created by averaging relevant items consistent with Jensen—Doss and Hawley (2010). During this process, missing data were evaluated; no variable had more than 5% missing data (see Tables S1 and S2).

R₁, Measurement

Because of modification to the wording of many items on the ASA-SP and to address our first research question—whether the ASA-SP maintained the psychometric properties of the original ASA—we conducted confirmatory factor analysis (CFA) to determine if the three scales formed a correlated traits model as identified by Jensen—Doss and Hawley (2010). Item-by-scale assignment is presented in Table S2. The CFA was conducted using the Lavaan package in R (v. 0.6-6; Rosseel, 2012). As the indicators in the model were on a Likert-scale, diagonal weighted least squares estimation was used. Jensen—Doss and Hawley (2010) permitted within scale residual covariances in their model; however, we chose not to set residual covariances in our model prior to estimation (Brown, 2015), and ultimately did not specify any such post hoc modifications to our models. As noted by Reise and colleagues (2010) psychological constructs are complex and are generally measured by heterogenous items that measure a single domain and multiple sub-domains. Additionally, Jensen–Doss and Hawley (2011) asserted that the ASA was developed as a “measure of child clinicians’ attitudes toward standardized assessment tools in general” (p. 480–481, emphasis added in this manuscript). Furthermore, these researchers referenced the ASA’s overall internal consistency reliability suggesting that they believe the ASA to be interpretable both at the subdomain and overall domain level. To account for the possibility that the information matrix could be further reduced, and to determine whether both the overall score and domain scores may be used we assessed the fit of a bifactor model (i.e., a three correlated traits model with an uncorrelated breadth factor on which all items load separately). To evaluate model fit, we examined robust estimates of chi-square (χ²), root mean square error approximation (RMSEA), standardized root mean square residual (SRMR), Comparative Fit Index (CFI), and the Tucker-Lewis Fit index (TLI). Decision criteria for determining model fit well were as follows: RMSEA ≤ 0.60, SRMR ≤ 0.80 (Hu & Bentley, 1999), CFI and TLI ≥ 0.90 (Raykov & Maroulides, 2000).

Item, factor and model level indices were also calculated to assess the dimensionality of the bifactor model. Specifically, item explained common variance (I-ECV) were calculated at the item level. At the factor level, explained common variance of a specific factor with respect to itself (ECV_SPEC), overall omega (ω; the variance in a score due to all sources of common variance), omega hierarchical for the total score (ωh; the variance in a score due to a common factor), and omega hierarchical for subscales (ωh/subscale; the variance in a score due to the intended group factor only), were calculated. Consistent with Reise, Bonifay, Haviland, (2013a) and endorsed by Watkins (2017), ωh and ωh/subscale estimates should be .50 or greater and preferably above .75.² At the model level, explained common variance of the general factor (ECV_GEN) and proportion of uncontaminated correlations (PUC), were calculated.

Attitudes

Given that this was the first time that school psychologists’ attitudes about standardized assessment were evaluated using this measure, we first described school psychologists’ responses to the ASA-SP using measures of central tendency for each scale and item. Subsequently, we assessed whether ASA-SP total and subscale scores were trivially different from a neutral response (a score of 3 on a 5-point scale) using a series of one-sided Welch’s t-tests. In each condition, the statistical hypothesis was that the mean score > 3. To adjust for multiple comparisons to a neutral value, we used false discovery correction and controlled for the completion of 4 Welch’s t-tests (Benjamini & Hochberg, 1995). All comparisons were completed using the rstatix package (version 0.7.0; Kassambara, 2021) in R.

Results

Research Question 1: Measurement properties of the Modified ASA Scale

Following our analysis procedure, the model converged normally, with adequate-to-good model fit across indicators (Brown, 2015). Robust χ²(206) = 481.324, p < .00 was not surprising; however, robust estimates of the comparative fit index (CFI = .887) and Tucker–Lewis Index (TLI = .874) were both just below the desired .90 criteria (Raykov & Maroulides, 2000). Root mean square error of approximation (RMSEA = .063 [.056 to .070]) was outside of our goal range (i.e., below .060; Hu & Bentler, 1999) and Standardized Root Mean Square Residual (SRMR = .075) was just under the .080 criteria. While likely adequate for the purposes of this project, we supposed that the residual covariances used by Jensen–Doss and Hawley (2010) suggested shared variance within the model beyond the scale-level factors that may support a total score and reduce the information matrix further. As such, we fit a bifactor model with the three subscales uncorrelated with one another and introduced a breadth factor, which was also uncorrelated to the other factors. For the bifactor model, robust χ²(187) = 325.625, p < .00, CFI = .947, TLI = .935, RMSEA = .045 [.037 to .053], and SRMR = .060, which suggests that the bifactor model fit the observed data matrix well—and perhaps slightly better.

Assessment of dimensionality indicted a unidimensional model, ECV_GEN = .63, ωh = .75, PUC = .88 (Reise, Scheines, Widaman, 2013b). Factor level coefficients of dimensionality and internal consistency are in Table 1. The general factor accounted for the majority of common variance (65%), while Practicality accounted for the next highest amount of ECV_SPEC (15%) followed by Psychometric Quality (13%). Benefit over Clinical Judgment explained a modest proportion of variance (bib_Stucky_and_Edelen_2014Stucky & Edelen, 2014). While no rules of thumb exist overall, each individual factor demonstrated adequate reliability (ω ≥ .50; Reise, Bonifay, & Haviland, 2013); the ωh and ωh/subscale coefficients suggest that only the General factor was reliable, and that the subscales displayed poor internal consistency after accounting for the general factor (Reise, Scheines, Widaman, 2013b). Furthermore, at the item level, 9 of the 22 items (41%) suggested a unidimensional item set (I-ECV > .80; Stucky & Edelen, 2014). Therefore, while the bifactor model suggested the best fit, the ASA-SP could be considered a unidimensional measure.

Table 1.

Indices of Dimensionality for the Bifactor Model.

	ECV_SPEC %	ω	ωH/ω(h)
General factor	64	.87	.86
Benefit over clinical judgment	7	.67	.13
Psychometric quality	13	.79	.19
Practicality	15	.74	.45

Notes. ECV_SPEC = explained common variance for specific factors; ω = omega; ωH/ω(h) = omega hierarchical related to General Self-Efficacy factor; values range from 0 to 1.

Research Question 2: Description and equivalency of ASA scores & items

The modified ASA was completed in full by 366 participants; four participants skipped one item and one participant missed two items. In total only six missing data points occurred on the ASA-SP. The mean and standard deviation of each scale and item are included in Table S2 while the density distribution of total and subscales are visualized in Figure 1. The mean and medians for all four scales were above 3.0; however, the density charts indicate that a number of participants reported neutral or positive attitudes toward standardized assessment.

Figure 1.

Density distributions of total and subscales.

To assess whether the participants’ attitudes were greater than a neutral position (Likert scale = 3), we conducted a series of four one-sided Welch’s t-tests with false discovery rate correction. These results are presented in Table 2. Differences between the average scale score and 3 ranged between 0.47 and 1.19, and were all statistically significant (p < .01). The effect sizes for these comparisons were also large, with a range between 0.81 and 2.90.

Table 2.

Welch’s T-Test Results for Each Scale Compared to Neutral Position.

Scale	n	T	df	d [95% CI]	Mean (SD)	Difference
Benefit over clinical judgment	371	15.58	370	0.81[.69, .94]	3.47 (0.58)	0.47
Psychometric quality	367	55.88	369	2.90 [2.64, 3.22]	4.20 (0.48)	1.19
Practicality	370	24.06	366	1.26 [1.13, 1.42]	3.60 (0.42)	0.61
Total score	366	37.93	365	1.98 [1.79, 2.22]	3.76 (0.38)	0.76

Note. Each t-test was statistically significantly at the p < .01 level after false discovery rate correction (Benjamini & Hochberg, 1995). The difference was calculated as the mean(scale)—3. Mean and standard deviation are reproduced here for clarity.

Discussion

The National Association for School Psychologists (NASP, 2020) and the American Psychological Association (APA, 2008) both advocate for the use of evidence-based practice (EBP) by their constituents. Research on the use of EBP in assessment (also known as EBA) amongst school psychologists is limited. This is unfortunate because EBA is vital for accuracy in labeling and diagnosis, which in turn allows practitioners to intervene with the children they serve. Effective assessment practices lead to more effective, intervention practices (Jensen–Doss & Hawley, 2010; Norcross et al., 2006; Youngstrom & Van Meter, 2016). Measuring perceptions are important because beliefs impact practice. While numerous studies examining clinical psychologists’ perception of EBA practices regarding standardized assessment exist, limited research on this topic has been published in our field.

A likely reason for this gap in the literature is the lack of psychometrically adequate measure of school psychologists’ perceptions of standardized assessment. The current study sought to address this need. The first stage of this study explored the psychometric qualities of an EBA research tool adapted for use with school psychologists (ASA-SP). Results using a solely school psychology sample did not strongly support Jensen–Doss and Hawley’s (2010) correlated traits model. A bifactor model, which posits the existence of an overall scale and subscales, as well as direct relationships between items and scales, and items and subscales (Reise et al., 2010) was also examined. Results from this model suggest that an overall ASA-SP score may be the best indicator of a school psychologists’ perception of standardized assessment. This is logical as most psychological constructs are complex and heterogeneous content items are generally required to measure a single construct (Reise et al., 2010).

However, there may be utility in using the subscales, particularly Practicality and Psychometric Quality, for research purposes. If researchers are interested in examining specific perceptions pertaining to these domains, this may be permissible as the bifactor model suggests an overall ASA-SP factor and specific factors. Due to low-to-moderate omega coefficients, researchers should interpret the Benefit Over Clinical Judgment subscale with caution as the variance in the score attributable only to participant’s perceptions regarding an instrument’s utility beyond clinical judgment alone was below recommended levels (Reis, Bonifay, Haviland, 2013a; Watkins, 2017). If researchers do choose to examine any of these subscales, they must do so with the recognition that these domains strongly represent overall ASA-SP.

In general, ratings from the ASA-SP indicate that school psychologists reported positive attitudes towards the use of standardized assessments. This would seem reasonable as school psychologists frequently use standardized measures (Benson et al., 2019; Farmer et al., 2021) and assessment using standardized measures makes up a large portion of a school psychologists’ training (Lockwood & Farmer, 2020). Furthermore, our sample reported significantly more positive attitudes on the ASA than Jensen–Doss and Hawley’s (2010), sample did on the original ASA across all scales especially Benefits over Clinical Judgment domain. However, because of modification to items and changes to the sample, direct comparisons are avoided.

Limitations

There are several limitations of this study. One pertains to the use of survey methodology. Survey research is prone to erroneous data due to participants responding in ways that they believe are socially desirable (e.g., social desirability). To control for this, we administered the survey anonymously and presented demographic items at the end of the survey. Survey participants may have misunderstood items, which may have led to inaccurate data. To counteract this, we started with Jenson–Doss and Hawley’s (2010)Jensen–Doss and Hawley’s (2010) survey and used a content expert panel and piloting to minimize error of this kind.

Another limitation involves the psychometric properties of the ASA-SP. As previously noted, the moderate omega values observed for the Benefits of Clinical Judgment subscale. A close inspection of the factor loadings of the items in the ASA scale suggested that a number of items have loadings below .30. Future researchers may wish to examine the ASA-SP with these items removed from the scale to improve its reliability and validity. We chose to leave the items in the scale for this project to approximate the original ASA (Jensen–Doss & Hawley, 2010) as closely as possible. Despite these possible limitations, we believe that this first study examining the perception of school psychologists’ perceptions is important because perceptions are indicators of practice (Ajzen, 1991).

In addition, there are possible generalizability concerns in the current study. All participants in this study were members of the National Association of School Psychologists (NASP). Active NASP members available to survey make up approximately 40% of all school psychologists, and several studies have documented that NASP members may not be representative of all school psychologists (Jimerson et al., 2009; Lewis et al., 2008; Goforth et al., 2021). In particular, non-members tend to be more diverse and use assessments in different ways than members. Given that assessment use is the crux of this study, findings may only be applicable to NASP members. In this way, these data represent a convenience sample, though one we believe is useful as a benchmark for establishing school psychologist’s attitudes toward standardized assessment. Future research may wish to survey both NASP and non-NASP members to examine the psychometric properties of the ASA-SP with a more representative sample.

Future Research and Implications

In addition to further examining the psychometric properties of the ASA-SP with a more representative sample, researchers may also wish to collect data regarding school psychologists’ assessment practices along with the responses to the ASA-SP. This may provide insightful information about the extent to which perceptions of EBA predict EBP. If the two are found to be related, trainers could create targeted interventions to improve trainees’ perceptions and, therefore, EBA practices. Researchers may also include survey items to determine how social pressure (i.e., pressure to practice consistent with norms) influences school psychologists’ choices to use EBA practices. Social pressure was one area of the TPB (Ajzen, 2005; Cooper et al., 2019) that was not evaluated in either the original Jensen–Doss and Hawley (2010) study or the current study, but is understood to be a factor in maintaining low-value practices (Montini & Graham, 2015). Understanding the role of social pressure could improve training/practice by providing information about how social norms impact EBA practices among trainees and practitioners. If social pressure is found to play a role in EBA, then interventions could be created to change the social norms of training programs to ones that promote EBA practices.

Additionally, studies may examine how assessment policy differences impact perceptions of standardized assessment. For example, it would appear informative to examine whether school psychologists from districts that require full assessment batteries during each special education evaluation have higher ASA-SP scores than practitioners from districts/schools that mandate the use of file reviews whenever possible. Additionally, researchers may wish to examine whether school psychologists in districts that mandate lower assessment caseloads have different ASA-SP scores than those whose districts required heavy ones. If differences in ASA-SP scores are associated with these variables, then this would have training implications. Specifically, training programs that wish to improve perceptions of EBA may wish to place students in districts with policies that are most conducive with higher ASA-SP scores.

Relatedly, training programs may find the ASA-SP useful in measuring their ability to improve perceptions of EBA throughout training. To do so, programs could administer the ASA-SP at various times throughout training (e.g., pre-practicum, after a year of practicum, and pre- or post-internship). Trainers could use these ASA-SP scores to adjust the practical experiences of individuals, cohorts, or of their program as a whole. However, as the ASA-SP has not been validated for use with trainees, future research should first examine the psychometric properties of the ASA-SP using a solely trainee sample.

Supplemental Material

sj-pdf-1-jpa-10.1177_07342829211057642 – Supplemental Material for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Supplemental Material, sj-pdf-1-jpa-10.1177_07342829211057642 for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools by Adam B. Lockwood, Ryan L. Farmer and S. Kathleen Krach in Journal of Psychoeducational Assessment

Supplemental Material

sj-pdf-2-jpa-10.1177_07342829211057642 – Supplemental Material for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Supplemental Material, sj-pdf-2-jpa-10.1177_07342829211057642 for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools by Adam B. Lockwood, Ryan L. Farmer and S. Kathleen Krach in Journal of Psychoeducational Assessment

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iDs

Adam B. Lockwood

Ryan L. Farmer

S. Kathleen Krach

Supplementary Material

Supplementary material for this article is available online.

Notes

References

Ajenz

(2005). Attitudes, personality, and behavior (2nd ed.). McGraw-Hill Education.

Ajzen

(1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T

American Psychological Association . (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285. https://doi.org/10.1037/0003-066X.61.4.271

American Psychological Association . (2008). Disseminating evidence-based practice for children & adolescents: A systems approach to enhancing care. https://www.apa.org/practice/resources/evidence/children-report.pdf#page=1&zoom=auto, 39,466.

Benjamini

Y.,

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x

Benson

N. F.

Floyd

R. G.

Kranzler

J. H.

Eckert

T. L.

Fefer

S. A.

Morgan

G. B.

(2019). Test use and assessment practices of school psychologists in the United States: Findings from the 2017 National Survey. Journal of School Psychology, 72, 29–48. https://doi.org/10.1016/j.jsp.2018.12.004

Benson

Lockwood

Farmer

(2021, August). National survey of training and assessment practices. Presented at the American Psychological Association (APA), Annual Convention, Virtual.

Bianco

(2005). The effects of disability labels on special education and general education teachers’ referrals for gifted programs. Learning Disability Quarterly, 28(4), 285–293. https://doi.org/10.2307/4126967

Bianco

M.,

Leech

N. L.

(2010). Twice-exceptional learners: Effects of teacher preparation and disability labels on gifted referrals. Teacher Education and Special Education, 33(4), 319–334. https://doi.org/10.1177/0888406409356392

10.

Cook

J. M.

Schnurr

P. P.

Biyanova

Coyne

J. C.

(2009). Apples don’t fall far from the tree: Influences on psychotherapists’ adoption and sustained use of new therapies. Psychiatric Services, 60(5), 671–676. https://doi.org/10.1176/appi.ps.60.5.671

11.

Cooper

Barkatsas

Strathdee

(2019). The theory of planned behaviour (TPB) in educational research using structural equation modelling (SEM) Barkatsas

Bertram

, (Eds.), (Vol. 1, pp. 139–162). Brill Sense.

12.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method (4th ed.). Wiley.

13.

Dombrowski

S. C.

McGill

R. J.

Farmer

R. L.

Kranzler

J. H.

Canivez

G. L.

(2021). Beyond the rhetoric of evidence-based assessment: A framework for critical thinking in clinical practice. School Psychology Review. Advance online publication. https://doi.org/10.1080/2372966X.2021.1960126

14.

Eysenbach

(2004). Improving the quality of web surveys: The checklist for reporting results of internet e-surveys (CHERRIES). Journal of Medical Internet Research, 6(3), e34. https://doi.org/10.2196/jmir.6.3.e34

15.

Farmer

R. L.

Goforth

A. N.

Kim

S. Y.

Naser

S. C.

Lockwood

Affrunti

(2021). Status of school psychology in 2020: Part 2, professional practices in the NASP membership survey. 5(3), 1–17. https://www.nasponline.org/Documents/Research%20and%20Policy/Research%20Center/RR_NASP-2020-Membership-Survey-part-2.pdf

16.

Goforth

A. N.

Farmer

R. L.

Kim

S. Y.

Naser

S. C.

Lockwood

Affrunti

(2021). Status of school psychology in 2020: A look at the NASP membership survey. NASP Research Reports. NASP Research Reports, 5(2), 1–17. https://www.nasponline.org/Documents/Research%20and%20Policy/Research%20Center/NRR_2020-Membership-Survey-P1.pdf

17.

Gross

T. J.

Farmer

R. L.

Ochs

S. E.

(2019). Evidence-based assessment: Best practices, customary practices, and recommendations for field-based assessment. Contemporary School Psychology, 23(3), 304–326. https://doi.org/10.1007/s40688-018-0186-x

18.

Higgins

E. L.

Raskind

M. H.

Goldberg

R. J.

Herman

K. L.

(2002). Stages of acceptance of a learning disability: The impact of labeling. Learning Disability Quarterly, https://doi.org/10.2307/1511187

19.

Hojnoski

R. L.

Morrison

Brown

Matthews

W. J.

(2006). Projective test use among school psychologists: A survey and critique. Journal of Psychoeducational Assessment, 24(2), 145–159. https://doi.org/10.1177/0734282906287828

20.

Hunsley

Lee

C. M.

Wood

J. M.

Taylor

(2015). Controversial and questionable assessment techniques. In Lilienfeld

S. O.

Lynn

S. J.

Lohr

J. M.

(Eds.), Science and pseudoscience in clinical psychology (pp. 42–82). The Guilford Press.

21.

Jensen-Doss

Haimes

E. M. B.

Smith

A. M.

Lyon

A. R.

Lewis

C. C.

Stanick

C. F.

Hawley

K. M.

(2018). Monitoring treatment progress and providing feedback is viewed favorably but rarely used in practice. Administration and Policy in Mental Health and Mental Health Services Research, 45(1), 48–61. https://doi.org/10.1007/s10488-016-0763-0

22.

Jensen-Doss

A.,

Hawley

K. M.

(2010). Understanding barriers to evidence-based assessment: Clinician attitudes toward standardized assessment tools. Journal of Clinical Child & Adolescent Psychology, 39(6), 885–896. https://doi.org/10.1080/15374416.2010.517169

23.

Jensen-Doss

A.,

Hawley

K. M.

(2011). Understanding clinicians’ diagnostic practices: Attitudes toward the utility of diagnosis and standardized diagnostic tools. Administration and Policy in Mental Health and Mental Health Services Research, 38(6), 476–485. https://doi.org/10.1007/s10488-011-0334-3

24.

Jimerson

S. R.

Stewart

Skokut

Cardenas

Malone

(2009). How many school psychologists are there in each country of the world?: International estimates of school psychologists and school psychologist-to-student ratios. School Psychology International, 30(6), 555–567. https://doi.org/10.1177/0143034309107077

25.

Kassambara

(2021). Rstatix: Pipe-friendly framework for basic statistical tests. [Computer Software]. https://cran.r-project.org/web/packages/rstatix/index.html

26.

Kranzler

J. H.

Maki

K. E.

Benson

N. F.

Eckert

T. L.

Floyd

R. G.

Fefer

S. A.

(2020). How do school psychologists interpret intelligence tests for the identification of specific learning disabilities? Contemporary School Psychology, 24(4), 445–456. https://doi.org/10.1007/s40688-020-00274-0

27.

Kratochwill

T. R.,

Stoiber

K. C.

(2002). Evidence-based interventions in school psychology: Conceptual foundations of the Procedural and Coding Manual of Division 16 and the Society for the Study of School Psychology Task Force. School Psychology Quarterly: The Official Journal of the Division of School Psychology, American Psychological Association, 17(4), 341. https://psycnet.apa.org/record/2003-01789-005

28.

Lee

Cerreto

F. A.

Lee

(2010). Theory of planned behavior and teachers’ decisions regarding use of educational technology. Journal of Educational Technology & Society, 13(1), 152–164. https://www-jstor-org.web.bisu.edu.cn/stable/jeductechsoci.13.1.152.

29.

Lewis

M. F.

Truscott

S. D.

Volker

M. A.

(2008). Demographics and professional practices of school psychologists: A comparison of NASP members and non–NASP school psychologists by telephone survey. Psychology in the Schools, 45(6), 467–482. https://doi.org/10.1002/pits.20317

30.

Lilienfeld

S. O.

Lynn

S. J.

Bowden

S. C.

(2018). Why evidence-based practice isn’t enough: A call for science-based practice. The Behavior Therapist/AABT, 41(1), 42–47. https://doi.org/10.1037/pro0000235

31.

Lindemann

(2021, August, 9). What’s the average survey response rate? [2021 benchmark]. Survey Anyplace. https://surveyanyplace.com/blog/average-survey-response-rate/

32.

Lockwood

Benson

Farmer

Klatka

(2022). Test use and assessment practices of school psychology training programs: Findings from a 2020 survey of U.S. faculty. Psychology in the Schools. https://doi.org/10.1002/pits.22639.

33.

Lockwood

A. B.,

Farmer

R. L.

(2020). The cognitive assessment course: Two decades later. Psychology in the Schools, 57(2), 265–283. https://doi.org/10.1002/pits.22298

34.

Lockwood

A. B.

Gross

T. J.

Farmer

R. L.

Loke

S. W.

(2019). Huber inventory of trainee self-efficacy: Assessment of measurement models. Psychology in the Schools, 56(6), 992–1003. https://doi.org/10.1002/pits.22230

35.

MacFarlane

K.,

Woolfson

L. M.

(2013). Teacher attitudes and behavior toward the inclusion of children with social, emotional and behavioral difficulties in mainstream schools: An application of the theory of planned behavior. Teaching and Teacher Education, 29, 46–52. https://doi.org/10.1016/j.tate.2012.08.006

36.

McGill

R. J.

(2019). Evidence-based assessment in school psychology: An introduction. Contemporary School Psychology, 23(2), 113–114. https://doi.org/10.1007/s40688-019-00240-5

37.

McGill

R. J.

Dombrowski

S. C.

Canivez

G. L.

(2018). Cognitive profile analysis in school psychology: History, issues, and continued concerns. Journal of School Psychology, 71, 108–121. https://doi.org/10.1016/j.jsp.2018.10.007

38.

Miller

L. T.

Bumpus

E. C.

Graves

S. L.

(2020). The state of cognitive assessment training in school psychology: An analysis of syllabi. Contemporary School Psychology. Advance online publication. https://doi.org/10.1007/s40688-020-00305-w

39.

Minner

(1989). Initial referral recommendations of teachers toward gifted students with behavioral problems. Roeper Review, 12(2), 78–80. https://doi.org/10.1080/02783198909553240

40.

Minner

(1990). Teacher evaluation of case descriptions of LD gifted children. Gifted Child Quarterly, 4(1), 37–39. https://doi.org/10.1177/001698629003400108

41.

Montini

T.,

Graham

I. D.

(2015). “Entrenched practices and other biases”: Unpacking the historical, economic, professional, and social resistance to de-implementation. Implementation Science: IS, 10, 24. https://doi.org/10.1186/s13012-015-0211-7

42.

National Association of School Psychologists . (2009). School psychologists’ involvement in assessment (Position Statement). Bethesda, MD: National Association of School Psychologists. https://www.nasponline.org/assets/Documents/Research%20and%20Policy/Position%20Statements/Involvement_in_Assessment.pdf

43.

National Association of School Psychologists . (2016). School psychologists’ involvement in assessment (Position Statement). Bethesda, MD: National Association of School Psychologists. https://www.nasponline.org/x36851.xml

44.

National Association of School Psychologists (2020). NASP 2020 professional standards https://www.nasponline.org/standards-and-certification/nasp-2020-professional-standards-adopted

45.

National Association of School Psychologists (NASP) . (1986). Rights without labels: A position statement. Executive Board / Delegate Assembly of the National Association of School Psychologists.

46.

National Institute of Mental Health strategic plan for research (2020). DHHS Publication No. 15–6368. https://www.nimh.nih.gov/about/strategic-planning-reports/index.shtml

47.

Norcross

J. C.

Beutler

L. E.

Levant

R. F.

(2006). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. American Psychological Association.

48.

Nulty

D. D.

(2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33(3), 301–314. https://doi.org/10.1080/02602930701293231

49.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013a). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437

50.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559. https://doi.org/10.1080/00223891.2010.496477

51.

Reise

S. P.

Scheines

Widaman

K. F.

Haviland

M. G.

(2013b). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26. https://doi.org/10.1177/0013164412449831

52.

Renzulli

J. S.,

Park

(2000). Gifted dropouts: The who and the why. Gifted Child Quarterly, 44(4), 261–271. https://doi.org/10.1177/001698620004400407

53.

Rolison

M. A.,

Medway

F. J.

(1985). Teachers’ expectations and attributions for student achievement: Effects of label, performance pattern, and special education intervention. American Educational Research Journal, 22(4), 561–573. https://doi.org/10.3102/00028312022004561

54.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02

55.

RStudio Team . (2020). RStudio: Integrated development for R. RStudio. http://www.rstudio.com/

56.

Sotelo-Dynega

M.,

Dixon

S. G.

(2014). Cognitive assessment practices: A survey of school psychologists. Psychology in the Schools, 51(10), 1031–1045. https://doi.org/10.1002/pits.21802

57.

Stucky

B. D.,

Edelen

M. O.

(2014). Using hierarchical IRT models to create unidimensional measures from multidimensional data. In Reise

S. P.

Revicki

D. A.

(Eds.). Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183-206). Routledge/Taylor & Francis Group.

58.

Swann

W.,

Snyder

(1980). On translating beliefs into action: Theories of ability and their application in an instructional setting. Journal of Personality and Social Psychology, 38(6), 879–888. https://doi.org/10.1037/0022-3514.38.6.879

59.

Thorndike

R. M.,

Thorndike-Christ

(2010). Measurement and evaluation in psychology and education. Prentice Hall. https://play.google.com/store/books/details?id=-1oASgAACAAJ

60.

Watkins

M. W.

(2017). The reliability of multidimensional neuropsychological measures: from alpha to omega. The Clinical Neuropsychologist, 31(6-7), 1113–1126. https://doi.org/10.1080/13854046.2017.1317364

61.

Woodcock

S.,

Vialle

(2011). Are we exacerbating students’ learning disabilities? An investigation of preservice teachers’ attributions of the educational outcomes of students with learning disabilities. Annals of Dyslexia, 61(2), 223–241. https://doi.org/10.1007/s11881-011-0058-9

62.

Yoshida

Chipman

J. J.

Bohn

McGowan

L. D.

Barrett

Christensen

R. H. B.

(2020). Create “Table 1” to describe baseline characteristics with or without propensity score weight. (0.12.0) [Computer software]. https://github.com/kaz-yos/tableone

63.

Youngstrom

E. A.

(2013). Future directions in psychological assessment: combining evidence-based medicine innovations with psychology’s historical strengths to enhance utility. Journal of Clinical Child and Adolescent Psychology, 42(1), 139–159. https://doi.org/10.1080/15374416.2012.736358

64.

Youngstrom

E. A.,

Van Meter

(2016). Empirically supported assessment of children and adolescents. Clinical Psychology: Science and Practice, 23(4), 327–347. https://doi.org/10.1111/cpsp.12172

65.

Zint

(2002). Comparing three attitude-behavior theories for predicting science teachers’ intentions. Journal of Research in Science Teaching, 39(9), 819–844. https://doi.org/10.1002/tea.10047

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.10 MB

0.21 MB

Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Abstract

Keywords

Evidence-Based Assessment

Perceptions of Assessment

Current Study

Attitudes towards Standardized Assessment scales for School Psychologists (ASA-SP) Psychometrics

Findings from ASA-SP

Methods

Recruitment & Participants

Instrument

Survey Design

Analyses

R1, Measurement

Attitudes

Results

Research Question 1: Measurement properties of the Modified ASA Scale

Research Question 2: Description and equivalency of ASA scores & items

Discussion

Limitations

Future Research and Implications

Supplemental Material

sj-pdf-1-jpa-10.1177_07342829211057642 – Supplemental Material for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Supplemental Material

sj-pdf-2-jpa-10.1177_07342829211057642 – Supplemental Material for Examining School Psychologists’ Attitudes Toward Standardized Assessment Tools

Footnotes

Declaration of Conflicting Interests

Funding

ORCID iDs

Supplementary Material

Notes

References

Supplementary Material

R₁, Measurement