Abstract
Given the ever-evolving role of school counselors and increasing demands placed on them in today’s sociopolitical climate, proficiency in program evaluation is more important than ever as school counselors advocate for their role and for comprehensive school counseling program delivery in schools. This article presents a reexamination of the purpose and importance of program evaluation and the development of an evaluator identity. This quantitative study investigated which factors contribute to school counselors’ identity as a program evaluator using self-report survey data on levels of self-efficacy, confidence, perceived importance of program evaluation, and interest in program evaluation. It also assessed the overall significance of each factor for engaging in program evaluation behaviors, as operationalized through data-based decision-making practices. Results include a predictive model that displays the contribution and significance of these factors on school counselor program evaluation and related data-based decision-making behaviors.
Introduction
School counselors deliver comprehensive school counseling programs within their educational settings, consisting of a complex array of services to promote students’ academic, career, and social/emotional development (ASCA, 2019; Carey et al., 2019). As such, program evaluation for school counselors has been a consistent topic of discourse within the profession for more than 3 decades (Astramovich & Coker, 2007; Carey et al., 2019; Dimmitt, 2009). School counselors, as educational leaders, face growing expectations to provide evidence of their accountability, and a salient way for school counselors to do this is through the ongoing process of program evaluation (Dimmitt, 2009; Sink, 2009). Rossi and colleagues (2018) broadly defined the discipline of program evaluation as the “application of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are designed to inform social action to improve social conditions” (p. 321). By nature, program evaluation is transdisciplinary (Scriven, 2008) and “as different fields incorporate evaluation into the repertoire of professional practice, different ideas are developed about what constitutes evaluation and for what purposes it should be used” (Trevisan & Carey, 2020, p. 2).
The standards for competent program evaluation in school counseling have yet to be established (Carey et al., 2019). Despite the clear and consistent call for school counselors to engage in program evaluation practices, the vision for what program evaluation looks like for school counseling as expressed by the American School Counseling Association (ASCA), the Council for Accreditation in Counseling and Related Educational Programs (CACREP), and state licensure examinations is, at best, “ambiguous and unconnected to best program evaluation practices” (Trevisan et al., 2020, p. 141). After decades of theorizing, Carey and colleagues formally defined program evaluation in the context of school counseling in 2019 as the knowledge and expertise in program evaluation activities, which includes: (a) involving stakeholders, (b) developing evaluation instruments, (c) utilizing quantitative and qualitative evaluation approaches, (d) reporting evaluation results, (e) demonstrating evaluation soft skills, and (f) awareness of evaluation ethics.
Professional Identity and Roles of the School Counselor
The absence of “evaluator” in school counseling professional identity research creates role ambiguity and responsibility conflict. School counselors must vacillate between the roles of educator, counselor, leader, advocate, consultant, collaborator, advisor, and systemic change agent, sometimes all within the same hour (Levy & Lemberger-Truelove, 2021). School counselors also play the vital role of program evaluator by using data to make informed decisions about planning, implementing, and assessing their comprehensive school counseling programs as they attend to the mental health, academic, social/emotional, and postsecondary needs of all students (Zyromski et al., 2021).
In their seminal work, DeKruyf and colleagues (2013) called for a conjoint identity where school counselors proudly wear both the “educator” and “counselor” hats simultaneously. Levy and Lemberger-Truelove (2021) have since updated the construct of school counselor professional identity, proposing a unified, nondual, nonhierarchical identity of educator–counselor. The educator–counselor identity is composed of the many roles and responsibilities required by school counselors to successfully implement comprehensive school counseling programs and to promote equity and access for all students. These roles and responsibilities include (a) leadership and advocacy; (b) counseling; (c) instruction, appraisal, and advisement; (d) consultation; (e) collaboration; and (f) referrals. Unfortunately, even in this seemingly comprehensive identity model, the importance of evaluation and the essential role of evaluator in school counseling identity development is still absent.
ASCA contributes to the ambiguity surrounding school counselors’ use of program evaluation by using the term assess in place of evaluation. The responsibility to assess is largely framed around the practice of data-based decision making. Although data-based decision making not explicitly defined, the importance of these practices is prioritized through the application of the ASCA National Model as school counselors implement the four key components: (a) define, (b) manage, (c) deliver, and (d) assess in their comprehensive school counseling programs (ASCA, 2019; Zyromski et al., 2021). Zyromski and colleagues (2021) provided an updated definition of data-based decision making, defining it as a multi-step process which includes the following: a) examining data to discover the unique needs of students; b) matching interventions to the identified student needs; c) evaluating the impact of those interventions on immediate, proximal, and distal outcomes; and d) applying continuous improvement cycles in support of increases in student wellness and life success.” (p. 2)
Although program evaluation and data-based decision making are different concepts, in analyzing this definition, program evaluation knowledge and skills come to the forefront as key components in successfully implementing the four steps of data-based decision making. As with program evaluators in other fields, collecting and analyzing data is essential for school counselors as they seek to recognize opportunity and achievement gaps for students, advocate for the dismantling of racist practices and policies within the school, and create necessary systemic change within schools and communities (Beasley & Ieva, 2022).
Previous Research on School Counselors as Program Evaluators
Previous literature on this topic has focused on measuring evaluation competency (Maras et al., 2013); measuring skills and interests in program evaluation (Astramovich, 2016); current practices in program evaluation (Dimmitt, 2010); training models for teaching counselors in training (Astramovich & Coker, 2007; Hausheer, 2019); and school counselors’ current knowledge in program evaluation (Trevisan et al., 2020). Although articles frequently address program evaluation competence in school counseling (Astramovich, 2016; Hausheer, 2019; Köse, 2019; Maras et al., 2013), few if any research articles have focused on how program evaluation is being taught and integrated into professional identity through counselor education programs.
Köse (2019) began by performing a literature analysis to provide clarity to the status of program evaluation competencies for school counselors using the competency framework of Stevahn et al.’s, 2005 taxonomy: systematic inquiry, reflective practice, professional practice, situational analysis, project management, and interpersonal competence. Upon completing this analysis, Köse combined the comprehensive list of competencies identified from the literature and divided them into three main categories or domains. These domains are field-specific competencies, technical competencies, and nontechnical competencies, providing the foundation for the taxonomy for school counseling program evaluation. Trevisan and Carey adopted this proposed taxonomy and continued to build upon it in their seminal work, Program Evaluation in School Counseling (2020).
The current study builds upon Köse’s work and is grounded in Bandura’s Social Cognitive Learning Theory (SCLT; Bandura et al., 1994). Central to SCLT is the concept of self-efficacy (Bandura, 2010). According to Bandura (2010), self-efficacy beliefs determine how people feel, think, motivate themselves, and behave. Mark et al. (2011) found that people with high efficacy are more likely to view difficult tasks as something to be mastered rather than something to be avoided, while those with weak efficacy are more likely to avoid challenging tasks and to focus on personal failings and negative outcomes. Using prior literature, specifically Maras and colleagues (2013) and Astramovich and Coker (2007), this study explicitly focuses on factors central to self-efficacy and how they relate to school counselor program evaluation and data-based decision-making practices.
The purpose of the current study was to investigate whether school counselors’ feelings, thoughts, beliefs, and attitudes around program evaluation, as they relate to SCLT and the concept of self-efficacy, play a major role in their engagement in related behaviors. SCLT posits that behavior, in this case program evaluation practices of school counselors in training, is determined by the individual’s thoughts, feelings, attitudes, and beliefs. Consistent with my postpositivist worldview, I used nonexperimental, quantitative survey design to investigate the following two research questions: 1. What are practicing school counselors’ self-reported levels of (a) evaluation self-efficacy, (b) confidence in conducting program evaluation, (c) perceived importance of program evaluation, (d) interest in program evaluation, and (e) evaluation values and beliefs? 2. Can a predictive model be created to determine the degree to which these five factors contribute to overall school counselor program evaluation behaviors as operationalized by data beliefs and practices?
Methodology
Research Design
This nonexperimental, quantitative survey design investigated the two research questions stated above. The first research question was exploratory in nature. Because no literature has previously examined these variables together, a crucial step prior to conducting any further analyses was to first understand how practicing school counselors self-report in each of these areas. To analyze the degree to which counselors self-reported their own self-efficacy, confidence, perceived importance of program evaluation, values, and interest in program evaluation, I utilized descriptive statistics to analyze responses on each of these variables.
To investigate the second research question, I conducted a linear regression analysis to create a predictive model of each variable’s contribution to school counselor program evaluation beliefs and practices. In the linear regression, the predictor variables were measured using subscales from the Effective Practices Survey (EPS; Maras et al., 2013) and the Program Evaluation Interest and Skills Assessment (PEISA; Astramovich, 2016) as described in Research Question 1. The criterion variable, school counselors’ data beliefs and practices, was measured using the Data and Accountability Beliefs and Practices Survey (DABAPS; Young & Kaffenberger, 2015).
Population
A power analysis was conducted to determine the necessary sample size for statistical significance with a confidence interval of p = .05 and a medium effect size. Using G*Power analysis with a medium effect size of .15, I identified that 134 participants were needed for significance in the multiple regression.
I used convenience and snowball sampling methods to recruit more than 134 participants, including posting recruitment fliers on social media platforms, emailing school counselors listed in school district directories, and emailing members of the state’s school counselor association using the published member directory. The sample included school counselors at the elementary, middle, and high school level in both public and private institutions statewide. Criteria for inclusion in the study included any school counselor who: (a) holds a master’s degree (or equivalent) in school counseling, (b) currently practices as a school counselor, and (c) currently works in this southeastern state.
Data Collection
Prior to participant recruitment and data collection, a university institutional review board (IRB) reviewed and approved the study. Upon gaining IRB approval, I began recruiting participants. Participant identities were kept anonymous and I collected only necessary demographic information. Upon agreeing to participation, the subjects in this study were asked to complete the following: (a) demographic questionnaire, (b) the PEISA (Astramovich, 2016), (c) the EPS (Maras et al., 2013), and (d) the DABAPS (Young & Kaffenberger, 2015). These assessments were compiled and available for participant completion using the data software system Qualtrics. The email invitation for participation included a web link to access the informed consent and online surveys.
The data was screened for completion prior to statistical analyses. All data were entered and analyzed using IBM SPSS Statistics (version 26), a software package commonly used in the social sciences. All demographic information and other qualitative data including age, race, gender, etc., were coded and entered quantitatively. The total scores and subscale scores from the PEISA, EPS, and DABAPS were collected for analysis in SPSS.
Sample
Of the 148 school counselors who began the survey instruments, one respondent did not complete the demographic questionnaire, resulting in a total sample of 147 participants (N = 147). Demographic questions included: age, gender, race/ethnicity, school type (i.e., public, private, charter), location (i.e., urban, rural, suburban), highest degree completed, number of years as a practicing school counselor, school’s current RAMP status, and grade levels served.
Instrumentation
Program Evaluation Interests and Skills Assessment
The PEISA is a 20-item instrument that assesses school counselors’ interests, skills, and perceived importance of program evaluation. The instrument uses a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree). The PEISA comprises four subscales: (a) Program Evaluation Skill Level, (b) Interest in Program Evaluation, (c) Program Evaluation Training Importance, and (d) Confidence in Conducting Program Evaluation. The scoring for the PEISA includes an individual score for each subscale.
For the purposes of this study, only the second, third, and fourth subscales were used, with individual scores for each of these subscales analyzed and reported. The highest possible score on each subscale is as follows: Interest in Program Evaluation, 25; Training Importance, 20; and Confidence in Program Evaluation, 15.
The Interest in Program Evaluation subscale contains eight items with a Cronbach’s α = .91. Sample items from this subscale are “I am interested in conducting an evaluation of my school counseling program” and “I am interested in developing my skills in program evaluation.” The Program Evaluation Training Importance subscale contains four items (Cronbach’s α = .92). Sample items include “School counselor education programs should train students to conduct program evaluations” and “Future school counselors should understand how to evaluate counseling programs.” The last subscale, Confidence in Program Evaluation, contains three items with a Cronbach’s α = .81. Sample items include “I am confident in my current ability in program evaluation” and “I understand the process of conducting an evaluation of my school counseling program.”
Effective Practices Survey
Maras and colleagues (2013) developed the EPS to “provide a complete and comprehensive measure of evaluation competency” (p. 102). The EPS was developed by adapting the Brief Self-Assessment of Essential Competences Required for Evidence-Based School Counseling (Dimmitt et al., 2007), by referencing existing literature on program evaluation, and through ongoing consultation with experts in the field. The EPS has 19 items that can be subdivided into four subscales. The items were assessed using confidence anchors of 1 (very unconfident/strongly disagree) to 6 (very confident/strongly agree). The EPS subscales are: (a) Evaluation Self-Efficacy, (b) Best Practices in Guidance Programs, c) Statistics, and d) Evaluation Values and Beliefs.
For the purposes of this study, I analyzed only the first and fourth subscales, and provided individual descriptive statistics for these two subscales. The highest possible scores are 42 on Evaluation Self-Efficacy and 18 on Evaluation Values and Beliefs. The first factor, Evaluation Self-Efficacy, consists of seven items with a reliability coefficient of α = .94. Sample items from this first subscale are: “I can use school data to identify student strengths and needs” and “I can develop a survey and use its results to make my guidance program better.” The fourth subscale, Evaluation Values and Beliefs, has a reliability coefficient of α = .92. This subscale includes only two items: “I believe evaluation leads to better student outcomes” and “I believe evaluation is an important ongoing activity for school counselors.”
Data and Accountability Beliefs and Practices Survey
The Data Beliefs and Practices Survey (DBAPS) was developed by Young and Kaffenberger in 2011 to better understand the data beliefs and practices of school counselors who have earned for their program the Recognized ASCA Model Program (RAMP) designation. The DBAPS is a 21-item survey measured using a 7-point Likert-type scale with the anchors ranging from 1 (strongly disagree) to 7 (strongly agree).
The first factor, consisting of 13 items, assesses school counselors’ perceptions of data practices. The second factor contains five qualitative items measuring school counselors’ data training and data sharing practices. The final two items are open-ended questions assessing school counselors’ motivation to use data and their understanding of the purpose of data (Young & Kaffenberger, 2011). Young and Kaffenberger were approached by ASCA for permission to utilize the DBAPS in training programs for school districts across the United States. After granting this approval, Young and Kaffenberger later revised the DBAPS in 2015 to include additional items to make the measure more comprehensive. The revised version of the DBAPS was renamed the Data and Accountability Beliefs and Practices Survey (DABAPS). An exploratory factor analysis was conducted to examine the weight of the factors. Although the DABAPS now contains six additional items, no additional factors were added to the revised instrument, strengthening the reliability of the original instrument.
The DABAPS version of the instrument has not been published in a journal. However, the face validity of the instrument remains very strong: It continues to be a valuable practitioner-based instrument used by ASCA in their ASCA National Model trainings and in creating the School Counselor Performance Appraisal self-assessment published in the fourth edition of the ASCA National Model book. This study utilized the revised DABAPS instrument as an outcome measure containing 19 total quantitative items.
Results
Participant Characteristics
The sample of school counselors (N = 147) included 21 males, 124 females, and 2 individuals who did not disclose a gender identity. Females represented 83.8% of the sample and males represented 14.2% of the sample. The age range was 24–70 years old, with a mean age of 40 (M = 40, SD = 10.66). The reported ethnicity of participants was 79 White (53.4%), 60 Black/African American (40.5%), 2 Hispanic (1.4%), and 1 Asian (0.7%); 1 respondent (0.7%) preferred to self-identify and 4 (2.7%) preferred not to disclose.
Regarding preparation, 67 (45.3%) participants held a masters’ degree, 56 (37.8%) earned a specialists’ degree, 15 (10.1%) earned a doctorate in education, and 9 (6.1%) held a doctorate of philosophy. Respondents’ average completed years of experience as a school counselor was 10.82 (SD = 7.96, range = 0–50, Mdn = 8, Mo = 7).
School Characteristics
The school counselor respondents’ reported school type included 141 (95.3%) public, 3 (2%) private and 3 (2%) charter schools. The reported school levels included 60 (40.5%) at the elementary school level, 48 (32.4%) at the high school level, 27 (18.2%) at the middle school level, 5 (3.4%) in K–12 settings, 2 (1.4%) in K–8 settings, and 5 (3.4%) other. In terms of school demographic environment, 94 (63.5%) respondents indicated suburban, 38 (25.7%) urban, and 15 (10.1%) rural. Of the respondents’ school counseling programs and RAMP, 78.4% (n = 116) did not previously or currently hold the RAMP designation, 7.4% (n = 11) previously earned RAMP but did not re-RAMP, 6.8% (n = 10) were currently RAMP programs, and 6.8% (n = 10) were preparing to apply for RAMP.
Research Question 1
Descriptive Statistics for Each Factor Subscale.
Note. PE = program evaluation.
Research Question 2
Multiple Regression Predicting School Counselors’ Data Accountability Beliefs and Practices.
Note. PE = program evaluation, N = 147.
Discussion
The purpose of the current study was to investigate whether school counselors’ self-efficacy, as measured by their feelings, thoughts, beliefs, and attitudes around program evaluation, are related to their engagement in data-based decision-making behaviors. The study had two primary aims described in the two research questions: 1. What are practicing school counselors’ self-reported levels of (a) evaluation self-efficacy, (b) confidence in conducting program evaluation, (c) perceived importance of program evaluation, (d) interest in program evaluation, and (e) evaluation values and beliefs? 2. Can a predictive model be created to determine the degree to which these five factors contribute to overall school counselor data beliefs and practices?
Because program evaluation is not adequately named, measured, or assessed in the field of school counseling, I chose to approximate the definition by measuring school counselors’ data-based decision-making behaviors. Data-based decision making has long been accepted as an essential practice for school counselors in developing, implementing, and assessing their school counseling programs (ASCA, 2019; Dimmitt et al., 2007; Zyromski et al., 2021). The critical skills and competencies that are required of school counselors to practice program evaluation in their professional roles has been well noted in previous research (Hatch et al., 2015; Poynton & Carey, 2006; Zyromski et al., 2021), yet continues to be overlooked in professional training and practice. This requires a further examination of school counselor identity, indicating the lack of recognition and importance of school counselors adopting the role of “evaluator.”
Through the lens of SCLT, my first research question sought to understand how school counselors self-report in comments related to their own self-efficacy, specifically in program evaluation. SCLT posits that an individual’s self-efficacy (feelings, attitudes, thoughts, and beliefs) about their capabilities directly impacts their behavior. In this study, understanding how school counselors think and feel and what they believe about program evaluation should theoretically align with their program evaluation practice. Although no scoring data can indicate a “high” or “low” score, the means for each of these variables were close to the maximum scores. This indicates that school counselors are confident in their abilities. The results of Research Question 2 can aid in understanding how these variables contribute to school counselors’ data-based decision making. Results of this study indicate that three of the five factors were statistically significant indicators of data-based decision-making behaviors. Confidence in Program Evaluation, Self-Efficacy in Program Evaluation, and Evaluation Values and Beliefs directly impacted how school counselors in this study utilized and practiced data-based decision making.
Previous research studies focused primarily on the knowledge and skills associated with program evaluation for school counselors (Astramovich, 2016; Maras et al., 2013). Although knowledge and skills are extremely important aspects of school counselor competence, the third component of competence—feelings, attitudes, thoughts, and beliefs—is not adequately addressed in the existing literature. These findings illuminate that knowledge and skills in a content area are important but do not adequately account for school counselors’ behaviors on their own. Feelings, attitudes, thoughts, and beliefs, comprising the third component of competence, have critical contributions in school counselors’ behaviors.
Implications and Future Research
Current research suggests that the creation of established taxonomy of school counselor competencies in program evaluation is the next step in training school counselors to become proficient evaluators. Köse’s work (2019) established a baseline for the field, technical, and nontechnical competencies that already exist within the literature. Trevisan and Carey (2020) have adopted this framework in Program Evaluation in School Counseling and provide tangible strategies, ideas, and resources for improving school counseling training programs in the area of program evaluation.
This study builds upon that work by diving deeper in the nontechnical competencies—beliefs and attitudes—necessary for program evaluation. The results of this study indicate that an individual’s confidence in program evaluation, self-efficacy in program evaluation, and values and beliefs about program evaluation play a vital role in their use of program evaluation practices. Understanding how these variables impact school counselor behaviors has important implications for how school counselors should be trained, beginning with their concept of school counselor identity. School counselor professional identity is taught throughout master’s training programs. It is integrated into general counseling courses (i.e., theories and communication skills), school-counseling-specific courses (i.e., introduction courses and advanced courses), and in field placements (i.e., practicum and supervision). Therefore, the evaluator identity also must be embedded and woven throughout each of these areas.
Further research is needed in the area of context-specific program evaluation competencies. Having explicitly defined competencies provides counselor education programs with a shared foundation for developing curricula and training experiences (Schwandt, 2015). Furthermore, counselor educators should carefully consider how these competencies are embedded throughout training programs.
School counseling program evaluator competencies will serve as an anchor for structuring program foundations and determining required courses. By systematically embedding such competencies in or across all courses, faculty can collectively create a cohesive program that equips students with the knowledge, skills, and dispositions they will need for successful professional practice (Stevahn et al., 2005). To continue teaching program evaluation to school counselors without defined competencies hinders future school counselors’ competence in effective program evaluation (Carey et al., 2019; Köse, 2019; Trevisan et al., 2020).
Limitations
The results of this study are limited by the nature of school counselor self-reports of their dispositional competencies in program evaluation. At this time, no instrument exists that measures demonstrated competency; therefore, this research relies on self-report of what school counselors believe about their competency. Numerous factors could impact one’s self-report such as social desirability, level of confidence, familiarity, etc. Another limitation lies within the scope of the study. This study was conducted with school counselors in one state in the southeastern United States. Generalizability for all states cannot be assumed from these results. However, this study does lay a solid foundation for future research to be done in this area.
The primary limitation of this study is the absence of a program evaluation outcome measure specifically for school counselors. In a literature review, Tate and colleagues (2014) identified 41 instruments measuring counselor competence in various areas. They were unable to identify any measures constructed to assess counselor competence in program evaluation, and therefore I chose to approximate a definition and utilize the DABAPS (Young & Kaffenberger, 2015). The original version of this instrument, the DBAPS (Young & Kaffenberger, 2011) had higher psychometric value, but the DABAPS is widely utilized by ASCA in their texts, professional development, and their own program evaluation as an outcome measure.
Conclusion
School counselor professional identity is a concept widely discussed among researchers in the field. For a decade, school counselors have adopted the idea of a dual identity: both educator and counselor (DeKruyf et al., 2013). Recently, a more comprehensive model of school counselor professional identity was presented. The nondual, nonhierarchical identity of educator–counselor proposed by Levy and Lemberger-Truelove (2021) adequately represents the various roles and duties of the school counselor. Unfortunately, it neglects the critical role of school counselor identity as a program evaluator, planning, implementing, and evaluating their comprehensive school counseling program. Seeing themselves as evaluators is imperative for school counselors as they collect, analyze, and utilize data to inform their programmatic decisions. Previous research in program evaluation focuses on the knowledge and skill components of competency. However, the current study illuminates the importance of a third component of competency: feelings, attitudes, thoughts, and beliefs. The study results indicate that this third layer of competency plays a significant role in school counselors’ behaviors when it comes to using data to make programmatic decisions. With these important findings in mind, it is critical that counselor educators and researchers acknowledge and adopt “evaluator” as a fundamental piece of school counselor identity and begin infusing this throughout school counseling training programs.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
