Measuring Evaluation Competency Among School Counselors

Abstract

School counselors must possess requisite evaluation competency to promote quality and accountability in their comprehensive counseling programs. Despite advances, the field lacks appropriate methods to measure evaluation competency. This article describes the development of a survey designed to measure evaluation competency among school counselors in Missouri and its use in the initial evaluation of a state mentoring program. Findings include initial support for the psychometric properties and four-factor structure of this survey as well as a preliminary assessment of evaluation competencies among participating school counselors. Implications focus on efforts to define, build, and measure evaluation competency in school counseling.

Keywords

program evaluation school counseling mentoring

Evaluation and accountability are not new to school counseling; Gysbers (2010) documents the rich history of efforts within the field to demonstrate that school counseling¹ programs lead to positive student outcomes. However, as noted by Gysbers and others (Dimmitt, 2010; Gysbers & Henderson, 2012), there is currently a significant demand for accountability within the field. Federal policy (e.g., No Child Left Behind), budget cuts, and a growing emphasis on data-based decision making in schools have increased the demand on school leaders and personnel to justify their programs, policies, and practices (Carey, Harrity, & Dimmitt, 2005; Dimmitt, 2010; Sink, 2009). The field has benefited from the rapid development, dissemination, and promotion of various evaluation models and tools as well as new national counselor competencies (i.e., School Counselor Competencies; American School Counselor Association [ASCA], 2008) and training models that prioritize evaluation (Dimmitt, Carey, & Hatch, 2007; Stone & Dahir, 2010). The ASCA National Model prioritizes evaluation as an essential function of school counselors, and this model serves as the guide for practicing school counselors across the country (ASCA, 2012). There is clearly a “revived vigor to prove that what school counselors do is vital and essential to the growth of school-aged youth” (Studer, Oberman, & Womack, 2006, p. 385). School counseling has taken important and necessary steps to address its somewhat difficult history and reputation with evaluation (Trevisan, 2002).

The field must continue to address major barriers to fully realize the promise of evaluation in school counseling. Outlined as part of an overarching approach to building evaluation capacity in the field through broader “contextual factors and system features” (Trevisan, 2002), the evaluation competency of individual school counselors is essential to successful evaluation practice. Hatch and Chen-Hayes (2008) suggested the success of school counseling practice hinges on the ability of school counselors to “learn new skills, to change outdated practices, and to design and implement program evaluation and action research studies showing program model component effectiveness” (p. 34). However, numerous studies suggest that school counselors lack the requisite knowledge, abilities, skills, and personal characteristics (e.g., attitude, self-efficacy) to perform the evaluation tasks expected of them (e.g., Burnham, Dahir, Stone, & Hooper, 2008; Dimmitt, 2010; Studer et al., 2006). Evaluation competencies are essential to successful implementation of the ASCA (2012) National Model. Given that school counselors often lack the competency to perform these functions, the future of evaluation in the field is dependent on advances in fostering evaluation competency among school counselors.

Measuring Evaluation Competency

The availability of accurate and useful measurements to assess evaluation competency is necessary to move evaluation forward in the field of school counseling. Such measures would be helpful in establishing a clearer idea of what competencies school counselors already possess, developing training activities, and evaluating and improving professional development within both preservice and in-service training contexts. The school counseling literature des-cribes measures that have assessed specific aspects of evaluation competency (e.g., beliefs/attitudes about evaluation and accountability, Astramovich, Coker, & Hoskins, 2005; frequency of assessment activities, Ekstrom, Elmore, Schafer, Trotter, & Webster, 2004) and associated frequency of use and training in assessment procedures more broadly (e.g., Blacher, Murray-Ward, & Uellendahl, 2005). Other measures include several items that more narrowly gauge aspects of evaluation competency as part of a broader assessment, often associated with the ASCA National Model (e.g., School Counseling Program Component Scale: Hatch & Chen-Hayes, 2008; Assessment of School Counselor Needs for Professional Development Survey: Burnham et al., 2008). Developed for different but complementary purposes, these measures inform parts of a comprehensive measure of evaluation competency in school counseling. For example, a number of items on the School Counseling Program Component Scale address evaluation skills and knowledge that could comprise a broader definition of evaluation competency. However, this survey addressed perceived importance of different components of the ASCA National Model rather than actual competence in performing these functions.

The School Counselor Attribute and Data Usage Survey (Holcomb-McCoy, Gonzalez, & Johnston, 2009) was designed to identify counselor dispositions (e.g., self-efficacy, openness to change) that predict data usage. Comprised primarily of existing scales (e.g., General Self-Efficacy scale; Schwarzer & Jerusalem, 1995) and adapted items from other scales (e.g., Bandura’s (1997) Teacher Self-Efficacy scale), this instrument also assessed frequency of data usage among school counselors. Notable for its focus on underlying dispositions that promote data usage, this assessment was not designed to assess perceived competency related to specific evaluation practices in school counseling (i.e., developing/administering surveys, changing practice based on findings). Similarly, the use of existing items to assess general self-efficacy circumvents a more thorough exploration of how items differentially capture components of perceived evaluation competency. The field may benefit from a measure designed specifically to assess evaluation self-efficacy.

Dimmitt, Carey, and Hatch (2007) developed the Brief Self-Assessment of Essential Competencies Required for Evidence-Based School Counseling Practice; it is a 24-item measure that assesses confidence in competencies related to problem description (e.g., “I can access data on school information systems”), outcome research use (e.g., “I can access and search the outcome research literature”), and intervention evaluation (e.g., “I can create a survey to meet program needs”). Although this assessment aligns most closely with the type of tool needed to accurately measure evaluation competency, it is being used exclusively as a personal and/or classroom learning tool (J. C. Carey, personal communication, May 14, 2009). This instrument could be used as a template to develop a measure that could have applications for promoting school counselor evaluation competency through research, training, and practice.

Although other measures of evaluation competency exist outside the field, definitions of evaluation competency vary widely by profession and discipline, and the process of building evaluation capacity through broader frameworks is thought to be highly context dependent (Stockdill, Baizerman, & Compton, 2002; Trevisan, 2002). As such, it is reasonable that not only should the field of school counseling develop distinct measures of evaluation competency that extend existing measures, but there may also be some value in further embedding such measures in unique state, regional, or even district contexts.

Professional School Counseling in Missouri

This project is situated within the unique context of Missouri, a state that has a long and rich history with school counseling. In the 1980s, Dr. Norm Gysbers developed a school counseling model in partnership with the Missouri Department of Elementary and Secondary Education—the Missouri Comprehensive Guidance and Counseling Program (MCGCP; see Gysbers, Lapan, Blair, Starr, & Wilmes, 1999 for a historical review)—which provided a foundation for the ASCA (2012) National Model. Within the MCGCP, school counselors focus on specific content areas as well as structural components, part of which includes evaluation activities.

Missouri has a strong state association of school counselors, counselor educators, and leadership at the state level specifically focused on school counselor education, training, and ongoing professional development. These partners have developed and promoted numerous resources to support full implementation of the MCGCP as well as tools Missouri school counselors can access to evaluate their guidance program. Missouri has also developed a mentoring program designed to provide first- and second-year school counselors with the support and technical assistance to fully acclimate to the field. Paired mentors and protégés (terminology specific to this mentoring program) work together on a number of projects, including developing and assessing professional goals, creating a calendar for the counseling program, and assessing current guidance program implementation to create a program evaluation plan.

Mentoring is a common practice in K–12 education to promote professional development and supervision among school-based practitioners, such as teachers (Fairbanks, Freedman, & Kahn, 2000), school counselors (Peace & Sprinthall, 1998), and school nurses (Houghton, 2003). Mentoring relationships often involve coaching, collaborative work, and skill demonstration (Schwille, 2008), and research has reported some positive results of mentoring. For example, Glazerman et al., 2010 found a significant relationship between mentoring intensity and teacher satisfaction.

The school counseling mentoring program in Missouri is intended to foster evaluation competency, particularly among second-year school counselors who are required to conduct a small-scale evaluation project in their schools. The mentoring program has used local, primarily qualitative evaluations to guide the ongoing improvement over the past few years. A logical next step is a more vigorous evaluation of the mentoring program, requiring the development of a valid tool such as the Effective Practices Survey to measure evaluation competency among Missouri school counselors.

The Effective Practices Survey

The Effective Practices Survey was developed to provide a complete and comprehensive measure of evaluation competency relevant to the unique context of professional school counseling in Missouri. The present study examined the factor structure of the Effective Practices Survey and addressed initial questions of survey validity using anticipated results based on levels of counseling experience (i.e., mentors, protégés). Specifically, research questions were (a) does the Effective Practices Survey have a reliable factor structure? and (b) do groups of practicing school counselors vary in their self-report of competency factors based on levels of experience as would be anticipated (e.g., mentors would have greater levels of competency than protégés)? Evaluation of the state mentoring program was a key driver in survey development, and findings from the current study provide tenuous baseline results to guide ongoing development and evaluation of the mentoring program.

Method

Participants

The current study included 226 Missouri school counselors participating in a statewide school counselor mentoring program. In this program, first- and second-year school counselors (i.e., “protégés”) are matched with mentor school counselors for a 2-year period. Participants were predominantly White (n = 212, 93.8%) and female (n = 205, 90.7%). The mean age of participants was 39.7 (SD = 11.47). Related to their role in the mentoring program, 20.4% (n = 46) were first-year protégés, 32.7% (n = 74) were second-year protégés, and 46.9% (n = 106) of the participants were mentor school counselors.

Procedures

Following University institutional review board approval, an e-mail message was sent to school counselors across the state of Missouri, who were participating in the mentoring program (N = 385). A representative from a statewide school counseling organization sent the e-mail to school counselors to request their participation in a brief survey. After consenting to participate, they completed the survey using an online survey administration tool. The response rate across counselors’ roles was 58.7%, with slightly higher response rates among second-year protégés (63.2%) when compared to mentors (57.9%) and first-year protégés (54.1%).

Survey

Participating school counselors received an initial pool of 19 items, which had been adapted from a previous survey (see Dimmitt et al., 2007). The existing literature on evaluation in school counseling and related fields guided survey adaptation, as well as ongoing consultation with experts in school counseling and the MCGCP. Some items were adapted to fit the Missouri context using language specific to the Missouri Department of Elementary and Secondary Education and/or the MCGCP; other items were reworded based on the first author’s experience in using the survey as a teaching tool to ensure items were user-friendly and not overly technical. Finally, other items were informed by the three sections of items in the original survey and combined several items into a small number of items intended to capture the underlying competencies assessed in the original survey.

Specifically, of the 19 items on the Effective Practices Survey, 8 items were nearly identical to the items on the Dimmitt et al. (2007) survey with slight modifications; 3 items assessed competencies described in the 6-item “Problem Identification” section of the original survey that focused on the use of data for planning; 2 items assessed competencies described in the 7-item “Outcome Research Use” section of the original survey that focused on the use of best practices; and 4 items expanded on the single data analysis question on the original survey (I can analyze evaluate data) to focus on specific approaches (e.g., correlation, t-tests). The final 2 items assessed attitudes/beliefs toward evaluation and were similar to items used in other surveys (e.g., Astramovich et al., 2005). These items were posed to school counselors on a Likert-type scale from 1 (very unconfident/strongly disagree) to 6 (very confident/strongly agree). This expanded the 3-point response scale used in the original survey. All items assessing school counselors’ confidence in evaluation skills were presented with the confidence anchors, while questions assessing school counselors’ endorsement of positive evaluation attitudes used the agree/disagree anchor points.

Data Analysis

The analysis plan for interpreting the survey data was as follows: First, we aimed to determine the psychometric properties of the scale. Given the lack of empirical evidence on evaluation competencies among school counselors, we conducted a principal components exploratory factor analysis (EFA) with no a priori assumptions of the factor structure to determine whether the items would load on a few common factors. We also examined reliability and the percentage of variance explained from this factor structure. Next, we examined mean scores on each factor to determine mean differences between factors across the entire sample to answer the question of whether school counselors rate their evaluation skills differently across domains. Next, we examined mean differences between factors within and across each mentoring group (e.g., mentors, first-year protégés, second-year protégés). Understanding mean differences between mentoring groups allows for a greater understanding of existing and developing evaluation competencies between groups with varying levels of experience. It also provides an initial, albeit tenuous, exploration of survey validity based on the logical assumption that counselors’ competency will increase with experience.

Results

EFA

As a first step, we performed an EFA using principal components analysis and a varimax rotation using the final 19 items. We retained four factors after examining the scree plot and locating those factors with eigenvalues above 1.0 (Kaiser, 1991). Specifically, the eigenvalues were as follows: 10.22% with 53.77% of variance explained (Factor 1); 1.96% with 10.30% of variance explained (Factor 2); 1.46% with 7.71% of variance explained (Factor 3); and 1.22 with 6.44% of variance explained (Factor 4). These factors accounted for a total of 78.2% of the variance in item scores.

Table 1 indicates the factor loadings obtained from the EFA. Following conventional criteria, items were retained on a factor due to high loading (>.40) on one factor and low cross loading (<.30) on all other factors (Tabachnick & Fidell, 2001). Experts in the field examined the resulting factor structure to name the factors based on conceptual similarities between items as well as to determine whether cross-loaded items should be retained on a different factor. Specifically, Item 17 (I know how to use technology [e.g., EZ Analyze, Excel] to manage and use data [e.g., create graphs, create databases]) was retained on Factor 3 (Statistics) due to conceptual similarity with other items on the factor. The following factors were derived and named using expert input and examination of the item factor loadings:

Table 1.

Factor Loadings for EFA With Varimax Rotation of Effective Practice Survey Items.

Item Text	Factor 1^a	Factor 2^b	Factor 3^c	Factor 4^d
1. I can use school data (such as attendance, discipline referrals, etc.) to identify student strengths and needs	.88	.32	.08	.07
2. I can access data on school information systems (such as attendance, discipline referrals, etc.)	.85	.29	.03	.02
3. I can use school data (such as attendance, discipline referrals, etc.) to inform my guidance program and activities	.84	.30	.12	.09
4. I can change my practice based on information I gain from research and other professional support (e.g., conference presentations, professional development)	.76	.36	.16	.15
5. I can develop a survey and use its results to make my guidance program better	.74	.32	.15	.22
6. I can create a PowerPoint presentation to share my evaluation results	.72	.16	.23	.22
7. I can access research and other professional support (e.g., conference presentations, professional development) that affects my practice	.71	.35	.24	.14
8. My guidance program includes measurable and objective goals	.27	.79	.15	.12
9. I can evaluate the extent to which my activities and interventions accomplish immediate learning objectives (such as GLEs)	.31	.78	.26	.15
10. I can use ongoing evaluation results to modify the guidance activities I use	.34	.78	.26	.21
11. I can describe how my guidance program and its activities connect to my district’s CSIP (Comprehensive School Improvement Plan) goals	.28	.77	.20	.00
12. I can use data to evaluate guidance program interventions and activities	.40	.74	.32	.20
13. I can use data to help plan interventions (e.g., in conjunction with student assistance teams)	.39	.71	.29	.19
14. I can use t-tests to evaluate my guidance program activities	.02	.24	.86	.08
15. I can use correlations to evaluate my guidance program activities	.17	.30	.84	.08
16. I can use descriptive statistics (means, standard deviations, percentages) to evaluate my guidance program activities	.25	.30	.78	.11
17. I know how to use technology (e.g., EZ Analyze, Excel) to manage and use data (e.g., create graphs, create databases)	.55	.09	.53	.10
18. I believe evaluation leads to better student outcomes	.23	.13	.07	.92
19. I believe evaluation is an important ongoing activity for school counselors	.13	.23	.15	.91

Note. EFA = exploratory factor analysis. The boldface values refers to primary loading on the factor.

^aEvaluation self-efficacy. ^bGuidance program. ^cStatistics. ^dEvaluation values/beliefs.

Factor 1 included items related to evaluation self-efficacy and included 7 total items. These items referred to school counselors’ ability to access and use relevant resources and data and then subsequently use these types of information in their professional practice. Within this factor, the internal consistency reliability was at an acceptable level (α = .94). Participants in the sample scored a mean of 5.34 on this factor (SD = 0.99).

Factor 2 included items related to guidance programs and included 6 total items. These items referred to school counselors’ ability to use best practices in their guidance program planning, such as including measurable and objective goals. Within this factor, the internal consistency reliability was at an acceptable level (α = .95). Participants in the sample scored a mean of 4.71 on this factor (SD = 1.1).

Factor 3 included items related to statistics and included 4 total items. These items referred to school counselors’ ability to utilize statistical knowledge and technology to promote more effective professional practice. Within this factor, the internal consistency reliability was at an acceptable level (α = .85). Participants scored a mean of 3.91 on this factor (SD = 1.19).

Factor 4 included items related to evaluation values and beliefs and included 2 total items. These items referred to school counselors’ beliefs in evaluation as a necessary and valuable job role. Within this factor, the internal consistency reliability was at an acceptable level (α = .92). Participants in the sample scored a mean of 5.27 on this factor (SD = 1.03).

The normality of factor means was tested using the Shapiro–Wilks statistic. Analyses indicated that all four factors deviated significantly from normality: Factor 1 (Shapiro–Wilks W = .65, p < .001), Factor 2 (Shapiro–Wilks W = .87, p < .001), Factor 3 (Shapiro–Wilks W = .97, p < .001), and Factor 4 (Shapiro–Wilks W = .73, p < .001) were all negatively skewed. As such, nonparametric tests were used for all between-factor and between-subject tests. Analysis of variance (ANOVA) tests were used for within-factor and within-subject tests. Due to the non-normal distribution of the data, separate ANOVAs were selected for analyses rather than multivariate analysis of variance.

Comparison of Factors Across and by Mentoring Program Roles

Several sets of ANOVA analyses were used to first examine whether average ratings on each factor differed for the entire sample—in other words, did school counselors overall rate their confidence in some factors differently than in others. Next, we repeated the ANOVA within each mentoring group (e.g., mentors, first-year protégés, second-year protégés) to determine whether factor mean scores differed within each mentoring role—whether mentors rated their confidence in some factors differently than in others. Mauchly’s test indicated that assumption of sphericity had been violated, χ²(5) = 50.7, p < .001, therefore degrees of freedom were corrected using Greenhouse–Geisser estimates of sphericity (∊ = .87).

Analysis revealed a significant within-subject effect for factor mean for all participants, F(2.61, 545.44) = 159.58, p < .001, η _p = .43, as well as a significant within-subject effect for factor mean by mentoring program role, F(5.22, 545.44) = 3.94, p < .01, η _p = .04. School counselors were more confident in the competencies captured by some factors than others, and this pattern was consistent when analyzed across the entire sample and also when analyzed within each mentoring group.

To determine mean differences between factors, planned pairwise comparisons for factor means among the entire sample were conducted with all possible combinations of factors. These comparisons revealed significant differences between all factors except Factors 1 and 4 (Tables 2 and 3). In sum, for the entire sample, the mean scores for evaluation self-efficacy and evaluation values/beliefs were significantly greater than both the guidance program mean score and the statistics mean scores. The guidance program mean score was significantly greater than the statistics mean score.

Table 2.

Factor Means for Total Sample and by Mentoring Program Role.

		Mentoring Program Role
	Total Sample	First-Year Protégé	Second-Year Protégé	Mentor
	(n = 212)	(n = 45)	(n = 68)	(n = 99)
Factors	M (SD)	M (SD)	M (SD)	M (SD)
Factor 1 (Evaluation Self-Efficacy)	5.31 (.07)	5.23 (.15)	5.23 (.12)	5.47 (.10)
Factor 2 (Guidance Program)	4.65 (.08)	4.49 (.16)	4.50 (.13)	4.96 (.11)
Factor 3 (Statistics)	3.88 (.09)	3.74 (.18)	3.93 (.14)	3.98 (.12)
Factor 4 (Evaluation Values/Beliefs)	5.32 (.07)	5.46 (.15)	5.31 (.12)	5.18 (.10)

Table 3.

Mean Differences Between Factor Scores for Total Sample and Mentoring Groups.

	Guidance Program				Statistics				Evaluation Values/Beliefs
	Total	First Year	Second Year	Mentor	Total	First Year	Second Year	Mentor	Total	First Year	Second Year	Mentor
Evaluation self-efficacy	.66**	.74**	.73**	.51**	1.4**	1.5**	1.3**	1.5**	00	−.23	−.08	.29**
Guidance program	—	—	—	—	.76**	.74**	.56**	.98**	−.66**	−.97**	−.81**	−.22*
Statistics	—	—	—	—	—	—	—	—	−1.4**	−1.7**	−1.4**	−1.2**

*p < .05. **p < .01.

To determine mean differences between factors for participants with varying levels of experience, an additional set of planned pairwise comparisons for all combinations of factor means within each participant role were conducted. The pattern of responses for first- and second-year protégés was consistent with that of the group as a whole: The mean scores for evaluation self-efficacy and evaluation values/beliefs were significantly greater than both the guidance program mean score and the statistics mean score. The guidance program mean score was significantly greater than the statistics mean score (Table 3). Results suggest a slightly different pattern of differences for mentors; there were significant differences between all factors for participants in this role at the .05 level (Table 3). However, the mean score for evaluation values/beliefs (Factor 4) for mentors was lower than all other groups and was lower than the mean score for evaluation self-efficacy for mentors (p < .01).

Comparison of Factor Scores Among Mentoring Roles

A Kruskal–Wallis test was conducted to evaluate the differences among the three mentoring program roles (mentor, first-year protégé, and second-year protégé) within factor scores. There was a significant difference between groups only on Factor 2 (guidance program), χ²(2, N = 224) = 9.16, p = .01. The three mentoring groups significantly differed from each other on mean self-reported confidence in their abilities to plan their guidance program according to best practices, such as using goals and objectives. Follow-up Mann–Whitney U tests were used to evaluate pairwise differences among the three participant roles on Factor 2 (guidance program). There was a significant difference between mentors and first-year protégés, U(150) = 1,796.00, Z = −2.44, p < .05, and mentors and second-year protégés, U(178) = 2,992.00, Z = −2.53, p < .05, with mentors having a higher score on this factor (M = 4.96, SD = .11) than both first-year protégés (M = 4.49, SD = .16) and second-year protégés (M = 4.50, SD = .13). There was not a significant difference between first- and second-year protégés on this factor score.

Discussion

The purpose of this study was to examine the psychometric properties and factor structure of a self-reported measure of school counselor evaluation competency. The development and initial investigation of this instrument was one component of broader efforts to strengthen the comprehensive evaluation of a mentoring program for school counselors in Missouri. The Effective Practices Survey was specifically developed both as a tool to evaluate the mentoring program in Missouri and as a measure to advance the research understanding of evaluation competencies in one state-specific context. While other measures have been developed for similar purposes and measuring similar constructs, the Effective Practices Survey specifically targets evaluation competencies that can contribute to the field’s increased attention to evaluation and data-based decision making. As an evaluation tool, it can serve as an important pre-/postassessment of school counselor confidence in evaluation competencies and can provide a needs assessment for ongoing training and professional development. As a research tool, it can be used in combination with other measures to determine overall counselor self-efficacy and its impact on student outcomes following a more thorough examination of the instrument.

The factor model provided an adequate fit to the data, and each factor had acceptable levels of internal consistency. We also found evidence to suggest that each factor captured different domains of evaluation competence. Four factors best described school counselors’ perceptions of their evaluation competency.

Evaluation Self-Efficacy described school counselors’ ability to access relevant resources (e.g., research, conference presentations) and data (e.g., attendance, office referral data) and subsequently use this information in their professional practice. Overall, school counselors in this study reported high levels of competence related to evaluation self-efficacy. Similar results have been found in studies assessing the frequency with which school counselors perform evaluation-related activities. Ekstrom, Elmore, Schafer, Trotter, and Webster (2004) assessed the extent to which school counselors used needs assessments to develop their school counseling programs and found that over 50% of counselors in the study regularly performed these activities. Holcomb-McCoy, Gonzalez, and Johnston (2009) also assessed the frequency with which school counselors use data as an integral part of their practice and found wider use of some sorts of data (e.g., to determine counseling groups) than others (e.g., to report to parents). Both studies included the frequency with which school counselors engaged in the practices, while the current instrument assessed self-reported confidence. Future research could concurrently investigate the frequency of evaluation practices and self-efficacy.

Guidance Program described school counselors’ ability to use best practices in their guidance program planning. This included using measurable and objective goals and evaluating the extent to which guidance activities accomplished goals set by local and state bodies. School counselors in the study generally reported strong skills in this area. This particular factor was formulated to be specific to the Missouri context, so this factor may need to be adapted to fit other contexts in terms of the state and local spheres of influence.

Statistics referred to school counselors’ ability to utilize statistical knowledge and technology to promote more effective professional practice. Specifically, this factor assessed the extent to which school counselors feel competent in using statistical procedures such as t-tests, correlations, and descriptive statistics. Of all factors, statistics was consistently rated as the lowest among school counselors in the sample. This echoes prior research—Ekstrom et al. (2004) included one item related to using statistics in evaluating guidance programs and found that few (31%) participants engaged in this practice. Similarly, 30% of the sample used technology to process data for evaluations. The area of statistics appears to be rated as a low-frequency, low-competence area that would likely be impacted by further training and practice on the technical aspects of evaluation.

Evaluation Values/Beliefs referred to school counselors’ beliefs in evaluation as a necessary and valuable job role. In this study, school counselors reported very positive views toward evaluation. Similarly, Astramovich, Coker, & Hoskins (2005) found that school counselors reported positive opinions about the value and rationale of program evaluation. However, Astramovich and colleagues found mixed opinions about the ability of school counselors to easily assimilate evaluation into their everyday practice.

Across the entire sample, school counselor rated their self-efficacy and beliefs in the importance of evaluation as significantly higher than their levels of application of evaluation in their guidance program or in their statistical knowledge. This suggests that while school counselors value the notion of evaluation and believe they possess some requisite competencies to perform evaluation activities, technical skills and actual use of evaluation appear to be lagging. These results suggest that further training and professional development might be helpful in empowering school counselors to use evaluation skills within everyday practice while facilitating knowledge development of statistical techniques. Programs such as the school counselor mentoring program in Missouri could also be used for direct in vivo experience in incorporating evaluation activities into practice with supervision and support. Mentoring programs are widely used in school settings to support ongoing professional development for school counselors, teachers, and other personnel (Curry & Bickmore, 2012; Ingersoll & Strong, 2011). Curry and Bickmore (2012) found that structured supports for novice counselors, including a formal mentor and high-quality professional development, positively affected new counselors’ feelings of “mattering.” Mentoring programs could play a significant role in fostering evaluation competencies among new school counselors entering the field.

The current study also explored differences on factor scores both within and across mentoring groups. Within particular mentoring groups, we found similar results for first- and second-year protégés: these groups rated higher confidence in their ability to use resources and data in evaluating their guidance program (evaluation self-efficacy) and their value of evaluation (evaluation values) than their confidence in their ability to use best practices in their guidance program (guidance program) and their ability to use statistics to evaluate their guidance program (statistics). The Missouri mentoring program is structured such that the majority of evaluation-focused professional development activities take place during the second year. Data were collected in early to mid-Fall, so these similarities are not surprising, given the limited exposure second-year protégés had to evaluation professional development.

Perhaps more notable were the significant factor comparisons between the newer school counselors and the mentoring group. Mentoring in schools and others contexts typically refers to a partnership between an expert and novice practitioner, with the underlying assumption that the expert mentor is more competent and confident than the novice protégé. In the current study, mentors rated themselves lower than the protégés on the evaluation values/beliefs factor, suggesting that newer school counselors may have more positive beliefs about evaluation than mentor school counselors. Given the tumultuous history of evaluation within the field of school counseling, and the relatively recent push toward changes in preservice education in evaluation for school counselors, this finding may highlight an important area of need related to in-service professional development for more seasoned school counselors (Trevisan, 2000, 2002). The current study also found that mentors had significantly higher scores on the guidance program factor compared to the protégés. More experienced school counselors may feel more competent in their ability to incorporate best practices for planning based on state and local mandates. In sum, the results suggest that it may be useful to compare evaluation competencies within and across groups to inform mentoring practices in school counseling.

Limitations

It is important to acknowledge several limitations regarding the current study. One limitation of the present study concerns the study context. This research was conducted in Missouri, a state that has unique history within the field and boasts a well-established infrastructure for building comprehensive school counseling programs across the entire state (Martin & Carey, 2012). While high levels of evaluation capacity in Missouri may limit some generalization of findings in this study, we believe this state context also provides a valuable opportunity to advance evaluation within the field.

We also purposefully modified the language on some survey items to match the language used in this state. Anecdotally, the first author found that the technical evaluation language used in other surveys (e.g., Dimmitt et al., 2007) masked higher levels of perceived competency among preservice school counselors within the classroom context. Adapting the survey to have more user-friendly language that aligned with the Missouri context helped students realize they had more evaluation skills and knowledge than initially thought. Similar modifications may be needed when using the tool in other states. Changes were minimal and likely did not have a great impact on the final structure; however, further research is needed to confirm the validity of this survey in other contexts. In particular, one important next step would be adapting the language of the Effective Practices Survey to align with the ASCA National Model and conducting further research on the factor structure as well as reported levels of competencies. Because the ASCA National Model provides an overarching framework for the practice of school counseling nationally, this step would expand the utility of this tool across other contexts.

Finally, it should be noted that the Effective Practices Survey is essentially a new research instrument with no prior research. The survey was adapted from an existing tool used exclusively for teaching purpose, and the psychometric properties of the original survey have not been explored to date. While the current study found a four-factor model that provided an adequate fit to the data with acceptable levels of internal consistency for each factor, further examination of the psychometric properties of the survey is needed. In particular, future studies should focus on exploring the validity of the Effective Practices Survey. The current study used varying levels of experience to compare levels of competency as one way to evaluate validity, but it is possible that we should not assume that competency increases with experience (as discussed in more detail earlier). Because this is the first review of this new instrument, we must be skeptical of all findings related to competencies of the current sample. Future research should focus on validating the Effective Practices Survey to ensure its psychometric properties are sound.

Practical Applications and Future Implications

These findings suggest that the evaluation tool, after further examination, could provide a useful self-assessment of counselor evaluation capacity both for program development and maintenance purposes and for research purposes. Brief and efficient tools to assess evaluation capacity are needed as school counselors attempt to expand their skills in this area and incorporate evaluation activities into their guidance programs. There are a number of potential uses for this measure and similar tools in building evaluation capacity within the field. For example, the survey could be used to inform ongoing efforts to better define what evaluation competency is for school counselors and the specific skills and knowledge required to effectively evaluate comprehensive guidance and counseling programs. A clear articulation of these competencies could further serve to align preservice and in-service evaluation training to create a more cohesive statewide evaluation capacity-building system. Trevisan (2002) and, more recently, Martin and Carey (2012) focus on the need for and role of statewide systems to build evaluation capacity among school counselors, and the Effective Practices Survey could be a useful tool in developing and evaluating this type of system. Consistent with any evaluation tool, it will be important to consider the ethical and legal implications of assessing individual counselor competencies within a broader statewide system that could focus on linking counselor evaluation competencies with student outcomes (e.g., academic achievement). Given recent efforts to connect teacher competencies with student performance in the classroom, it will be important to ensure that an instrument like the Effective Practices Survey is used to assess counselor competency as conceptualized within a statewide framework of supports and barriers.

The Effective Practices Survey may also have specific applications within the ASCA National Model. As discussed above, the current survey could be adapted to reflect components of the ASCA National Model and align with terminology used within the model. Similar to potential efforts at the state level, this adapted and further-validated survey could be used to develop a nationally representative sample of school counselors’ evaluation competencies across state contexts. This type of information would allow further comparison of state school counseling models and inform national efforts to support effective school counseling programs.

The survey has also recently been used at a smaller scale in district-level consultation activities and preservice counselor education courses. The Effective Practices Survey was used to assess a yearlong consultation effort aimed to develop evaluation competency among school counselors in one Missouri school district. For this project, the survey was completed initially as a self-assessment by school counselors who used findings to guide discussions of their individual strengths and weaknesses as well as trends across counselors in the district. As an example, counselors in this district discussed their collective gaps in capacity and confidence regarding the use of statistics when evaluating their school counseling activities. Further conversations focused on their prior reliance on a single counselor to “crunch the numbers” who left the district, and they realized that they all needed to have a basic level of understanding of and confidence in using statistics. The Effective Practices Survey was administered at the conclusion of the project to evaluate individual progress on evaluation competencies as well as the overall effectiveness of the program. This small pilot study points to possible uses of the Effective Practices Survey as a tool for professional development.

Finally, the measure has been used in the classroom to assess instruction, model the use of survey tools in evaluation, teach simple statistics, and create opportunities for self-assessment and reflection. Future research could explore the Effective Practices Survey as a teaching tool, and counselor educators may be particularly interested in the practical application of the instrument in the classroom. While additional research is needed to explore the content and uses of the Effective Practices Survey, the development and initial investigation of the scale suggests it may be a useful tool to assess the evaluation competency of school counselors.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ralph Bedell Fund, Department of Educational, School, and Counseling Psychology, University of Missouri.

Note

References

American School Counselor Association. (2008). School counselor competencies. Retrieved from http://www.schoolcounselor.org/files/SCCompete-ncies.pdf

American School Counselor Association. (2012). The ASCA national model: A framework for school counseling programs (3rd ed.). Alexandria, VA: Author.

Astramovich

R. L.

Coker

Hoskins

W. J.

(2005). Training school counselors in program evaluation. Professional School Counseling, 9, 49–54.

Bandura

(1997). Self-efficacy: The exercise of control. New York, NY: Freeman.

Blacher

J. H.

Murray-Ward

Uellendahl

G. E.

(2005). School counselors and student assessment. Professional School Counseling, 8, 337–343.

Burnham

J. J.

Dahir

C. A.

Stone

C. B.

Hooper

L. M.

(2008). The development and exploration of the psychometric properties of the assessment of school counselor needs for professional development survey. Research in the Schools, 15, 51–63.

Carey

Harrity

Dimmitt

(2005). The development of a self-assessment measure of a school district’s readiness to implement the ASCA national model. Professional School Counseling, 8, 305–312.

Curry

J. R.

Bickmore

(2012). School counselor induction and the importance of mattering. Professional School Counseling, 15, 110–122.

Dimmitt

(2010). Evaluation in school counseling: Current practices and future possibilities. Counseling Outcome Research and Evaluation, 1, 44–56.

10.

Dimmitt

Carey

J. C.

Hatch

(2007). Evidence-based school counseling. Thousand Oaks, CA: Corwin Press.

11.

Ekstrom

R. B.

Elmore

P. B.

Schafer

W. D.

Trotter

T. V.

Webster

(2004). A survey of assessment and evaluation activities of school counselors. Professional School Counseling, 8, 24–30.

12.

Fairbanks

C. M.

Freedman

Kahn

(2000). The role of effective mentors in learning to teach. Journal of Teacher Education, 51, 102–112. doi:10.1177/002248710005100204

13.

Glazerman

Isenberg

Dolfin

Bleeker

Johnson

Grider

Jacobus

(2010). Impacts of comprehensive teacher induction: Final results from a randomized controlled study (NCEE 2010-4027). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

14.

Gysbers

N. C.

(2010). School counseling principles: Remembering the past, shaping the future, A history of school counseling. Alexandria, VA: American School Counselor Association.

15.

Gysbers

N. C.

Henderson

(2012). Developing and managing your school guidance and counseling program (5th ed.). Alexandria, VA: American Counseling Association.

16.

Gysbers

N. C.

Lapan

R. T.

Blair

Starr

M. F.

Wilmes

(1999). Closing in on the statewide implementation of a comprehensive guidance program model. Professional School Counseling, 2, 357–366.

17.

Hatch

Chen-Hayes

S. F.

(2008). School counselor beliefs about ASCA national model school counseling program components using the SCPCS. [Article]. Professional School Counseling, 12, 34–42. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.5330/PSC.n.2010-12.34

18.

Holcomb-McCoy

Gonzalez

Johnston

. (2009). School counselor dispositions as predictors of data usage. Professional School Counseling, 12, 343–351. Retrieved from https://dx-doi-org.web.bisu.edu.cn/10.5330/PSC.n.2010-12.343

19.

Houghton

(2003). A mentoring program for new school nurses. The Journal of School Nursing, 19, 24–29. doi:10.1177/10598405030190010501

20.

Ingersoll

R. M.

Strong

(2011). The impact of induction and mentoring programs for beginning teachers: A critical review of the research. Review of Educational Research, 81, 201–233.

21.

Kaiser

H. F.

(1991). Coefficient alpha for a principal component and the Kaiser-Guttman rule. Psychological Reports, 68, 855–858.

22.

Martin

Carey

J. C.

(2012). Evaluation capacity within state-level school counseling programs: A cross-case analysis. Professional School Counseling, 15, 110–122.

23.

Peace

Sprinthall

(1998). Training school counselors to supervise beginning school counselors: Theory, research, and practice. Professional School Counseling, 1, 2–8.

24.

Schwarzer

Jerusalem

(1995). Generalized self-efficacy scale. In Weinman

Wright

Johnston

(Eds.), Measures in health psychology: A user’s portfolio. Causal and control beliefs (pp. 35–37). Windsor, England: Nfer-Nelson.

25.

Schwille

(2008). The professional practice of mentoring. American Journal of Education, 115, 139–167.

26.

Sink

(2009). School counselors as accountability leaders: Another call for action. Professional School Counseling, 13, 68–74.

27.

Stockdill

S. H.

Baizerman

Compton

D. W.

(2002). Toward a definition of the ECB process: A conversation with the ECB literature. New Directions in Evaluation, 93, 7–25.

28.

Stone

C. B.

Dahir

C. A.

(2010). School counselor accountability: A MEASURE of student success (3rd ed.). Upper Saddle River, NJ: Prentice Hall.

29.

Studer

J. R.

Oberman

A. H.

Womack

R. H.

(2006). Producing evidence to show counseling effectiveness in the schools. Professional School Counseling, 9, 385–391.

30.

Tabachnick

B. G.

Fidell

L. S.

(2001). Using multivariate statistics. Boston, MA: Allyn and Bacon.

31.

Trevisan

(2000). The status of program evaluation expectations in state school counselor certification requirements. American Journal of Evaluation, 21, 81–94.

32.

Trevisan

M. S.

(2002). Evaluation capacity in K-12 school counseling programs. American Journal of Evaluation, 23, 291–305.