The Assessment of CACREP Core Curricular Areas and Student Learning Outcomes Using the Counseling Competencies Scale

Ethical Responsibility of Counselor Education and Supervision Programs

The ACA (2005) Code of Ethics contains ethical guidelines for counselors, counselors-in-training, counselor educators, counseling supervisors, and counseling researchers with responsibilities for the assessment of counseling competencies. Specifically, counselor educators are expected to discuss their expectations regarding counseling competencies with counseling students, and assess and provide formative and summative feedback to students regarding their progress in developing the competencies (Standard F.9.a.). In addition, the ethical code notes the need for counselors-in-training to have self-awareness of their abilities to provide counseling services and seek professional help when they are impaired (Standards F.8.a.; F.8.b.). Therefore, both counselor educators and their students have an ethical responsibility to monitor the development of their counseling competencies and provide systematic remediation in areas where students lack proficiency.

Accreditation Standards and Student Evaluation

The CACREP (2009) Standards state that counselors-in-training “develop professional counselor identity and master the knowledge and skills to practice effectively” (p. 1). The counselor development process begins with the screening of applicants during the admission process, in order to assess applicants’ dispositions and their potential to develop interpersonal relationships (CACREP, 2009, Section I, Standard K). Then, throughout the counselor training experience, CACREP-accredited counselor education programs need to demonstrate how SLOs are measured by the program and met by the students (CACREP, 2009, Section I, Standard AA), including conducting a “systematic developmental assessment of each student’s progress” focused on academic performance, and personal and professional development throughout counseling preparation (CACREP, 2009, Section I, Standard P, p. 5). When students are identified as being inappropriate for the program, counselor educators engage in a process to assist students with their “transition out of the program” (CACREP Section I, Standard P, p. 4). Therefore, counselor educators need formalized processes for (a) screening applicants, (b) evaluating students, (c) communicating their feedback to students to support their learning and development, and (d) developing remediation plans to address specific student deficiencies.

Counselors-in-training apply the knowledge and skills acquired during counseling content courses (e.g., Theories of Counseling) into their practice during their professional practice courses (Practicum and Internship) within their preparation program (CACREP, 2009). During students’ practicum and internship experiences, counselor educators and supervisors evaluate students’ performance, providing formal formative and summative evaluation (CACREP, 2009, Section III, Standard F.5 & Standard G.6). Therefore, counselor preparation programs must have an established procedure that identifies the process used to formally evaluate counseling students’ counseling competencies during their practicum and internship experiences. Thus, both ACA (2005) and CACREP (2009) note the significance of evaluation in students’ development of counseling competencies throughout their preparation program. Nevertheless, a need exists for a psychometrically sound instrument to assess counseling SLOs.

CCS

The CCS (Swank, 2010; Swank et al., 2012; UCF Counselor Education Faculty, 2009) began as an initiative at a research university by the counselor education faculty. The faculty recognized a need for the development of a psychometrically sound assessment that measured counseling competencies in a comprehensive manner. After an extensive literature review, the faculty identified that various instruments (e.g., Eriksen & McAuliffe, 2003; Hill Counselor Verbal Response Category System–Revised [HCVRCS], Hill, 1978; Counseling Skills scale [CSS], Skilled Counseling scale [SCS], Urbani et al., 2002) existed focusing on the assessment of counseling competencies in specific areas (counseling skills); however, the faculty were unable to identify any existing assessments that evaluated counseling competencies in a comprehensive manner. Thus, the faculty began the process of developing a comprehensive assessment to measure counseling competencies (Swank, 2010, Swank et al., 2012).

The initial version of the counseling competency assessment was the Counselor Skills and Professional Behavior scale (CSPBS; UCF Counselor Education Faculty, 2004). After evaluating the CSPBS, the faculty recognized that the instrument needed revisions in operationally defining the assessment items and the scoring system. Thus, the CSPBS was revised into the CCS in 2009 (Swank, 2010, Swank et al., 2012).

The CCS items were constructed through a review of the counseling literature and were reviewed by a panel of experts (counselor educators), which provided measures of content validity. The items are related to the CACREP (2009) Standards and the ACA (2005) Code of Ethics. The rubric format, contained within the CCS, has five response categories: (a) harmful, (b) below expectations, (c) near expectations, (d) meets expectations, and (e) exceeds expectations for assessing all areas within the instrument. The five response categories were designated to distinguish between varying levels of development in regard to counseling competencies (Table 1 presents sample CCS items). Two models of the CCS (midterm and final evaluation models) were developed per an exploratory factor analysis (EFA; Swank et al., 2012), which was conducted to measure construct validity. The final CCS model is used for the purpose of this manuscript as it is a summative assessment of learning in the core curricular areas and addresses SLOs. The CCS assesses counseling competencies in five domains that are supported by the literature (Eriksen & McAuliffe, 2003; Kerl, Garcia, McCullough, & Maxwell, 2002; Rogers, 1957; Urbani et al., 2002: (a) counseling skills; (b) professional dispositions; (c) professional behaviors; (d) the counseling relationship; and (e) assessment and application, which were identified through an EFA (see Figure 1). The counselor educator rater reviews counseling sessions to assess counseling competencies in two domains (counseling skills and counseling relationship). The remaining three domains of the CCS (professional dispositions, professional behaviors, and assessment and application) are assessed through the observation of the student during clinical and nonclinical coursework, supervision, and other interactions with the student. In addition, some items within the counseling relationship domain (e.g., professionalism, boundaries) are assessed through the observation of the counseling students during their counselor training experience.

OPEN IN VIEWER

Figure 1.

Final CCS five-factor model.

OPEN IN VIEWER

Table 1.

Sample CCS Items (Swank et al., 2012)

Skill, Behavior, or Disposition	Specific Counseling Descriptors	Exceeds Expectations/Demonstrates Competencies (8)	Meets Expectations/Demonstrates Competencies (6)	Near Expectations/Developing toward Competencies (4)	Below Expectations/Insufficient/Unacceptable (2)	Harmful (0)
Reflection B	Reflection of Feelings	Demonstrates appropriate use of reflection of feelings as the primary approach (85%)	Student demonstrates appropriate use of reflection of feelings appropriately (70%)	Demonstrates reflection of feelings inconsistently and is not matching the client	Demonstrates limited proficiency in reflecting feelings or often inaccurate	Judgmental, dismissing, overshoots
Professional and personal boundaries	Maintains appropriate boundaries with supervisors, peers, and clients	Demonstrates consistently strong and appropriate boundaries	Demonstrates appropriate boundaries	Demonstrates appropriate boundaries inconsistently	Demonstrates inappropriate boundaries	Harmful relationship with others
Application of theory to practice	Demonstrates knowledge of counseling theory and its application in his or her practice	Demonstrates a strong understanding of the counseling theory(ies) that guides his or her therapeutic work with clients	Demonstrates an understanding of the counseling theory(ies) that guides his or her therapeutic work with clients	Demonstrates inconsistent understanding of the role of counseling theory in his or her therapeutic work	Demonstrates limited understanding of counseling theory and its role in his or her therapeutic work with clients	Harmful use of theoretical principles

The initial exploration of the psychometric properties of the CCS yielded a strong internal consistency reliability (.933 for the overall instrument and a range of .850 to .989 for the individual factors; Swank et al., 2012). Additionally, in measuring criterion-related validity, a moderate correlation was found between the final total score on the CCS at the end of a practicum experience and the final counseling practicum course grade explaining 17% of the variance (r = .407, p < .01). The correlation between the CCS and earned practicum course grade provides initial support for utilizing the CCS to identify students’ academic and clinical achievement. Finally, the CCS interrater reliability correlation was (r = .570), appearing to be low, supporting the need for effective training in the utilization of the CCS to strengthen the interrater reliability (Swank, 2010, Swank et al., 2012). Nevertheless, the overall initial investigation of the CCS supports the psychometric soundness of the instrument, while emphasizing the need for continued examination and refinement of the instrument.

With the development of the CCS, counselor educators and supervisors have a mechanism to promote the professional and personal growth and development of counselors-in-training. Additionally, the CCS establishes consistent competencies and expectations to assist counselor educators in evaluating SLOs in the area of counseling competencies. Therefore, the CCS supports counselor educators in fulfilling ethical responsibilities to support counseling students’ learning and development while gatekeeping for the profession and bolstering programmatic evaluation processes aligned with CACREP (2009) and NCATE (2008) Standards.

The CCS as a Measure of SLOs

The CCS (Swank, 2010; Swank et al., 2012; UCF Counselor Education Faculty, 2009) was designed for counselor educators to assess students’ development of counseling competencies throughout the counselor training program. Initially, counselor educators may use the identification of specific dispositions within the CCS to evaluate the potential of applicants during the admission process. Within the program, counselor educators and supervisors may utilize the CCS to assess counseling students’ competencies in specified courses, while choosing to assess only specific items within the CCS in other courses. Specifically, the CCS is designed for utilization in its entirety during practicum and internship courses; however, the skills section and other identified items (e.g., self-awareness, openness to feedback) are useful for assessing personal and professional development and appropriate use of counseling skills during a counseling techniques course. In addition, counselor educators may use the CCS to assess students’ dispositions and behaviors in other courses throughout their preparation program, in order to systematically conduct a developmental assessment of all students’ progress throughout their program of study, as required by CACREP (2009; Section I, Standard P).

In addition to faculty evaluations, doctoral students, and internship site supervisors can be trained to evaluate counseling students’ competencies utilizing the CCS during practicum and internship experiences. Moreover, counseling students can learn how to assess their own counseling competencies through the use of the CCS. Training videos and a detailed manual were developed to assist counselor educators, supervisors, and students in scoring and interpreting the CCS, which may strengthen interrater reliability (Swank, 2010).

Counselor educators and supervisors use the CCS to evaluate SLOs in the area of counseling competencies during students’ practicum and internship experiences, aligning with CACREP (2009; Section III, Standard F.5 & Standard G.6). The CCS is used for formal evaluation twice during each semester of Practicum and Internship (midterm and at the end of the semester). The CCS provides students and supervisors with concrete expectations regarding supervision, matching new supervisees’ developmental needs (Bernard & Goodyear, 2009; Stoltenberg & McNeill, 2010). The supervisor completes the CCS, which includes a narrative section for documenting students’ strengths and areas for improvement, in addition to completing the scoring rubric. After completion of the CCS, the supervisor meets with the supervisee to review the evaluation and discuss the student’s ongoing development. The supervisor–supervisee CCS discussion process supports open communication, strengthening the supervisory relationship (Bernard & Goodyear, 2009). Furthermore, clear and concrete supervisory feedback identifying students’ strengths and areas for improvement may contribute to students’ increased confidence and self-efficacy, and lower anxiety (Daniels & Larson, 2001).

Following the review of the CCS, both the supervisor and the supervisee sign the evaluation. The student’s signature does not signify agreement with the supervisor’s ratings; rather, the student’s signature signifies that the student was present for the review of the evaluation and that the student understood the CCS ratings and the additional comments documented on the assessment. In addition, both the supervisor and supervisee maintain a signed copy of the CCS, supporting accountability for all involved parties. Finally, a copy of the final CCS from students’ practicum experiences is given to their internship supervisors, promoting programmatic continuity. Thus, a formal procedure is established for assessing counseling competencies using the CCS, which supports transparency and fosters students’ investment in the evaluation process (Foster & McAdams, 2009).

Practicum and internship supervisors may also ask their students to complete a self-rating, using the CCS, prior to reviewing their supervisors’ evaluations. Counselor educators and supervisors use the students’ self-rating as a teaching tool to assist students in developing skills in self-assessment and self-awareness of their counseling competencies, which may contribute to their personal and professional development that are emphasized within the CACREP (2009) Standards (Swank et al., 2012). Accurate self-assessment may enhance counseling supervision when utilized concurrently in practice (Morrissette, 1999). In addition, self-assessment helps counseling students recognize ineffective responses and improve counseling skills (Bernard & Goodyear, 2009). Self-assessment also promotes confidence in working with future clients (Steward, Breland, & Neil, 2001). Furthermore, the development of self-evaluation skills may assist counseling students in becoming reflective practitioners as they continuously assess and reflect upon their counseling competencies beyond their counselor preparation program, throughout their counseling careers. Thus, the CCS provides counselor educators and supervisors with an instrument to enhance pedagogy, in addition to using the CCS as an evaluation method.

Counselor educators may use the CCS to prepare counseling students for the formative and summative evaluation process occurring during their practicum and internship experiences. For example, counselor educators may evaluate SLOs during a counseling techniques course using the CCS (e.g., reflection of feelings). Instructors for counseling techniques courses may find the CCS effective in assessing counseling competency in the area of counseling skills when evaluating their students’ role-played counseling sessions. Developmentally, the expectations regarding students’ appropriate use of counseling skills are based on their practice in their preparation program (e.g., counseling techniques course, limited skill development; counseling internship, proficient skill development); therefore, higher ratings are expected as students progress throughout their preparation program. Using the CCS in counseling techniques courses provides students an opportunity to become familiar with the instrument and start recognizing their strengths and areas for improvement with regard to counseling competencies. Additionally, repeated use of the CCS in various courses provides an opportunity for multiple mastery experiences, which enhances counselor self-efficacy (Daniels & Larson, 2001) and programmatic continuity. The use of the CCS provides counselor educators with the opportunity to have counseling students evaluate their peers’ counseling competencies, in addition to their own counseling competencies. Evaluating a peer’s counseling competencies allows counseling students the opportunity to practice offering feedback to colleagues in a written format, which could then be communicated verbally to peers.

The CCS addresses various objectives within the eight common core curricular areas and specific SLOs within the CACREP (2009) Standards. Table 2 presents sample CCS items that are aligned to CACREP core domains and SLOs. Therefore, counselor educators may utilize domains of the CCS to assess SLOs in other courses within the counselor education curriculum, providing a mechanism to document strengths and concerns of students to present to the students and also discuss with other faculty during formal student review processes. For example, counselor educators may use the CCS to assess students during the experiential component of a group counseling course, in regard to several areas (e.g., counseling skills, self-awareness, emotional stability, openness to feedback). Within the multicultural counseling and development class, counselor educators may assess these areas in addition to multicultural competencies. Furthermore, the CCS supports consistency and continuity in the evaluation of SLOs, enhancing programmatic effectiveness and student development.

OPEN IN VIEWER

Table 2.

Sample CCS Items That Correspond to CACREP (2009) Core Curricular Areas and SLOs

CCS Items and Descriptor	CACREP (2009) Core Curricular Area	CACREP (2009) SLOs
Professional Ethics: Adheres to the ethical guidelines of the ACA, ASCA, and IAMFC, including practices within competencies	Professional Orientation and Ethical Practice (Section II, G.1.b; G.1.j)	Demonstrates the ability to apply and adhere to ethical and legal standards in clinical mental health counseling (Clinical Mental Health, B.1)
Self-awareness: Demonstrates an awareness of his or her own belief systems, values, needs, and limitations (herein called “beliefs”) and the effect of “self” on his or her work with clients	Social and Cultural Diversity (Section II, G.2.b; G.2.d; G.2.e; G.2.f)	Demonstrates the ability to recognize his or her own limitations as a marriage, couple, and family counselor and to seek supervision or refer clients when appropriate (Marriage, Couple, and Family, D.6)
Professional literature: Researches therapeutic intervention strategies that have been supported in the literature and research	Research and program evaluation (Section II, G.8.a; G.8.e)	Understands the outcome research data and best practices identified in the school counseling research literature (School Counseling, I.5)

Limitations

Despite advantages of using the CCS, the instrument does not fully address all SLOs identified by the CACREP (2009) Standards; therefore, it is crucial to use the CCS with other assessments. Also, a significant challenge with evaluating SLOs is consistency in ratings (interrater reliability). The CCS promotes objectivity through the use of the scoring rubric with operationally defined items, reducing the subjectivity associated in rating counseling competencies (Moskal & Leydens, 2000). However, concerns regarding consistency were evident in the low interrater reliability correlations, ranging from .436 to .570, within the initial exploration of the psychometric properties of the CCS (Swank, 2010; Swank et al., 2012). Eriksen and McAuliffe (2003) recommend training to increase interrater reliability, which may include reviewing the manual and using the counseling training sessions to practice scoring the CCS. However, interrater reliability remains a challenge for instruments that are designed to assess areas that are difficult to quantify, such as counseling competencies. In addition to extensive training, counselor education programs may address reliability concerns through the implementation of a policy, which requires at least one additional evaluator to assess counseling students’ demonstrated competencies during their clinical courses. In addition, the CCS is the only instrument for measuring SLOs in the area of counseling competencies. Therefore, counselor educators are cautioned in using the CCS as a sole source for decision making, as stronger evaluation processes include multiple measures of a construct such as counseling competencies.

In summary, counselor preparation programs have the responsibility of training competent and ethical counselors, which presents challenges for counselor educators and supervisors. Nevertheless, the CCS (Swank et al., 2012) offers counselor educators a mechanism for assessing their students’ development of counseling competencies throughout their program of study, supporting CACREP (2009) Standards and ACA (2005) Code of Ethics. In addition, the CCS presents counselor educators with a resource to use in teaching students about the construct of counseling competence and to develop their self-awareness and ability to self-assess. Thus, the CCS, with initial empirical support, provides counselor educators with a comprehensive assessment to evaluate SLOs in the domain of counseling competencies.