Abstract
The 2009 Council for Accreditation of Counseling and Related Educational Programs Standards call upon counselor educators to assess student learning outcomes (SLOs) throughout the curricula. This article provides an overview of content and process considerations for identifying SLOs; developing assessments; creating measures; collecting, managing, and reporting data; and making meaning and implementing changes at student- and program levels. A case example is provided.
As we enter the third decade in this era of accountability in higher education, the assessment movement is growing in perceived legitimacy among faculty members, centralized policies are requiring greater accountability within higher education, regulation focus is shifting from state mandates to accrediting bodies, and related assessment technology is booming (Ewell, 2009). As part of this movement, regional and specialty accreditation bodies have switched focus from an input-based process focused on what programs do to an outcome-based approach in which stakeholders examine what students learn and do as a result of their experiences (Council for Higher Education Accreditation [CHEA], 2011; Urofsky, 2009). Provezis (2010) found that the seven regional accrediting bodies were more alike than different in that they were following the Council for Regional Accrediting Commission’s Principles for Good Practices, requiring attention to student learning outcomes (SLOs), expecting public disclosure of findings, expressing concerns about deficits in SLO assessment across institutions, and providing assistance to institutions in the SLO assessment process. In light of this awareness, Kuh and Ewell (2010) named accreditation as “the primary vehicle for quality assurance in American higher education and the major driver of learning outcomes assessment” (p. 23).
Expectations for SLO assessment are not limited to institutional assessment; CHEA’s (2010) most recent Policy and Procedures document requires that all recognized accrediting organizations provide evidence that they have “standards or policies that include expectations of institutional or program quality, including student achievement, consistent with mission” (p. 5). Engagement in assessment activities is often linked to accreditation requirements (Kuh & Ikenberry, 2009); however, a national study of program-level assessment practices indicated that academic programs are assessing learning outcomes to improve programs and instruction (Ewell, Paulson, & Kinzie, 2011). The authors noted that these assessment practices are undersupported in many ways and urged professional associations to take the lead in educating stakeholders about student learning assessment procedures and strategies for success.
It is within this context that the Council for Accreditation of Counseling and Related Educational Programs (CACREP) Standard I.AA (2009) specified that program faculty members “. . . engage in continuous systematic program evaluation indicating how the mission, objectives, and student learning outcomes are measured and met” (p. 8). Individuals familiar with the 2001 Standards will note that Standards I.AA.1, 2, 3, 5, and 6 are nearly identical to 2001 requirements and will require little change in program functioning. However, the addition of Standard I.AA.4, along with the specification that programs “provide evidence that student learning has occurred” for program area SLOs (pp. 18, 24, 30, 36, 40, 47), has the potential to change the face of counselor education quite drastically. No longer can programs simply use syllabi to document where they teach core curricular experiences or rely upon indirect global assessments (Urofsky, 2009) such as satisfaction indices, employment rates, or the National Counselor Examination (NCE) to demonstrate evidence of effectiveness. In essence, counselor education programs are now being held accountable for programmatic outcomes in ways that parallel requirements for agencies and school counseling programs.
Ideally, assessment of SLOs should be an “enactment of the moral and professional commitments of faculty, staff, and institutions, rather than as a reaction to externally imposed obligations” (Blaich & Wise, 2011, p. 7). Although faculty involvement and buy-in are critical to the development of authentic assessment processes (Banta, Griffin, Flateby, & Kahn, 2009; Provezis, 2010), faculty response to the accountability movement has ranged from wholehearted endorsement to resigned acceptance to angry resistance (Ewell, 2009; Hussey & Smith, 2008). In response to findings of lackluster faculty involvement and support for assessment (Kuh & Ikenberry, 2009), Hutchings (2010) speculated that this disconnection may be fueled by perception of assessment as an externally mandated practice, alienating language, lack of faculty preparation for evaluation tasks, lower overall value for promotion and tenure processes, and a perception that assessment does not make a difference.
Given that a review of the counselor education literature failed to yield even one article regarding assessment of SLOs in our profession, we believe many counselor educators may benefit from a roadmap that demonstrates how to shift paradigms from input-based to outcome-based accountability while contextualizing this process as one that will ultimately enhance counselor educator and practitioner competence. In the following sections, we provide a framework for SLO evaluation in counselor education in which faculty members identify SLOs; develop assessments; create evaluation measures; collect, manage, and report data; and make meaning and implement changes related to data. Within each section, we propose both content and process considerations for faculty members.
Identifying SLOs
Just as researchers must identify research questions prior to developing research studies, the first step in developing a comprehensive SLO assessment system is identifying common expectations for knowledge and skill attainment (Banta et al., 2009). In the broadest sense, an SLO is “a stated expectation of what someone will have learned” (Driscoll & Wood, 2007, p. 5) that answers the question “What do your students need to be able to DO ‘out there’ (in the rest of life) that we are responsible for in the classroom?” (Stiehl & Lewchuk, 2008, p. 6). It is important that the SLOs identified represent a “clearly articulated, collective conception” of the ideal candidate so that programs may use “intentionality and coherence . . . to cultivate those qualities” (Association of American Colleges and Universities [AACU], 2008, p. 1). In essence, what students do, know, or value, must be linked to the mission and goals of the program and profession (CACREP, 2009; CHEA, 2011; Gardiner, Corbitt, & Adams, 2010), so SLOs may be aligned with curriculum and pedagogy employed within the program (Banta et al., 2009).
The 2009 CACREP standards specify between 36 (Counselor Education and Supervision) and 69 (School Counseling) SLOs for which each program must assess each student. All master’s level program area standards include SLOs regarding Foundations; Counseling, Prevention, and Intervention; Diversity and Advocacy; Assessment; and Research and Evaluation. The Standards provide additional SLO categories for Addiction (Diagnosis), Career (Program Promotion, Management, and Implementation; Information Resources), Clinical Mental Health (Diagnosis), and School (Academic Development, Collaboration and Consultation, and Leadership) Counseling program areas. The doctoral level Counselor Education and Supervision SLOs are organized into categories regarding Supervision, Teaching, Research and Scholarship, Counseling, and Leadership and Advocacy.
The SLOs required by CACREP vary in specificity and program area focus. For example, “Demonstrates the ability to use procedures for assessing and managing suicide risk” (pp. 20, 32, 37, 41, 49) emerges as a SLO for all master’s level program areas except Career Counseling, whereas “Knows how to design, implement, manage, and evaluate a comprehensive school counseling program” (p. 45) is unique to School Counseling. Although we believe the 2009 Standards represent a significant step forward in keeping with the current state of higher education, the standards at times fall victim to what Hatfield (2009) dubbed “laundry lists” and “Rubrik’s cube-like outcomes” that can be difficult or impossible to evaluate (p. 3). For example, Clinical Mental Health Counseling (CMHC) Standard D.5, “Demonstrates appropriate use of culturally responsive individual, couple, family, group, and systems modalities for initiating, maintaining, and terminating counseling” (p. 32) might be better presented in a 5 × 3 matrix that encompasses 15 learning outcomes. As the standards continue to evolve, counselor education faculty will need to be creative and persistent when conceptualizing how to go about understanding and assessing SLOs.
Process Considerations
Current practices in educational facilities throughout the United States have contributed to a perception that SLO assessment is about standardized testing rather than a meaningful process that may result in stronger student achievement (AACU, 2008). Counselor educators may consider the ways in which SLO evaluation has the potential to enhance quality of instruction and, in turn, counselor effectiveness. In addition, we encourage counselor educators to think critically about content areas that are and are not included in the current edition of the SLOs. Because the SLOs provided by CACREP are minimal standards, they may not always capture the essence of unique programs. A careful understanding of the SLOs required by CACREP will help programs determine whether they want to supplement the assessment process by adding program-specific SLOs. Because CACREP already requires that programs assess numerous SLOs per program area, faculty members must balance identification of unique SLOs with consideration regarding feasibility of managing additional dimensions.
After ensuring an adequate understanding of the SLOs to be assessed, program faculty will need to determine who will be responsible for spearheading the development and implementation of the assessment plan. As will be discussed in the following pages, the planning process can be intense, and universities seldom provide release time, administrative staff, or other resources to assist with program-level evaluation (Ewell et al., 2011). Because faculty engagement is key to success of the process (Hutchings, 2010), programs may wish to develop an SLO evaluation committee comprised of representatives from each program area and field experience component.
For centuries, educators have been requiring students to engage in a variety of tasks that assist in acquisition and demonstration of knowledge and skills related to course content. As they begin the SLO evaluation planning process, counselor educators may take inventory of the ways in which they are already assessing learning outcomes within the unique culture of the program. Key questions for the inventory include: (a) to what degree do the CACREP Standards capture the SLOs most important to our faculty, (b) where in the curriculum do we cover the foundation knowledge, skills, and processes necessary for the attainment of each SLO, (c) in what ways to we need to modify our curricula to ensure adequate foundations, (d) where and when is best to assess this learning outcome, and (e) in what ways are we already assessing this SLO?
Developing Assessments
Once SLOs are aligned with pedagogy, faculty members may begin developing corresponding assessment methods and measures (Banta et al., 2009). There is wide-reaching agreement that there is no one-size-fits-all approach to SLO assessment (AACU, 2008; Provezis, 2010; Urofsky, 2009). Nationally, the most frequently employed program-level assessments include: portfolios; course-level assessments; rubrics; external judges; student, employer, and alumni interviews; and employer, local, alumni, and national surveys (Kuh & Ikenberry, 2009). Prior to the 2009 CACREP standards, counselor education programs relied almost exclusively on indirect measures of student learning for program evaluation. In essence, programs were assessing perceptions of student learning rather than measuring actual student learning or performance (Martell & Calderon, 2005; Price & Randall, 2008). Within the context of SLO evaluation, indirect measures such as interviews and surveys will not substitute for direct measures of student learning key to SLO assessment (Hatfield, 2009). However, information obtained from indirect measures of students, site supervisors, and employers perspectives as required by CACREP (2009) Standard I.AA can help counselor educators make meaning from the results of direct measures. Information on implicit qualities of learning, such as feelings, attitudes and values, can be obtained through surveys (i.e., graduates, employers, supervisors, alumni), graduation and retention rates, exit interviews, and disposition reviews. By utilizing both indirect and direct measures from multiple sources, counselor educators can triangulate data and construct a comprehensive picture that is reliable and consistent.
Of key concern for SLO assessment is distinguishing one-shot testing from quality assessment in which students may collaborate, utilize resources, and integrate learning (Stiehl & Lewchuk, 2008). Meyers and Nulty (2009) charged educators with developing
courses in ways that provide students with teaching and learning materials, tasks and experiences which: (1) are authentic, real-world and relevant; (2) are constructive, sequential and interlinked; (3) require students to use and engage with progressively higher order cognitive processes; (4) are all aligned with each other and the desired learning outcomes; and (5) provide challenge, interest and motivation to learn. (p. 567)
In turn, they proposed that students will arrive at a place of deep learning while simultaneously demonstrating SLOs necessary for success. A national study of program-level assessment plans found that most programs are already assessing some degree of SLOs via capstone experiences (Ewell et al., 2011). This finding is in keeping with Urofsky’s (2009) encouragement for educators to take a “multi-pronged” and integrated approach to skills assessment (p. 8) and AACU’s (2008) statement that quality learning assessment should include “students’ authentic and complex performances in the context of their most advanced studies: research projects, community service projects, portfolios of student work, supervised internships, etc” (p. 9).
Types of Assessments
Counselor educators will likely call on students to engage in a combination of four assessment activities when developing integrated SLO evaluation plans: course-based assessments, field experiences, comprehensive examinations, and portfolios. In each case, program faculty members must develop the assessment activity with clear and intentional links to the SLOs faculty members wish to capture in the process.
Course-based assessments
Course-based assessments require students to demonstrate what they value, know, and can do through actual work completed in courses (Gardiner et al., 2010; Price & Randall, 2008). Pringle and Michel (2007) suggested that learning goals or outcomes be “broken down into specific characteristics or traits that can be measured” (p. 203). Examples of course-based assessments include but are not limited to papers, presentations, speeches, graded assessment items, and tests (Price & Randall, 2008). Within the context of counselor education, role-plays and demonstrations may also serve as important course-based assessments. As will be discussed in the next section, faculty members evaluate performance on course-based assessment SLOs with specially designed rubrics or rating forms.
Course-based assessments lend themselves most readily to assessment of knowledge SLOs, emerging skills and practice SLOs, and skills and practices that students may not have ample opportunity to demonstrate within practicum and internship. Given the sheer number of SLOs required in the 2009 CACREP Standards, instructors may develop course-based assessments that capture multiple SLOs within one program area or, when activities are located in the common core, across program areas. For example, a course required for all marriage, couple, and family counseling students may include an examination upon which students demonstrate proficiency with four or five knowledge SLOs within the program area. A research and evaluation instructor may call upon students across program areas to demonstrate knowledge of program evaluation by creating evaluation plans for programs in their respective settings, thus meeting several standards within each program area. Finally, an instructor who realizes that students may not have an opportunity to engage in diagnosis following a crisis, disaster, or trauma-causing event may create an opportunity to simulate such an assessment.
Field experiences
The minimum of 700 hours of field experiences required for graduation from CACREP-accredited programs provides a rich opportunity to assess in vivo application of Skills and Practices SLOs. Program faculty will likely embed assessments of observations, conceptualizations, and field experience projects into field experience requirements. In many cases, these observations will be documented via field experience rating forms completed by instructors or site supervisors. While this allows for consideration of authentic experience, confidentiality considerations may limit the degree to which faculty members are privy to direct experiences and can maintain artifacts as evidence of student attainment of SLOs. In this case, faculty members may create additional assessments of learning via field experience, such as deidentified case conceptualizations that will serve as additional evidence of SLO attainment.
Comprehensive examinations
Comprehensive examinations provide an opportunity to assess overall attainment of SLOs and integration across program areas; however, examinations must be tailored to match specific SLOs and provide sufficient opportunity to assess depth of understanding and, when relevant, student ability to apply and synthesize information. Although it is possible that multiple-choice examinations will assess some SLOs, the CACREP Standards require much more depth than is typically associated with closed-ended formats.
Given that over 300 college and universities were utilizing the Counselor Preparation Comprehensive Examination (CPCE) as an exit examination (www.nbcc.org), we suspect that many programs may be using CPCE results and NCE pass rates as program evaluation measures. Essentially, the CPCE provides feedback regarding general performance on the eight CACREP core curricular areas. Although the CACREP SLOs assume some degree of mastery of these core areas, CACREP SLOs are much more specific and dynamic, thus limiting usefulness of the CPCE to general assessment of overall performance and feedback to students regarding examination areas in which they are strong or need development.
Portfolios
Similar to counselor education programs, recent changes in teacher education accreditation standards require programs to “systematically collect, analyze, and use data from a wide variety of sources across the duration of a student’s experience” (Swan, 2009, p. 627). Teacher education programs in colleges and universities have a long history of using performance portfolios as a collection of artifacts and reflective learning tool for students (Banta et al., 2009; Beck, Livne, & Bear, 2005; Strudler & Wetzel, 2005, 2008). Portfolio assessment has evolved from simply a method to evaluate students’ performance on standards-based curriculum to a method that also generates data to evaluate programs’ practice, quality, and outcomes for accreditation purposes (Meyer & Latham, 2008). Banta et al. (2009) noted advantages of portfolios include opportunities to link course and field experience, inclusion of voluntary artifacts, and opportunity to assess complex outcomes via reliable rubrics. Disadvantages include trade-offs between contextual validity reliability and technology considerations. Due to the comprehensive nature of portfolio assessment, we recommend it as a promising method of collecting and managing counselor education program data and SLO information.
Historically, the use of hard copy portfolios in teacher education student and program assessment has been prevalent (Swan, 2009). Much of the criticism of using hard copy portfolios is related to the amount of time in constructing, formatting, and managing information included in them. In the past decade, teacher education programs have moved to requiring electronic portfolios to address these criticisms (Bartlett, 2002; Gathercoal, Love, Bryde, & McKean, 2002; Smith, Harris, & Sammons, 2001; Williams, Wetzel, & Wilhelm, 2004; Yancey, 2001). Advances in computer technologies have made using electronic portfolios more viable than using hard copy portfolios in higher education settings (Swan, 2009).
One of the first considerations in using an electronic assessment portfolio is to clarify the purpose of the portfolio (Strudler & Wetzel, 2008). Counselor education faculty need to consider if the portfolio is to act as a reflective learning tool, a tool to document students’ progress with meeting standards, a tool to document the program’s effectiveness, or a tool that addresses all of these issues. In order for assessment to be as efficient as possible, we recommended that the electronic portfolio be used for all of these purposes. By incorporating direct (i.e., course-specific rubrics) and indirect (i.e., alumni surveys, supervisor evaluation) measures into the portfolio system, programs can design meaningful learning experiences and track SLOs along with more general program-level evaluation requirements.
To begin the process of collecting data, the program and/or college needs to determine how the portfolio will be created. Many colleges of education use Microsoft Office applications and various Web authoring applications to create portfolios (Strudler & Wetzel, 2005). However, there are commercial web-based systems (i.e., Blackboard, Foliotek, Live Text, Tk20) that use a centralized database for storage and management of information. In making this decision, programs and colleges need to consider costs to student and school, campus technology infrastructure, and continual user support (Wetzel & Strudler, 2005). Additional considerations for collecting and managing data will be presented in the section regarding data collection and management.
Process Considerations
Because good assessment builds in complexity and expectation (AACU, 2008), it is critical that all faculty members are involved in the process of developing authentic assessments (Banta et al., 2009). Faculty involvement and buy-in are especially important when an assessment activity is shared by several faculty members, is tasked to adjunct or affiliate faculty members, or is located outside of the course context. In addition, faculty members will need to decide when and where is most appropriate to assess (e.g., in curriculum, at major transition point, at multiple points; Ewell, 2009), which individual or group of individuals is responsible for assessing (Banta et al., 2009), and how to implement the process systematically (Hatfield, 2009). Because SLO evaluation is a form of evaluation methodology, it is not unreasonable to plan for a multimethod, multiple source data collection that includes measures assessed by multiple raters (Davidson, 2005). Triangulating data will allow for integration of information about the students and program, while providing information on reliability and validity of the measure (Jonsson & Svingby, 2007; Moskal & Leydens, 2000; Stellmack, Konheim-Kalkstein, Manor, Massey, & Schmitz, 2009).
Creating Evaluation Measures
Due to this shift in focus from program-level to SLO evaluation, educators are beginning to attend to the development of course- and portfolio-embedded measures that assess specific SLOs (Martell, 2007) via performance criteria linked to CACREP standards or student dispositions that have numerically, weighted performance indicators. These assessment measures provide data to program faculty and accreditation bodies (i.e., CACREP, NCATE, and other regional higher education accreditation agencies) that demonstrate the degree to which students have learned knowledge and skills representative of counselors, counselor educators, and supervisors (Price & Randall, 2008). Typically, evaluation measures include rubrics and observation forms.
Rubrics
Rubrics have been defined as documents that “articulate the expectations for assignments by listing the criteria, and describing levels of quality from excellent to poor” (Reddy & Andrade, 2010, p. 435). Specifically, they include three essential features: (1) evaluation criteria; (2) quality definitions; and (3) a scoring strategy (Popham as cited in Reddy & Andrade, 2010). The evaluation criteria include indicators of the content of the assignment (Parke, 2001). This information informs the student and rater of specific parameters for performance (Reddy, 2007). In addition to delineating criteria, the CACREP standards applicable to the course need to be linked to each indicator. If the course is part of the curriculum for several counselor education programs, specific program standards (e.g., School, Clinical Mental Health) should also be delineated.
Quality definitions in a rubric consist of determining the merit of the student’s performance on the measure (Davidson, 2005; Reddy & Andrade, 2010). In essence, the definition is a descriptive explanation of a certain level of skill or practice, such as exceeds, meets, or does not meet expectations. Quality definitions allow faculty to convert descriptive performance into evaluative conclusions in a reliable manner (Davidson, 2005). In this case, counselor education faculty can interpret the degree to which students demonstrate knowledge or skills/practices that encapsulate specific CACREP standards.
Finally, the scoring strategy of the rubric refers to performance, as a whole, on the assessment (Reddy, 2007). Rubrics for course assignments can include component and overall performance scoring with or without weights assigned to each criterion. In addition, faculty members can determine criteria for success on the overall task to include a cutoff score or specification that the student not have any “does not meet expectations” or “inadequate performance” ratings on the rubrics. Please see Table 1 for a sample rubric used to evaluate a course-based SLO assessment regarding assessing and managing suicide risk.
Sample Rubric for Suicide Assessment Demonstration
Although rubrics have been commonly used in teacher education, changes in accreditation standards and accountability policies have resulted in a recent emphasis on rubrics across other higher education programs (Dunn, Mehrotra, & Halonen, 2004; Thaler, Kazemi, & Huscher, 2009). Some faculty members may have to adjust to this process of creating rubrics for assignments that assess SLOs. The rationale that rubrics can serve instructional as well as evaluative purposes may reinforce the efforts of counselor educators in the creation and implementation of rubrics and other direct measures of student learning (Andrade, 2005). For example, the first author has found that using rubrics in courses provides students with a road map for success and decreases the amount of class time spent talking about expectations.
Observation Forms
As with development of rubrics, faculty members who create observation forms must do so while considering the specific criteria to be assessed, quality indicators or definitions, and overall scoring strategy. Often, faculty members may draw from resources for creating rubrics. Whereas faculty members may rate rubrics based on an artifact or portfolio submitted at one point in time, individuals may complete observation forms based on a body of work (e.g., experience of a candidate throughout a supervisory experience). In essence, the observation form is an assessment instrument that requires a degree of validity and reliability; unfortunately, we are unaware of any validated measures that correspond to the CACREP Skills and Practices SLOs.
Process Considerations
Key process considerations in creating measures include establishment of validity and reliability of measures to ensure they distinguish among levels of achievement and training of faculty to use tools in ways that promote reliability and validity (Hatfield, 2009). For the measures to produce meaningful data beyond the immediate context, faculty members need to begin the process with a common understanding of acceptable performance. Generally speaking, use of more descriptive quality definitions will help faculty articulate expectations more precisely; however, specificity may come at the expense of flexibility of creativity. Similarly, faculty members need to consider whether to create a common metric to use with assignments that delineate the same quality definitions and scoring strategies for specific components and overall evaluation, respectively. Although we believe it convenient to agree to a common metric to use throughout the program so ratings have consistent meaning (e.g., 0 = does not meet expectations, 1 = meets expectations, and 2 = exceeds expectations), faculty members may wish to trade this consistency for freedom. Additionally, faculty need to consider which measures will be individually and collectively rated; faculty may discover that course-based measures for specific assignments are more appropriate for individual rating and culminating activities (e.g., comprehensive examinations, portfolios) are more appropriate for collective ratings by multiple stakeholders including faculty and site supervisors. Once measures are developed, program faculties will need to determine optimal ways to train faculty members in consistent use of measures and evaluate the degree to which measures use data consistent with faculty member experiences and priorities.
Collecting, Managing, and Reporting Data
Careful assessment planning and instructional delivery culminates in collecting, managing, and reporting the evidence of SLOs, thus requiring a comprehensive system for collecting and organizing data (Hatfield, 2009). Based on requirements of the 2009 CACREP standards, documenting evidence of student learning has become a rigorous task of counselor education programs. Thankfully, counselor education programs do not need to reinvent the process of collecting, managing, and reporting this copious amount of information.
Data Collection and Management Tools
Options for collecting data vary based on program size and design. Smaller programs or programs that rely upon a culminating portfolio may find it feasible to collect data and aggregate via spreadsheets or databases at key points within a student’s program. Larger programs and those that rely on a series of course-based assessments may find it nearly impossible to keep track of student artifacts and instructor measures throughout the program. In either case, counselor education programs may have adequate support from their program assessment personnel within their colleges and may not need a commercial system. If unsure, we suggest that the program investigate what is needed to collect, manage, and report their program results. Given the current state of SLO demands in higher education, it is highly unlikely that the counselor education program is the only one facing increased expectations. The university may already have access to commercial data collection systems, and program faculty may simply need to enlist the support of college administration in determining what resources are available to implement or access a commercial system. If a commercial system is not available, program leadership may consider inviting vendors to present their product to program faculty and administration will help clarify how the systems will address the needs of the program.
Once the system for collecting data is determined, all faculty members need to buy into this process. Similar to how the counselor education faculty has to adjust to this new level of assessment of SLOs, it also has to adjust to changes in how data are collected and managed. Successful implementation comes from dean and program-level leadership (Wetzel & Strudler, 2005). Additionally, a learning curve should be expected of students, faculty, and program support personnel on learning how to navigate a commercial SLO assessment system that requires students to upload assignments, faculty and supervisors to grade or evaluate students, and program administrators and support personnel to monitor student outcomes and program progress. Learning to navigate this system successfully allows for easier management of data within the system.
Reporting Data
Collecting direct and indirect measures of student learning will provide the counselor education program faculty with information regarding how students have learned and applied CACREP standards. Beyond determining documentation of specific-SLO measures, program faculty will need to decide on the timeline for collecting and reviewing program-wide data (i.e., dispositions, standardized assessment data from certification, and licensure boards). Once those decisions are made and data are collected, data must be aggregated and disaggregated to demonstrate how the program is meeting standards (Strudler & Wetzel, 2005).
An advantage of a portfolio system is the ability to examine efficiently counselor education students as a whole population (aggregate) or by specific characteristics (disaggregate; Meyer & Latham, 2008). Although commercial electronic portfolio systems make it easier to manage data once initial implementation, college-created portfolio systems can also provide data on students within and across program areas (i.e., school, addictions, mental health). Programs may also be interested in examining student data across cohorts, graduates, and alumni.
Once data are collected and reviewed by the program faculty, stakeholders (i.e., college administrators, site supervisors, community advisory board members, alumni, and students) should be informed of the results. Additionally, CACREP (2009) requires programs to report results to the public. Hard copy or electronic reports should be created on an annual basis and sent to the identified stakeholders. A web-based version of the report can be created and integrated into the counselor education program’s website for public view. The program evaluation summary report may include key evaluation data (e.g., alumni survey, site supervisor evaluations, licensure pass rates) and SLO attainment for each program area. Additionally, modifications based on this data need to be outlined for each program area. An example of a program summary report can be found at the University of North Texas (2012) Counseling Program website.
Process Considerations
Once faculty members reach this stage in the SLO evaluation process, process considerations become more technical and removed from the program planning process. For example, faculty members need to decide how specifically to collect the data and who is responsible for ensuring accurate completion of assessment commitments on a semesterly and annual basis so that specific tasks do not get left behind. Programs that rely on adjunct and affiliate faculty members will need to determine the best ways to facilitate buy-in, orient to expectations, and help individual faculty members understand how specific activities link to the bigger picture. In our experience, the university or college may provide a staff person who can work with faculty to engage in these tasks. Faculty members who are tasked with day-to-day data collection and management of data may work with administrators to negotiate release time or graduate assistance as compensation for this responsibility.
Making Meaning and Implementing Changes
Within the higher education assessment literature, there seems to be a degree of consensus that focus has been on finding ways to collect and manage data rather than on using the data to inform practice, leaving Blaich and Wise (2011) to advise that “assessment data has legs only if the evidence collected rises out of extended conversations across constituencies about (a) what people hunger to know about their teaching and learning environments and (b) how the assessment evidence speaks to those questions” (p. 12). Similarly, Kuh and Ikenberry (2009) referred to the integration of assessment and pedagogy as “yet unrealized goals” (p. 27).
According to CACREP (2009), data collected in the counselor education program need to be systematic, with a demonstrated use for program improvement. Hence, the faculty has to consider data on course-specific assignments for remediating concerns with specific students as well as making improvements in course delivery, content, and application. This includes examining how and if all content standards are being met; examining program implementation, such as admissions, remediation, field experiences, certification/licensure, retention, and graduation; and determining appropriate program modifications to address deficits identified. Reviewing the data should occur once a year at a minimum. Certainly, the ability to show action on results of assessment and use data to improve teaching and learning should be at the center of our assessment practices (Ewell, 2009; Hatfield, 2009). Faculty members will need to consider how to utilize data to make decisions at both the student and program levels.
Student-Level Decisions
At the heart of CACREP (2009) SLO requirements is an expectation that programs can determine when specific students are not meeting SLOs and take action to remedy the concern. Thus, programs need not demonstrate that every student meets every learning outcome; if all students were engaging with material on the same level, we would suspect concerns with validity of the evaluation plan. Program faculty will need to find ways to integrate the systematic program evaluation plan with the program’s “systematic developmental assessment of each student’s progress throughout the program, including consideration of the student’s academic performance, professional development, and personal development. . .” (I.P, p. 4) and corresponding student retention policy. Certainly, student-level decisions should be grounded in best practice literature regarding student review and retention (e.g., Foster & McAdams III, 2009; Kress & Protivnak, 2009; McAdams III & Foster, 2007; McAdams III, Foster, & Ward, 2007). See Appendix A for a sample policy statement regarding integration of SLO assessment with program review and retention.
Program-Level Decisions
In addition to assisting with individual student review, SLO evaluation activities should result in aggregate data that help faculty members make meaning regarding overall strengths and weaknesses of the program. Of course, quality of data generated links directly to validity of student assessments and corresponding measures. Thus, faculty members should first ascertain the degree to which the assessments are yielding useful data. For example, if all students are assessed as “exceeds expectations” on a measure, it is likely that the instrument lacks sensitivity and that users are not discriminating among levels of SLO attainment. Similarly, sharp drops in SLO performance from one experience to the next or poor interrater agreement within an assessment may indicate need for a change in the SLO evaluation plan or implementation rather than the curriculum.
Once program faculty members have determined that the assessments and measures are valid, they may begin evaluating the degree to which students are meeting SLOs within and across program areas. For example, if program faculty note relatively higher ratings on diagnosis Knowledge SLOs compared to diagnosis Skills and Practice SLOs, it may be that coursework regarding diagnosis would benefit from enhanced opportunities to put understanding into practice. Relatively lower field experience ratings on the School Counseling leadership items may indicate an opportunity to revisit curricula or implement minifield experiences around school counselors’ roles as system change agents. For all program changes, faculty members will need to document the specific change, methods for evaluating the effectiveness of the change, and corresponding changes to the program evaluation plan.
Process Considerations
Process considerations for making meaning and implementing changes on the student level include determining who is responsible for facilitating and documenting the overall student review (e.g., instructor who notes a concern, student advisor, student at program gateway). As part of this process, program faculty members will need to define satisfactory progress or performance as it corresponds to student grade and performance on specific assessment measures. In some instances, program faculty may consider “success” in overall context of an assignment, in other instances a “does not meet expectations” on a field experience item regarding ethical behavior may necessitate immediate, intensive remediation efforts. If using a commercial data collection system, faculty members will also need to set a schedule for routine evaluation of data so they act on information in a timely manner.
On the program level, faculty members need to determine when and to change assessments and corresponding measures. Because mid-stream changes to items makes comparison across time and students difficult or impossible, faculty members will need to find optimal balance for making changes and keeping the program fresh without changing assessments and measures simply for the sake of change. In addition, the program faculty will need to determine who is responsible for aggregating and reporting the data. Although it may be tempting to charge this task to one administrative leader, faculty members may find themselves more connected to the process if they take turns examining data and making recommendations related to specific program areas or experiences.
Case Example
At the time the 2009 CACREP Standards were released, the first author was serving as program coordinator for a large CACREP-accredited Counseling Program at a public university in the southwest United States. The following case example is provided to illustrate the process through which the program faculty developed and implemented each element of the SLO evaluation plan.
Identifying SLOs
Shortly after the release of the 2009 CACREP Standards, the program’s master’s and doctoral curriculum committees began meeting to evaluate curricular changes necessary to ensure adherence to the new standards. Meanwhile, the program engaged in a large-scale alumni and employer follow-up study to determine constituent perception of graduate preparation in key SLO areas. Because the program was not routinely assessing SLOs throughout program areas, the faculty used the results of the comprehensive program review to guide curricular changes.
Once the program solidified curricular changes necessary for transition to the 2009 CACREP Standards, the program coordinator provided the faculty with a matrix that included a list of all SLOs, curricular experiences in which material relevant to the SLO were covered, and a space to indicate existing course assignments and field experience activities that corresponded to the SLO. When faculty members were unable to locate ways in which courses provided a foundation for SLOs, the lead instructor for the course was charged with adapting the course accordingly. This resulted in a series of program-endorsed “CACREP syllabi” that included attention to core curricular experiences to be included (II.G Standards) as well as SLOs to be assessed in the course or field experience.
Developing Assessments
Upon completing the SLO tasking matrix and adjusting curricula accordingly, the program faculty began the process of determining which assessment procedures were already in place and which procedures would need to be added or changed to meet new standards. The faculty noted that they were already assessing a number of master’s level SLOs via course-based papers, projects, and examinations that tended to stay fairly consistent across course sections and semesters; however, assignments varied in degree of specificity, and some SLOs were not addressed at all. During the next phase of the process, lead instructors were charged with enhancing or developing course-based assessment procedures to correspond with CACREP SLOs. Upon developing the assessments, instructors presented the proposed course-based plans for approval by the faculty during a series of program retreats. Once approved, all instructors agreed to require the assignment as a “key assessment” in all sections of the course until agreed upon otherwise. As questions regarding academic freedom arose, the faculty discussed freedom to vary methods for teaching required concepts and weight of the activity in the larger scope of the course. The faculty agreed that faculty members assigned to the corresponding courses would be responsible for assessing all course-based assessments.
In addition to offering a number of course-based assessments, the program faculty was already utilizing field experience rating forms for university and site supervisors. However, the rating forms tended to be focused on demonstration of micro skills rather than the more systemic skills highlighted within the CACREP (2009) SLOs. The faculty agreed to modify existing field experience assessments to better capture Skills and Practices performed by students in culminating experiences. This resulted in a series of field experience rating forms corresponding to each accredited program area. Similarly, the program faculty noted strengths in field experience assessments of doctoral student counseling experiences; however, they realized they did not have standardized supervisor evaluation of supervision nor were they routinely utilizing student teaching observations. In order to better document performance regarding these SLO areas, the program faculty agreed to develop new field-based measures for the doctoral program.
Upon further exploring the SLOs that needed to be assessed, the program faculty considered utility and feasibility of employing a comprehensive examination within the master’s program areas. The faculty was using the CPCE in fulfillment of a university requirement for a Master of Science comprehensive examination and, in turn, using results to evaluate general learning within CACREP core curricular areas. Due to program size and the reality that students are focused on field experience and licensure examinations at the end of their programs, the faculty decided against changing to a comprehensive examination focused on CACREP SLO attainment. In contrast, the faculty was already using a doctoral qualifying examination to assess student learning regarding five areas that corresponded to the program’s doctoral level objectives. Upon further examination, the faculty determined that the doctoral qualifying examination could be modified slightly to serve as an assessment of multiple doctoral SLOs. Thus, the faculty revised the assessment and agreed to maintain the practice of using three blind raters to triangulate assessment of doctoral level SLO mastery.
Finally, the program faculty considered whether a portfolio assignment would be an appropriate assessment within the program context. Given the size and complexity of the master’s program, the faculty decided against a program-wide portfolio requirement. However, the school counseling faculty noted a unique opportunity to organize school counseling SLOs via course-based portfolios. Thus, miniportfolios were developed and implemented as course-based assessments in two school counseling courses. Given the smaller size and more general nature of SLOs regarding the doctoral program, the faculty decided to incorporate a doctoral program portfolio to highlight student attainment of SLOs regarding supervision, teaching, counseling, scholarship, and professional activity. In turn, the program operationalized program expectations for each area to include (a) reflective statements to accompany each major area, (b) specific experiences to be completed, (c) examples of artifacts to support evidence of learning in course-based assessments and field experiences, (d) procedures for developing the portfolio, and (e) the specification that the portfolio be presented to the students’ doctoral advisory committee for evaluation at the time of dissertation proposal and defense.
Creating Measures
Upon agreeing to assessment activities for each accredited program area, the faculty began the process of developing corresponding measures for evaluating SLO attainment. At the beginning of the process, the program coordinator invited the university’s Assessment and Measurement specialist to provide an in-service training regarding SLO assessment measures and resources available to faculty members. In several cases, program faculty members had already developed rubrics and were able to share examples with faculty members who were discovering rubrics for the first time. Given the degree of autonomy valued by the program faculty, time constraints, and the number of rubrics already in existence, the faculty agreed to create measures using whatever scoring system they found most relevant for course-related activities. In addition, the program’s clinical experiences committee worked together to revise a series of field experience rating forms so that they captured Skills and Practices items specific to each program area. After developing the measures, faculty members presented them to the entire faculty for approval; the program coordinator also evaluated measures independently to ensure that they tied logically to the SLOs they were intended to assess. In turn, the faculty agreed to use measures for a trial period of 1 year after which measures would be evaluated and, if needed, modified to better meet the needs of students, faculty, and site supervisors.
Collecting, Managing, and Reporting Data
During the semester in which the faculty was developing assessments and measures for SLO evaluation, the College adopted a commercial SLO management system, Tk20, to assist with NCATE reaccreditation. In turn, administrators agreed to provide the program with access to Tk20 for SLO assessment. The program coordinator worked with administrative staff to upload all master’s level course-based assessments, rubrics, and field-based observation forms to the system so that they could be rolled out for use in courses and field experiences. In addition, the faculty notified students regarding the need to purchase a subscription to the program and worked with faculty, adjuncts, and site supervisors to ensure timely and appropriate use of the assessment system. At the beginning and end of each semester, the program coordinator reminded faculty of their agreed upon assessment tasks and checked system reports to identify instructors who needed additional assistance fulfilling their SLO commitments. In addition, she downloaded reports of field experience ratings and routed reports to corresponding faculty members so they could identify specific students and sites of concern.
Making Meaning and Implementing Changes
As part of the program’s evaluation plan, the program faculty agreed to review SLO and other program-level assessment at a faculty meeting in October of each year. In preparation for this meeting, the program coordinator provides faculty leaders with summary data from Tk20 regarding assessments in their areas. In turn, the faculty leaders summarize data for the official report and come to the faculty meeting prepared to present results and discuss possible changes to assessment procedures, curricula, and program policies.
Conclusions
As we have illustrated, the SLO evaluation cycle includes identifying SLOs; developing assessments; creating measures; collecting, managing, and reporting data; and making meaning and implementing changes. Researchers suggest that implementation of SLO assessment systems should take place in incremental, manageable steps, possibly spanning 2–3 years (Meyer & Latham, 2008; Wetzel & Strudler, 2005). Additionally, programs should not wait until technology is available in creating a portfolio. Many elements of the portfolio can be created and implemented prior to adapting it to an electronic version. Finally, faculty members are encouraged to support each other through this process. Some individuals may be able to grasp the vision of the systematic evaluation process more quickly than others. These individuals need to develop a teaching and helping stance with colleagues who may be frustrated with measurement tasks. Reminding each other of the main focus of program improvement and student learning will provide the impetus for learning this process of systemic evaluation. Unfortunately, programs across the U.S. report lacking institutional support and expertise to do good program assessment (Ewell et al., 2011). Thus, counselor education faculties will need to decide how to integrate SLO assessment into the regular work of teaching and learning within our service-heavy profession. Certainly, faculty members may advocate for additional resources and find ways to value the intensive work associated with SLO assessment when it comes to promotion and tenure (Hutchings, 2010).
As counselor educators adjust to the new practice of SLO evaluation practices, they may work with professional associations such as the Association for Assessment in Counseling and Education and Association for Counselor Education and Supervision to share experiences and resources via a centralized clearinghouse. Certainly, much research is needed regarding the impact of SLO evaluation on counselor education. In particular, researchers may seek to understand the impact of SLO evaluation on student learning and counselor competency; the effectiveness of specific approaches to counselor education; and the validity and reliability of assessment methods and measures tailored to CACREP SLOs.
Footnotes
Appendix A
The Counseling Program faculty uses the following methods to engage in systematic, developmental assessment of each student’s progress.
Further, the counseling program faculty has adopted a series of key assessments that require students to demonstrate knowledge and learning outcomes for the course. Students upload key assessments to Tk20, and instructors evaluate progress and identify areas for remediation using standardized rubrics. Students not meeting learning outcomes associated with courses may be required to do additional work before moving on. See course syllabi for those assignments designated as “KEY.”
In addition to the program gateways identified above, students are evaluated throughout the Counseling Program on the criteria of knowledge, personal characteristics, interpersonal skills, and counseling skills to determine whether the progress is adequate, if remedial work is needed, or if the student should be withdrawn from the program. At any faculty meeting, a faculty member may bring up a student about whom the faculty member has concerns. In addition, the faculty specifically addresses concerns at the last faculty meeting of each semester. Any decision involving remediation or withdrawal follows the procedure specified for reviewing student’s professional competency that appears on the following pages.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.