Outcome-Based Assessment in Counselor Education

Abstract

The 2009 Council for the Accreditation of Counseling and Related Educational Programs Standards present a move to outcome-based assessment of student learning outcomes. The authors developed an assessment system for counselor education programs through a Delphi study (n = 9). The proposed assessment system and recommendations for implementation using existing program assignments and developing program practices around the proposed system are presented. Future research areas are outlined including opportunities to further develop the proposed system and explore other means of systematic program assessment of student learning outcomes.

Keywords

CACREP assessment standards student learning outcomes

The Council for the Accreditation of Counseling and Related Educational Programs (CACREP) require programs to submit evidence of student mastery of CACREP Standards (CACREP, 2009). Programs are to create assessment systems with consistent assessment measures to ensure accuracy in the interpretation and application of any assessment system (Bresciani, 2008; Brindley, 2001; Kelly, 2011). Consistent with the research on outcome-based assessment (e.g., Brindley, 2001; Donnelly, 2007; Frank & Danoff, 2007; Jansen, 1998), the authors conducted a Delphi study to develop a template for a comprehensive assessment of student learning that can be implemented in CACREP-accredited programs.

CACREP-accredited programs tend to demonstrate commitment to continuity in curriculum and assessment, and simultaneously exercise faculty academic freedom in their delivery and evaluation of course material as evidenced by the myriad and diverse ways that programs meet standards. It may be challenging then for counseling faculty within a program to use the same form to evaluate different assignments. Methods of assessment, therefore, become important to balance programmatic consistency with faculty flexibility (Bresciani, 2008).

A comprehensive portfolio is a commonly documented assessment method of students’ work. Portfolios provide a comprehensive overview of students’ experiences and professional preparation for use with employment applications and interviews (Heath, 2005; Pecheone, Pigg, Chung, & Souviney, 2005; Rhyne-Winkler, & Wooten, 1996; Strudler & Wetzel, 2008) or simultaneous assessment of several curricular experiences (Rogers & Chow, 2000). They may be useful for assessing and promoting counselor development (Baltimore, Hickson, George, & Crutchfield, 1996; Cobia et al., 2005) and multicultural competency (Coleman, Morris, & Norton, 2006). Although portfolios could be useful assessment tools, there are challenges. For example, portfolios may have elements that overlap with accreditation standards, although not necessarily consistently, and often present only one method of standards assessment in the absence of other validated approaches, such as comprehensive assessment systems. Further, constructing a portfolio may be a cumbersome process for students to compile and for faculty to review (Boes, 2001; Bresciani, 2008; Dhonau & McAlpine, 2005; Hauser, Koutouzos, & Olson, 2005; Paradise et al., 2011). Given these challenges, it is imperative that counselor educators consider more comprehensive assessment systems that link clearly with CACREP Standards.

Method

Delphi methodology was selected for this study because it is a systematic approach to soliciting and integrating expert input and thus allows for compilation of perspectives toward the development of a comprehensive assessment system. Steps of a Delphi study involve a small monitor team identifying and soliciting expert participants, creating and distributing a survey or questionnaire to these participants, synthesizing results, and modifying the survey or questionnaire for additional rounds of responses. In each successive round, participants view and further comment on the previous responses of the entire sample until consensus is obtained (Clayton, 1997; Linstone & Turoff, 1975).

Participants

For the current study, the identified experts were CACREP program liaisons for community counseling, clinical mental health counseling, and school counseling–accredited master’s programs. The authors began with these program areas because they reflect the greatest number of CACREP-accredited programs. The authors sent an initial invitation to 171 CACREP liaisons, and 20 liaisons initially responded to the e-mail solicitation (response rate = 11.7%). Of these, 18 completed and returned the informed consent agreement, and in turn were sent electronically the first round survey materials. Three liaisons declined to participate, and six did not respond to the initial mailing or follow-up requests. In total, nine liaisons participated in the study. The original nine participants were invited to participate in each round, although some did not participate in a given round (see Table 1).

Table 1.

Participant Demographics^a

Characteristic	Round 1 (n = 9)	Round 2 (n = 7)	Round 3 (n = 7)	Round 4 (n = 9)
Sex	M = 5; F = 4	M = 4; F = 3	M = 4; F = 3	M = 5; F = 4
Race/ethnicity	White = 9	White = 7	White = 6; Other = 1	White = 9
No. of years as counselor educator, M (Range)	16.78 (7–32)	15.71 (7–31)	20.57 (7–31)	19.57 (10–31)
No. of years as CACREP liaison, M (Range)	5.33 (1–20)	3.57 (1–5)	5.71 (2–20)	6 (3–18)
Research level	I = 2	I = 1	I = 3	I = 2
	II = 5	II = 4	II = 2	II = 3
	III = 1	III = 0	III = 2	III = 2
	NA = 1	NA = 2	NA = 0	NA = 1
CACREP programs	CC = 9	CC = 7	CC = 6	CC = 6
	SC = 9	SC = 7	SC = 7	SC = 7
	SA = 1	SA = 1	SA = 1	SA = 0
	CE = 3	CE = 2	CE = 3	CE = 3
	CMHC = 1	CMHC = 0	CMHC = 1	CMHC = 1
Total no. of students, M (Range)	89.33 (46–140)	94 (50–150)	86.28 (42–124)	108.86 (60–155)

Note: CC = community counseling; CE = counselor education and supervision; CMHC = clinical mental health counseling; SA = student affairs; SC = school counseling.

^aBased on participants’ self-report responses to demographic questionnaire presented in each round of the study. Differences in responses are attributed to participants’ alterations in responding to questionnaire items in each round of the study.

Procedure

For each round, the authors utilized e-mail to distribute a template assessment system, a short list of open-ended questions related to the template, and a demographic questionnaire. The original template was developed by the first author based on CACREP core curricular areas. The template was a portfolio-type system that delineated areas where students would be assessed for standards mastery. The template was modified and distributed in each successive round based on participant feedback. Participants were given approximately 3 weeks to complete and return the materials in each round. Two weeks after each round of materials were sent, participants received a reminder about the approaching target date for completion. Another reminder was sent after the target date passed. Each participant received 10 dollars per completed round.

Round 1

Participants reviewed the assessment template and indicated at least one assignment used in their programs that would correspond with each core area. The authors asked them to explain if the assignment was expected of all students or limited to students in certain concentration areas. Participants were asked for initial impressions of the template, ideas about how to aggregate student data, and thoughts about the use of portfolios and other assessment methods of CACREP program area standards. Participants identified alternate assessment methods in this round of the study.

Round 2

Participants reviewed the sample template with participant-generated suggested assignments, rated their likelihood of implementation (to determine feasibility of assignments generated by their peers), and offered alternative assignments. The results of this process yielded a large set of proposed assignments that participants believed addressed general areas of assessment. Participants rated their perceived likelihood that their programs would implement proposed assignments, yielding a more manageable list of alternatives. The three top-rated assignments were identified for each category for further feedback and analysis in the next round of the study. The authors selected three assignments to address the highest rated content and to narrow down the scope of responses. These assignments will be further discussed in the results from the final round of the study.

Round 3

Participants identified CACREP program area standards addressed by each of the top-rated assignments identified in the previous round. They also suggested a specific assessment method and a measure of meeting the standard for each assignment. Participants then turned their attention from specific assignments to examine challenges of implementing and/or utilizing programmatic assessment systems suggested by the group in previous rounds. Participants were prompted to think about these comments and reconsider ratings in Round 2 for likelihood of implementation. For each of the assessment systems, participants again identified program area standards addressed and suggested a specific measure (assignment) of meeting the standard.

Round 4

The final round was confirmatory in nature. The final assessment system with suggested assignments was presented to participants for additional comments and recommendations. Participants were asked for commentary on the implementation of rubrics, including when and how within the program they may be applied. Following the completion of this round, the authors developed a final assessment system.

Results

The proposed assessment system contains 26 suggested assignments organized according to the eight CACREP core areas that assess all Community Counseling, Clinical Mental Health Counseling, and School Counseling program area standards (see Appendix A). The results from each round of the study are presented below to provide context for the final product.

Round 1

The first round produced a large data pool that served as a foundation for subsequent rounds. Concerns regarding portfolios were stated, including the potential for scripting programs and curricula. Initial comments included challenges of a parallel system to National Council for the Accreditation of Teacher Education (NCATE), as well as the workload that a portfolio system would present to program faculty. Participants shared some of their programs’ current assessment processes and offered recommendations for adaptation to generate outcome data compatible with the 2009 standards. The following participant comments highlight the sentiment regarding current assessment practices: “This [portfolio] approach ‘stretches’ some of the assignments and tries to make them cover more standards than logic dictates. Assessment works better with more targeted measures.” “A portfolio approach makes absolute sense, however to be reasonable and useful an Assessment Plan would be better served if representative artifacts were selected from the field of possible outcome measures in both program area standards (where outcome measures are required) and core standards (where outcome measures are not required).” and “We use portfolios in our school program presently. I believe they are useful, however, I am often concerned by the amount of time required to complete them professionally. . .often we find students losing out on the deeper learning because of their requirement to produce a binder to prove that they learned (ironic sabotaging cycle).”

Participants did suggest that the template presented a readable format and manageable data collection system. Other comments regarding data collection and aggregation included the following: “The template would be useful for constructing syllabi, but I am not sure it would be useful for our data collection needs. I think this template would be better for evaluating the course not the student.” “We would likely develop a more detailed rubric per requirement, score each student’s artifact/assignment by that rubric, and enter that data to be then aggregated statistically.” and “We write up more of a qualitative type response once we review the students’ portfolios. We talk about what they learned (as demonstrated by the portfolio), etc.”

In the initial round, participants also identified assessment systems they currently use or would recommend. Participants proposed assessment systems including pre–post assessments, field-based evaluations of counseling skills, use of technology, comprehensive examinations, portfolios, the National Counselor Examination (NCE), and practicum and internship skills assessments.

Round 2

Feedback indicated that the proposed system is consistent with CACREP accreditation requirements. Participants stated that the participant-generated assignments provided in the study are application based and provide evidence of achievement, creating a means of assessment. The feedback suggested that perhaps the present system could be used as a check for training experience rather than a portfolio. Participants noted that the developing system demonstrated a reflective process, stressing counselor (rather than curriculum or program) development. In response to a question about methods of assessment, participants overwhelmingly identified the importance of counseling skills ratings by field supervisors, followed by other means such as the NCE and an online data collection system such as one used in NCATE processes.

Round 3

The authors asked participants to identify the specific program area standards (Clinical Mental Health Counseling and School Counseling) assessed by the top-rated suggested assignments for each content area (see Appendix A). Participants’ recommendations involved the use of this system as a formative evaluation process. One participant commented on continued concerns of assessment systems:

…really worry about the one-size fits all approach to the level of standardizing we do. . .we worry that there is room/need for counselors with very different strengths and weaknesses, and we seem to be pushing every little thing into a one-size box. . .and we worry about the loss of some strengths and even some weaknesses that may be part of what makes the world go around.

Participants noted that “global assessment measures such as the NCE” would not work for the purposes of program area assessment. They reiterated the work-intensive nature of portfolios, and suggested a modified system might be more amenable and consistent with the spirit of assessment of student learning outcomes. Concerns were raised regarding interrater reliability of supervisor evaluations and a need for consistency in this and other evaluation methods.

The subjective nature of assessment in general, by faculty, adjunct instructors, and site supervisors, was also noted as an area of concern in the final assessment procedure:

In the second round of this study, some participants were critical of single assessor procedures. I do not see this as an issue, because students have multiple professors and supervisors during the course of their studies, therefore in the big picture they are not evaluated only by one person or one measure. It also was apparent that many of the assignments described in the study could be applicable to numerous CACREP specialty standards depending on the construction and the details of the assignment.

Round 4

A few participants in this last round indicated that they use rubrics and portfolios in their programs, and likewise addressed the challenges and limitations of each. As stated by one participant, students may be giving faculty “exactly what they think they want you to hear, no more and no less.” Participants advocated for a modified assessment system in which the proposed system could be used formatively within the program rather than at the end of student experiences:

It mostly is very comprehensive. I wonder if there are ways to employ rubrics to encompass more areas of evaluation. Wouldn’t it be nice to not have to track the number of proposed rubrics? I fear faculty will get annoyed with reminders and that persons “in charge” of maintaining these records would be very stressed out over time.

The authors also asked participants to identify other suggested assignments that might be used to enhance core or program area standards. As a result, an interview with a practicing counselor to address the career development core content area and an interview with a practicing school counselor regarding assessment usage for the Assessment core content area were added as additional examples.

Most participants recommended standardized rubrics to be used across the program for each assessment. The development of uniform rubrics and interrater reliability were again stated as concerns by participants. Participants rated the likelihood of their programs’ implementation of a system similar to that developed in the study. The mean rating, on a Likert-type scale of 1–5 (1 = not at all likely to 5 = very likely) was 3.92. Participants’ narratives indicated the need for further development of the proposed system, programmatic consideration of alternative methods of assessment, and development of systems that align with other college and university accreditation requirements (e.g., NCATE, regional accreditation).

Discussion

The purpose of this study was to develop a proposed assessment system for measuring student learning outcomes of the 2009 CACREP Clinical Mental Health Counseling and School Counseling program area standards. The new standards require that programs demonstrate a system of assessing student learning outcomes, not necessarily the data to demonstrate their students’ accomplishment of specific standards (see Urofsky & Bobby, this issue). The latter is an intent of outcome-based assessment and can be used by programs for continued curricular and student development.

The final proposed formative assessment system evolved from the development of an author-generated template to a more comprehensive and formative approach, listing possible assignments. The 26 suggested assignments address all of the program area standards for Clinical Mental Health Counseling and School Counseling programs. Data and assessments can be aggregated as assignments are implemented throughout the program. This system offers flexibility in selecting assignments that meet the standards and are consistent with program objectives and curricular experiences. There is an added measure of consistency with the use of program-identified assignments and assessments. The seven additional recommended artifacts may also be applied and used in some combination with the assignments in the main part of the system.

A distinguishing feature of Delphi methodology is the very nature of expert development of a final product (Linstone & Turoff, 1975). CACREP liaisons may be deemed such by experts, given their involvement in addressing accreditation standards and developing assessment systems in their programs. The proposed model offers multiple suggested assignments based on varying programs’ experiences and perspectives. There is flexibility in the application of the system. For example, programs may identify current assignments and determine comparable program area standards listed in the proposed assessment system. Likewise, for areas in which programs are lacking, proposed assignments from the template may be applied.

The authors are sensitive to concerns raised by participants relating to the difficulty of one model universally fitting all programs. The nature of the suggested assignments is both broad enough to encompass programs’ unique characteristics and specific enough to capture program area standards. As reinforced in this study, global measures such as the NCE (or the student version, Counselor Preparation Comprehensive Exam) are valuable in assessing core knowledge. The types of assignments suggested in the template will be needed to assess knowledge and skills of program area standards more specifically. The final proposed assessment system of this study is a reflection of the experiences and structures of nine programs. Programs have unique characteristics and will demonstrate individuality when selecting and utilizing assignments to assess student learning of standards.

The actual assessment and aggregation of data were not the primary focus of the study. The focus was to develop a proposed assessment system for practical use. Nevertheless, participants provided thoughts about data collection and assessment processes in the final round of the study. Rubrics, with the limitations noted, could be designed on a three-point system to measure students’ achievement of the proposed standard areas. For example, an item on a rubric for a personal statement of counseling could assess the student’s understanding of the history, philosophy, and trends in clinical mental health counseling (Clinical Mental Health Counseling program area standard A.1). Ratings such as the following may be employable: Rating 1 could indicate the student exceeded this knowledge area in the written statement, Rating 2 that the student met the knowledge standard, and Rating 3 that the student did not demonstrate understanding of this knowledge area in the assignment.

In the end, participants developed a meaningful and comprehensive system of assessment in which all program area standards can be assessed through a menu of suggested assignment options (see Appendix A). Programs can select specific assignments that meet their curricular and student learning objectives and implement them as part of their assessment systems. The intent is for programs to identify assignments in each area and determine how they will be implemented within the program. For example, programs may use site supervisor evaluations to address several program area standards. It is hoped then that the program will develop a uniform rubric or other assessment that will be used by all site supervisors at a determined point or points within the program. The importance of consistency is stressed. Although there are limitations to interrater reliability, a consistent assessment measure should be used for all students to gauge their development in these program area standards. Further development of this first proposed assessment system will require exploration of rubric development and implementation.

Limitations

As with all research, there were limitations to the present study. Delphi methodology by nature is lengthy and requires tremendous effort on the part of participants. This study spanned over a year of time. Some rounds of data collection, in particular Round 3, required a significant amount of time and work on the part of participants. Two participants did not take part in this round, perhaps due to the nature of the expectations or other circumstances unrelated to the study. Response fatigue may be a reality and limitation to the results of the study. In a study requiring significant thought and contributions, participants can manifest fatigue by providing minimal responses that do not convey the complexity or true nature of their thoughts. There is depth to the data given the comments provided, yet not all perspectives were necessarily fully conveyed. The Delphi process of participant commentary on all participant responses attempts to control for full expression of perspective; yet, it is possible that not all were fully accounted for in the final product development. Additionally, there is limited diversity in the study participants, who the authors identified as experts. Although programs were diverse in nature, the predominantly European American perspectives and potential for biased results must be noted.

Researcher bias must also be considered. The principal investigator began this study with the intent of developing a portfolio system. The authors attempted to control researcher bias through the use of language outside of portfolio and intended solicitation of other assessment systems. The limited research on assessment systems beyond portfolios limited the availability of other options familiar to participants. Further development of outcome-based assessment of student learning outcomes will need to adapt the systems recommended by participants in the present study. Finally, the proposed system has not been piloted by programs to determine the accuracy of program area standards measurement and the assessment system process.

Future Research

Outcome-based assessment is new to counselor education and still a relatively new system across disciplines. Further research regarding student learning outcomes is warranted, and the special issue of this journal is a step in this direction. Systems used in other disciplines may be more closely studied and replicated in CACREP-accredited programs. Whether the proposed model from the present study or another system developed elsewhere, pilot studies of assessment systems are recommended. As programs have submitted assessment plans with CACREP accreditation reviews, a systematic study of these plans may be undertaken to explore elements of development and execution of systems.

A review of the literature indicates that a particularly salient focal point for future studies is the potential burden, monetary and otherwise, a program or institution will incur when implementing a new assessment system (Bresciani, 2008; Kelly, 2011; Paradise et al., 2011). Concerns about the resources needed for success in assessing student outcomes and implementing the 2009 CACREP Standards indicate research in these areas could benefit educators and program coordinators. Qualitative examination of experiences with student learning outcomes and assessment processes may yield recommendations for implementation and execution. Student understanding of the methods used for outcome-based assessment in individual programs should also be examined. Recognizing the potential impact of student understanding of outcome-based assessment on student input may be of importance. For example, a semilongitudinal study could explore how a model for assessment such as the one proposed here influences or alter students’ approaches to completing a course of study.

Future research may explore the system of data collection through electronic, paper, or combined means. Concerns about interrater reliability, consistency among faculty and supervisors, and application of systems across varying sized programs should be addressed as well. A more comprehensive examination of accreditation programs and their systems may include feasibility studies, ratings of assessment systems across program sizes and settings. Programs submitting materials for accreditation review under the 2009 CACREP Standards are working with these issues and may serve as platforms for further development of systems. Data aggregation systems, although not a focus of the present study, might be researched as well for implementation and continued development of accreditation standards. Related, a review of students’ achievement of program area standards may prove useful to future revisions to the CACREP Standards.

As stated previously, rubrics present challenges to counselor educators and site supervisors. A comprehensive study of the development of rubrics is warranted to provide programs with guidance on the process and implementation. Research from related fields may likewise provide insight into the development and piloting of rubrics to assess specific program area standards as well as student learning outcomes more comprehensively.

Conclusion

Outcome-based assessment in counselor education will present new challenges and opportunities to programs. As the profession and indeed academia embrace accountability the reality and responsibility is upon us to demonstrate that graduates are prepared to work as professional counselors. The present study produced one proposed model for assessing student learning outcomes of the 2009 CACREP Clinical Mental Health and School Counseling program area standards. The proposed comprehensive assessment system is based on a systemic study of perceptions of representatives of accredited programs, perceived experts in the field (Linstone & Turoff, 1975). The proposed system fulfills the CACREP 2009 standards requirement for a program-wide system of assessing student learning outcomes. Program faculty are encouraged to engage in dialogue regarding their own curricula and demonstration of student learning to further enhance experiences and ultimately the services provided to counseling consumers.

Footnotes

Appendix A

Additional Assessment System Items to Address Program Area Standards

Assignment/Artifact	CMHC Program Standards	SC Program Standards
Checklist completed by site supervisor (Checklist items aligned to program area standards)	A7, A8, A10, B1, B2, C2, C3, C4, C7, C8, C9 D1, D2, D3, D4, D5, D6, D7, D8, D9, E4, E5, F1, F2, F3, G1, G2, G3, H1, H2, H3, H4, I2, I3, J2, J3, K1, K2, K3, K4, K5, L1, L2, L3	A2, A3, A5, A7, B1, B2, C1, C2, C3, C4, C5, D1, D2, D3, D4, D5, F1, F2, F3, F4, G1, G2, G3, H1, H2, H3, H4, H5, K2, K3, I3, J2, J3, L1, L2, L3, M4, M5, M6, N1, N2, N3, N4, O3, O5, P1, P2
Video taped case presentations	A9, A10, B1, C1, C2, C4, C5, C6, C7, C8, D1, D2, D3, D4, D5, D6, D7, D8, D9, E1, E2, E3, E4, E5, E6, F3, G1, G2, G3, G4, H1, H2, H3, H4, K1, K2, K3, K4, K5, L1, L2, L3	B1, C1, C3, C5, D1, D2, D3, D4, D5, F1, F4, G1, G2, G3, H1, H2, H3, H5, K2, K3, L1, L2, L3, M1, M2, M3, M4, M5, M6, M7, N1, N2, N3, N4, N5
Treatment notes	A2, A8, C2, C4, C7, C8, D1, D2, D4, D5, D6, D7, D8 G1, G2, G3, G4, H1, H2, H3, H4, K1, K2, K3, K4, K5, L1, L2	A2, B.1, C3, D3, F.3,4; G.1-3; H.1-5; K.1-3, M.1-5, N.1-3
Case report	A2, A6, A8, A9, B1, C2, C4, C6, C7, C8, D1, D2, D4, D5, D6, D7, D8, E1, E4, F3, G1, G2, G3, G4 H1, H2, H3, H4, K1, K2, K3, K4, K5, L1, L2, L3	A2, A6, B1, C1, C2, C3, C4, C6, D3, E2, E4, G1, G2, G3, H1, H2, H3, H4, H5, L3, M1, M2, M3, M4, M5, N1, N2, N3
Site contract (in field handbooks)	A2, A3, A8, C5, C6, C7, C9	A2, A7, C6, M1, M2, M3, M4, M5, M6, M7, O5
Internship field agreement	A2, A3, A8, C5, C6, C7, C9	A2, A7, C6, M1, M2, M3, M4, M5, M6, M7, O5
Case study in internship	A2, A6, A8, A9, B1, C2, C4, C6, C7, C8, D1, D2, D3, D4, D5, D6, D7, D8, F3, G1, G2, G3, G4, H1, H2, H3, H4, K1, K2, K3, K4, K5, L1, L2, L3	A6, B1, C1, C2, C3, C4, C6, D1, D2, D3, D4, D5, E2, E4, F3, F4, G1, G2, G3, H1, H2, H3, H4, H5, K1, K2, K3, L3, M1, M2, M3, M4, M5, N1, N2, N3

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by a Council for the Accreditation for Counseling and Related Educational Programs (CACREP) research grant.

References

Baltimore

M. L.

Hickson

George

J. D.

Crutchfield

L. B.

(1996). Portfolio assessment: A model for counselor education. Counselor Education and Supervision, 36, 113–121.

Boes

S. R.

(2001). Portfolio development for 21st century school counselors. Professional School Counseling, 4, 229–231.

Bresciani

M. J.

(2008). Implementing outcomes-based assessment process in a major metropolitan community college district. Community College Journal of Research and Practice, 32, 970–984.

Brindley

(2001). Outcomes-based assessment in practice: Some examples and emerging insights. Language Testing, 18, 393–407. doi:10.1177/026553220101800405

Clayton

M. J.

(1997). Delphi: A technique to harness expert opinion for critical decision making tasks in education. Educational Psychology, 17, 373–386. doi:10.1080/0144341970170401

Cobia

D. C.

Carney

J. S.

Buckhalt

J. A.

Middleton

R. A.

Shannon

D. M.

Trippany

Kunkel

, (2005). The doctoral portfolio: Centerpiece of a comprehensive system of evaluation. Counselor Education and Supervision, 44, 242–254.

Coleman

H. L. K.

Morris

Norton

R. A.

(2006). Developing multicultural counseling competence through the use of portfolios. Journal of Multicultural Counseling and Development, 34, 27–37.

Council for Accreditation of Counseling and Related Educational Programs. (2009). 2009 Standards. Alexandria, VA: Author.

Dhonau

McAlpine

(2005). An electronic portfolio for the ACTFL/NCATE teacher program standards in the second language methods course. Foreign Language Annals, 38, 69–76. doi:10.1111/j.1944-9720.2005.tb02454.x

10.

Donnelly

(2007). Australia's adoption of outcomes based education: A critique. Online Submission (ERIC Document Reproduction Service No. ED501707).

11.

Frank

Danoff

(2007). The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Medical Teacher, 29, 642–647. doi:10.1080/01421590701746983

12.

Hauser

Koutouzos

Olson

(2005). The standards-based school leader digital portfolio. International Journal of Learning, 12, 311–316.

13.

Heath

(2005). Are you ready to go digital? The pros and cons of electronic portfolio development. Technology Connection, 23, 66–70.

14.

Jansen

(1998). Curriculum reform in South Africa: A critical analysis of outcomes-based education. Cambridge Journal of Education, 28, 321–331. doi:10.1080/0305764980280305

15.

Kelly

(2011). Faculty-driven change: How to get buy-in, make progress. Academic Leader, 27, 4–8.

16.

Linstone

H. A.

Turoff

(1975). The Delphi method: Techniques and applications. London, England: Addison-Wesley.

17.

Paradise

L. V.

Lolan

Dickens

Tanaka

Tran

Doherty

(2011). Program coordinators react to CACREP standards. Counseling Today, 53, 50–52.

18.

Pecheone

Pigg

Chung

Souviney

(2005). Performance assessment and electronic portfolios. Clearing House, 78, 164–176. doi:10.3200/TCHS.78.4.164-176

19.

Rhyne-Winkler

M. C.

Wooten

(1996). The school counselor portfolio: Professional development and accountability. School Counselor, 44, 146–150.

20.

Rogers

Chow

(2000). Electronic portfolios and the assessment of student learning. Assessment Update, 12, 4–5.

21.

Strudler

Wetzel

(2008). Costs and benefits of electronic portfolios in teacher education: Faculty perspectives. Journal of Computing in Teacher Education, 24, 135–142.

22.

Urofsky

R. I.

Bobby

C. L.

(2012). The evolution of a student learning outcomes focus in the CACREP standards in relation to accountability in higher education. Counseling Outcome Research and Evaluation, 3.