Program Evaluation in Doctoral-Level Counselor Education Preparation

Sampling

To locate possible institutions with CE programs, an archival research procedure supplemented with current website search methodology was utilized (LaVelle & Donaldson, 2010). Specifically, the research team composed of the lead author, a research CE professor, and a doctoral CE student developed the sampling frame by creating an Excel™ spreadsheet of CACREP-accredited university-based CE training programs located in the United States. Data were aggregated by geographical region and other demographic descriptors. The nation-wide sampling frame and unit of analysis included 100% of the doctoral-level (PhD and EdD) CE programs (N = 81). In other words, 81 institutions/universities with a CE program at the doctoral level were included in this study. Of these doctoral-level programs, 76 (93.8%) were CACREP accredited and 5 (6.2%) were in the process of accreditation. Because these institutions are obligated to comply with CACREP (2009, 2016) standards regarding program evaluation preparation for doctoral students, these units were targeted for the study. Non-CACREP accredited doctoral-level CE programs were not included in the sample.

Applying the Association for Counselor Education and Supervision regional categorization of CE programs as a guide, of the 81 doctoral programs purposefully sampled, 44 (54.3%) were situated in the Southern region of the United States. The other programs represented North-Central (n = 21, 25.9%), Rocky Mountain (n = 8, 9.9%), North Atlantic (n = 6, 7.4%), and Western (n = 2, 2.5%) regions. Sixty-three (77.8%) of the institutions awarded PhDs, 12 (14.8%) granted EdD, and 6 (7.4%) offered both a PhD and an EdD track. Of the institutions reviewed, 58 (71.6%) were public and 23 (28.4%) private. The majority of CE programs were campus based (n = 75, 92.6%) rather than delivered online (n = 6, 7.4%). Some doctoral programs were perhaps offered in a hybrid format, delivering coursework both on-campus and online, but this option was not discernable from the institutional websites.

Procedures

Each institution with a doctoral-level CE program was identified through the CACREP website in May 2017. Database inclusion criteria were kept to a minimum. Each institution’s doctoral CE program (a) needed to be CACREP accredited or in process and (b) webpages provided sufficient information including program description and foci, degree awarded, coursework, course descriptions, and graduation requirements. Based on an abbreviated version of LaVelle and Donaldson’s (2010) Internet-based webpage/document analysis technique, researchers first employed the Chrome™ webpage browser to locate each doctoral-level CE program within participating institutions. Next, using each institution’s internal search capabilities, descriptors such as ‘‘program evaluation” and “research and evaluation” were searched. Internal links to CE program literature were viewed and analyzed. Institutional demographic data were collected as well as any information that referred to program evaluation in the curriculum and pedagogy was noted on a spreadsheet. This matrix with predefined categories served as a rubric to catalogue data (e.g., institution, address, PhD/EdD, website, program evaluation coursework, delivery style, and program evaluation content). Researchers reviewed any online materials (e.g., program descriptions, course catalogues and descriptions, program handbooks, and syllabi) provided by each institution’s website. Curriculum was broadly defined as any program evaluation and research coursework, assignments, activities, and so on that were indicated on the program webpages. Pedagogical information (methods of teaching, course delivery style, etc.) relating to evaluation and research were entered on the spreadsheet as well. Researchers were trained, and to maintain consistency and rigor, analyses and findings were regularly reviewed and consensus arrived at.

Recommendations

Based on this status report, it appears that nationally accredited CE doctoral programs that aim to equip their students to be competent evaluators are advised to modify their curriculum and, perhaps, instructional methods. Plentiful recommendations have been offered over the years to rectify the situation in CE (e.g., Astramovich & Coker, 2007; Hosie, 1994; Schaffer & Atkinson, 1983; Wheeler & Loesch, 1981). Instead of reiterating them here, the suggestions for educational modifications provided below stem largely from current information and literature. However, before specific refinements are proposed and implemented, program evaluation and counselor educators should establish a close partnership to ascertain their shared needs, goals, and processes.

Recommendation 1: Conduct a focused audit of existing research and program evaluation skill development courses and curriculum.

The need to improve the research and evaluation component of CE programs has been repeatedly affirmed in CE publications with little or no effect (e.g., Borders, Wester, Fickling, & Adamson, 2014; Heppner et al., 2016; Peterson et al., 2016; Sink, 2009; Wheeler & Loesch, 1981). The findings from this study provide the first data-based evidence that little improvement has occurred. The majority doctoral-level CE programs surveyed have minimal expectations for their students, certainly not enough for graduates to conduct rigorous and effective evaluations. The situation is perhaps due, in part, to the vague and limited number of CACREP (2009, 2016) program evaluation standards. Without more specificity, CE programs maintain their accreditation even with insufficient coursework in program evaluation.

To begin the process of restructuring, CE leadership and faculty should first conduct a focused program audit to identify gaps in evaluation-related coursework offered within the CE unit and by other departments. Keeping in mind CACREP (2016) standards, a crosswalk of goals, curricular offerings, and pedagogical strategies should be developed. This gap analysis should be conducted periodically, occurring at least every 3–5 years, and not necessarily in conjunction with CACREP reaccreditation activities. There are numerous resources to guide the process (e.g., Smith & Goodwin, 2014).

Including program evaluation scholars from outside the unit will enhance the viability of the audit outcomes. Together, these educators should revisit all pertinent curriculum, course materials, and pedagogy in terms of program evaluation skill development. Sample questions for an initial cross-discipline discussion may include How do program evaluation requirements and instruction meet the present needs of the CE profession and how are they aligned with best practice? In what ways is coursework aligned with CACREP (2016) standards for program evaluation? How are course objectives differentiating skill development in program evaluation at the master’s and doctoral levels? How are course objectives assessed and are they competency based? How do faculty document student attainment of specified skills in program evaluation? Are these methods sufficient? What course(s) should be offered by counselor versus program evaluation educators? In what ways can CE faculty partner with program evaluation educators on course development and curricular refinements, particularly to best meet current CACREP standards regarding research and evaluation? How does the coursework foster evaluative thinking (ET)?

Recommendation 2: Grounded on established program evaluation standards/competencies for practice, revise the evaluation component of CE..

Utilizing the audit results and ensuing discussions, the next phase is to implement the restructuring process, again in partnership with university program evaluation educators. A way to make certain the comprehensiveness and relevancy of program modifications is to frame them around specific research-based program evaluation competencies (Stevahn, King, Ghere, & Minnema, 2005). Specifically, faculty from CE and program evaluation units should review and propose refinements to coursework and instruction around the widely used and extensive program evaluation standards developed the Joint Committee on Standards for Educational Evaluation (Stufflebeam & Madaus, 1983; Yarbrough, Shulha, Hopson, & Caruthers, 2011). The 30 Joint Committee standards are arranged into five dimensions: utility, feasibility, propriety, accuracy, and evaluation accountability. Another suitable option is to use AEA’s (2004) Guiding Principles for Evaluators. The principles set out 25 expectations for evaluators covering these general domains of evaluation work: systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare. In their entirety, the standards and principles convey the need to consider the unique content and skills (e.g., evaluation models, managing processes and procedures, stakeholder involvement, program context, and political accountability) involved in program evaluation.

To further assist and simplify the cross-academic unit collaboration on curriculum and instructional planning and implementation, counselor and program evaluation educators may want to consult the recently developed PECM presented in Table 1. The authors designed the matrix to efficiently combine, adapt, and abbreviate AEA’s (2004) Guiding Principles and the Joint Committee’s standards (Yarbrough et al., 2011) into one user-friendly matrix. In developing the PECM, the content of the standards was matched to those guiding principles with comparable subject matter, and subsequently, reworded for clarity and relative succinctness. For example, the utility standard U2 attention to stakeholders (Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation.) was matched with the guiding principle B.2. (To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence.) The resulting PECM competency reads: Be socially, politically, and culturally sensitive when selecting and deploying evaluation methods, processes, and procedures as well as when sharing findings and recommendations. Moreover, to establish a level of content and face validity for the PECM’s competencies, input from two experts in CE program evaluation was solicited. These scholars represented two different Research I public universities with CACREP accredited graduate programs in counseling.

OPEN IN VIEWER

Table 1.

Core Program Evaluation Competencies Related to CACREP Standards and School-Based PE Leadership Skill Areas.^a

Competency	Applicability Level			School-Based PE Leadership Skill Areas
Program Evaluators-in-Training Will Develop the Skills Needed to	CACREP (2016, p. 14) Standards Section 2.8	Master	PhD/EdD	Counseling Model Development	External Evaluators of Counseling Programs	Conduct Policy-Related Evaluations	Curricular Decision-Making	Graduate-Level PE Instruction
1.0 Professional practice and effective communication
1.1 Conduct evaluation in accordance with ethical standards and legal requirements	j. Ethical and culturally relevant strategies for conducting, interpreting, and reporting the results of research and/or program evaluation	√	√		√	√	√	√
1.1 Be socially, politically, and culturally sensitive when selecting and deploying evaluation methods, processes, and procedures as well as when sharing findings and recommendations	j. Ethical and culturally relevant strategies for conducting, interpreting, and reporting the results of research and/or program evaluation	√	√		√	√	√	√
1.2 Communicate effectively with stakeholders at all stages of the evaluation process		√	√		√	√	√	√
1.3 Collaborate and consult with stakeholders in an open and honest manner through evaluation stages		√	√		√	√	√	√
1.4 Clearly disclose considerations, procedures, data collection and analysis methods, results, and recommendations/implications that are consistent with the aims and objectives of the evaluation		√	√		√	√	√	√
1.6 Sensitively and prudently share with stakeholders recommendations with likely positive and negative repercussions/implications resulting from the evaluation		√	√	√	√	√	√	√
1.7 Write and submit quality evaluation proposals and formative and summative reports			√	√	√	√	√	√
1.8 Engage in self-reflective practice, assessing one’s evaluation knowledge, skills, and dispositions and pursue professional development activities to improve performance		√	√	√	√	√	√	√
2.0 Systematic analysis and data-based practice
2.1 Understand and appropriately use the knowledge base (evaluation models, terms, concepts, theories, and assumptions), processes and procedures	g. Designs used in research and program evaluation	√	√	√	√	√	√	√
2.2 Understand and apply evidence-based practice	b. Identification of evidence-based counseling practices	√	√
2.3 Locate and apply for evaluation opportunities			√		√	√		√
2.4 Understand and apply, as appropriate, qualitative, quantitative, and mixed research methods and associated strategies for valid and reliable data collection, analyses, and reporting	f. Qualitative, quantitative, and mixed research methods i. Analysis and use of data in counseling	√	√	√	√	√		√
2.5 Assess school-based settings in terms of potential strengths and limitations or barriers related to the evaluation and implement of recommendations for practice	c. Needs assessments	√	√	√	√	√	√	√
2.6 Design quality evaluation and related materials, incorporating relevant research literature and theory	a. The importance of research in advancing the counseling profession including how to critique research to inform counseling practiced. Development of outcome measures for counseling programs		√	√	√	√	√	√
2.7 Deploy relevant evaluation model, questions, and data collection and data analysis	h. Statistical methods used in conducting research and program evaluation	√	√	√	√	√		√
3.0 Program Management
3.1 Use evaluation resources (e.g., fund management and budgeting, materials, and personnel utilization) efficiently and wisely		√	√		√	√		√
3.2 Document work coherently and fully at each stage of the evaluation		√	√		√	√		√
3.3 Conduct evaluation in timely manner documenting steps and actions		√	√		√	√		√
3.4 Use relevant technology as required		√	√		√	√		√

Note. PE = program evaluation.

^aFor brevity, the table is referred to as Program Evaluator Competency Matrix; competencies were adapted and abridged from AEA’s (2004) Guiding Principles for Evaluators, Joint Committee on Standards for Educational Evaluation (as reproduced in Yarbrough et al., 2011), and CACREP (2016) Standards (section 2.8. research and program evaluation); Heppner et al. (2016), Stevahn et al. (2005).

After incorporating outside feedback, PECM’s competencies were mapped to (a) CACREP (2016, p. 14) Standards for Research and Evaluation (see Section 2: Professional Counseling Identity); (b) their applicability level (i.e., Are they primarily applicable to master’s or doctoral level students or both?); and (c) school-based program evaluation leadership skill areas (Carey & Trevisan, in press). The resulting version of the PECM offers 19 core competencies organized into three broad domains: professional practice and effective communication, systematic analysis and data-based practice, and program management. Indicators were added to show users whether the competency and CACREP standard should be taught primarily at the master’s or doctoral level. For counselor educators preparing doctoral-level school counselors, the competencies are also linked with Carey and Trevisan’s program evaluation leadership skill areas for counselor educators. These include (a) counseling model development, (b) external evaluators of counseling programs, (c) policy-related evaluations, (d) curricular decision-making, and (e) graduate-level program evaluation instruction. These categories assist CE and program evaluation educators to determine where individual competencies fit into core topical areas in CE preparation.

The PECM has multiple practical applications for graduate-level counselor and program evaluation educators. For instance, the tool should prove useful to faculty members as they make decisions related to program evaluation curriculum development and instructional revisions. PECM competencies can also be converted into course objectives and serve as the basis for a program evaluation skill attainment rating scale for graduating CE doctoral students. The matrix can also serve as a guiding document to design a formative or summative questionnaire, allowing CE faculty to survey program graduates about material they have retained and are using in the field. These data should be shared with program evaluation instructors to further refine their work with CE students. Specific benchmarks could be developed for each competency to further delineate skill development. Subsequently, counselor and program evaluation educators would rate the students at one of four levels of proficiency: 1. learning (student understands knowledge base), 2. emerging (student demonstrates basic skills); 3. assisting and supporting (student can support and collaborate with evaluation project leadership); and 4. leadership (student demonstrates the professional skill set to oversee an evaluation project). To summarize, the PECM can be employed by program evaluation and counselor educators as a tool for aligning evaluation curriculum and instruction with professional standards (see Table 1).

Recommendation 3: Through collaboration with research and program evaluation faculty and field-based evaluators, revisit curriculum and pedagogy to affirm that students’ possess appropriate research and data analysis skills and become “evaluative thinkers.”

One potential implication of the findings from the current study is that counselor educators may underestimate the skill set needed to be a successful evaluator. Given the lack of CE program requirements in this area, they may view program evaluation as a collection of skills that can be gained with only a modicum of instruction and practical experience. However, as Buckley, Archibald, Hargraves, and Trochim (2015) pointed out, effective program evaluators are not only skilled in designing and implementing program evaluation activities, but they are encouraged to also deploy ET. These scholars defined ET as

critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action. (p. 378)

To foster evaluative/critical thinking in CE students, there are a number of instructional methods counselor and program evaluation educators might find effective. The relevant literature indicates that socratic questioning, journal writing, collaborative, contextual or field-based training, experiential (“hands on”) learning (e.g., semester-long program evaluation internships), mentoring, and problem and project-based learning are beneficial (e.g., Abrami et al., 2015; Granello, 2000; Griffith & Frieden, 2000). Moreover, learning activities should be intentionally planned to access doctoral-level CE students’ previously established skill set, particularly as they tap into ET. For example, these graduate students should be relatively proficient with consultation, systems thinking, communication, open questioning, self-reflection, intuition, diagnosis, and action/intervention planning. In other words, program and counselor educators can nurture ET in CE doctoral students by focusing their instruction and classroom activities on the transfer of learning from “counselor thinking” to “ET.” It should be noted, however, for students to reach this level of evaluative functioning, they may need two specific program evaluation courses that address foundational and advanced-level skill development, respectively, as well as a variety of pertinent field-based experiences (LaVelle & Donaldson, 2010).

Research Caveats and Suggestions for Future Research

As LeVelle and Donaldson (2010) elucidated, a review of university webpages is a less than an ideal strategy to grasp the full extent of program information and offerings. In some cases, the webpages analyzed by the researchers appeared to be dated, incomplete, and difficult to navigate. Locating applicable information was a challenge and many sites lacked specifics on course requirements and content relating to program evaluation. The investigators also relied on key search terms such as “program evaluation” and “research and evaluation.” It is more than likely these explorations failed to capture the entire scope of evaluation preparation offered within CE programs. To maintain uniformity and rigor in the review process, the researchers did not survey the small number of non-CACREP-accredited doctoral CE programs. Thus, estimates regarding the level of program evaluation training in CE programs across the country are imprecise.

Future research in this area should include interviewing a sample of doctoral program CE coordinators and faculty to determine the accuracy and completeness of online program materials. These interviews could also explore whether (a) program evaluation is adequately addressed in the curriculum and (b) the coursework and associated activities are meeting current CACREP standards in this area. Researchers could combine CE program materials with interview data to map CACREP evaluation standards with doctoral program evaluation goals and outcomes from actual coursework including assignments and assessments.