Program Evaluation in Doctoral-Level Counselor Education Preparation

Abstract

Within the context of intentional collaboration between doctoral-level program evaluation and counselor educators, this article addresses the need for advanced training in program evaluation for preservice doctoral-level counselor education (CE) students. To determine the current level of evaluation training provided to doctoral students attending a nationally accredited CE program, 81 institutions and their program websites and accompanying materials were analyzed. Approximately 25% of these graduate programs appeared to provide a modest level of training. Over 50% of these units did not fully identify what program evaluation preparation was required or offered to students. Recommendations to enhance the program evaluation component of counselor educator training are offered. To assist program evaluation and counselor educators to restructure training, an instructional planning and implementation tool (Program Evaluation Competency Matrix) is provided.

Keywords

program evaluation doctoral-level counselor education programs preparation competencies

For decades, program evaluation has been an essential component of American education. Evaluators are regularly employed to document the value and limitations of schooling processes and procedures as well as educational programs and innovations. Results from program evaluations also guide policy making and planning. Given the widespread application of evaluation within educational praxis, it is not surprising that mental health providers and school-based counselors find themselves under additional pressure from the public and health-care insurance companies to demonstrate the efficacy of their program services and client interventions (Astramovich & Coker, 2007; Astramovich, Coker, & Hoskins, 2005; Heppner, Wampold, Owen, Wang, & Thompson, 2016; Reupert, McHugh, Mayberry, & Mitchell, 2012; Schaffer & Atkinson, 1983; Tyler, 1983; Wheeler & Loesch, 1981). In fact, counselors serving under the auspices of their respective professional counseling organizations are required to be adept at conducting program evaluations and using accountability and evidence-based practices (e.g., American Counseling Association [ACA]; Baker, 2012; American Mental Health Counseling Association [AMHCA], 2016; American School Counselor Association [ASCA], 2012a, 2012b).

The challenge to meet the need for expertly prepared program evaluators and researchers has grown steadily (Astramovich & Coker, 2007; LaVelle & Donaldson, 2010). In fact, academics and leaders from a variety of human services (e.g., counseling, education, clinical psychology, and social work) and organizations (e.g., American Educational Research Association, American Evaluation Association [AEA], ACA; AMHCA, ASCA) posit that advanced training in evaluation-related skills should be a higher priority (e.g., Astramovich & Coker, 2007; Hosie, 1994; LaVelle & Donaldson, 2010; Reupert et al., 2012; Royse, Thyer, & Padgett, 2016; Sink, 2009; Trevisan, 2000, 2002; Vacc & Rhyne-Winkler, 1993; Wheeler & Loesch, 1981).

Many universities offer graduate degrees in program evaluation, while others merely provide related coursework in this area. Program evaluation educators are largely positioned within separate units within Colleges of Education, and to a lesser extent, Psychology Departments (LaVelle & Donaldson, 2010). As LaVelle and Donaldson showed in their national-wide study, 87 universities met the cross disciplinary and programmatic demands within a single evaluation course. This single-course strategy seems to be insufficient to address the relatively unique program evaluation needs of doctoral-level counselor education (CE) students. For instance, future counselor educators in university settings have to be prepared to successfully implement a variety of program evaluation tasks (e.g., monitoring and evaluating the competency and learning of student counselors, departmental assessment of program quality, accreditation program self-studies, clinic-based results evaluation, program audits, faculty and staff evaluation, K–12 school district program evaluation, program evaluation research, and grant writing) that cannot be covered fully in a generic course designed for graduate student across multiple disciplines (Brott, 2006; Bryant, Druyos, & Strabavy, 2013; Haberstroh, Duffy, Marble, & Ivers, 2014; Hosie, 1994; Schaffer & Atkinson, 1983).

The appeal for enhanced program evaluation training is also consistent with the ethical standards for professional counseling practice. For example, ACA (2014), serving largely community/mental health counselors, stated in their ethical codes that all practitioners “take steps to ensure the competence of their work and protect others from harm” (section C.2.b., p. 8) and “continually monitor their effectiveness as professionals and take steps to improve when necessary” (section C.2.d., p. 8). School-based counselors, in the same way, are obligated by their professional ethical standards to (a) “provide effective, responsive interventions to address student needs” (section A.1.h., ASCA, 2016, p. 1) and (b) “conduct school counseling program evaluations to determine the effectiveness of activities supporting students’ academic, career and social/emotional development through accountability measures, especially examining efforts to close information, opportunity and attainment gaps” (section A.13.i; ASCA, p. 5).

School-based counselors are also called upon to effectively collaborate within systemic intervention programs such as multitiered systems of support (e.g., positive behavior and interventions supports; Cook, Lyon, Kubergovic, Wright, & Zhang, 2015) and comprehensive school counseling programs (ASCA, 2012a, 2012b; Gysbers & Henderson, 2012). These school-wide support systems are heavily dependent on data-driven or evidence-based decision-making and interventions designed to assist all students, not just those who are at risk of academic failure (Cook et al., 2015; Ziomek-Daigle, Goodman-Scott, Cavin, & Donohue, 2016). It is therefore incumbent upon school and mental health counselors to gain the requisite research and evaluation skill set to implement proven interventions and to appraise program, client, and student outcomes (Gysbers & Henderson, 2012; Kratochwill & Shernoff, 2004; Peterson, Hall, & Buser, 2016; Sink, 2016).

Beyond the need for well-trained master’s-level counseling practitioners who can assess the impact of their interventions, doctoral CE students pursuing faculty or research positions ought to be well versed in program evaluation. These skills will be called upon in a number of capacities. For instance, when applying for grants from well-established funding agencies (e.g., U.S. Department of Education; National Institute of Mental Health; and Gates, Kellogg, and Templeton foundations), a strong program evaluation component is generally required for proposals. Moreover, future graduate-level counselor educators may want to act as evaluation consultants for in-patient clinical facilities, outpatient mental health agencies/organizations, and educational institutions. Such consulting services necessitate expertise in program evaluation. Evaluation skills are also needed for common faculty tasks such as curriculum development, accreditation self-studies, and yearly program review.

Another impetus for doctoral-level counselor educators to provide students with quality program evaluation coursework involves meeting national accreditation standards. The Council for the Accreditation of Counseling and Related Educational Programs (CACREP, 2016), the primary national accrediting agency for university-based graduate CE programs, expects these units to carefully arrange their curriculum’s scope and sequence according to its standards (CACREP, 2009, 2016). For graduate CE departments offering a doctoral degree, program administrators and faculty are required to provide clear evidence that preservice students have attained proficiencies associated with research and program evaluation (see CACREP Standards, section 6.b.4. research and scholarship). CE doctoral students, upon degree completion of their program of study, are expected to be competent in quantitative and qualitative research designs, data analysis procedures and processes, measurement and instrument design, and models and methods of program evaluation. It is important to note that CACREP (2016) offers substantial latitude to program administrators on how the specific program evaluation standard (section 6.b.4.f.) should be delivered to and demonstrated by students.

Considering that CE evaluation preparation literature is sparse, particularly at the doctoral level (e.g., Astramovich et al., 2005; Hosie, 1994), it remains unclear how coursework addresses CACREP (2009, 2016) research- and evaluation-related standards. What anecdotal evidence there is, suggests that course content directly referring to the development of program evaluation skills is inadequate. While the literature is dated, much of the instruction appears to be focused on entry- or master’s degree-level counseling professionals, leaving more advanced training to be acquired in the field or through professional development activities (Trevisan, 2000, 2002).

The primary aims of this article are 2-fold. First, we briefly report on the findings from a descriptive study exploring the current state of evaluation training in CACREP-accredited doctoral-level CE programs. Second, several workable recommendations to enhance current evaluation preparation for program evaluation and counselor educators at the doctoral level are provided. Within this context, a planning and implementation tool called the Program Evaluator Competency Matrix (PECM) is introduced. It is our hope that the results of the study will lead to improvements in the program evaluation training for students pursuing a PhD/doctorates of education (EdD) in CE. This study sought to document existing program evaluation requirements by answering these research questions: (a) To what extent, do current CACREP-accredited or near-accredited doctoral programs in CE include program evaluation in their curriculum? (b) How are program evaluation components delivered to doctoral-level CE students?

Method

Sampling

To locate possible institutions with CE programs, an archival research procedure supplemented with current website search methodology was utilized (LaVelle & Donaldson, 2010). Specifically, the research team composed of the lead author, a research CE professor, and a doctoral CE student developed the sampling frame by creating an Excel™ spreadsheet of CACREP-accredited university-based CE training programs located in the United States. Data were aggregated by geographical region and other demographic descriptors. The nation-wide sampling frame and unit of analysis included 100% of the doctoral-level (PhD and EdD) CE programs (N = 81). In other words, 81 institutions/universities with a CE program at the doctoral level were included in this study. Of these doctoral-level programs, 76 (93.8%) were CACREP accredited and 5 (6.2%) were in the process of accreditation. Because these institutions are obligated to comply with CACREP (2009, 2016) standards regarding program evaluation preparation for doctoral students, these units were targeted for the study. Non-CACREP accredited doctoral-level CE programs were not included in the sample.

Applying the Association for Counselor Education and Supervision regional categorization of CE programs as a guide, of the 81 doctoral programs purposefully sampled, 44 (54.3%) were situated in the Southern region of the United States. The other programs represented North-Central (n = 21, 25.9%), Rocky Mountain (n = 8, 9.9%), North Atlantic (n = 6, 7.4%), and Western (n = 2, 2.5%) regions. Sixty-three (77.8%) of the institutions awarded PhDs, 12 (14.8%) granted EdD, and 6 (7.4%) offered both a PhD and an EdD track. Of the institutions reviewed, 58 (71.6%) were public and 23 (28.4%) private. The majority of CE programs were campus based (n = 75, 92.6%) rather than delivered online (n = 6, 7.4%). Some doctoral programs were perhaps offered in a hybrid format, delivering coursework both on-campus and online, but this option was not discernable from the institutional websites.

Procedures

Each institution with a doctoral-level CE program was identified through the CACREP website in May 2017. Database inclusion criteria were kept to a minimum. Each institution’s doctoral CE program (a) needed to be CACREP accredited or in process and (b) webpages provided sufficient information including program description and foci, degree awarded, coursework, course descriptions, and graduation requirements. Based on an abbreviated version of LaVelle and Donaldson’s (2010) Internet-based webpage/document analysis technique, researchers first employed the Chrome™ webpage browser to locate each doctoral-level CE program within participating institutions. Next, using each institution’s internal search capabilities, descriptors such as ‘‘program evaluation” and “research and evaluation” were searched. Internal links to CE program literature were viewed and analyzed. Institutional demographic data were collected as well as any information that referred to program evaluation in the curriculum and pedagogy was noted on a spreadsheet. This matrix with predefined categories served as a rubric to catalogue data (e.g., institution, address, PhD/EdD, website, program evaluation coursework, delivery style, and program evaluation content). Researchers reviewed any online materials (e.g., program descriptions, course catalogues and descriptions, program handbooks, and syllabi) provided by each institution’s website. Curriculum was broadly defined as any program evaluation and research coursework, assignments, activities, and so on that were indicated on the program webpages. Pedagogical information (methods of teaching, course delivery style, etc.) relating to evaluation and research were entered on the spreadsheet as well. Researchers were trained, and to maintain consistency and rigor, analyses and findings were regularly reviewed and consensus arrived at.

Results and Discussion

The study attempted to document the level of program evaluation training provided in fully or in process CACREP accredited doctoral-level CE programs. After carefully studying each institution’s website and associated materials, all 81 doctoral-level CE programs met the criteria for inclusion in the study. In general, most program webpages summarized their overall degree focus and its components relatively well. The researchers found that only 10% (n = 8) of CE programs offered a mandatory course that singularly focused on program evaluation. In most units, the required class was titled Program Evaluation or something similar (e.g., Evaluating Programs). Approximately 15% (n = 11) of the CE programs infused program evaluation instruction as part of a required research class. A sample infused class was titled Qualitative Research and Program Evaluation. Additionally, classes that did not mention program evaluation in the title, but their course descriptions included this material, were placed in the infused category. About 13% (n = 10) of the reviewed CE programs offered to doctoral students a program evaluation course as one of their research electives. If program evaluation instruction was a requirement or an option, it appears that the content was largely delivered face-to-face on campus. Nearly, 62% (n = 47) of the reviewed programs did not readily identify the ways CACREP program evaluation standards were being addressed.

Given the increasing exigency for counselor accountability and need for well-prepared program evaluators in the counseling profession over the past several decades, the findings are disconcerting (Astramovich & Coker, 2007; Brott, 2006; Reupert et al., 2012; Schaffer & Atkinson, 1983; Wheeler & Loesch, 1981). Of the institutions reviewed, only 25% required their doctoral students to take either a stand-alone program evaluation course or a research course that included program evaluation content and instruction. The remainder of CE programs appeared to place less curricular emphasis on CACREP evaluation-related standards. These findings reflect, in part, what Schaffer and Atkinson (1983) reported some 35 years ago. Their survey of 216 CE programs, principally at the master’s level, showed that program evaluation preparation was a low priority in comparison with instruction on scientific research methods. With the dearth of research in this area, the possible reasons for the continuing lack of commitment of counselor educators to program evaluation are speculative. Perhaps the confusion between what constitutes scientific research and program evaluation methods remains an issue (Healy, 2000; Wheeler & Loesch, 1981). CE program designers may assume that doctoral students who earned their master’s degree in counseling from an accredited institution had previously gained a sufficient skill set in research and evaluation. It is also plausible that doctoral CE programs offer minimal advanced training in program evaluation because CACREP (2009, 2016) standards in this area are nonspecific and few in number. Whatever the reasons for CE programs to be less committed to adequately training future counselor educators and counselors for entry level and advanced program evaluation responsibilities, it is important that the conditions hindering effective preparation are identified and resolved.

Recommendations

Based on this status report, it appears that nationally accredited CE doctoral programs that aim to equip their students to be competent evaluators are advised to modify their curriculum and, perhaps, instructional methods. Plentiful recommendations have been offered over the years to rectify the situation in CE (e.g., Astramovich & Coker, 2007; Hosie, 1994; Schaffer & Atkinson, 1983; Wheeler & Loesch, 1981). Instead of reiterating them here, the suggestions for educational modifications provided below stem largely from current information and literature. However, before specific refinements are proposed and implemented, program evaluation and counselor educators should establish a close partnership to ascertain their shared needs, goals, and processes.

Recommendation 1: Conduct a focused audit of existing research and program evaluation skill development courses and curriculum.

The need to improve the research and evaluation component of CE programs has been repeatedly affirmed in CE publications with little or no effect (e.g., Borders, Wester, Fickling, & Adamson, 2014; Heppner et al., 2016; Peterson et al., 2016; Sink, 2009; Wheeler & Loesch, 1981). The findings from this study provide the first data-based evidence that little improvement has occurred. The majority doctoral-level CE programs surveyed have minimal expectations for their students, certainly not enough for graduates to conduct rigorous and effective evaluations. The situation is perhaps due, in part, to the vague and limited number of CACREP (2009, 2016) program evaluation standards. Without more specificity, CE programs maintain their accreditation even with insufficient coursework in program evaluation.

To begin the process of restructuring, CE leadership and faculty should first conduct a focused program audit to identify gaps in evaluation-related coursework offered within the CE unit and by other departments. Keeping in mind CACREP (2016) standards, a crosswalk of goals, curricular offerings, and pedagogical strategies should be developed. This gap analysis should be conducted periodically, occurring at least every 3–5 years, and not necessarily in conjunction with CACREP reaccreditation activities. There are numerous resources to guide the process (e.g., Smith & Goodwin, 2014).

Including program evaluation scholars from outside the unit will enhance the viability of the audit outcomes. Together, these educators should revisit all pertinent curriculum, course materials, and pedagogy in terms of program evaluation skill development. Sample questions for an initial cross-discipline discussion may include How do program evaluation requirements and instruction meet the present needs of the CE profession and how are they aligned with best practice? In what ways is coursework aligned with CACREP (2016) standards for program evaluation? How are course objectives differentiating skill development in program evaluation at the master’s and doctoral levels? How are course objectives assessed and are they competency based? How do faculty document student attainment of specified skills in program evaluation? Are these methods sufficient? What course(s) should be offered by counselor versus program evaluation educators? In what ways can CE faculty partner with program evaluation educators on course development and curricular refinements, particularly to best meet current CACREP standards regarding research and evaluation? How does the coursework foster evaluative thinking (ET)?

Recommendation 2: Grounded on established program evaluation standards/competencies for practice, revise the evaluation component of CE..

Utilizing the audit results and ensuing discussions, the next phase is to implement the restructuring process, again in partnership with university program evaluation educators. A way to make certain the comprehensiveness and relevancy of program modifications is to frame them around specific research-based program evaluation competencies (Stevahn, King, Ghere, & Minnema, 2005). Specifically, faculty from CE and program evaluation units should review and propose refinements to coursework and instruction around the widely used and extensive program evaluation standards developed the Joint Committee on Standards for Educational Evaluation (Stufflebeam & Madaus, 1983; Yarbrough, Shulha, Hopson, & Caruthers, 2011). The 30 Joint Committee standards are arranged into five dimensions: utility, feasibility, propriety, accuracy, and evaluation accountability. Another suitable option is to use AEA’s (2004) Guiding Principles for Evaluators. The principles set out 25 expectations for evaluators covering these general domains of evaluation work: systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare. In their entirety, the standards and principles convey the need to consider the unique content and skills (e.g., evaluation models, managing processes and procedures, stakeholder involvement, program context, and political accountability) involved in program evaluation.

To further assist and simplify the cross-academic unit collaboration on curriculum and instructional planning and implementation, counselor and program evaluation educators may want to consult the recently developed PECM presented in Table 1. The authors designed the matrix to efficiently combine, adapt, and abbreviate AEA’s (2004) Guiding Principles and the Joint Committee’s standards (Yarbrough et al., 2011) into one user-friendly matrix. In developing the PECM, the content of the standards was matched to those guiding principles with comparable subject matter, and subsequently, reworded for clarity and relative succinctness. For example, the utility standard U2 attention to stakeholders (Evaluations should devote attention to the full range of individuals and groups invested in the program and affected by its evaluation.) was matched with the guiding principle B.2. (To ensure recognition, accurate interpretation and respect for diversity, evaluators should ensure that the members of the evaluation team collectively demonstrate cultural competence.) The resulting PECM competency reads: Be socially, politically, and culturally sensitive when selecting and deploying evaluation methods, processes, and procedures as well as when sharing findings and recommendations. Moreover, to establish a level of content and face validity for the PECM’s competencies, input from two experts in CE program evaluation was solicited. These scholars represented two different Research I public universities with CACREP accredited graduate programs in counseling.

Table 1.

Core Program Evaluation Competencies Related to CACREP Standards and School-Based PE Leadership Skill Areas.^a

Competency	Applicability Level			School-Based PE Leadership Skill Areas
Program Evaluators-in-Training Will Develop the Skills Needed to	CACREP (2016, p. 14) Standards Section 2.8	Master	PhD/EdD	Counseling Model Development	External Evaluators of Counseling Programs	Conduct Policy-Related Evaluations	Curricular Decision-Making	Graduate-Level PE Instruction
1.0 Professional practice and effective communication
1.1 Conduct evaluation in accordance with ethical standards and legal requirements	j. Ethical and culturally relevant strategies for conducting, interpreting, and reporting the results of research and/or program evaluation	√	√		√	√	√	√
1.1 Be socially, politically, and culturally sensitive when selecting and deploying evaluation methods, processes, and procedures as well as when sharing findings and recommendations	j. Ethical and culturally relevant strategies for conducting, interpreting, and reporting the results of research and/or program evaluation	√	√		√	√	√	√
1.2 Communicate effectively with stakeholders at all stages of the evaluation process		√	√		√	√	√	√
1.3 Collaborate and consult with stakeholders in an open and honest manner through evaluation stages		√	√		√	√	√	√
1.4 Clearly disclose considerations, procedures, data collection and analysis methods, results, and recommendations/implications that are consistent with the aims and objectives of the evaluation		√	√		√	√	√	√
1.6 Sensitively and prudently share with stakeholders recommendations with likely positive and negative repercussions/implications resulting from the evaluation		√	√	√	√	√	√	√
1.7 Write and submit quality evaluation proposals and formative and summative reports			√	√	√	√	√	√
1.8 Engage in self-reflective practice, assessing one’s evaluation knowledge, skills, and dispositions and pursue professional development activities to improve performance		√	√	√	√	√	√	√
2.0 Systematic analysis and data-based practice
2.1 Understand and appropriately use the knowledge base (evaluation models, terms, concepts, theories, and assumptions), processes and procedures	g. Designs used in research and program evaluation	√	√	√	√	√	√	√
2.2 Understand and apply evidence-based practice	b. Identification of evidence-based counseling practices	√	√
2.3 Locate and apply for evaluation opportunities			√		√	√		√
2.4 Understand and apply, as appropriate, qualitative, quantitative, and mixed research methods and associated strategies for valid and reliable data collection, analyses, and reporting	f. Qualitative, quantitative, and mixed research methods i. Analysis and use of data in counseling	√	√	√	√	√		√
2.5 Assess school-based settings in terms of potential strengths and limitations or barriers related to the evaluation and implement of recommendations for practice	c. Needs assessments	√	√	√	√	√	√	√
2.6 Design quality evaluation and related materials, incorporating relevant research literature and theory	a. The importance of research in advancing the counseling profession including how to critique research to inform counseling practiced. Development of outcome measures for counseling programs		√	√	√	√	√	√
2.7 Deploy relevant evaluation model, questions, and data collection and data analysis	h. Statistical methods used in conducting research and program evaluation	√	√	√	√	√		√
3.0 Program Management
3.1 Use evaluation resources (e.g., fund management and budgeting, materials, and personnel utilization) efficiently and wisely		√	√		√	√		√
3.2 Document work coherently and fully at each stage of the evaluation		√	√		√	√		√
3.3 Conduct evaluation in timely manner documenting steps and actions		√	√		√	√		√
3.4 Use relevant technology as required		√	√		√	√		√

Note. PE = program evaluation.

^aFor brevity, the table is referred to as Program Evaluator Competency Matrix; competencies were adapted and abridged from AEA’s (2004) Guiding Principles for Evaluators, Joint Committee on Standards for Educational Evaluation (as reproduced in Yarbrough et al., 2011), and CACREP (2016) Standards (section 2.8. research and program evaluation); Heppner et al. (2016), Stevahn et al. (2005).

After incorporating outside feedback, PECM’s competencies were mapped to (a) CACREP (2016, p. 14) Standards for Research and Evaluation (see Section 2: Professional Counseling Identity); (b) their applicability level (i.e., Are they primarily applicable to master’s or doctoral level students or both?); and (c) school-based program evaluation leadership skill areas (Carey & Trevisan, in press). The resulting version of the PECM offers 19 core competencies organized into three broad domains: professional practice and effective communication, systematic analysis and data-based practice, and program management. Indicators were added to show users whether the competency and CACREP standard should be taught primarily at the master’s or doctoral level. For counselor educators preparing doctoral-level school counselors, the competencies are also linked with Carey and Trevisan’s program evaluation leadership skill areas for counselor educators. These include (a) counseling model development, (b) external evaluators of counseling programs, (c) policy-related evaluations, (d) curricular decision-making, and (e) graduate-level program evaluation instruction. These categories assist CE and program evaluation educators to determine where individual competencies fit into core topical areas in CE preparation.

The PECM has multiple practical applications for graduate-level counselor and program evaluation educators. For instance, the tool should prove useful to faculty members as they make decisions related to program evaluation curriculum development and instructional revisions. PECM competencies can also be converted into course objectives and serve as the basis for a program evaluation skill attainment rating scale for graduating CE doctoral students. The matrix can also serve as a guiding document to design a formative or summative questionnaire, allowing CE faculty to survey program graduates about material they have retained and are using in the field. These data should be shared with program evaluation instructors to further refine their work with CE students. Specific benchmarks could be developed for each competency to further delineate skill development. Subsequently, counselor and program evaluation educators would rate the students at one of four levels of proficiency: 1. learning (student understands knowledge base), 2. emerging (student demonstrates basic skills); 3. assisting and supporting (student can support and collaborate with evaluation project leadership); and 4. leadership (student demonstrates the professional skill set to oversee an evaluation project). To summarize, the PECM can be employed by program evaluation and counselor educators as a tool for aligning evaluation curriculum and instruction with professional standards (see Table 1).

Recommendation 3: Through collaboration with research and program evaluation faculty and field-based evaluators, revisit curriculum and pedagogy to affirm that students’ possess appropriate research and data analysis skills and become “evaluative thinkers.”

One potential implication of the findings from the current study is that counselor educators may underestimate the skill set needed to be a successful evaluator. Given the lack of CE program requirements in this area, they may view program evaluation as a collection of skills that can be gained with only a modicum of instruction and practical experience. However, as Buckley, Archibald, Hargraves, and Trochim (2015) pointed out, effective program evaluators are not only skilled in designing and implementing program evaluation activities, but they are encouraged to also deploy ET. These scholars defined ET as

critical thinking applied in the context of evaluation, motivated by an attitude of inquisitiveness and a belief in the value of evidence, that involves identifying assumptions, posing thoughtful questions, pursuing deeper understanding through reflection and perspective taking, and informing decisions in preparation for action. (p. 378)

Given the limited program evaluation training provided to CE doctoral students, it is unlikely that current students achieve a satisfactory level of critical/ET to be effective evaluators. Together with the development of metacognitive and higher order inquiry skills, program evaluators need to demonstrate foundational and advanced knowledge in research-related skills. Students should develop proficiency in quantitative, qualitative, and mixed research methods, complex statistical procedures, and psychometrics including test and questionnaire development and implementation (Figueredo, Olderbak, Schlomer, Garcia, & Wolf, 2013; Peterson et al., 2016). Research faculty and program evaluation instructors can assist CE faculty to create the requisite learning opportunities where ET skills can be developed. Comprehensive and well-designed research and evaluation coursework, cross-discipline seminars, and workshops may be helpful here. As further described below, CE students tend to appreciate instruction that emphasizes active learning, small groups, and real-world activities.

To foster evaluative/critical thinking in CE students, there are a number of instructional methods counselor and program evaluation educators might find effective. The relevant literature indicates that socratic questioning, journal writing, collaborative, contextual or field-based training, experiential (“hands on”) learning (e.g., semester-long program evaluation internships), mentoring, and problem and project-based learning are beneficial (e.g., Abrami et al., 2015; Granello, 2000; Griffith & Frieden, 2000). Moreover, learning activities should be intentionally planned to access doctoral-level CE students’ previously established skill set, particularly as they tap into ET. For example, these graduate students should be relatively proficient with consultation, systems thinking, communication, open questioning, self-reflection, intuition, diagnosis, and action/intervention planning. In other words, program and counselor educators can nurture ET in CE doctoral students by focusing their instruction and classroom activities on the transfer of learning from “counselor thinking” to “ET.” It should be noted, however, for students to reach this level of evaluative functioning, they may need two specific program evaluation courses that address foundational and advanced-level skill development, respectively, as well as a variety of pertinent field-based experiences (LaVelle & Donaldson, 2010).

Research Caveats and Suggestions for Future Research

As LeVelle and Donaldson (2010) elucidated, a review of university webpages is a less than an ideal strategy to grasp the full extent of program information and offerings. In some cases, the webpages analyzed by the researchers appeared to be dated, incomplete, and difficult to navigate. Locating applicable information was a challenge and many sites lacked specifics on course requirements and content relating to program evaluation. The investigators also relied on key search terms such as “program evaluation” and “research and evaluation.” It is more than likely these explorations failed to capture the entire scope of evaluation preparation offered within CE programs. To maintain uniformity and rigor in the review process, the researchers did not survey the small number of non-CACREP-accredited doctoral CE programs. Thus, estimates regarding the level of program evaluation training in CE programs across the country are imprecise.

Future research in this area should include interviewing a sample of doctoral program CE coordinators and faculty to determine the accuracy and completeness of online program materials. These interviews could also explore whether (a) program evaluation is adequately addressed in the curriculum and (b) the coursework and associated activities are meeting current CACREP standards in this area. Researchers could combine CE program materials with interview data to map CACREP evaluation standards with doctoral program evaluation goals and outcomes from actual coursework including assignments and assessments.

Concluding Remarks

As Reupert, McHugh, Mayberry, and Mitchell (2012) advised, program evaluation is an indispensable research and accountability method. Program evaluation done well generates an evidence base to assist counselor educators and researchers, as well as counselors, policy makers, and funders to reach consensus on the most effective ways to implement school-based systemic innovations and mental health counseling strategies. Whether students matriculating from doctoral-level CE programs retain the essential research and evaluation skills to meet the increasing demand for quality evaluations remains an open question. The findings from the current study suggest that the preponderance of CE units lack sufficient program evaluation requirements and coursework, especially at the doctoral level, to adequately prepare quality evaluators. If graduates desire advanced training, it largely occurs through extended fieldwork and professional development activities. To improve the current situation, several tentative recommendations were proffered. First, CE program leadership and faculty, in partnership with program evaluation educators, should conduct an audit aimed at documenting the unit’s program evaluation requirements, objectives, curriculum, and instruction. Next, after identifying missing elements and areas necessitating further attention, research-based program evaluation standards should be consulted and deployed as the restructuring framework. The PECM was introduced as one useful tool to accomplish this task. The matrix is applicable to the evaluation work of counselor educators, practicing counselors, and graduate-level preparation of future evaluators. Finally, we suggest that CE and program evaluation educators design their curriculum and instruction to foster ET in students.

In summary, academics and clinicians in counseling, education, and other related fields are obligated to have facility with program evaluation processes, procedures, and methods. CACREP (2009, 2016) standards, albeit brief and nonspecific, as well as the ethical codes of the counseling profession require a high level of accountability. Based on the current study, doctoral-level counseling students may not be receiving the requisite skill set to properly conduct program evaluations. Should this be the case, without advanced preparation outside university settings, graduates will find it challenging to document the effectiveness of their counseling programs, services, and activities. Moreover, future counselor educators will be ill-prepared for program evaluation requirements associated with the faculty role. CE programs with assistance from program evaluation educators need to heed the longstanding appeal for improved preparation in research and program evaluation skills. Both sets of educators could partner to establish courses and relevant training materials, whether they be in vivo or online, that follow standards and guidelines established by AEA and meet the requirements of the latest CACREP accreditation standards. Hopefully, this article reinvigorates this important discussion and cross-departmental collaboration, engendering substantial CE program reform.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Abrami

P. C.

Bernard

R. M.

Borokhovski

Waddington

D. I.

Wade

C. A.

Persson

(2015). Strategies for teaching students to think critically: A meta-analysis. Review of Educational Research, 85, 275–314.

American Counseling Association. (2014). 2014 ACA code of ethics. Retrieved from https://www.counseling.org/resources/aca-code-of-ethics.pdf

American Evaluation Association. (2004). Guiding principles for evaluators. Retrieved from http://www.eval.org/d/do/594

American Mental Health Counseling Association. (2016). AMHCA standards for the practice of clinical mental health counseling. Retrieved from http://connections.amhca.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=e6b635b0-654c-be8d-e18c-dbf75de23b8f

American School Counselor Association. (2012a). ASCA national model: A framework for school counseling programs (3rd ed.). Alexandria, VA: Author.

American School Counselor Association. (2012b). Position statement: The school counselor and comprehensive school counseling programs. Retrieved from https://www.schoolcounselor.org/asca/media/asca/PositionStatements/PS_ComprehensivePrograms.pdf

American School Counselor Association. (2016). ASCA ethical standards for school counselors. Retrieved from https://www.schoolcounselor.org/asca/media/asca/Ethics/EthicalStandards2016.pdf

Astramovich

R. L.

Coker

J. K.

(2007). Program evaluation: The accountability bridge model for counselors. Journal of Counseling and Development, 85, 162–172.

Astramovich

R. L.

Coker

J. K.

Hoskins

W. J.

(2005). Training school counselors in program evaluation. Professional School Counseling, 9, 49–54.

10.

Baker

(2012, 12 1). A new view of evidence-based practice. Counseling Today. Retrieved from http://ct.counseling.org/2012/12/a-new-view-of-evidence-based-practice/

11.

Borders

L. D.

Wester

K. L.

Fickling

M. J.

Adamson

N. A.

(2014). Research training in doctoral programs accredited by the council for accreditation of counseling and related educational programs. Counselor Education & Supervision, 53, 145–160.

12.

Brott

P. E.

(2006). Counselor education accountability: Training the effective professional school counselor. Professional School Counseling, 10, 179–188.

13.

Bryant

J. K.

Druyos

Strabavy

(2013). Gatekeeping in counselor education programs: An examination of current trends. VISTAS: Online, 51, 1–13. Retrieved from https://www.counseling.org/docs/default-source/vistas/gatekeeping-in-counselor-education-programs.pdf?sfvrsn=7f6e77b5_13

14.

Buckley

Archibald

Hargraves

Trochim

W. M.

(2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36, 375–388.

15.

Carey

Trevisan

(in press). School counselor education and program evaluation. American Journal of Evaluation.

16.

Cook

C. R.

Lyon

A. R.

Kubergovic

Wright

D. B.

Zhang

(2015). A supportive beliefs intervention to facilitate the implementation of evidence-based practices within a multi-tiered system of supports. School Mental Health, 7, 49–60.

17.

Council for Accreditation of Counseling and Related Educational Programs. (2009). CACREP standards. Retrieved from http://www.cacrep.org/wp-content/uploads/2013/12/2009-Standards.pdf

18.

Council for Accreditation of Counseling and Related Educational Programs. (2016). CACREP standards. Retrieved from http://www.cacrep.org/wp-content/uploads/2012/10/2016-CACREP-Standards.pdf

19.

Figueredo

A. J.

Olderbak

S. G.

Schlomer

G. L.

Garcia

R. A.

Wolf

P. S. A.

(2013). Program evaluation: Principles, procedures, and practices. In Little

T. D.

(Ed.), The Oxford handbook of quantitative methods, Vol. 1: Foundations (pp. 332–360). New York, NY: Oxford University Press.

20.

Granello

D. H.

(2000). Contextual teaching and learning in counselor education. Counselor Education and Supervision, 39, 270–283.

21.

Griffith

B. A.

Frieden

(2000). Facilitating reflective thinking in counselor education. Counselor Education and Supervision, 40, 82–93.

22.

Gysbers

N. C.

Henderson

(2012). Developing and managing your school guidance and counseling program (3rd ed.). Alexandria, VA: American Counseling Association.

23.

Haberstroh

Duffey Marble

Ivers

N. N.

(2014). Assessing student-learning outcomes within a counselor education program. Counseling Outcome Research and Evaluation, 5, 28–38. doi:10.1177/2150137814527756

24.

Healy

(2000). Knowing what works: Program evaluation. New Directions for Student Services, 90, 57–65. doi:10.1002/ss.9005

25.

Heppner

P. P.

Wampold

B. E.

Owen

Wang

K. T.

Thompson

M. N.

(2016). Research design in counseling (4th ed.). Boston, MA: Cengage Learning.

26.

Hosie

T. W.

(1994). Program evaluation: A potential area of expertise for counselors. Counselor Education & Supervision, 33, 349–356.

27.

Kratochwill

T. R.

Shernoff

E. S.

(2004). Evidence-based practice: Promoting evidence-based interventions in school psychology. School Psychology Review, 33, 34–48.

28.

LaVelle

J. M.

Donaldson

S. I.

(2010). University-based evaluation training programs in the United States 1980–2008: An empirical examination. American Journal of Evaluation, 31, 9–23. doi:10.1177/1098214009356022

29.

Peterson

C. H.

Hall

S. B.

Buser

J. K.

(2016). Research training needs of scientist-practitioners: Implications for counselor education. Counselor Education and Supervision, 55, 80–94.

30.

Reupert

McHugh

Mayberry

Mitchell

(2012). Training mental health promotion workers in programme evaluation. Journal of Psychiatric and Mental Health Nursing, 19, 31–39.

31.

Royse

Thyer

B. A.

Padgett

D. K.

(2016). Program evaluation: An introduction to an evidence-based approach (6th ed.). Boston, MA: Cengage.

32.

Schaffer

J. L.

Atkinson

D. R.

(1983). Counselor education courses in program evaluation and scientific research: Are counselors being prepared for the accountability press? Counselor Education and Supervision, 23, 29–34. doi:10.1002/j.1556-6978.1983.tb00585.x

33.

Sink

C. A.

(2009). School counselors as accountability leaders: Another call for action. Professional School Counseling, 13, 68–74.

34.

Sink

C. A.

(2016). Incorporating a multi-tiered system of supports into school counselor preparation. Professional Counselor, 6, 203–219.

35.

Smith

C. F.

Goodwin

(2014). A guided empowerment self-audit as a school improvement strategy. Research in Higher Education Journal, 25. Retrieved from http://files.eric.ed.gov/fulltext/EJ1055326.pdf

36.

Stevahn

King

J. A.

Ghere

Minnema

(2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59.

37.

Stufflebeam

D. L.

Madaus

G. F.

(1983). The standards for evaluation of educational programs, projects, and materials. In Madaus

G. F.

Stufflebeam

D. L.

(Eds.), Evaluation in education and human services (pp. 395–404). New York, NY: Springer Netherlands.

38.

Trevisan

M. S.

(2000). The status of program evaluation expectations in state school counselor certification requirements. American Journal of Evaluation, 21, 81–94.

39.

Trevisan

M. S.

(2002). Evaluation capacity in K-12 school counseling programs. American Journal of Evaluation, 23, 291–305.

40.

Tyler

R. W.

(1983). A rationale for program evaluation. In Madaus

D. L.

Scriven

M. S.

Stufflebeam

D. L.

(Eds.), Evaluation models: Viewpoints on educational and human services evaluation (Vol. 6, pp. 67–78). Boston, MA: Kluwer-Nijhoff.

41.

Vacc

N. A.

Rhyne-Winkler

M. C.

(1993). Evaluation and accountability in counseling services: Possible implications for a midsize school district. The School Counselor, 40, 260–266.

42.

Wheeler

P. T.

Loesch

(1981). Program evaluation and counseling: Yesterday, today, and tomorrow. Journal of Counseling & Development, 59, 573–578.

43.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.

44.

Ziomek-Daigle

Goodman-Scott

Cavin

Donohue

(2016). Integrating a multi-tiered system of supports with comprehensive school counseling Programs. Professional Counselor, 6, 220–232.