Evaluator Skill Acquisition

Abstract

This article presents the results of a survey conducted in an attempt to examine evaluators’ perceptions of the contributions that different educational experiences make to the development of their evaluation skill sets. Self-identified new and graduate student evaluators were asked to describe their evaluation training experiences and to indicate how important each experience was to the development of particular evaluation competencies. A paired-comparison analysis was conducted to determine the scaling of the contribution each educational experience made to the development of each competency. Results show that the perceived contribution of different training experiences changes depending on which evaluation competency is being developed. Pedagogical implications of these findings are also discussed.

Keywords

evaluator training evaluator competencies professional issues mentoring fieldwork

Evaluation demands myriad skills of its practitioners (Dewey, Montrosse, Schroter, Sullins, & Mattox, 2008; Stevahn, King, Ghere, & Minnema, 2005), and training experiences of various kinds contribute to the development of essential evaluator competencies. While there are many articles that explore evaluator competencies or describe evaluation training programs, there is little empirical work explaining how novice evaluators acquire the competencies necessary to practice evaluation effectively. The hope is that the work presented in this article can guide future empirical investigations into the means by which evaluators develop their competencies.

Using the Stevahn et al. (2005) taxonomy of evaluator competencies as a guide, this study explores how different educational experiences (specifically: coursework, mentorship, fieldwork, and participation in professional activities) contribute to the development of these competencies. Understanding this process has two implications for the field. First, it grants insight into evaluators’ perceptions of their competency development. Second, it could influence evaluation pedagogy.

In their recent article cataloguing evaluation training programs, LaVelle and Donaldson (2010) posed this question, “perhaps our concern may shift from ‘where will the next generation be trained?’ to ‘how can we best prepare the next generation for the challenges they will face?’” (p. 21). This study takes steps toward furthering that conversation by investigating evaluators’ perceptions of the ways educational experiences impact their competency development. Specifically, this study explores (1) the educational experiences of new and graduate student evaluators and (2) how new and graduate student evaluators perceive the relative contributions that different educational experiences make to developing their evaluation competencies.

Evaluator Competencies and Evaluation Training

There has been considerable work in the field on guidelines for both desired evaluator practice and essential evaluator competencies. The former are legitimized in the Guiding Principles for Evaluators developed by a task force of the American Evaluation Association (1995), and the Program Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation (Yarbrough, Shulha, Hopson, & Caruther, 2011). However, as a field, we have yet to reach consensus on the latter (Ghere, King, Stevahn, & Minnema, 2006). Several evaluation scholars have suggested that establishing evaluator competencies could benefit the practice of evaluation in several ways, including improving training, enhancing reflective practice, advancing research on evaluation, and professionalizing the field (Altschuld, 1999; Stevahn et al., 2005). Stevahn, King, Ghere, and Minnema’s (2005) taxonomy of essential competencies for program evaluators is the most recent and comprehensive effort in detailing competencies. The authors outlined six categories of evaluator competencies: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal competence.

Discussions of evaluator competencies naturally lead to considerations of evaluation training programs. Research on evaluation training generally falls into two categories: (1) empirical work describing and cataloging evaluation programs as they already are (Altschuld, Engle, Cullen, Kim, & Macce, 1994; Engle, Altschuld, & Kim, 2006; LaVelle & Donaldson, 2010) or (2) conceptual work discussing what the content and structure of evaluation training should be (Davis, 1986; Fitzpatrick, 1994; Ghere et al., 2006; Mertens, 1994; Stufflebeam, 2001). Both types of studies rest on the premise that proper evaluation training is essential to the health of the field (Stufflebeam, 2001). LaVelle and Donaldson (2010) highlight this point in saying, “Evaluators are made, not born, and an extended period of training is necessary to master the evaluation-specific skills and knowledge necessary to provide quality service to clients, and be socialized into the professional frameworks, standards, and ethical guidelines” (p. 10).

Several scholars have focused their attention on the substance and structure of evaluation training programs. A frequent theme in these explorations is that evaluation training would benefit from alignment with competencies or the particular skill set deemed essential for program evaluators (Davis, 1986; Fitzpatrick, 1994; Ghere et al., 2006; Mertens, 1994; Stufflebeam, 2001). For example, Fitzpatrick (1994) calls for a needs-assessment approach to constructing an evaluation training program. She specifically looks to the model provided by professional schools and argues that “using the professional school model forces us when we develop our curriculum to focus on the competencies required of practicing evaluators. Thus, our curriculum becomes [. . . ] a coherent program aimed at producing graduates who have the knowledge and skills needed to perform the kinds of evaluations that organizations need now and in the future” (pp. 43–44).

Despite the body of work surrounding both evaluator competencies and evaluation training programs, there is little literature exploring how evaluation is learned from the perspective of the evaluation student or novice evaluator. Specifically, we lack awareness of how the variety of educational experiences offered in evaluation training programs contributes to the development of essential skills and competencies.

The Study

This study seeks to document participant educational experiences and evaluation competencies. Of particular interest are the perceptions of the relative importance of the contributions of different educational experiences to the development of different evaluation competencies.

Participants

The population pool consisted of 572 American Evaluation Association members who self-selected into the Graduate Student and New Evaluators Topical Interest Group (TIG). This group was targeted because evaluators who self-identify as graduate students or new evaluators are likely to have more recent education experiences and would be better able to recall their training. Underlying this choice is the idea that these responses would most accurately represent the current state of evaluation education in the field.

The e-mail distribution list for this TIG was acquired in the winter of 2011. Each member of this group was contacted via e-mail shortly thereafter and invited to participate in a confidential online survey (four of these e-mails were returned as undeliverable). The online survey remained active for 2 weeks, and all TIG members were sent one additional e-mail reminding them to participate. As an incentive, respondents were entered into an opportunity drawing for a $50 gift card. E-mail addresses for the drawing were collected separately from survey responses to ensure survey confidentiality.

Of the 568 delivered e-mails, 179 people completed the survey, representing a 31.5% response rate. Eleven of these respondents started, but did not complete the survey. Their answers are included in analyses only when possible and appropriate. There is a potential sample selection bias that may limit the generalizability of these findings. It is unknown whether or not there are systematic differences in education, experience, and/or beliefs between those who chose to respond to the survey and those who did not. As such, the findings from this analysis should be considered with this limitation in mind.

A broad range of educational and work experiences are represented in the sample. Table 1 displays education level obtained by years of practicing evaluation.

Table 1.

Education Obtained by Years of Experience Practicing Evaluation.

	Years practicing evaluation
Highest level of education obtained	<1	1–2	3–4	5–6	>6	Totals	%
Bachelor’s degree	3	1	2	0	0	6	3
Master’s degree	11	24	18	4	6	63	35
PhD or other doctorate	1	0	3	3	7	14	8
In-process master’s	9	9	1	2	0	21	12
In-process PhD or other doctorate	15	15	29	9	6	74	42
Totals	39	49	53	18	19	178^a
%	22	28	30	10	11

^a n = 178. One participant did not answer this question.

The fact that 80% of respondents had fewer than 5 years of work experience and 44% of respondents described themselves as current students indicates that overall, the education and work experience of the population of respondents is consistent with the title of the TIG (Graduate Student and New Evaluators) selected for examination. However, there were several instances in which respondents did not match the profile for the intended study population. Specifically, in 13 cases, respondents had more than 6 years of work experience and are not presently enrolled in school. A sensitivity analysis demonstrated that these respondents did not answer questions in ways that differed significantly from those respondents with less education and experience. Therefore, their responses are included in further analyses.

Participants were asked to indicate the focus of their degree (and encouraged to mark all responses that applied, leading to total numbers that surpass the number of respondents). A wide variety of disciplines are represented in the sample. Participants most commonly indicate earning degrees in evaluation or educational research (n = 75), education (n = 59), and research methods (n = 52). Other responses include psychology (n = 37), public health (n = 24), public administration (n = 19), public policy (n = 17), and social work (n = 13).

Respondents also elaborated on their employment status, setting, and roles. One hundred and sixty respondents (79%) are employed either full- or part-time. Participants were presented with choices describing various employment settings and job functions and were asked to select as many options as described them. Twenty-eight percent of participants (n = 39) chose more than one response to describe their employment setting and 43% (n = 60) chose more than one response to indicate their job function. This suggests that evaluators wear many hats and fill many roles while executing their job responsibilities. As shown in Table 2, participants are most commonly found in academia (n = 66), as that was selected more than twice as frequently as any other employment setting. For respondents who indicated academia as a part of their job function, nearly 3 times as many identified as researchers (n = 42) than professors (n = 10).

Table 2.

Respondents’ Employment Setting and Job Function.

Employment setting	n	%	Job function	n	%
Academia	66	47	External program evaluator	51	37
Nonprofit	29	21	Researcher	42	30
Consulting	24	17	Internal program evaluator	33	24
Research organization/think tank	22	16	Academia–researcher	28	20
K–12	17	12	Consultant	23	17
Public health	16	12	Other	15	11
Other	11	8	Program developer	12	9
			Academia–professor	10	7
			Program operator	8	6

Note. n = 139.

Survey Instrument

Respondents were asked to complete a web-based evaluator skill acquisition survey designed primarily to measure two constructs—educational experiences and evaluator competencies—and the relationship between them. To develop the educational experiences construct, current graduate students and evaluation faculty members were asked to generate lists of the educational experiences they participated in during their training. The items on this list were then divided into four broad categories believed to capture evaluation educational experiences: coursework, mentorship, fieldwork, and participation in professional activities. The evaluator competencies constructed in the survey are based, with some modification, on the evaluator skill taxonomy developed by Stevahn et al. (2005). The competencies are methodological knowledge, theoretical knowledge, project management skills, effective communication skills, and contextual consideration. Table 3 lists the categories of educational experiences and evaluator competencies used in the survey. Each of these categories was defined by way of the example items listed in the table. Survey questions further solicited information to aid in understanding participants’ perceptions about how educational experiences foster the development of specific evaluation competencies.

Table 3.

Definitions of the Survey Constructs.

Educational experiences	Items from survey
Coursework	quantitative methods qualitative methods mixed methods	evaluation theory evaluation procedures research design
Mentorship	mentored by an advisor mentored by other faculty/staff mentored by a more advanced student	mentored by other professional served as a mentor to someone else
Fieldwork	engaging stakeholders describing the program designing an evaluation	gathering evidence analyzing data justifying conclusions communicating findings
Participation in professional activities	member of AEA or other professional organization	attend conferences attend workshops
Evaluation competencies
Contextual consideration	evaluation use respecting uniqueness of client/site	understanding political influences involving stakeholders
Project management skills	budgeting working with clients/stakeholders	writing agreements supervising training
Effective communication skills	communicating with stakeholders	conflict resolution negotiation
Methodological knowledge	research design	collecting and analyzing data
Theoretical knowledge	history of evaluation evaluation models	evaluation trends evaluation philosophies theories of evaluation

The survey instrument was developed over a 6-month period in consultation with both seasoned evaluators and measurement experts. The survey was designed to minimize both the time required to complete it and any potential effects that might have resulted from the ordering of questions. The survey was tested and piloted among graduate students in an evaluation program. Modifications were made based on initial results and suggestions offered by pilot participants. The final version of the survey included questions about respondents’ work experiences, educational background, the activities they participated in during their schooling, their confidence in various aspects of conducting evaluations, and their perceptions of educational experiences’ contributions to the development of various competencies. The survey consisted of 55 items across 16 pages and was designed to take between 10 and 15 min to complete.

Analysis

A paired-comparison analysis was used to order the four educational experiences along a continuum of perceived contribution to the development of each of the five evaluation competencies. By using individuals’ choices between paired options, the analysis facilitates the ordering of the educational experiences along a scale.

Paired-comparison analysis is based on Thurstone’s law of comparative judgment (as described in Edwards, 1957). It is a technique most often used for constructing attitude scales for which there is no physical or objective measure. The law assumes that for any given attitude statement, there is a most frequent reaction to that statement for a particular individual, referred to by Thurstone as the modal discriminal process. For example, an individual will find a particular statement either favorable or unfavorable most of the time. However, the law allows that an individual will not always have the same reaction to that statement. Rather, the assumption is that the different reactions (discriminal processes) will be normally distributed about the modal discriminal process (Edwards, 1957).

We can illustrate this concept through a hypothetical example using items from the evaluator skill acquisition survey. If we wanted to develop a scale for the usefulness of educational activities in developing a particular evaluation competency, we may present “Fieldwork” as an item and ask if it is favorable or unfavorable to developing contextual consideration skills. An individual will find that statement either favorable or unfavorable most, but not all of the time. If we were to plot the multiple responses (discriminal processes), they would be normally distributed about the most frequent response (the modal discriminal process).

We can extend the law of comparative judgment to consider the relationship of multiple items to a single scale. Presenting 2 items to a large group of individuals and asking them to indicate which is more favorably related to the development of a particular competency will result in one of three outcomes: (1) the 2 items will be considered favorable in equal measure; (2) Item A will be considered favorable more often than Item B; and (3) Item B will be considered favorable more often than Item A. The latter two cases will result in a separation of modal discriminal processes along the scale depicting the development of contextual consideration.

This can be further explained using the contextual consideration skills competency. Consider “Fieldwork” as Item A and “Coursework” as Item B. If we were to plot the results of asking a group of individuals to judge which item, A or B, is more favorably related to developing contextual consideration skills on a graph (with contextual consideration skills as the horizontal axis), we would see a separation between their distributions. Specifically, the modal discriminal process of Item A would have a higher value than that of Item B. This means “Fieldwork” is considered more related to the development of contextual consideration than “Coursework.”

Survey respondents were presented with pairs of educational experiences and asked to choose between them (as described above). This resulted in an empirical frequency of the number of times statements are judged more or less favorable in relation to one another. Thus, this procedure allowed an ordering of educational experiences (coursework, mentorship, fieldwork, and participation in professional activities) along a continuum of their perceived contribution to the development of a particular evaluation competency (contextual consideration, project management skills, effective communication skills, methodological knowledge, and theoretical knowledge).

Limitations

This study has limitations worthy of mention. The choice of the new and graduate student TIG as the sample from which to draw participants, while deliberate, does limit the generalizability of these findings. The choice excludes both evaluators who have been working in the field for an extended period of time and those who may have entered “through the back door” (practitioners who may have extensive training and experience in another discipline, but who have found themselves undertaking evaluation as a part of their job responsibilities). However, the intention of this study is to inform a discussion about evaluation training. As such, the choice of this TIG was guided by an interest in representing the present condition of evaluation training through the eyes of those with more recent experiences, rather than to document the experiences of evaluator training generally.

The sampling choice also necessitated a modification of the taxonomy of essential competencies for program evaluators (Stevahn et al., 2005); the taxonomy was not represented in its entirety in the survey instrument. Neither professional practice nor reflective practice was included in this study. The defining characteristics of these competencies (adherence to evaluation standards, ethics, and awareness of personal evaluation expertise and evaluation needs) and their descriptions suggested that they were competencies evaluators would establish throughout their careers (rather than primarily through specific evaluation training). In both instances, the essential competencies described under the domains (e.g., “aware of self as an evaluator—knowledge, skills, and dispositions;” and “conveys personal evaluation approaches and skills to clients”) implied a level of experience in the field that was beyond that of the target population. This is not to say that new and graduate student evaluators are not actively developing in these skill areas, just that this was not the main focus of this study.

Furthermore, using a paired-comparison analysis forces survey respondents to make choices between educational experiences. Several participants in this study point out the challenge in having to make a choice between experiences that they consider overlapping and/or complementary. However, the study needed to be designed in this way to facilitate the scaling of educational experiences. Exploration into the ways in which educational experiences support and complement one another was beyond the scope of this investigation, but is an important topic for future studies of evaluator competency acquisition.

Finally, this study does not explore qualitative differences in novice evaluators’ training experiences. The survey asked respondents to indicate the numbers of courses they have taken, the types of mentorship experiences they participated in, the different professional activities they engaged in, and the contents of their fieldwork experience. However, there is most likely a wide range in how these experiences might be characterized that the survey was unable to capture. Differences in types and qualities of educational experiences may influence how respondents rank their educational experiences. Again, this can be the subject of future research.

Findings

Respondents were asked a variety of questions about their educational experiences. Specifically, participants were asked to select, from among a list of 10 choices, the experiences they participated in during school that contributed to developing essential competencies as a program evaluator. After selecting the activities, respondents were asked to rank these choices according to which experiences were believed to contribute in the most important ways to their competency development. Respondents were able to select as many choices as applied to them (therefore, they indicated different total numbers of activities). Results are shown in Table 4.

Table 4.

Evaluators Rankings of Educational Experiences That Contribute to Their Skill Development.

	Rank										Totals
Activity	1	2	3	4	5	6	7	8	9	10	N	%
Coursework	43	25	30	17	14	4	1				134	87
Fieldwork	56	51	8	10	1	1					127	82
Conferences	4	11	22	24	25	6	7	1			100	65
Working with a mentor	27	20	14	12	8	5	2	1			89	58
Self-directed learning	8	11	21	14	15	15	3	2			89	58
Thesis/dissertation	7	13	12	18	4	5	3	1			63	41
Work placement	13	16	17	6	2	1	2				57	37
Group work	2	3	7	7	4	4	2	4		1	34	22
E-learning	1	1	2	2	5	5	2	3			21	14
Working as a mentor			1	1	2	2	4	1	2		13	8
Other	1		1	1							3	2

Note. n = 154.

Numbers and percentages in the Totals column reflect the total percentage of participants who chose this option. Numbers in boldface represent the most frequent common rank for that activity.

Table 4 illustrates that coursework, fieldwork, participation in professional activities (e.g., “Conferences”), mentorship, and self-directed learning are perceived to be the most significant educational experiences. The activities are listed in descending order according to frequency of selection (activities at the top of the table are more common choices than those at the bottom). Not only are coursework, fieldwork, and working with a mentor among the most commonly cited training experiences, the majority of respondents who choose these options also rank them as the most important contributing factor to their evaluation competency development.

A paired-comparison analysis was used to determine the relative contribution of four of these educational experiences (coursework, fieldwork, working with a mentor, and participation in professional activities) to the development of respondents’ competencies. Participants were presented with pairs (offering each educational experience against every other) and asked to indicate which made the most significant contribution to their competency development.

Scaling of Educational Experiences’ Contribution to Competency Development

Participants were asked whether or not they had engaged in each of the four categories of educational experiences. Only those who indicated participation in all four educational experiences (coursework, mentorship, fieldwork, and participation in professional activities) were able to continue on with the survey. Sixty-eight respondents (38% of the sample) met this criterion¹ and were further asked to specify the components of each educational experience. Descriptive tables for each experience can be found in Appendix A. Respondents were presented with all possible pairs of the four educational experiences (for a total of 6 pairings) and asked to indicate which of the pair made the most important contribution to the development of a particular competency.

The paired-comparison analysis allows us to depict the perceived contribution of the different educational experiences (on a scale from −1 to 1) for each evaluation competency. These scale values are constructed as follows. First, we note the frequency with which each educational experience is chosen over every other. We then convert this frequency to a proportion by dividing by number of judgments. These proportions are then converted to z-scores,² each of which represents the probability of choosing one educational experience over another (in terms of standard deviation). We then calculate the mean z-score for each educational experience. The values obtained represent the relative strength of the contributions of each educational experience to that evaluator competency.

Table 5 provides these data for one of the evaluation competences—contextual consideration—and displays (1) the number of times each educational experience is perceived to contribute more to the development of contextual consideration than the other experience listed in the column (F), (2) that number represented as a proportion (p in parentheses), (3) the tabled z-values for the proportions, and (4) the mean (scale value) for each educational experience. The scale values facilitate comparing the perceived contributions of each educational experience to the others—here, professional activities and coursework (with negative scale values) are believed to contribute less to the development of contextual consideration than mentorship and fieldwork (with positive scale values).

Table 5.

Frequencies, Proportions, and z-Scores Estimating Educational Experiences’ Contributions to the Development of Contextual Consideration.

Educational experience	Professional activities	Coursework	Mentorship	Fieldwork
Educational experience	F(p)	F(p)	F(p)	F(p)
Professional activities	34(0.500)	39(0.574)	50(0.735)	61(0.897)
Coursework	29(0.426)	34(0.500)	49(0.721)	57(0.838)
Mentorship	18(0.265)	19(0.279)	34(0.500)	46(0.676)
Fieldwork	7(0.103)	11(0.162)	22(0.324)	34(0.500)
z-Values
Professional activities	0.000	0.187	0.628	1.265
Coursework	−0.187	0.000	0.586	0.986
Mentorship	−0.628	−0.586	0.000	0.457
Fieldwork	−1.265	−0.986	−0.457	0.000
Sums	−2.080	−1.385	0.757	2.708
Means (scale values)	−0.520	−0.346	0.189	0.677

Note. N = 68.

Figure 1 plots the scale values for educational experiences according to evaluation competency. The educational experiences are found along the X-axis (in arbitrary order), and the Y-axis shows the z-score for the mean probability of choosing that educational experience over every other (the exact values, ranging from −1 to 1, are shown in the table below the graph). Each line on the graph represents a different competency (the lines are only added to the figure to help the reader trace each competency). Respondents perceive those educational experiences with positive scale values (above the horizontal line indicating 0 on the Y-axis) as contributing more to their competency development than those that appear below the horizontal line. For example, if we examine the contextual consideration competency, we see that individuals perceive participation in professional activities (z = −0.52) and coursework (z = −0.35) to contribute less to the competency development than mentorship (z = 0.19) and fieldwork (z = 0.68). Note that when summed, these z values total 0.

Figure 1.

Deviations from the scale mean of perceived contributions of educational experiences to evaluation competencies.

Based on the data presented in the graph, we can draw inferences about both the evaluator competencies and the educational experiences. Considering the competencies, the graph reveals two distinct patterns (represented by solid and dashed lines) describing the relative perceived contributions of educational experiences to competency development. These patterns divide the competencies along the lines of those that describe evaluator skills and those that describe evaluator knowledge. Regarding the skill-oriented competencies (contextual consideration, project management skills, and effective communication skills), represented by solid lines on the graph, we observe the same ordering of educational experiences along the scale of their perceived contribution; fieldwork is considered by respondents to be the greatest contributing experience, followed by mentorship, coursework, and participation in professional activities. In addition to this identical ranking, we see marked similarities between the calculated scale values of each of the educational competencies (indicated in the table below the graph). Survey respondents paint a very consistent picture—to develop contextual consideration, project management skills, and effective communication skills, evaluators need opportunities to participate in practical evaluation experiences in the field. Furthermore (but to a lesser extent), having experience working with a mentor will also aid in the development of these competencies. Coursework and participation in professional activities, on the other hand, are not considered as important when it comes to developing these areas.

There is a striking difference between the pattern described above and the ones we see for the knowledge-oriented competencies (methodological knowledge and theoretical knowledge). For these competencies, coursework is perceived to have the greatest impact on development. However, the ordering (in terms of perceived contribution to competency development) differs by competency for the remainder of the educational experiences. In the case of methodological knowledge, fieldwork follows coursework in the order of perceived contribution, and then is followed by mentorship and participation in professional activities. For the development of theoretical knowledge, mentorship is second, followed by participation in professional activities. Fieldwork is considered to contribute the least to the development of that competency.

The graph and data presented in the accompanying table can also guide a discussion about the different educational experiences. Participation in professional activities is the only educational experience with a negative scale value for all of the competencies. It also ranks lowest of all the other educational opportunities in developing four of the five competencies (the exception is the development of theoretical knowledge, for which fieldwork ranks lowest). However, this finding should not lead us to conclude that participation in professional activities (membership in AEA or other professional organizations, or attending workshops and conferences) has no value in developing evaluation competencies. There are potential alternate explanations for the observed response patterns regarding participation in professional activities. First, we may have seen participants respond more favorably to participation in professional activities, had competencies related to professional practice and/or reflective practice been included. This exclusion was purposeful (as explained above), but it is reasonable to hypothesize that were this competency included in a similar survey distributed to seasoned professionals, we would find greater value attributed to participation in professional activities. Furthermore, a flaw of this type of survey study is a limitation in exploring nuance. One survey respondent addressed this and offered the following statement, “I had a sense that there was something missing from the questions that didn’t adequately tap into the benefit for me of participation in professional associations.” A qualitative exploration into these educational experiences would supplement this study and offer greater insight into how participation in the professional community of evaluators can enhance competency development.

The response patterns regarding the relationship between coursework and the development of the various competencies confirm what was anticipated: The knowledge base for the technical tasks of conducting an evaluation can largely be acquired in the classroom. However, there is a belief in the field that suggests becoming a skilled evaluator is something greater than the sum of the technical training components (Schwandt, 2008). The findings about coursework lend some credibility to this assertion in that novice evaluators did not rate the contributions of coursework very highly when it came to the development of their contextual consideration, effective communication, and project management skills.

Like coursework, respondents’ ratings of fieldwork are widely discrepant based on which competency is being considered. Fieldwork, on the whole, is considered to contribute more to the development of evaluation competencies than any other educational experience. Furthermore, it is considered the most important educational experience for developing contextual consideration, project management skills, and effective communication. It ranks second in its contribution to the development of methodological knowledge; however, it ranked last when respondents considered its contribution to their theoretical knowledge. This last point is interesting in light of considerable calls issued in the field for increased research on the translation of evaluation theory into practice (Mark, 2008; Miller, 2010; Smith, 1993). There seems to be consensus that studying evaluation theory has implications for what happens in the field, but little attention given to understanding how work in the field might influence the ways evaluators think about evaluation theory. However, in an investigation of an informal evaluation discussion group, Christie and Rose (2003) discussed how “the play back and forth between the theory and the real-world leads to a reinvestigation, an advancement, and a refinement of the theoretical concept” (p. 239). Their work seems to suggest that the relationship between learning about evaluation theory and participating in fieldwork is a two-way street, but perhaps some sort of mediation is necessary to make that connection explicit.

Mentorship is an interesting case because fewer respondents indicated engaging in these experiences (being mentored by an advisor, other faculty or staff member, more advanced student, professional, or acting as a mentor to someone else) than in coursework, fieldwork, or participation in professional activities. Only 53% of survey respondents indicate participating in mentorship as a part of their evaluation training (compared to 88% who took courses, 78% who participated in professional activities, and 75% who engaged in fieldwork). This finding was unanticipated, and somewhat troubling, given that respondents who did have mentors rank it second among educational experiences according to perceived contribution to the development of all but one competency (methodological knowledge being the exception). Furthermore, mentorship was the only educational experience with a positive scale value (indicating greater importance to the development) for all competencies. Both in evaluation and beyond, mentorship is touted as an essential experience in graduate student development. Green and Bauer (1995) document the positive effects of mentoring in their study of engineering graduate students. Those who had mentors during their studies were found to be more productive and involved than those who did not. Given these findings, the low percentage of respondents in this survey who indicate participating in mentorship is cause for concern.

Implications for Evaluation Training

If, as a field, we are committed to this notion of evaluator competencies, we need to be more deliberate and intentional about the types of educational experiences provided to students. We lack both awareness of how educational experiences are currently offered through graduate education programs nationwide, and an accepted standard of how these opportunities should take form. The implications for the field are thus: Our training of evaluators through graduate programs would benefit from a more detailed and deliberate approach that intentionally incorporates a variety of educational experiences.

The findings presented in this article suggest that in order to adequately prepare evaluators, training should incorporate coursework, fieldwork, and mentorship. While it is likely that many graduate evaluation training programs offer all of these elements, the accepted minimum standard for defining such programs only includes coursework. LaVelle and Donaldson (2010) used as the inclusion criterion for their study, a definition of an evaluation training program as offering “two or more courses with the word ‘evaluation’ in the title” (p. 13)—the same inclusion criteria used in 1994 by Altschuld, Engle, Cullen, Kim, and Macce. Based on the findings presented here, offering two evaluation-specific courses is necessary, but not sufficient to ensure that evaluators are equipped with essential competencies to practice in the field. This is not to say that the programs included in these studies do not offer educational experiences beyond coursework, only that we do not know about these additional offerings, and have not yet sought to find out about them in a systematic way.

The findings presented here support the notion that we would benefit from more directed and deliberate articulation of necessary educational experiences outside the classroom. Perhaps, it is time to consider a model for formal evaluator training programs to ensure that degrees conferred are meaningful and adequately support future work in the field. Fortunately, models for both fieldwork and mentorship exist in other professional training programs to guide these discussions in the evaluation community.

Fieldwork

Outside of evaluation, many fields that provide graduate-level training to future practitioners include contractual work experiences. Examples of these types of programs include residency in medicine, internships in clinical psychology, and student teaching in teacher education. Shared by these fields, but lacking in evaluation are universal, minimum standards that ensure adequate completion of a field experience, although these standards are described and monitored in different ways for each discipline. In medical education, the institutions that sponsor medical residents are the object of accreditation by the Accreditation Council for Graduate Medical Education. The organization outlines policies governing selection and sponsorship of residents, the appropriate allocation of resources, and educational goals and support (Accreditation Council for Graduate Medical Education, 2007). Clinical psychology programs are accredited by the Office of Program Consultation and Accreditation of the American Psychological Association. The guiding principles for accreditation of programs in professional psychology outline specific requirements for graduate students’ internships including length of time in the field, appropriate partner organizations, articulated education and training objectives, and supervision requirements (Office of Program Consultation and Accreditation Education Directorate, 2007). Teacher education programs vary state by state, but there are several national voluntary accrediting bodies. One of these is the National Council for Accreditation of Teacher Education (NCATE)—recognized as a professional accrediting body by both the U.S. Department of Education and the Council for Higher Education Accreditation. NCATE outlines a framework for teacher candidates’ clinical practice that describes entry and exit requirements, knowledge demonstration, and skills application across several areas. This framework is further expanded according to specific content areas to include additional details relevant to particular subjects (National Council for Accreditation of Teacher Education, 2008).

The structured field experiences detailed above offer many examples that could be used to develop a fieldwork model in evaluation. Using the findings presented here as a guide, we could determine a model that supports fieldwork experiences through which participants gain skills for contextual consideration, project management, and effective communication. To design a fieldwork experience that adequately supports student learning in these areas, we would need to consider the appropriate length of time in the field, the breadth and depth of responsibilities, and the desired learning objectives. Additional research about what and how students learn through their work in the field is necessary to support these determinations. Furthermore, setting standards for fieldwork is incomplete without a discussion about supervision. Another commonality among the other disciplines that include fieldwork as a key component in their training is utilizing appropriate supervision to ensure successful experiences.

Mentorship

The anticipated benefits of fieldwork would not be achievable without supervision and guidance from a more experienced practitioner. It is common practice in many graduate education programs to pair students with an academic advisor. However, the role and responsibilities of a mentor outside the classroom and school-bound learning experiences is unclear in evaluation. The literature about mentorships in evaluation is largely case study–based, and almost makes good mentoring seem serendipitous rather than an intentional and directed activity.

Previous research has described mentors as knowledgeable and supportive; they model expectations, provide feedback, treat students as professionals, and share opportunities (Gredler & Johnston, 2001; Levin-Rozalis & Rosenstein, 2003). As is the case with fieldwork, good mentorship is something nearly universally accepted as a best practice in the field. However, the results presented in this study indicate that we may be underutilizing mentoring as an educational experience (given that only 53% of respondents indicate participating in mentoring activities as a part of their training). Even if mentorship is more widespread than these findings suggest, evaluation novices are not recognizing it as such. As in the case with fieldwork, more specificity in outlining mentorship will help us ensure the value of this educational experience to students.

These findings suggest that mentoring plays a crucial supporting role in developing evaluator competencies—it is considered the second most important educational experience in the development of all competencies with the exception of methodological knowledge, for which it is considered third most important. Given this, it is possible that the most powerful role a mentor can fill is to facilitate learning connections across students’ educational experiences. In order for this to be effective, mentors would need to work with students regarding both their coursework and their fieldwork. This could be accomplished in several ways. It may mean that we adopt a multiple mentorship model, a practice recommended in occupational therapy training (Nolinske, 1995), or that one mentor is responsible for assisting students to synthesize the learning obtained through a variety of experiences. Even if a particular mentor cannot be in the field with a student, the experiences in the field can provide fodder for conversations, and mentors can help students map the experiential knowledge that they gain back to the learning obtained in the classroom.

The conversation in the field about evaluator competencies is ongoing. Some scholars question their utility and wonder if it is truly possible to capture a definition of a competent evaluator through a list of skills and practices. In his article, Educating for Intelligent Belief in Evaluation, Schwandt (2008) expressed this sentiment thus, “We cannot simply teach about the ways and means of doing evaluation, we must also convey the idea of evaluation as a practical, intellectual disposition and outlook on social and political life” (p. 145). This caution is well taken, but it is possible to codify the ways we train evaluation students to ensure appropriate breadth and depth of educational experiences without jeopardizing the “intelligent belief” Schwandt espouses. Indeed, striking a balance between the intellectual and vocational aspects of evaluation should be a chief concern as we move forward in refining evaluation training programs. But we must take steps to be more deliberate about the educational opportunities we are providing to evaluation students if we are to foster these qualities and competencies in our practitioners.

Footnotes

Appendix A

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Notes

References

Accreditation Council for Graduate Medical Education. (2007). Common program requirements. Retrieved from www.acgme.org/acWebsite/dutyHours/dh_dutyhoursCommonPR07012007.pdf

Altschuld

J. W.

(1999). The case for a voluntary system for credentialing evaluators. American Journal of Evaluation, 20, 507–517.

Altschuld

J. W.

Engle

Cullen

Kim

Macce

B. R.

(1994). The 1994 directory of evaluation training programs. New Directions for Program Evaluation, 62, 71–94.

American Evaluation Association, Task Force on Guiding Principles for Evaluators. (1995). Guiding principles for evaluators. New Directions for Program Evaluation, 66, 19–26.

Christie

C. A.

Rose

(2003). Learning about evaluation through dialogue: Lessons from an informal discussion group. American Journal of Evaluation, 24, 235–243.

Davis

B. G.

(1986). Overview of the teaching of evaluation across the disciplines. New Directions for Program Evaluation, 29, 5–14.

Dewey

J. D.

Montrosse

B. E.

Schroter

D. C.

Sullins

C. D.

Mattox

J. R.

(2008). Evaluator competencies: What’s taught versus what’s sought. American Journal of Evaluation, 29, 268–287.

Edwards

A. L.

(1957). Techniques of attitude scale construction. New York, NY: Appleton-Century-Crofts.

Engle

Altschuld

J. W.

Kim

(2006). 2002 Survey of evaluation preparation programs in universities: An update of the 1992 American evaluation association-sponsored study. American Journal of Evaluation, 27, 353–359.

10.

Fitzpatrick

(1994). Alternative models for the structuring of professional preparation programs. New Directions for Program Evaluation, 62, 41–50.

11.

Ghere

King

J. A.

Stevahn

Minnema

(2006). A professional development unit for reflecting on program evaluator competencies. American Journal of Evaluation, 27, 108–123.

12.

Gredler

M. E.

Johnson

R. L.

(2001). Lessons learned from the directed evaluation experience. American Journal of Evaluation, 22, 97–104.

13.

Green

S. G.

Bauer

T. A.

(1995). Supervisory mentoring by advisers: Relationships with doctoral student potential, productivity and commitment. Personnel Psychology, 48, 537–562.

14.

LaVelle

J. M.

Donaldson

S. I.

(2010). University-based evaluation training programs in the United States 1980–2008: An empirical examination. American Journal of Evaluation, 31, 9–23.

15.

Levin-Rozalis

Rosenstein

(2003). A mentoring approach to the one-year evaluation course. American Journal of Evaluation, 24, 245–259.

16.

Mark

M. M.

(2008). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In Smith

N. L.

Brandon

(Eds.), Fundamental issues in evaluation (pp. 111–134). New York, NY: Guilford Press.

17.

Mertens

D. M.

(1994). Training evaluators: Unique skills and knowledge. New Directions for Program Evaluation, 62, 17–27.

18.

Miller

R. L.

(2010) Developing standards for empirical examinations of evaluation theory. American Journal of Evaluation, 31, 390–399.

19.

National Council for Accreditation of Teacher Education. (2008). Professional standards for the accreditation of teacher preparation institutions. Retrieved from www.ncate.org/Portals/0/documents/Standards/NCATE%20Standards%202008.pdf

20.

Nolinske

(1995). Multiple mentoring relationships facilitate learning during fieldwork. The American Journal of Occupational Therapy, 49, 39–43.

21.

Office of Program Consultation and Accreditation Education Directorate, American Psychological Association. (2007). Guidelines and principles for accreditation of programs in professional psychology. Retrieved from www.apa.org/ed/accreditation/about/policies/guiding-principles.pdf

22.

Schwandt

T. A.

(2008). Educating for intelligent belief in evaluation. American Journal of Evaluation, 29, 140–151.

23.

Smith

N. L.

(1993). Improving evaluation theory through the empirical study of evaluation practice. Evaluation Practice, 14, 237–242.

24.

Stevahn

King

J. A.

Ghere

Minnema

(2005). Establishing essential competencies for program evaluators. American Journal of Evaluation, 26, 43–59.

25.

Stufflebeam

D. L.

(2001). Interdisciplinary Ph.D. programming in evaluation. American Journal of Evaluation, 22, 445–455.

26.

Yarbrough

D. B.

Shulha

L. M.

Hopson

R. K.

Caruthers

F. A.

(2011). The program evaluation standards: A guide for evaluators and evaluation users (3rd ed.). Thousand Oaks, CA: Sage.