Arguments for a Common Set of Principles for Collaborative Inquiry in Evaluation

Abstract

In this article, we critique two recent theoretical developments about collaborative inquiry in evaluation—using logic models as a means to understand theory, and efforts to compartmentalize versions of collaborative inquiry into discrete genres—as a basis for considering future direction for the field. We argue that collaborative inquiry in evaluation is about relationships between trained evaluation specialists and nonevaluator stakeholders (i.e., members of the program community, intended program beneficiaries, or other persons with an interest in the program) and that practice should, in the first instance, be sensitive to stakeholder interests and context, and it should be principle-driven.

Keywords

collaborative inquiry program evaluation complexity theory principles of practice

The Problem

Collaborative approaches in evaluation are increasingly recognized as important in the development evaluation as well as the evaluation contexts in developed countries. For example, recent critiques of development evaluation have pointed to the overrepresentation of donor interests, as opposed to the interests of local program communities and the privileging of the accountability function of evaluation (e.g., Carden, 2010; Hay, 2010). Adherence to conventional, evaluation designs and approaches to impact evaluation that tend to privilege rigor over the importance of context (e.g., randomized control trials [RCTs] and quasi-experimental designs) has been framed as potentially disruptive to program development and implementation and oppressive with regard to the learning function of evaluation (Dahler-Larsen, 2009; Preskill, 2008).

Recent think tank sessions at American and European evaluation professional meetings have led to the identification of a variety of alternative approaches to the statistical counterfactual in the context of impact evaluation (e.g., Rugh, Steinke, Cousins, & Bamberger, 2010). The importance of understanding context for programs and interventions has been underlined as being essential to such evaluation. Similar arguments derive from developments in complexity theory as applied to evaluation (see, e.g., Patton, 2010; Ramalingam & Jones, 2008; Rihani, 2002; Woodhill, 2007).

Some of the alternative modes of inquiry that were identified by Rugh et al. (2010; e.g., most significant change technique and contribution analysis) lend themselves quite readily to participatory and collaborative approaches to understanding program effects. In this light, some would argue—and we count ourselves among them—that learning from evaluation, a recognized strength of collaborative approaches, is a form of accountability in and of itself.

In North America, participatory and collaborative approaches to evaluation are the most often used approach for cross-cultural inquiry associated with interventions for aboriginal and indigenous peoples (Brant-Castellano, 1986; Chouinard & Cousins, 2007; Hoare, Levy, & Robinson, 1993; Jackson, McCaskill, & Hall, 1982). There is little doubt that such approaches are well suited to cross-cultural evaluation contexts as opposed to more traditional social sciences approaches. As an example, Cousins et al. (2010) integrated participatory evaluation (PE) with an approach similar to the most significant change technique (Davies & Dart, 2005; Serrat, 2009) in a multiple case study design investigating the effects of a national Canadian strategy for aboriginal youth suicide prevention.

With evident growth in the popularity and credibility of collaborative approaches to inquiry, it is increasingly important to consider ways to advance the field along theoretical and practical lines. Along with many others, we have been actively engaged in such pursuits. Yet we find that some recent developments in this respect may not always be as productive as initially thought. In this article, we review and critique two such developments: using program logic models to represent collaborative evaluation theory and compartmentalizing collaborative approaches. Following a review and comment on these developments, we propose some ideas about the development of fundamental principles for collaborative inquiry, principles that could serve as guidelines to practice. But first we provide some thoughts about the evolutionary trajectory of collaborative inquiry in our field.

Background

Collaborative inquiry in evaluation traces its origins to development evaluation, research, and theory of over 30 years ago (Whitmore, 1998a). An international participatory research network was set up in the 1970s, with headquarters in India, and the first of a series of major international seminars was held in Tanzania in 1979 (Hall, 1975, 1981, 1992; Kassam & Mustapha, 1982). Rapid rural appraisal (RRA; Chambers, 1981), participatory research, and participatory action research (PAR) are all forerunners of what has come to be known as PE taking place in Latin America (Alcocer et al., 1997; Fals-Borda, 1981, 1992, 2006), Asia (Armonia & Campilan, 1997; Fernandes & Tandon, 1981; Tandon, 1981, 2008), Africa (Kassam & Mustafa, 1982; PAMFORK, 1997), and elsewhere (Campos & Coupal, 1996; Estrella & Gaventa, 1998; Feuerstein, 1986; Jackson & Kassam, 1998; Narayan, 1994). These alternative approaches developed at least in part as a reaction to positivist models of inquiry that were seen as exploitive and detached from urgent social and economic problems.

In 1993, David Fetterman introduced “Empowerment Evaluation” to the North American evaluation community in his presidential address at the annual conference of the American Evaluation Association (AEA) in Dallas (Fetterman, 1994). Given the radical departure of empowerment evaluation from more conventional evaluation approaches, this proposed new direction for evaluation was as courageous as it was thought provoking. Perhaps, not surprisingly, empowerment evaluation met with stiff opposition from evaluation luminaries including Stufflebeam (1994) and Scriven (1997). Not long after Fetterman’s address, an AEA Topical Interest Group was created called Collaborative, Participatory and Empowerment Evaluation and its membership has been growing ever since.¹ Also in the early to mid-90s we started to publish papers about a participatory approach to evaluation, which had as its goals program problem solving and improvement and the enhancement of evaluation use (Cousins & Earl, 1992, 1995; Shulha & Wilson, 1995).

Coupled with the earlier work taking place in international development contexts, the menu of choices with respect to collaborative modes of applied research and evaluation was growing rapidly. This diversity of activity, and attendant conceptual overlap and confusion, prompted us to think about fundamental theoretical aspects of collaborative inquiry, which we ultimately published as a chapter in New Directions for Evaluation titled “Framing Participatory Evaluation” (Cousins & Whitmore, 1998). That paper, as it turns out, was widely received and ultimately selected to be reprinted in a 20-year anniversary issue of AEA’s New Directions for Evaluation (Matheson, 2007).

We would argue that there were two important contributions in that paper. First, we differentiated among forms of collaborative inquiry in evaluation with different ideological interests. On one hand, there is transformative PE (T-PE), an approach that is normative in form and function and associated with such ideals as emancipation, empowerment, and self-determination. On the other hand, there is practical PE (P-PE), a pragmatic problem-solving approach to evaluation that promotes the use of evaluation findings and process. While T-PE is predominantly grounded in a political justification and P-PE in pragmatic considerations, the two are not mutually exclusive; and they both embrace a third, philosophical rationale that is about the development of deeper levels of meaning through collaboration. The two streams represent different emphases or desired end points and are therefore at different points on a common continuum.

As a second important contribution, we built on earlier work (Cousins & Earl, 1992) to provide a tool (see Figure 1) for differentiating alternative collaborative approaches along three fundamental dimensions: control of technical evaluation decisions (evaluator vs. stakeholder), stakeholder selection for participation (diverse group vs. homogeneous primary user group), and depth of participation by stakeholders (participation in selected aspects vs. comprehensive participation in technical evaluative processes). The idea is that any given collaborative inquiry project, at any given point in time, could be described by locating its coordinates it in three-dimensional space (see Figure 1).

Figure 1.

Dimensions of form in collaborative inquiry (from Cousins & Chouinard, 2012).

As we had hoped, the three-dimensional process framework became the focus for ongoing analysis, discussion, and debate with colleagues and students. Eventually, we came to the view that one of these dimensions—stakeholder selection—was confounded and therefore conceptually inadequate (Weaver & Cousins, 2004). We ultimately teased apart this dimension into three distinct dimensions of form or inquiry processes. The resulting framework consisted of five dimensions of form: we unpacked stakeholder selection into three more parsimonious dimensions—manageability (manageable vs. unmanageable), power relations (conflicted vs. neutral), and stakeholder selection (diverse vs. homogenous)—and added them to existing dimensions of control of technical decision making and depth of participation. Figure 2 details the specific elements of the framework, which we argued ameliorated the aforementioned confound.

Figure 2.

Five dimensions of form in collaborative inquiry (from Weaver & Cousins, 2004).

Again, any given collaborative inquiry, at any given point in time can be described according to its location on each of the five dimensions. The results can be displayed using a “radar-gram” to show variation across the five dimensions. Figure 3 shows hypothetical differentiation among P-PE, T-PE, and more conventional stakeholder-based evaluation. We have since used this reformulation to describe and compare collaborative evaluation projects (e.g., Cousins, 2005).

Figure 3.

Hypothetical dimensions of form in collaborative inquiry (adapted from Weaver & Cousins, 2004).

Some years later, Daigneault and Jacob (2009) took issue with this development and built a solid argument as to why the original formulation was conceptually superior. We are persuaded by their argument, which goes something like this: generally in the field, PE is conceptually sloppy stacked up against the eight criteria for evaluating the functionality of a concept offered by Gerring (1999 cited by Daigneault & Jacob, 2009). The most serious shortcomings are parsimony, internal coherence, and external differentiation (i.e., effectively differentiating between participatory and nonparticipatory inquiry). According to the authors, the three dimensions put forward by Cousins and Whitmore (1998) “. . . happen to be PE’s fundamental attributes or constitutive dimensions” (p. 334). They are all necessary conditions for PE and jointly sufficient for membership in this category. Other dimensions such as Weaver and Cousins’ (2004) “manageability” and “power relations” add value through augmenting analytic power but according to Daigneault and Jacob “. . . they do not constitute the necessary attributes of PE” (p. 336). Manageability and power relations do not define PE because they can apply equally to nonparticipatory approaches such as conventional technocratic program evaluations. Control of technical decision making, stakeholder selection and depth of participation are therefore seen as constitutive of PE. The authors also argue that the three dimensions add an impressive degree of internal coherence and are able to successfully differentiate participatory and nonparticipatory inquiry (i.e., external differentiation). Further, they can accommodate a wide range of approaches to collaborative inquiry including empowerment evaluation, democratic evaluation, and fourth-generation evaluation.

Based on this analysis, Daigneault and Jacob (2009) provide an approach to measuring PE that seems meritorious and worthy of empirical application in research on collaborative inquiry. We like these ideas, but perhaps what we take first and foremost from their careful critique and analysis is that “we had it right the first time.” While attributes such as manageability and power relations are important considerations about PE—as are others such as multiplicity in methods or capacity building (Burke, 1998)—they are not fundamental to the conceptualization of collaborative inquiry.

Since that time, as foreshadowed above, two separate streams of activity have prompted us to conclude that it may be time to pause and perhaps rethink new directions for collaborative inquiry in evaluation in the interest of advancing the field. We now turn to our review and critique of recent efforts to (i) represent theory in collaborative inquiry using logic models and (ii) compartmentalize different families or genres of collaborative inquiry.

Contemplating the Logic of Logic Models

Marv Alkin has contributed to the development and understanding of theory in evaluation perhaps as much as or more than anyone in the field. Among his many accomplishments, he sponsored and captured “debates in evaluation” among noted contributors (Alkin, 1990), provided a 25-year retrospective on theory in evaluation (Alkin, 1991), and more recently published Evaluation Roots (Alkin, 2004, in press) an edited volume that features the evaluation theory tree, a conceptual framework codeveloped with Tina Christie (Alkin & Christie, 2004).

At the 2007 meeting of AEA in Denver, Alkin invited one of us (Cousins) to participate in a panel session that involved other contributors and some of Marv’s graduate students (Hansen, Luskin, and Wallace). The focus for the session was a series of draft logic models generated by the students that corresponded to respective “evaluation theories:” P-PE (Cousins), transformative evaluation (Mertens), utilization-focused evaluation (UFE; Patton), and empowerment evaluation (Fetterman). In the session, the students presented the models and then invited reaction from the respective theory proponents. The initial idea led to further work and development on the part of Alkin and his students. Papers were drafted for a session at AEA in 2010, comparing and contrasting various theoretical perspectives using visual representations (Hansen & Wallace, 2010; Luskin, 2010). This time the focus was on P-PE (Cousins), value-engaged evaluation (Greene), and emergent realist evaluation (Mark, Henry, and Julnes). Ultimately, products from this line of inquiry, including discussant and commentary articles, will appear as a special issue (Alkin, Vo, & Hansen, in press).

At the core of it, this work is motivated by an interest in understanding the similarities and differences among various evaluation approaches or theories using logic models as a basic visual representation. Through using such representations, salient features of respective approaches can be identified, thereby illuminating the anticipated relations between evaluators and stakeholders and assisting evaluation practitioners who choose to follow a particular theorist’s prescriptions (Hansen & Wallace, 2010). Five basic elements are depicted: assumptions, evaluation context, evaluation activities, evaluation consequences/effects, and external factors (Hansen, Alkin, & Wallace, in press). The authors assert that through operational specificity, the approach may assist evaluation practitioners with implementation, aid in the identification of theoretical deficiencies, and facilitate research on evaluation.

A separate, parallel initiative was recently undertaken by Harner (2012) in his doctoral research. Harner had a similar objective to the UCLA group but focused exclusively on developing a model of TP-E and relied on the collection of original data from evaluation theorists and practitioners. His study employed a multiple methods, cascading design and generated a logic model representation of T-PE that elaborated principles, actions, and outcomes of the process. The design permitted him to differentiate the relative weights of variables associated with T-PE practice.

According to Miller (in press) in her commentary on the UCLA papers to be published in the special issue, the use of a standardized approach to develop logic models of evaluation theories provides a unique way to represent each theory’s relative emphasis on its ideological, operational, and intervention components. These are the elements or sublogics of evaluation theory offered by Smith (2010, cited by Miller, in press). P-PE as an example, when compared with value-engaged and emergent realist theories, turns out to most closely align with the interventionist theoretical element. Miller elaborated other potential benefits of the logic modeling approach such as facilitating comparative analysis among theories, illuminating training needs for evaluators, and exposing conceptual inconsistencies or paradoxes within theories. It could be argued that Miller’s analysis would apply to Harner’s (2012) results as well.

We understand the merit and the potential benefits of such approaches to visualizing theory. Yet at some level, with respect to collaborative inquiry, we have discomfort with this line of inquiry. That the approach centers on the conceptualization of PE as “a theory” or “a model” reflects one of our chief concerns. We have long taken the position that PE is an alternative “approach” to evaluation. Studies that attempt to codify P-PE, T-PE, or any other approach to collaborative inquiry in evaluation as a logic model run the risk of misrepresenting them as something less fluid and flexible, and perhaps more prescriptive, than they were ever intended to be. In our opinion, the interests and forms of collaborative inquiry are best informed by the contextual exigencies emerging from the community of program practice and the information needs of various actors associated with the program. Such considerations ultimately shape, through negotiation, the expressed purposes and form that the inquiry will take. For example, in contexts where there is a relatively high degree of agreement about the goals of an intervention and that evaluation can assist with the development and improvement of the program, a P-PE approach that focuses on balanced control, limited stakeholder diversity, and substantial depth of participation might be warranted (see, e.g., Cousins & Shulha, 2008). Likewise, through involving diverse groups of stakeholders in evaluation knowledge production, T-PE might assist in developing community capacity for inquiry and a sense of self-determination in contexts where agreement about goals is not high or where capacity for collective understanding and problem solving is constrained (see, e.g., Kar & Chambers, 2008). But these collaborative inquiry processes, in our mind, are up for negotiation with the relevant and interested members of the program community.

In her remarks, Miller (in press) laid out similar concerns. She suggested that the logic modeling approach runs the risks of underrepresenting context, marginalizing cultural and cross-cultural considerations, and privileging a mechanistic representation to the detriment of capturing the dynamic character of evaluation practice. In our view, context, culture, and dynamic negotiated practice are part and parcel of collaborative inquiry in evaluation. As such, Miller’s concerns apply to efforts to develop visual representations of evaluation theory with respect to collaborative inquiry in evaluation. Having said that, we do acknowledge Alkin’s point that evaluation theories represented visually are ideals and their application in practice will be very much influenced by context.

Yes, we do portray the ideal or most likely mode of each theory in our logic models. As you know from my writings, I am very concerned about context and understand that context shapes and modifies that ideal approach (personal communication, Marvin Alkin, November 15, 2011).

Nonetheless, we remain reluctant to commit to an understanding of P-PE or T-PE as a theory or model, as opposed to an approach and we think of context very much as a starting point for defining the practical application of collaborative inquiry in evaluation. Miller’s concerns are also quite relevant to another stream of activity that gives us pause, namely, efforts to compartmentalize collaborative inquiry in evaluation.

Rethinking Compartmentalization

At the 2009 and 2010 annual AEA conferences, the Collaborative, Participatory and Empowerment (CPE) Topical Interest Group (TIG) has openly addressed the question of how best to differentiate among these three family members. One of us (Shulha) has been directly involved in, and become increasingly more uncomfortable with, these discussions. At the 2010 meeting, the group appeared to be reaching consensus about differentiating these approaches (personal communication with David Fetterman and Abe Wandersman, November, 2010) mostly along the lines of one of our dimensions of form in collaborative inquiry, technical control. These ideas continue to move forward, most recently in a March 2011 series of webinars sponsored by the CPE-TIG (Fetterman & Wandersman, 2011; Rodriguez-Campos & O’Sullivan, 2011; Zukoski & Luluqisen, 2011) and in articles published as part of a special issue of a peer reviewed journal (O’Sullivan, 2012; Rodriguez-Campos, 2012).

Collaborative evaluation, it is argued by CPE-TIG proponents, is about evaluators in leadership roles working with stakeholders to produce evaluative knowledge. The evaluator is “in charge” and the stakeholder participant’s role is to engage with other participants and the evaluator. As recently put by Rodriguez-Campos (2012),

[Evaluators] also contribute to a genuinely collaborative atmosphere, i.e. one in which everyone feels represented in an appropriate and fair way. Even though evaluators are in charge of the collaborative evaluation, they create an ongoing engagement between evaluators and stakeholders. (p. 227, our emphasis)

Identifiable benefits of collaborative evaluation are strong evaluation designs, enhanced data collection, and analysis and results that stakeholders understand and use (Rodriguez-Campos & O’Sullivan, 2011; O’Sullivan, 2012). Books by O’Sullivan (2004) and Rodriguez-Campos (2005) are identified as exemplars of the approach.

Empowerment evaluation (Fetterman, 2001; Fetterman & Wandersman, 2005), on the other hand, is located at the other end of the control spectrum. It is about divesting control of technical decision making to stakeholders at the outset. “People take charge of their evaluation with the assistance of an empowerment evaluator” (Fetterman & Wandersman, 2011). The benefits associated with empowerment evaluation are capacity building, the production of measurable outcomes, contributions to sustainability, and enhanced knowledge utilization.

The representation of the third member of the CPE-TIG family, PE, remains somewhat of an enigma to us. What sets this approach apart, we thought we understood, was that control begins with the evaluator but is divested to program community members over time and with experience (personal communication with David Fetterman and Abe Wandersman, November, 2010), yet no mention of this is made in the webinar handout (Zukoski & Luluqisen, 2011), although the authors support a balance of control shared between evaluators and stakeholders, which does differentiate it from the other CPE family members. The PE webinar handout seems to more or less privilege PE as an overarching framework with practical and transformative thrusts. For example, it lists such “methods” as empowerment evaluation, PAR, and UFE as part of its repertoire. The rationale for PE, according to the webinar leaders, is long, varied, and really quite all encompassing: identify locally relevant questions; improve program performance; empower participants; build capacity; develop leaders and build teams; sustain organizational learning and growth; transform and improve programs; and use findings to create action plans to make improvements.

Contrary to our colleagues in CPE-TIG, we prefer to frame collaborative evaluation or collaborative inquiry in evaluation as an umbrella term that encompasses such approaches as PE, transformative evaluation, empowerment evaluation, democratic deliberative evaluation, PAR, and fourth-generation evaluation. But this is not our central concern with the business of compartmentalization exercises; rather, we are unclear as to the principal motivations for this activity. Why is it important to have sharp distinctions among these approaches and to whose benefit? Possibly it is because some evaluators self-identify with particular approaches. If this is so, does it imply that “empowerment evaluators” or “collaborative evaluators” bring their specific approach with them—3-step, 10-step, and step-by-step—to each opportunity for collaborative inquiry in evaluation? Again, we underscore that it is the exigencies of context and the information needs of program community members that ought to shape the inquiry. What comes out in terms of what the evaluation practitioner actually does will be different, as any given context will demand, and negotiation is essential. To that end, going in with preconceived notions about what to do, and hows to do it inevitably carries with it some risks. For example, in related discussion Donna Mertens commented:

I am sure I would not agree that turning control over to stakeholders would be a transformative participatory approach; it could be chaos. There needs to be a partnership rather than a relinquishing of responsibility on the part of the evaluator. (personal communication, December 2010)

We share the concern. In a context that does not warrant such an approach, it runs the risk of setting people/stakeholders up with false expectations and ultimately may end up doing more harm than good, however unintended. It is essential that evaluators engaging in any form of collaborative inquiry seriously consider the potential for unintended effects and their implications for programs and communities (Whitmore, 1998b).

Reconsidering Direction for Collaborative Inquiry in Evaluation

To this point we have reviewed two distinct developments in the field concerning the theory and practice of collaborative inquiry in evaluation. Our conclusions are recapped as follows: First, we recognize the merit and potential benefits associated with using visual representations to understand and compare theories of evaluation. Yet we have concerns that portraying collaborative evaluative inquiry as a theoretical model rather than an approach represents it as being more rigid and pre-ordinate and less dynamic and flexible than it was ever intended to be, remembering that the complexities of any given context cannot be adequately captured in such a modeling process. Second, we find an investment in compartmentalizing genres of collaborative, participatory, and empowerment evaluation unwarranted and ultimately unproductive. We fail to understand the utility of this direction as a means of advancing theory and practice in the field.

Our conclusions lead us to pause and rethink future direction for the field. We conclude this article with some thoughts about the essential aspects of collaborative inquiry in evaluation and suggested strategies to move toward a principle-driven conception of practice.

The Essential Aspects of Collaborative Inquiry in Evaluation

We recently considered the essential aspects of PE (Shulha, 2010), which apply more generally to collaborative inquiry in evaluation. Thinking about the latter as a class of approaches rather than as methods, models, or techniques allows evaluators to foreground the purpose, people, and context of their collaborative work. In our view, this must be the starting point for any collaborative process.

Considerations of purpose help us to understand the call—either implicitly or explicitly—for collaborative inquiry in the first place. What are the information needs of the program community and to what extent do they vary? How has context shaped those information needs? How can evaluation, and specifically collaborative approaches, help stakeholders to meet identified needs? What are the advantages of the collaborative approach over others for meeting the needs? Purposes may vary widely both within and across stakeholder groups. To whose interests should the inquiry attend and why? The call for collaborative inquiry is mediated by context and this implies social, historical, ecological, and cultural complexities. Collaboration must therefore be negotiated between evaluators and members of the program community, broadly defined, if collaborative inquiry in evaluation is to be meaningful, productive, and healthy for communities of program practice.

As a platform for negotiation we rely on our primary justifications for collaborative inquiry, described above. Political, pragmatic, and philosophical rationales for collaborative inquiry are not mutually exclusive but they align in varying degrees with stakeholder predispositions and ultimately shape the collaborative process and desired consequences toward either more practical (e.g., P-PE) or transformative (e.g., T-PE) ends. Practical motivations would include meeting demands for both accountability for responsible action and learning for improvement and positive change. Transformative interests include, for example, capacity building through developing evaluation habits of mind, questioning assumptions, and challenging the status quo (process use). That collaborative evaluative inquiry can generate new knowledge and insights and can contribute directly to ongoing developmental decisions and processes are other potential benefits and drivers that may shape the inquiry.

Figure 4 helps us to grasp the essential processes of negotiation and relationship building. It is through dialogue and deliberation among evaluators and stakeholder communities that the complexities of context are understood and that the principal drivers for collaborative inquiry are identified, critiqued, and clarified. It is this process that will ultimately shape the inquiry and set the stage for deciding control, diversity of participation, and stakeholder engagement with the inquiry.

Figure 4.

Essential features of collaborative inquiry in evaluation (adapted from Shulha, 2010).

Understanding and embracing the concepts in Figure 4 and the relationships among them help us move beyond concerns about codifying, describing, or differentiating specific forms of collaborative inquiry and into a space where the ultimate form and shape of participation is principle-driven, not method-, model-, or specific approach-driven. It is in this direction that we would choose to invest in the interests of advancing the field.

Figure 4 represents the program context as the ever-present filter through which subsequent activities and decisions flow. Understanding the context in which we work is central to what we do, why we do what we do and how, or the methods we use. As a context changes so will the appropriateness of decisions around the purpose and form of collaboration. The Cynefin framework (Snowdon & Boone, 2007) can be helpful in clarifying our understanding of the contexts in which we operate. “Cynefin” is a Welsh word (pronounced ‘coon-ev’in) meaning “habitat, acquainted, accustomed or familiar, being both noun and adjective, and thus requiring context to understand its meaning in any given instance” (Patton, 2010, p. 106). Snowdon and Boone (2007) developed this to guide organizational decision making and Patton (2010) has adapted it to the evaluation context. In Figure 5, we draw upon both in the interest of informing our thinking about collaborative approaches to inquiry.

Figure 5.

A modified Cynefin framework (Adapted from Snowdon & Boone, 2007 and Patton, 2010).

As we can see in the lower right hand sector of Figure 5, In simple situations, evaluation can be quite straightforward; for example, one measures the difference before and after an intervention. A controlled, predictable environment is assumed and clear cause and effect relationships are relatively easily discernible. There is a high degree of agreement about what the problem is and what should be the path to the solution. The evaluator assesses the facts of a situation, categorizes them, and draws conclusions based on established practice. “Best practices” are established and it is assumed that what worked in one context will work in another.

In a complicated context (see Figure 5 upper right sector), there may be more than one right answer and though there may be a clear cause and effect relationship, it may not be entirely evident (Snowdon & Boone, 2007). An evaluator must analyze a situation and look at the pros and cons of different options or possibilities. The context is not as controllable as in a simple situation, but it does nonetheless have some degree of predictability. Good practice rather than best practices is more appropriate here.

Complex situations are unpredictable and in constant flux. As shown in the upper left sector of Figure 5, Data are incomplete, there is no right answer; rather, over time, patterns can be discerned and a path forward emerges. In complex contexts, there are many opportunities for creativity and innovation; therefore, instead of attempting to impose a given method or draw conclusions too quickly, evaluation practice remains receptive to the unanticipated. Evaluators focus on identifying the initial conditions, monitoring and documenting what emerges, providing timely feedback, facilitating reflective practice among stakeholders, and embedding evaluative thinking in the process (Patton, 2010).

In chaotic situations (see Figure 5 lower left sector), searching for right answers is pointless. There is no time for input; someone must take charge and decide what to do. The most important thing is to act immediately to “stop the bleeding” and establish some kind of order. An evaluator has a limited role here, other than to make immediate recommendations or even decisions, if leadership is otherwise lacking (Snowdon & Boone 2007).

The multiple and evolving contexts in which evaluators work are more often than not complex; that is, they are dynamic, constantly changing, and unpredictable; what Zimmerman (2000) characterizes as inherent in “the messiness of real life.” Given this, it is far more productive to embrace the uncertainty, and rather than trying to control a situation, to seek out the unexpected, or what Guijt (2008) calls “surprises,” as sources of learning. The key in complexity thinking is that it invites us to change the metaphor from “systems as machines” to “systems as living entities” (Zimmerman, 2000). In the field of evaluation, Patton regards complexity as “the great unexplored frontier” (Patton, 2010).

Complex systems do have patterns of behavior that appear stable within a larger stream of organizing activities. What is lost in focusing on the regularity of program patterns is how the individuals who give these patterns their meaning are continuously addressing pressures that emerge within and around them—essentially learning (Davis, Sumara, & Luce Kapler, 2008). When evaluators enter such systems they will, to a greater or lesser degree, cause a disturbance in the stream if not in one or more program patterns. Evaluators who are sensitive to the adaptations that arise in response to their presence and demonstrate their own willingness to learn appear better positioned to understand the contexts in which they are operating, the optimal purposes for evaluation, and the deliberations necessary to yield an appropriate form of collaboration.

Toward a Coherent Set of Principles for Collaborative Inquiry in Evaluation

We conclude with sentiments that the field would be best served by serious work to develop principles of practice that allow ample flexibility to do what seems best given diversity in stakeholder interests, contextual complexity, cultural diversity, evaluator–stakeholder relations, and the like. In short, collaborative inquiry in evaluation is about approaches that should remain dynamic and adaptable to the exigencies of the evaluation context.

We acknowledge that there are prior attempts to consider or develop principles of practice in this regard. Cousins and Whitmore (1998), for example, identified several categories of ideas that ought to be taken into account in collaborative inquiry, ideas that could easily serve as clues to principles of practice. The categories were power relations and their ramifications, ethics, participant selection, technical quality, cross-cultural issues, training, and conditions enabling PE. Fetterman and Wandersman (2005) tackled the problem in a more direct way. Through a serious process of consultation, dialogue, and deliberation they generated a list of 10 principles. The process was laudable. But, despite their overt commitment to empowerment evaluation and the fostering of self-determination and transformative ends, the scope of the resulting set of principles extended well beyond this perspective. A review reveals that many would be appropriate in contexts where empowerment was not the driving force for evaluation. In addition, simultaneous adherence to all of the principles is most likely impossible.

Yet a directed approach, of the sort used by Fetterman and Wandersman is what is required. What is needed now, in our view, is a collaborative developmental process, involving evaluators and stakeholders to establish a working set of principles for collaborative inquiry as one holistic approach in evaluation. The principles would not be written in stone but rather they would be the subject of continuous analysis and renewal through dialogue and systematic inquiry. The strategies for developing these principles need to be fully and openly collaborative (we need to walk the talk) engaging diverse groups of people in an open and democratic, dialogic process. Moreover, we would propose that a set of working principles be subject to field testing and inquiry and that such inquiry should be, in and of itself, collaborative.

Conclusion

The delineation of a specific set of principle-development strategies is beyond the scope of this article, and it would not appropriately reflect the collaborative process we advocate. But, in addition to a reconsideration of the essential aspects of collaborative inquiry, this general direction is what we would offer as the article’s main contribution. To begin the conversation, we initiated a dialogic process with a think tank at the 2011 AEA conference. Follow-up conversations and perhaps even systematic inquiry are emerging from the initial think tank.

We have criticized recent developments in the field that are intended to advance theory and practice in collaborative inquiry in evaluation. Our intention has been to inform an agenda for future direction for the field. It is our hope that the ideas presented here will at least engender ongoing constructive discussion and dialogue.

Footnotes

Authors’ Notes

A previous version of this article was presented at the Global Assembly of the International Development Evaluation Association (IDEAS), Amman, Jordan, April 2011. The authors thank Nathalie Gilbert for her assistance with the article.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Note

References

Alcocer

Lizárraga

Delgadillo

. (1997). A survey of the practice of participatory monitoring and evaluation methods in Bolivia. Paper prepared for the International Workshop on participatory monitoring and evaluation: Experiences and lessons. Cavite, Philippines.

Alkin

M. C.

(1990). Debates on evaluation. Thousand Oaks, CA: Sage.

Alkin

M. C.

(1991). Evaluation theory. In McLaughlin

Phillips

(Eds.), Evaluation at quarter century. Chicago, IL: University of Chicago Press.

Alkin

M. C.

(Ed.). (2004). Evaluation roots: Tracing theorists’ views and influences. Thousand Oaks, CA: Sage.

Alkin

M. C.

(Ed.). (2012). Evaluation roots: Tracing theorists’ views and influences (2nd ed.). Thousand Oaks, CA: Sage.

Alkin

M. C.

Christie

C. A.

(2004). An evaluation theory tree. In Alkin

M. C.

(Ed.), Evaluation roots: Tracing theorists’ views and influences (pp. 12–65). Thousand Oaks, CA: Sage.

Alkin

M. C.

A. T.

Hansen

(not yet published). Using logic models to facilitate comparisons of evaluation theory. Evaluation and Program Planning.

Armonia

R. C.

Campilan

D. M.

(1997). Participatory monitoring and evaluation: The Asian experience. Regional overview paper for the International Workshop on participatory monitoring and evaluation: Experiences and lessons. Cavite, Philippines.

Brant-Castellano

(1986). Collective wisdom: Participatory research and Canada’s native people. Convergence: An International Journal of Adult Education, 19, 50–53.

10.

Burke

(1998). Evaluating for change: Reflections on participatory methodology. In Whitmore

(Ed.), Understanding and practicing participatory evaluation. New Directions in Evaluation, No. 80 (pp. 43–56). San Francisco, CA: Jossey-Bass.

11.

Campos

Coupal

(1996). Who are the question makers? A participatory evaluation handbook. New York, NY: UNDP Office of Evaluation and Strategic Planning.

12.

Carden

(2010). Introduction to the forum on evaluation field building in South Asia. American Journal of Evaluation, 31, 291–221.

13.

Chambers

(1981). Rapid rural appraisal: Rationale and repertoire. Public Administration and Development, 1, 95–106.

14.

Cousins

J. B.

(2005). Will the real empowerment evaluation please stand up? A critical friend perspective. In Fetterman

D. M.

Wandersman

(Eds.), Empowerment evaluation principles in practice (pp. 183–208). Thousand Oaks, CA: Sage.

15.

Cousins

J. B.

Chouinard

(2012). Participatory evaluation up close: A review and integration of research-based knowledge. Charlotte, NC: Information Age Press.

16.

Cousins

J. B.

Shulha

L. M.

(2008). Complexities in setting program standards in collaborative evaluation. In Brandon

Smith

(Eds.), Fundamental issues in evaluation (pp. 139–158). New York: Guilford.

17.

Cousins

J. B.

Decent

Kenny

Moore

Pruden

Sanderson

. (2010, June). Multiple case study of community initiatives: National aboriginal youth suicide prevention strategy. Ottawa, Canada: Centre for Research on Evaluation and Community Services.

18.

Cousins

J. B.

Earl

L. M.

(1992). The case for participatory evaluation. Educational Evaluation and Policy Analysis, 14, 397–418.

19.

Cousins

J. B.

Earl

(1995). Participatory evaluation in education: Studies in evaluation use and organizational learning. London, England: Falmer.

20.

Cousins

J. B.

Whitmore

(1998). Framing participatory evaluation. In Whitmore

(Ed.), Understanding and practicing participatory evaluation. New Directions in Evaluation, No. 80 (pp. 3–23). San Francisco, CA: Jossey Bass.

21.

Chouinard

J. A.

Cousins

J. B.

(2007). Culturally competent evaluation for Aboriginal communities: A review of the empirical literature. Journal of Multidisciplinary Evaluation, 4, 40–57.

22.

Dahler-Larsen

(2009). Learning-oriented educational evaluation in contemporary society. In Ryan

K. E.

Cousins

J. B.

(Eds.), Sage international handbook on educational evaluation (pp. 307–322). Thousand Oaks, CA: Sage.

23.

Daigneault

P.-M.

Jacob

(2009). Toward accurate measurement of participation: Rethinking the conceptualization and operationalization of participatory evaluation. American Journal of Evaluation, 30, 330–348.

24.

Davies

Dart

(2005). The ‘most significant change’ (MSC) Technique: A guide to its use. Cambridge, UK Authors, http://www.mande.co.uk/docs/MSCGuide.pdf.

25.

Davis

Sumara

Luce-Kapler

(2008). Engaging minds: Changing teaching in complex times (2nd ed.). New York, NY: Routledge, Taylor & Francis Group.

26.

Estrella

Gaventa

(1998). Who counts reality? Participatory monitoring and evaluation: A literature review. Institute for Development Studies Working Paper 70. Brighton, England: IDS, University of Sussex. Retrieved from www.ids.ac.uk

27.

Fals Borda

(1981). Science and the common people. Journal of Social Studies, 11, 1–21.

28.

Fals Borda

(1992). Evolution and convergence in participatory action research. In Friederes

J. S.

(Ed.), A world of communities: Participatory research perspectives (pp. 14–19). North York, ON: Captus University Publications.

29.

Fals Borda

(2006). The north-south convergence: A 30 year first person assessment of PAR. Action Research, 4, 351–358.

30.

Fernandes

Tandon

(1981). Participatory research and evaluation: Experiments in research as a process of liberation. New Delhi, India: Indian Social Institute.

31.

Fetterman

D. W.

(1994). Empowerment evaluation. Evaluation Practice, 15, 1–15.

32.

Fetterman

(2001). Foundations of empowerment evaluation. Thousand Oaks, CA: Sage.

33.

Fetterman

Wandersman

(Eds.). (2005). Empowerment evaluation principles in practice. New York, NY: Guilford.

34.

Fetterman

Wandersman

(2011). Webinar on participatory evaluation. Part 2 of CPE-TIG sponsored series. Retrieved from www.eval.org

35.

Feuerstein

M. T.

(1986). Partners in evaluation: Evaluating development and community programs with participants. London, England: Macmillan.

36.

Gerring

(1999). What makes a concept good. A critical framework for understanding concept formation in the social sciences. Polity, 31, 357–393.

37.

Guijt

(2008). Seeking surprise: Rethinking monitoring for collective thinking in rural resource management. Wageningen, Netherlands: University of Wageningen.

38.

Hall

(1975). Participatory research: An approach for change. Convergence, An International Journal of Adult Education, 8, 24–31.

39.

Hall

(1981). Participatory research, popular knowledge and power: A personal reflection. Convergence, An International Journal of Adult Education, 14, 5–17.

40.

Hall

(1992). From margin to centre? The development and purpose of participatory research. American Sociologist, 23, 15–28.

41.

Hay

(2010). Evaluation field building in South Asia: Reflections, anecdotes, and questions. American Journal of Evaluation, 31, 222–231.

42.

Hansen

Wallace

T. L

(2010). Creating visual representations of evaluation theories. Working paper (draft). Los Angeles, CA: University of California.

43.

Hansen

Alkin

M. C.

Wallace

(Epub ahead of print). Depicting the logic of three evaluation theories. Evaluation and Program Planning.

44.

Harner

M. A.

(2012). Theory building through praxis discourse: A theory- and practice-informed model of transformative participatory evaluation. Unpublished doctoral dissertation, Claremont Graduate University, Claremont: CA.

45.

Hoare

Levy

Robinson

M. P.

(1993). Participatory action research in Native communities: Cultural opportunities and legal obligations. Canadian Journal of Native Studies 13, 43–78.

46.

Jackson

E. T.

McCaskill

Hall

(Eds.). (1982). Learning for self-determination: Community based options for Native training and research. Canadian Journal of Native Studies, 2, 1–9.

47.

Jackson

E. T.

Kassam

(1998). Knowledge shared: Participatory evaluation in development cooperation. West Hartford, CT: Kumarian.

48.

Kar

Chambers

(2008). Handbook on community-led total sanitation. Brighton, England: Plan International UK & Institute for Development Studies, University of Sussex. Retrieved from www.communityledtotalsanitation.org/sites/communityledtotalsanitation.org/files/cltshandbook.pdf

49.

Kassam

Mustafa

(1982). Participatory research: An emerging alternative methodology in social science research. Toronto, Canada: ICAE.

50.

Luskin

R. J. C.

(2010). Comparing the outcomes of three theories of evaluation. Paper presented at the annual meeting of the American Evaluation Association, San Antonio.

51.

Matheson

(Ed.). (2007). Enduring issues in evaluation: 20th anniversary of the association between NDE and AEA. New Directions for Evaluation. No. 144. San Francisco, CA: Jossey-ssssBass.

52.

Miller

R. L.

(Epub ahead of print). Logic models: A useful way to study theories of evaluation practice? Evaluation and Program Planning.

53.

Narayan

(1994). Participatory evaluation: Tools for managing change in water and sanitation. World Bank Technical Paper No. 207. Washington, DC: World Bank.

54.

O'Sullivan

R. G.

(2004). Practicing evaluation: A collaborative approach. Thousand Oaks, CA: Sage.

55.

O'Sullivan

R. G.

(2012). Collaborative evaluation within a framework of stakeholder-oriented evaluation approaches. Evaluation and Program Planning, 35, 518–522.

56.

PAMFORK. (1997). Participatory Methodologies Forum in Kenya. Report of the workshop in using participatory methodologies for monitoring and evaluation. Nairobi, Kenya: South-South Sharing Forum.

57.

Patton

M. Q.

(2010). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York, NY: Guilford Press.

58.

Preskill

(2008). Evaluation’s second act: Spotlight on learning. American Journal of Evaluation, 29, 127–138.

59.

Ramalingam

Jones

Reba

Young

.) (2008). Exploring the science of complexity: Ideas and implications for development and humanitarian efforts. Working Paper 285. 2nd ed. London, England: Overseas Development Institute (ODI).

60.

Rihani

(2002). Complexity systems theory and development practice: Understanding non-linear realities. London, England: Zed Books.

61.

Rodriguez-Campos

(2005). Collaborative evaluation: A step-by-step model for the evaluator. Tamarac, FL: Lumina Press.

62.

Rodriguez-Campos

(2012). Advances in collaborative evaluation. Evaluation and Program Planning, 35, 523–528.

63.

Rodriguez-Campos

O’Sullivan

(2011). Webinar on collaborative evaluation. Part 3 of CPE-TIG sponsored series. Retrieved from www.eval.org

64.

Rugh

Steinke

Cousins

J. B.

Bamberger

(2010). Summary of discussion of the ‘Alternative to the Statistical Counterfactual Think Tank.’ Paper presented at the American Evaluation Association, San Antonio.

65.

Scriven

(1997). Empowerment evaluation examined. Evaluation Practice, 18, 165–175.

66.

Serrat

(2009). The Most Significant Change Technique. Knowledge Solutions, 25 (January), 1–4, www.adb.org/knowledgesolutions.

67.

Shulha

(2010, Nov.). Essential aspects of participatory evaluation. Paper presented at the annual meeting of the American Evaluation Association, San Antonio.

68.

Shulha

Wilson

(1995). Collaborative evaluation case example. In Cousins

J. B.

Earl

(Eds.), Participatory evaluation in education: Studies in evaluation use and organizational learning. London, England: Falmer.

69.

Snowdon

D. J.

Boone

M. E.

(2007). A leader’s framework for decision making. Harvard Business Review, 85, 69–76.

70.

Stufflebeam

D. L.

(1994). Empowerment evaluation, objectivist evaluation, and evaluation standards: Where the future of evaluation should not go and where it needs to go. American Journal of Evaluation, 15, 321–338.

71.

Tandon

(1981). Participatory research in the empowerment of people. Convergence, 14, 20–29.

72.

Tandon

(2008). Participation, citizenship and democracy: Reflections on 25 years of PRIA. Community Development Journal, 43, 284–296.

73.

Weaver

Cousins

J. B.

(2004). Unpacking the participatory process. Journal of Multidisciplinary Evaluation, 1, 19–40.

74.

Whitmore

(1998a). Understanding and practicing participatory evaluation. New Directions in Evaluation, No. 80. San Francisco, CA: Jossey-Bass.

75.

Whitmore

(1998b). We need to rebuild this house. The role of empowerment evaluation of a Mexican farmers’ cooperative. In Jackson

E. T.

Kassam

(Eds.), Knowledge shared: Participatory evaluation in development cooperation (pp. 217–230). West Hartford, CN: Kumarian Press.

76.

Woodhill

(2007). M&E as learning: Rethinking the dominant paradigm: In deGraaf

Camerson

Sombatpanit

Pieri

Woodhill

(Eds.), Monitoring and evaluation of social conservation and watershed development projects (pp. 83–107). Enfield, New Hampshire: Science Publishers.

77.

Zimmerman

(2000). A complexity science primer: What is complexity science and why should I learn about it? Retrieved from www.plexusinstitute.com/edgeware

78.

Zukoski

Luluqisen

(2011). Webinar on participatory evaluation. Part 2 of CPE-TIG sponsored series. Retrieved from www.eval.org