Abstract
The term “evaluation framework” and other associated terms are common in the practice and the discipline of program evaluation. These terms correspond to a variety of meanings across contexts and organizations and, thus, tend to lack overall consistency. In response, this article provides a model to analyze frameworks for program evaluation organized around four dimensions. The model states that a framework for evaluation is an intellectual framework made of concepts and/or theories (first dimension: types of ideas) about an object related to evaluation (second dimension: object), where the said concepts and theories can be positive and/or normative (third dimension: analytical perspective). These three dimensions provide the means to describe, explain, or judge an evaluation-related matter. A fourth and optional dimension, the institutional character of a framework, allows an evaluation framework to become a form of regulation for behaviors related to program evaluation (fourth dimension: institutional dimension).
Introduction
Those involved in the practice and the discipline of program evaluation are not unfamiliar with the appellations “framework,” “frame of reference,” “evaluation framework,” and the like. We encounter these terms in supranational, international, national, and local organizations, in governments, and in evaluation associations. They are present in the education, health, environment, and development sectors, among others.
While organizational examples for the use of these terms abound, the actual objects they are referring to do not necessarily demonstrate much consistency or similarity. For instance, the Organisation for Economic Co-operation and Development (OECD, 2014) created its Framework for Regulatory Policy Evaluation, an extensive guide of more than 150 pages, with expert advice chapters and supporting thematic bibliographies, to assist “countries in systematically evaluating the design and implementation of regulatory policy” (p. 13). The United Nations Office for Disaster Risk Reduction (2015) refers to its Monitoring and Evaluation Framework, a 26-page document with definitions and organizational and methodological indications, as a way to “provide a consistent approach to the monitoring and evaluation” of its programs (p. 2). The Framework of Results and Key Success Factors, from the Canadian International Development Agency (2004), defines success through high-level questions and themes for the interventions supported by the organization. In this case, the framework is limited to a three-page document found inside the agency’s evaluation guide (p. a1). In the state of Victoria in Australia, a two-page color flyer titled Corrections Victoria Evaluation Framework 2015–2018 presents several evaluation principles (in terms of evaluation themes and the organization of the evaluation function) on one side and a logic model of the implementation process and alleged outcomes sequence on the other (Department of Justice & Regulation, 2014).
Evaluation associations also make use of frameworks. The Australasian Evaluation Society’s (2014) Evaluators’ Professional Learning Competency Framework identifies categories of competency intended to guide evaluation practitioners. In New Zealand, the Aotearoa New Zealand Evaluation Association (n.d.) adopted a similar document called Evaluator Competencies, which was explicitly referred to as a framework during its development period.
In peer-reviewed journals, evaluation frameworks are usually seen as a component of the measurement and data management strategy rather than a way to organize evaluation activities. For both Chen (2001) and Cooksy (1999), frameworks appear as program theories or logic models that guide the choice of indicators and data collection. In contrast, Ekins and Medhurst (2006) use the term for the set of indicators utilized in their evaluation instead of referring to a prior guiding set of ideas, such as a logic model. The term framework also appears on the cover of evaluation textbooks. Markiewicz and Patrick’s (2016) Developing Monitoring and Evaluation Frameworks is a detailed guide to organizing evaluation activities, such as developing a program theory, establishing evaluation questions, and reporting and using plans. With their book Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs, Mark et al. (2000) adopt a drastically different notion of framework; their framework is the scientific and political justification of an evaluation approach driven by the ideal of social betterment.
These self-declared frameworks differ greatly in size, purpose, and object, which poses a certain challenge in defining what an evaluation framework is. Moreover, one could question why other similar documents could not also be called frameworks: the Development Assistance Committee (DAC) set of criteria from the OECD (1991) or the Program Evaluation Standards from the Joint Committee on Standards for Educational Evaluation (Yarbrough et al., 2011) are two well-known examples falling in this category.
Evidently, associating a term with a particular meaning is ultimately arbitrary, as language is, at its core, a shared system of symbols (Hollihan and Baaske, 2015). In that sense, there is nothing inherently invalid in calling or not calling something an evaluation framework, as long as the term is agreed upon by a group of users. From there, clarifying the meaning of frameworks in evaluation could be considered a purely academic endeavor.
Nonetheless, the specific circumstances of evaluation practice call for a consistent terminology in that matter. Evaluation contexts involve not one but several groups of actors, like funders, program hosts, beneficiaries, and evaluators. Each comes with its own understanding of evaluation frameworks arising from its own specific evaluation traditions and associated terminology, which does not foster the natural emergence of shared understanding around frameworks. This alone makes for a great deal of complexity, but in the specific case of frameworks, this complexity is multiplied, as several types of objects, each with its own means of action and areas of influence, can bear the title of evaluation framework.
This multiplicity of interpretations and objects can impair evaluation in two ways: first, there can be a mismatch between people’s goals and the capacity of the framework they are developing or agreeing upon to support these goals; second, disparate understandings regarding applicable frameworks in a situation may affect the predictability and effectiveness of the relationships to be governed by the frameworks.
Generally speaking, skillful use of evaluation language is considered by some an essential competency for evaluation practice (e.g. Patton, 2000) and even finds its way in the Canadian Evaluation Society’s (n.d.) set of Competencies for Canadian Evaluators. In regard to evaluation language, it is not proposed here that the use of framework-related terms in the field should be regulated; instead, it is suggested that a clear point of reference for the term “framework” will help evaluators and stakeholders clarify their communications across different interpretations of the term and, ultimately, seek out or develop frameworks in a mindful way.
This article endeavors to be this point of reference. It addresses the terminological challenge by providing a model to recognize, understand, and analyze frameworks in program evaluation. To do so, the model acknowledges a variety of types of framework and offers a conceptual rationale for their inclusion. The model provides a core definition for all frameworks as well as parameters to differentiate between frameworks based on their positive or normative perspective, their object, and their institutional status. This allows frameworks to be differentiated from non-frameworks, allows for identification of frameworks of different natures, and allows users to efficiently harness the potential of different framework types.
Throughout, the phrase “framework for evaluation,” instead of the more specific “evaluation framework,” is used as a generic term for all frameworks associated with evaluation. The choice is justified since some frameworks relevant to evaluation are not necessarily frameworks with evaluation as their core matter. In addition, unless otherwise specified or necessarily implied by context, the term evaluation will stand for program evaluation where it is used in the following sections. This study is situated within the domain of ideas related to program evaluation, and not the more general domain of evaluation.
Intellectual frameworks and insights from research
The first step in establishing what frameworks for evaluation are is to specify what is meant by frameworks in general, regardless of the realm of activity. While the literal meaning of framework, a “skeletal, openwork, or structural frame” (Merriam-Webster, 2018), is a reminder of their intrinsic organizing nature, the focus of this article is rather on intellectual frameworks. From that standpoint, the Merriam-Webster (2018) defines a framework, in the intellectual sense, as “a basic conceptional structure (as of ideas)” or as “a set of ideas, conditions, or assumptions that determine how something will be approached, perceived, or understood.” In other words, a framework is a set of ideas that provides a perspective on something. It is that understanding that is adopted for the sake of this analysis. In that sense, being an intellectual framework, independently of the level of sophistication, is the most fundamental characteristic of all frameworks for evaluation.
Still, it must be acknowledged that if, at their core, frameworks are intellectual objects, they need to be made explicit to be identified or communicated. Therefore, in practical terms, frameworks often take the form of written statements or even graphic representations about a given set of ideas, and it is these representations that are usually identified as the frameworks themselves. This requirement for practical, explicit communication is recognized by Markiewicz and Patrick (2016), for whom a “monitoring and evaluation framework” is defined as “both a planning process and a written product designed to provide guidance to the conduct of monitoring and evaluation functions over the life span of a program or other initiative” (p. 1, emphasis added).
The adopted definition does not include any requirement of validity or sophistication for the ideas that form it. A framework is a framework as long as a set of ideas exists and provides a perspective about something, regardless of sophistication. Therefore, relying on a framework is not a guarantee of a better, more valid, more penetrating, or more relevant understanding of the matters being considered. This depends, of course, on the qualities of the set of ideas, with the possibility that a framework includes biases that are detrimental to understanding.
The concept of framework is well recognized in research, and the literature on research methodology provides a few insights into the nature and the roles of frameworks. Fundamentally, a research framework is one that provides a perspective on the object of the research. Lester (2005) defines a research framework as “a basic structure of the ideas (i.e., abstractions and relationships) that serve as the basis for a phenomenon that is to be investigated.” Research frameworks play several roles in conducting research, including the formulation of research questions, the development and justification of a methodology, and the interpretation of data (Imenda, 2014; Lester, 2005). These characteristics can play a part in research tasks undertaken in program evaluation but can also apply to research on evaluation as a social phenomenon.
The literature on research methodology refers to two types of research frameworks: conceptual frameworks and theoretical frameworks. While these two terms are sometimes used interchangeably, a useful distinction can be drawn on the basis of their analytical contributions. On one hand, a conceptual framework provides understanding about concepts and how they are related in terms of meaning. For Miles and Huberman (1994), it “lays out the key factors, constructs, or variables, and presumes relationships among them” (p. 440). A theoretical framework, on the other hand, refers to a theory rather than just concepts (Imenda, 2014), where theory means “a set of interrelated propositions, concepts and definitions that present a systematic point of view of specifying relationships between variables with a view to predicting and explaining phenomena” (Fox and Bayat, 2007, as cited in Imenda, 2014). To summarize, a conceptual framework establishes the concepts associated with the research object, and a theoretical framework addresses the matter of causality between the concepts. In that respect, this article, which clarifies the meaning of various concepts for evaluation, could serve as a conceptual framework for research on evaluation, but could not serve as a theoretical framework, as it does not deliver causal explanations related to these concepts.
Following the reasoning found in this section, the first parameter of frameworks for evaluation can be simply stated this way: a framework for evaluation is an intellectual framework, made of concepts and/or theories, that provides a perspective on an evaluation matter.
Normative or positive analytical perspectives
In addition to their conceptual and theoretical aspects, intellectual frameworks can be characterized according to a positive or normative perspective. The adjective normative refers to claims that are based on values: they are about how things ought to be. A normative claim is one that judges the world or one that presents the world as it should be from a value perspective; it, therefore, relies on values, norms, or standards. In contrast, a positive claim is one that describes the world as it is. Therefore, such a claim refers to facts, like measurements or causal explanations (Samuelson and Nordhaus, 2010).
For example, to say that 10 percent of an evaluation association’s membership is made of indigenous members is a positive claim. An explanation that states that indigenous enrollment in evaluation associations is encouraged by a strong sense of belonging is another positive claim. Both previous claims simply posit how the reality of evaluation associations is, without appreciating its value. However, to say that the indigenous membership in an evaluation association should correspond to the proportion of the indigenous population in the general population for the sake of fairness, or that the current 10 percent of representation is unfair, is a normative claims since it is about the appreciation of the value of evaluation associations.
If an intellectual framework is a set of ideas that provides a perspective on something, it is possible to classify it as either normative or positive, depending on the analytical perspective of its constitutive ideas.
The normative/positive dichotomy is a revealing distinction for frameworks in general, but even more when it comes to evaluating, as the normative dimension is core to this activity. Taylor (1961), in his treatise Normative Discourse, puts normativity at the heart of the act of evaluating, where the evaluation of an object is necessarily made by a reference to some norm. For Taylor, attributing value to an object must proceed from an understanding of what value entails for a class of objects, which is precisely the role of the norm.
The distinction between normative and positive claims is linked to the “is-ought” problem from the philosopher of science David Hume in his A Treatise of Human Nature. For Hume, normative claims are fundamentally invalid, as they cannot be derived from positive claims. His stance tends to permeate modern social sciences, which are in some ways at odds with the concept of value. 1 The inclusion and development of normative models and theories are nonetheless openly pursued in some disciplines, like economics (Fleurbaey, 2016), normative ethics, or axiology. Ultimately, whether or not normative ideas and normative frameworks are valid, they do exist, and they represent an important point of differentiation among frameworks, since presenting and judging are two fundamentally different analytical endeavors.
As for the evaluation discipline, the role of normativity is acknowledged through the concepts of criteria, standards, and evaluation synthesis, as seen notably in the work of Scriven (1980), Fournier (1995), Davidson (2005), Mathison (2005), or Julnes (2012a, 2012b). If an evaluation is understood as an inquiry that leads to a judgment on the object examined (Mathison, 2005; Scriven, 1991), this necessarily makes it a normative inquiry. Such a normative inquiry must rely on a set of ideas about the value of the evaluand, that is, its criteria, standards, their weighting, and their justification, in order to appreciate the factual reality of the evaluand. These ideas, together forming the conceptual or theoretical foundations to judging the evaluand, correspond in all points to a normative framework. In that sense, a normative framework is an evaluation framework in the strictest sense of the term, and vice versa.
Nevertheless, not all frameworks in the context of a program evaluation are necessarily made of normative ideas. Both normative and positive perspectives can be used: a positive framework can help establish the factual dimensions of the program before it is examined through a normative perspective in order to establish a judgment. For example, a behavioral theory of policy tools (Schneider and Ingram, 1990) could help reveal the causal mechanisms involved in a new rent control regulation to develop a program theory and eventually inform the data collection. This could be accompanied by a normative theory from welfare economics which would, in turn, provide a rationale to judge the regulation’s results (Salamon, 2002).
Based on the distinctions proposed in this section, the second parameter of frameworks for evaluation can be stated this way: a framework for evaluation can offer a positive or normative analytical perspective on an evaluation matter to accordingly help understand that matter as it is or as it should be.
The object (topic of the framework)
Although the discussion above provides a clearer understanding of the general nature of frameworks, a framework remains incomplete if we do not consider its object; in the end, it is the object of the framework that makes it a framework for evaluation.
Three types of objects are referred to in this analysis. As indicated in the Venn diagram in Figure 1, the selection includes the intervention to be evaluated, the practice of evaluators, and the larger program evaluation function (as the social organization of program evaluation activities). To these three objects, our diagram adds a universal category that captures all possible objects related to program evaluation. All of these objects can be analyzed under normative and positive considerations, which opens the possibility of positive and normative frameworks for each of them. In addition, the Venn diagram illustrates a system of overlaps and inclusions between the categories. First, ideas regarding the practice of evaluators or the organization of the program evaluation function may include considerations regarding the analysis of evaluands. Second, the program evaluation function can be concerned with the work of evaluators (e.g. an evaluation plan could define the contribution of evaluators), and vice versa (e.g. an evaluation approach for evaluators could include provisions regarding the governance of evaluation).
Venn diagram of objects addressed by frameworks for evaluation.
The selection of objects used as the topic of frameworks is based on objects commonly contemplated in the evaluation practice and discipline, as illustrated by the self-declared frameworks listed in the introduction. Consequently, the present selection is a group of plausible objects that can reveal some roles played by frameworks in evaluation. Nevertheless, there is no pretense that the selection is a comprehensive one.
First object: Evaluand (intervention)
The study of interventions benefits from numerous positive theoretical and conceptual frameworks. The previous section has already established that the act of evaluating can benefit from both positive and normative frameworks.
Positive theories from the social sciences provide frameworks to identify and understand the causal mechanisms involved with the intervention and/or the social setting the intervention attempts to modify. This contribution from the social sciences about the causality of programs is acknowledged by the evaluation literature (e.g. Donaldson and Lipsey, 2006; Vaessen and Leeuw, 2010). In addition, models linked to system thinking, as well as practical logic model guidelines that derive from them (e.g. W.K. Kellogg Foundation (2006)), can break down the intervention into its various systemic dimensions, which in turn can guide the collection of administrative and impact data. Moreover, specific social constructs and their associated data collection instruments can help establish the empirical domain of an intervention. All these frameworks remain, by themselves, positive frameworks, as they help describe and explain interventions, but do not provide the intellectual means to judge them.
As mentioned earlier, a program evaluation is, in the strictest sense, a normative inquiry about an intervention and, as such, relies on normative frameworks to inform judgment of the intervention’s value. Such frameworks can be found in general normative concepts of performance like effectiveness and efficiency; high-level evaluation conceptual models like the logic of evaluation (Fournier, 1995; Scriven, 1980) or models related to evaluation synthesis (Julnes, 2012a, 2012b); normative theories and conceptual models about social interventions from the social sciences (e.g. normative economics, public choice, feminist theories); ethics frameworks; or pre-established sets of criteria (like the DAC criteria). To these, we may add frameworks that are much more methodologically operational, such as assessment instruments intended to capture valued constructs, for example, well-being or quality-of-life instruments (e.g. Cooke et al., 2016).
Second object: Program evaluators and their work
The choice of program evaluators and their work among the objects discussed in this study is justified by the evaluation discipline’s own focus: in the literature, evaluation tends to define itself as a field of practice with evaluators, rather than value determination, at the center of its concerns. Practice and evaluators are so important that evaluation theories themselves are often defined by referring to them. For example, Shadish et al. (1991) suggest that the “purpose of program evaluation theory is to specify feasible practices that evaluators can use to construct knowledge of the value of social programs [. . .]” (p. 36). This is echoed by Christie’s own understanding where “prescriptive theories are intended to offer principles, rationales, and organization for the procedural choices made by evaluators . . . to orient practitioners to the issues and problems with which they must deal” (Christie, 2012).
Nevertheless, in order to understand the precise delineations of the present category of objects for this study, the meaning of evaluation practice should be distinguished from the work performed by evaluators. Sociology produced several studies of the notion of practice, with the contributions of influential theorists like Giddens, Bourdieu, and Foucault. In an effort to capture the main characteristics of practice theory emerging from these developments, Reckwitz (2002) defines practice as
a routinized type of behaviour which consists of several elements, interconnected to one other: forms of bodily activities, forms of mental activities, “things” and their use, a background knowledge in the form of understanding, know-how, states of emotion and motivational knowledge. (p. 249)
This definition of practice as a routinized type of behavior could correspond to the work of evaluators if there was a clear activity to be routinized, like the act of determining the value of programs, as suggested by Scriven’s (1991) strict definition of evaluation as a value determination activity. But the actual work performed by evaluators in the field has very inconstant boundaries; it spans from value determination to any kind of analytical work or advice that can support organizations and their programs. This echoes the diversity of definitions for evaluation itself as well as the numerous roles associated with evaluators (e.g. Ryan and Schwandt, 2002). In that context, referring to the work of evaluators captures a reality with a greater correspondence with the empirical situation of evaluators than the sociological notion of evaluation practice. It must be mentioned that the term evaluation practice is used for the remaining of this study for the convenience of the reader, as this is the term of preference currently employed in the evaluation literature. However, the specific implied meaning will be of the work of evaluators.
What the evaluation discipline tends to call evaluation theories or prescriptive theories is a predominant example of normative frameworks applied to the object of evaluators and their decisions. These theories can occasionally be identified through other terms, like evaluation models and evaluation approaches (Owen, 2006), as long as they address the matter of evaluation practice. Observers of the field, decision-makers, or evaluators themselves could use these theories to assess the pertinence and quality of the contribution of evaluators or to prescribe actions to be taken by evaluators.
This is not to say that everything in the evaluation literature that bears the name evaluation theory, model, or approach is entirely made of normative components applied to practice. Some of these contributions may be entirely positive. For example, a conceptual model regarding the types of participatory evaluations, in essence a positive contribution, is distinct from a discourse that promotes one form or another type of participation, in essence a normative contribution; however, both can be associated with the same approach and even be found in the same corpus (e.g. Cousins and Earl, 1992).
Alongside evaluation theories are the various references adopted by evaluation associations to guide the work of evaluators, such as standards of practice, sets of competencies, and ethics guidelines for evaluation. As these documents have a prescriptive message regarding what evaluators should know or do, they also count as normative frameworks for practice.
These evaluation theories, approaches, models, and professional documents are obvious normative frameworks for evaluation practice because they emerge from the evaluation field itself and are specific to it. Nevertheless, they are far from being the sole perspective on evaluators and their practice. Research ethics obligations are another type of potential normative framework, particularly for the interactions between evaluators and evaluation participants when evaluation occurs in a university research context. Equally, positive theories and models from social sciences can support the research on evaluation practice by explaining and clarifying, for example, interactions with stakeholders, the notion of profession, conflicts of interest, and other notions and phenomena that constitute evaluation practice.
Third object: Evaluation function
Evaluation function is simply defined here as the organization and the conduct of evaluation in society. This overlaps with, but is distinct from, the practice of evaluation by evaluators. Not all work performed by evaluators is related to evaluation in the strict sense of the term, and not all evaluation-related activity includes evaluators. Certain tasks, such as the decision to evaluate a program or planning a budget for evaluation resources, are strategic decisions that may happen without an evaluator’s involvement. An evaluation itself does not necessarily imply the involvement of an evaluator.
Many frameworks calling themselves evaluation frameworks mainly address the evaluation function. This can include prescriptions about the selection and planning of evaluations, the distribution of responsibilities among various actors, the breakdown of the evaluation administrative or research process, the identification of a shared terminology, the various forms and conditions of communication of the results, as well as the budget and resources for program evaluation (Arbour, 2016). The usual prescriptive focus of these documents makes them normative frameworks.
Other contributions from the social sciences, especially from economics, public administration, and political science, can provide positive and normative analytical tools to study evaluation function in society. For instance, the notion of “institution” from the Public Choice school in political economics, which serves as the basis for the analysis in the next section, has been used before to study evaluation institutions (e.g. Marceau and Turgeon, 1994). Some frameworks developed for the study of the institutionalization of evaluation even emerge from the evaluation discipline itself (e.g. Furubo et al., 2002; Jacob et al., 2015), notably for the study of evaluation policies (Trochim, 2009).
This section addressed the third parameter of frameworks for evaluation, that is, that such a framework addresses an evaluation object. It did so through identifying three possible and predominant classes of object in evaluation, namely interventions, evaluation practice, and the evaluation function in society. Frameworks addressing these objects appear under different guises, including social science theory, evaluation theory, and professional standards.
Institutional frameworks
So far, it has been posited that a framework for evaluation is an intellectual framework with a positive and/or a normative perspective on one or several objects related to evaluation. At the intersection of intellectual perspectives (concepts and theories, normative and positive) and evaluation objects (evaluand, evaluators, evaluation function), all kinds of sets of ideas can qualify as frameworks for evaluation and these can fulfill a variety of purposes for a variety of observers. While different frameworks support various agents and their goals (like an evaluation, a research on evaluation, or the monitoring of a program), the notion of purpose is irrelevant for the characterization of strictly intellectual frameworks: a framework provides a perspective, and its user and the user’s intentions do not change anything about the nature of that framework. For example, a single model or approach on participatory evaluation can guide a researcher to establish research questions and a methodology to investigate evaluation practice, lead an evaluator’s self-reflection, or help frame an evaluation contract by an evaluation commissioner. In all these cases, the framework remains ultimately the same: intellectual frameworks are not purpose-specific.
On some occasions, it is expected for a framework to go beyond the intellectual realm of perspectives and to influence evaluation reality: By integrating the institutional dimension, frameworks become a form of intervention that regulates the behaviors related to program evaluation. This section looks at frameworks as institutions or, more specifically, as institutional frameworks for program evaluation.
The perspective on institutions adopted here is the one proposed by Vincent and Elinor Ostrom (1980, 1986), which associates institutions with the idea of rules. Rules “refer to prescriptions commonly known and used by a set of participants to order repetitive, interdependent relationships” (Ostrom, 1986: 5). Prescriptions themselves are about what actions (or outcomes) are required, prohibited, or permitted (Ostrom, 1980, 1986).
An important aspect of rules is that they can be conceived as interventions. Rules are human artifacts; as such, they are made by humans and can be modified by humans (Ostrom, 1980). They are created with a particular goal: they are the result of implicit or explicit efforts to achieve order and predictability in human behavior (Ostrom, 1991). For E. Ostrom (1986), while behavioral and physical laws cannot be changed, rules can be, which allows us to bring social changes through the control of the structure of incentives in a social situation.
Crawford and Ostrom (2005) developed a comprehensive conceptual model to analyze institutions. They further divided institutions into three “institutional statements,” namely, shared strategies, norms, and rules, where each one bears a stronger institutional weight than the last due to the incremental combination of institutional components. In their model, the shared strategy has an attribute (the identity of who is concerned with the prescription), a deontic (a modal verb that requires, prohibits, or permits) and an aim (the behavior or outcome that is the object of the deontic). The norm adds to these three components the condition or conditions that supersede the application of the prescription. Finally, what distinguishes a rule from the two other institutional statements, and ultimately gives it more influential weight, is the presence of a consequence for not following the prescription that is attached to the rule, known as the or else component.
The following example illustrates the various types of institutional statements in a program evaluation context. Consider the statement “evaluators should make explicit judgments in their reports.” This is a shared strategy where “evaluators” is the attribute, “should” the deontic, and “make explicit judgments in their reports” the aim. If the statement evolves to become “evaluators should make explicit judgments in their reports when the evaluation is commissioned by the board of administrators,” the section “when the evaluation is commissioned by the board of administrators” is a condition for the application of the prescription, which makes the statement a norm. Finally, if the statement is instead “evaluators should make explicit judgments in their reports when the evaluation is commissioned by the board of administrators or they will not be considered for evaluation contracts for a period of three years,” the addition of a consequence for non-compliance in the form of a 3-year ban is an or else particle, which makes the statement a rule.
Following the notions of institutions and institutional statements, a few conclusions can be drawn about institutional frameworks for program evaluation. The first is that all institutional frameworks must include prescriptions, since the deontic particle is common to all of them. This makes them necessarily fall within the category of normative frameworks, since prescriptions are about how things should be. The second is that these prescriptions are not simply expressed as part of theoretical debates on evaluation; rather, they are used in the real world to regulate evaluation-related behaviors. The third and final conclusion is that to correspond to the strongest institutional statement, that is, the rule, an institutional framework for evaluation must also state consequences in case of non-compliance. A practical implication of these characteristics is that an institutional framework for evaluation is more likely to be adopted by an entity that has the authority to make a framework an instrument of order, such as a government, a legislative body, or a professional order.
As a matter of fact, many frameworks adopted by organizations in the context of evaluation are institutional frameworks that address the object of the evaluation function. However, the sole fact of being adopted by an organization does not make a framework an institutional framework if it is not used to order behavior. The DAC set of criteria, for example, is not an institutional framework for evaluation even though it is adopted by the OECD. It is simply presented by the OECD as a way to inform evaluation activities and is not part of an institutional attempt to control the evaluation activities of its member countries. Nonetheless, if the same set of criteria is adopted by a national organization as a shared strategy, a norm, or a rule to control its evaluation activities, then it becomes an institutional framework for evaluation or a part of such framework.
A similar analysis can be made of the regulation of the evaluation practice. Many normative frameworks regarding evaluators’ behaviors are adopted by evaluation associations around the world, but most of them miss the requirements of a system of institutional statements. A rare exception is the Credentialed Evaluator designation from the Canadian Evaluation Society (2014), which regulates the right to use a specific title. Frameworks of this sort are more likely to be shared strategies, norms, or even merely informative propositions without any institutional status. In the end, institutional frameworks for evaluation are not about one object in particular; they are about turning a set of ideas into institutional statements to order human behaviors involved in evaluation.
Many forms can be used to establish a framework and communicate its institutional status. Depending on its audience and who establishes it, a framework may take the form of an evaluation contract, an evaluation policy, an executive order, a government regulation, or a public administration law (such as the Budget and Accounting Act of 1921 that established the Government Accountability Office in the United States). In the case of rules, Ostrom specifies that what matters ultimately is not the embodiment in a formal document but the notion of working rule. If a rule is formulated in a law but is never intended to be enforced, there is no rule; a working rule only exists when there is a real chance of accountability (Ostrom, 1986). Therefore, the mere presence of an evaluation policy in a governmental department may not necessarily bear much weight if no one recognizes its status or if its promoters do not have the capacity to enforce it. However, the absence of an explicit, written document does not imply that there is a complete absence of a rule or another institutional statement. For example, the unwritten customs in place in a certain evaluation market could be clear about what would be considered a problematic delivery and lead to no further contractual deals with a consultant.
This section discussed the fourth parameter of frameworks for evaluation with the help of lessons drawn from institutional analysis. Frameworks for evaluation either have an institutional dimension or do not. When they do, they incorporate a set of prescriptions to order evaluation behavior.
Summary and conclusion
The discussion at the heart of this article originated in the conceptual struggle to identify and understand frameworks in the field of evaluation. The article addressed the issue by providing a model to analyze frameworks for program evaluation, organized around four constitutive dimensions. The summary of learnings from this article is illustrated in Figure 2.
Summary of the constitutive dimensions of frameworks for evaluation and their communication.
From our analysis, we can conclude that a framework for evaluation is an intellectual framework, made of concepts and/or theories (types of ideas) about an object related to evaluation (object), where the said concepts and theories can be positive and/or normative (analytical perspective). A framework for evaluation provides the means to describe, explain, or judge an evaluation-related matter under the purview of the user of the framework, while not being defined by the goals pursued by the user. In addition, an institutional framework for evaluation is a normative framework of a certain institutional status that regulates behavior related to program evaluation (institutional dimension). In institutional frameworks for evaluation, the intellectual aspect of the framework is associated with the specific purpose of ordering evaluation behavior.
As shown through the various examples encountered in this article, the sources for frameworks are varied and go beyond the evaluation field, from the academic world to leading evaluation organizations and professional organizations. While the form of communication (e.g. a scientific article or an evaluation policy) does not define the intellectual or institutional nature of a framework, it can provide indications regarding its institutional status, as regulatory and contractual documents, for example, tend to be associated with the required authority and intent to regulate evaluation behaviors.
The analytical model proposed in this article provides the means to unlock the intellectual and institutional nature of different frameworks for evaluation, thanks to a broad characterization occurring at the intersection of its four parameters. This ultimately leads to a better understanding and more conscious choices about them.
Clarifying the definition of frameworks for evaluation is but one step in the study of their role in the evaluation field. First, in descriptive terms, the fundamental distinctions established in this article can lead to a more refined approach to frameworks’ empirical account, either by classifying different frameworks with additional precision or by recognizing the presence of various institutional and non-institutional frameworks in a single document. Second, the model developed here can contribute to the study of frameworks’ influence on practice: the institutional/non-institutional divide as well as the three types of institutional statements force us to consider entirely different objects in causal terms: simple ideas about practice (non-institutional) are not likely to provoke the same consequences on evaluation behavior as fully enforced institutions. These distinctions can be associated with relevant causal theories from the social sciences to eventually calibrate our expectations and inform individuals’ and organizations’ choices of frameworks. Third, having identified general types of frameworks, it is now possible to discuss what makes a framework valuable within its own category. Prescriptions about frameworks cannot be entirely universal, and the distinctions provided by this model offer a way to identify varying criteria of merit.
In practice, a framework has an impact because someone decides to adopt, adapt, or develop that framework in a given evaluation context. This leads in many cases to locally developed logic models, evaluation plans, evaluation policies, and many other products associated with the term “evaluation frameworks.” The further study of the intellectual quality of these locally developed frameworks and of their elaboration process is crucial for the field of evaluation as they represent ideas in action with a real potential for influence on the delivery of quality program evaluations. This is even more critical when frameworks have an institutional dimension, as they are actual interventions on the social phenomenon of evaluation. In this case, understanding the causal functioning of these regulatory interventions on evaluation behavior is key to improve our effectiveness in shaping evaluation toward our collective expectations.
Footnotes
Acknowledgements
The author wishes to recognize the contribution of Associate Professor John M. Owen and Associate Professor Richard Marceau for their useful feedback on different versions of this paper.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by departmental resources. This work was supported by the Research Development Award of the Melbourne Graduate School of Education (grant number 01-4600-06-xxxx-000319-RDQ-12-01).
