Frameworks for program evaluation: Considerations on research,practice,and institutions

Introduction

Those involved in the practice and the discipline of program evaluation are not unfamiliar with the appellations “framework,” “frame of reference,” “evaluation framework,” and the like. We encounter these terms in supranational, international, national, and local organizations, in governments, and in evaluation associations. They are present in the education, health, environment, and development sectors, among others.

While organizational examples for the use of these terms abound, the actual objects they are referring to do not necessarily demonstrate much consistency or similarity. For instance, the Organisation for Economic Co-operation and Development (OECD, 2014) created its Framework for Regulatory Policy Evaluation, an extensive guide of more than 150 pages, with expert advice chapters and supporting thematic bibliographies, to assist “countries in systematically evaluating the design and implementation of regulatory policy” (p. 13). The United Nations Office for Disaster Risk Reduction (2015) refers to its Monitoring and Evaluation Framework, a 26-page document with definitions and organizational and methodological indications, as a way to “provide a consistent approach to the monitoring and evaluation” of its programs (p. 2). The Framework of Results and Key Success Factors, from the Canadian International Development Agency (2004), defines success through high-level questions and themes for the interventions supported by the organization. In this case, the framework is limited to a three-page document found inside the agency’s evaluation guide (p. a1). In the state of Victoria in Australia, a two-page color flyer titled Corrections Victoria Evaluation Framework 2015–2018 presents several evaluation principles (in terms of evaluation themes and the organization of the evaluation function) on one side and a logic model of the implementation process and alleged outcomes sequence on the other (Department of Justice & Regulation, 2014).

Evaluation associations also make use of frameworks. The Australasian Evaluation Society’s (2014) Evaluators’ Professional Learning Competency Framework identifies categories of competency intended to guide evaluation practitioners. In New Zealand, the Aotearoa New Zealand Evaluation Association (n.d.) adopted a similar document called Evaluator Competencies, which was explicitly referred to as a framework during its development period.

In peer-reviewed journals, evaluation frameworks are usually seen as a component of the measurement and data management strategy rather than a way to organize evaluation activities. For both Chen (2001) and Cooksy (1999), frameworks appear as program theories or logic models that guide the choice of indicators and data collection. In contrast, Ekins and Medhurst (2006) use the term for the set of indicators utilized in their evaluation instead of referring to a prior guiding set of ideas, such as a logic model. The term framework also appears on the cover of evaluation textbooks. Markiewicz and Patrick’s (2016) Developing Monitoring and Evaluation Frameworks is a detailed guide to organizing evaluation activities, such as developing a program theory, establishing evaluation questions, and reporting and using plans. With their book Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs, Mark et al. (2000) adopt a drastically different notion of framework; their framework is the scientific and political justification of an evaluation approach driven by the ideal of social betterment.

These self-declared frameworks differ greatly in size, purpose, and object, which poses a certain challenge in defining what an evaluation framework is. Moreover, one could question why other similar documents could not also be called frameworks: the Development Assistance Committee (DAC) set of criteria from the OECD (1991) or the Program Evaluation Standards from the Joint Committee on Standards for Educational Evaluation (Yarbrough et al., 2011) are two well-known examples falling in this category.

Evidently, associating a term with a particular meaning is ultimately arbitrary, as language is, at its core, a shared system of symbols (Hollihan and Baaske, 2015). In that sense, there is nothing inherently invalid in calling or not calling something an evaluation framework, as long as the term is agreed upon by a group of users. From there, clarifying the meaning of frameworks in evaluation could be considered a purely academic endeavor.

Nonetheless, the specific circumstances of evaluation practice call for a consistent terminology in that matter. Evaluation contexts involve not one but several groups of actors, like funders, program hosts, beneficiaries, and evaluators. Each comes with its own understanding of evaluation frameworks arising from its own specific evaluation traditions and associated terminology, which does not foster the natural emergence of shared understanding around frameworks. This alone makes for a great deal of complexity, but in the specific case of frameworks, this complexity is multiplied, as several types of objects, each with its own means of action and areas of influence, can bear the title of evaluation framework.

This multiplicity of interpretations and objects can impair evaluation in two ways: first, there can be a mismatch between people’s goals and the capacity of the framework they are developing or agreeing upon to support these goals; second, disparate understandings regarding applicable frameworks in a situation may affect the predictability and effectiveness of the relationships to be governed by the frameworks.

Generally speaking, skillful use of evaluation language is considered by some an essential competency for evaluation practice (e.g. Patton, 2000) and even finds its way in the Canadian Evaluation Society’s (n.d.) set of Competencies for Canadian Evaluators. In regard to evaluation language, it is not proposed here that the use of framework-related terms in the field should be regulated; instead, it is suggested that a clear point of reference for the term “framework” will help evaluators and stakeholders clarify their communications across different interpretations of the term and, ultimately, seek out or develop frameworks in a mindful way.

This article endeavors to be this point of reference. It addresses the terminological challenge by providing a model to recognize, understand, and analyze frameworks in program evaluation. To do so, the model acknowledges a variety of types of framework and offers a conceptual rationale for their inclusion. The model provides a core definition for all frameworks as well as parameters to differentiate between frameworks based on their positive or normative perspective, their object, and their institutional status. This allows frameworks to be differentiated from non-frameworks, allows for identification of frameworks of different natures, and allows users to efficiently harness the potential of different framework types.

Throughout, the phrase “framework for evaluation,” instead of the more specific “evaluation framework,” is used as a generic term for all frameworks associated with evaluation. The choice is justified since some frameworks relevant to evaluation are not necessarily frameworks with evaluation as their core matter. In addition, unless otherwise specified or necessarily implied by context, the term evaluation will stand for program evaluation where it is used in the following sections. This study is situated within the domain of ideas related to program evaluation, and not the more general domain of evaluation.

Intellectual frameworks and insights from research

The first step in establishing what frameworks for evaluation are is to specify what is meant by frameworks in general, regardless of the realm of activity. While the literal meaning of framework, a “skeletal, openwork, or structural frame” (Merriam-Webster, 2018), is a reminder of their intrinsic organizing nature, the focus of this article is rather on intellectual frameworks. From that standpoint, the Merriam-Webster (2018) defines a framework, in the intellectual sense, as “a basic conceptional structure (as of ideas)” or as “a set of ideas, conditions, or assumptions that determine how something will be approached, perceived, or understood.” In other words, a framework is a set of ideas that provides a perspective on something. It is that understanding that is adopted for the sake of this analysis. In that sense, being an intellectual framework, independently of the level of sophistication, is the most fundamental characteristic of all frameworks for evaluation.

Still, it must be acknowledged that if, at their core, frameworks are intellectual objects, they need to be made explicit to be identified or communicated. Therefore, in practical terms, frameworks often take the form of written statements or even graphic representations about a given set of ideas, and it is these representations that are usually identified as the frameworks themselves. This requirement for practical, explicit communication is recognized by Markiewicz and Patrick (2016), for whom a “monitoring and evaluation framework” is defined as “both a planning process and a written product designed to provide guidance to the conduct of monitoring and evaluation functions over the life span of a program or other initiative” (p. 1, emphasis added).

The adopted definition does not include any requirement of validity or sophistication for the ideas that form it. A framework is a framework as long as a set of ideas exists and provides a perspective about something, regardless of sophistication. Therefore, relying on a framework is not a guarantee of a better, more valid, more penetrating, or more relevant understanding of the matters being considered. This depends, of course, on the qualities of the set of ideas, with the possibility that a framework includes biases that are detrimental to understanding.

The concept of framework is well recognized in research, and the literature on research methodology provides a few insights into the nature and the roles of frameworks. Fundamentally, a research framework is one that provides a perspective on the object of the research. Lester (2005) defines a research framework as “a basic structure of the ideas (i.e., abstractions and relationships) that serve as the basis for a phenomenon that is to be investigated.” Research frameworks play several roles in conducting research, including the formulation of research questions, the development and justification of a methodology, and the interpretation of data (Imenda, 2014; Lester, 2005). These characteristics can play a part in research tasks undertaken in program evaluation but can also apply to research on evaluation as a social phenomenon.

The literature on research methodology refers to two types of research frameworks: conceptual frameworks and theoretical frameworks. While these two terms are sometimes used interchangeably, a useful distinction can be drawn on the basis of their analytical contributions. On one hand, a conceptual framework provides understanding about concepts and how they are related in terms of meaning. For Miles and Huberman (1994), it “lays out the key factors, constructs, or variables, and presumes relationships among them” (p. 440). A theoretical framework, on the other hand, refers to a theory rather than just concepts (Imenda, 2014), where theory means “a set of interrelated propositions, concepts and definitions that present a systematic point of view of specifying relationships between variables with a view to predicting and explaining phenomena” (Fox and Bayat, 2007, as cited in Imenda, 2014). To summarize, a conceptual framework establishes the concepts associated with the research object, and a theoretical framework addresses the matter of causality between the concepts. In that respect, this article, which clarifies the meaning of various concepts for evaluation, could serve as a conceptual framework for research on evaluation, but could not serve as a theoretical framework, as it does not deliver causal explanations related to these concepts.

Following the reasoning found in this section, the first parameter of frameworks for evaluation can be simply stated this way: a framework for evaluation is an intellectual framework, made of concepts and/or theories, that provides a perspective on an evaluation matter.

Normative or positive analytical perspectives

In addition to their conceptual and theoretical aspects, intellectual frameworks can be characterized according to a positive or normative perspective. The adjective normative refers to claims that are based on values: they are about how things ought to be. A normative claim is one that judges the world or one that presents the world as it should be from a value perspective; it, therefore, relies on values, norms, or standards. In contrast, a positive claim is one that describes the world as it is. Therefore, such a claim refers to facts, like measurements or causal explanations (Samuelson and Nordhaus, 2010).

For example, to say that 10 percent of an evaluation association’s membership is made of indigenous members is a positive claim. An explanation that states that indigenous enrollment in evaluation associations is encouraged by a strong sense of belonging is another positive claim. Both previous claims simply posit how the reality of evaluation associations is, without appreciating its value. However, to say that the indigenous membership in an evaluation association should correspond to the proportion of the indigenous population in the general population for the sake of fairness, or that the current 10 percent of representation is unfair, is a normative claims since it is about the appreciation of the value of evaluation associations.

If an intellectual framework is a set of ideas that provides a perspective on something, it is possible to classify it as either normative or positive, depending on the analytical perspective of its constitutive ideas.

The normative/positive dichotomy is a revealing distinction for frameworks in general, but even more when it comes to evaluating, as the normative dimension is core to this activity. Taylor (1961), in his treatise Normative Discourse, puts normativity at the heart of the act of evaluating, where the evaluation of an object is necessarily made by a reference to some norm. For Taylor, attributing value to an object must proceed from an understanding of what value entails for a class of objects, which is precisely the role of the norm.

The distinction between normative and positive claims is linked to the “is-ought” problem from the philosopher of science David Hume in his A Treatise of Human Nature. For Hume, normative claims are fundamentally invalid, as they cannot be derived from positive claims. His stance tends to permeate modern social sciences, which are in some ways at odds with the concept of value. ¹ The inclusion and development of normative models and theories are nonetheless openly pursued in some disciplines, like economics (Fleurbaey, 2016), normative ethics, or axiology. Ultimately, whether or not normative ideas and normative frameworks are valid, they do exist, and they represent an important point of differentiation among frameworks, since presenting and judging are two fundamentally different analytical endeavors.

As for the evaluation discipline, the role of normativity is acknowledged through the concepts of criteria, standards, and evaluation synthesis, as seen notably in the work of Scriven (1980), Fournier (1995), Davidson (2005), Mathison (2005), or Julnes (2012a, 2012b). If an evaluation is understood as an inquiry that leads to a judgment on the object examined (Mathison, 2005; Scriven, 1991), this necessarily makes it a normative inquiry. Such a normative inquiry must rely on a set of ideas about the value of the evaluand, that is, its criteria, standards, their weighting, and their justification, in order to appreciate the factual reality of the evaluand. These ideas, together forming the conceptual or theoretical foundations to judging the evaluand, correspond in all points to a normative framework. In that sense, a normative framework is an evaluation framework in the strictest sense of the term, and vice versa.

Nevertheless, not all frameworks in the context of a program evaluation are necessarily made of normative ideas. Both normative and positive perspectives can be used: a positive framework can help establish the factual dimensions of the program before it is examined through a normative perspective in order to establish a judgment. For example, a behavioral theory of policy tools (Schneider and Ingram, 1990) could help reveal the causal mechanisms involved in a new rent control regulation to develop a program theory and eventually inform the data collection. This could be accompanied by a normative theory from welfare economics which would, in turn, provide a rationale to judge the regulation’s results (Salamon, 2002).

Based on the distinctions proposed in this section, the second parameter of frameworks for evaluation can be stated this way: a framework for evaluation can offer a positive or normative analytical perspective on an evaluation matter to accordingly help understand that matter as it is or as it should be.

The object (topic of the framework)

Although the discussion above provides a clearer understanding of the general nature of frameworks, a framework remains incomplete if we do not consider its object; in the end, it is the object of the framework that makes it a framework for evaluation.

Three types of objects are referred to in this analysis. As indicated in the Venn diagram in Figure 1, the selection includes the intervention to be evaluated, the practice of evaluators, and the larger program evaluation function (as the social organization of program evaluation activities). To these three objects, our diagram adds a universal category that captures all possible objects related to program evaluation. All of these objects can be analyzed under normative and positive considerations, which opens the possibility of positive and normative frameworks for each of them. In addition, the Venn diagram illustrates a system of overlaps and inclusions between the categories. First, ideas regarding the practice of evaluators or the organization of the program evaluation function may include considerations regarding the analysis of evaluands. Second, the program evaluation function can be concerned with the work of evaluators (e.g. an evaluation plan could define the contribution of evaluators), and vice versa (e.g. an evaluation approach for evaluators could include provisions regarding the governance of evaluation).

OPEN IN VIEWER

Figure 1.

Venn diagram of objects addressed by frameworks for evaluation.

The selection of objects used as the topic of frameworks is based on objects commonly contemplated in the evaluation practice and discipline, as illustrated by the self-declared frameworks listed in the introduction. Consequently, the present selection is a group of plausible objects that can reveal some roles played by frameworks in evaluation. Nevertheless, there is no pretense that the selection is a comprehensive one.

Institutional frameworks

So far, it has been posited that a framework for evaluation is an intellectual framework with a positive and/or a normative perspective on one or several objects related to evaluation. At the intersection of intellectual perspectives (concepts and theories, normative and positive) and evaluation objects (evaluand, evaluators, evaluation function), all kinds of sets of ideas can qualify as frameworks for evaluation and these can fulfill a variety of purposes for a variety of observers. While different frameworks support various agents and their goals (like an evaluation, a research on evaluation, or the monitoring of a program), the notion of purpose is irrelevant for the characterization of strictly intellectual frameworks: a framework provides a perspective, and its user and the user’s intentions do not change anything about the nature of that framework. For example, a single model or approach on participatory evaluation can guide a researcher to establish research questions and a methodology to investigate evaluation practice, lead an evaluator’s self-reflection, or help frame an evaluation contract by an evaluation commissioner. In all these cases, the framework remains ultimately the same: intellectual frameworks are not purpose-specific.

On some occasions, it is expected for a framework to go beyond the intellectual realm of perspectives and to influence evaluation reality: By integrating the institutional dimension, frameworks become a form of intervention that regulates the behaviors related to program evaluation. This section looks at frameworks as institutions or, more specifically, as institutional frameworks for program evaluation.

The perspective on institutions adopted here is the one proposed by Vincent and Elinor Ostrom (1980, 1986), which associates institutions with the idea of rules. Rules “refer to prescriptions commonly known and used by a set of participants to order repetitive, interdependent relationships” (Ostrom, 1986: 5). Prescriptions themselves are about what actions (or outcomes) are required, prohibited, or permitted (Ostrom, 1980, 1986).

An important aspect of rules is that they can be conceived as interventions. Rules are human artifacts; as such, they are made by humans and can be modified by humans (Ostrom, 1980). They are created with a particular goal: they are the result of implicit or explicit efforts to achieve order and predictability in human behavior (Ostrom, 1991). For E. Ostrom (1986), while behavioral and physical laws cannot be changed, rules can be, which allows us to bring social changes through the control of the structure of incentives in a social situation.

Crawford and Ostrom (2005) developed a comprehensive conceptual model to analyze institutions. They further divided institutions into three “institutional statements,” namely, shared strategies, norms, and rules, where each one bears a stronger institutional weight than the last due to the incremental combination of institutional components. In their model, the shared strategy has an attribute (the identity of who is concerned with the prescription), a deontic (a modal verb that requires, prohibits, or permits) and an aim (the behavior or outcome that is the object of the deontic). The norm adds to these three components the condition or conditions that supersede the application of the prescription. Finally, what distinguishes a rule from the two other institutional statements, and ultimately gives it more influential weight, is the presence of a consequence for not following the prescription that is attached to the rule, known as the or else component.

The following example illustrates the various types of institutional statements in a program evaluation context. Consider the statement “evaluators should make explicit judgments in their reports.” This is a shared strategy where “evaluators” is the attribute, “should” the deontic, and “make explicit judgments in their reports” the aim. If the statement evolves to become “evaluators should make explicit judgments in their reports when the evaluation is commissioned by the board of administrators,” the section “when the evaluation is commissioned by the board of administrators” is a condition for the application of the prescription, which makes the statement a norm. Finally, if the statement is instead “evaluators should make explicit judgments in their reports when the evaluation is commissioned by the board of administrators or they will not be considered for evaluation contracts for a period of three years,” the addition of a consequence for non-compliance in the form of a 3-year ban is an or else particle, which makes the statement a rule.

Following the notions of institutions and institutional statements, a few conclusions can be drawn about institutional frameworks for program evaluation. The first is that all institutional frameworks must include prescriptions, since the deontic particle is common to all of them. This makes them necessarily fall within the category of normative frameworks, since prescriptions are about how things should be. The second is that these prescriptions are not simply expressed as part of theoretical debates on evaluation; rather, they are used in the real world to regulate evaluation-related behaviors. The third and final conclusion is that to correspond to the strongest institutional statement, that is, the rule, an institutional framework for evaluation must also state consequences in case of non-compliance. A practical implication of these characteristics is that an institutional framework for evaluation is more likely to be adopted by an entity that has the authority to make a framework an instrument of order, such as a government, a legislative body, or a professional order.

As a matter of fact, many frameworks adopted by organizations in the context of evaluation are institutional frameworks that address the object of the evaluation function. However, the sole fact of being adopted by an organization does not make a framework an institutional framework if it is not used to order behavior. The DAC set of criteria, for example, is not an institutional framework for evaluation even though it is adopted by the OECD. It is simply presented by the OECD as a way to inform evaluation activities and is not part of an institutional attempt to control the evaluation activities of its member countries. Nonetheless, if the same set of criteria is adopted by a national organization as a shared strategy, a norm, or a rule to control its evaluation activities, then it becomes an institutional framework for evaluation or a part of such framework.

A similar analysis can be made of the regulation of the evaluation practice. Many normative frameworks regarding evaluators’ behaviors are adopted by evaluation associations around the world, but most of them miss the requirements of a system of institutional statements. A rare exception is the Credentialed Evaluator designation from the Canadian Evaluation Society (2014), which regulates the right to use a specific title. Frameworks of this sort are more likely to be shared strategies, norms, or even merely informative propositions without any institutional status. In the end, institutional frameworks for evaluation are not about one object in particular; they are about turning a set of ideas into institutional statements to order human behaviors involved in evaluation.

Many forms can be used to establish a framework and communicate its institutional status. Depending on its audience and who establishes it, a framework may take the form of an evaluation contract, an evaluation policy, an executive order, a government regulation, or a public administration law (such as the Budget and Accounting Act of 1921 that established the Government Accountability Office in the United States). In the case of rules, Ostrom specifies that what matters ultimately is not the embodiment in a formal document but the notion of working rule. If a rule is formulated in a law but is never intended to be enforced, there is no rule; a working rule only exists when there is a real chance of accountability (Ostrom, 1986). Therefore, the mere presence of an evaluation policy in a governmental department may not necessarily bear much weight if no one recognizes its status or if its promoters do not have the capacity to enforce it. However, the absence of an explicit, written document does not imply that there is a complete absence of a rule or another institutional statement. For example, the unwritten customs in place in a certain evaluation market could be clear about what would be considered a problematic delivery and lead to no further contractual deals with a consultant.

This section discussed the fourth parameter of frameworks for evaluation with the help of lessons drawn from institutional analysis. Frameworks for evaluation either have an institutional dimension or do not. When they do, they incorporate a set of prescriptions to order evaluation behavior.

Summary and conclusion

The discussion at the heart of this article originated in the conceptual struggle to identify and understand frameworks in the field of evaluation. The article addressed the issue by providing a model to analyze frameworks for program evaluation, organized around four constitutive dimensions. The summary of learnings from this article is illustrated in Figure 2.

OPEN IN VIEWER

Figure 2.

Summary of the constitutive dimensions of frameworks for evaluation and their communication.

From our analysis, we can conclude that a framework for evaluation is an intellectual framework, made of concepts and/or theories (types of ideas) about an object related to evaluation (object), where the said concepts and theories can be positive and/or normative (analytical perspective). A framework for evaluation provides the means to describe, explain, or judge an evaluation-related matter under the purview of the user of the framework, while not being defined by the goals pursued by the user. In addition, an institutional framework for evaluation is a normative framework of a certain institutional status that regulates behavior related to program evaluation (institutional dimension). In institutional frameworks for evaluation, the intellectual aspect of the framework is associated with the specific purpose of ordering evaluation behavior.

As shown through the various examples encountered in this article, the sources for frameworks are varied and go beyond the evaluation field, from the academic world to leading evaluation organizations and professional organizations. While the form of communication (e.g. a scientific article or an evaluation policy) does not define the intellectual or institutional nature of a framework, it can provide indications regarding its institutional status, as regulatory and contractual documents, for example, tend to be associated with the required authority and intent to regulate evaluation behaviors.

The analytical model proposed in this article provides the means to unlock the intellectual and institutional nature of different frameworks for evaluation, thanks to a broad characterization occurring at the intersection of its four parameters. This ultimately leads to a better understanding and more conscious choices about them.

Clarifying the definition of frameworks for evaluation is but one step in the study of their role in the evaluation field. First, in descriptive terms, the fundamental distinctions established in this article can lead to a more refined approach to frameworks’ empirical account, either by classifying different frameworks with additional precision or by recognizing the presence of various institutional and non-institutional frameworks in a single document. Second, the model developed here can contribute to the study of frameworks’ influence on practice: the institutional/non-institutional divide as well as the three types of institutional statements force us to consider entirely different objects in causal terms: simple ideas about practice (non-institutional) are not likely to provoke the same consequences on evaluation behavior as fully enforced institutions. These distinctions can be associated with relevant causal theories from the social sciences to eventually calibrate our expectations and inform individuals’ and organizations’ choices of frameworks. Third, having identified general types of frameworks, it is now possible to discuss what makes a framework valuable within its own category. Prescriptions about frameworks cannot be entirely universal, and the distinctions provided by this model offer a way to identify varying criteria of merit.

In practice, a framework has an impact because someone decides to adopt, adapt, or develop that framework in a given evaluation context. This leads in many cases to locally developed logic models, evaluation plans, evaluation policies, and many other products associated with the term “evaluation frameworks.” The further study of the intellectual quality of these locally developed frameworks and of their elaboration process is crucial for the field of evaluation as they represent ideas in action with a real potential for influence on the delivery of quality program evaluations. This is even more critical when frameworks have an institutional dimension, as they are actual interventions on the social phenomenon of evaluation. In this case, understanding the causal functioning of these regulatory interventions on evaluation behavior is key to improve our effectiveness in shaping evaluation toward our collective expectations.