Framing Evaluation in Reality: An Introduction to Ontologically Integrative Evaluation

Introduction

For over 30 years, informal and formal calls have been issued for the field of evaluation to address implicit bias in western ¹ evaluation theory and practice. More recently, the role of western designed evaluation methods in non-western settings and the field of evaluation's connection to systemic oppression sparked discussion among the American Evaluation Association (AEA) and African Evaluation Association (AfrEA) membership. Members drew attention to the field's historical neglect of institutionalized discrimination and called for a renewed commitment to honest conversation about the field's history and a move toward greater inclusivity of Indigenous knowledge and evaluation methodologies (Bowman, 2018; Njovu, 2018; Williams, 2018). Earlier, Scriven (2010) prophetically called for a reconceptualization of evaluation from the ground up (para 1), “a radical shift in the framework of our thinking about a substantial subject matter area, that is, a rejection and/or redefinition of the most fundamental assumptions” (para 2). Highlighting the field's fundamental ontological and epistemological assumptions, Shadish et al. (1991) argue that the field's “sins of omission or commission” (p. 43) resulted in failure to transparently address western ontological and epistemological assumptions and active oppression of other ways of knowing. This failure reflects the evaluation field's foundation of ontological and epistemological agnotology—that is, the cultural production of ignorance (Proctor, 2008)—making Shadish et al.’s (1991) warning of delayed resolutions for ontological, epistemological, methodological debates, and limited theoretical applicability and relevance (p. 43) prophetic.

Delaying resolution of ontological and epistemological agnotology results in several risks to the field of evaluation. These risks include: (a) unpreparedness of western-trained evaluators engaging non-western communities with alternate knowledge systems (House, 2015; Shadish & Cook, 1998); (b) alienation of Indigenous evaluators from the mainstream evaluation discourse (Bowman, 2018; Njovu, 2018; Williams, 2018); (c) irrelevancy and uselessness of evaluation findings to the communities in which programs operate (Contu & Girei, 2014); and (d) deception in M&E reporting practices, which threatens the reliability, validity, trustworthiness, and usefulness of the evaluation (Bornstein, 2006; Clark, 2010; Contu & Girei, 2014; Douma & Hilhorst, 2012; Hilhorst & Douma, 2018; Kite & Davy, 2015; Traore, 2016). Regrettably, “terms like ontology and epistemology bore many evaluators, because they conjure up images of sterile philosophical debates … [and] most evaluators see debates about ontology and epistemology as tangential to their work” (Shadish et al., 1991, p. 42), thus delaying risk mitigation.

The current focus on cultural competence and cultural responsiveness marks an attempt to mitigate these risks (Bamberger et al., 2012; Beisheimetal, 2014; Bird & Osland, 2006; Conner, 2004; Garakani & Peter, 2016; Hanberger, 2010; Hood, 2004; King et al., 2004; Kirkhart, 2010; Kurtz, 2013; LaFrance, 2004; LaFrance et al., 2012; LaFrance & Nichols, 2010; Lombe et al., 2013; Mato, 2011; Mertens, 2009; Mertens & Wilson, 2019; Murphy & Dixon, 2012; SenGupta et al., 2004; Symonette, 2004; Tilley & Gormley, 2007). Yet, while a step in the right direction, this shift does not guarantee ontological competence and risks more evaluative harm befalling non-western communities. Harm occurs when applying Hanberger's (2010) and/or AEA's (2011) standards of cultural competence without rigorous ontological interrogation, as doing so may result in no more than obeisance of the evaluator in adhering to local cultural protocols. Indeed, in their critical review of international development partnerships, Contu and Girei (2014) argue that the meaning of partnership as most often deployed perpetuates power inequalities and neoliberalism despite surface assumptions of “reciprocity, shared objectives, mutuality and dialogue” (p. 215). More damaging, cultural competence in some cases may even become a foil for western hegemony. Kincheloe and Steinberg (2008) describe this tension aptly when they state, “Walking the well-intentioned road to hell, Western scholars dedicated to the best interests of Indigenous peoples often unwittingly participate in the Western hegemonic process” (p. 141).

This risk—the perpetuation of western hegemonic processes grounded in a narrowed western ontology—threatens shared learning and co-creation of solutions to the wicked problems (Rittel & Webber, 1973)—socially complex, multicausal, solution-resistant problems (Hopson & Cram, 2018, p. 5)—of our day. To actualize solutions to these problems, we need to go beyond the paradigmatic address of the ontological assumptions within the field (Chilisa, 2012; Guba & Lincoln, 1989; Mertens & Wilson, 2019) and pursue ontological competence. Alluded to by only a few authors (Lovanio & Wallace, 2007; Meredith et al., 2012; Schaefer et al., 2012; Turton, 1997), ontological competence requires that we dig deeper to identify the ways in which our own ontologies impact our answers to questions regarding the planning, conducting, and reporting of research/evaluation (Mato, 2011). Here, I present a framework for conceptualizing ontological competence and a tool designed to help us enact it in our theory, practice, and axiology. Kovach (2009) calls on non-Indigenous researchers to “help create entry points for Indigenous knowledges to come through” (p. 7) and presents seven steps non-Indigenous academics can take to support Indigenous research and evaluation. Of these seven, this paper most directly supports the decolonization of western hearts and minds and the rejection of western-centric notions of knowledge built upon narrowed ontological claims.

Ontological Competence

AEA’s (2011) statement on cultural competence encourages evaluators to acknowledge the complexities of cultural identity, recognize the dynamics of power, eliminate bias in language, and employ culturally appropriate methods. Although each is a vitally important step to take, the emphasis on cultural competence may serve as a red herring, distracting us from the more challenging goal of ontological competence ² . Despite the address of paradigmatic ontological assumptions within the field (Chilisa, 2012; Guba & Lincoln, 1989; Mertens & Wilson, 2019), ontological competence remains an underdeveloped concept.

Ontological competence requires ontological reflexivity and responsiveness. It requires that we (1) continuously interrogate our ontological stance, (2) be open to changes in our ontological stance, (3) be knowledgeable and respectful of the ontological stance of others, and (4) commit to not privileging our ontological stance over that of others. Ontological competence goes beyond the ontological assumption of a “multifaceted world” (Mertens & Wilson, 2019, p. 165) grounded in social, cultural, and historical realities and asks us to examine the foundational, metaphysical ontologies from which these realities emerge, that is, the subconscious ↔ conscious and immaterial (spiritual) ↔ material dimensions. Despite the challenges of representing metaphysical multidimensionality in print, Figure 1 serves as a conceptual guide to assist with the interrogation of ontological positionality, both personally and paradigmatically. In locating our comfort zone within these ontological dimensions, we begin the process of thinking critically about our ontological location and the forces that direct our engagement with each dimension.

OPEN IN VIEWER

Figure 1.

Reality dimensions concept map.

For many trained within the secularized west, ontological comfort zones largely reside within the material-conscious reality dimensions. This boundedness threatens ontological competence and creates tension when a western-trained evaluator works among or with others affirming a broader ontology. In this case, neglecting ontological competence leaves questions about the foundational differences between peoples’ understanding of what counts as real and what counts as knowledge left unaddressed and threatens the reliability, validity, trustworthiness, and usefulness of the evaluation (Clark, 2010; Contu & Girei, 2014; Douma & Hilhorst, 2012; Hilhorst & Douma, 2018; Kite & Davy, 2015; Traore, 2016).

In contrast, ontological (and epistemological) concerns are fundamental to Indigenous scholars (Chilisa, 2012; Pierotti, 2011; Smith, 2012; Wilson, 2008). Chilisa (2012) points out, “The researcher's perceptions of reality, what counts as knowledge and values, have an impact on the way research questions are conceived, research approaches, data-gathering instruments, analysis, and interpretation and dissemination of research findings” (p. 2). Emphasizing the validity of immaterial realities (i.e., spiritual realities)—a crucial element to Indigenous frameworks (Hopson & Cram, 2018; Smith, 2012)—Kovach (2009) writes, “The spiritual, holistic nature of Indigenous knowledges can be problematic for the more traditional, empiricist approach to knowledge. Not all academic researchers will embrace Indigenous knowledges if doing so is too far from their level of comfort” (p. 170).

Given this, to what extent can western and non-western evaluators function collaboratively to achieve evaluative goals (e.g., the SDGs) when using approaches and methodologies grounded in conflicting ontologies? Although numerous evaluation approaches advance cross-cultural collaboration and competence (e.g., transformative, culturally responsive, utilization focused, and empowerment evaluation), both may not go far enough to reverse methodological colonialism (Contu & Girei, 2014; Dohn, 2014; Smith, 2012; Wilson, 2008). Speaking as both a natural scientist and evaluator, here I argue that greater inclusivity of Indigenous knowledge and evaluation methodologies requires deep interrogation of the ontological assumptions undergirding western evaluation's epistemologies, methodologies, theories, and axiology.

The Nature of Reality

The current crisis in the field of evaluation—not tapping into the fullness of human capacity for understanding (G. Julnes, personal communication, October 4, 2021)—rests upon ontological disagreements over expressed certainty of that which exists and is known. These disagreements center on the validity of knowledge claims falling outside the conventional western naturalistic framework. Hofweber (2014) states that ontology encompasses the study of what there is, the general features thereof, and questions concerning these general features (p. 12). What there is, that is, what is real, is that which exists and that which exists is contextually knowable—we know something exists because we engage it within a specific context. Simply put, existence is contextual interaction. Furthermore, although interaction may be unknown by the interacting entities at a given time, it is potentially knowable at some level (e.g., molecular, cellular, individual, ecological, and organizational) given the availability of the appropriate instruments (internal or external) of perception (conscious or subconscious) to measure the interaction and the necessary pathways of communication to relay the information. Thus, that which exists may be known but is not necessarily known and that which does not exist is not known and cannot be known. Therefore, we may express certainty over that which exists and is known, but we cannot express certainty over that which exists but is unknown nor that which does not exist.

If we accept that what exists is contextually knowable, then we must also accept that our knowledge of what exists is context dependent; that is, dependent on our position in time and space and the availability of the necessary perception, measurement, and communication tools (Russell, 2010). Thus, based on our positionality, we can know what exists, but we can never be certain of what does not exist. Moreover, “what exists” exists regardless of our knowledge of it. Therefore, the question of existence is more a question of knowledge than a question of existence, for it is knowledge of an entity that brings that entity out of the shadows of existence and into the subconscious or conscious known reality.

Ultimately, our ability to perceive and measure interaction limits our subconscious or conscious awareness of, subsequent knowledge of, and reaction to what exists. While interactions occur within individuals at a subcellular and cellular level at the unconscious level, here we are interested in interactions between entities that relay information to our subconscious or conscious awareness and result in further interactions which either negate, nullify, or intensify the original interaction. Neuroscience research suggests that decision-making rests within our subconscious mind, not our conscious mind, and that subconscious brain activity “specifically encodes how the subject is going to decide” (Soon et al., 2008, p. 545). This suggests that knowledge or recognition of an entity need not be known to us at a conscious level for it to impact our decision-making. For this reason, our understanding of existence requires a contextualized subconscious or conscious awareness—incorporated easily in Indigenous paradigms, yet a challenge for western conscious-bound, material paradigms.

Generally, western ontologies accept evidence of what exists primarily based on sensory interactions with an object, and western philosophy rests upon this foundation of empiricism. Positivism, representing the most bounded knowledge paradigm, epitomizes this empirical approach and limits existence to that which can be physically perceived through the senses at a conscious level. However, the lack of sensory engagement need not rule out the possibility of existence, and sensory engagement alone may not guarantee existence if our sensory perceptions deceive us. Because of this, most current research in the social and natural sciences operates within the post-positivist paradigm and adheres to the principle of falsifiability (Popper, 1959/2002). Paradoxically, despite this foundational commitment to falsifiability within the western knowledge framework, western ontology purports that apart from empirical knowledge all other knowledge claims are false. Thus, in practice western-derived knowledge systems remain grounded in narrow conceptions of a material reality, bounded by limited definitions of what counts as sensory observation and therefore what counts as acceptable evidence of existence. Challenging both positivism and post-positivism's ontology, constructivism and its offshoots (e.g., interpretivism, critical realism) define all reality as socially constructed and emphasize the contextualization of knowledge production. Despite the clear differences between post-positivism and constructivism, both evolved from within a western philosophical framework and fall short of full ontological interrogation (Kovach, 2009).

At the most basic level, awareness of ontological frameworks that embrace subconscious ↔ conscious and immaterial ↔ material reality spectrums better position western evaluators working in non-western communities to serve program stakeholders. When working within non-western communities, if evaluation protocols do not incorporate Indigenous ontologies, epistemologies, and methodologies, then conclusions regarding program efficacy risk misrepresenting participants’ lived experiences with the program and the true impact thereof (Contu & Girei, 2014; LaFrance, 2004). To determine this effect, evaluators seek to know and understand that which exists (e.g., a program and its inputs, outputs, outcomes, and stakeholders) as demonstrated through engagement with the program and participants. To be relevant, this evaluation work—that is, the determination of that which exists—requires ontological grounding and flexibility.

Considering that evaluation is fundamentally a determination of what exists and how we engage that existence, evaluators need to interrogate their own perceptions of reality and ways of knowing the world before engaging with communities that hold to different ontological and epistemological principles. Unfortunately, an emphasis on evaluation practice over evaluation philosophy steers many away from interrogating the ontological foundation of their training. Review of the leading western evaluation texts (e.g., Alkin, 2013; Bamberger et al., 2012; Rossi et al., 2019; Russ-Eft & Preskill, 2009; Stufflebeam & Coryn, 2014), reveals this emphasis on practice. Even Mertens and Wilson (2019) devote only a few paragraphs to ontological considerations despite acknowledging that most debates within the field arise from ontological differences. When comparing the ontological and epistemological focus of non-western evaluation texts (Chilisa, 2012; Kovach, 2009; Wilson, 2008) to western evaluation texts, the ontological silence within the latter becomes even more pronounced.

The assumption of a shared secularized, material ontology privileges western ways of knowing and alienates those embracing multiple realities and multiple ways of knowing (LaFrance, 2004). Given the great risk this poses to non-western communities, the paucity of deep ontological considerations in the western evaluation literature and the current call for greater flexibility in evaluative thinking, the evaluation field may benefit from a framework that explicitly addresses ontological diversity and challenges evaluators to interrogate their own ontological assumptions while encouraging them to be welcoming to that of others.

In response to this, I introduce Ontologically Integrative Evaluation (OIE) and propose six ontological standards for evaluation. OIE supports the work of many who have contributed significantly to the production of evaluation approaches designed to give voice to the underrepresented and marginalized (e.g., Brisolara et al., 2014; Fetterman, 1994; LaFrance & Nichols, 2010; Mertens, 2009) and seeks to advance the 5Rs of evaluation—relationships, responsibility, relevance, restoration, and reverence (Hoffman, 2013; Hopson & Cram, 2018). Acknowledging these significant contributions, OIE partners with them in providing evaluators a systematic way to explicitly address ontological assumptions such that space is created for multiple ontologies within the evaluation.

Six Ontological Standards for Evaluation

To address the lack of attention given to ontological negotiations within the field of evaluation and to provide western-trained evaluators a tool to systematically interrogate their ontological leanings, OIE advances ontological competence and grounds it upon six ontological standards for evaluation.

Ontological Fluidity: Ontological fluidity refers to the shift in one's ontological stance that occurs over time in response to changes in position/location and subsequent new interactions with the material and nonmaterial world. OIE recognizes the contextualized nature of ontological commitment and the need for continual ontological reflection and interrogation as context changes.

The OIE Concept Model

While ontological questions traditionally focus on “what there is” (i.e., what exists), Hofweber (2014) points out that the question of “what there is” is not necessarily the same as asking “what is real” or even “what is fundamental” (p. 15). Pondering “what there is” requires engagement with an entity to objectively address the existence of that entity while “what is real” requires an understanding of our own and others’ subjective experiences with “what there is.” Through the interconnection of “what there is” and “what is real,” questions regarding “what is fundamental,” that is, what is necessary, can then be addressed. OIE, as shown in Figure 2, systematically cycles through these three ontological considerations—what there is, what is real, and what is fundamental—and applies them to each of the phases of program evaluation. In seeking answers to these questions, OIE makes explicit what otherwise may remain implicit, that is, ontological biases. Such ontological interrogation goes beyond the paradigmatic recognition of single/multiple and objective/subjective realities, requiring interrogation of the deeper ontological foundations of these realities. Below, I introduce the philosophical rationale for each phase of the OIE framework. ³ Representing OIE's ontological integration, Table 1 emphasizes the need for evaluators to be attentive to empirical data, people's experiences, and relationships both between and with people, place, and power and thus serve to help explicate OIE's three phases.

OPEN IN VIEWER

Figure 2.

Ontologically Integrative Evaluation (OIE) concept model.

OPEN IN VIEWER

Table 1.

Ontological integration within Ontologically Integrative Evaluation (OIE).

Ontologically Integrative Evaluation	Phase I: What there is Projected Realities	Phase II: What is real Perceived Realities	Phase III: What is fundamental Principal Realities
Ontology
Post-positivist	Single reality grounded in material context	Single, objective experience	Data driven
Constructivist	Multiple constructed realities grounded in material & social context	Multiple, subjective experiences	Experience driven
Indigenous	Multiple constructed realities grounded in material, social, & spiritual context	Multiple, subjective experiences bounded together in relationship between living and nonliving	Relationship driven
Transformative	Multiple constructed realities grounded in historical realism	Multiple, subjective experiences impacted by power and privilege	Human rights driven

OIE Phase I: “What There Is” and Projected Realities

In alignment with an evaluation's preparation phase, evaluators first engage “what there is” through an examination of a program's projected realities. Projected realities consist of all the objective manifestations of a program that project into the material and virtual space external to the program. As such, these projected realities are accessible to any observer and not limited to a single mode of representation. For example, these projections may appear as billboards, websites, blogs, podcasts, commercials, archival documents, or promotional materials. Regardless of their manifestation, projected realities often extend locally, regionally, or globally, as they traverse through virtual and material space. Because projected realities are in motion, traveling out from programs into external spaces, they may be transformed by the medium of the space through which they travel. Yet, at any single point in time, an observer engages an objective representation of the program (e.g., picks up a program flyer or listens to a podcast)—a projected reality. Thus, projected realities represent objective manifestations of programs shaped by subjective space.

Acknowledging that OIE Phase I accepts that programs possess material elements, we can turn to material science for help in understanding the complexities of projected realities and their relevance to evaluation. From physics, we know that light or sound traveling through different mediums will experience changes in speed, direction, and form due to refraction, and the degree of difference between the mediums determines the degree of these changes. To explain, consider two identical pencils resting on a table side by side. Both pencils appear linear and cylindrical. Now, fill a clear glass with water and place one of the pencils vertically in the glass of water so that only half of the pencil is submerged. Examine the pencil, paying close attention to its appearance at the water–air interface. The identical pencil sitting on the table still appears as a uniformly linear cylindrical object, but the pencil partially submerged in water, that is, positioned within two different mediums, now appears as two objects misaligned with one another. In both scenarios, the object under consideration, the pencil, exists, but the visual projection of its existence varies due to the variances in the surrounding medium which impact the object's projected realities. In the case of the pencil, a single unified projected reality becomes multiple realities when its positionality changes. This simple exercise demonstrates that the context of an object determines its projected reality and that changes in context may lead to multiple projected realities, each being distortions of the other.

Applying this to program evaluation, as we accept that the pencil exists, the evaluator accepts that a program exists. This existence is verified through interaction between the evaluator and the program, that is, interaction with the projected realities of the program. But, like the pencil, the projections evaluators interact with may vary dependent upon the mediums through which they travel (e.g., social media, archival documents, and organizational websites). Therefore, awareness of a program's transmitted projected realities depends on both the context of the program and the context of the evaluator—shifting either positionality changes the projected realities. Additionally, while projected realities exist in material space, they need not originate from material space as some may form in transit through virtual space. Given the plethora of mediums through which projected realities flow, OIE Phase I challenges evaluators to explore diverse mediums (e.g., social media, website, public documents, etc.) when seeking to understand a program's projected realities prior to direct engagement with the program.

Having established that a program's projected realities depend upon the medium of transmission, OIE Phase I not only calls an evaluator to acknowledge these various transmitted projected realities, but also to consider further complexities of projection. For one, although projected realities may project from varying contexts (i.e., material, social, spiritual), they may not differ from each other if they transmit through the same medium. Therefore, we must consider that a single projected reality may not have a single source of projection, that is, a single projected reality may result from the convergence of individual projected realities traveling through the same or different mediums and converging with one another. Furthermore, a single projected reality may itself pass through multiple mediums or the same medium in a new manner and exhibit divergence when observed.

Mertens (2009) uses the prism as a metaphor for transformative evaluation, a metaphor useful here as well. Consider white light as it passes through a prism. Before transmitting through the prism, the white light projects as a single entity, white light. If transmitting directly through the prism, the white light retains its white light projected reality. However, upon transmission through the prism at an angle, the white light separates into multiple wavelengths of color. Thus, what can be perceived as a single reality, white light, can also be disaggregated and perceived as multiple realities, colors of many wavelengths. Analogous to this divergence of wavelengths, a program may project a singular reality when transmitted through a given medium in a specific manner, but when transmitted differently through the same medium (e.g., multiple pathways within a single website) or through a different medium (e.g., brochure vs. video) divergence occurs and multiple projected realities emerge. These examples demonstrate that the nature of the medium and the interaction of the program and the medium determine if the projected reality is identical to or a distortion of its source.

To build upon these complexities, consider that projected realities may only provide a composite perspective of a program. Programs themselves contain many components (e.g., organizational units, stakeholders, etc.) and each component projects its own reality which may or may not be perceived when transmitted through the same medium as the composite program. However, these internal projections, although impacting the external projected realities, are not accessible to the evaluator prior to direct engagement with the program. Given this, OIE Phase I focuses on the externally projected realities while acknowledging these internal projections. Key to this stage, then, is awareness of the dimensionality of a program's projected reality, a commitment to examination of the many transmission mediums through which a program's projections pass, and consideration of how these projections interact with one another.

Thus, at any given point in time, projected realities take on objective identities shaped by subjective spaces in which numerous projected realities may interact. Recognizing this, OIE Phase I integrates the objective collection of program artifacts characteristic of empirically centered post-positivism, the continuous self-interrogation characteristic of constructivism, the relational contextuality and accountability characteristic of Indigenous paradigms, and the acknowledgment of the impact of power and privilege on reality constructions characteristic of the transformative paradigm. Grounded in an unbounded ontological perspective, OIE engages material and spiritual and conscious and subconscious representations of “what there is” and encourages evaluators to consider multiple mediums of projection as well as multiple source points of projection. This grounding facilitates the interrogation of western assumptions about reality which accept either one knowable reality or multiple socially constructed realities, but which deemphasize realities based on interactions between the material/immaterial and the biotic/abiotic.

Through such ontological integration, evaluators position themselves to better understand “what there is” during the preparation phase of the evaluation. However, given the dynamic nature of projected realities, OIE Phase 1 must be revisited throughout the evaluation to ensure the evaluation represents the living nature of the program. Because projected realities traverse through a program's external environment and may undergo transformations along the way, after identifying projected realities the next step of OIE addresses “what is real.”

OIE Phase II: “What Is Real” and Perceived Realities

Following the initial phase of determining “what there is,” evaluators next must determine “what is real” through an examination of stakeholders’ programmatic perceived realities. OIE Phase II explores the subjective experiences of stakeholders through examination of stakeholder perceptions of the program, that is, perceived realities. While individuals disconnected from the program may encounter the program's projected realities (e.g., picking up a flyer or visiting a program's website before making direct contact with program staff or participants), only the perceptions of individuals engaged at some level with the program contribute meaningfully to an evaluative understanding of “what is real.” Forming within an entity and grounded in experiences, perceived realities become known to evaluators as they interact with stakeholders during the design, sampling, and data collection phases of an evaluation. Because of the subjective nature of perceived realities, within any given program the number of multiple perceived realities may be quite numerous. Based upon an objective experience common to many which becomes subjectively embedded within an individual, perceived realities reflect objective experiences interpreted through an individual's beliefs and values. In this way, perceived realities originate from within individuals and represent interactions between perceived realities of the past and perceived realities of the present. As such, the perceived reality exists apart from the objective reality yet interacts with it. Collectively, then, perceived realities reflect the integration of experiences and the valuations thereof. Additionally, because ontological assumptions are grounded upon belief systems (Hofweber, 2014), examination of “what is real” reveals stakeholders’ belief systems which in turn reflect stakeholders’ value systems. Thus, to determine “what is real” evaluators must explore the beliefs and values of all program stakeholders, for a stakeholder's value system influences how (or if) they engage a program, how long they remain involved with the program, and how they perceive the program's efficacy.

Indeed, the success of any program is determined by stakeholder engagement and response to the program and is thus intricately tied to individual values and beliefs, both of which reflect ontological assumptions. Consider the participant stakeholder. Participants do not enter programs as empty slates (i.e., tabula rasa), but rather as individuals with belief and value systems previously formed. Ultimately, programs aim to elicit a behavior change (or to reinforce a behavior) in participants and behavior changes require shifts in beliefs and values. For example, consider a microenterprise development program that aims to increase participants’ financial self-sufficiency. If a participant values community wealth over individual wealth, a participant may perceive the program a success because it enabled them to contribute more to the financial security of their community even though a program evaluation may determine the program unsuccessful because the participant's held individual wealth seemed to minimally improve. In this example, the participant's valuation of relationships reflects their relational ontology and a program advancing individualism seems ill-fit in such a community. The underlying value assumption that financial independence or self-sufficiency is good directly contrasts with the participant's underlying value assumption that community wealth is good. Furthermore, the program, developed on the value of individualism, implicitly advances the notion that financial interdependency is bad. When applying OIE to this program, determining stakeholders’ perceived realities and comparing them to the program's projected realities reveals the underlying beliefs and values of all stakeholders and assists the evaluator in identifying the multidimensionality of “what is real” within the program.

Given that perceived realities are experiential and that experiences become encoded in individuals as memories, we can look to neuroscience to assist in our understanding of OIE Phase II. From neuroscience, we know that memories may or may not be identical to the objective reality which triggers them. In fact, not requiring a complete experience to apply meaning to it, the human brain heuristically fills in sensory gaps to create a perception of an experience based on past experiences. Indeed, it is this operational heuristic strategy of the brain that causes one experience to remind us of another, or that accounts for two individuals having the same objective experience but possessing drastically different perceived realities of that experience. Through a process of iterative comparisons, the brain “makes sense” of incoming experiences coded as sensory input and creates a perceived reality from that input. These perceptions can be both conscious and subconscious, the latter more deeply embedded and longer retained than the former. Neuroscientists call this subconscious perceptual memory nondeclarative memory and attribute the inner work of valuation to it. That is, assigning value to a perception occurs largely in the subconscious and is linked to valuations of past experiences (Squire & Dede, 2015, p. 8). What this tells us regarding perceived reality is that its creation and valuation occurs at the subconscious level and reflects past experiences. Thus, as an objective experience is engaged, an individual subconsciously interprets the experience as a perceived reality of that experience and affixes value to it. Because of this link between perception, valuation, and the subconscious, OIE Phase II challenges western notions of reality limited to material consciousness and directs evaluators to account for more expansive perceived realities of stakeholders when determining their programmatic valuation.

Furthermore, understanding that values reflect beliefs and that beliefs in turn reflect ontological assumptions, it behooves evaluators to explore both their own and others’ ontological commitments. Evaluators must be able to follow the trail from ontological assumptions to beliefs to values so they can meaningfully evaluate a program's efficacy. Evaluating programs that seek to invoke behavior changes (i.e., a value change) requires ontological authenticity (Chilisa, 2012; Guba & Lincoln, 1989; Lincoln & Guba, 2000). To be ontologically authentic, evaluators must acknowledge the value systems of all stakeholders as revealed through their perceptions of “what is real” about the program, that is, the perceived realities.

Differing in their understanding of “what is real,” the major paradigms when integrated can provide a pathway to better understanding stakeholders’ perceived realities. Because OIE Phase II focuses on the subjective experiences of stakeholders within the program, it most closely aligns with constructivist, Indigenous, and transformative ontological frameworks while still recognizing that programs consist of concrete, objective experiences as emphasized by the post-positivist paradigm. Yet, because each time an objective experience is encountered and a subjective experience becomes linked to it, applying a constructivist lens draws evaluators into an examination of these linked perceived realities and how they influence an individual's engagement with a program. Additionally, considering that programs and individuals exist within complex socio-politico-cultural ecosystems comprised of both biotic and abiotic entities, Indigenous ontologies draw an evaluator's attention to more expansive interactions which assist in determination of “what is real.” For example, the same program offered in different ecological environments may result in different perceived realities based on the varying physical (abiotic) settings of the program if the setting influences a participant's perceived reality. Indigenous ontologies also provide a space for inclusion of spiritual–material interactions as exemplified as a perceived reality formed through a revelatory experience. This revelatory experience, whether occurring as part of the program or outside of it, becomes integrated into the network of experiences the brain uses to construct an individual's perception of reality. Thus, Indigenous paradigms encourage evaluators to consider how relationships between the material-immaterial and the abiotic-biotic impact stakeholder perceptions. Furthermore, interrogating perceived realities through the lens of power and privilege as called for by the transformative paradigm broadens evaluator understanding of the contexts, needs, and priorities of all stakeholders.

To summarize, OIE Phase II acknowledges the single, objective experience and links that experience to a perceived reality of a program stakeholder. Recognizing that a single, objective experience (e.g., all participants completing the same activity) may manifest as multiple perceived realities, OIE accounts for these differences as varying perceived realities grounded in varying held ontologies. As previously demonstrated, different individuals may perceive the same program in different ways, which suggests that any programmatic effect rests more on the perceived reality of programmatic experience than on the objective experience common to all. Only after evaluators have gathered these perceived realities, can they begin to address the third ontological consideration, “what is fundamental.”

OIE Phase III: “What Is Fundamental” and Principal Realities

The third phase of OIE addresses the key ontological question “what is fundamental” by identifying the principal realities of the program. Principal realities are those realities necessary for the program to be successful and are determined during the data analysis phase of an evaluation. As many programs operate under time and resource constraints, program evaluations grounded in “what is fundamental” aim to identify those program attributes deemed essential (i.e., fundamentally necessary) for the program to achieve its short and long-range outcomes. As such these attributes represent the principal realities associated with the program—the projected and perceived realities necessary for program efficacy. Having identified the projected and perceived realities in OIE Phase I and II respectively, the goal of analysis during OIE Phase III is to locate points of intersection among stakeholders’ experiences (perceived realities) and program attributes (projected realities). These points of intersection—alignment between “what there is” and “what is real”—represent “what is fundamental,” that is, the principal realities of the program which are of central importance and without which the program would be nonoperational and ineffective. In this manner, principal realities emerge from the intersection or relationship between projected realities and perceived realities.

Approaching program evaluation through an ontological framework that explores the relationships between projected and perceived realities to reveal the program's principal realities serves to identify the ontological assumptions of all involved with the program, inclusive of the evaluator. With a focus on the relationships between realities, OIE Phase III calls evaluators to seek an integrative understanding of the program as a network of interacting realities. Through this process, an evaluator may discover that the actualized outcomes of the program differ from the stated/desired program outcomes (i.e., perceived realities differ from projected realities) or that stakeholders hold varying valuations of program outcomes (i.e., variation between perceived realities).

Interactions between perceived and projected realities influence an evaluator's determination of a program's merit, worth, and significance. For example, consider two youth development programs, a sport for development and peace (SDP) program and an English language literacy program. The SDP program which markets decreased antagonistic interactions between formerly combatant youth may result in increased conflict between youth due to the competitive nature of sport. Or the English language literacy program aimed at increasing participants’ mastery and use of the English language may result in youth networking with other native speakers and distancing from the language assimilation goals of the program. In both cases, program administrators and funders may determine the program unsuccessful despite participant youths’ determination that the program succeeded in meeting their social needs. Because of such varying valuations of perceived realities, OIE Phase III encourages evaluators to ontologically interrogate both unintended and intended outcomes when determining “what is fundamental.”

With an emphasis on relationships, OIE Phase III identifies principal realities through the integration of objective data (reflective of single, objective experiences), experiential data (reflective of multiple, subjective experiences), and relational data (reflective of interwoven experiences). To examine this integration, let us revisit the SDP program. Consider output data indicating the number of youths participating and the number of antagonistic interactions that occur during a soccer match (e.g., number of yellow or red cards issued). This type of objective data, characteristic of post-positivism, provides a baseline for understanding principal realities and often is represented by projected realities in the form of organizational records or marketing materials. In this case, the SDP program's website may represent this objective reality as a graphic showing a decrease in violent acts between youths on the pitch, suggesting that sport effectively promotes peace. However, through the collection of experiential data and integration of the subjective perceived realities of youth participants, it may become clear that the decrease in violence resulted not from sport itself but from other interventions the program provided. In this case, divergence between projected and perceived realities reveals a lack of consensus regarding the program's principal realities and requires that the evaluator dig deeper into “what is fundamental.”

Determination of “what is fundamental” requires that an evaluator take caution not to simply identify perceived realities. Rather, in alignment with the Indigenous paradigm, deep analysis of the relationships between perceived realities and projected realities guards against program misrepresentation. In the SDP example, program administrators may perceive that sport effectively decreases violence among formerly combatant youths and may move to divert more programmatic funds into sporting equipment and/or coaching staff. Whereas youth may perceive that the provisioning of food, shelter, and peer/adult mentors provides a secure, safe space for them to coexist, removing the external pressures that contribute to violent interactions. Or a youth may share a revelatory experience that compelled a behavior change independent of program participation. Exploring the relationships between these varying perceived realities, coupled by their projected realities (e.g., youth and program social media postings), creates space for the emergence of principal realities. For example, the integration of projected and perceived realities may reveal that meeting the material and emotional needs of youth off the field increases their trust and acceptance of one another as manifested by greater sportsmanship on the field. In this way, OIE Phase III analysis moves sport from its centrality and replaces it with material and emotional support as the principal reality upon which the program should be further developed.

Through such exploration of the divergence/convergence between projected and perceived realities, principal realities emerge. In this way, OIE's focus on the intersection between projected and perceived realities embraces Indigenous and constructivist ontologies, while not neglecting the existence of objective realities. Viewing these principal realities through a transformative ontological lens, evaluators can then interrogate “what is fundamental” to determine how principal realities contribute to the well-being of various stakeholders. Capable of expanding to include other paradigms, OIE requires that evaluators explicitly identify the ontological paradigms they are integrating and take account of the ways the integration directs each phase of the evaluation.

Concluding Remarks

In practice, OIE cycles through the traditional stages of evaluation—preparation, data collection, data analysis, reporting, and use—but engages each stage through the lens of ontology. Designed to bring to light evaluators own and others’ ontological assumptions, application of OIE moves evaluators iteratively through the evaluation process with constant return to the three underlying ontological considerations: what there is, what is real, and what is fundamental. These three questions when systematically addressed reveal three categories of program realities: projected realties, perceived realities, and principal realities. Through its emphasis on the foundational ontological questions and how they impact the act of evaluating, OIE encourages evaluators to embrace multilogicality in their evaluation practice and provides a guide for doing so.

Unfortunately, too often ontological assumptions go unexamined during program evaluation. Because rigorous ontological interrogation remains outside the most popular evaluation approaches, even evaluators sensitive to ontological diversity lack support to maintain an ontological emphasis throughout the evaluation. As a result, program evaluations conducted within a narrowly defined western ontological framework introduce risks of unpreparedness, alienation, irrelevancy, and deception. Thus, rather than aligning an approach/paradigm to a specific ontology, I propose that grounding an evaluation—regardless of selected approach/paradigm—upon the three ontological questions better supports the work of evaluators committed to social and restorative justice. By focusing on key ontological questions throughout the evaluation process, OIE continuously prompts evaluators to interrogate their own ontological assumptions and to consider the ontological stance of others. Particularly where western-based organizations operate programs in non-western communities, OIE may help to move evaluators along the path from cultural competence toward ontological competence in ways that preferentially benefit Indigenous evaluators and marginalized communities.

Stating that “it may be counterproductive to reject applicable Western ideas simply because they originate from the West” (p.178), Chilisa (2012) opens the door for the integration of best practices from post-positivist, constructivist, Indigenous, and transformative paradigms as encouraged by OIE. Not linked to any single evaluation approach, OIE provides space for ontological fluidity and synthesis while upholding to a systematic evaluative approach that is critical and self-reflective. Grounded in the six Ontological Standards for Evaluation and incorporating a constant return to and interaction with the grounding ontological questions, “what there is,” “what is real,” and “what is fundamental,” OIE approaches evaluation from an ontologically integrated and iterative framework and helps to ensure an evaluation's ontological validity. While originally conceived for western evaluators as a tool to advance ontological authenticity, OIE's potential may extend to all evaluative work as all realities are contextually known. At its core, OIE seeks to provide a philosophical foundation for evaluators seeking ontological justice through their practice such that they fulfill their ontological vocation.