Social Ontology and Evaluation— A Comment on “Framing Evaluation in Reality: An Introduction to Ontologically Integrative Evaluation”

The premise of Jennifer Billman's article is startling but plausible: western evaluators are ontologically incompetent and, as a result, a pervasive bias against non-western ways of thinking characterises the contemporary Evaluation Society (Dahler-Larsen, 2012). This is a serious charge since evaluation is fundamentally ontological: its remit consists of making claims about what “it” is (the evaluand), whether “it” is real and how fundamental “it” is.

There is little doubt that the prevailing western evaluation consensus ignores subconscious and immaterial realities and rejects the spiritual dimensions of indigenous thinking, its close connection to nature, and its communitarian ethics. It privileges a Cartesian approach to science. Thus, evaluators committed to the view that randomised control trials are a gold standard dominated evaluation practice at the creation of the discipline and they remain influential among neo-liberal evaluation commissioners.

Furthermore, in today's global market society, fee dependent evaluation practitioners are not inclined to question the social relevance of intervention goals and they mostly view the detrimental environmental consequences of economic growth as inevitable side effects. Nor do they normally challenge the vested interests that commission evaluations or resist the lure of indicator fetishism associated with goal achievement evaluation. Unsurprisingly, even culturally sensitive evaluators are prone to question the validity and reliability of indigenous knowledge frameworks.

This said, the overarching solution offered by Jennifer Billman (this issue, pp. 90–108) i.e., the adoption of an Ontologically Integrative Evaluation (OIE) model, leaves a lot to be desired. To be sure, no one can object to evaluation conceived as a vocation committed to fluidity, authenticity, validity, synthesis, and justice. But these highly abstract notions do not readily translate into serviceable good practice guidelines. Nor can they help resolve a host of thorny ontological questions that have so far eluded broad-based agreement among eminent philosophers.

The intersection between logic and ontology is riddled with unresolved dilemmas about the limits of language; the challenges associated with the study of “what there is” (e.g., is there a God?); the inconclusive debates about the necessity of “scientific” verification; the manifold obstacles that stand in the way of ascertaining what is fundamental; the learned disagreements as to whether the structure of the world can be objectively established and/or whether it is only a reflection of subjective processes, etc.

Billman's elusive OIE model fails to address or resolve these basic philosophical questions. Nor does it probe the social dimension of ontology that lies at the very core of the evaluation enterprise. Yet, putting social ontology to work would help evaluators fulfil their remit and it would extricate them from the bizarre, confusing, and ultimately fruitless palavers that have discouraged evaluators to resort to philosophy when faced with the concrete questions elicited by their daily practice.

Specifically, the monolithic OIE model ignores the fact that evaluators’ beliefs about the nature of social reality vary widely. They fall into two major categories:

Evaluators for whom reality exists independent of the observer (the domain of empiricism, positivism, objectivism, realism, and foundationalism); and

Just as the constructivists, the post-positivists believe that evaluation cannot be value-neutral, i.e., they view bias as undesirable but inevitable given the diversity of human beliefs that shape the questions evaluators are mandated to address, as well as their distinctive approaches to the analysis of evidence, and its interpretation.

For the positivists, social facts can be studied through the quantitative methods of the physical sciences. Not so for the constructivists who favour qualitative methods and for whom social facts only exist as a construct of human perceptions, beliefs, and conventions embedded within political structures.

Crucially, the ontological “clockwork” conception of the universe that underlies the positivists’ epistemological stance does not recognize the dynamic complexity of a natural and human world that displays emergence, positive and negative feedbacks, self-organization, phase transitions, criticality, and chaotic behavior. Hence, calling on evaluators to “frame evaluation in reality” implies respect for indigenous evaluation and its corollary: familiarity with the new science of complexity.

Positivists have lost ground in academia, ¹ but they still hold their ground in mainstream evaluation practice. They focus on fact finding, data gathering, and theory testing. They purport to be value-free. They are not inclined to challenge the establishment. They act as evaluation consultants, advisers, or proactive facilitators. By contrast, the constructivists adopt a more independent stance since they recognize that any sensory experience must be mediated by the concepts wielded by the observer.

For the positivist social facts are “out there” while for the constructivist, they are constructed and need interpretation. This stark ontological difference led to paradigm wars that were eventually settled under the “big tent” of evaluation practice when advocacy of mixed methods allowed evaluators to choose their ontological affiliation (Datta, 1994).

This uneasy truce did not bridge the ontological divide between positivists and constructivists. ² But its pragmatic ambiguity had the singular advantage of facilitating the expansion of the evaluation occupation in the global market society and it did not stand in the way of claiming that evaluation was a distinctive discipline as well as a trans-discipline that informs and strengthens all the social sciences and draws on them as appropriate (Scriven, 1994).

Whereas positivists view evaluation as subject to the same rules as the physical sciences, constructivism holds that social inquiry is radically different from the exploration of nature: social science is distinctive and committed to the proposition that there is no objective social truth, i.e., social knowledge and truth are creations of the mind and do not exist independent of the observer (Schwandt, 1998).

For the constructivist, a priestly vestment is more than an item of clothing: it signals theological influence. A voting machine is more than a mechanical device: it is a valued instrument of electoral democracy. A ten-dollar bill is more than a piece of paper: it is a trusted means of exchange in the market economy. Hence, brutal natural facts that are independent of human perception are fundamentally different from stubborn social facts shaped by human values.

What most distinguishes social facts from natural facts is that they are clustered in institutions that define roles, norms, and expectations and provide the structures and processes that the performance of social functions requires. Institutions assign roles and codify behaviour. They shape the social environment and are shaped by it. They interact, evolve, adapt, and sometimes disappear to make room for new institutions. Evaluation is an institution.

The privileged role of institutions in society holds the key to the constructivist social ontology project (Hay, 2016). Institutions facilitate standardized and reciprocal patterns of human behaviour geared to the achievement of intended social outcomes. They combine hierarchy and participation. They emerge from complex interplays among individuals and groups in ways designed to facilitate cooperation, discourage free riding, and minimize transaction costs.

Institutions would not materialize absent tacit management of expectations and reliance on control processes. Ideas, paradigms, and conventions make for a highly contingent operating environment. This means that institutions evolve. Initial conditions are highly influential, path dependence contributes to social stability while complex feedback mechanisms contribute to volatility and uncertainty. As a result, social reality is shaped and reshaped by the changing perceptions of individuals and groups, a complex process.

For the constructivist, institutions are human constructions which interact in ways that generate unpredictability so that evaluation to be effective as a social institution in pursuit of the truth must be developmental and interactive (Patton, 2010). Furthermore, unlike positivist evaluation, constructivist evaluation is duty bound to confront power since power invariably generates its own truth and/or seek effective control over knowledge production (Foucault, 1980).

In sum, constructivism characterises progressive and principled evaluative inquiry. It interrogates power by unearthing truth and disseminating inconvenient social facts. It illuminates the collective action dilemmas that plague humanity. It resists capture. ³ It recognizes that resilient colonial mindsets must be banished by challenging the neo-liberal myths of contemporary capitalism. It currently co-exists with positivist evaluation, the approach favoured by decision makers.

This inconclusive outcome implies a long twilight struggle given the triumph of neoliberalism around the world. Neo-liberal commissioners define evaluation as a market good rather than a public good, thus facilitating evaluation capture (Mathison, 2018). Power holders issue terms of reference that privilege achievement of predetermined objectives. They resist stakeholders’ perspectives. They subscribe to the epistemological premise that value-free distancing from the object of study is essential to avoid bias. They hold on to what I contend are antiquated views about the superiority of quantitative methods. ⁴

By contrast, constructivist evaluators are highly sensitive to the context. They respect indigenous cultures and acknowledge that infinite and regenerative connections among open systems characterize the natural and social world. They oppose the conception of society conceived as a closed system that demands infinite growth through ruthless exploitation of natural resources. They embrace complexity thinking as a strategy for pushing back the frontiers of knowledge (Mikulecky, 2001).

Indeed, just as indigenous evaluation, constructivist evaluation gives pride of place to the relational context in the evaluation process. It rejects the illusion of scientific omniscience—just as quantum physicists do. It adopts transdisciplinary qualitative methods that give pride of place to genuine participation and dialogue. It promotes sustainability by respecting nature and all living beings. It works effectively in a wide range of cultural contexts. It puts interactions within the global, national. and local community at the centre of the evaluation stage in ways that amplify the voices of all stakeholders.

Fortunately, the evaluation community has begun to reconsider its practices, its policies, and its priorities (van den Berg et al., 2021). How then does constructivist ontology relate to indigenous evaluation? It paves the way for a rapprochement between western and non-western mindsets. A convergence between western and non-western ways of thinking would be a providential outcome given the existential threat of climate change and the imperative of evaluation transformation that it implies.

Although indigenous societies are highly diverse, they have many things in common. They seek peaceful relations with other groups. They share material goods. Their culture is egalitarian, and this prevents anyone from dominating. This high trust, cooperative culture contrasts with the competitive, adversary culture of a market society that has promoted economic growth at the expense of nature and remains committed to a linear, postpositivist conception of evaluation.

Hegemonic ideas have, for far too long, generated social injustice and environmental destruction. A decolonized evaluation agenda inspired by indigenous values would task evaluators to confront global forces, to work at the macro level to identify the interests being served, address transboundary issues, unpack the role of governments, corporations, and civil society organizations, and contribute to genuinely democratic reforms aimed at social equity and environmental sustainability.

Such an agenda would rely on “no strings attached” independent funding to break free from narrow cost-benefit assessment methods that treat environmental and social costs as inevitable side effects of providential economic growth. It would engage in meta-evaluation by subjecting influential evaluations to systematic assessment. It would unearth the interests they serve, expose their biases, unveil their value criteria, and examine their policy prescriptions through a sustainability lens.

In sum, future evaluation policy directions should push the boundaries of the current authorizing environment, break the chains of fee dependence, reject the global neoliberal tenets embedded in terms of reference defined by vested interests, and adapt the principles of indigenous evaluation to the unprecedented requirements of systemic change and global policy reform. This would amount to a fifth wave of evaluation diffusion (Vedung, 2010) that would put values at the very centre of the discipline, transform evaluation methods, and pioneer new evaluation governance models designed to protect evaluation from capture by vested interests.