Critical Essay: Meta-analysis: A critical realist critique and alternative

Ontology

Observed (empirical) events or states of affairs are the ultimate phenomena about which quasi-empiricists collect data – for example, size of organizations; presence of teamwork; being female; employee performance. If these events are observed (or proxied) in terms of quantity or degree, they become variables – that is, quantified events. The ontology consists, therefore, of observed events or states or affairs that are unique, unconnected or atomistic.

Epistemology

Whilst quantitative empirical researchers are probably aware that the variables they measure represent causal mechanisms, broadly conceived, their focus is always on the events these mechanisms generate. ⁶ If, as presumed, particular knowledge is gained through observing events, more general or ‘scientific’ knowledge is gained only if these events manifest themselves as regularities in the flux of events or states of affairs. ⁷ This is usually styled ‘whenever event x₁….x_n then event y’ or y = f(x₁….x_n).

Together, this ontological and this epistemological position implies a ‘flat’ ontology – the assumption that all that exists are events (or actions) and people’s perceptions of these events (Table 1).

OPEN IN VIEWER

Table 1.

A ‘flat’ ontology.

Domain	Entity
Empirical	Experiences and perceptions
Actual	Events and actions

Methods

The method of quasi-empiricism seeks to generate predictions, typically in the form of hypotheses to be refuted or supported via the collection of quantitative data. The only phenomena that feature in quantitative empirical research are those capable of being transposed into variables – that is, the quantified expression of events. What cannot be quantified adequately is omitted.

Aetiology

The notion of causation pre-supposed by quasi-empiricism is referred to as the regularity view of causation. As its ontology is of observed atomistic events, its concept of causality cannot be conceived of in terms of anything other than events and their regularity. As the epistemology of quasi-empiricism is reliant upon identifying event regularities, its conceptualization of causation requires knowledge of event regularities. To know the cause of increased organizational performance is, for example, to know that it is regularly preceded by the introduction of a bundle of HRM practices. More generally, to know the cause of event y, requires us to know (no more than) that event x, or events x₁, x₂…x_n, is/are regularly conjoined to event y.

It is worth adding that conclusions are often, usually implicitly, given a universal and general ‘twist’, along with a spurious precision. For example, in their analysis of performance related pay (PRP) (Gielen et al., 2010: 291) write that: ‘PRP increases productivity at the firm level by 9%’. It is not clear if this is understood as a ‘one off’, or whether this is supposedly generalizable to all firms. If the latter, the ‘9%’ looks to be an example of spurious precision.

Open and closed systems

The quasi-empiricist commitment to causality as regularities in the flux of events requires that social or organizational systems are theorized or modelled as if they are closed systems, defined thus: Parts of the socio-economic world characterized by regularities in the flux of events (or states of affairs) of the form ‘whenever event x then event y’, or y = f(x) are closed systems, and parts of this world not so characterized are open systems (see Bhaskar, 1978; Fleetwood, 2016; Lawson, 1995). Crucially, statistical techniques like regression analysis not only presuppose, but only work in, closed systems. Methodologically speaking, quantitative empirical researchers of organization studies must ‘engineer’ closed systems (only in theory, because a real open system such as an organization cannot be closed) so they can write things like:

Hypothesis 1: Empowerment-enhancing bundles [of HRM practices] will be positively correlated with business outcomes. (Subramony, 2009: 748)

This translates to ‘whenever empowerment-enhancing bundles (EEB), then business outcome’, or ‘outcome = f(EEB)’ and, by definition, this describes a closed system. Variations in regularity are generally specified probabilistically or stochastically, as random processes occurring in the ontic domain. Probability is a measure of the likelihood of an event occurring. The re-conceptualization of stochastic event regularities using the concepts of probability, might be styled ‘whenever event x, then on average event y’, or y = f(x + ϵ) or more accurately, ‘whenever the realized value of the (independent) variable measuring event or state of affairs x, then the conditional mean ⁸ of the (dependent) variable measuring event or state of affairs y’. The error term (ϵ) presents random influences on the dependent variable y and consequently converts the mathematical model linking y to the x into a stochastic or statistical model representing the population of interest (Downward, 2016: 210). If an empirical researcher managed to identify a stochastic event regularity (perhaps over a restricted space/time) then s/he will have identified a stochastically closed system. Henceforth, we use the phrase ‘event regularities, probabilistically specified’, to refer to the kind of associations identified by statistical techniques such as regression analysis and MA (Fleetwood, 2016).

Theory

A theory is often said to have a predictive dimension containing statements delivering predictions such as ‘y will follow x’; and an explanatory dimension containing statements delivering ‘explanation’ that amounts to the same thing. ‘Theory’, then, becomes reduced to a set of statements designed to enable predictions, usually, in the form of hypotheses. We describe this as ‘theory’ – that is, with scare quotes – because, in our example, a ‘theory’ that explains an increase in organization performance is reduced to a statement to the effect that ‘a bundle of HRM practices were introduced’. Whilst other information, perhaps identifying the relevant causal agentially enacted mechanisms, is sometimes included, it is, strictly speaking, not necessary. This is sometimes referred to as ‘ultra-empiricism’ or ‘measurement without theory’.

Prediction and (lack of) explanation

Prediction for quasi-empiricism is based upon induction from past regularities in the flux of events. This conflates prediction and explanation. This illicit conflation is commonly referred to as the ‘symmetry thesis’, whereby the only difference between explanation and prediction relates to the direction of time (i.e. if x predicts y, then x is said to ‘explain’ y). For example, if the introduction of team-working was found to predict an increase in profitability, then the former would be said to ‘explain’ the latter. This conflation manifests itself in the way independent variables are commonly referred to as ‘explanatory variables’, and/or ‘predictors’ of the magnitude of dependent variables. This is, however, a misconception. Imagine that a regression analysis identifies an association between the introduction of team-working and an increase in profitability, or put another way, imagine that the introduction of team-working predicts the increase in profitability. Is anything explained by this? The answer is no. A prediction, even a successful one, explains nothing. A regression analysis, even one that successfully identifies an association between independent and dependent variables, does not reveal why the association comes about and, therefore, lacks explanatory power.

Summary

The lack of explanatory power in individual quantitative empirical studies, rooted as they are in quasi-empiricist meta-theory, is the result of their commitment to the particular chain of meta-theoretical concepts – that is, an ontology of events or states of affairs; causality as event regularity; epistemology based upon identifying event regularities probabilistically specified; a method of building theoretically closed systems to engineer the event regularities that generate predictions to be tested qua hypotheses; and theory as sets of statements that ‘set up’ the event regularities as predictions, which are then conflated with ‘explanations’. As these studies cannot generate explanations, they cannot generate theoretically informed and empirically substantiated explanations either. Unfortunately, this problem is not restricted to individual quantitative empirical studies but, as we will see below, carries over into MA more generally.

We envisage two potential responses from advocates of MA. First, they might find, demand, or carry out individual studies, including quantitative studies, which do have explanatory power, or insist on them being used as the appropriate basis for MA. Second, they might counter-argue that MA does not lack explanatory power. All MA has sections referred to as ‘theory’, ‘literature review’, ‘hypothesis building’, or some such, and it is in these sections that theoretically informed and empirically substantiated explanations can be found. Unfortunately, these responses will not work. Apropos the first response, whilst qualitative empirical research is essential in the search for theoretically informed and empirically substantiated explanations (Ackroyd and Karlsson, 2014; see also Edwards et al., 2014), it is precisely this material that is excluded from MA:

. . . weed out all those papers that do not report data . . . as well as those studies that are based on the analysis of qualitative data (e.g., ethnographies . . . and case studies). Getting rid of these types of papers is straightforward. (Ellis, 2010: 98)

Furthermore, in order to find or carry out quantitative studies that do have explanatory power, they would have to be rooted in an alternative meta-theory, one not committed to the chain of meta-theoretical concepts noted above. Yet, quantitative empirical researchers cannot just abandon their commitment to this or that meta-theoretical concept, because these concepts only ‘work’ as a complete package. The alternative, which we propose later, is that we should abandon this entire chain of meta-theoretical concepts, and replace it with an alternative.

Illustrating the meta-theoretical problems with meta-analysis

To illustrate our critique, we have selected two recent, highly-cited examples of meta-analyses, published in top ranked journals in the authors’ areas of interest. In the first article, Reichl et al. (2014) explore the relation between work–nonwork conflict and burnout by conducting a MA of 86 relevant studies, which allows for an analysis of 220 coefficients. In the second, Subramony (2009) explores the relationship between bundles of human resource (HR) practices and specifically defined organizational outcomes. This is achieved through a MA of 65 relevant studies, which allows for an analysis of 239 separately reported effect sizes. Both are examples of ‘best practice’ MA and the criticisms we raise apply to all the examples of MA we are familiar with.

Reichl et al. (2014)

First, in a (half-page) section entitled ‘theoretical framework’, Reichl et al. mention ‘several theoretical reasons to expect relations between work–nonwork conflict and burnout’ (2014: 982–983). After a very short theoretical discussion they refer the reader to six sources where, presumably, the theoretically informed and empirically substantiated explanations informing their MA might be found. Further inspection, however, reveals this not to be the case. One study is just another MA; two are ‘standard’ quantitative studies seeking empirical regularities; three offer theoretical insight, but are not qualitative empirical studies, and two are extremely dated. Their ‘theoretical framework’ section, then, offers little or nothing in the way of theoretically informed and empirically substantiated explanation.

Second, Reichl et al.’s MA tells us that work–nonwork conflict was correlated with emotional exhaustion and cynicism, but these relations were moderated by gender, age, family characteristics and cultural norms. They are aware of ‘important gaps in our knowledge about underlying processes [i.e. causal mechanisms] and moderating variables’ (p. 980), and their remedy is to obtain ‘theoretically derived moderators’ – that is, to theoretically derive the moderating causal mechanisms. Whilst this looks like a potential source of theoretically informed and empirically substantiated explanation informing their MA, further inspection reveals this not to be the case. Apropos the moderator variable gender: one study is a theoretically informed quantitative analysis; four are ‘standard’ quantitative studies, despite one having ‘multi-method’ in the title; and two are dated. Concerning the moderator variable family characteristics: two are ‘standard’ quantitative studies and two are meta-analyses. For the ‘age’ variable, there is only a ‘standard’ quantitative study. Apropos the moderator cultural norms: four studies are ‘standard’ quantitative studies; one is another MA; and two are overviews/reviews. None of these references offer the kind of theoretically informed and empirically substantiated explanation that would be needed to derive the moderating causal mechanisms. This point is developed in more detail later.

Subramony (2009)

Let us turn our attention now to the other example of MA: Subramony’s article on HRM bundles and firm performance:

[HRM] bundles consisting of multiple complementary practices are typically considered superior to individual best practices in influencing firm performance. This study investigates the relationship between three such bundles (empowerment, motivation, and skill-enhancing) and business outcomes . . . Although it makes conceptual sense to categorize individual HRM practices into these bundles, there is insufficient empirical evidence regarding both their proposed synergistic properties and the magnitude of bundle effects on firm performance. I propose to bridge this gap in the strategic HRM literature by investigating the relationship between empowerment, motivation, and skill bundles and various business outcomes; clarifying the synergistic properties of these bundles by comparing their effects to those of individual HRM practices; and demonstrating the usefulness of these bundles in relation to high-performance work systems (HPWSs). (Subramony, 2009: 745–746, emphasis added)

To say that there is insufficient empirical evidence regarding (π), the proposed synergistic properties of bundles of HRM practices, is entirely correct. Subramony’s observation that there is insufficient empirical evidence regarding (Ω), the magnitude of bundle effects on firm performance, has valid and invalid elements. It is invalid in the sense that there are actually many quantitative empirical studies seeking to identify the magnitude of bundle effects on firm performance. It is, however, valid in the sense that what evidence there is does not support the existence of the statistical association he believes exists. Subramony proposes to ‘bridge this gap’ by (a) investigating the relationship between these bundles and business outcomes; (b) clarifying the synergistic properties of these bundles by comparing their effects to those of individual HRM practices; and (c) demonstrating the usefulness of these bundles in relation to HPWS. Notice, however, that there are two ‘gaps’ – that is, (π) and (Ω). At best MA can deal with (Ω) by engaging in (a) and (c). What MA cannot do, however, is deal with (π) via (b). It cannot bridge the gap of insufficient empirical evidence regarding the proposed synergistic properties of bundles of HRM because to do this would require theoretically informed and empirically substantiated explanations of why empowerment, motivation, and skill-enhancing HRM practices cause increased performance. MA cannot get anywhere near delivering explanations of this kind.

This said, as with Reich et al., Subramony’s MA is not entirely devoid of theoretically informed explanations – although few of them are empirically substantiated. He writes:

The combination of multiple empowerment-enhancing practices . . . is likely to be synergistic because of the potential complementarities among these practices. For instance, allowing autonomous work teams to manage the production of a component or provision of a specific service can enhance employees’ sense of responsibility and autonomy within the constraints of their work role. Additionally, the provision of voice and upward feedback mechanisms can help employees view themselves as part of a larger organizational system, leading them to engage in discretionary behaviors, including suggesting improvements to the products, services, or work processes; assuming increased responsibilities; and volunteering (e.g., serving on joint management-worker task forces). Also, the presence of multiple empowerment-related practices is likely to signal a coherent organization wide commitment to employee empowerment, which is likely to result in reciprocation in the form of in-role and extra-role behaviors. (Subramony, 2009: 748)

Subramony’s brief explanations for the existence of synergies are not unreasonable, but any competent researcher in this field could come up with reasonable explanations about dis-synergies. The fact is, we do not really know which is the case because there are insufficient theoretically informed and empirically substantiated explanations of the proposed synergies. Moreover, MA brings us no closer to obtaining these explanations because it focuses our attention on identifying statistical associations, such as that underlying ‘Hypothesis 1: Empowerment-enhancing bundles [of HRM practices] will be positively correlated with business outcomes’ (Subramony, 2009: 748).

Let us consider an example of how statistical techniques used in MA can often lead us further into obscurity:

By calculating the composites of relevant effect sizes within each study, I created the empowerment, motivation, and skill bundles. For instance, if a given study provided correlations between training and productivity and selection and productivity, a single composite score was created to reflect the combined effect of both the skill-enhancing practices of training and selection on productivity. (Subramony, 2009: 753)

Subramony takes past research showing correlations between training and productivity, and selection and productivity, and combines them into a single composite score reflecting the combined effect of training and selection on productivity. Whatever the advantages of doing this are, they have to be weighed against the dis-advantages. And the main disadvantage is this; to know that there are correlations between training and productivity, and selection and productivity, is not to explain anything – that is, we remain in the dark as to why these correlations come about. But then to combine them into a single score reflecting their combined effect on productivity leaves us with an even more complex statistical association about which we actually understand even less. We are moving further away from generating theoretically informed and empirically substantiated explanations, not getting closer to them.

Note that this has nothing to do with missing moderating or mediating variables. Indeed, if it turned out that additional moderating or mediating variables were needed, the problems would get even worse: we would end up with yet more variables, and yet more associations between them, and be no closer to deriving theoretically informed and empirically substantiated explanations.

Moreover, what can be done, practically, with Subramony’s argument that:

. . . firms can benefit from the adoption of high-performance HRM practices . . . as long as these practices also are complementary. Thus, instead of simply increasing the number of HRM practices . . . firms could derive positive returns by enhancing synergy among these practices. (Subramony, 2009: 759)

The only way this finding could be of substantive, or practical, use would be if it enabled a HR manager to successfully predict (solely on the basis of past event regularities) that the implementation of a bundle of complementary high-performance HRM practices would be followed by increased organizational performance in some future period. Even if a HR manager was prepared to implement the bundle, s/he would need to know a great deal more about how exactly to ‘enhance synergy among these practices’ than can be provided in such research. For the practitioner, therefore, these exhortations require a peculiar leap of faith that diminishes their own insight, experience and expertise. The implication is that the HR manager should sacrifice any experienced insight as to why certain HR practices may, or may not, work in their own context, and instead follow the numbers. One is left feeling that the HR professional might be usefully replaced with an algorithm.

Summary

The ‘A critical realist critique of meta-analysis’ section established that the lack of explanatory power that characterizes individual quantitative studies, rooted in quasi-empiricist meta-theory, is the result of its commitment to a particular chain of meta-theoretical concepts. Unfortunately, this problem carries over into MA. Despite MA having dedicated ‘theoretical’ sections, the latter carry little in the way of explanation and, therefore, can offer little prospect of theoretically informed and empirically substantiated explanations. The explanations contained in MA are as lacking in explanatory power as the individual quantitative studies upon which they are based. Allow us to make the point more forcefully: if one individual quantitative study lacks explanatory power, then synthesizing scores of them does not increase the explanatory power.

Does this mean that all attempts to synthesize existing research are doomed to failure? We think not, but only if we turn to an alternative approach, CRS, which is rooted in an entirely different meta-theory. It is to this that we now turn.

Ontology

As well as the actual and the empirical (Table 1), CRs recognize the existence of the ‘deep’ (Table 2). This stratified ontology is also emergent, meaning that entities existing at one ‘level’ are rooted in, but irreducible to, entities existing at another ‘level’. For example, the social is rooted in, but irreducible to the biological, which is rooted in, but irreducible to the chemical, which is rooted in, but irreducible to the atomic, and so on (Elder-Vass, 2010). Social reality is also transformational; Agents reproduce or transform a set of pre-existing mechanisms. Society continues to exist only because agents reproduce or transform the mechanisms that causally condition their social actions.

OPEN IN VIEWER

Table 2.

A stratified or laminated ontology.

Domain	Entity
Empirical	Experiences and perceptions
Actual	Events and actions
‘Deep’	Structures, mechanisms, tendencies, powers

In a social world, characterized by stratification, emergence, transformation and, typically, configurations of interacting causal mechanisms, it is unsurprising to find partial, approximate, rough-and-ready regularities or patterns in the flux of events. Following Lawson (1997; 2003: 81–83 and 105–107), we refer to these as ‘demi-regs’, which can be styled as ‘whenever event x, then sometimes, but not always event y’; for example, ‘women sometimes, but not always, look after children more than men’. A system wherein demi-regs predominate, is an open system. Thus, whilst any explanations CR generate should ‘fit’ with the statistical record, the statistical record explains nothing in itself (see also Porpora, 2015).

Epistemology

With the recognition that events do not often manifest as regularities and that something must govern an irregular flux of events, the emphasis of CR investigation switches from the domains of the empirical and actual to the deep: to the causal mechanisms that govern the flux of events. For example, we noted the claim that the introduction of PRP increases productivity at the firm level by ‘9%’ (Gielen et al., 2010: 291). CRs might re-focus attention towards the mechanisms by which the relationship might occur: the motivation of some, but not all, individuals to increase their effort towards those metrics that are being measured, or the impact of labour relations on such motivation. In weighing up explanations, CRs accept the possibility of judging between competing claims because they reject the claim that to accept epistemic relativism is to accept judgemental relativism. That said, there is no gainsaying the difficulty involved with this, especially when such judgement requires far more than simply carrying out statistically-based hypothesis testing to see which competing theories have greater explanatory power. ¹⁰

Aetiology

The parts of the social world not characterized by event regularities (i.e. open systems) are still governed by something. This something cannot be a law as this would produce constant regularities. So instead, CRs use the term tendency to depict the (transfactual) way of acting of a thing (or things) with properties (Fleetwood, 2009). A tendency is not an empirically observable pattern, as a tendency can be in play and yet not manifest itself empirically, as it can be counteracted by other mechanisms (Fleetwood, 2012: 248).

To illustrate causation, CRs seek what Fleetwood and Hesketh (2010) refer to as ‘thick explanation’ – that is the kind of explanation that requires hermeneutic information – that is, information relating to a range of human cognitive activities such as understanding, intention, purpose, meaning, interpretation, reason and so on. We do not, however, know what the cause of the action is, one does not understand it, until we know the intention that underlies it; that is, until we know why the agent did what s/he did. If, to explain an action is to give a causal account of it, then to explain an action is to give an account of why the actor did what s/he did. Whilst exploring motivations is always difficult, these can be explored using interviews. Sims-Schouten and Riley (2014) and Smith and Elger (2014) show, for instance, how interview-based research facilitates the probing of agent’s own understandings of causal relations in organizational contexts.

Method

As the social world is an open system, mechanisms cannot be induced or deduced, but must instead be retroduced. Retroduction ‘consists of a movement . . . from the conception of some phenomenon of interest to a conception of some totality or thing, mechanism, structure or condition that is responsible for that given phenomena’ (Lawson, 2003: 145). It usually involves asking a specific kind of question: ‘What thing, if it existed, might account for the existence of P?’ and might end up identifying Q as the thing in question. Retroduction is used when we are relatively ignorant about the mechanisms in operation that are causing the phenomena under investigation. When there is little or no existing theory to act as a guide, we must take a voyage of discovery, make hypothetical conjectures, requiring the ‘scientific imagination’ (see Lewis, 1999). We use what we do know to explain what we do not know.

Open and closed systems

In open systems, theoretically informed claims must be framed in transfactual terms. Transfactual claims cannot, however, be empirically substantiated by testing quantitative hypotheses. Consider two hypotheses: the first is typical of quasi-empiricism and the second is transfactual:

The intuition underlying Hypothesis 1 is something like ‘workers assembled into a team raise the probability that profit will increase’. This presumes the existence of a (ontic) stochastic regularity, which can be re-conceptualized probabilistically, between assembling workers into a team, and the resulting increase in profit. Hypothesis 1 can be tested using ‘normal’ statistical techniques. In complete contrast, the intuition underlying Hypothesis 2 is something like ‘workers assembled into a team have the causal power to raise profit, but sometimes this power is actualized and sometimes it is not’. This gives rise to a demi-regularity, rather than a stochastic regularity, and thus cannot be re-conceptualized probabilistically. Hypothesis 2 cannot be tested using the ‘normal’ statistical methods rendering quasi-empirical methods such as MA unsuitable for open-systems theorizing.

Prediction and explanation

CRs hold that, in open systems ‘thick’ explanation is probably our only guide to the future. If, for example, one can uncover, and explain, the causal mechanisms (e.g. HR practices) that, when drawn upon by workers and managers, increase organizational performance, then one has an explanation of the increase in performance. Such an explanation would allow one to understand the tendencies generated when workers and managers engage with HR practices. If one understands these tendencies one can make tendential predictions.

Importantly, and in contrast to the empiricist tradition, which focuses only on what actually happens, powers or tendencies for CRs are transfactual, and therefore point to the potential of entities. Thus, given the appropriate context (i.e. products, production regimes, labour relations systems), one mechanism may have more potential to increase performance than another, even if this potential is continually negated by countervailing tendencies. This is important because, unlike MA, it points to theorizing the possibilities of future social events, caused by agentially enacted mechanisms, even if these events have not occurred in the past.

Theory

For CRs, theory consists of statements that deliver causal explanations. We can illustrate this by returning to our previous example: if we want to explain the tendency for team-working to increase productivity, we might look to existing theory about the relations within teams, seeking to develop new insights about (i) exactly how teams (as bundles of causal mechanisms) raise productivity; (ii) how agents are engaged with them; and (iii) the complex set of interactions between the bundles themselves and between the agents.

Summary

In sum, a CR chain of meta-theoretical concepts can be contrasted to that of quasi-empiricism in which MA is rooted (Table 3).

OPEN IN VIEWER

Table 3.

Comparative aspects of quasi-empiricism and critical realism.

	Quasi-empiricism	Critical realism
Ontology	Atomistic, observable, events. No agency-structure approach, only rational agents as individuals.	Social world is stratified or laminated, emergent, transformational, systemically open, complex, becoming, processual and relational. Agents and structures (i.e. mechanisms) are distinct but related.
Epistemology	Knowledge derives from (a) observing (b) event regularities.	Knowledge derives from uncovering causal mechanisms. Epistemically but not judgementally relativist.
Aetiology	Humean: causality as event regularity. Laws, law-like relations and functional relations.	Separates Humean causality from causality as powers and tendencies. Powers and tendencies replace laws, law-like relations and functional relations.
Methodology and research technique	Some version of the covering law method. Engineering event regularities via theoretically closed systems. Exclusively quantitative data and techniques such as regression, analysis of variance, factor analysis and meta-analysis.	Causal-explanatory. Explanation comes via uncovering and understanding causal mechanisms. Ethnography, participant observation, in-depth interviewing critical discourse analysis, action research, archaeology, deconstruction and genealogy accepted. Mainly uses qualitative techniques, but the role of (some) quantitative techniques is debated.
Objective	Prediction. To construct and test predictions and hypotheses to establish whether claims are true or false.	Explanation. Claims must be empirically substantiated, but quantitative hypothesis testing is inappropriate.
Explanation	Explanation confused with prediction.	Explanation is ‘thick’ – an account of the operation of causal mechanisms. Explanation is not confused with prediction.
Prediction	Prediction based on inductive generalizations. Spurious precision.	Tendential prediction based on knowledge of causal mechanisms. Tendential prediction is not precise, but not spurious either.
Theory	Vehicle for delivering predictions.	Vehicle for delivering causal-explanatory accounts.
Mode of inference	Deduction and induction.	Retroduction.

With the meta-theoretical framing completed, we can now take the first tentative steps to show how the above CR meta-theory might be used to guide CRS.

Define the scope of the synthesis

The scope of CRS will often be in the form of a question like: ‘how does mechanism M, when enacted by agent A, tend to alter outcome O?’. This approach will also work for clarification questions, such as ‘what are the properties of mechanism M?’ or ‘why does outcome O often occur in context P?’. As CRs accept systemic openness, CRS is not only restricted to providing a ‘thick’ explanation of the agentially enacted mechanism, but also of the different contexts in which the mechanism might generate a tendency to O – or qualitatively different versions of O – as well as the unintended consequences of mechanism M’s tendency. This also permits consideration of the ways in which the outcome might react back, in a later time period, on mechanism M and agent A.

In clarifying the purpose of the review, it is also useful to know the mechanisms that are claimed or assumed in the relevant literature or policy. For example, in exploring the question do bundles of HRM practices improve performance?, it might be useful to identify the assumptions that are made, or theories that are drawn upon, when this is presumed to be the case, as these provide a focus for the review that can be explored systematically later. This step is not always necessary.

Search for, and appraise, the evidence

As CRS focuses on identifying agents and mechanisms, it need not restrict itself to statistical studies or indeed studies from any specific discipline, including CR. For example, O’Mahoney (2011) reviews the social constructionist identity literature, much of which explicitly rejects realism, to retroduce the entities, powers, and mechanisms involved with identity construction. Indeed, the ecumenical nature of CR’s review of the literature allows it to use this breadth to identify similar causal mechanisms working in a variety of contexts. Moreover, as Ackroyd and Karlsson suggest, the CR researcher is marked by their ‘eclecticism’ when it comes to matching innovative methods to collect new data indicative of the existence and character of causal mechanisms (2014: 22). For example, Pawson (2002) – which is expanded upon below – seeks to understand the impact of ‘public disclosure’ on recalcitrant behaviour (i.e. does ‘naming and shaming’ work?). To investigate this, he drew on a wide range of public disclosure policies, from Megan’s Law and school league tables, to hospital star ratings and naming prostitute’s clients. This allows exploration of similar mechanisms but in very different contexts, permitting the identification of the particular contexts that were more likely to generate a tendency for disclosure to affect behaviour.

In collecting studies, quantitative work (e.g. those using regression analysis to identify statistical associations) should be treated with caution. Instead of dismissing them, however, we would check to see if, in addition to the (non-explanatory) statistical data, there is something that might help us to create theoretically informed and empirically substantiated explanations. Instead, we are far more favourably disposed to past qualitative empirical research. In both cases (and recalling the ‘A critical realist critique of meta-analysis’ section) we would be asking ourselves: does this past research help us deepen our understanding of the appropriate agents and mechanisms, how agents and mechanisms interact, and the other mechanisms (i.e. ‘the context’) that dispose this agent to interact with one mechanism and not another?

CR accepts that different disciplines may use different terms to describe similar mechanisms – though where these terms differ they may be more or less accurate. For example, ‘enculturation’, ‘socialization’, ‘institutionalization’, ‘indoctrination’, ‘learning’, and ‘disciplining’, might be used in different traditions to describe the ways in which societies inform and (re)create the individuals that inhabit them. Such terminological diversity should always be critically appraised, as such terms are not apolitical, and, for example, can range from strong managerialism (‘workers can learn to be more efficient’) to critical (‘workers are indoctrinated through induction programmes’). However, such diversity should also be embraced as providing potentially useful alternative perspectives on how the agentially enacted causal mechanisms operate and relate. More specifically, terms captured in the review may operate at different (sometimes emergent) levels – for example, socialization and learning may be different (and related) forms of indoctrination.

In short, then, a CR review of a subject should cast its net wide, searching not only for key words (such as HRM, bundles and performance) but also examining historical texts, and different disciplines for similar mechanisms that may have operated in different contexts. Thus, for the literature review, the search terms and sources would usually be wider than we might expect in a standard structured literature review (Tranfield et al., 2003). Once the relevant literature has been collected, and this would usually be an ongoing process, it needs to be appraised, both in terms of ensuring the research actually addresses the mechanism(s) under study, and its internal validity – that the data actually supports the conclusions it makes.

Extract and synthesize findings

The purpose of a CRS analysis is to identify the agentially enacted causal mechanisms by reviewing extant literature. In CR-oriented studies, these mechanisms will often (though not always) be explicit. However, when reviewing non-CR-oriented literature the analysis can often proceed in two steps. The first is to identify the agentially enacted mechanisms stated within the literature, and the second is to retroduce further conditions of possibility for these. It is important to note that CRS does not require a rejection of any research that is not CR in orientation. Let us consider two examples of this latter point.

First, O’Mahoney’s (2011) review, mentioned above, takes the statements of ostensibly anti-realist authors concerned with identity construction to identify the key agentially enacted mechanisms involved. For example, he draws upon an article by Thomas and Davies (2005) that details how Kate, a personnel manager in the police service, draws on discourses of femininity and parenthood to resist performative employment discourses. O’Mahoney’s first step is to identify the context stated explicitly by the authors, including Kate herself, her job, the police, and the various discourses that, for CRs, are causal mechanisms. The second step is to retroduce implied mechanisms and powers, such as the power of the police service to employ and discipline workers, and terminate contracts, and the (agential) power of Kate to learn skills and reproduce them. We also learn much about discourses – for example, that they can be resisted, that some discourses (such as femininity) exist in tension with others (such as masculinity), and that individuals exercise some form of free-will in choosing to engage with or resist them. This allows O’Mahoney to argue that, contrary to the anti-realist protestations of social constructivists, their research can contribute towards the kind of theoretically informed and empirically substantiated explanations sought in CRS.

Second, in seeking to understand the impact of ‘public disclosure’ on recalcitrant behaviour Pawson (2002) examines the policy literature to identify the mechanisms at play in the literature when culprit behaviour is worsened and policies fail (Figure 1).

OPEN IN VIEWER

Figure 1.

Mechanisms of public disclosure on culprit resistance (simplified from Pawson, 2002).

He then reviews the history and operation of the various disclosure policies to identify when the mechanisms lead to positive outcomes (Figure 2). For example, he notes that:

Megan’s Law swept onto the statutes following the enormous public outcry at the brutal death [of a child]. The courts responded to the wave of sentiment that ‘something must be done’ and were thus able to brush aside the constitutional challenges forwarded by minor lobbies. (Pawson, 2004: 39)

OPEN IN VIEWER

Figure 2.

When disclosure policies lead to successful outcomes (simplified from Pawson, 2002).

Here, Pawson uses contrastive theory building to identify patterns rather than laws (we would say ‘demi-regs’) about the potential of public disclosure policies to achieve their aims. The more tentative and less certain language here is also worth noting, especially in contrast with the ‘9%’ of MA detailed earlier:

Although popularly known as ‘naming and shaming’, public disclosure outcomes in these cases do not seem to depend, in the long term anyway, on the dishonour of the culprits . . . Public disclosure is meant to change behaviour – but seems effective only in relation to what organises that behaviour in the first place. What is more, in each of the [cases], it is the information providers rather than the public who are the key agents of change. (Pawson, 2004: 44)

In terms of synthesis, Pawson takes a comparative approach to identifying the mechanisms that link X and Y, and provides a ‘thick’ description of how and why these work in different circumstances.

Whilst no-one has yet carried out an explicit CRS, some CRs have implicitly started to go down this route. Three can be cited as examples. First, Fleetwood’s (2014, 2016) attempt to build a CR-oriented alternative model of labour markets draws upon a body of existing theoretical and empirical research, which he refers to as ‘the socio-economics of labour markets’. The key point to note is Fleetwood’s rejection of existing quantitative empirical research that is rooted in quasi-empiricism because it contributes little or nothing to the generation of theoretically informed and empirically substantiated explanations of the way labour markets work. In contrast, Fleetwood accepts the ‘socio-economics of labour markets’ because it consists of existing qualitative empirical research that contributes to the generation of theoretically informed and empirically substantiated explanations of the way labour markets work. Whilst implicit, Fleetwood’s work on labour markets might be thought of as a rudimentary CRS.

Second, Vincent’s (2011) work, in this journal, focuses attention on emotional experiences at work, the organization control mechanisms that seek to influence these experiences, and how different contextual conditions (M^c) affect both organizational control systems and worker experiences. Whilst the article is not explicitly either CR or CRS, it offers a form of analysis that is highly consonant with the approach outlined here. The article maps the structural conditions and agential dispositions that affect emotional displays at work, and how these combine to explain experiences. It highlights, in particular, how employers’ regulation and rewarding of workers’ emotional displays interacted with workers’ conformity (or not) with organizational interests and rule systems. The article then considers the contextual conditions that impel different types of control system and experience, for example, by highlighting the circumstances in which workers are rewarded for specific emotional displays at work. Overall, this article contributes by developing theoretically informed and empirically substantiated insights about the way emotions are managed, experienced and enacted at work, offering another rudimentary CRS.

Third, Dirpal (2015) starts from the position that past quantitative empirical research on the HRM-Performance cannot explain why HRM practices are linked to performance. He re-theorizes HRM practices to develop the concept of an ‘HRMechanism’ (i.e. HRM practice + causal mechanism) before applying qualitative research techniques to investigate what would normally be considered a quantitative research programme. Thus, he offers a (meta) theoretically informed piece of qualitative research into six HRMechanisms: team-working, corporate culture, empowerment, work–life balance, performance appraisal and reward. What makes Dirpal’s research interesting for our purposes is how he uses past qualitative empirical research as a quasi-CRS. What he lacks, initially, is a sophisticated understanding of exactly how HRM practices may or may not work to influence organizational performance. He turns to the existing literature to glean any theoretically informed and empirically substantiated insights, and uses them to frame his interviews. He finds that team-working, performance appraisal and work–life balance, generate powers/tendencies to increase organizational performance, whereas corporate culture, empowerment and rewards generate neutral powers/tendencies vis-a-vis organizational performance. Moreover, he generates causal explanations of exactly what these HRMechanisms do to generate these powers/tendencies.