Interpretive Quantitative Methods for the Social Sciences

Introduction

Methodological disputes between what we now call positivist and interpretive research strategies are as old as social science itself, if not older. They are at least as old as the late 19th-century German methodenstreit pitting analytical against historicist social science and extend to the late 20th-century tension over the relative prestige of quantitative versus quantitative research. In sociology such tensions continue today between scholars who see research as something scientific, positivist, analytical and quantitative on one side and scholars who see research as something humanistic, interpretive, historicist and qualitative on the other. Accordingly, entire national sociologies are associated with quantitative or qualitative methods; different journals cater to each side; research methodology influences people’s perceptions of the publishability of research; national research assessments accord differing prestige to research in different traditions; research funding authorities explicitly target the funding of either scientific or humanistic research projects. There is a widespread perception (Maxwell, 2010: 475) backed up by systematic research (Collyer, 2013: 254–6) that such distinctions are strongly biased in favour of research that is perceived as scientific/positivist/analytical/quantitative over research that is perceived as humanistic/interpretive/historicist/qualitative.

This reputational hierarchy plays out in practical ways that have the potential to shape the future trajectory of sociology and other social science disciplines. In 2010 the United Kingdom’s Economic and Social Research Council (ESRC) published the results of a root-and-branch review of UK sociology conducted in conjunction with the UK Council of Heads and Professors of Sociology (HaPS) and the British Sociological Association (BSA), the International Benchmarking Review of UK Sociology (ESRC, 2010). All three sponsors of the review are prestige associations led by highly esteemed leaders of the profession. These are unambiguously ‘elite’ groupings within UK academia, if not necessarily UK society. In its section on research methods, the sociology benchmarking review concluded that:

The upshot of our assessment is, that innovative work continues to be done in developing qualitative methods, but that British sociology remains weak in quantitative methods … we address primarily what we see to be the major deficiency, namely the relative neglect of quantitative methods.

Of course statistical methods are not the only valid mode of inquiry, and each of the social sciences also embraces its own theoretical and qualitative approaches. But, arguably, statistical methods form a common core of social science. (ESRC, 2010: 23)

A further section on ‘The future of sociology: What kind of discipline?’ focused entirely on the need for more quantitative methods, and one of the five major policy recommendations of the report was that the UK should expand quantitative methods training. Clearly, these elite academic groups (the ESRC, BSA and HaPS) are in no doubt that they would like UK sociology to move decisively towards a stronger focus on quantitative research. Similar sentiments are expressed in the ESRC benchmarking reviews of politics and international studies (ESRC, 2007) and (especially) of human geography (ESRC, 2013). Interestingly, the ESRC benchmarking reviews of the (now) highly quantitative disciplines of economics (ESRC, 2008) and psychology (ESRC, 2011) made absolutely no mention of the need for more qualitative research and training to balance the disciplines (in fact, the mere existence of qualitative research in psychology was problematised). Heterodox scholars and even departments do exist both in economics and in psychology, but from the perspective of the ESRC and other elite institutions an overwhelmingly quantitative social science seems not to be a problem in need of a solution. Reinforcing this interpretation, since the completion of these benchmarking reviews the Nuffield Foundation (in cooperation with the ESRC and the Higher Education Funding Council for England) has announced a major programme (‘Q-Step’) to fund the creation of 53 new quantitatively-oriented university posts across the social sciences.

Is this a problem for sociology? The privileging of quantitative approaches to social science has obvious implications for the future of qualitative research and the viability of individual social science careers. Perhaps less recognised, it also has important implications for the character of quantitative sociology itself. Not far beneath the surface of the push for a more quantitative sociology is the elite preference for a social science that is more scientific, positivist and analytical in its world-view. That this is an elite agenda in the sense of the expressed preferences of the leaders of peak associations is clear from the quotations above. That this is an elite agenda in the sense of the implicit preferences of the rich and powerful has often been suggested as well (Lather, 2004). In this context the scientism of psychology and mathematicism of economics are seen as models to which the social sciences should aspire. The elite agenda for more quantitative social sciences (most transparent in, but not limited to, the UK) is in effect an elite agenda for more analytical social sciences that frame research questions in terms of mathematical symbols and answers them using quasi-experimental hypothesis tests.

This is a problem for sociology, because the analytical approach to social science can often result in sophisticated but sterile research. Leaving aside the potentially catastrophic implications for qualitative research and qualitative sociologists’ careers, the unreflexive importation of methodological conventions from economics and the sciences is simply inappropriate for most social science, and particularly sociological, research environments. This article outlines an alternative interpretive approach to quantitative social science (in general) that can be applied especially to sociology (in particular). The first section below lays out the differences between interpretive social science and two forms of positivism: the logical positivism that dominates economics and the empirical positivism that dominates psychology. The second section links interpretive quantitative methods to the discovery of the data generating processes that underlie observed social science variables. The third section explains how the interpretive triangulation of results from multiple analyses can shift the focus of quantitative research from the modelling of variables to the uncovering of data generating processes. The fourth section draws a connection between interpretive understandings of causality and the formulation of more realistic policy implications from research. The article concludes with an interpretive prescription for a more meaningful, more understandable, more applicable practice of quantitative methods in the social sciences.

Positivist and Interpretive Research Frameworks

Rightly or wrongly, quantitative social science has long been identified with self-consciously positivist approaches to the philosophy, rhetoric and methodology of research. Bryman (1984) explicitly criticises this identification while citing numerous examples of it (and lamenting its increasingly wide acceptance). Philosophically, this identification implies that quantitative social scientists assume that there exists a knowable objective reality that is represented reasonably well by the variables that are used in statistical analyses (and their relationships). Rhetorically, this identification implies that quantitative social scientists routinely adopt the experimental language of null and alternative hypotheses about which probabilistic conclusions (‘statistical significance’) can be drawn. Methodologically, this identification implies that quantitative social scientists rely on statistical controls to simulate how independent variables might affect dependent variables under ceteris paribus counterfactual conditions in order to make claims about causality. Taken to their extreme, these positivist precepts can result in formalistic research that lacks substantive meaning, is difficult to understand and cannot easily be translated into practical social policies.

The exact meaning of positivism is the topic of many passionate debates, but two strands stand out in the social sciences. The first strand is the logical positivism of Comte (1844/1957), the idea that social truths can be arrived at through reasoning from basic principles. This has largely disappeared from contemporary social science practice outside economics but has become the dominant research paradigm in economics, where strong mathematical theorisation predominates. Logical positivism in the social sciences requires very strong assumptions based on theory in order to compensate for the relatively low quality of social science data. The second strand is the empirical positivism of Popper (1957), the idea that theories must be falsifiable and should be subjected to empirical testing. Though Popper vehemently objected to the labelling of his approach as ‘positivist’ (Kageyama, 2003), it has come to be identified as positivist (or post-positivist) by others. It is the dominant research paradigm in experimental psychology. Its applicability outside the laboratory is open to question, since hypotheses can never be tested unambiguously with observational data.

These two strands of positivism share a common foundation in that both demand a high degree of a priori theorisation from the researcher. They differ in the aggressiveness of their claims about causality. Popperian empirical positivism makes very modest claims about causality, since it permits no systematic generalisation beyond the particular setting in which the data being analysed have been generated. Comteian logical positivism supports much broader claims about causality, since it begins from the premise that causal mechanisms can be theorised as universally valid.

If both logical and empirical positivism depend on strong a priori theorisation, other strands of scientific enquiry can be imagined in which theories are instead built up from data. The grounded theory approach that pervades qualitative sociology is just such a strand. Grounded theorisation is an interpretive approach to social science research in which both a priori theorisation and claims about causality are intentionally kept to a minimum. In describing the technique, Glaser (2002) emphasises the emergence of concepts through the ‘constant comparison’ of results from the theoretically informed sampling of data. While grounded theorisation is clearly anti-positivist, it need not be anti-quantitative. When Denzin and Lincoln (2011: 2) assert that ‘the interpretive traditions of qualitative research commit one to a critique of the positivist or post-positivist project’ they are absolutely correct, except in their assumption that interpretive methods should be identified with qualitative research. In fact, Glaser and Strauss (1967: 17) in their seminal book on grounded theory explicitly recognised this:

Our position in this book is as follows: there is no fundamental clash between the purposes and capacities of qualitative and quantitative methods or data. What clash there is concerns the primacy of emphasis on verification or generation of theory – to which heated discussions of qualitative versus quantitative data have been linked historically.

If it is possible to combine low levels of a priori theorisation with modest claims about causality, it is also possible to combine low levels of a priori theorisation with aggressive claims about causality. This is the approach taken by emerging ‘big data’ approaches to social science that rely on the computationally intensive crunching of large amounts of data to discover supposed truths about human behaviour. The promise that truth can be discovered from data through black-box data mining – without the complications of potentially critical social theory – is presumably at the root of elite fascination with (and funding of) this approach. Ironically, the emerging field of big data social science is in some ways similar to the interpretive grounded theory approach, which Glaser and Strauss (1967: 1) described as ‘the discovery of theory from data’. The two approaches differ not so much in their research philosophies as in their research personalities: interpretive researchers are unwilling to make the grandiose claims about causality that are the bread and butter of big data research, leaving the field open to non-sociologists and indeed non-social scientists to frame the interpretation of patterns found in large administrative databases (Burrows and Savage, 2014: 3).

The characteristics of these four approaches to social science research – logical positivism, empirical positivism, grounded interpretive theorisation and black-box data mining – are summarised graphically in Figure 1. Judged on its own rhetoric, much published quantitative social science would seem to fall in the lower-right corner of empirical (Popperian) positivism: hypotheses are ‘tested’ to determine the ‘statistical significance’ of ‘results’ about the relationships among variables, acting out what Gigerenzer et al. (2004) evocatively label the ‘null ritual’. But this ritual does not accurately reflect typical research practices. As Greiffenhagen et al. (2011: 104) show in a one-of-a-kind ethnography of sociological research methodology, actual quantitative sociologists engage in extensive give-and-take in shaping their analytical strategies that is ‘not on the record’ (they hesitate to use the pejorative ‘off the record’) of their final research reports. As they describe the process:

Models do not build themselves any more than they interpret themselves; it is neither a predominantly mechanical nor purely deductive process. Of course, some standard techniques are involved; they are not starting from scratch. But choices still have to be made, and these are frequently based on intuitions, hunches and ideas of what is needed that have not yet been fully rationalized. (Greiffenhagen et al., 2011: 103)

OPEN IN VIEWER

Figure 1.

Four possibilities in the relationship between theory and data.

Such processes can hardly be described as Popperian.

Kincheloe and Tobin (2009: 514) liken positivism in the social sciences to a ‘zombie’ that ‘walks the socio-political and educational landscape shaping the way we think, what we see in the world, and, of course, how we produce knowledge’. Nonetheless, the rhetoric of positivism is alive and thriving and in contemporary quantitative sociology. This may be attributable to the perceived higher prestige of the natural sciences where experimental research prevails (Clegg, 2010). That positivist research philosophy, rhetoric and methodology are more prevalent in quantitative than in qualitative sociology may be due in part to the fact that in qualitative sociology the interpretive role of the researcher is so prominent that it would be rhetorically indefensible not to problematise it. Thus, the study of researcher reflexivity has historically been the preserve of qualitative research (Alvesson and Sköldberg, 2009).

Quantitative sociology is probably never strictly positivist in practice, but it is frequently positivist in philosophy, rhetoric and methodology. This is unfortunate, but it is not inevitable. A more interpretive philosophy, rhetoric and methodology of quantitative sociology might result in a more realistic and policy-relevant quantitative sociology than we have today. Philosophically, quantitative research might benefit from grounding theories in the full complexity of the data rather than forcing data into predefined boxes that may not truly reflect reality. Rhetorically, quantitative research might benefit from presenting research results in ways that recapitulate actual research practices. Methodologically, quantitative research might benefit from using a variety of numerical and statistical methods to triangulate towards robust conclusions, whether or not they are statistically ‘significant’ in conventional statistical tests. These implications of a shift from a positivist to an interpretive framework for quantitative social science are summarised in Table 1.

OPEN IN VIEWER

Table 1.

Contrasting typical characteristics of positivist versus interpretive research frameworks.

Dimension	Positivist framework	Interpretive framework
Philosophy	Assigns meaning to variables on the basis of the requirements of the research	Allows variables to posses emergent properties that may not correspond to the requirements of the research
Rhetoric	Follows or recommends a linear path from theory through operationalisation to results and interpretation	Follows or recommends a feedback pattern that iteratively grounds theory in data and vice versa
Methodology	Uses or recommends the use of a single appropriate technique for modelling relationships among variables	Uses or recommends the use of a variety of methods in order to understand fully relationships among latent concepts

For this shift to occur (and to be successful), quantitative researchers must be willing to question the purpose of statistical modelling. An interpretive perspective suggests that statistical models should be used to help the data ‘tell its story’. From this standpoint, the goal of statistical modelling is ‘to discover, explain or interpret or to fashion a more systematic way of understanding what, at the outset, appears to be an obscure, perhaps ambiguous human event or situation’ (Goldstein, 1991: 104, though referring in the original to qualitative research). Quantitative methodologists might think of this as a form of data reduction but at heart it is a kind of quantitative verstehen. Weber identified several types of verstehen, but in its primary usage it refers to the explanation of action in terms that are subjectively meaningful to the actor (Martin, 2000: 18). The parallel conceptualisation of verstehen in quantitative methods is to understand the patterns exhibited by the data in the terms of the data themselves – that is, without imposing interpretations that are not native to the data. Interpretive quantitative methodology is all about the observed data and how they came to be generated.

Data Generating Processes

Positivist quantitative modelling in the social sciences usually takes for granted the stylised fact that variables cause other variables. In reality, however, social action and societal processes occur at a level at least one step removed from the variables that sociologists are able to observe (Coleman, 1990: 12). Sociological variables themselves are thus observed manifestations of unobserved actions and processes. These unobserved phenomena cause other unobserved phenomena, which are associated with their own variables. The distinction between stylised practice and literal reality is illustrated graphically in Figure 2 using the example of education (operationalised as years of schooling) and income (operationalised as pounds per year). Variable-based quantitative studies routinely estimate models that are analogous to Model A on the left side of Figure 2: the variable ‘education’ causes the variable ‘income’. A more nuanced approach suggests that the true models that researchers should be estimating are analogous to Model B: phenomenon ‘X’ (reflected in the variable education) causes phenomenon ‘Y’ (reflected in the variable income). These two models are not interchangeable. In terms of both statistical estimation and interpretive meaning they are quite different, and different in their implications for policy. For example, in Model A a social policy intervention that directly increases levels of education would be expected to raise incomes. In Model B, it might have no effect on incomes whatsoever – depending on whether or not it also affected X.

OPEN IN VIEWER

Figure 2.

Two models of causality underlying the relationship between education and income.

This problem is endemic to observational research. Meehl (1978: 806) goes so far as to argue that in the social sciences ‘the null hypothesis is quasi-always false’ because in the social sciences substantive theories are almost always several conceptual stages removed from our formal statistical hypotheses (Meehl, 1978: 824). It is usually unavoidable that the practical operationalisations of sociological theories should be somewhat removed from the micro-level mechanisms that (explicitly or implicitly) underlie the operation of those theories. This difficulty is exacerbated by the fact that in many cases the data that happen to be available for analysis represent only a tiny portion of the relevant but mostly unobserved data about society and sociologists’ research subjects. The available data are the artefacts of uncertain and generally unobservable (but nonetheless theorisable) data generating processes.

This is a deepening of the position taken by Goldthorpe (2001). A data generating process is a real world mechanism that is theorised to have created the data that sociologists are able to observe. Multiple data generating processes might fit the same observed data. Returning to Figure 2, the unobserved phenomena X and Y that are operationalised using variables for education and income might be labelled ‘ambition for success’ and ‘level of economic productivity’ in functionalist study but labelled ‘mastery of society’s hidden curriculum’ and ‘class position in capitalist society’ in a critical study. Without additional variables, statistical modelling cannot directly distinguish between these two operationalisation choices in the simple scenario summarised in Figure 2. This places a duty of care on the researcher to recognise that an interpretive process has taken place.

The classic example of research involving an unobservable data generating process comes from epidemiology. In 1854 the British physician John Snow famously identified a polluted well in Soho as the source of a London cholera outbreak based on data on the distance of households from the well (observed X) and the presence of illness in the household (observed Y). He could not directly observe with the technology of his time the bacteria in the well-water that actually caused the disease (unobserved X) or the bacteria in the intestines of the victim of the disease (unobserved Y). Nonetheless he correctly interpreted the observable data to the extent that an appropriate policy intervention could be effected. Today epidemiologists can directly observe the pathogens that generate infection, but as the cholera example shows it is possible to formulate effective policy even when the actual data generating processes involved are unknown and unobservable. An interpretive research framework that prioritises discovery over testing is indispensable to this process.

Specific micro-level data generating processes in sociology are usually unknown and almost always unobservable. There is no microscope for human behaviour. As Figure 1 suggests, there are four feasible approaches to handling this problem that have been adopted in different social science settings. It is feasible to study only highly circumscribed kinds of behaviour for which data generating processes (‘treatments’) can be imposed experimentally in the laboratory; this is the dominant approach in experimental psychology. It is feasible simply to assume a particular data generating process; this is the dominant approach in mainstream economics. It is feasible to generate theories about data generating processes algorithmically through automated decision-making rules, but this approach requires extraordinarily large amounts of data relative to the complexity of the data generating processes to be uncovered (‘big data’). Finally, it is feasible to interrogate the available data by trying out multiple variable operationalisations and running multiple statistical models in an attempt to shed at least some light on the particular data generating processes that underlie the observed data. In practice, this is the dominant approach in quantitative sociology. The problem is that in quantitative sociology this theory discovery process is informal, almost covert (Greiffenhagen et al., 2011). Published quantitative research almost always repackages results as if they had been arrived at through positivist approaches, even when they have not (Gigerenzer and Marewski, forthcoming).

Measurement, Modelling, Triangulation and Reflexivity

A defining characteristic of interpretive methodology is the blurring of the boundary between measurement and modelling. Qualitative interpretive sociologists may not put it in exactly those terms, but that is probably precisely because qualitative researchers tend not to distinguish between measurement and modelling as distinct research activities. In positivist quantitative sociology, by contrast, researchers usually enforce a strict rhetorical distinction between the two: first variables are measured; second the relationships connecting variables are modelled. But once it is accepted that variables do not cause other variables (as such) the possibilities for concepts to cause other concepts and the varieties of ways in which those concepts might be measured are endless. Even in ideal situations in which quantitative sociologists are working with truly unbiased sample survey data, the variables available in those surveys represent incomplete and highly biased records of the research subjects’ lives. The observed relationships among these observed variables are the measurement tip of the causal iceberg. The task of interpretive social science is to surmise what lies unobserved beneath.

An important tool for this is triangulation. Triangulation involves the collation and comparison of data from multiple sources at multiple levels. To continue the education and income example of Figure 2, the observed correlation between the two variables might be triangulated by examining things like:

Such supporting analyses go far beyond the standard inclusion of control variables and interaction terms in statistical models and demand more the kind of sociological imagination championed by C. Wright Mills. In typical positivist quantitative sociology, control variables and interaction terms serve mainly as checks on the statistical robustness of the hypothesised relationships of interest: if the coefficient remains significant in the face of all controls, the results are judged likely to be valid. This approach may sometimes be sufficient, but it may not always be very enlightening. Exploratory triangulation is likely to be much more effective at uncovering the true data generating processes that have produced the relationships that are observable in the available data.

An interpretive approach to statistical modelling is likely to result in statistical analyses that are mathematically less complicated but understandings that are substantively more interesting than those that result from positivist approaches. In the interpretive approach measurement and modelling decisions emerge from the idiosyncrasies of the data rather than being imposed over the idiosyncrasies of the data. The relationships among observed variables are located holistically within the larger societal data generating processes that created the data in the first place. This encourages quantitative sociologists to think more about the societies (as opposed to the statistics) in which their data are embedded. It also encourages them to think reflexively about the biases that might be inherent in the available data, taking stock both of the societal prejudices that may have resulted in the available data having been collected and of their own prejudices in selecting those data for analysis.

Reflexivity (or reflectivity) in qualitative research is understood as a process that involves researchers examining their own personal roles in the research process (Alvesson and Sköldberg, 2009: 9) and Ryan and Golden (2006) suggest that quantitative researchers can benefit just as much from personal reflexivity as do qualitative researchers. Anderson (1989: 254–5) influentially identified reflexivity in the qualitative practice of critical ethnography as involving: ‘a dialectical process among (a) the researcher’s constructs, (b) the informants’ commonsense constructs, (c) the research data, (d) the researcher’s ideological biases, and (e) the structural and historical forces that informed the social construction under study’. A slight reformulation of Anderson’s list for the quantitative social sciences would result in guidelines for a reflexive research practice in which researchers engaged in dialectical inquiry into (a) the researcher’s constructs of the concepts that they measure using observed variables, (b) the commonsense meanings of those variables, (c) the research data, (d) the researcher’s ideological biases and (e) the structural and historical forces that underlie the research. Ironically, though qualitative sociologists may find it difficult to flesh out Anderson’s list, information about and insights into all five of these factors are close at hand for most quantitative sociologists. Reflecting on these five aspects of the research process would likely improve their understandings of their own research and through this their understandings of society. Reflexivity, unavoidable in qualitative sociology, should be enthusiastically embraced in quantitative sociology.

Causality and Policy

The relationship between education and income depicted in Figure 2 is one of the strongest and most robust relationships in any of the social sciences. For decades social scientists, politicians and educators encouraged students to focus on their schoolwork with the message that higher education was the key to high income and job security. Since 2007, however, new graduates in countries across Europe and North America have faced sometimes staggering levels of youth unemployment exacerbated by the high levels of debt that many students took out on the promise of finding well-paying jobs. In this they join millions of their peer graduates in developing countries who have long experience of the disconnect between education and employment. For millions of graduates around the world education has opened new vistas into a promised land of fulfilling adult roles in society only to shunt them aside into deadening routine work at low pay – when they can find jobs at all. How can this be? Are graduates’ real-world experiences of the job market merely due to random error in the established relationship between education and income? Or is there something more to it?

As every methodologist knows, correlation does not necessarily imply causality. In the observational research designs that prevail in sociology there is no random assignment and thus it is impossible to ascribe causal effects unambiguously to the independent variable of interest. This gives rise to the problem of endogeneity: it is always possible that some unobserved phenomenon is a common cause of both the independent variable of interest and the dependent variable in a model. While many instrumental variable techniques have been developed to address the problem of endogeneity, they ultimately depend on very demanding model assumptions (Dunning, 2008). From a purely empirical standpoint, endogeneity can never be ruled out in observational research. As a result, the phenomenality of observed statistical relationships is always in doubt and causality is always ambiguous. What this implies for new graduates is that social scientists cannot tell them with certainty that the high incomes of highly educated people are causally attributable to the educations they received, or that achieving higher education will result in higher income.

The standard approach to understanding causality in sociology is the counterfactual approach of Winship and Morgan (2007). The counterfactual approach conceptualises observational data in experimental terms, with the causal independent variable of interest even referred to as a ‘treatment’ variable. Counterfactual causal modelling relies on control (‘conditioning’) variables to reconstruct what relationships would have been observed (‘potential outcomes’) under various counterfactual circumstances. It is essentially a formalisation of the common practice of controlling for all credible alternative explanations. The counterfactual approach grows out of the graphical tradition of structural equation modelling (Pearl, 2000) in which causal models are conceptualised as path diagrams that connect all of the relevant variables. In essence, the problem of causality boils down to drawing the right paths from variable to variable in the available data.

This kind of causal modelling starts from the (implicit) assumption that variables cause other variables. If instead it is acknowledged that variables are epiphenomenal and the true data generating processes that underlie most sociological studies operate at least one level below the level of observed variables, this assumption is untenable. Counterfactual causality also suffers from the practical difficulties that the number of credible alternative explanations is always infinite and that the data required for operationalising alternative explanations are usually not available. But without counterfactual causality quantitative sociology has no conceptual mechanism for handling endogeneity. In the limited and rare scenarios in which reasonably credible ‘natural experiments’ (Dunning, 2012) can be constructed, sociology can revert to Popperian empirical positivism (and its attendant literal counterfactuality) as an orienting framework. The rest of the time, endogeneity is usually unresolvable unless sociologists are willing to accept the kinds of highly contrived model assumptions that are routinely accepted by mainstream economists.

If the very structure of sociological data ensures that causality can never be unambiguously resolved, what should sociologists tell prospective students about their income earning prospects? Without wading into the epistemology of causality, questions like this challenge sociologists to use survey data to provide practical guidance for solving real-world problems. Quantitative sociologists must somehow use data pertaining to the past to come to understandings in the present about the relative likelihoods of various outcomes in the future. Experimental techniques are no panacea here, since social processes (unlike physical processes) are likely to operate differently in the near future from how they operated in the recent past. Once one accepts that society is always changing, positivist approaches to understanding society that depend on high levels of a priori theorisation lose much of their applicability.

Heckman (2005) neatly captures these challenges in his three-category typology of policy evaluation problems. Heckman’s typology is summarised here in Table 2, reproduced from Babones (2014: 209). Some policy challenges are easily solved using conventional counterfactual causality implemented in a positivist framework for quantitative data analysis. Heckman calls these problems of internal validity (P1), and they are fundamentally quasi-experimental in character: government policy changed and welfare either went up or went down. Problems of external validity (P2) are more difficult to deal with: one country changed a policy and this had an observed effect; thus it might be surmised that the same policy change would have the same effect in another country. Heckman’s third type of policy challenge (P3), however, is the most common. Heckman did not give it a name, but in Babones (2014) it is characterised as ‘divination’. It arises whenever it is necessary to predict the effect of an unprecedented policy change in an unprecedented environment. This is the type of challenge faced by our students who must make education decisions now on the basis of past predictions about future employment prospects.

OPEN IN VIEWER

Table 2.

An interpretation of Heckman’s (2005) three modes of policy formation.

Policy challenge	Heckman’s description (quoted)	Heckman’s solution (summarised)
Problem P1: ‘Internal validity’	‘Evaluating the impact of historical interventions on outcomes including their impact in terms of welfare.’	Apply standard counterfactual ceteris paribus reasoning
Problem P2: ‘External validity’	‘Forecasting the impacts … of interventions implemented in one environment in other environments, including their impacts in terms of welfare.’	Apply existing models developed from historical data to new situations in the hope that the model still applies; if this is not satisfactory, treat as a P3 problem
Problem P3: Divination (?) (unlabelled by Heckman)	‘Forecasting the impacts of interventions … never historically experienced to other environments, including their impacts in terms of welfare.’	Decompose new environments into basic building blocks that have already been studied, then use these building blocks to assemble a conjectural solution (to the extent that past relationships still hold)

Though Heckman himself may be a mainstream economist, Heckman’s P3 policy challenges are most effectively addressed through interpretive research that triangulates towards as full an understanding as possible of the data generating processes that operate one or more levels below the surface of observable social reality. In this approach, endogeneity is not really a threat to causal modelling because the starting assumption is that all variables are endogenous. In the interpretive approach variables are not caused by variables but arise endogenously from the data generating processes theorised by the researcher. In the example of education and income, interpretive research would seek to uncover what it is about a society that gives rise to the observed correlation between education and income in that society. The modelling of variables can shed light on this question, but by itself it cannot answer it.

This implies that difficult research and policy questions – Heckman’s P3 challenges – can only be answered conjecturally by human experts, not definitively by statistical models. Here researcher reflexivity comes to the fore. The goal of interpretive research is not really to answer research questions. The goal of interpretive research is to develop the expertise of the researcher. The decomposition of new environments into basic building blocks that have already been studied and the assembly of those building blocks into conjectural policy solutions is what human experts do. The practice of interpretive data analysis helps them learn how to do it better.

The development of researcher expertise is perhaps the only coherent rationale for an academic publishing system based on peer review with no auditing of the actual results being reported, in which many journal articles are never cited and many academic books are printed in runs of fewer than 500 copies. In this understanding of the discipline, sociology does not progress from finding to finding in an accumulation of results along the lines of a Kuhnian normal science, nor does it experience true Kuhnian paradigm shifts (Kuhn, 1962). Instead, individual sociologists accumulate knowledge and individual sociologists experience intellectual biographical shifts. Thus although sociology is no closer to predicting next year’s events than it was 50 years ago, sociologists (hopefully) have much more sophisticated understandings of society than did sociologists 50 years ago. These more sophisticated understandings better equip today’s sociologists to advise on policy than they would have been able to had they never engaged in research. When society needs answers, it does not turn to published journal articles. It turns to people.

Conclusion

With the corpus delicti in front of you, you do not say, ‘Here is evidence against the hypothesis that no one is dead.’ You say, ‘Evidently someone has been murdered.’ (Berkson, 1942: 326)

The ESRC-led International Benchmarking Review of UK Sociology and ensuing Q-Step funding programme represent a transparent elite attempt to impose a more quantitative national sociology in the United Kingdom (Platt, 2012). Similar (though perhaps less blatant) pressures exist in most other countries. It is hard to see why elite funding bodies should seek to give preference to any particular type of data, but easy to see why elites of all kinds would seek to give preference to modes of inquiry that are widely perceived as functional over modes of inquiry that are widely perceived as critical. One suspects that the hidden agenda of the ESRC and similar bodies is not the imposition of the use of quantitative data as such but the imposition of the positivist research paradigms that are closely associated with the use of quantitative data. This becomes obvious from a comparison of the relatively critical ESRC review of sociology (ESRC, 2010) with the absolutely hagiographic ESRC review of economics (ESRC, 2008) – conducted at a time when even the Queen was asking how economists got things so wrong.

This article is not the first to advocate that researchers should continue to embrace quantitative methods while nonetheless diluting their positivism. Sophisticated methodologists like Gorard (2006) have always recognised the central role of researcher judgement in statistical modelling and mixed-methods approaches arose from an explicit attempt to bridge the positivist–interpretivist divide (Johnson and Onwuegbuzie, 2004). Advocates of the computationally intensive techniques that collectively fall under the rubric of ‘analytical sociology’ go even further in their attempts to derive interpretation from mathematics (Hedström and Bearman, 2009). In fact the analytical sociology school – which includes practitioners of agent-based modelling, complex adaptive systems, game theory and social network analysis – can be scathing in its criticism of conventional positivist quantitative social science. For example, Byrne (2012: 13) concludes in his review of the 2010 International Benchmarking Review of UK Sociology: ‘[W]e need a quantitative programme which actually corresponds to social reality and that is not to be found in statistical methods which reify variables and consider causality in linear terms.’ Both Byrne and the analytical school are wrong to give up on classical statistical techniques like regression analysis – they are far more flexible and powerful than the systematic comparison approaches Byrne advocates – but they are right to conclude that these techniques as currently used are producing results that have little value for society.

Contra Byrne (2012), classical statistical techniques can produce quantitative sociological analyses that are meaningful, understandable and applicable and contra Savage and Burrows (2007) classical statistical techniques provide quantitative sociology with the incredible scope and power that can only be achieved through the application of multivariate statistical models to data derived from sample surveys. Classical statistical methods are not the problem in quantitative sociology. The problem is that in most cases quantitative sociology unnecessarily retains the philosophical, rhetorical and methodological baggage of the experimental sciences. An interpretive quantitative sociology would leave this baggage behind. Interpretive approaches can make quantitative analyses more meaningful by providing a strong philosophical foundation that researchers can use to counter the pressure towards the use of ever more arcane statistical models, whether or not they are appropriate to the research questions asked or data at hand. They can make quantitative analyses more understandable by providing a strong rhetorical foundation that researchers can use to justify the straightforward presentation of research outcomes in line with how they actually emerged in the course of research practice instead of feeling compelled to reconstruct their research in artificial terms ‘as if’ it had been conducted in line with predefined expectations. Finally, they can make quantitative analyses more applicable by providing a strong methodological foundation for the acceptance of a reflexive research practice that translates directly into the development of the kinds of expertise required for formulating policies that can accommodate complex social realities.

Sociology would benefit more from increasing the sophistication of its people than from increasing the sophistication of its statistics. Involvement in quantitative research is an important way for sociologists to increase their levels of sophistication, along with involvement in qualitative research, involvement in theorisation, involvement in teaching and involvement in public outreach. Interpretive and reflexive modes of engaging in all of these activities are more likely to result in the development of higher levels of sophistication and expertise than are positivist and unreflexive modes. When former US Federal Reserve Board chairman Alan Greenspan told the US Congress in 2008 that ‘[t]hose of us who have looked to the self-interest of lending institutions to protect shareholders’ equity (myself especially) are in a state of shocked disbelief’, sophisticated social scientists of all disciplines collectively rolled their eyes. Alan Greenspan’s economics relied on a high degree of a priori theorisation to make very strong claims about causality that were comically implausible and turned out to be tragically wrong. Most sociologists, quantitative as well as qualitative, prefer to make more modest causal claims. They should adjust their research philosophies, rhetoric and methodologies to match.