UK Sociology and Quantitative Methods: Are We as Weak as They Think? Or Are They Barking up the Wrong Tree?

We are frequently told that UK sociology at all levels from the basic training of undergraduates through to the actual research output of the discipline has a problem with quantitative methods. Most recently, the International Benchmarking Review of UK Sociology jointly sponsored by the ESRC, the BSA and the Heads and Professors of Sociology asserted that:

The upshot of our assessment is … that British Sociology remains weak in quantitative methods (which might well be a matter of tiers, with one tier of high level research methods reflected in sociological engagement with NCRM [the National Centre for Research Methods] and the Researcher Development Initiative, and a lack of diffusion into mainstream sociology departments as the other tier). …

That UK sociology makes relatively little use of statistical methods, and the study of their epistemic status, is problematic both in neglecting a sociological object of inquiry and in its engagement with the other social sciences in the international domain. (BSA, HaPS and ESRC, 2010: 23)

These are not isolated comments. Although the benchmarking review was generally complimentary about UK sociology, larded through it are critical comments relating to the UK discipline’s quantitative capacity in terms both of its research programme and what is delivered by it in teaching. This is identified in relation to business user views (section 2.2.8: 16) and in relation to the future of sociology as a discipline (section 5.2: 38) Indeed, one of the five final recommendations of the international panel was precisely to:

… Strengthen access to and offers [sic] (at least basic) training in statistical empirical analysis and other quantitative methods, as well as access to advanced research training especially for those thematic and ‘core’ sociological areas where interdisciplinary and/or international comparative work requires expertise in quantitative methods. (2010: 41)

There is little to quarrel with in the simple assertion that UK sociology needs to engage with the quantitative. Of course there is an issue of lack of competence. Many UK sociologists are to all intents and purposes essentially innumerate. They lack fundamental mathematical skills and combine fear of quantitative work with an all too often scornful dismissal of it. However, the authors of the Benchmarking Report do not simply diagnose. They prescribe in definite terms:

The social sciences have a shared interest in the scientific study of people, grouped into aggregates in various ways. In pursuit of this interest, sociology has in common with economics, psychology and political science the extensive use of statistical methods applied to data on individuals and groups, both for descriptive purposes and for the development and testing of hypothesized causal processes. Of course statistical methods are not the only valid mode of inquiry, and each of the social sciences also embraces its own theoretical and quantitative approaches. But, arguably statistical methods form the core of social science. (2010: 23)

Whilst it is entirely appropriate to agree that failure to engage the majority of UK undergraduate sociology students with statistical methods is an issue, and their effective innumeracy – (see the ESRC examination of the teaching of quantitative research methods at undergraduate level in UK social science, MacInnes, 2009) – is a real problem, ¹ the assertion of ‘statistical methods’ as the core of social science is absolutely contentious.

The Gulbenkian Commission’s reporting on the future of the social sciences stated that:

In many ways the most severe problems have been with the three more nomothetic social sciences [here economics, psychology and sociology]. In taking the natural sciences as a model, they have nurtured three kinds of expectations that have proved impossible to fulfill as stated in universalist form: an expectation of prediction; an expectation of management; both in turn premised on an expectation of quantifiable accuracy. … the nomothetic social sciences built themselves on the premises that social achievements can be measured, and that the measurements themselves can be agreed upon universally. (1996: 50)

This is, to say the least, a fundamental challenge to the quantitative programme across the social sciences as that programme has been implemented up to now. The possibility of a different quantitative programme remains. There is no sense of this issue in the Benchmarking Report. Note that the Gulbenkian Commission challenged the whole process of measurement itself, whereas the benchmarking review simply took this for granted and proceeded to an endorsement of statistical methods, and by clear implication statistical methods essentially similar in form to those deployed in economics and psychology and in much of non-UK sociology, as ‘the core of social science’. There are real arguments to be had about measurement – see for example Desrosières’ critique founded in an actor network frame of reference in relation to social statistics as a whole (1998). These are simply ignored here. What is being asserted is that UK sociologists should deploy methods of the form of, for example, structural equation modelling in their work. They don’t do so very much, so UK sociology has a deficit. There is no real critical consideration of whether a good reason for the lack of employment of such methods is that, to put it plainly, they are essentially useless. The issue is merely one of lack of competence, of a deficit in UK sociology which needs to be rectified.

However, we really do need to consider whether the solution to that problem is to go down a road which in the view of the Gulbenkian Commission (and very much of the present author) has been a complete dead end. These matters have been at the heart of an epistemic critique of the status of the most commonly deployed statistical methods which has been written in part by UK sociologists. These critiques are absolutely not arguments against the use of quantitative methods in social research. Rather, they raise, both in attack and defence, the value of the dominant methods of quantitative explanation – that is, the mode of quantitative investigation and representation of the social world which depends on the assertion of the variable as something which generally can be understood outwith the case and which understands change to be linear and incremental rather than non-linear and qualitative. Implicitly, sometimes even explicitly, those attacking the dominant mode of linear statistical reasoning have also questioned the use made of the concept of probability in relation to causation in the social sciences.

To illustrate this critique we can consider Pawson’s assertion that:

What is needed is a much more refined quantitative method which is sensitive to the phenomenological and relativist critiques of the last two decades but which is self-confident enough not to be led up a thousand garden paths in the belief that some methodological salvation lies with some yet-to-be discovered epistemological miracle. (1989: 2)

Actually, the garden paths we are being led up by the benchmarking review have scarcely considered epistemology at all. Rather, in practice, they are all about method as technique. In so far as they pay attention to measurement they do so in terms of issues of validity which never challenge the reality of the variates they deploy. So the question is always are we measuring this thing properly, as opposed to whether there is anything there, which has an independent and real character, to measure in the first place. The emphasis on ‘independent and real’ is necessary because we certainly can and should measure but ought to have a very different conception of the things we measure. They are attributes which have no reality outwith the cases. Cases themselves can be understood in complexity terms as real open systems. If we understand our measurements to be of attributes then the measurements describe traces of the systems as they are at one or more time points, rather than ‘variables’ with independent causal powers (Abbott, 1998).

Goldthorpe is often regarded as a radical defender of quantitative approaches and certainly Oxford/Nuffield Sociology, in considerable part under his influence, is generally understood as a lonely bastion of the quantitative in the howling wilderness of qualitative and relativistic British sociology. However, that would be a simplistic interpretation of what is a rather sophisticated consideration of the issues under discussion here. Having endorsed Van den Berg’s assertion that sociological theory has become a kind of self-referential intellectual game ‘wholly divorced from the concerns of practicing social researchers’ (2000: 4), ² Goldthorpe continues:

… as a scarcely surprising reaction, many researchers have formed strongly negative views of general theory – and have then, by an unfortunate and unwarranted extension, tended to become dismissive of theory in general. This attitude they express less in explicit critique than in assumptions implicit in their practice: for example, the assumption that research findings can largely ‘speak for themselves’ without need of theoretical explanation, and that where data are given quantitative form causal relationships can in any event be determined directly from statistical analysis. Thus as van den Berg further observes (1998: 206), the ‘autonomization’ of theory here finds its equivalent in the autonomization of technique and in a way that is ‘no less absurd or damaging to the discipline as a whole.’ (Goldthorpe, 2000: 4)

What Goldthorpe wants is a theory-informed empiricism and for him that requires a turn to rational action theory. Be that as it may, he explicitly endorses a US-based critique of causal modelling methods in sociology and goes on to agree with those expressing this position asserting that:

… the whole statistical technology that has underpinned the sociological reception of the idea of causation as robust dependence, from Lazarsfeldian elaboration through to causal-path analysis, should be radically re-evaluated. That is to say, instead of being regarded as a means of inferring causation directly from data, its primary use should rather be seen as descriptive, involving the analysis of joint and conditional distributions in order to determine no more than patterns of association (or correlation). Or, at the very most, representations of the data might serve to suggest causal accounts, which, however, will need always to be further developed theoretically and then tested as quite separate undertakings. (2000: 152–3)

Goldthorpe’s suggested approaches include log linear methods which do not distinguish between independent and dependent variables. These methods still depend on an acceptance of the simple reality of the variables and can only deal with complex causation through the fitting of interaction terms, although in practice this is seldom done. Hedstrom takes the same line on causal modelling which he defines as: ‘… the use of large-scale non- experimental data to estimate parameters of statistical models, which are then interpreted in causal terms’ (2005: 101). ³ He proposes a move towards agent-based modelling. ⁴ However, if we read through the quantitative literature which addresses topics of interest to sociologists we are far more likely to encounter causality in relation to categorical level data explored, and in fact generally explained, through techniques of logistic regression which do precisely require a distinction between caused and causal. A survey of articles in the main UK journal of sociology’s cognate discipline social policy, the Journal of Social Policy, found that, in the limited number of pieces where techniques went beyond simple description, this was almost invariably the approach adopted. Sophisticated description is interesting but doing it on the basis of variables considered as real seems less productive than, for example, the use of clustering techniques which can be deployed when measurements are understood not as real in themselves but as traces of underlying real systems (see Byrne, 2002, for a development of this argument).

The status and nature of the quantitative programme is not just an important argument in relation to modes of scientific reasoning and empirical investigation. It also plays largely in the politics of the UK academy where the ESRC has constantly harped on – no other expression will do – about the deficiencies in quantitative skills among researchers and research students across the social sciences other than in the disciplines of economics and psychology. By implication, and frequently in explicit terms, those disciplines have been identified as exemplars from which the rest of the social sciences, including sociology, have a good deal to learn. This piece will leave the deficiencies of quantitative modes of investigation and causal reasoning in psychology for another day and/or to other hands, but some harsh but necessary comments will be made in relation to that programme in economics. The recent ESRC process for the establishment of selective and elite Doctoral Training Centres, whilst paying some attention to the need for all research students regardless of discipline to have some training in qualitative methods, placed a great deal more emphasis on advanced training in quantitative methods. An examination of the current investments in quantitative methods funded by the ESRC with the objective of building a ‘world class social science research base’ demonstrates an absolute preponderance of linear modelling techniques targeted on longitudinal data sets with an emphasis on micro data. There is nothing wrong with the collection of longitudinal data. ⁵ The collection of such data is invaluable for the exploration of social trajectories and retroductive search for causality across them – but it is noticeable that less attention has been paid to the collection of data on social institutions and other entities, other than to data which at best is in the form of statistical aggregates at the micro-level.

The Benchmarking Report noted that: ‘… a standard undergraduate methods course will include as much time critiquing the use of quantitative methods as teaching them (although critique presupposes an understanding of what is critiqued’ (BSA, HaPS and ESRC, 2010: 23). We can agree that critique requires understanding. However, so does use. There is little evidence that most of those who utilize methods based on developments of the bivariate correlation (Ragin and Rihoux, 2004a, note methods up to and including structural equation modelling, and they might have added multi-level modelling, all derive from this) actually understand the ontological claims which are fundamental to their deployment. ⁶ Certainly there seems to be minimal evidence that most economists are ever taught anything which questions the fundamental assumptions of the techniques they apply. There is something fundamentally wrong with the uncritical propagating of methods which a substantial body of social scientists are now prepared to assert are incommensurate with the nature of social reality and in particular with processes of causation in social reality. Let us rehearse the attack mounted on them.

First we have to question the reality of variables as entities which can in any way be detached from the real social ‘cases’ which are the object of social research. This is not to question variation or the value of the measurement of variation. Rather it is to question the ontological status of variate entities existing in and of themselves. Here it is useful to think of the difference between variables considered as real in and of themselves and the idea of attributes (which, in terms of complexity theory approaches to the dynamic character of social entities, can be considered as variate traces, i.e. attributes which differ not only across cases but for cases through time) that are intrinsically inseparable from real cases and have no separate existence or causal powers. Abbott notes that in the future social scientists might express surprise that:

The people who called themselves sociologists believed that society looked the way it did because social forces and properties did things to other social forces and properties. … Sociologists called these forces and properties ‘variables’. Hypothesizing which of these variables affected which others was called ‘causal analysis’. … what made social science science (original emphasis) was the discovery of these ‘causal relationships’. (1998: 148–9)

Goldthorpe’s entirely appropriate questioning of the value of regression-based modelling in relation to the exploration of causality does not take on board this fundamental issue. It implies more than just a recognition of the value of statistical description but rather requires a very different understanding of how social causality can be represented in quantitative terms. Abbott (1998) suggests a turn to simulation and Hedstrom (2005) takes the same route. Ragin (1987) argues for an understanding of causality based on set theoretic relationships and has developed variants of Qualitative Comparative Analysis (QCA) as a tool for investigating such relationships. This is wholly compatible with a generative and contingent understanding of causation in the realist tradition in that it allows for causes to be both multiple and complex. Conventional regression-based methods can only deal with complex causation very clumsily through the insertion of interaction terms and even this is seldom done in published work. They cannot deal with multiple causation at all.

Another fundamental issue is the reliance of conventional regression-based methods on linear relationships. The great majority of approaches assume simple linearity – that is, that the transformation in the value of a dependent variable is an incremental function of the transformation in one or more of the variables causal to that variable. Associated with this is the emphasis on partialling out the discrete contribution of variables in the causal set. Of course, there may be transformations in the ‘independent’ variables. They may be expressed as logs, squares or indeed any available algebraic function. The whole purpose of such transformations is to introduce linearity into the regression equation. The point remains that the focus is on proportionality expressed in terms of linear dependency. Why is this such a problem? Because in the social realm the changes that matter are not incremental changes of degree but fundamental changes of kind. Why is this the case? Because the social world is essentially composed of complex systems in relation to which human agency has causal power. Certainly, quantity can become transformed into quality. We do have tipping points but linear methods in general cannot handle such transformation and the approaches which can (for example, threshold functions as described by Seawright, 2004) are clumsy and complex in comparison with much simpler and more direct approaches such as a combination of time-ordered cluster analysis to establish kind and QCA to explore causality. Threshold function approaches do not address the issue of multiple causation – more than one way for effects to become manifested – and, as with other regression-based techniques, can only allow for complex causation through the insertion of interaction terms.

The above implies that in the social realm, and indeed in all intersections (e.g. ecological) of the natural and social realms, we are dealing with complex systems with emergent properties. That means that effects are to be understood in terms of the state of such systems through time; that is, in terms of trajectories. We are interested in relative stability of state – in complexity terms, systems moving through a multi-dimensional torus attractor or, in plain English, jiggling about a bit. We are even more interested in changes of kind, in phase shifts, which involve radical transformation of state. For complex social systems, which may be constituted by anything from individual human beings to the whole global social order, ⁷ the nature of trajectories and the establishment of what is causal to those trajectories is a crucial task for social science. To say that properties of complex systems are emergent means that they cannot be established by an analytical strategy of reduction to component parts. In terms of searching for causality, this means that causality cannot be established by assigning partial contributions to discrete variables, even in the less common instances where we are dealing with real variables extrinsic to our systems of interest, as is sometimes the case in relation to policy interventions. We always have to think in terms of complex cause. The methods which characterize contemporary quantitative social science are, with the exception of simulation approaches, ⁸ generally incapable of addressing emergence.

A further issue emerges in relation to the reliance of statistical methods on probabilistic conceptions of cause. There is a very real place for probability in quantitative social science as the basis of inference – the process of making statements about a population on the basis of information about only part of it, on the basis of a sample. However, to understand causes as probabilistic is to make a major error. It is to endorse what Desrosières (1998) identifies as the frequentist or objective concept of probability in contrast to the subjective or epistemic perspective. Bayesian methods adhere to the subjective/epistemic position. Probability is a matter of what we do not know and we attempt to eliminate ignorance and produce better accounts; we deal in a science of clues. In contrast, the frequentist approach seeks to establish nomothetic universal laws expressed in probabilistic terms. Note that this is not the same objective as bench experimental science which seeks to establish nomothetic laws expressed in deterministic terms. Indeed, this contrast between statistical law making and experimental law making was precisely what led Znaniecki (1934) to propose analytic induction as a better way for establishing regularities in the social world. There does seem to be a belief among most statisticians that the world has to be understood in probabilistic terms; for example, in a severe critique of QCA Lieberson notes:

… the fact that QCA is less prepared to allow for chance and probabilistic processes than is the case in many of the hard sciences. This ought to make us stop and wonder. If we are operating in a probabilistic universe, and if we recognize that there are errors in data and that almost surely the comparisons involve influences that are not comparable or necessarily or always measurable (even if imperfectly), then it can be bothersome that the data table provided in QCA cannot ascertain whether the observed pattern includes a stiff dose of random results and leads to massive over-interpretation. Indeed the procedures do not rule out the possibility that the observations are all a random matter and / or that none of the causal variables were even measured. (2004: 13)

That passage hits all the wrong bullseyes at once. It reifies variables – the emphasis on validity and reality of the variable is central to the argument. It asserts in an essentially Platonic fashion the fundamentally probabilistic character of all reality. And it lacks any sense of complexity in relation to causation. Ragin and Rihoux (2004b) in their reply to this and other criticisms point out that Ragin’s fuzzy set QCA does allow for probabilistic estimates, ⁹ but I for one read the development of those estimates as a recognition of the possibility of sampling error rather than as an endorsement of any fundamentally probabilistic character to social causation. In many instances in social science we have all the cases. This is not only true in relation to small and medium N universes, for example all advanced post-industrial nation states, ¹⁰ but also for large N sets, for example all secondary schools in England or all city regions in the European Union. ¹¹ In such instances we do not even need to engage with probabilistic reasoning in relation to issues of inference from samples.

Let us turn to that exemplary quantitative social science, economics. If only all other social scientists had the quantitative skills of economists, what a wonderful world we would have with really hard developments of nomothetic laws and a consequent real ability to engage in valid social predictions. Or NOT! The fundamental basis of economics’ quantitative programme has been challenged both by critics drawn from within the discipline itself – the generic group of heterodox economists – and by the actual development of social reality. Ormerod in The Death of Economics (1994) asserted that:

… orthodox economics is in many ways an empty box. Its understanding of the world is similar to that of the physical sciences in the Middle Ages. A few insights have been obtained which will stand the test of time, but they are very few indeed, and the whole basis of conventional economics is deeply flawed. (1994: ix)

He goes on to describe the general character of academic economics as: ‘virulently esoteric chat’ (1994: 67). This would be bad enough if we were dealing only with representations of the social world, but we have to remember that it is this ‘virulently esoteric chat’ which legitimizes the nonsense that competitive markets are the solution to all public policy issues. As Ormerod has remarked (2010), it is this nonsense which has brought us to the brink of financial meltdown and global economic depression. Contemporary academic economics is highly quantitative but behind what Joan Robinson (1954) called ‘thickets of algebra’ there is an emperor with no clothes at all. Ormerod has turned to what we might call in Morin’s terminology ‘restricted complexity’ for a way out of this morass. For an alternative methodological programme see Downward and Mearman (2007: 77), who propose: ‘Retroduction as mixed-methods triangulation in economic research’ with a view to ‘reorienting economics into social science’. So, the UK ESRC privileges training in conventional quantitative economics by paying research students a premium if they engage with this hard quantitative programme and allows the development of PhD programmes in which two years from four are devoted to rigorous training in wholly useless formal modelling procedures.

Again bluntness is appropriate. The conventional quantitative programme in the social sciences has told us very little of real interest. Goldthorpe argues that:

… various cases can also be cited in which the chief statistical accomplishment has been to identify and characterize important social regularities that were hitherto unappreciated, or incorrectly understood, by in effect separating out these regularities from their particular context. (2001: 11)

One can agree absolutely with Goldthorpe’s insistence on the value of ‘sophisticated description’ as a necessary precursor of any attempt at causal explanation without accepting the validity of the separation out of particular context. Context is everything, something demonstrated absolutely in relation to the issue of social mobility where the formal establishment under Goldthorpe’s influence of an atemporal and acontextual specification of social class has caused considerable difficulties. ¹² Of course there is a reason for searching for atemporal and acontextual regularities. It is so that generative mechanisms can be established, in Goldthorpe’s case a version of rational action theory, which have a nomothetic status. That is to say they apply in all cases. Let us be clear here. If we understand the social as composed of complex systems with emergent properties, then the search for general nomothetic mechanisms is pointless. That is not to say that by a combination of process tracing and systematic comparison, by a historical and narrative-driven approach to investigating cause, we cannot establish causal patterns which hold over sets of cases which have similar path dependencies. Working in that way offers a particularly fruitful approach to causal investigation but will necessarily be retroductive and multi-method in form.

The authors of the UK’s Academy of Social Sciences report Great Expectations (ACSS Commission, 2002) were distinctly huffed by the Gulbenkian Commission’s severe critique of the achievements of the nomothetic social sciences as quoted above, but their petulant assertion that many practitioners of these disciplines take a different view carries little weight. In the immortal words of Mandy Rice Davies: ‘Well they would, wouldn’t they.’ We really do need a quantitative programme in sociology, and in the spirit of the Gulbenkian Commission’s arguments across an opened up social science as a whole. However, we will not get a useful quantitative social science if we think that much of what has been done under that label actually provides us with any sensible guide as to how to proceed in the future.

Of course there are arguments to be had about what constitutes a valid empirical programme in the social sciences in general and in sociology in particular. The point is that the benchmarking review, which is likely to have a very significant impact on the future development of UK sociology, did not address these arguments. Indeed, it might even be considered that by criticizing the amount of attention devoted in taught programmes to critiques of methods then it rejected their significance as a whole. It has to be said that 20 or perhaps even 10 years ago, much of the criticism of the quantitative on methodological grounds presented in taught programmes varied between the naïve and the pig ignorant. This is much less the case today. Bryman’s immensely useful Quantity and Quality in Social Research (1988) did much to improve both argument and, given its accessibility, teaching in these areas. However, these developments were not considered in any real sense in the report. To put it plainly, the report seemed to be based on an understanding of the critique of the quantitative as rejectionist whereas now in the UK it is rather a critique which argues for a different kind of quantitative programme. Of course there are arguments to be had here and they really require book-length treatment rather than brutal summary in short articles. A particular position has been advance above, essentially one which argues for a complex realist meta-theory and suggests that inter-alia QCA-style systematic comparison provided a technique of quantitative investigation which is consonant with that fundamental understanding of social reality and causality. Other positions exist. Arguments among them matter. We will not go forward in a good way if the quantitative issue is understood only in terms of deficit in techniques and if we accept that conventional statistical methods form ‘the core of social science’.