Clear,rigorous and relevant: publishing quantitative research articles in Work,employment and society

Empiricism

To set out an empiricist approach to quantitative analysis in sociology, we turn to John Goldthorpe, arguably the pre-eminent British sociologist undertaking research using quantitative methods, and a powerful advocate of an empiricist approach to quantitative data analysis. Goldthorpe (2001) makes the case that there are three broad approaches to quantitative analysis, each based on a set of assumptions about epistemology. He argues that two of these approaches, ‘causation as robust dependence’ and ‘causation as consequential manipulation’, are unsatisfying from a sociological perspective, but that a third approach, ‘causation as generative process’, offers a better way forward for sociological research.

To outline briefly Goldthorpe’s argument; ‘causation as robust dependence’ relates to a positivist or neo-positivist approach to statistical analysis still much used in work psychology and related areas of business and management studies. The key idea in this approach is that while correlation does not necessarily equal causation, there can be no causation without correlation, therefore an absence of correlation can be used to falsify a theory. Inferences about causality are then made on the basis of theory that is not falsified by the analysis. The problem here is that there are often numerous partially overlapping theories that might explain an observed correlation. Therefore the researcher risks ending up in a tautological position where statistical analysis informed by theory is structured in a way that means results seem to support that theory. However it is not actually possible to test directly the causal relationships specified by the theory with the available data. While it is possible to reject a theory on the basis of this type of analysis, theoretical complexity means that it is often not possible to deduce causal explanations from it.

By contrast, ‘causation as consequential manipulation’ is designed to overcome these weaknesses. It involves using randomized control trials or natural experiments to draw stronger inferences about causality by directly observing the effects of causes. In recent years, this approach has become the dominant research paradigm within labour economics. Goldthorpe identifies five key problems with this approach. First, there may be ethical issues that prevent an experimental design being employed to investigate many sociologically interesting questions. Second, adequately measuring all of the factors that might influence the outcome of the experiment is tricky in practice and these factors may not be independent of one another. Therefore results may be biased as a result of these omitted variables and failure to account for interdependence. Third, variables (e.g. race and gender) that cannot easily be manipulated via an experiment have causal powers too, but this approach does not allow them to be observed or measured. Fourth, the approach obscures the role of human agency behind the experimental manipulation of structure so it is not possible to decide the extent to which causal powers rest with structure compared to agency. Fifth, because the experiment takes place in a very specific social context which is itself affected by the experiment, the extent to which results can be generalized to other social contexts is not clear. For these reasons, Goldthorpe argues that this experimental approach is unlikely to occupy more than a marginal position in sociological research.

Goldthorpe labels the third approach ‘causation as generative process’. The essential idea is that there are causal mechanisms operating at a ‘more microscopic level’ which generates the causal effects observed through statistical analysis, although these mechanisms might not themselves be observable at the time of the analysis. Therefore no set of statistical analyses to establish causal relationships will ever be definitive, as there is always scope for more fine-grained analysis that tests the underlying causal mechanisms. The challenge for researchers seeking to explain the cause of things is both to identify statistical relationships between variables that might suggest causality, and also to identify and test for the underlying generative processes that cause the statistical relationships.

On the basis of this approach, Goldthorpe elucidates what he sees as a distinctly sociological methodology for undertaking statistical analysis. He identifies three stages, which may in practice overlap with each other. First, establish evidence that the phenomena posited by theory exist and are common enough to have causal effects. The first phase requires essentially descriptive analysis using techniques that may be similar or identical to those used by researchers working in the positivist research tradition. However, rather than using the results of this analysis for developing causal explanation as a positivist would, causal explanation comes from the next two stages of the process. The first stage is important because it prevents causal analysis based on mistaken suppositions (i.e. it allows inadequate theories to be falsified) and because it may establish previously unappreciated empirical regularities which require explanation. Second, to hypothesize the generative processes at the level of social action by spelling out as fully as is practical the causal mechanisms that might explain the regularity. Third, test the hypotheses. This stage involves looking for a new set of empirical regularities at a different level of analysis which would either support or falsify the hypothesized generative mechanisms. Even if support were found, it would ‘suggest’ rather than ‘confirm’ that the hypothesized generative process was the cause of the original empirical regularity, because new evidence may yet falsify the hypothesis or suggest alternative generative mechanisms.

To spell out the difference between this sociologically informed approach and the dominant positivist and neo-positivist models of quantitative research in psychology or economics; in the latter disciplines, the usual starting point is a theoretical model that is then tested so allowing causality to be inferred. By contrast, following Goldthorpe’s approach the starting point is to use statistical analysis to identify empirical regularities that confirm the existence of a sociologically interesting phenomenon, then to develop and test theories that might explain the regularity in a provisional way.

Critical realism

Critical realism as a philosophy of social science is critical of empiricist and positivist approaches to quantitative research on ontological grounds. However, in contrast to post-structuralist ontologies, critical realists maintain that the scientific study of society is possible. The starting concern of critical realism is Bhaskar’s (1975) observation that much scientific activity was centred around the search for ‘constant conjunctions’ (universal regularities), but that such constant conjunctions do not in fact exist. Consequently the realist critique rests on the observation that there are no universal rules of social behaviour, therefore there is no point seeking universal theories to explain social behaviour. If experiments to uncover the causes of social behaviour are based on the flawed assumption that there are universal rules, results will be wrongly interpreted as revealing universal rules, when in fact they are contingent on a specific social context which might not be replicable and which it may not be possible or feasible to measure and model accurately. Furthermore, it is not possible to measure accurately all causal factors through survey analysis, because causes are not always directly observable (Kemp and Holmwood, 2003: 167; Olsen and Morgan, 2005: 256).

It is immediately apparent that there is a high degree of sympathy between critical realist critiques of positivism and Goldthorpe’s critiques of ‘causation as robust dependence’ and ‘causation as consequential manipulation’. However, a shared critique of specific approaches to quantitative and statistical analyses does not necessarily mean that there is a shared understanding of the way in which quantitative and statistical analysis can contribute to sociological analysis. Indeed, many critical realists have been weary or dismissive of quantitative analysis as a research methodology (e.g. Fleetwood, 2001; Lawson, 1997; Sayer, 1992). In response, Kemp and Holmwood (2003) and Olsen and Morgan (2005) have advocated distinctively critical realist approaches to quantitative analysis within sociology, which we believe overlap to a considerable extent with the approach advocated by Goldthorpe. Specifically, there is a clear overlap between Goldthorpe’s position and Kemp and Holmwood (2003: 178), who argue that statistical analysis can be used to identify partial regularities (demi-regs in the language of critical realism) and that these demi-regs may provide a basis for testing theories of the causal power of social structures by falsifying theories that do not explain the demi-regs. Similarly, Olsen and Morgan (2005: 278) stress the importance of looking inside the ‘black box’ to identify causal processes that explain statistical associations, an argument compatible with Goldthorpe’s focus on identifying and testing generative mechanisms.

Despite these areas of agreement, Olsen and Morgan’s critical realist position contains important points of difference with Goldthorpe’s empiricism. First, there is a more strongly developed sense of scepticism about the ability of survey data to represent complex social realities. Olsen and Morgan (2005: 276) argue that surveys capture ‘ficts’ – attempts to represent reality which may be untrue or subject to error, rather than facts. Nevertheless, they argue survey data are useful in providing traces and outlines of an underlying reality. Because of this scepticism about survey data, Olsen and Morgan go further than Goldthorpe in arguing for a mixed methods approach and the use of induction from qualitative research in developing causal explanations and identifying causal mechanisms. Despite these differences, there is a shared belief in the need to establish and test for the underlying causal mechanisms and (if for different reasons) the provisional and contingent nature of research results between Olsen and Morgan and Goldthorpe. Therefore, despite differences over ontology, in practice there is a strong degree of overlap in ideas about what constitutes good quantitative research between Goldthorpe’s empiricism and critical realist advocates of quantitative analysis.

From these areas of overlap we can deduce six principles to guide quantitative sociological research: first, sophisticated statistical analysis of survey data can be used to describe empirical regularities in order to confirm the existence of and investigate associations between different social phenomena; second, on the basis of this description it is possible to test and reject theories which do not correspond to the empirical regularities; third, to explain what causes the empirical regularities, it is necessary to look inside the ‘black box’, to hypothesize what the causal mechanisms might be and then test these empirically. This may involve induction and mixed methods research; fourth, research programmes constructed in this way will always be provisional in their conclusions. New empirical regularities or new evidence on potential causal mechanisms may lead to revised conclusions. Limitations of data or measurement, specifically the difficulties involved in accurately representing complex, multi-layered social realities through survey data, may prevent all potential generative mechanisms from being tested; fifth, therefore researchers should display a critical awareness of the limitations of their data and analysis, in terms of sample, schemes of classification, measurement instruments and methods; sixth, given these limitations, it is important to be transparent in explaining and justifying methods by considering the robustness of results to changes in module specification or statistical technique.

We do not expect the preceding analysis to win over sceptics who doubt that complex social realities can be adequately categorized and measured through survey data, but we note in passing that the perspective outlined above is not merely an abstract exercise in summarizing epistemological reasoning, but an approach that is being used by sociologists of work and employment. For example, it closely corresponds with the approach that Leslie McCall (2005) has advocated and employed in the study of intersectionality, work and employment, and the approach can be seen in operation in any number of recent articles published in WES, for example Brynin and Güveli’s (2012) analysis of the ethnic minority pay gap in Britain and Morgan et al.’s (2013) mixed methods study of job satisfaction among frontline healthcare workers. What are the practical implications of this approach for our expectations of articles submitted to WES?

One of the major reasons quantitative articles are rejected is that reviewers and editors judge that they do not make sufficient contribution to sociology. Articles that are essentially works of labour economics, social psychology or atheoretical industrial relations are unlikely to be accepted. In a number of areas (e.g. unemployment and its effects, issues of wages and skill, the relationships between work, health and well-being) the boundaries between the sociology of work and other social science disciplines and fields are fuzzy. In these circumstances, articles often fail because although it is possible to discern a relevance to the sociology of work and employment within them, this is left implicit or hidden behind a facade of theory from other disciplines. In order to be judged to make a clear contribution to sociology, articles need to be framed in explicitly sociological terms or to be able to demonstrate a clear relevance to sociology. There needs to be evidence of sustained engagement with sociological theory or with sociologically interesting questions and existing sociological research throughout the article. Articles may be grounded firmly in sociological theory (e.g. Gallie et al., 2012; White, 2012) or consider sociological theories alongside theories from other disciplines and fields in order to answer sociologically interesting questions (e.g. Holman, 2013; Jones and Wass, 2013; Reeskens and Van Oorschot, 2012).

It is essential that articles include a succinct and clear description of the data. When research is based on a survey, the most important pieces of information to include are how the sample was drawn and response rates, specifically whether or not the sample was drawn randomly, and the population from which the sample was drawn. This information is essential if readers are to understand the extent to which the results can be generalized. It is important to remember that large-scale social surveys tend to have potentially problematic issues with aspects of their design and execution. It is important that limitations arising from these issues are acknowledged and discussed (Timming, 2010). It is unlikely that articles which are based on samples of convenience will be accepted for publication, because with samples drawn in this way, we cannot know the extent to which results can be generalized beyond the sample. A possible exception to this rule would be if a researcher is seeking to test a specific generative mechanism, so has constructed a convenience sample where that mechanism is more likely to be observed. Articles based on whole populations, administrative data or other forms of ‘big data’ captured from company records of workers’ real time behaviour are also welcomed. It is also important that measures used in the analysis are clearly and concisely explained, possibly in the form of a summary table, so that the reader can make a judgement about how well measures and categories used are likely to capture the complex social reality that they seek to measure. This is particularly important when key measures take the form of multi-item scales. Authors must be clear about the analytical procedures used to generate the scale, for example which form of factor analysis was used, and why this method was selected. Results of factor analyses used in data reduction should be included so that readers can understand the process. Robinson and Pendleton (2010) provide an example of where this has been done particularly clearly and effectively.

Statistical methods need to be appropriate to the type of data being analysed. For example, if research is based on regression analysis, then the models estimated must be appropriate to the form of dependent variable. Ordinary Least Squares (OLS) regression is appropriate when the independent variable is continuous and normally distributed. If the dependent variable is dichotomous, probit or logit estimation are more appropriate, while if the dependent variable takes the form of an ordered scale, ordered probit or logit would normally be the appropriate analytical method. Other forms of regression analysis may be used depending on the form of the independent variable (e.g. poisson, interval regression, tobit). If OLS analysis is used when an ordinal dependent variable is being analysed, authors must provide a clear rationale for this, and be able to demonstrate through sensitivity analyses that their results are not an artefact of the statistical method. Panel econometric methods should be used when analysing panel data.

Regression analysis is one of the more commonly used statistical methods, and it is one that the majority of the WES readership will have some familiarity with. If an article uses an alternative form of statistical or econometric analysis, for example structural equation modelling (SEM), multi-level modelling or more unusual or novel forms of regression analysis including panel analysis, it is important to give the reader information on what the method tells us and why this method has been used, that is, what does it tell us that a more typical regression analysis would miss or get wrong? Panel analysis, SEM and multi-level modelling can, if used appropriately, be used to facilitate careful exploration of causal complexity, and can therefore contribute to the development of causal understanding over and above what can be achieved by simpler forms of analysis, but the reasons for using these methods need to be clearly explained so that the reader can judge the value that they add. An example of where this is done effectively is Schober and Scott’s (2012) analysis of attitude–practice dissonance in the transition to parenthood. A large part of the WES readership are not expert in quantitative methods, therefore it is important that methods are explained in language accessible to a general reader without a background in statistics or econometrics. Some econometric notation may be desirable to communicate succinctly and precisely the form of the analysis, but this should be kept to a minimum. Notation is not a substitute for clear and accessible prose explanations.

We would particularly encourage the submission of articles or research notes that outline and explain the benefits of novel quantitative methods to sociological research. Similarly, articles or research notes that provide a critical perspective or re-evaluate existing widely used quantitative research methods are welcome (e.g. Mood, 2010).

In an era where it is perceived that the pressure to publish is resulting in increasing numbers of academic journal articles with results that are not robust enough to be replicated by other scientists (The Economist, 2013) it is important to explain methods transparently so that they can be replicated by other interested researchers. All variables included in a model must be clearly explained (perhaps in the form of a summary table) so that another researcher making reasonable assumptions and working on the same data could construct them. Correlations between variables and standard errors should be reported (if this is not possible because of space constraints, this information should be included in an appendix made available via the WES website) so that results can be included in meta-analysis studies. Authors should specify the software and, where relevant, the commands used to generate results. Authors should be prepared to make their data and syntax used to code and analyse the data available to other interested researchers (or in the case of licensed or restricted data, the syntax only). Intermediate stages in the analysis (e.g. factor analysis to identify latent variables and create scales) should be reported.

It is important that articles report meaningful analysis and tell a clear story. As most quantitative research projects will have produced far more results outputs than could possibly be accommodated in an 8000 word journal article, or be easily digested by a reader, it is necessary to be judicious in the selection of results to report in the article. Authors should focus on reporting those results that are theoretically relevant and which therefore add to the story or argument that the article is developing. However, this selection process should not obscure the wider research process that underpins the results; for example results that do not accord with initial hypotheses should be acknowledged. Additional analysis, particularly sensitivity analyses which check the robustness of the results, should be referred to in the text. If both dependent and independent variables in an analysis are from the same respondent and particularly if they are based on similar types of scale responses (for example an analysis of the relationship between self-reported effort and self-reported autonomy), it is important to acknowledge this limitation and it is good practice to explain the steps taken to account for or check for the influence of common method bias (see Lindell and Whitney, 2001; Podsakoff et al., 2003 for reviews of the relevant statistical methods). Where scale variables are used following factor analysis or other forms of data reduction, the sensitivity of results to alternative approaches to data reduction should be investigated and described. Authors should have these additional analyses available in the form of an appendix that can be supplied to other interested researchers or published on the journal website alongside the article.

All results tables should be easily interpretable without reference to the text, that is, variable labels in the table should readily understandable (e.g. ‘unemployment rate’ not U_rate), table titles should be descriptive of the contents of the table, model diagnostic statistics and indicators of statistical significance should be included within the table and notes explaining any aspects of the table not readily understandable to the reader should be added beneath it. Results included in tables should be reported in a form that can be interpreted by a general reader. For example, coefficients from probit or ordered probit analyses are not easily interpretable, so should be transformed to marginal effects, odds ratios or exponentiated coefficients as appropriate. Graphical representation of results is encouraged if it makes them clearer and easier to understand. Robinson and Smallman (2013) provide an exemplary case of how to present data and results concisely and clearly.

Following presentation of results, articles should re-engage with the theory(s) that motivated them. The crucial point, following from the analysis of epistemology above, is that researchers need to move beyond simple empiricism and crude positivism to consider both what their results say for the plausibility of theory, and the additional theoretical and empirical work required to advance causal understanding further.