The Methodological Divide of Sociology: Evidence from Two Decades of Journal Publications

Abstract

Past research indicates that Sociology is a low-consensus discipline, where different schools of thought have distinct expectations about suitable scientific practices. This division of Sociology into different subfields is to a large extent related to methodology and choices between qualitative or quantitative research methods. Relying on theoretical constructs of the academic prestige economy, boundary demarcation and taste for research, we examine the methodological divide in generalist Sociology journals. Using automated text analysis for 8737 abstracts of articles published between 1995 and 2017, we discover evidence of this divide, but also of an entanglement between methodological choices and different research topics. Moreover, our results suggest a marginally increasing time trend for the publication of quantitative research in generalist journals. We discuss how this consolidation of methodological practices could enforce the entrenchment of different schools of thought, which ultimately reduces the potential for innovative and effective sociological research.

Keywords

natural language processing research methodology scientometrics sociology of science sociology of Sociology

Introduction

Despite the landscape of sociological research being in constant flux (Moody and Light, 2006; Oromaner, 2008), long-lasting epistemological demarcations between schools of thought exist and raise the question of how to conduct research properly (Au, 2018; Burawoy, 2005; Byrne, 2012; Münch, 2018; Payne et al., 2004). One of the deepest entrenchments between rival camps is the methodological field (Münch, 2018; Payne et al., 2004; Smelser, 2015). This divide spans predominantly between qualitative and quantitative research methods (Byrne, 2012; Turner, 1998; Williams et al., 2017). One result of this divide is that Sociology is a low-consensus discipline torn between rival camps aligning themselves epistemologically either to the natural sciences or the humanities (Leahey and Moody, 2014; Puddephatt and McLaughlin, 2015; Turner, 2006; Varga, 2011). At the same time, these camps are located in different domains of the philosophy of science. One is associated with the humanities and aligned to constructivism, logical induction and theory-building in the sense of Berger and Luckmann (1991) or Glaser and Strauss (2017), while another one is related, for example, to positivism, deduction and falsification in Popper’s (2008) sense. These alignments are deeply linked with the way of conducting research, are thus not reflected by the scholars and manifest in the scholarly discourse held in publication outlets (Moksony et al., 2014; Vanderstraeten, 2010). In turn, this linkage lowers the chances for consensus formation and the successive generation of knowledge in Sociology even further (Boyns and Fletcher, 2005; Burawoy, 2005; Collins, 1989, 1994; Payne et al., 2004; Smelser, 2015; Turner, 2006, 2016; Williams et al., 2017 for an overview on this discussion).

These divisions not only resulted in myriad different topics and research pursued, but also in the emergence of epistemologically demarcated schools of thought (e.g. the ‘Chicago School’ and ‘Columbia School’) and dominant research paradigms (Collins, 1994; Kuhn, 1996). These paradigms combine a limited number of theories, methods, epistemologies and research topics. Paradigms are a focal point for the emergence of research networks and preferences for topics and publication outlets (Moksony et al., 2014; Vanderstraeten, 2010). Paradigmatic alignments also bear the potential of conflict within Sociology and often revolve around nationally embedded epistemic cultures with own approaches drawn from the domain of the philosophy of science. In the United Kingdom, the so-called paradigm wars of the 1980s put a strong emphasis on qualitative methods that are still present today (Bryman, 2008; Gage, 1989). Another example is found in Germany. Here, the so-called ‘Positivismusstreit’ of the 1960s was fought between representatives of the paradigms of critical rationalism and critical theory (Adorno, 1987). The outcome led to a deepening divide between scholars applying qualitative and quantitative methods as well as antagonizing views on theory that still exist today (Münch, 2018).

At the same time, the academic publication market and the perceived focus of publication outlets unintentionally enforce the entrenchment between different schools of thought. For instance, journals might signal a focus on educational outcomes and thus attract a disproportionally large number of scholars who are academically socialized within a research paradigm focusing on quantitative methods and specific types of rational choice theory. If the journal publishes a large number of articles that are arguably linked to the research paradigm, these scholars are incentivized to focus even more on the respective outlet. Since publication outlets provide the main stage for the competition between scholars and paradigms, entrenchments are expected to be most visible in these outlets, especially those said to represent the discipline as a whole (Collins, 1994; Turner, 1998, 2016).

For the above-mentioned reasons, this article examines the existence of a methodological divide in generalist Sociology journals. We focus on generalist interest publication outlets as they are increasingly important for the dissemination of sociological knowledge, and thus increasingly bear the potential to deepen the entrenchment between paradigms (Moksony et al., 2014; Münch, 2018; Puddephatt and McLaughlin, 2015). Moreover, general interest Sociology journals cover research from a wide range of subfields and are therefore relevant for a large body of the research community.

Specifically, we investigate the following research questions:

Is a methodological divide reflected in generalist Sociology journal publications? (RQ1)

If so, to what extent is a methodological divide reflected in tastes for certain paradigms in different Sociology journals (RQ2a) and publication trends over time? (RQ2b)

To answer these questions, we apply three theoretical concepts to explain the methodological divide seen in publication outlets. First of all, we scrutinize Merton’s (1968, 1988) concept of the academic prestige economy. This concept highlights the mechanisms to gain reputation for bestowing knowledge on the discipline that is hierarchized within the paradigms, national context and overall, transnational discourse of Sociology. Second, the concept of boundary demarcation is applied to frame this methodological divide as an actively ongoing interaction between schools of thought, paradigms and outlets (Lamont and Molnár, 2002; Pachucki et al., 2007). Finally, we use the concept of taste proposed by Bourdieu (1989) to explain the stability of the linkage between paradigms and publication outlets, which accounts for the methodological divide. All three mechanisms combined therefore account for the emergence and consolidation of a symbiosis between articles submitted, publication outlets and paradigms.

To measure the methodological divide empirically, we rely on quantitative and automated methods of text analysis. Using unidimensional scaling of 8737 abstracts from articles published between 1995 and 2017, we find evidence of this divide, but also of an entanglement between methodological choices and different research topics (RQ1). This methodological divide is reflected in taste for research on the part of different publication outlets (RQ2a). Moreover, our results suggest a marginally increasing time trend for the publication of quantitative research in generalist journals (RQ2b). They also support the interlinkage between different paradigms and publication outlets and provide evidence of the entrenchment between different schools of thought.

Theoretical Concepts and Expectations

In academia, scholars constantly try to push the frontiers of knowledge. Yet, it is precisely this overall mission of academia that establishes an academic prestige economy (Merton, 1968, 1988), which is responsible for the distribution of reputation by creating prestige hierarchies. Prestige hierarchies are based on outreach and perceived relevance of knowledge added by academic peers. These hierarchies are tied to scholars, publication outlets (e.g. by journal impact factors), departments (e.g. REF-profiles, ranking positions) and paradigms. Hierarchies immanent to the prestige economy also create a market for ideas with reputation as its currency. This market is additionally divided in segments revolving around national scientific cultures, accounting for differing perceptions of relevant topics, of what counts as qualitative or quantitative approach and of what theory to use (Erola et al., 2015; Williams et al., 2017). Analogous to other markets, the prestige economy is regarded as a symbolic market and tends to produce monopolies or oligopolies (Bourdieu, 1985: 18), which depend on the linkage between methods, topics, theories and epistemologies favoured by scholars. Also, belonging to a nationally anchored section of the prestige economy makes it more likely for scholars of the respective country to aim for publication and getting published in outlets located in their own country and instantiation of the prestige economy (Ylijoki et al., 2011). However, with regard to the methodological divide, the prestige economy itself does not predict a dominance of either quantitative or qualitative methods.

The prestige economy will work and produce oligopolies and durable hierarchies only if boundaries between different discourses, paradigms and references to other disciplines are drawn (Leahey and Moody, 2014; Leahey et al., 2010; Moody and Light, 2006). Such boundaries are drawn, stabilized and perpetuated by schools of thought, which provides a congruent approach to research topics from a point of view unique to them. These boundaries are then utilized as means of distinction and sources of worth and orientation for scholars belonging to the same school of thought (Bourdieu, 2004: 55–71). In line with Lamont and Molnár (2002) and Pachucki et al. (2007), such strategies can be coined boundary demarcation. Boundary demarcation aims at raising the worth of one’s own paradigm as against competing paradigms by monopolizing the access to sections of the scholarly discourse while preventing others from accessing these sections. One way to do so is to publish repeatedly in journals about a number of topics defined as relevant by the peers of the same school of thought, but also by the academic community as a whole. By doing so, demarcations are drawn that are of use as a signalling device to discourage scholars associated with other schools of thought to publish in outlets involved. Editors and reviewers of these journals have to choose among increasingly homogeneous submissions, thereby reinforcing certain paradigms without actively aiming to do so (Erola et al., 2015: 389). This linkage reinforces the effects of boundary demarcation and makes the journals beacons for the respective paradigm interacting with the academic prestige hierarchy. The reciprocity between boundary demarcation by scholars and publication outlets is expected to drive, deepen and consolidate the association between publication outlets and paradigms within the period under scrutiny. We therefore expect that a methodological divide is reflected in generalist Sociology journals (RQ1).

Within disciplinary boundaries, hierarchies provided by the prestige economy and constraints emerging from daily boundary demarcation, scholars develop a taste for research. Taste is defined as a set of dispositions in relation to fields (e.g. academia), enabling actors to take part in the field (Bourdieu, 1989: 19–20; Bourdieu and Wacquant, 1992: 203–204). Fields are usually present at national level, but are able to span national boundaries (Schmitz et al., 2017). Therefore, scholars located in different sectors of their respective fields aim to gain reputation at either national or transnational level. At the same time, they are strongly influenced by national specificities of the academic field, including competition among schools of thought and paradigms. In the academic field we may speak of different forms of taste for research that are visible as a distinct combination of methods, theories and research topics. In addition, taste for research also includes preferences for publishing the findings in certain forms of publications such as journal articles or monographs. Such forms of publication are expected to be read by different audiences that in turn are located in different segments of the prestige economy (e.g. readers of the American Journal of Sociology versus readers of Gender and Society). Since the methods used are linked with research paradigms and aim at audiences with a similar taste for research, we expect that the methodological divide is reflected in tastes for certain paradigms in different Sociology journals (RQ2a).

Since academia is not an autonomous field, we expect three mechanisms to interfere with the academic prestige economy leading to a tendency for quantitative methods to become more dominant over time. First, external pressures such as contingent funding, quantification of research productivity (e.g. impact factors) and demands formulated by stakeholders to tailor research findings influence the prestige economy and urge scholars to publish as frequently as possible in the most prestigious outlets (Feller, 2009: 334–340; Gläser and Laudel, 2016; Münch, 2014: 31–65; Wieczorek et al., 2017). As will be discussed in more detail below, scholars using quantitative methods are better able to adapt to these circumstances. Second, to cope with external pressures, scholars develop strategies to publish more efficiently, such as working in larger groups. This improves workload distribution and ultimately increases the number of co-authored publications (Lee and Bozeman, 2005). According to Moody (2004: 235), ‘co authorship is more likely in specialties that admit to an easier division of labour. Research method seems particularly important, showing that quantitative work is more likely to be coauthored than non-quantitative work’. As can be seen in supplementary material S6, this also applies to publications analysed in this article. In addition, quantitative methods allow scholars to systematically reuse gathered data, thus reducing the effort required to produce further articles. Even if there are attempts to address this issue by scholars using qualitative methodology, for example, via the Qualidata archive (i.e. Lampropoulou and Myers, 2013), databases containing quantitative data are more prevalent and include more datasets (e.g. the Harvard Dataverse). Third, the application of quantitative methods is mostly associated with survey data. Such data are collected and prepared for usage by smaller research groups, as well as governmental bodies and large research institutes. While the former sometimes field their own small-scale surveys, the latter two systematically provide scholars with financial resources, research infrastructures and access to official and administrative data. These factors make the division of labour both necessary and more likely, while the division of labour increases publication output over time (Abramo et al., 2009; Hunter and Leahey, 2008). Ultimately, we expect scholars focusing on ‘theory-building’ and qualitative methods to have a taste for publishing in national publication outlets in line with Erola et al. (2015). This is because topics focusing on the explanation of country-specific research questions mostly have audiences in the respective countries, thus addressing country-specific instances of the prestige economy. The contributions do not have to be translated to reach the specific audiences. In addition, every language has its own subtleties that are not easy to translate, adding to the author’s taste for publishing in outlets or monographs in their mother tongue. This way, their theoretical concepts do not face the danger of being depreciated in foreign languages. Moreover, focusing on a limited national scope of research enables the application of qualitative methods that are more suitable for in-depth analysis, further adding to the expected growing importance of quantitative methods in generalist Sociology journals. Hence, we expect that the academic prestige economy leads to an increasing trend towards publications applying quantitative methods (RQ2b).

Data

To scrutinize the methodological divide for sociological research, we compiled a dataset of abstracts published in generalist Sociology journals listed in the Social Science Citation Index (SSCI). We focused on SSCI journals with high impact factors for which publication data are available via the Scopus database. These journals cover research from a wide range of subfields and are relevant for a large body of the research community. In light of issues related to the peer-review system and concerns regarding the replicability of scientific work, Sociologists increasingly consider open access journals as alternative publication outlets. In addition to journals with high impact factors, we therefore also include articles published in the recently established open access journal Sociological Science.¹

Our strategy for measuring a methodological divide relies on abstracts as textual data. To this end, we further restricted our dataset to articles published between 1995 and 2017, as abstracts were often not available before 1995. We rely on abstracts for two reasons. First, abstracts are more often available than full texts. Second, processing entire articles and extracting the main text-bodies is challenging due to differences in citation style, footnotes, endnotes, table formats and style requirements. After removing non-article publications such as book reviews, our final dataset includes 8737 publications. Figure 1 shows all journals and the corresponding number of articles in our dataset.

Figure 1.

Number of articles published between 1995 and 2017 by journal.

It becomes apparent that journals such as Social Science Research and Social Forces published over 1000 articles between 1995 and 2017. In comparison, the Annual Review of Sociology, which only publishes one issue per year, and the open access journal Sociological Science, which started publishing in 2014, have a low cumulative output of article publications.

Although we believe that our data selection criteria result in an adequate representation of general and renowned journals for sociological research, we are aware that our selection is to some extent subjective. After all, the population of generalist Sociology research is not well defined and Sociology is an interdisciplinary and heterogeneous field framed by different tastes of research. However, we consider other approaches for case selection, which are arguably more objective, to be more problematic. For example, another strategy for identifying sociological work relies on categorizations of bibliometric services, for example, considering all publications from journals that are categorized as Sociology journals in the SSCI (e.g. Traag and Franssen, 2016). However, this approach results in the inclusion of articles that most scholars would not consider sociological research. Past research has shown that categorization schemes of publication data are problematic for the Social Sciences (Harzing, 2013). As an example, Annals for Tourism Research is listed as one of the top Sociology journals by impact factor in the SSCI 2016, but most scholars would not identify it as a Sociology journal. For this reason, we stick to our manual selection of publication outlets and corresponding articles.

Methods

To measure the methodological divide empirically, we rely on quantitative and automated methods of text analysis.² In preparation, we applied a range of common preprocessing techniques (for a comprehensive overview, see Grimmer and Stewart, 2013). At first, we removed content that is captured by the Scopus database, but is not part of the actual abstracts, such as keywords and copyright information. Next, we utilized the linguistic technique of lemmatization on all abstracts to group different words with a similar semantic meaning. For example, applying lemmatization to the words walk, walked, walking results in a single feature walk. We also removed very rare words (appearing in fewer than 50 abstracts) as well as stop words (e.g. the or which) with no semantic meaning. We then converted all abstracts to a bag of words format, where words are normalized to lowercase and the information about word order is discarded. Finally, we normalized differences for UK and US spellings, for example, for the terms labor and labour, and applied a collocation detection algorithm to identify phrases such as United States or statistically significant. Our preprocessing results in 2125 different features and a cumulative count of 625,066 terms within all 8737 abstracts.

Analysing the qualitative–quantitative divide in general, but also across journals and over time, requires a model for estimates on the word level as well as estimates on the document level. Such estimates could be produced with a large variety of complex methods, such as probabilistic topic models (Blei, 2012) or extensions such as structural topic models (Roberts et al., 2014). Unfortunately, these models require an a priori choice about the number of dimensions to extract. Instead, we make use of a more simplistic model imposing fewer parameter inputs, which has been widely applied for Political Science research: the wordfish model (Slapin and Proksch, 2008). Wordfish is a count-based model and assumes a Poisson distribution of word frequencies for positioning texts and corresponding authors on a unidimensional scale. It is most commonly used to position political actors on ideological scales, such as Economic-Left versus Economic-Right or Authoritarian versus Libertarian. In our case, we expect that the extreme points of our unidimensional scale capture the polar opposition between qualitative work versus quantitative work associated with different schools of thought. More specifically, the scale should represent a continuum between these methodologies, where different research paradigms are positioned along this scale. The functional form of the wordfish model is as follows:

\begin{array}{l} w o r d c o u n t_{i j} ~ P o i s s o n (λ_{i j}) \\ λ_{i j} = e x p (α_{i} + ψ_{j} + β_{j} * θ_{i}) \end{array}

Applied to Sociology publications, this formula models the number of times abstract i includes term j. More precisely, α is a document-level fixed-effect, controlling for some abstracts including more terms than others. ψ is the word-fixed-effect, controlling for some terms being used more frequently than others. β is the estimated weight for each term used to position the documents on the unidimensional scale. θ is the estimate for each document (abstract) on the unidimensional scale. Distributions for parameters of interest are included in our online supplementary material S1. We conducted several validity checks outlined below to make sure that the unidimensional scale captured by our model is in line with our theoretical assumptions. Finally, we examine not only whether we can observe a methodological divide in general (RQ1), but also its relation to different tastes for research between journals (RQ2a) and a change of methodological preferences over time (RQ2b). To model journal effects as well as time trends, we use ordinary least squares (OLS) regressions to model the variability of scaling estimates for each abstract (θ). As explanatory variables we include the corresponding journals as a categorical variable (RQ2a), and the year of publication (RQ2b) as either a linear predictor or a more flexible spline function.

Results

Before turning to the scaling model results for analysing the methodological divide, it is worth illustrating different journal preferences. Although the journals in our dataset are considered publication outlets for general interest Sociology, tastes for certain research topics or methodologies vary substantially. After preprocessing all abstracts, we computed key terms for each journal with χ² independence tests.³ Table 1 includes the top 10 key terms along with the number of times these terms were included in the corresponding journals. Unsurprisingly, key terms not only illustrate differences regarding methodology, for example, indicated by terms such as survey and depth interview, but also regarding regions and countries of interest. The term sweden was used 69 times in publications from Acta Sociologica, a journal with special interest in Nordic countries. In a robustness check, which is available in our supplementary material S5, we accounted for author-related country effects for our scaling model estimates. Besides country-specific terms, it becomes apparent that research topics of interest also vary. For example, terms related to collective action, such as movement, social movement and mobilization, are key terms for the journal Social Problems when compared to all other journals.

Table 1.

Top 10 key terms for each journal determined by χ² independence tests. Numbers in parentheses denote term frequencies within abstracts from the corresponding journals.

Journal	Key terms
Acta Sociologica	sweden (69), norwegian (44), swedish (51), norway (41), unemployment (59), unemployed (31), welfare state (52), voluntary (25), country (130), social capital (56)
American Journal of Sociology	network (207), law (106), organizational (134), diffusion (50), genetic (49), state (206), market (126), firm (86), organization (159), structural (98)
American Sociological Review	firm (169), exchange (140), penalty (60), tie (129), foreign (64), market (192), managerial (45), u.s. (152), wage (120), organization (256)
Annual Review of Sociology	sociological (89), direction (33), literature (78), theoretical (72), sociologist (51), future (53), field (66), sociology (96), attention (56), focus (102)
British Journal of Sociology	modernity (83), sociology (279), uk (76), british (104), bourdieu (72), britain (80), class (279), society (186), reflexive (34), sociological (162)
European Sociological Review	country (416), germany (116), socio economic (105), german (99), educational (244), labor market (226), education (301), unemployment (115), employment (231), european country (57)
Social Forces	black (306), disaster (52), marital (96), concentration (74), southern (51), church (58), religious (212), mexican (53), neighborhood (247), fertility (61)
Social Problems	social problem (68), movement (217), coalition (61), victim (49), social movement (115), activist (59), mobilization (62), political opportunity (31), frame (82), threat (74)
Social Science Research	estimate (264), college (275), immigrant (457), measure (434), survey (368), child (620), adolescent (220), student (361), non (294), us (122)
Sociological Forum	essay (51), frame (92), depth interview (38), police (65), community (169), boundary (81), color (28), mental (30), racism (32), volunteer (35)
Sociological Quarterly	frame (88), identity (149), narrative (56), body (58), discourse (69), newspaper (28), race (128), art (27), stereotype (27), movement (110)
Sociological Science	online (24), experiment (25), percent (24), score (18), fact (20), income (39), enforcement (10), ask whether (8), item (11), correlation (10)
Sociology	sociology (370), british (149), uk (108), explore (310), argue (428), young people (105), class (377), identity (287), draw (287), sociological (240)

To interpret results from our scaling model, we first visualize term fixed-effects (ψ) and weights (β) in Figure 2.

Figure 2.

Term weights and fixed effects from the wordfish scaling model fitted to Sociology abstracts.

Each point in the plot represents one term from our model. In order to make the visualization readable, we only display labels for a selection of terms.

As can be seen, the fixed effects capture how frequently terms are used within publications. Terms like work, theory and analysis are commonly used in many publications regardless of their research topic and methodological approach, which highlights the role of theoretical discussions and analysis for the field of sociology. Due to the high frequency of such terms, their contribution to the unidimensional scaling estimate is low. In comparison, the higher the absolute term weight, the stronger the signal for the position of an abstract (θ) on the unidimensional scale.

On the far left-hand side, terms like discourse, discursive and framing illustrate that this end of the scale represents predominantly qualitative publications. As the majority of sociological work with textual data applied qualitative research methods, the term text appears on this end as well. In contrast, terms on the right-hand side, for instance statistically significant, matching, multilevel and unobserved, represent predominantly quantitative research. In line with our arguments on taste for research and boundary demarcation, our results suggest that the methodological divide is strongly entangled with certain research topics and cultures. For instance, the terms social movement and protest on the left-hand side suggest that corresponding articles deal with collective action. In accordance with the notion of boundary demarcation and tastes for research, the relation between methods and theories is highlighted by an occurrence of terms such as weber, bourdieu, neoliberal and social theory on the qualitative spectrum. On the more quantitative side, many terms are related to family, life course and inequality studies, for example, parent, cohabitation and income dynamics. The relation between these subjects and quantitative methodology seems reasonable, as for instance life course studies often require expensive panel survey and advanced quantitative analysis.

Another interesting finding is that the terms qualitative and quantitative have very low term weights and therefore are not good indicators of either qualitative or quantitative methodology. Altogether, we found that ‘quantitative’ is included in only 200 abstracts and ‘qualitative’ in 256. These terms are to a high extent general and include a multiplicity of methods. However, subsuming multiple methods renders the two terms relatively inefficient for drawing symbolic boundaries. In this sense, they are not suitable for boundary demarcation. This argument is corroborated, since 86 abstracts among the 200 articles using the term ‘quantitative’ also mention the term ‘qualitative’. Examining related abstracts reveals that these terms are rather used to describe mixed method designs, indicating their low applicability to conduct boundary demarcation. Other methodological indicators like multilevel or discourse are more specific, and thus enable us to draw sharper distinctions between and within paradigms. Therefore, these terms signal different tastes for research associated with different quantitative and qualitative methods. For instance, the term multilevel is associated with hierarchical data structures and matching is associated with methods like propensity-score-matching for causal analysis.

Another validity check for our scale covers the inspection of abstracts from two publications that are representative of either end of the unidimensional scale as indicated by their very low/high θ (document level) values. The first abstract is highly representative of the qualitative end of the scale (Croteau and Hicks, 2003):

This article examines framing processes in a coalition context. We argue that, if they are to avoid debilitating frame disputes, coalitions seeking social change face the unique task of developing a ‘consonant frame pyramid,’ which aligns coalition, organizational, and individual frames. We show the usefulness of these concepts for understanding framing through analysis of a local coalition formed to better meet the needs of the homeless. Using information gathered via interviews, participant observation, and documents, we describe several of the organizational frames that were contending for adoption as the larger coalition’s frame, and detail two of the frame disputes that developed. We explore the source of these frame disputes and the factors that influenced the emerging coalition frame. We contend that organizations vary in their ability to promote a coalition frame due to varying degrees of power, different organizational structures, and varying degrees of ‘fit’ between organizational frames and the existing political opportunities and constraints. These findings reveal coalition frames as the emergent products of ongoing intra- and inter-organizational dynamics, and help specify framing’s links to mobilizing structures and political opportunity.

In comparison, the second abstract is highly representative of the quantitative end of the scale (Brown et al., 2006):

We extend prior research on the association between premarital cohabitation and marital outcomes by investigating whether covenant marriage, which entails more stringent requirements for divorce, minimizes the deleterious effects of cohabitation on subsequent marital quality and stability. Using a unique longitudinal data set of covenant and standard newlywed couples in Louisiana, we find that covenant marriage does not modify the effects of premarital cohabitation on marital instability, happiness, dependency, or divorce for either wives or husbands. In fact, once we control for sociodemographic characteristics, premarital relationship factors, and marital factors, the relationships between premarital cohabitation and marital outcomes reduces to non-significance, suggesting that selection factors largely account for the deleterious effects of premarital cohabitation on marital success.

Additional representative abstracts are available in our online supplementary material S2. Combined with our word-level estimates from Figure 2, these abstract inspections suggest a methodological divide in sociological research, which is strongly entangled with research paradigms and associated symbolic boundaries (RQ1).

One potential concern about our approach is that increasing the dimensionality of our model might produce results that would not clearly indicate a methodological divide anymore. For instance, one might argue that an additional dimension could capture the poles theoretical and empirical. In an additional robustness check, we therefore used correspondence analysis for dimensionality reduction to project all articles in a two-dimensional space. The results from this model, which are available in the online supplementary material S4, are in line with our findings: the first dimension again uncovers a methodological divide with more qualitative terms on the one pole of the corresponding axis and more quantitative terms on the other pole of the axis.

Having shown that our text-based approach is robust to increasing the dimensionality of the model, we use OLS regressions to further analyse whether the methodological divide is related to publication outlets (RQ2a) and time trends (RQ2b). We fitted three models, one with a linear time trend and two with differently specified splines. Model fit indicators, available in our regression table in the online supplementary material S3, suggest no improvements for models with non-linear variables. Hence, the predicted values shown in Figure 3 are produced with the linear model. Furthermore, all predictions are estimated while holding the remaining variables from the linear model at their empirically observed values. It is important to note that, similar to the word-level estimates above, scaling values are not directly interpretable. For instance, a predicted scaling value of 0.8 does not capture any substantial meaning, such as 80 publications authored by a certain sociologist. Nevertheless, the magnitude of different scaling estimates can be compared. Subfigure 3(a) illustrates that Social Science Research and European Sociological Review predominantly publish articles positioned at the quantitative end of our scale. In comparison, the British journals Sociology and British Journal of Sociology focus on qualitative work. In line with the assumptions derived from the interaction between the prestige economy and external influences, predictions shown in subfigure 3(b) suggest an increasing tendency towards quantitative methods. However, the magnitude of this trend is much lower in comparison to the journal effect. This indicates that the entanglement between paradigms and journal outlets inherent in the notion of taste for research mitigates the effects of the prestige economy to a certain extent. Ultimately, the relatively low magnitude of the time trend signifies the depth of the entrenchments between paradigms relying on qualitative and quantitative methods.

Figure 3.

Predicted values for abstract scaling positions by journal (a) and publication year (b) with 95% confidence intervals.

As argued above, embedded instances of prestige hierarchy and tastes for research may both be related to nationality, which ultimately could also affect the preference of authors for qualitative or quantitative methodology. As an additional robustness check, we therefore examined whether the methodological divide is related to the nationality of authors. For all publications in our dataset for which affiliations are available, we used the Google Geocoding Application Programming Interface to retrieve country annotations of author affiliations. More details on this procedure and the corresponding analysis are available in supplementary material S5. Overall, 76 per cent of all author affiliations are located either in the United Kingdom (UK) or the United States (US). Using these data, we created indicators for the relative share of UK and US affiliations for each article. We then included them in additional OLS models together with journal indicators and the publication year. Findings suggest that the nationality effect of authors is weak at best, where publications from UK-based authors are slightly more often positioned at the qualitative end of the methodological dimension. Even when controlling for author affiliations, results for journal effects and time trends do not change.

Conclusions

We examined the presence of a methodological divide of the low-consensus discipline Sociology using data from two decades of generalist Sociology journals. We scrutinized the theoretical concepts of academic prestige economy, boundary demarcation and taste for research to account for the linkage of paradigms and publication outlets. Our findings not only identify a methodological divide, but also a marginally increasing preference for quantitative methodology in generalist publication outlets. At the same time, the methodological divide is accompanied by a differentiation in research topics. These topics are also associated with different methods used, indicating that topics are addressed to a different extent by paradigms and schools of thought. In this sense, boundary demarcation enables scholars to draw distinctions between competing paradigms by utilizing publication outlets aligned to the different paradigms. The notion of prestige economy further suggests that, albeit all included publication outlets are prestigious in their own right, there are outlets that are addressed by scholars using a combination of certain topics with either qualitative (e.g. social movements and discourse) or quantitative methods (e.g. family structure and longitudinal survey). These oppositions also follow the differences in the alignment to the philosophy of science, for example, qualitative, constructivism and inductive versus quantitative, positivistic and deductive, making different outlets beacons for scholars of different schools of thought. All these factors combined account for both the clear-cut differences among journals and the relatively low magnitude of the time trend, while possibly explaining the marginalization of mixed methods approaches.

The entanglement between methodology and topics of interest is, to some extent, unsurprising, mirroring the entrenchment between different paradigms. For instance, some important research questions regarding inequalities over the life course require quantifiable data, for example, panel surveys, collected over longer periods of time. Analysing such data requires predominantly quantitative methods such as regression techniques. Likewise, many crucial questions regarding inequalities over the life course cannot be answered with survey data and would instead require more qualitative approaches.

If the entanglement between methodology and topics is fuelled by boundary demarcation and tastes for research interacting with prestige economy, the generation of innovative and impactful (sociological) research might suffer. To give an example, our research (see Figure 2) indicates that, at least for sociological work, text as data is strongly related to qualitative research methodology.⁴ Our task of measuring a methodological divide across more than 8700 abstracts would be very labour intensive with a qualitative approach. Moreover, although a qualitative approach could certainly shed light on other interesting aspects of the methodological divide, it would be more difficult to quantify time trends and differences between journals. Beyond this example, the entanglement between paradigms and publication outlets covered by different tastes for research bears the risk of narrowing down the way in which topics could legitimately be addressed.

With regard to our empirical approach, the identification of a methodological divide is robust to applying correspondence analysis with two dimensions instead of only one from the wordfish model. Furthermore, results do not change when controlling for country effects and the nationality of authors is used as explanatory factors for methodological preferences. Moreover, our findings are in line with word-level visualizations from a blog post by scholars who also examined Sociology publications (Traag and Franssen, 2016).

Nevertheless, our work is limited in as far as we only examined publications from generalist Sociology journals, since some scholars prefer to publish in monographs and edited volumes, often written in their mother tongue. Our data cannot infer whether, for instance, the increasing trend towards quantitative research in generalist journals also applies to different types of publications. Also, data limitations do not allow us to analyse time trends for methodological preferences separately for each journal. Future research could examine whether methodological trends are consistent across different journals. Furthermore, abstract data cannot fully grasp the usage of mixed methods approaches. We noticed that articles relying on the application of two or more methods often do not mention the concepts of mixed methods or every method used in the abstract, which makes it difficult to identify such work in an automated way.⁵ Similar problems arise for ‘big data’ research, which we are not addressing specifically in this article as it is hard to define in the first place and also makes use of mostly quantitative, but also qualitative or mixed methodology.

Regarding the role of theory, most sociological publications involve thorough theoretical passages. As can be seen in results from our scaling model (Figure 2) the term ‘theory’ is one of the most common terms. Although positioned slightly more towards the qualitative spectrum, it is used frequently in both qualitative and quantitative work. In comparison to other fields like economics, sociologists also rely more often on words to formulate theories rather than formulas. Nevertheless, many quantitative articles include at least some kind of mathematical formulization. Unfortunately, our methodological approach is insufficient to measure mathematically formalized theory development as formulas are in general not included in abstracts. We do not have access to many full texts in order to extract and analyse formulas. Scholars with better data at hand might be able to address this shortcoming in order to further examine the role of theory and formulization for sociological publishing.

Despite these limitations, our findings show that the entrenchment between different schools of thought, accompanied by boundary demarcation and different tastes for research, is fundamentally connected with the discipline of Sociology itself. Sociology, just like other disciplines, is kept alive by a rigorous exchange of ideas rooted in different paradigms and by combining theories and methods to solve societal problems. Future research should focus on how the mechanisms manifest themselves within different paradigms and how they prohibit the exchange of ideas. Otherwise, Sociology might increasingly give way to an archipelago of sub-disciplines with their own domains of knowledge but a limited ability to produce new insights.

Supplemental Material

SOC853146_supplementary_material – Supplemental material for The Methodological Divide of Sociology: Evidence from Two Decades of Journal Publications

Supplemental material, SOC853146_supplementary_material for The Methodological Divide of Sociology: Evidence from Two Decades of Journal Publications by Carsten Schwemmer and Oliver Wieczorek in Sociology

Footnotes

Acknowledgements

We thank Richard Münch, Brigitte Münzel, Michael Eberhardt, Isabella Czedik-Eysenberg and the three anonymous reviewers for important criticisms and suggestions that significantly improved our article.

Funding

The research is funded by the German Federal Ministry of Education and Research [funding code: 01PU17021A].

Replication Material

Data and code for replication are available at Harvard Dataverse: .

Supplemental Material

Supplemental material for this article is available online.

Notes

ORCID iD

Carsten Schwemmer

Carsten Schwemmer is a research fellow at the Weizenbaum Internet Institute, Berlin. He is also a PhD candidate in the field of Computational Social Science and a lecturer for the chair of Political Sociology at University of Bamberg, Germany. His research focuses on applying computational methods for ethnic minority and social media studies. He is particularly interested in natural language processing, data mining and the development of research software.

Oliver Wieczorek is a research associate and doctoral student at the Chair of Sociology, especially Sociological Theory at University of Bamberg. He is a member of the research project ‘ABD – Factors of career success in the natural sciences and the social sciences’, funded by the German Federal Ministry of Education and Research. His research interests include sociological theory, relational sociology, sociology of science, social network analysis, comparative macrosociology, quantitative and qualitative methods of empirical research, sociology of elites, political sociology and life course analysis.

References

Abramo

D’Angelo

Di Costa

(2009) Research collaboration and productivity: Is there correlation? Higher Education 57(2): 155–171.

Adorno

(ed.) (1987) Der Positivismusstreit in der deutschen Soziologie. 12. Aufl. Sammlung Luchterhand 72. Darmstadt: Luchterhand.

(2018) Sociology and science: The making of a social scientific method. The American Sociologist 49(1): 98–115.

Benoit

Nulty

(2016) quanteda: Quantitative analysis of textual data. R package version 0.9 8.

Berger

Luckmann

(1991) The Social Construction of Reality: A Treatise in the Sociology of Knowledge. Harmondsworth: Penguin UK.

Blei

(2012) Probabilistic topic models. Communications of the ACM 55(4): 77–84.

Bourdieu

(1985) The market of symbolic goods. Poetics 14(1–2): 13–44.

Bourdieu

(1989) Social space and symbolic power. Sociological Theory 7(1): 14–25.

Bourdieu

(2004) Science of Science and Reflexivity. Cambridge: Polity.

10.

Bourdieu

Wacquant

LJD

(1992) An Invitation to Reflexive Sociology. Chicago, IL: University of Chicago Press.

11.

Boyns

Fletcher

(2005) Reflections on public Sociology: Public relations, disciplinary identity, and the strong program in professional Sociology. The American Sociologist 36(3–4): 5–26.

12.

Brown

Sanchez

Nock

, et al. (2006) Links between premarital cohabitation and subsequent marital quality, stability, and divorce: A comparison of covenant versus standard marriages. Social Science Research 35(2): 454–470.

13.

Bryman

(2008) The end of the paradigm wars. The SAGE Handbook of Social Research Methods. London: SAGE, 13–25.

14.

Burawoy

(2005) For public Sociology. American Sociological Review 70(1): 4–28.

15.

Byrne

(2012) UK Sociology and quantitative methods: Are we as weak as they think? Or are they barking up the wrong tree? Sociology 46(1): 13–24.

16.

Collins

(1989) Sociology: Proscience or antiscience? American Sociological Review 54(1): 124–139.

17.

Collins

(1994) Why the social sciences won’t become high-consensus, rapid-discovery science. Sociological Forum 9(2): 155–177.

18.

Croteau

Hicks

(2003) Coalition framing and the challenge of a consonant frame pyramid: The case of a collaborative response to homelessness. Social Problems 50(2): 251–272.

19.

Erola

Reimer

Räsänen

, et al. (2015) No crisis but methodological separatism: A comparative study of Finnish and Danish publication trends between 1990 and 2009. Sociology 49(2): 374–394.

20.

Feller

(2009) Performance measurement and the governance of American academic science. Minerva 47(3): 323–344.

21.

Gage

(1989) The paradigm wars and their aftermath: A ‘historical’ sketch of research on teaching since 1989. Educational Researcher 18(7): 4–10.

22.

Glaser

Strauss

(2017) Discovery of Grounded Theory: Strategies for Qualitative Research. Abingdon: Routledge.

23.

Gläser

Laudel

(2016) Governing science: How science policy shapes research content. European Journal of Sociology/Archives Européennes de sociologie 57(1): 117–168.

24.

Grimmer

Stewart

(2013) Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis 21(3): 267–297.

25.

Harzing

A-W

(2013) Document categories in the ISI Web of Knowledge: Misunderstanding the social sciences? Scientometrics 94(1): 23–34.

26.

Hunter

Leahey

(2008) Collaborative research in Sociology: Trends and contributing factors. The American Sociologist 39(4): 290–306.

27.

Kuhn

(1996) The Structure of Scientific Revolutions. Chicago, IL: University of Chicago Press.

28.

Lamont

Molnár

(2002) The study of boundaries in the social sciences. Annual Review of Sociology 28(1): 167–195.

29.

Lampropoulou

Myers

(2013) Stance-taking in interviews from the Qualidata Archive. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research 14(1). Available at: https://dx-doi-org-s.web.bisu.edu.cn/10.17169/fqs-14.1.1813 (accessed 13 May 2019).

30.

Lê

Josse

Husson

(2008) FactoMineR: An R package for multivariate analysis. Journal of Statistical Software 25(1): 1–18.

31.

Leahey

Moody

(2014) Sociological innovation through subfield integration. Social Currents 1(3): 228–256.

32.

Leahey

Keith

Crockett

(2010) Specialization and promotion in an academic discipline. Research in Social Stratification and Mobility 28(2): 135–155.

33.

Lee

Bozeman

(2005) The impact of research collaboration on scientific productivity. Social Studies of Science 35(5): 673–702.

34.

Legewie

Schaeffer

(2016) Contested boundaries: Explaining where ethnoracial diversity provokes neighborhood conflict. American Journal of Sociology 122(1): 125–161.

35.

Leifeld

(2013) texreg: Conversion of statistical model output in R to LATEX and HTML tables. Journal of Statistical Software 55(8): 1–24.

36.

Lüdecke

(2018) ggeffects: Tidy data frames of marginal effects from regression models. Journal of Open Source Software 3(26): 772. Available at: https://doi.org/10.21105/joss.00772 (accessed 13 May 2019).

37.

Merton

(1988) The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis 79(4): 606–623.

38.

Merton

(1968) The Matthew effect in science. Science 159(3810): 56–63.

39.

Moksony

Hegedűs

Császár

(2014) Rankings, research styles, and publication cultures: A study of American Sociology departments. Scientometrics 101(3): 1715–1729.

40.

Moody

(2004) The structure of a social science collaboration network: Disciplinary cohesion from 1963 to 1999. American Sociological Review 69(2): 213–238.

41.

Moody

Light

(2006) A view from above: The evolving sociological landscape. The American Sociologist 37(2): 67–86.

42.

Münch

(2014) Academic Capitalism: Universities in the Global Struggle for Excellence. Routledge advances in Sociology 121. New York: Routledge.

43.

Münch

(2018) Soziologie in der Identitätskrise: Zwischen totaler Fragmentierung und Einparadigmenherrschaft. Zeitschrift für Soziologie 47(1): 1–6.

44.

Oromaner

(2008) Intellectual integration and articles in core Sociology journals, 1960–2000. The American Sociologist 39(4): 279–289.

45.

Pachucki

Pendergrass

Lamont

(2007) Boundary processes: Recent theoretical developments and new contributions. Poetics 35(6): 331–351.

46.

Payne

Williams

Chamberlain

(2004) Methodological pluralism in British Sociology. Sociology 38(1): 153–163.

47.

Popper

(2008) The Logic of Scientific Discovery. Repr. 2008 (twice). Routledge classics. London: Routledge.

48.

Puddephatt

McLaughlin

(2015) Critical nexus or pluralist discipline? Institutional ambivalence and the future of Canadian Sociology. Canadian Review of Sociology/Revue canadienne de sociologie 52(3): 310–332.

49.

Roberts

Stewart

Tingley

, et al. (2014) Structural topic models for open-ended survey responses. American Journal of Political Science 58(4): 1064–1082.

50.

Schmitz

Witte

Gengnagel

(2017) Pluralizing field analysis: Toward a relational understanding of the field of power. Social Science Information 56(1): 49–73.

51.

Slapin

Proksch

(2008) A scaling model for estimating time-series party positions from texts. American Journal of Political Science 52(3): 705–722.

52.

Smelser

(2015) Sources of unity and disunity in Sociology. The American Sociologist 46(3): 303–312.

53.

Traag

Franssen

(2016) Revealing the quantitative–qualitative divide in Sociology using bibliometric visualization. CWTS blog.

54.

Turner

(1998) Must sociological theory and sociological practice be so far apart? A polemical answer. Sociological Perspectives 41(2): 243–258.

55.

Turner

(2006) Explaining the social world: Historicism versus positivism. The Sociological Quarterly 47(3): 451–463.

56.

Turner

(2016) Academic journals and Sociology’s big divide: A modest but radical proposal. The American Sociologist 47(2): 289–301.

57.

Vanderstraeten

(2010) Scientific communication: Sociology journals and publication practices. Sociology 44(3): 559–576.

58.

Varga

(2011) Measuring the semantic integrity of scientific fields: A method and a study of Sociology, economics and biophysics. Scientometrics 88(1): 163–177.

59.

Wickham

(2016) tidyverse: Easily install and load ‘tidyverse’packages [Software].

60.

Wieczorek

Beyer

Münch

(2017) Fief and benefice feudalism: Two types of academic autonomy in US chemistry. Higher Education 73(6): 887–907.

61.

Williams

Sloan

Brookfield

(2017) A tale of two sociologies: Analyzing versus critique in UK Sociology. Sociological Research Online 22(4): 132–151.

62.

Ylijoki

O-H

Lyytinen

Marttila

(2011) Different research markets: A disciplinary perspective. Higher Education 62(6): 721–740.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.60 MB