Prologue

Symposium on Big Data—Innovations in Computational Social Science

With the development of the Internet and the worldwide web, our lives have become inundated with vast amounts of information—we live in a “big information” or “big data” era. Take sports, where all sorts of new information is available on player performance (see e.g., Epstein 2013). Michael Lewis’s (2003) popular book Money Ball (as well as the movie), for example, highlights the work of SABR-metrician Bill James (SABR = Society for American Baseball Research) in the development of new “metrics” to evaluate player success in major league baseball (see Albert 2010). This occurred in other sports as well.

More and more information is available to be analyzed, which places demands on innovative analytic techniques. In science, developments can be seen in the availability of huge databases. In genomics, space exploration, marine biology, and nearly every science, researchers are confronted with big data (Pearl 2018). Even closer to home, in the social sciences, we have become inundated with data. U.S. federal census data are increasingly available in one form or another, including images of the original census enumerations (see e.g., Ancestry.com). Maintaining the traditional standards for collection of population-based big data, the Census Bureau and Bureau of Labor Statistics collaborate to produce a number of public data sets on the U.S. population, including the quarterly CPS (Current Population Surveys). Another example is the American Community Survey (ACS), an ongoing survey of the Census Bureau, which includes more than 3.5 million respondents per year (U.S. Census Bureau 2015). Several cohort studies of labor force participants are increasingly available (see Bureau of Labor Statistics 2019). And with the continuation of many existing sources of social monitoring data (e.g., the U.S. General Social Survey, the German ALLBUS, the British Social Attitudes Study, and the World Values Survey, to name just a few) as well as new large-scale multinational data projects, such as the European Social Survey or the International Social Survey Program (ISSP), along with the digitization of many archival records, new demands are being placed on how we approach data—social science data are bigger than ever before. The bottom line here is that “big data social science” has arrived, and it will put major demands not only on how we think about testing hypotheses about society but also on how we develop curricula for training students, how we use computers, and how we communicate with one another. Ultimately, the hope is that big data will help us answer big questions.

One tantalizing example of a massive new big data resource is Google Ngrams (see Michel et al. 2011), an ambitious collaboration between Google and many of the world’s largest libraries and book publishers, which intends to digitize every existing book ever printed. The corpus currently contains 361 billion words dating back five centuries to the 1500s. The more than 15 million scanned and digitized books, in several languages, represent approximately 12 percent of books ever published. The process of developing the Ngram database is straightforward. Each book is scanned, digitized via optical character recognition, and the extracted words are added to the Ngram database along with key metadata. The Ngram database contains all words appearing at least 40 times in the entire corpus; it includes the year in which the word appeared in a published text, overall frequency of occurrence, number of pages on which the word appeared, and number of unique books containing the word. To date, Google has supplied searchable American English databases (one-gram, two-gram, three-gram, four-gram, and five-gram databases). The one-gram database contains single words (e.g., fish), the two-gram database contains combinations of up to two words that were adjacent to each other in the scanned books (e.g., catching fish), the three-gram database contains combinations of three adjacent words (e.g., likes catching fish), and so on. The potential of Google Ngrams for the study of society is great, and some scholars have used the term culturomics to refer to the use of these new digitized resources in the study of trends, regional differences, and the human cultural genome (Michel et al. 2011). ⁹ In the area of methodology, one could, for example, examine historical trends in the use of methodological or scientific terms like survey, sample, experiment, statistic, empiricism, validity, causation, or placebo (see Alwin 2013).

New approaches that merge computer science, visualization tools, and traditional statistics are being developed to accommodate these growing resources, and in keeping with these observations, this year’s symposium highlights the exploration of nontraditional data resources, including text and image data from social media, and the computational analysis of textual meaning using natural language processing. The term big data has come to be associated with many of the new large data sets; the authors of the two articles included here, along with the commentaries and replies, focus on examples of how big data resources can be used to promote new understandings in traditional research areas. In addition to considering big data issues, these two papers also embody a modern approach to data analysis involving “computational social science” (e.g., Evans and Aceves 2016; Nardulli, Althaus, and Hayes 2015; Nelson 2017). The existence of large amounts of text have created a “demand for natural language processing and machine learning tools to filter, search and translate text into valuable data” (Evans and Aceves 2016:21).

The symposium focuses on potentially new resources for sociology, including automated analysis of image and text data from social media and data mining of newspaper text using natural language processing. As these articles reveal, machine learning and data mining approaches are at the cutting edge of methodological developments in sociology and social science more generally, and they are squarely in the intersection of what is traditionally thought of as a divide between quantitative and qualitative data. “CASM: A Deep-Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media,” by Han Zhang and Jennifer Pan (hereafter ZP), goes beyond just text. They analyze collective action events in China using both image and text data. The study of social movements in general and the study of protest events in particular traditionally rely on news media reports as a source of data (see e.g., Martin, Rafail, and McCarthy 2017). The authors introduce a new approach—CASM (collective action from social media)—tailored to the study of collective action and protest events that happen in authoritarian regimes and that are not reported in traditional media. CASM uses text and images from social media data and deep learning algorithms to identify offline collective action events in real time. This approach harnesses digital technologies and deep learning algorithms to reveal posts that likely discuss offline collective action events. Deep learning (or deep structured learning or hierarchical learning) is a branch of supervised machine learning that uses a framework of artificial neural networks to create algorithms that are “trained” to identify collection action events.

Deep learning algorithms are just beginning to be used in social science research, in particular in research using image data. ZP's innovative use of these methods expands on this emerging strand of social science research by using deep learning for image and textual classification. This article should also be read against the backdrop of the fact that the Chinese government places restrictions on traditional and new media to limit subversion of its authority. The article implements CASM for China using social media data from Sina Weibo, the largest social media platform in China, with over 445 million monthly active users. ZP describe the advantages and disadvantages of using social media as a data source for identifying collective action events, and they explore the external validity of this target data source with other event data sets. The article also evaluates the impact of censorship on the quality of the CASM-China data, and it considers how computer science methods can be made more practical and usable in the social sciences. In summary, deep learning algorithms have helped make significant advances in many machine learning tasks, especially tasks related to analysis of images and text, such as image classification, multiple object detection automated image captioning, voice recognition, and parts-of-speech tagging.

The commentaries on the ZP article, by Swen Hutter, Pamela Oliver, and Zachary C. Steinert-Threlkeld, offer a great deal of praise for these new approaches to protest-event analysis but not without some cautionary admonitions. The commentaries agree that ZP have given the sociological community three “gifts” (in Hutter’s words): (1) a two-stage classifier of text and images, (2) a large data set on offline protests reported online during a specific historical period in China, and (3) an accessible text that explains their methods. On the critical side, Hutter focuses on the conceptual issues of the coding unit, the identification of duplicates, and relations between online and offline dynamics. Oliver similarly lauds ZP’s contributions. She observes the potential for human error and discusses the various advantages and disadvantages to strategies of compiling protest event data. Steinert-Threlkeld suggests ZP’s work represents “the new frontier” in the study of collective action. He highlights their contributions in including image data in addition to text and holds that the debate about the quality (bias) of various types of event protest data (multimodal event data) reflects a promising new approach to the problems of bias. The collective conclusion is that these new methods will assist in the development of automated systems for coding protest events that permit greater accuracy and analytic power. The commentaries agree that ZP’s work is already helping shape future innovations in protest event research, and in the words of Steinert-Threlkeld, “the future of event data is bright, and one of the brightest areas is using social media images” to develop new event protest databases.

The second article in the symposium, “Analyzing Meaning in Big Data: Performing a Map Analysis Using Grammatical Parsing and Topic Modeling” by Jan Goldenstein and Philipp Poschmann (hereafter GP), offers another example of the application of analysis of big data using computational methods that perform text analysis through natural language processing. They perform a “map analysis” of textual meaning in which they use a “layered” approach combining “grammatical parsing” to discover communication structures and “topic modeling” to assess semantic context. We consider the combination of these two approaches to text analysis to be one of the unique contributions of their article. They creatively apply this layered approach to examine how the meaning of the concept of “corporate responsibility” has changed in the United States.

These methodological tools go by a number of terms. Text mining is the most commonly applied term, but this does not necessarily convey the variety of approaches. The underlying technique that enables contemporary approaches to content analysis involves natural language processing (or NLP), which is a field of specialization that focuses on interactions between human language and computational linguistics and artificial intelligence. The authors combine grammatical parsing using NLP tools and topic modeling in what they refer to as a “map analysis” of different layers of meaning. In machine learning and natural language processing, grammatical parsing refers to methods of breaking down a text into its component parts of speech. Topic modeling, on the other hand, is a text-mining tool for the discovery of latent semantic structures in a body of text. Topic modeling is a popular approach to the measurement of cultural meaning (see Mohr, Wagner-Pacifici, and Breiger 2015).

GP’s article provides an overview of the current state of formal text analysis in sociology, pointing to existing sources on the topic. In so doing, the article raises the question of whether text analysis should be free of subjective (ex ante) interpretations. GP also provide an introduction to the concept of “maps” in text analysis (to wit: a map involves analysis of textual characteristics without any subjective assumptions, and it presents a focused version of the entire text corpus from the perspective of the chosen textual characteristics). They present the simultaneous consideration of two layers of textual meaning using this map analysis approach. They first obtain a grammatical parsing that reflects communication structures, and then they follow this with a layer of topic modeling on top of the structural elements.

This article introduces an approach that suggests a combination of text-mining tools that address different layers of meaning. In their empirical application, GP analyze 15,371 newspaper articles published in the New York Times and Washington Post from 1950 to 2013 on the topic of corporate responsibility. They used state-of-the-art Stanford CoreNLP software (Manning et al. 2014) to enable their supervised learning approach to perform grammatical parsing. They then coupled this with their own topic modeling software that implements content analysis of the semantic patterns. This article provides a fully developed NLP software application implementing grammatical parsing and topic modeling and enables the large-scale analysis of communication structures (i.e., semantic triplets) and hidden semantic patterns in texts.

Commentaries by expert text analysts, Burt L. Monroe (a political scientist) and Laura K. Nelson (a sociologist), provide additional perspectives on the GP article. Both Nelson and Monroe raise concerns about the transparency of Goldenstein and Poschmann’s research with respect to their use of current text-mining tools, but they clearly see some value in the approach. Nelson focuses on transparency in the context of research and the analytical steps in a text-analysis project to ensure the reproducibility of results. While maintaining that GP’s research “exemplifies how to expand our text analysis tool-kit by applying computation tools to address an important sociological question,” she takes serious issue with the assumptions on which the approach rests, namely, that computational methods can supplant subjective approaches. Monroe focuses on transparency regarding data inspection and the credibility of results. He suggests the authors may have “fallen prey” to some pernicious measurement traps in the text analytic techniques they apply, and he raises concern that the authors may not have “captured the meaning they assert.” Although the invited commentaries are critical of the approach taken by GP, we conclude that the GP article usefully illustrates how recently introduced methodological frameworks for the study of cultural content can be used in sociology. They show that various tools can be used in combination with multiple layers of text analysis approaches. Ultimately, GP argue for a combination of text-mining approaches with scaling inductively grounded in a “close reading” of content. Any given study has certain limitations, and the area of content analysis (i.e., text mining using computational approaches) has a great deal more disagreement than agreement on the optimal approach. Whatever its limitations, this article illustrates the variety of applications of text mining using various forms of natural language processing, and as a representative of this new genre of research, it portends promising applications of text mining in sociology in the future.

Framework for Model Comparisons

The article in this section by Trenton D. Mize, Long Doan, and J. Scott Long, “A General Framework for Comparing Predictions and Marginal Effects across Models,” tackles the problem of comparing predictions or effects across models, an issue that is receiving substantial attention in the discipline writ large (e.g., Mood 2010; Mustillo, Lizardo, and McVeigh 2018). Statistically oriented social scientists have been pursuing the question of how best to compare across models for at least a half century (e.g., Chow 1960), particularly within nonlinear models and other complex scenarios. Sociological methodologists have long led advances on these sorts of topics (Sobel 1986). In this new advance to the literature, the authors argue for a general and flexible approach to model comparisons that permits comparison of marginal effects across studies or across groups within a study. The framework they propose subsumes a wide range of cross-model comparisons. They begin with a clear statement of the problem, review the importance of assessing marginal effects, and show that these issues often plague researchers trying to draw comparisons across models, studies, and data sets. They propose the use of seemingly unrelated estimation (SUEST) to combine estimates from multiple models, testing the equality of predictions and effects across models. A focus on changes in the predictions—the very thing current sociological research is increasingly attuned to—dovetails with the primary concerns of recent critics of common practice. The article applies this framework to an array of problems for binary, continuous, count, ordinal, and nominal dependent variables.

The seemingly unrelated regression (SUR) model, or seemingly unrelated regression equations (SURE) model, is an approach borrowed from econometrics (Zellner 1962) wherein multiple equations (multiple dependent variables) with potentially different sets of exogenous explanatory variables are analyzed (see Felmlee and Hargens [1988] for an introduction to the approach in sociology). The term seemingly unrelated refers to the fact that each equation can be considered on its own and estimated separately (Hargens 1988). In fact, one might say they are “seemingly related” because the disturbances in these equations are in general correlated. The SUR model is a generalization of a regression model in which there are multiple equations for the outcomes of interest. Mize and colleagues utilize this model to develop tests of cross-model differences, and to demonstrate the utility of the approach, they present examples and applications that illustrate how their framework can be used in a variety of concrete situations. Importantly, their proposed approach offers direct tests of between-model differences in the coefficients of interest, built from a principled statistical framework that can apply across a diverse array of situations of interest in contemporary social research. Their approach promises to provide a framework that can improve our methods of comparing effects across models, and it seems to provide an answer to many hotly debated methodological issues in the discipline. We expect readers will share in the excitement about its applicability to many research questions.

Multilevel Models

The data collection explosion of the past half century has made it easier for researchers to examine units that are nested, or otherwise embedded, within some other type of unit. Examples abound: multiple respondents surveyed in different years, respondents surveyed in different years in different countries, students observed in different grades in a school or in different schools, and so on. Such data permit the analysis of interesting sociological questions about how social contexts influence individuals. Accordingly, given the volume of data available to contemporary sociologists and the theoretical importance of questions about the ways context affects individual lives, a robust methodological literature has evolved for working with such data. This literature is voluminous, but there remain important areas for further development. Volume 49 of Sociological Methodology features two articles on topics related to hierarchical or multilevel modeling that advance different aspects of these approaches, both geared to practitioners and both recognizing problems in the standard toolkit of approaches to such data.

The article “Getting the Within Estimator of Cross-Level Interactions in Multilevel Models with Pooled Cross-Sections: Why Country Dummies (Sometimes) Do Not Do the Job,” by Marco Giesselmann and Alexander W. Schmidt-Catran, examines the limits of a common approach to studying hierarchically organized data. When analyzing cases where survey respondents are nested within country-years, which are in turn, of course, nested within countries, it is common for researchers to estimate within-country effects that seek to account for unobserved heterogeneity. Past research on this topic has demonstrated the desirable properties of such an approach for understanding country-level characteristics, but this work has devoted insufficient attention to the performance of cross-level interactions in these models. Using a combination of algebraic logic, Monte Carlo simulation experiments, and other arguments, the authors demonstrate that unobserved correlated moderators can be a substantial source of bias. The authors guide readers through the limitations of the standard approaches and discuss a new specification of the country fixed-effects estimator that allows researchers to obtain unbiased within-country estimates of such cross-level interactions when there is unobserved effect heterogeneity between countries. In addition, the article offers some thoughts on the decomposition of effects into within and between components in a random-effects framework. To highlight the importance of these results for practitioners, the authors reevaluate a previously published study that used a country fixed-effects approach to study the relationship between labor market policies and subjective well-being among people who are unemployed. Given the growing availability of international research data and interest in questions about national differences (in policy, economy, and social situations), this article will be of broad interest.

The second article that deals with issues pertaining to hierarchical or multilevel modeling, “Assessing Differences between Nested and Cross-Classified Hierarchical Models,” by David Melamed and Michael Vuolo, gives a detailed overview of the distinctions between cross-classified and fully nested models. Data are cross-classified when units at one level appear in multiple units at another level; for instance, when students are measured in two separate schools, each student’s record exists in two schools. In such cases, the data are not fully nested; that is, there is no way to organize the data such that each higher level unit contains all observations of each lower level unit, as would be the case if, for instance, no student ever switched schools but each student might appear multiple times (e.g., at different grades) in a given school. Cross-classified data are challenging to model in a computationally efficient manner. Prior work has proposed a number of shortcuts to make these problems analytically tractable, but a broad understanding of the tradeoffs these shortcuts entail is lacking. This article explicitly examines what such shortcuts mean for parameter estimates. Using a Monte Carlo framework, the authors conduct a series of simulation studies. They find that when outcomes are continuous, there are substantial differences between different shortcut approaches in the standard errors and the explained variances, indicating the likelihood of challenges for hypothesis testing; for binary outcomes, there are similar but less consequential issues. The authors conclude with a call for humility: When data are cross-classified but researchers model them as fully nested, it is imperative to explicitly discuss the limitations associated with this approach. Whenever feasible, cross-classified models should be used. However, this is not always possible, which means future scholarship should prioritize work on developing more computationally efficient approaches to handling such data.

Models for Diffusion in Social Networks

That actors are embedded in networks of social relations is a fundamental sociological insight that has generated a vast literature. In this line of research, the topic of diffusion—the spread of ideas, information, resources, behaviors, diseases, and other items through network ties—features increasingly prominently. Network models of diffusion have witnessed an explosion of research by sociologists and others in recent decades (Bearman, Moody, and Stovel 2004; Centola 2010; Centola and Macy 2007; Christakis and Fowler 2007; Goldberg and Stein 2018), building on classic insights about such processes (Coleman, Katz, and Menzel 1957; Granovetter 1973; Rogers 1962). Despite several methodological advances in the study of diffusion, two topics remain particularly underexplored: measurement error, specifically the uncertain accuracy of measured ties between actors that are developed from survey responses or other sources, and the dynamic co-evolution problem, whereby both actors’ attributes and network ties change over time in ways that impede the ability to distinguish cases where actors’ attributes change in response to diffusion from ties to peers with different attributes from cases where actors change their ties in response to their peers’ attributes.

The article “Social Space Diffusion: Applications of a Latent Space Model to Diffusion with Uncertain Ties,” by Jacob C. Fisher, tackles the first problem in studies of social network diffusion: Errors are inherent in the measurement of social networks. Such errors might arise from numerous sources, including but not limited to caps on the number of peers respondents can nominate in the “name generator” portion of a network survey, poor recall when selecting from a list all the people to whom one is tied, or non-real connections observable in online digital trace data (e.g., when two people are nominally connected on a social media platform but each “mutes” or otherwise suppresses posts from the other, making it unlikely they will see each other’s posts despite the appearance of a connection). Even if network ties were measured accurately at one point in time, networks are dynamic and may not reflect actual ties that exist at other points. In any case, networks are never measured perfectly, but studies of diffusion rarely recognize this. Fisher’s article proposes a new approach to address these limitations. He suggests using a latent space model of social networks based on observed ties and other features of the data to create a predicted network, where each tie is assigned a probability of existing, then simulating diffusion on the predicted network, with diffusion more likely over the ties with higher probabilities than the ties with lower probabilities. Testing this proposed method on longitudinal data from a set of schools in rural Iowa and Pennsylvania (the PROSPER study), he finds that a diffusion model based on the proposed latent space approach predicts attitude changes 10 percent better than a diffusion model using the measured network alone. These results suggest there is substantial promise for studies of diffusion to incorporate a more probabilistic framework that recognizes and explicitly addresses the innate measurement error that accompanies network data.

The second article on social networks in this volume, “No Longer Discrete: Modeling the Dynamics of Social Networks and Continuous Behavior,” by Nynke M. D. Niezink, Tom A. B. Snijders, and Marijtje A. J. van Duijn, deals with the second problem in studies of network diffusion: co-evolution. The co-evolution of actor attributes and network ties has certainly received a fair amount of methodological attention (e.g., Snijders 1996), but a central limitation of these studies is that they require actor attributes to be discrete. The article by this trio offers a methodological advance that does away with this requirement. To achieve this, the authors use stochastic differential equations. The article also contains other innovations likely to be of interest to researchers studying co-evolving network dynamics, including a measure of the amount of variance explained and a clear walk-through of how to interpret parameter estimates. To demonstrate the validity of this approach and offer a road map for others to implement these procedures, the authors use data from the Teenage Friends and Lifestyle study, which surveyed a cohort of 160 12- to 13-year-old students at a secondary school in Glasgow over a two-year period in the 1990s. Specifically, they study the co-evolving dynamics in the friendship network, individuals’ self-esteem, and alcohol use. Many readers will find the appendices of special interest, where the authors give a step-by-step description of how to estimate and interpret such models. These innovations will be very useful to researchers who wish to apply these methods to their own studies.

Notes and Comments

Despite limited space, the editors see justification for the inclusion of relevant commentary on previously published SM articles to improve the level of discourse in sociological methodology. In this volume we publish two comments, one by Benjamin F. Jarvis on an article published in SM volume 42 by Elizabeth Bruch and Robert Mare, “Methodological Issues in the Analysis of Residential Preferences and Residential Mobility” (see Bruch and Mare 2012). Jarvis’s comment (prepared with input from Bruch and Mare), “Estimating Multinomial Logit Models with Samples of Alternatives,” reviews the use of what sociologists and epidemiologists call conditional logistic regression but which economists and choice modelers call multinomial logistic regression (MNL). The comment corrects the sociological record with regard to methodological developments in econometrics and provides updated guidance for sampling in MNL models. Jarvis concludes that the advice given to researchers by Bruch and Mare (2012) may lead to bias in coefficient estimates in residential mobility research and the sampling corrections they encourage are not recommended.

A second comment, by Arvid Sjölander and Yang Ning, builds on an earlier SM article by one of the authors (Sjölander 2017). The earlier article developed an important extension of the case-time-control design, a tool that has existed in the epidemiological literature for the analysis of treatment effects, controlling for measured time-varying covariates and unmeasured covariates that are constant within persons over time. Prior to Sjölander’s article, the design was restricted to two time points and a binary exposure variable. His approach allows for an arbitrary number of time points and covariates and a nonbinary exposure. The present paper, “A General and Robust Estimation Method for the Case-Time Control Design,” presents a further improvement in the estimation strategy. The authors draw attention to the restrictiveness of the estimation methods proposed by Sjölander (2017) and offer an alternative: the conditional maximum-likelihood (CML) estimation method. The new method provides consistent estimates when the previous approach did not. This novel approach relaxes to a large extent the restrictive assumptions in the estimation strategy proposed originally. Through a series of simulations and the analysis of real data, the present article shows that their new estimation approach performs well under a range of scenarios relative to Sjölander’s (2017) estimation method.