Adapting computational text analysis to social science (and vice versa)

First difference: Supervised vs. unsupervised machine learning

Topic modeling and many other text-analysis tools have their roots in machine learning. Typically, in machine learning problems, one has a class of cases of known type and a class for which the type is unknown (Almeydin, 2014). One divides the former into a “training set” and a “testing set”; develops a model based on the former that is predictively effective for the latter; and applies that model to classify the cases for which the type is unknown. An excellent example of this approach is Jockers and Mimno (2013), who used supervised topic models (“supervised” refers to models based on cases of known type) to identify the gender of anonymous or pseudonymous authors of 19th-century novels. Supervised models have had a wide range of practical applications (think, e.g. the Netflix challenge, where contestants used supervised-learning models to improve Netflix’s ability to recommend to users films that they would enjoy).

Arguably, however, the major strides in computational text analysis in recent years—and the ones of greatest use to social scientists—have entailed the development of unsupervised approaches to exploring the latent structure of textual corpora: Latent Dirichlet Allocation and related approaches like Correlated Topic Models and Dynamic Topic Models (Blei, 2012). Social scientists have used such models productively, for example in identifying innovative patents (Kaplan and Vaikli, 2014), assessing the political leanings of economists (and correlating these with political behavior and research conclusions) (Jelveh et al., 2014), and discerning previously unrecognized continuities among historical waves of the US women’s movement (Nelson, 2014). But unlike supervised models, for which straightforward means of validation using held-out samples are available, assessing the quality of solutions from unsupervised models is more challenging, with criteria that are more varied and less definitive.

In other words, the shift from supervised to unsupervised models, especially in some areas of greatest interest to social scientists, requires many of us to move outside our comfort zone in accepting interpretive uncertainty and to develop robust ways to interpret and validate the results of our models. ² There have been promising developments in this respect (e.g. Grimmer and Stewart, 2013) and room for more progress. The key may be starting with interpretive methods borrowed from the humanities, but then disciplining the results through statistical validation. For example, a model interpretation, combined with information external to the corpus, may lead one to hypothesize that the prevalence of some topics should be associated with particular classes of authors, or to anticipate temporal patterns in topic prevalence (DiMaggio et al., 2013).

Second difference: Machine-learning vs. statistical explanation

Whereas social scientists customarily obsess over causality and rely on formal tests of statistical significance, computer scientists using supervised models focus on results. The first topic-model presentation I attended used the method to identify public records particularly likely to require redaction, out of a set of records too immense for humans to screen by hand. The only measure that mattered was whether the models improved prediction (which they did).

Even where held-out samples are not available, however, computer scientists seem to devote more attention to designing models and less on statistical validation (at least in the social-scientific sense) of model solutions. I suspect that this (admittedly stylized) difference is reinforced by variation in disciplinary skill sets: Computer scientists can write new algorithms faster than most social scientists can learn them. In the tradition of supervised machine learning, this makes sense because if you create a better model, you will be rewarded instantly with better results. This approach carried over (at least at first) into unsupervised models, as computer scientists wrote new algorithms at an alarming rate. From a computer-science standpoint, this made sense: algorithms tend to solve big problems, rather than nibble at the edges of small ones.

Social scientists, by contrast, tend to learn a method and adapt it to their needs, in large part because their learning costs are much higher. ³ Social scientists therefore tend to be more interested in model-testing and curation than computer scientists (though there are exceptions on both sides (e.g. Boyd-Graber et al., 2014)), attempting to get the most out of the programs they have mastered. From this standpoint, social scientists have good reason to invest in corpus curation, because it may be their most efficient way to boost the quality of their results.

There is another difference, which is likewise consistent with the supervised-learning roots of text modeling, but has more to do with underlying approaches to explanation. Many computer scientists (with notable exceptions (e.g. Pearl, 2009)) seem less concerned with causality and with model confirmation than are many social scientists. It is not that they care less about getting models right; rather they understand “getting it right” in a different (and I am beginning to suspect more useful) way than do most social scientists, focusing on model plausibility, utility, and descriptive, as opposed to causal, validation. Social scientists accustomed to obsessing over model specification and statistical significance may find this emphasis frustrating, at least at first. (As a novice topic-model user, I literally could not believe that there was no target function I could use as a simple goodness-of-fit criterion to choose among model solutions.) Ultimately, however, the computer-science perspective is liberating, as it forces us to recognize real interpretive uncertainty and seek out appropriate and substantively relevant forms of validation fitted to specific research goals.

Third difference: Computer scientists trust humans more than social scientists do

From Alan Turing (1950) onward, much work in computer science, especially in Artificial Intelligence, has sought to create algorithms that can replicate the results of human problem solving. The difficulty of this task has generated great respect among many computer scientists for the human brain (which, as computers go, is an impressive piece of hardware). This tradition has influenced Natural Language Processing and, especially, Sentiment Analysis, where human reasoning is routinely described as a “gold standard” against which algorithmic output should be judged. Tasks like the Netflix Challenge induce computer scientists to write programs that try to simulate human evaluation processes; and much research on sentiment analysis employs as training sets human reviews from websites like Yelp or Rotten Tomatoes that combine text and summary evaluations (for a thoughtful review see Liu, 2010).

By contrast, social scientists, at least those who have paid attention to work in cognitive psychology, are deeply suspicious of human judgment. I, for one, harbored the hope (of which my computer science colleagues have disabused me) that computational analysis could free us from dependence on pesky humans, whose judgments are clouded by hard-wired errors of reasoning (Kahneman, 2003); schematic (Nisbett and Wilson, 1977) and ideological priors (Graham et al., 2009); and vulnerability to emotional environments (Shiller, 2015), priming (Gilbert, 1991), stress (Hammond, 2000), poverty (Mullainathan and Shafir, 2013), pride (Tourangeau and Yan, 2007), and prejudice (Hardin and Banaji, 2013).

As anyone who has ever hand-coded text or supervised others in doing so is aware, measures of inter-rater reliability are correlated directly with the insignificance of the information one wants to extract from the data. Give two coders of, say, a corpus of articles describing episodes of civil unrest, a factual question (e.g. where did the demonstration take place?) and they are very likely to agree. Ask them something more interesting (what grievances did participants express?) and convergence will be more difficult to achieve. Likewise, human coders are good at straightforward linguistic tasks, such as distinguishing between homonyms with very different meanings. But they are less trustworthy when differences of perspective shape the emotional context framing their judgments. For example, asked to code the affective tone of the sentence, “the Isis fighters celebrated their rout of the Iraqi troops,” an Isis supporter is more likely to discern positive sentiment than would a coder sitting in Washington or Erbil. Computer scientists who treat human judgments as a gold standard and social scientists who see in sentiment-analysis programs an antidote to human imperfection will often be disappointed, as algorithms and humans seem to be bad at pretty much the same types of tasks. For now at least, we can savor the irony that each group is most skeptical about the entities (algorithms or people) with which its expertise is most closely associated.

What do social scientists need?

Three priorities follow from these observations, two for topic models and one for sentiment analysis:

We need better ways to choose among potential models of a textual corpus. Some outstanding work is already being done on model diagnostics by both computer scientists (Boyd-Graber et al., 2014; Lau et al., 2014; Mimno and Blei, 2011) and social scientists (Roberts et al., 2014). The challenge is that different criteria point in different directions: statistical goodness-of-fit measures tap conformity to model assumptions that are unrealistic descriptions of the world, but are useful as yardsticks for measuring variation in texts produced by different authors or at different times. Measures of model robustness can identify solutions at the center of a model space (e.g. by measuring distances between solutions and using multi-dimensional scaling to project them onto a space), but the relationship between centrality and model quality has yet to be established (Marshall, 2013). Even solutions that successfully predict types in testing samples may not be the “best” by criteria of semantic interpretability (Chang et al., 2009). Approaches that focus on the fit or robustness of topics that are analytically important to the investigator seem particularly promising for many purposes (Mimno et al., 2011).

Engagement with computational text analysis entails more than adapting new methods to social-science research questions. It also requires social scientists to relax some of our own disciplinary biases, such as our preoccupation with causality, our assumption that there is always a best-fitting solution, and our tendency to bring habits of thought based on causal modeling of population samples to interpretive modeling of complete populations. If we do, the encounter with machine learning may pay off not just by providing tools for text analysis, but also by improving the way we use more conventional methods. To be sure, there are risks in going against the grain. But this is a great time for social scientists to get involved in computational text analysis, because the field is relatively young, the challenges are intellectually captivating, and there is still time to influence the shape that these methods will take as they enter the social sciences.