Abstract
Han Zhang and Jennifer Pan (this volume, pp. 1–57) have made two important contributions in their article “CASM: A Deep Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media.” First, they use high-quality methods to generate an important data set about protests in China for a 7.5-year period, from January 1, 2010, to June 30, 2017, from Twitter-like Sina Weibo posts, and they have provided detailed methodological information to validate the data. This dataset promises to be a rich source for analysis of Chinese protests. It also appears they will be able to update the protest series, at least until China’s controls or the Weibo platform itself change enough to block their method. Second, they have provided a model for how other researchers could go about collecting information about protests. Even though some of their specific methods cannot transfer to other settings, their general approach is well worth emulating. At the same time, their careful methodology reveals the ongoing issues involved in compiling consistent and unbiased information about protests that all researchers should attend to.
Reports of protests are needles in haystacks of irrelevant material. The landmark Dynamics of Collective Action data set was constructed by humans who read New York Times articles for 1960 to 1995 (Soule and Earl 2005). At an estimated 170 articles a day (Meyer 2016), they read 62,050 articles for each year for 35 years for an estimated 2,171,750 articles. They found a total of 23,616 events in these articles, a hit rate of perhaps 1.1 percent. Patrick Rafail used keywords to identify possible protest events in news sources from 20 cities and scanned more than a million retrieved articles to locate the roughly 11,000 that were read and coded, a hit rate of about 1 percent (Martin, Rafail, and McCarthy 2017). Zhang and Pan report that they found only one reported protest in human coding of a random sample of 20,000 Weibo posts, or an estimated hit rate of 0.02 percent.
Computers are more efficient and get less bored in sifting through haystacks. Zhang and Pan provide an exemplary model of documenting the steps in their data creation process and performing a series of careful checks on the validity of their work. To train a computer to find the protest needles, they took advantage of Wickedonna, an existing human-curated list of 67,502 protests with 240,521 associated text Weibo posts and 233,288 images and videos. They acknowledge that the Wickedonna creators did not document their selection criteria.
Zhang and Pan rely on heavy-duty image- and text-processing neural network algorithms that are beyond the technical capacity of most social scientists, including me. But even for social scientists who cannot hope to do this kind of programming themselves, the authors provide an excellent model for how social scientists could work with qualified programmers to merge good social science research design with advanced programming. Their work also tells social scientists what kind of artificial intelligence work is possible as they go looking for programmer collaborators or look to upgrade their own skills.
One important feature of Zhang and Pan’s work is that they classified the posts in several passes. First, they obtained a list of the 50 most frequent words from the Wickedonna data to use as the initial search criteria to select a subset of 9.5 million posts. These words are a combination of “form” words (synonyms for various protest types) and “issue” words such as wage arrears, homeowners, and migrant workers. From human coding, they estimated that the rate of false positives is 93 percent, a positive hit rate of 7 percent.
They put these through several passes of artificial intelligence training with Wickedonna posts as positive data. The first set of negative data were 200,000 randomly selected geolocated Weibo posts. Results of this training were used to select 450,000 posts from the 9.5 million with the relevant keywords that had a very low probability of having a protest, which were used as a second set of training data in the first-stage classifier. This was better but still had a lot of false positives. Then they human-coded a random sample of false positives and, on this basis, eliminated all posts from government agencies. Next, they human-coded 40,000 posts that the classifier gave a probability of 0.8 or higher to, finding about 24 percent referred to protests and 76 percent did not. The 76 percent that had keywords but were not protests were used as the negative training data; the 24 percent that were protests, supplemented by Wickedonna posts, were the positive training data. This second stage reduced the false positives substantially. After the second-stage coding, they estimated that about 79 percent of the posts CASM classifies as protests really are, and 90 percent of the protests identified by humans can be captured by CASM. Not perfect, but very good.
Humans are also imperfect at identifying protests from media traces. Although researchers agree in recognizing certain prototypical actions as protests, they disagree about exactly what criteria to use and how to adjudicate borderline “edge” cases. To be classified as a protest, an event must meet multiple criteria about the actors, the issue, and the form. The Soule team defined relevant events as involving a claim by more than one person in a public sphere, and they included lawsuits and petitions. My team defines protests as nonelite actors’ advocating about collective issues (not just personal grievances) using unconventional forms and excludes lawsuits. Zhang and Pan define protests as having powerful targets, affecting the interests of the actors, being contentious and in physical space, and involving at least three people. Given that Zhang and Pan trained their system to the Wickedonna data, and we do not know the Wickedonna definition of a protest, I think it is fair to say that the real criterion is the vaguely defined “we know it when we see it” criterion that we all use interpretively. Honestly, I do not see any way around this definitional blurriness and think only that we need to recognize the lack of precision at the edges as we do our work.
In terms of improving the China data, there are possible tweaks around the edges. First, I was confused about why there needed to be at least eight segmented words to code a post. I would think a lot of protests could be described in a few words like “Protest unfair evictions.” Second, it appears that only 32 percent of the posts contained codable geographic information that could allow attribution to a county, and 46 percent had to be discarded because not even the prefecture could be identified. It appears that only date, location, issue, and form are identifiable from the posts. It is not clear whether there is any hope of identifying event size.
I wonder whether there could be another round of tweaking the search. Having determined that a lot of false positives came from government sources and thus discarding all government posts, it would be reasonable to verify that government posts are not a big source of Wickedonna protest posts. Also, because 45 percent (15.8 percent/35 percent) of the 35 percent of Wickedonna protests not found by CASM were due to their not having any of the 50 keywords used in the initial search, one wonders whether there is more to be learned. Would a few more keywords capture a significant fraction? Or are they just too idiosyncratic? People are quite able in all languages to describe protests using creative and innovative vocabularies.
This last point leads to a consideration of the implications of this work for other researchers. A common strategy in searching electronic news archives is to begin with protest-relevant words because, otherwise, we have far too much irrelevant dross to search. But then we miss posts or articles that do not use those words. But even as one wants a machine algorithm to find “everything,” we must remember that humans make mistakes, too. The definitions of what is or is not a protest event are disputed, and for many events, knowledgeable researchers will disagree about whether it should be classified as a protest even after a long discussion.
Even though many of their specific techniques are tied to their case, Zhang and Pan’s strategy for approaching the problem provides important insights for other researchers. Apart from finding or generating some source of positive training data and either learning advanced programming or collaborating with programmers who know the advanced methods, the other implications seem to be seriously thinking about the negative training data, using images where they exist, and classifying in multiple passes, a first “coarse” pass to identify sources that are not obviously irrelevant, then a finer pass to eliminate more false positives.
We need to consider seriously the pluses and minuses of different kinds of sources. Zhang and Pan convincingly argue that social media is likely to be an important source in authoritarian regimes, where the news media cannot be trusted to report on dissent. In less authoritarian spaces where the media are independent, we need to consider the trade-offs in the limitations of short-form social media versus long-form news reports as sources. Each has advantages and disadvantages. The short forms in social media are often extremely sparse in their information, and in nonauthoritarian contexts, a large share of social media content involves sharing mass media sources. Mass media news sources are more likely to describe events in greater detail. On the other hand, social media are the only way to find out about events that the mass media are ignoring. Short-text posts also have the advantage of being easier to obtain electronically and to classify (because they lack a lot of extraneous material), but this same sparseness puts sharp limits on what can be learned from them. Most of the Weibo posts lack geolocation, which makes figuring out where the event occurred a problem. Twitter similarly has geolocation turned off by default as a privacy measure. Social media images are more likely to be contemporaneous and pegged to the day they are posted, but not always. As Zhang and Pan note, they have to classify multiple posts from the same county or prefecture on the same day as being the same event, a decision that many other researchers make. News media more often mention when an event happened, but at times the mention is imprecise (e.g., “last week”). Automated coding of news media often assumes the event location is the dateline, but this assumption is often false. News media are essential for studying the past; social media are rapidly changing.
In the long run, the ideal would be to develop protocols that allow events collected in different ways from different sources to be merged. This is how event catalogs started, with researchers consulting multiple sources and merging that information into lists of events by date and location and issue that could tell a story that no single source was telling. This, in turn, requires data collection protocols that keep track of the original sources of information, including all the multiple sources for any given event, so that alternative sources can be checked against each other. Even in the Weibo data, the ratio of posts to events was 2.01, and doubtless some events were posted about many times while the majority got only one post.
To end where I began, Zhang and Pan have made a significant contribution that has many elements worth emulating, even as the very quality of their work highlights the important difficulties that remain in compiling protest event data.