CASM: A Deep-Learning Approach for Identifying Collective Action Events with Text and Image Data from Social Media

3.2.1. Convolutional Neural Network for Image Classification

We use a CNN for image classification. A CNN is a model that consists of a series of operations, called layers, where each layer takes the output of the previous layer as input and, after performing some operation on it, passes the output to the next layer. CNNs get their name from the operation of convolution—element-wise multiplication between matrices followed by summation. The output of convolution is called a feature map, and each layer can contain multiple feature maps. After each convolution, an activation function is applied to introduce nonlinearity because most real-world data are nonlinear, yet convolution is a linear operation. The most common activation function for CNNs is rectified linear unit (ReLU), an element-wise operation applied per pixel to replace all negative pixel values with zero, producing a rectified feature map. Feature maps and rectified feature maps are high dimensional, so spatial pooling is often applied to reduce dimensionality and so features can be identified regardless of their position in an image (translation invariance). Generally, multiple convolution layers are used to extract useful features from the raw data. After these layers, one or more fully connected layers, often a type of neural network called a multilayer perceptron, learns nonlinear combinations of the features generated from the convolutional layers and uses all features to classify the image. ¹⁹ The final fully connected layer typically uses a softmax activation function, or normalized exponential function, to generate the output value.

There are many variants of CNNs, which differ based on the architecture of the network as well as parameters such as the number of filters, the filter size, and stride. The architecture we use is called VGGNet, also known as VGG or VGG-16, which we chose based on its conceptual simplicity, ease of implementation, and wide-ranging applications (Simonyan and Zisserman 2015). ²⁰ VGGNet uses 16 convolutional layers to extract features and three fully connected layers to perform classification. VGGNet uses small filter sizes (3 × 3) and more layers (16) instead of larger filter sizes (7 × 7) and fewer layers, as was common in previous models (Krizhevsky, Sutskever, and Hinton 2012). ²¹ In VGGNet, a ReLU operation follows each convolutional layer, and max pooling is performed after the 2nd, 4th, 7th, 10th, and 13th convolutional layers.

VGGNet was trained on a set of 1.2 million images, classified into 1,000 categories. We do not use the entire pretrained VGG model because human faces and crowds (not to mention collective action events) are not included among the 1,000 categories VGGNet was originally trained for. Instead, we trained and fine-tuned the last four convolutional layers with our own data, as illustrated in Figure 1. ²²

OPEN IN VIEWER

Figure 1.

Illustration of our convolutional neural network architecture for image classification.

We do not change the first 12 convolutional layers of VGGNet because they identify more basic features of images (e.g., edges, circles); subsequent layers use these basic features to learn more complex features (e.g., human faces, signs, placards) specific to our task. The structure of the last four layers we use is consistent with the original VGG architecture in terms of filter size and stride. After the convolutional layers, we added a fully connected layer with ReLu activation and dropouts and a second fully connected layer with logistic sigmoid function to output the binary-class probability. ²³ The final CNN output is a probability between 0 and 1, where 1 means the image is certain to represent offline collective action and 0 means the image does not represent offline collective action.

We train this model to minimize cross entropy loss because our output is binary:

L ( p , y ) = − 1 N ∑ n = 1 N [ y n ( log ( p n ) + ( 1 − y n ) log ( 1 − p n ) ] ,

(1)

where p is the output of predicted probability, y is the labels of the training data, and N is the number of images; p_n is the nth output, and y_n is the nth ground truth label. We minimize cross-entropy loss by using an adaptive gradient-based optimization algorithm (Kingma and Ba 2014). ²⁴

3.2.2. Convolutional and Recurrent Neural Network for Text Classification

To classify our text data, we use a model that combines convolutional layers with a recurrent neural network (RNN) with long short-term memory (LSTM) architecture. Recurrent neural networks are used extensively in dealing with sequential data, and they have set the standard for performance on natural language processing tasks such as speech recognition and machine translation (Bahdanau et al. 2014; Mikolov et al. 2010; Sak, Senior, and Beaufays 2014). RNNs are a type of model that performs the same operation repeatedly on sets of sequential inputs. Central to an RNN is a state vector that accepts an input and the previous state to produce a new state and output. The shortcoming of “vanilla” RNNs is they are difficult to optimize due to the effect of vanishing gradients (Pascanu, Mikolov, and Bengio 2013). LSTMs were created as a special kind of RNN that can learn long-term dependencies in a computationally tractable manner with cell vectors that control what information from the previous sequence of operations is retained (Hochreiter and Schmidhuber 1997).

The architecture we use combines an embedding layer, convolutional layers, an LSTM layer, and two fully connected layers (Sainath et al. 2015; Wang, Jiang, and Luo 2016; Xiao and Cho 2016; Zhou et al. 2015). Figure 2 shows this architecture.

OPEN IN VIEWER

Figure 2.

Illustration of our convolutional and recurrent neural networks with long short-term memory architecture for text classification.

The embedding layer, shown on the left of Figure 2, is an operation to provide each word in our text data with a dense representation of the word and its relative meanings (Peters et al. 2018). Although social media text is short, the set of words, or vocabulary, used in social media is large. This means that when social media texts are represented as a bag-of-words—an unordered set of words—not only is word ordering lost, but the vector that represents the text is exceptionally sparse. With word embeddings, words are represented by dense vectors, where a vector represents the projection of the word into a continuous vector space and the position of the word in that space is learned based on other words that surround it in the text. In our case, each word is represented by a 128-dimension word vector we trained with a continuous skip-gram model using 20 million Weibo posts (Mikolov et al. 2013). ²⁵ This embedding layer can be thought of as a type of transfer learning, where we use the word vector model to add information about each word in our data.

The convolutional layers in the CNN-RNN extract features just like the convolutional layers of the CNN. ²⁶ For image analysis, the matrices where convolution is performed are subregions of the image’s pixels. For text analysis, convolution is performed on subregions of a matrix of words of fixed dimension. We use an n×n matrix, where n is the length of the vector that represents each word (128), because no Weibo post exceeds 128 words and 128 has nice mathematical properties (128 = 2⁷). If the number of words in the text is less than n, extra rows of zeros are added as part of a process called padding, which is standard when implementing CNN for text classification. For example, the very left of Figure 2 shows a short text, “I saw a protest in Wukan,” which in our case results in a matrix of size 6 × 128. An additional 122 rows of zeros are added to generate a 128 × 128 matrix. Then, as with image data, subregions are defined by filter size and stride. ²⁷ Here, we also use a ReLu activation function, add max pooling after each convolution layer, and apply dropout.

Instead of using convolutional layers for feature extraction, we could have defined features ourselves, such as using bag-of-words or n-grams as features. We use convolutional layers for feature extraction rather than bag-of-words features to avoid losing word order and information about grammatical syntax. For example, CNNs can capture as features phrases such as “defend my rights” that are meaningful beyond the words that compose the phrase, even when such phrases appear across diverse social media posts: “I protest to defend my rights,”“Why should I defend my rights?,” or “Company X can defend my rights as a consumer.” We use convolutional layers rather than n-grams because n-grams exponentially increase the size of the vocabulary, introducing a high level of noise (Tan, Wang, and Lee 2002).

We use LSTM on top of the convolutional layers because LSTMs perform better in preserving long-range dependencies within sentences and short texts (Sutskever et al. 2014). Long-range dependencies matter because meaning in a sentence or social media post is often determined by words that are not very close together. For example, a social media post such as “The people in the square were wearing ponchos during the protest because of the heavy rain” is about people protesting, not about people wearing ponchos, and LSTM is more likely to capture the long-range dependence between people and protest. Our LSTM layer is fixed to a bidirectional LSTM, which scans the inputs in forward and reserve order, preserving the proceeding and following features (Schuster and Paliwal 1997). ²⁸

Finally, similar to the CNN model, there are two fully connected layers: The first applies the ReLU activation function and dropouts, and the second is a logistic sigmoid function that outputs the binary-class probability of whether the post’s text is discussing real-world collective action. To train this model, we again minimize cross-entropy loss with an adaptive gradient-based optimization algorithm.

3.2.3. First-Stage Classifier

The first-stage classifier uses the CNN model described in Section 3.2.1 to classify image data and the CNN-RNN classifier described in Section 3.2.2 to classify text data. In this first stage, we train the CNN model using a random sample of 230,000 images ²⁹ from the Wickedonna data set as our positive training data (i.e., examples of images that pertain to collective action events). We use a random sample of 230,000 images from geolocated Weibo posts as the negative training data (i.e., examples of images that do not relate to collective action). We train the CNN-RNN model using the 240,521 text-based posts from the Wickedonna data set as the positive training data. We use a random sample of approximately 200,000 geolocated posts from Weibo as the first negative training data. ³⁰ This random sample of posts is extremely unlikely to contain discussion of collective action events or many keywords from K. If we only use these data as the negative training data, then the positive training data would all contain protest-related words and the negative training data would not, which would bias the classifier into making predictions about collective action based on whether a protest-related word is present. To ameliorate this issue, we use approximately 450,000 posts that contain keywords from K but have a very low likelihood of being about collective action as the second negative training data set. ³¹

After training the CNN and CNN-RNN models with these data, we use the trained models to make predictions about the approximately 9.5 million Weibo posts containing at least one of the words in K. If the input Weibo post only contains text, then the CNN-RNN model generates the predicted probability that the text relates to offline collective action (p_text). If the input post contains text and images, the CNN-RNN model generates the predicted probability that the text relates to offline collective action (p_text), and the CNN model generates the predicted probability that images relate to offline collective action (p_image). When multiple images are associated with one social media post, we take the largest predicted probability as p_image. The 9.5 million posts include just under 3.6 million images.

Figure 3 shows 11 images whose predicted probabilities as assigned by the CNN in the first stage range from 0 to 1.0. These images are those whose predicted probabilities are closest to the integer values listed in the figure. For example, the third picture from the left (a night street scene) is the image with predicted probability closest to 0.2. Figure 3 suggests the image classifier has construct validity. Images with higher predicted probabilities of relating to collective action contain crowds, signs, placards with text, and government buildings. The appearance of a picture containing text (third image from the right, with predicted probability of 0.8) is not surprising as Chinese social media users often post images of text to discuss sensitive topics in an attempt to avoid censorship.

OPEN IN VIEWER

Figure 3.

Images with their predicted probabilities of relating to collective action generated from the convolutional neural network in the first-stage classifier (some images are cropped).

When a Weibo post contains both p_text and p_image, we combine the two predicted probabilities to obtain a single predicted probability for each post. How the probabilities are combined depends on two tuning parameters (α and β):

p = { ( p text + α · p image ) ( 1 + α ) · β if the post has images , p text otherwise .

(2)

As seen in Equation 2, α controls how much information we should borrow from text versus image data. If α is higher, more weight is placed on the output of the image classifier. β controls how much extra up-weight we give to posts that contain both text and images. The intuition here is that images in social media posts can be informative. Of 10,000 human-coded posts that contain protest-related words, ³² only 23.9 percent contain images in addition to text, but among posts identified as related to collective action in human-coded data, 56.9 percent contain images in addition to text. This suggests protesters may post pictures strategically when publicizing their efforts on social media.

We use cross-validation to select the optimal values of α and β. ³³ The optimal α and β for the first-stage classifier are 0.44 and 1.10, respectively. The α is smaller than 1, which means relatively more information is extracted from the CNN-RNN model of text, but β is larger than 1, confirming our intuition that a post with both text and images is more likely to be about collective action than are posts with only text.

We evaluate the performance of the first-stage classifier with cross-validation and out-of-sample validation. Cross-validation is the dominant approach for evaluating machine-learning systems of event detection (Hanna 2017; Nardulli et al. 2015). The training data are split into k equal subsets (we use k = 5). Each subset is used to calculate precision and recall with the rest used for training, and this process is repeated k times. The advantage of cross-validation is that class labels are already known for the training data, so precision and recall can be directly estimated. The first-stage classifier performs extremely well in cross-validation, with a maximum F₁ score of 0.96 (precision = 0.95, recall = 0.96). ³⁴ The first-stage classifier vastly outperforms random guess, and above the range of F₁ scores (0.6 to 0.8) for existing supervised machine-learning systems for event classification (Adams 2014; Hanna 2017).

The problem with cross-validation is that the training data could differ from the data researchers ultimately want to apply the classifier on. For example, the positive training data used for CASM in China could be based on a definition of collective action that differs from ours and draws from a broader range of data sources. Therefore, precision and recall based on cross-validation can paint a rosier picture of the algorithm performance than is warranted. To address this problem, we conduct out-of-sample validation with 10,000 posts to mimic the context where the classifier will be used, thus providing a more realistic evaluation of the system. We take a stratified random sample of 200 posts per each keyword from the 9.5 million posts collected between 2010 and 2017 (these posts are not used during training). We trained human coders to code each of the 10,000 sampled posts as discussing a collective action event or not per our definition. ³⁵ We assess the performance of our classifier based on this independent validation set. The first-stage classifier achieves a maximum F₁ score of 0.69 (precision = 0.66, recall = 0.73). Figure 4 shows the precision-recall curve of the CNN image classifier alone, the CNN-RNN text classifier alone, and the combined classifier from the first stage based on out-of-sample validation. ³⁶ There is often a trade-off between precision and recall, but Figure 4 shows that the text-based classifier outperforms the image classifier, and the combined classifier outperforms both across the precision-recall curve.

OPEN IN VIEWER

Figure 4.

Precision-recall curve of the image-based convolutional neural network model (dotted line), text-based convolutional and recurrent neural networks with long short-term memory model (dotted dash line), and the combination of the text and image classifier (solid line) from the first-stage classifier based on out-of-sample validation.

The first-stage classifier can correctly identify posts about collective action. Figure 5 shows two posts about collective action that are identified by the first-stage classifier as such (true positive). The first stage also correctly classifies posts containing words K that are not related to collective action. Figure 6 shows a post unrelated to collective action that is classified as not relating to collective action (true negative). ³⁷

OPEN IN VIEWER

Figure 5.

Weibo posts about collective action correctly classified as related to collective action (true positives) from the first-stage classifier.

OPEN IN VIEWER

Figure 6.

Weibo post containing one of the words in K, “surrounded by a mob,” correctly classified as not related to collective action (true negative) from the first-stage classifier.

Overall, the performance of the first-stage classifier is strong, but we wanted to do better than correctly classifying 66 percent of posts as collective action (precision = 0.66) and correctly identifying 73 percent of the collective action posts from T_K (recall = 0.73). In particular, we want to improve precision to make sure more posts we classify as collective action meet our definition of collective action. To do so, we need to reduce the number of false positives. We systematically examined the false positives from the first-stage classifier by taking a random sample of 2,000 false positives and examining them by hand. We found two types of false positives. The first were posts made by government-related social media accounts that describe how public grievances are resolved. Figure 7 shows such a false positive, where a local government is publicizing how it was able to resolve worker grievances over owed wages. Reading the post in Figure 7, it is possible that workers did protest and then the government responded; however, this post does not meet our definition of collective action because it does not provide direct evidence of worker protests.

OPEN IN VIEWER

Figure 7.

Post made by a government Weibo account incorrectly classified as related to collective action (false positive) from the first-stage classifier.

Social media posts that describe issues, problems, and grievances that can lead to collective action but collective action is not apparent in the posts are the second type of false positive. For example, Figure 8 shows a Weibo post made by someone whose house was demolished. This individual was not able to reach an agreement with the government on compensation for their house, so someone (presumably at the government’s behest) destroyed the building in the middle of night. The image that accompanies the text shows the ruins of the building.

OPEN IN VIEWER

Figure 8.

Weibo post that does not meet our definition of collective action, incorrectly classified as related to collective action (false positive) from the first-stage classifier.

Housing demolition appears frequently as a motive for collective action, but in the case of Figure 8, the Weibo post does not provide evidence of collective action as we define it (see Section 3). As a result, this post is incorrectly classified as a false positive. The following two quotes are examples of text-only social media posts incorrectly classified in the same way:

What is the government of this country doing! Forced demolition, forced land-taking, corruption, and taking bribes? (这个国家的政府到底是干什么的!是不是强征强拆，贪污受贿?)

What is law enforcement? Why arrest those who are just trying to demand their wages? The police are recklessly arresting people, beating people without trying to distinguish right from wrong! Is rightfully demanding wages a crime? (什么执法过当? 人家只是讨薪干嘛抓人? 警察不问青红皂白就胡乱抓 人打人! 人家正当讨薪是罪犯吗?)

These posts contain words describing issues that often motivate collective action in China—forced demolition (强拆), forced land-taking (强征), corruption (贪污), and unpaid wages (讨薪)—and words that frequently appear in discussions of collective action—police (警察), arrest (抓), and beating (打人). The first-stage classifier is not effective in distinguishing social media posts about collective action from posts that talk about the same issues, complaints, and grievances but do not meet our definition of collective action. The main reason is that in the training data, the negative examples (posts unrelated to collective action) are much less likely than the positive examples to contain the words, phrases, and images related to issues motivating protest.

To address the first type of false positive, we exclude posts that come from government or Chinese Communist Party accounts. This includes accounts of national and subnational governments, party offices, bureaucracies, and state media outlets. We acknowledge that removing all government posts may decrease the number of collective action events we identify if the government discusses collective action events online that nongovernment users of social media do not discuss online. However, we believe this scenario is relatively rare, and we want to prioritize precision over recall.

To address the second type of false positive, we use a second-stage classifier trained on data with a larger number of negative examples—posts that discuss issues and grievances such as housing demolition, police, and corruption but do not describe collective action events.

3.2.4. Second-Stage Classifier

The second-stage classifier uses the same CNN model as the first-stage classifier (the CNN model whose architecture and training method is described in Section 3.2.1). ³⁸ However, we retrain the CNN-RNN model for the second stage (the architecture and training method remain the same as described in Section 3.2.2). A team of research assistants coded 40,505 posts with predicted probability greater than 0.8 from the first-stage CNN-RNN model. ³⁹ Four undergraduate and master’s students who are native Chinese speakers identified by hand posts related to offline collective action. ⁴⁰ Among the 40,505 posts, 9,761 pertained to offline collective action, and 30,744 did not. These 30,744 negative examples are crucial because they are much more likely to discuss issues and grievances that can motivate collective action than are the negative examples used to train the models in the first stage. To make this training data balanced, we took a sample of 20,983 (30,744 – 9,761) posts from the Wickedonna data set, so there are also 30,744 positive examples. We used these posts to train the CNN-RNN model for the second-stage classifier.

We use the CNN and CNN-RNN models to make predictions of 1,286,514 posts (associated with 1,069,113 images) with predicted probability above 0.70 from the first-stage classifier. ⁴¹ We set the probability threshold to maximize recall before application of the second-stage classifier. ⁴²

Like in the first stage, if a Weibo post contains both text and images, we combine the two predicted probabilities to obtain a single predicted probability for each post using Equation 2. The optimal α and β for the second-stage classifier are 0.22 and 1.14, respectively. As in the first stage, α is smaller than 1, suggesting that relatively more information is based on text classifiers, and again, β is larger than 1, suggesting that posts with both text and images are indeed slightly more likely to be about protest than are posts with only text. Note that α is decreasing from the first to the second stage, suggesting the information taken from images is relatively less important in the second stage. This makes sense because many of the errors in the first stage were due to protest-related words appearing in the text of the posts, and we retrained the CNN-RNN model to better deal with text data. We consider a post as being related to collective action if the combined predicted probability is greater than 0.66, which we selected to maximize the F₁ score based on out-of-sample validation.

In cross-validation, the two-stage classifier performs extremely well, with a maximum F₁ score of 0.94 (precision = 0.93, recall = 0.94). For out-of-sample validation, we again use our test set of 10,000 posts (described in Section 3.2.3). The left panel of Figure 9 shows the precision-recall curve based on random guess (dot-dash line), cross-validation (dotted line), and out-of-sample validation (solid line) from the two-stage classifier. ⁴³

OPEN IN VIEWER

Figure 9.

Precision-recall curve based on random guess (dot-dash line), cross-validation (dotted line), and out-of-sample validation (solid line) from the two-stage classifier (left panel); precision-recall curve of the first-stage classifier (dotted line) and two-stage classifier (solid line) (right panel).

As expected, precision and recall are better for cross-validation than for out-of-sample validation, but both are strong. The right panel of Figure 9 compares the out-of-sample performance of the first-stage classifier along with the performance of the two-stage classifier. The out-of-sample performance of the two-stage classifier is much better than that of the first stage alone, with a maximum F₁ score of 0.84 (precision = 0.79, recall = 0.90). This means that at the end of the two-stage classification process, 79 percent of the posts CASM predicts to be about collective action are indeed about collective action, and CASM captures 90 percent of the human-coded collective action events from our out-of-sample validation data of 10,000 posts. Note that we cannot ascertain “true recall”—that is, to what extent our classifier can retrieve the underlying pool of posts about collective action found in all of social media, because the rarity of posts about collective action makes the creation of a human-validated data set based on all social media posts unfeasible.

From this two-stage classifier, we identify a total of 508,707 T_protest posts out of 9.5 million that are likely discussing collective action between January 1, 2010, and June 30, 2017.

3.3.1. Identifying Location

Identifying the location of a post is less straightforward. Below the central level, China is divided administratively into provinces, provinces into prefectures, prefectures into counties, and counties into townships. ⁴⁴ We want to locate collective action events within these administrative divisions because collective action events in China often involve the government. Even when the target of protest is not the government, protesters often ask for government intervention. It thus makes sense to align location with existing administrative boundaries to identify unique collective action events.

When Weibo users make a post, they have an option to share the exact location where the post is being made. Only 3.3 percent of posts in T_protest (16,770) have this attribute. When this information is available, we use the longitude and latitude from the Weibo metadata to attribute the post to the proper county.

When precise geolocation data are not available, we extract location information from the text of the post. We took a list of names of provinces, prefectures, and counties from China’s National Bureau of Statistics, ⁴⁵ and we looked for these names in the text of posts in T _protest . We found usable geolocation data for 53.8 percent (273,950) of posts in T_protest. We take a conservative approach and discard the posts in T_protest that we cannot geolocate. This means the number of collective action events we identify will be an underestimate relative to the posts we identify.

When we compare the county and prefecture location identified by our text-based extraction method against the location identified through longitude and latitude, we find that our method performs well—the county or prefecture name extracted from the text of the Weibo post matches the county or prefecture identified by longitude and latitude 95 percent of the time.

3.3.2. Identifying Events

To identify events, we combine posts by location and day. When we can identify the county, we consider all posts made within the same county on the same day, defined as a 24-hour period from 12:00 a.m. to 11:59 p.m. China time, to be the same event. When we cannot identify the county associated with a post but can identify the prefecture, we consider all posts made within the same prefecture on the same day to be the same event. We consider posts located to a county to be distinct events from posts made the same day that we can locate to the prefecture above that county (recall that prefectures are subdivided into counties). We group by day because few protests in our hand-coded data are reported on social media for more than one day.

To illustrate this approach in practice, suppose five posts reference Prefecture A on January 1. Two of those posts do not contain county names. Among the three posts that do contain county names, two reference County X, and one references County Y. According to our grouping method, there are three collective action events on January 1: one event in Prefecture A (described in two posts), one in County X (described in two posts), and one in County Y (described in one post).

There are a number of shortcomings to this method. When more than one collective action event occurs on the same day in a prefecture or county, this method undercounts the number of events. This method can also artificially inflate the number of events. Using the previous example—the two posts referencing Prefecture A without mentioning any county names could reference the collective action event occurring in County X or in County Y, and we would mistakenly count these as separate events. ⁴⁶ This method assumes equivalence between date of the post and date of the event when in reality there may be multiple posts about the same event on different days even if the event is confined to a single day. Finally, this grouping method misses cross-regional protests, which are rare in China but would be of substantive interest.

From the 508,707 posts about collective action in T_protest, we can identify location for 273,950 posts, from which we identify 136,330 unique collective action events. Going forward, we refer to this data set of 136,330 unique events as the CASM-China data set. On average, each collective action event is discussed in 2.01 posts. This suggests CASM is able to recover collective action events that receive limited overall attention on social media. ⁴⁷

In future iterations of CASM, we hope to explore alternative methods of post grouping and improve the identification of unique events. For example, we could use additional location information, such as well-known locations that are not administrative regions (e.g., Tiananmen Square, Beijing Railway Station, Zhejiang University), and we could experiment with grouping based on issue in addition to location and time.