Crime rate detection using social media of different crime locations and Twitter part-of-speech tagger with Brown clustering

2.1.1 Twitter data collection

The collection of data from Twitter accounts has been done from January 2014 to November 2018 by scrolling the search outputs for different accounts of seven locations (Ghaziabad, Chennai, Bangaluru, Chandigarh, Jammu, Gujarat, and Hyderabad) of India. The latitude and longitude are given for the considering locations in Table 1 (extracted from Google Map Online). 3801 tweets were collected and saved to the database. We find keywords on social media sites that match keywords and record these matched patterns. This is a word tagging operation. The identification of keywords is very significant in our analysis. Therefore, we determine the words which are directly related to the crime. The following keywords are selected for the analysis of crime rate by using social media as follows: ‘Murder’, ‘Crime’, ‘Encounter’, ‘Hit and Run’, ‘Rape’, ‘Fight’. 999 tweets are filtered for the first location. The keyword “fight” is a topmost word shared by the number of users on Twitter. In the second location, keyword “crime” is widely used by the users. The number of tweets collected for the seven crime cities is shown in Table 2.

The reason to consider these seven locations for the study is because these locations are located in the most popular Indian states for International and Domestic Tourist and the security for Domestic and international tourist is the major concern for Indian Government. According to the report of National Crime Records Bureau, out of these seven locations, 5 locations (Ghaziabad, Chennai, Bangaluru, Gujarat, and Hyderabad) are located in the top most crime rate states in India.

Twitter API is also the part of study for the verification of the proposed approach, we measure the latest crime trends for the highest (Ghaziabad, Uttar Pradesh) and lowest crime cities (Jammu) in India. The measurement of the latest crime trends has been recorded for the time duration of 7 days (23, January 2019-30, January 2019) by using TAGS v6.1.

Handling crime records is important for detecting and preventing crime in any country. Years by years, Indian police is involving in many new approaches for improving the efficiency of crime records system. In 1985, Indian Government constitutes a new task force to maintain the crime records for different locations in India and after the task force recommendations, Indian Government constituted the National Crime Records Bureau (NCRB).

NCRB records are useful for collection of real crime data for the determined locations. The real crime data for the determined locations were collected from the security agencies which are useful to check the accuracy of the present research. Real crime data for seven cities are shown in Table 3 for 2014, 2015 and 2016. Combined datasets of Tweets collected and crime according to NCRB can be shown in Table 4.

The data labeling process is to read Tweets carefully and try to extract the required information. The data sharing on Twitter is useful for detecting many facts to predict the election results, to predict the weather for any location, to track high profiles and also to predict the stars’ publicity. Here, we perform this operation to detect the crime rate. Few of recorded tweets that show the criminal activity performed at different locations are given below for instance:

The post shows that an encounter is going on between the terrorists and security forces in Jammu and Kashmir.

In the above post, twitter user shares the information about the “Rape” case of a Russian Lady in Manali.

2.2.2 Data tagging

The part-of-speech is a major problem in the online conversational text. Here, we use large-scale unsupervised word clustering and new lexical features to make our research more accurate. In [20], the authors released a new version of datasets that helps to solve the problem of part-of-speech in online conversational text. These clusters and annotation guidelines are available at http://www.ark.cs.cmu.edu/TweetNLP. By using these features, we can achieve state-of-the-art tagging results on twitter tasks [20]. As known, the tweets are the sentiments or intentions of people. Sometimes, they use some non-alphanumeric characters while typing these tweets. Indeed, these characters need to be eliminated or removed. To solve this issue, we use a new Twitter part-of-speech tagger building proposed by Olutobi et al. [23].

2.2.3 Tagger

This tagging model is a Markov Model of first order entropy. Decoding and training are extremely efficient for this model [24]. The tag probability is y _t that is conditioned to the input sequence x and y_t - 1. p ( y t = k | y t - 1 , x , t ; β ) ∝ exp ( β yt - 1 , k ( trans ) + ∑ j β j , k ( obs ) f j ( x , t ) )(1)

The transition features used for each label pair and base observation features are extracted from token t and neighboring tokens. A Viterbi algorithm {O (|x| K2)} is used for prediction [25]. Here, K is the number of tags. For t = 1... |x| : y t ˆ ← arg max k p ( y t = k | y t - 1 ˆ , x , t ; β )(2)

In training, for a tagged tweet (x, y), the model is the summation over the y _t (observed token tags), each condition on observed previous tags and the tweets being tagged. ℓ ( x , y , β ) = ∑ t = 1 | x | log p ( y t | y t - 1 , x , t ; β )(3)

The parameters L1-capable variant of L-BFGS and β [26, 27] are optimized: arg min β - 1 N ∑ 〈 x , y 〉 ℓ ( x , y , β ) + R ( β )(4)

The number of tokens and total ranges over tagged tweets (x, y) are represented by the N. Elastic net regularization of [28] here is the combination of L₁ and L₂ penalties, note that j corresponds to the indexes over the features. R ( β ) = λ 1 ∑ j | β j | + 1 2 λ 2 ∑ j β j 2(5)

Finally, to eliminate noisy or irrelevant features, a small value of L₁ penalty should be used.

Basically, a linear transformation Y = BX is generated from X⟶Y with help of non-singular matrix B.

Jacobian of Transformation DX DY with positive sign is |B|. Therefore, all differential elements are connected by the relation, dy 1 dy 2 . . . dy n = | B | dx 1 dx 2 . . . dx n(6) where dY=|B| dX, ∀B is an orthogonal matrix, |B|=1.

Now, based on two types of orthogonal matrix, the approach of transformation will take two forms.

Condition 1. If B is a complete orthogonal matrix, then X’X⟶Y’Y and ( X - μ ) ′ ( X - μ ) → ( Y - ξ ) ′ ( Y - ξ )(7) where μ indicates mean value and ξ = Bμ. This is conserving distances.

Condition 2. If B is a partitioned matrix ∀B _i is n _i  × n and ∑n _i  = n then B = ( B 1 · · B n )(8) Also B i B j ′ = 0 , where i ≠ j .

Such that sub-matrices are orthogonal to each other, but they are not orthogonal by themselves. According to equation (8), transformation is rewritten as, Y 1 = B 1 X , . . . . . . . , Y k = B k X(9) where Y₁, . . , Y _k are exclusive subsets of new variables. As per transformation of Quadratic Forms X ′ X → Y 1 ′ C 1 Y 1 + . . . . . . . . . + Y k ′ C k Y k ( X - μ ) ′ ( X - μ ) → ( Y - ξ 1 ) ′ C 1 ( Y 1 - ξ 1 ) + . . . + ( Y k - ξ k ) ′ C k ( Y k - ξ K ) .(10) where C i = ( A i A i ′ ) - 1 and ξ = A _i μ.

Equation (10) shows the transformation with n number of splits of quadratic forms in exclusive subset of the new variables. When B is fully orthogonal and each row is orthogonal to every other row, then splitting occurs fully.

Let Q be a Quadratic form in n variable, and it is a set of n-dimensional data to be clustered, x₁, x₂, …, x _n is a homogeneous quadratic function of the variables. Such that Q = ∑ i = 1 n ∑ j = 1 n a ij x i x j = X ′ AX(11) where X is the column vector of variables. As per problem, sample data Q is n-dimensional Q ={ x₁, …, x _j , …, x _k  }. Each datum x _j  (1 ≤ j ≤ k) is an n-dimensional point.

x j = ( x j 1 , . . . . . . . , x ji , . . . . . . x jn ) ∀ 1 ≤ i ≤ n .

Assume that D is a hypercube and also assume that all data point x _j  (1 ≤ j ≤ q) belong to D. Therefore, D = ∏ 1 ≤ j ≤ n s i t i [ a j ≤ min 1 ≤ j ≤ q ( x ji ) , b j > max 1 ≤ j ≤ q ( x ji ) ](12)

Each interval [s _i , t _i ] is split into length of l _i . Therefore, total number of intervals is P _i . The partition of each [s _i , t _i ] is defined as [ s i t i ] = ⋃ 0 ≤ m ≤ p i [ s i + ml i , s i + ( k + 1 ) l i ](13) where l i = t i + S i p i . Partition P is defined as: P = ∏ 1 ≤ i ≤ n P i

This strips over the domain D. Each slice P _r  (1 ≤ r ≤ p) is defined as: P r = ∏ 1 ≤ i ≤ n [ α r , i , α r , i + l i ](14) where α_r,i = s _i  + P_r,il _i and 0 ≤ P_r,i < P _i

The centroid of each slice P _r is an n-dimensional data point x _r . x r = ( x r , 1 , . . . . . x r , i , . . . . . x r , n ) And x r , i = α r , i + l i 2 ∀ 1 ≤ i ≤ n(15)

The partition P points x _r  (1 ≤ r ≤ p) defined set Was a sampling data space in a discrete fashion. Every initial data point x _j  (1 ≤ j ≤ k) belongs to one slice of the domain D, and the data is connected with centroid of this slice. Thus, each data is connected with one point of W.

The clustering method is obtained via Brown clustering [29] for a long set of unlabeled tweets. This algorithm divides words into a set of 1000 clusters. Brown clustering generates effective features in addition to that it provides variants that are much better than some other older models and comparisons according to the study in [30]. Therefore, this algorithm makes easier to scale a big amount of data. Let us see an example: A 1 - - - Nt n 0 t nottttt _ n ot _ A 2 - - - / / u / / you ( you iyou A 3 - - - wh 0 ehowhodeho

Here, A1 represents the same cluster, in which few challenging twts are given. A1 Cluster represents the word “not”, A2 cluster represents the word “you” and A3 cluster is used for the word “who”. By using this approach, challenging words can be easily modified and used for our analysis. Note that, sometimes, users can use some acronyms like “prbly” that represents the “probably”. These kinds of acronyms can be decoded by using this clustering approach.

Table 5 shows an example of paths for clusters that can be used for most frequent words. Here, the cluster path B1 shows the string of some challenging words, B1 represents the cluster for the word “always”. Another cluster B2 represents the word “gone”, B3 corresponds to the word “who” and finally B4 indicates the word “I”. This approach is very useful for sorting the challenging words and creating a set of data for our analysis.

As known, the tweets are the sentiments or intensions of people [31]. Sometimes, they use some non-alphanumeric characters while typing these tweets. Thus, these characters need to be eliminated or removed. To accomplish data cleaning process, the following steps are playing an important role.

Step 1. Converting Tweets to lowercase letters.

We convert these tweets to lower case letters and then use them at our database. Some examples of these tweets are shown in Table 6.

Step 2. Stop words need to be removed.

For data cleaning process, the stop words (such as so, is, its etc.) need to be removed from the recorded tweets. Some examples of these tweets are shown in Table 6.

Step 3. Explain abbreviations.

As it is in notice that many social media users mostly used the abbreviations in their tweets instead of full meaning. These abbreviations have to be clearly explained for data cleaning process. Some examples of these tweets are shown in Table 6.

Step 4. Adopt clustering method

Data clustering is very necessary to clean the data recorded from social media. Clustering method has been explained in section 2.2.6. Some examples of these tweets are shown in Table 5.

Step 5. Remove emojis and special character.

For the data cleaning process, emojis and special character have to be removed from the recorded tweets. Some examples of these tweets are shown in Table 6.

This converting process provides us to obtain an authentic and accurate data sheet. Thus, by using the same methodology, the data can be collected for different locations and used for a comparative analysis with the real crime data for different locations.