Information entropy based event detection during disaster in cyber-social networks

Abstract

The demand for Cyber Social Networks has increasingly become the main source of information propagation due to the rapid growth of micro-blogging activity between socially connected people. The process of detecting disaster events, in huge volumes, on fast-streaming platform is quite challenging. In this paper, an information entropy based event detection framework is proposed to identify the event and its location by clustering relatively high-density ratio of tweets using Twitter data. The Shannon entropy of target users, location, time intervals and hashtags are estimated to quantify the dissemination of events as “how-far about” in real- world using entropy maximization inference model. The geo-tagged (spatial) tweets are extracted for a specified time period (temporal) to identify the location of an event as “where-when about”; and visualizes the event in geo-maps. The evaluation parameters of Entropy, Cluster Score, Event Detection Hit and False Panic Rate during four major disaster events are identified to illustrate the effectiveness of the proposed framework. The retweeting activity of the Twitter user is classified as human signatures and bots. The experimental outcome determines the scope and significant dissemination direction of finding events from a new perspective which demonstrates 96% of improved event detection accuracy.

Keywords

Cyber-social networks event detection geo-tag spatiotemporal Shannon entropy

1 Introduction

In recent years, Cyber-Social Networking (CSN) platforms like Twitter, Facebook, Sina Weibo and Tumblr have encapsulated huge chunks of information comprising insightful user opinions about various events [1]. The problem of evaluating unstructured data in user generated content has ubiquitous applications, including the identification of conversation topics and abnormal events [2]. The online events that match with actual real-time events are of diverse temporal and spatial scales, particularly disasters like earthquakes, floods, landslides, etc. to deliver prior alerts, support, and immediate recovery [3, 4]. The CSN act as communication tool and double up as informative podiums that catch real human voices reaching out during disasters. However, the information about ongoing events and sensitive issues that remain unnoticed. Providentially, the CSN is used as new additions to measure the impact of disaster events due to wide usage of smart phones [5]. The geographically linked CSN data containing geo-tags is accepted as a trustworthy source for detecting disaster and provides wide range of accessible services [6]. In 2015, a study explored the retweeting activity geographically from collective Twitter population around the globe during Hurricane Sandy [7] disaster. They used feature-rich classifiers for determining the microblogs which are most relevance to specified event [8]. The events could be recognized as abnormal spikes in activity [9] by monitoring the normal flow of microblogs using change point detection approaches [10]. It determines the intensity of the sensitive and abnormal event that was deciphered in the course of fluctuations in content. The annotation scheme [11] for identifying relevant tweets after disasters usually carried out using keyword-based collections without location features. The tweets were semantically examined to detect the overlapping event communities by relative density-ratio estimation in FuSeO model [12]. The community clusters were created using keyword-based ontology which consist of most appropriate hashtags and keywords. The challenge of quantifying the in-depth traces of event information dissemination is resolved in our proposed work by introducing new optimization parameters. The information entropy based framework is proposed in this paper which uses the metadata of a tweet to detect disaster event diffusion and comprehend online user behaviour during crises event. The proposed framework is specific to the Twitter platform by considering the retweeting activity of online users to address the challenges involved.

This paper is organized as follows. The existing event detection models are discussed in Section 2. The Shannon entropy maximization inference model to quantify event is postulated in Section 3. The proposed event detection framework is presented in Section 4. The experiments and results are discussed in Section 5. The conclusion and future work is discussed in Section 6.

2 Related Work

The various methods to evaluate CSN data to build supportive frameworks and models for online event detection have been proposed over time. In early 2015, a system for sensing complex social systems was introduced which use Bluetooth-enabled mobile telephones data to measure information using Shannon entropy threshold method [13]. The social patterns were recognized across different context namely regular user activity and user connectivity to detect event locations profoundly. A graph based entropy models was deployed in [14] to study information flow in a social entity using Part-of-speech tagger. It focused on node entropy in the social entity graphs with random error pruning. The event detection studies in [15, 16] using Twitter data explored the critical survey of all possible event detection methodologies. It classified the event detection techniques into specified or unspecified and the task of detection as retrospective or new event detection, methods of detection as supervised and unsupervised. The tweets with its meta features were combined with ‘retweeting’ dynamics [17] to examine the behaviors of Twitter users was considered by Ghosh quantification model which merely based on tweet content. The activities were categorized using user generated content namely tweets and retweets. It investigated the process of spreading specific pieces of information using keyword sequence and the user’s viral diffusion behaviors which is used for a comparison with our work. The Ghosh model categorized the tweeting and re-tweeting activity in Twitter as newsworthy information, robotic activity advertising and promotion, campaigns.

The event was identified using topic summarization on microblog features [18] as a bipartite graph. The bipartite graph was generated to capture the relationship between two topics discussed during simultaneous time period to summarize events. The node entropy in the bipartite graph was used with consistent event pair to merge events. In [19], the Twitter hashtags are utilized to indicate events based on which each user node (contributed authors) entropy was mined. The entropy based regression model [20] was used for identifying trending topics which follows binomial distribution.

The popular Sina Weibo compute Cosine similarity to evaluate similarity of the influential corpus [21]. The burst corpus which repeatedly occur in the batches of incoming messages were closely related with clusters. The methodology resulted in 75% accuracy which is relatively low than the proposed approach. The temporal dynamics, including lifecycles and tipping-points, of tweets’ popularity were used for event detection. The Mention-Anomaly-Based Event Detection (MABED), that depend simply on tweets and the cluster score to assess the magnitude of event impact over the crowd [22]. MABED approach detected sub-events by performing spatial and temporal random indexing on textual descriptors.

In the field of disease outbreaks, the CSN agent [23] estimated the information gain in the event location, which evaluates potential time of influenza affected zones using keyword entropy. The study syntactically parsed tree and classified to detect disease reporting sentences in Twitter. An unsupervised Bayesian model [24], extracted the structured representations of events in the form of quadruples namely entity, keyword, date, location and further categorized the extracted events into event types. A general query language models [25, 26] extracted events through a series of machine-learned approaches which is adopted in our work. For our experimentation, the event detection process considers Twitter parameters namely location co-ordinates, place check-ins, geo-tags are used for spatial information which is used in our work.

3 Information entropy based quantification model

The proposed work deals with Twitter user activity which is transformed into an information theoretic method by calculating the information entropies to detect events. The entropy maximization inference quantification model is shown in Fig. 1. It takes Twitter tweets and Re-tweets as input. The process uses the objective function and entropy parameters; namely, Hashtag Entropy (E_H), Time Interval Entropy (E_T), User Entropy (E_U), and Location Entropy (E_L) to detect events. The motivation towards the event detection problem is two-fold: first, maximizing information entropy which minimizes the overall prior knowledge established in the distribution of tweets; second, the proposed inference model which provides maximum entropy value over a period of time. The final convex-constrained Shannon entropy E optimization formulation of given function as follows, $Max (H (E_{H}) + H (E_{U}) + H (E_{T}) + H (E_{L}) + \partial)$ (1) $\partial = μ_{E} + λ X$ (2) where λ = 1, μ_E = 1 which are used as tuning parameters with constraints H (E) ≠0, ∂≠0. It takes the values to 1, because the entropy cannot take negative or non-zero values respectively. With these constraints, the following two prior knowledge is added with the proposed for maximizing the inference.

Tweeters and Retweeters

CSN users are strongly correlated to carry information to extremes in terms of tweets and retweets. The tweets disseminate an event at any time interval from a location.

Entropy Distribution

The entropy distribution given in Equation (1) and Equation (2) that measures the propagation of the trending event from source. The users impulsively create more influence because of network size and density, as well as a tendency on the part of the followers to retweet posts whenever an event occurs.

Fig.1

Entropy Maximization Inference Model.

3.1 Time Interval Entropy

The set of Twitter posts of a user u ∈ U with a topic word is represented as P_u,word. Let n_u,p be the total number of posts by users. Assume b represent the type of post where b = 1 indicates normal tweets, b = 2 retweets (RT) and b = 3 replies (RE). The time interval of a particular user u, who posts ‘p’ with the topic word ‘W’ from (t - 1) to t, is expressed as Δt_u,p,t where Δt_u,p,0 = 0, because the first post is assumed to be at the starting time. The regularity of the traces of tweets is measured using the time interval entropy on topic words. The frequency of the word (W) in a given time-interval Δt_u,p,t is calculated as follows. $N_{u, p, t} = \frac{Δ t_{u, p, t}}{r} \supset W$ (3) where r = 1, 2 . . . , n_u,p contains keywords W_i to determine the tweet arrival rate. The time interval entropy on topic words is given as follows.

$E_{T} = - \sum_{k = 0}^{m_{u, p}} p_{T} (N_{u, p, t}) \log (p_{T} (N_{u, p, t}))$ (4)

3.2 Similar user entropy

The measure of similar user distribution on topic words represent similar user entropy. Let the random variable D represent a distinct user in trace T_i with all the possible values {d₁, d₂,. . . , d_n}, The number of retweets from user U_i be found in trace T_i, whereP_F represents the probability density function of D, such that P_F (d_i) gives the probability of a retweet being generated by user U_j, and the frequency of user U_i ∈ U, with post M to user U_j ∈ U.

$E_{U} = - \sum_{i = 1}^{n} p_{F} (n_{U_{i}, M, U_{j}}) \log p_{FT} (n_{U_{i}, M, U_{j}})$ (5)

3.3 Hashtag entropy

The hashtag entropy is a measure of significant hashtags in a set of tweets, defined as hashtag entropy. Consider N tweets containing the hashtag (HT) used frequently by K number of users generated g₁, g₁,. . . g_k times relevant to a particular event, $E_{H} = - \sum_{i = 1}^{n} \frac{g_{i}}{N} \log (\frac{g_{i}}{N})$ (6)

3.4 Location entropy

The frequency of the location geo-tagged by diversified users is captured using location entropy. Let {l : l₁, l₂,. . . , l_n} be the set of locations which are geo-tagged in tweets by a set of users {u : u₁, u₂,. . . , u_n} is formulated as $E_{l} = - \sum_{u \in u_{l}} \frac{c_{l, u}}{c_{l}} \log (\frac{c_{l, u}}{c_{l}})$ (7) where u_i is the set of distinct users in location l and $\frac{c_{l, u}}{c_{l}}$ is the fraction of tweets location that belongs to user u.

3.5 Tweet-cluster score

The cumulative score of the tweets that discuss about the same event is computed using score of keyword (KS), tweet score (TS) and cluster score (CS) as follows. $K S (w) = {\begin{cases} v a l u e (w), \forall w \\ 0, o t h e r w i s e \end{cases}$ (8)

The value of each word relevant to a disaster is assigned to a value in dictionary. Based on word score, the tweet score is computed as follows. $TS (w_{1}, w_{2}, . . . w_{n}) = min (\frac{1}{n} \sum_{i = 1}^{n} KS (w_{i}), 1)$ (9)

The tweet score is calculated for each keyword which is normalized to interval 0 to 1. Moreover, the score of cluster is computed as follows. $CS (t_{1}, . . ., t_{n}) = \sum_{i = 1}^{n} TS (t_{i})$ (10)

The higher value cluster score, the more the cluster leads to be informative event. Therefore, the entropy is calculated for each cluster which estimates the quantum of information spread from each cluster.

4 Proposed event detection framework

The proposed work identifies the disaster-related event based on the tweet corpus and evaluates the dynamics of the tweeting activity during the disaster. The framework performs change point detection, burst corpus identification, followed by location detection. The tweets collected from Twitter are stored in a tweet repository, and accessed iteratively to retrieve the most recently collected tweets. The system is pre-configured with corpus library for referencing burst words. The usage history of the word w, so the system recognizesthe non-bursty and bursty occurrence of the word w over specified time interval. During pre-processing, the most relevant tweets arfitered by eliminating noisy and irrelevant tweet corpuses. Further, stop words, stemming, and tokenization, including lookups is performed using WordNet dictionary [27]. The keywords in the identical subset appear in the similar topic of discussions. Instead of detecting the most similar words, the system identify the event clusters that are most similar to preceding clusters in order to detect an on-going event. The proposed event detection framework is shown in Fig. 2.

Fig.2

Proposed Event Detection Framework.

The tweet is converted into vector format using bag-of-words vector space approach. After pre-processing, the burst corpus are identified the meta-features namely text, location, hashtag, time, url, and images. The tweets assign to a value, using a set of 5 distinctive word feature: words, tags, links, mentions and users. Each cell value in the matrix from 0 to 1 reflects the importance of the particular term in the specific tweet. The Document Incidence (DI) is the number of tweets in which the word appears. The Global Frequency Rate (GFR) is the total number of times the word appears within the tweet dataset between the First Interval (FI) and Last Interval (LI). The tweets assign to a value, each cell value in the matrix from 0 to 1 reflects the importance of the particular term in the specific tweet. Temporal Burst Ratio (TBR) is obtained by calculating the Z-score of the term between intervals to detect change point detection.

Local-Global (LG) Ratio of the term is computed as the ratio of local (interval) frequency to global document frequency. Event Link Ratio (ELR) is the ratio between the total amount of tweets containing links related to an event and the total number of tweets for that interval. The lesser value of ELR, the fewer the links between events and, vice versa for highly-linked events with a value nearing unit value. A high Z-score means that the word is unusually more frequent and therefore likely to be a good descriptive word diffused, or else that rare topics (high homogeneity index) are being discussed within the interval from event location. It results in determining the behavioural homogeneity among Twitter users. Finally, the event clusters are formed using K- means clustering using event detection algorithm.

Algorithm 1

Algorithm: Event Detection Algorithm

Input: TW: a set of tweets meta data

Output:

{E_{i}}_{i = 1}^{n}

Event clusters

Begin

1: Compute the TBR for a significant corpus in a time interval (t, Δt) in all pairs of tweets (T₁,. . . T_n) to find change-point detection.

2: Extract location from Geo-tagged location tweets.

3: Compute k-means clustering to find similar tweets with maximum cluster score.

4: Calculate entropy parameters: E_H, E_T, E_U, E_L

5: Evaluate Event Detection Hit P_hit and False Panic Rate P_Fp.

6: Compute the ELR to retain meaningful event clusters

{E_{i}}_{i = 1}^{n}

End

The event clusters are formed using k – means clustering based on the similarity measure. The similar users are clustered using the Jaccard Coefficient Index, which is used to find the distance between user pairs. The resultant k-means clusters are assigned a score called Cluster Score in Equation (10), to retain meaningful event label using the dataset tabulated in Table 1.

Table 1

Summary of Disaster Events Identified using Four Datasets

#	Event Dataset	Period	Location	Geo-tagged Tweets	Avg. No. of tweets	Avg. No. of Retweets	HOM
1	Flood	01-17 Dec 2015	Chennai	1526	4799	860	91%
2	Flood	11-20 July 2016	Assam	1590	5189	561	85%
3	Landslide	3-10 June 2016	Uttarakhand	2879	4181	327	83%
4	Earthquake	4 Jan 2016	Manipur	3571	4875	549	86%

5 Experiments and Results

5.1 Online User behavioral analysis

Tweets with the highest burst ratio for a specific time interval are filtered. For the experiments, a training dataset collected using Twitter API comprises a total of around 20,000 tweets captured during various disasters between 2015 – 2016, in which 16,000 tweets are valid ones that identify 3 disasters namely Flood, Landslide and Earthquake events (Table 1) tweet arrival rate λ= 100 tweets per minute. The attributes considered for the analytics are Twitter User ID, Time, Time-zone, Tweet, Geo-location, URL, retweet count, replies, followers. After pre-processing the tweets, word indexes are constructed to compute the statistics model for the distinct parameters. Similar users with similar content are clustered using the Jaccard Coefficient Index [28]. The TF-IDF is computed for 3-grams since uni-gram approach cannot significantly distinguish the event occurrence Initially, the tweets are clustered as similar users who retweeted similar tweets. Homogeneity Index (HOM) parameter is the percentage value in which tweets within a particular interval use the same keywords. These categories are used as class labels and the associated words in tweets are trained for each class. The overall statistics of word indexes are given in Table 2. It represents the usage of burst keywords and location names to detect the event type as ‘flood’ and its location as ‘Assam’.

Table 2
Tweet Word Statistics (Dataset: Assam Flood Tweets)

Word DI GFR Z Score LG TBR HOM(%) ELR

Assam 698 554 4.57 99.1 94.5 87.1 0.89

flood 577 493 3.74 95.11 81.1 79.5 0.78

affected 415 425 4.99 97.25 87.7 86.4 0.76

rescue 411 345 3.41 81.27 71.4 81.2 0.70

situation 475 367 5.87 98.45 81.0 88.9 0.61

government 234 145 3.12 84.54 78.4 67.1 0.75

relief 188 121 2.41 75.68 69.4 76.1 0.78

IndiaNews 186 147 3.25 89.10 81.4 65.1 0.68

waterlog 214 139 4.11 68.19 61.2 67.9 0.77

Word	DI	GFR	Z Score	LG	TBR	HOM(%)	ELR
Assam	698	554	4.57	99.1	94.5	87.1	0.89
flood	577	493	3.74	95.11	81.1	79.5	0.78
affected	415	425	4.99	97.25	87.7	86.4	0.76
rescue	411	345	3.41	81.27	71.4	81.2	0.70
situation	475	367	5.87	98.45	81.0	88.9	0.61
government	234	145	3.12	84.54	78.4	67.1	0.75
relief	188	121	2.41	75.68	69.4	76.1	0.78
IndiaNews	186	147	3.25	89.10	81.4	65.1	0.68
waterlog	214	139	4.11	68.19	61.2	67.9	0.77

The observations made for the dynamics of retweeting activity are captured in four distributions: time interval, hashtag, location, user distribution and similar user. The arrival rate (time/hour) and the frequency of retweeting activity is recorded using Chorus TweetVis [29]. It shows that the popular retweeting frequency of Twitter accounts such as India News (online news posting on Twitter), BarkhaDutt (a popular India journalist and columnist on NDTV) and PTTV Online (a popular Tamil television online) is examined in our experiments is shown in Fig. 3. The human signature on Twitter with inter-arrival rates of different lengths are likely to be equally distributed, where India News, BarkhaDutt and PTTV online tweeted on Twitter with a significant distribution of retweeting frequencies by their followers. The automated activity of bot account namely Bloombrg Newsish shown in Fig. 4, retweet at regular intervals of time, culminating in an isolated maximum peak in distribution. Figure 5 represents the retweeting activity by distinct users and hashtags which shows the maximum retweets of location based hashtags. The time line analysis of tweet distribution for Assam flood-related tweets from July 1, 2016 to August 21, 2016 is shown in Fig. 6 which records the tweets spiked out with maximum homogeneity.

Fig.3

Retweeting Activity of News Channels (Dataset #1).

Fig.4

Retweeting Activity of Bot (Dataset #1).

Fig.5

Retweeting activity by Distinct Users and Hashtags (Dataset #1).

Fig.6

Timeline Analysis (Tweet Arrival) during Assam Flood Event.

The human signature on Twitter with inter-arrival rates of different lengths are likely to be equally distributed, where India News, BarkhaDutt and PTTV online tweeted on Twitter with a significant distribution of retweeting frequencies by their followers (Fig. 3). The retweeting activity during Assam flood dataset shows the actual number of tweets spiked, out of which only 40% of tweets contains informative content (URLs, hashtags, geo-tag) and signifies 89% homogeneity of similar users who uses similar set of keywords to represent an event. In addition, the tweets are classified as two classes: Class A (without multimedia content) and Class B (with multimedia content) using naïve bayes classifier. The entropy values from Table 3 show that users retweet more multimedia content such as images, videos, and audio links rather than raw text tweets without multimedia content. The procedure of spreading specific pieces of information, including corpus sequence and viral diffusion behaviours, is used for a comparison with our work. The activities are characterized as two distinct categories of retweeting activity on Twitter: robotic activity and newsworthy information. The arrival rate (time/hour) and the frequency of retweeting activity. In contrast, Bloombrg Newsish (bot account) [30] shows an automated retweeting frequency at regular intervals of time, culminating in an isolated maximum peak in distribution which correlate with the entropy maximization inference model. The most significant parameters for burst detection are identified to the tweet arrival rate λ, and the number of tweets N containing word.

Table 3

Evaluation of Entropy (Class A Vs Class B)

User Name	Time Int. Entropy		RT-Time Int. Entropy		User Entropy		Hash tag Entropy		Similar User Entropy		Location Entropy
CLASS	A	B	A	B	A	B	A	B	A	B	A	B
India News	1.75	2.04	3.41	5.95	5.12	7.51	2.1	4.26	1.98	3.2	2.1	3.4
BarkhaDutt	2.89	4.02	6.1	8.04	3.20	5.21	2.45	4	5.26	7.2	3.1	4.7
BloombrgNewish	0.04	0.12	0.00	0.20	0.00	0.00	0.00	0.00	0.00	0.00	0.41	0.7
PTTVOnline	2.45	4.25	1.04	3.56	2.74	3.84	0.89	1.25	0	1.36	2.74	3.8

5.2 Location Extraction and Prediction from Geo-Tagged Tweet Content

Geo-tags are extracted from tweets using the Geo-filter option offered by the Twitter Streaming API [31]. The user’s current location is assumed to be the ground truth for the event location occurring in the most recent time. In some cases, there is a chance for the event to have occurred at a particular location and the user who tweets about the event might in different location. The geographical information which is extracted using Named Entity Recognizer (NER). In order to obtain the geographical coordinates, the output of NER for specific location is Geo-coded. Geo-tags associated with trending hashtag frequency increases in events and trends, reflecting obsessed user attention concerning the area of the event. The information is investigated in a different way, including what and whose content the individuals chose to broadcast. The Twitter-based relationships are examined if they are newly formed because of the disaster. It is found that the general population in the tactic of the disaster support is based on retweeting local tweets and those with locally-significant data. The location is extracted from the tweet using the Stanford NER [32] and the Geonames API [33] is used to carry out the Geo-parsing and Geo-coding processes.

The local terms for next level of geographic places are turned out to be difficult. Hence, a heuristic for selecting local terms is considered. First, the word statistics for the extracted location and the count of Twitter users who frequently used those words in their tweets is calculated. The threshold is fixed for words that frequently occur in tweets by at least the N number of the people in that location. The value of N is an optimally chosen parameter, N = 10, i.e., at least 10 users should be found frequently using the term ‘T’ from a particular location. Secondly, the geographic bounds of event location are determined by conditional probability of the users (who used the said term in relation to the event), their location and the location of the event are compared. The visualization of specific location is plotted on a map with its label, as shown in Figs. 7(a) and 7(b). Finally, the difference of conditional probabilities between user location and event location is calculated and compared with a threshold. If the comparison result is successful, it ensures that the word used by the user is most relevant to an event at a particular location. The extraction of the location of the tweet message is shown in Table 4, with the maximum likelihood of the location where the user at event place.

Fig.7

Visualization of Event using Geo-Coded Tweets.

Table 4

Location Estimation using the Maximum Likelihood (user and event location)

User ID	Location Keyword	Location Type	Likelihood		Latitude	Longitude
			User	Event
1	Guindy	India, Tamil nadu	0.94	0.97	N 13°0^′29^′	E 80°13^′10^′′
2	Varadharajpuram	Tamil nadu, Chennai	0.84	0.97	N 13°0^′29^′	E 80°13^′10^′′
3	Kedarnath	India, Uttarakhand	0.83	0.89	N 30°47^′42^′′	E 79°4^′10^′′
4	Nagaon	India, Assam	0.75	0.94	N 26°21^′1^′′	E 92°41^′32^′′

5.3 Evaluation methodology

The effectiveness of the proposed framework is evaluated concerning the safety-critical in terms of the probability of event detection hits P_hit, and false panic rates P_Fp. Effective event detection and disaster response establishes a good balance between P_hit and P_Fp. The P_hit and P_Fp are computed as $P_{hit} = \frac{number of eventdetection successes}{number of targets}$ (11)

$P_{Fp} = \frac{number of false panic alters}{number of non - targets}$ (12)

The tuning parameters, namely, keyword similarity η = 5, temporal.τ = 0.61 minutes, locality ξ = 10 and hashtag ω = 5 are fixed. The clustering algorithm is applied in 3 scenarios by consolidating these major features (i) Location+Time (LT), (ii) Location + Time + Tweet Content (LTC), and (iii) Location + Time + Content + Hashtag (LTCH). The Cluster Score (CS), Event Detection Hit (EDH), and False Panic Rate (FPR) is measured. The results in Table 5 show that the LTCH gives a high cluster score in Equation (10) through better accuracy (EDH) in detecting events with a minimal FPR. The CS and EDH values in the proposed approach is relatively high compared with Qian method [21], Guille approach [22].

Table 5

Experimental Result- Cluster Score (CS), Event Detection Hit (EDH), and False Panic Rate (FPR)

Algorithm\Features	Location + Time (LT)			Location + Time + Content (LTC)			Location + Time + Content + Hashtag (LTCH)
	CS	EDH	FPR	CS	EDH	FPR	CS	EDH	FPR
Qian Method [21]	0.70	0.61	0.19	0.76	0.67	0.23	0.78	0.72	0.15
Guille Method [22]	0.73	0.75	0.14	0.79	0.80	0.45	0.79	0.77	0.10
Proposed Approach	0.81	0.84	0.09	0.81	0.87	0.10	0.91	0.96	0.07

The hashtags are categorized in two ways – with entropy less than 0.5 (Entropy A – with multimedia content) and equal to or greater than 0.5 (Entropy B - without multimedia content). The time decays, the probability of relevance hash tags with higher entropy remains to trend, whereas the low entropy hash tags fade away with time. It explains that the real human activity turns up to show high entropy value, whereas robotic (bots) activity turns down to show very low entropy. The entropy values are estimated for all events with respect to Location, User, Hashtag and Time interval using Equation (4), Equation (5), Equation (6) and Equation (7), which is shown in Fig. 8 for all 4 disaster datasets. It evidents that during disasters, users tend to reach CSN to tweet and retweet, and consistently embedded with the names of popular users/activists, hashtags, URLs, mentions, and locations.

Fig.8

Entropy values for Event.

The probability of relevance hash tags with higher location entropy and hashtag entropy remains to trend, whereas the low entropy for user and time interval which predicts important news sources of disasters on Twitter. It quantifies how vital the hashtags and location specific geo-tags are used for disaster news diffusion. The scikit-learn using python [35, 36] and machine learning library [37] is deployed to perform clustering of 20k samples into k = 15 clusters with k-means for 4 events.

The validation of the proposed framework is evaluated using Kappa co-efficient [34], the achieved agreement is 91%. It is evident that the proposed quantification model results in high accuracy in detecting flood-related disasters. The proposed framework is tested that are able to identify an event in diverse temporal and spatial location of interest. The events related to disasters are detected that are influential against the uncertain and noisy information present in the data between different time intervals. The experiments are conducted on a desktop computer with modest hardware (one i7 – processor, 16 GB memory) to collect tweet datasets of size 1 GB with the number of tweets n = 16000, and requiring non-linear time to compute the word index statistics and perform machine learning algorithms.

The cumulative entropies of popular, typical users are calculated using the quantification method and the proposed method. The proposed quantification method gives higher quantification accuracy when compared with the Ghosh quantification method [17] which is shown in Fig. 9. Meaningful event clusters such as Event 1(Assam Flood2016), Event 2 (Uttarakhand Landslide 2016), Event 3 (Chennai Flood 2015), and Event 4 (Manipal Earthquake) and their accuracies are examined to ascertain the differences between the two methods. The Ghosh quantification model results in a moderate accuracy of 73%. whereas the proposed quantification method shows up to 96%. The scalability of the proposed framework depends on the runtime of the maximum entropy optimization parameters.

Fig.9

Performance analysis of proposed work.

6 Conclusion and future work

The framework for event detection in real-time CSN is proposed and the levels of information spread is quantified using Shannon Entropy for a time-bounded Twitter activity. The entropy maximization inference model exhibits the demand of transforming CSN into information spreading platform. The proposed framework detects four major events during 2015-2016, which clusters the Twitter data, showing an event detection accuracy of 96%. which is acceptably high. The results demonstrate the advantage of adopting Shannon entropy maximization inference model to quantify event in the CSN. Based on the retweeting activity, the Twitter users are classified as humans and bots. The integration of hashtag and location parameter with keywords generated by users significantly improves the event detection accuracy to the top hierarchy. It establishes the importance of quantifying information disseminating features involved in Twitter data.

In future work, the framework can be extended to support the tracking of disaster event in CSN with respect to the user’s native language in big data enabled ecosystem to support the scalability. The proposed approach is well suited to other sources of Geo-social content with substantial modifications.

References

Sakaki

, Okazaki

, Matsuo

Earthquake shakes Twitter users: Real-time event detection by social sensors, Proceedings of the International conference on World wide web, 2010, USA.

Imran

, Castillo

, Diaz

, Vieweg

, Processing social media messages in mass emergency: A survey, ACM Computing Surveys47(4) (2015), 67–72.

Gao

, Wang

, Padmanabhan

, Yin

, Cao

, Mapping spatiotemporal patterns of events using social media: A case study of influenza trends, International Journal of Geographical Information Science32 (2018), 425–449.

Pei

, Gong

, Shaw

S.L.

, Ma

, Zhou

, Clustering of temporal event processes, International Journal of Geographical Information Science27 (2013), 484–510.

Hughes

A.L.

, Palen

, Twitter adoption and use in mass convergence and emergency events, International Journal of Emergency Management6 (2009), 248–260.

Ifrim

, Shi

, Brigadir

, Event detection in twitter using aggressive filtering and hierarchical tweet clustering, Proceedings of the Second ACM Workshop on Social News on the Web, 2014, 1–7.

Kogan

, Palen

, Anderson

K.M.

, Think local, retweet global: Retweeting by the geographically-vulnerable during Hurricane Sandy,Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, 2015, 981–993.

Stowe

, Paul

M.J.

, Palmer

, Palen

, Anderson

, Identifying and categorizing disaster-related tweets, Proceedings of the Fourth International Workshop on Natural Language Processing for Social Media, 2016, 1–6.

, Sun

, Cong

, On predicting the popularity of newly emerging hashtags in twitter, Journal of the Association for Information Science and Technology64 (2013), 1399–1410.

10.

Liu

, Yamada

, Collier

, Sugiyama

, Change-point detection in time-series data by relative density-ratio estimation, Neural Networks43 (2013), 72–83.

11.

Palen

, Anderson

K.M.

, Crisis informatics - New data for extraordinary times, Science353(6296) (2016), 224–225.

12.

Kianian

, Khayyambashi

M.R.

, Movahhedinia

, FuSeO: Fuzzy semantic overlapping community detection, Journal of Intelligent & Fuzzy Systems32 (2017), 3987–3998.

13.

Lin

, Divergence measures based on the Shannon entropy, IEEE Transactions on Information Theory37 (1991), 145–151.

14.

Shetty

, Adibi

, Discovering important nodes through graph entropy the case of enron email database,Proceedings of the 3rd international workshop on Link discovery, 2005, 74–81.

15.

Deng

, Qiao

, Li

, Zhang

, Wang

, An Overview of Event Extraction from Twitter, International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2015, 251–256.

16.

Atefeh

, Khreich

, A survey of techniques for event detection in twitter, Computational Intelligence, 31 (2015), 132–164.

17.

Ghosh

, Surachawala

, Lerman

, Entropy-based classification of ‘retweeting’ activity on twitter, Neurocomputing23 (2011), 234–242.

18.

Long

, Wang

, Chen

, Jin

, Yu

Towards Effective Event Detection, Tracking and Summarization on Microblog Data, In: Wang

, Li

, Oyama

, Hu

, Qian

(eds) Web-Age Information Management, WAIM, 2011 Lecture Notes in Computer Science, Springer68 (2011), 652–663.

19.

Cui

, Zhang

, Liu

, Ma

, Zhang

, Discover breaking events with popular hashtags in twitter, Proceedings of the 21st ACM international conference on Information and knowledge management, 2012, 1794–1798.

20.

Benhardus

, Kalita

, Streaming trend detection in twitter, International Journal of Web Based Communities9 (2013), 122–139.

21.

Qian

, Ma

, Xia

, He

, Xu

, Zhou

, Towards modeling popularity of microblogs, Frontiers of Computer Science7 (2013), 171–184.

22.

Guille

, Favre

, Event detection, tracking, and visualization in twitter: A mention-anomaly-based approach, Social Network Analysis and Mining5 (2015), 1–18.

23.

Gao

, Wang

, Padmanabhan

, Yin

, Cao

, Mapping spatiotemporal patterns of events using social media: A case study of influenza trends, International Journal of Geographical Information Science32 (2018), 425–449.

24.

Zhou

, Chen

, Zhang

, He

, Unsupervised event exploration from social text streams, Intelligent Data Analysis21 (2017), 849–866.

25.

Cheng

, Wicks

, Event detection using Twitter: A spatio temporal approach, PloS one9 (2014), e97807.

26.

Dork

, Gruen

, Williamson

, Carpendale

, A visual backchannel for large-scale events, IEEE Transactions on Visualization and Computer Graphics16 (2010), 1129–1138.

27.

Miller

, WordNet: An electronic lexical database, MIT Press, 1998.

28.

del Moral

M.J.

, Chiclana

, Tapia

J.M.

and Herrera-Viedma

, A comparative study on consensus measures in group decision making, International Journal of Intelligent Systems5 (2018), 1–19.

29.

Brooker

, Barnett

and Cribbin

, Doing social media analytics, Big Data & Society3 (2016).

30.

Ferrara

, Varol

, Davis

, Menczer

, Flammini

, The rise of social bots, Communications of the ACM59 (2016), 96–104.

31.

McFedries

, Twitter tips, tricks, and tweets, Wiley Publishing, 2010.

32.

Vatant

, Wick

, Geonames ontology, Dostupné, 2012http://www.geonames.org/ontology/ontology_v3.

33.

Manning

, Surdeanu

, Bauer

, Finkel

, Bethard

, McClosky

, The Stanford CoreNLP Natural Language Processing toolkit, Proceedings of 52nd annual meeting of the association for computational linguistics: System demonstrations, 2014, 55–60.

34.

Mukherjee

S.P.

, Sinha

B.K.

, Chattopadhyay

A.K.

, Statistical Assessment of Agreement, Statistical Methods in Social Science Research, 2018, 61–68.

35.

Pedregosa

, Varoquaux

, Gramfort

, Michel

, Thirion

, Grisel

, Blondel

, Prettenhofer

, Weiss

, Dubourg

, Vanderplas

, Scikit-learn: Machine learning in Python, Journal of Machine Learning Research12, (2011), 2825–2830.

36.

Hackeling

, Mastering Machine Learning with scikit-learn, Packt Publishing Ltd, 2017.

37.

Mitchell

, Machine Learning, McGraw Hill, 1997.