Event type induction using latent variables with hierarchical relationship analysis

Abstract

The conventional approach to event extraction requires predefined event types and their corresponding annotations to train event extractors. However, these prerequisites are often difficult to satisfy in real-world applications. To automatically induct event types, most work has been devoted to clustering event triggers, where a cluster of event triggers is represented as an event type. Some works use trigger semantics, while others use co-occurrence relationships to cluster triggers. However, the clustering results of event triggers obtained by the above work are not sufficiently detailed in describing event types, making it difficult to accurately determine the corresponding event types manually. This paper proposes an open-domain event type induction framework that automatically discovers a set of event types from a given corpus. Unlike previous work on event trigger clustering, this paper takes into consideration the hierarchical relationship of event types to partition the event trigger clusters into event mains and subtypes. The framework employs a latent variable-based neural generation module and a semantic-based clustering module, the former of which obtains event trigger clusters representing the main types of events by jointly projecting the co-occurrence and semantic information of event triggers into a latent space for event type latent variable mining, and the latter of which further divides these event trigger clusters into event subtypes based on semantic information. Finally, experiment results show that, compared with the benchmark model, the ETGen-Clus can improve event type quality scores of 6.23% and 3.11% on the two datasets, respectively.

Keywords

Event extraction event type induction deep learning latent variable

1. Introduction

Event extraction from text has received considerable research attention. Traditional event extraction tasks [6,11,30] typically require a predefined event schema, which consists of event types and argument roles and can be extracted using supervised machine learning models such as deep neural networks based on manually annotated data. In practice, however, the annotation process of event schema extraction is recognized as being very resource-intensive in terms of time and money. In addition, the coverage of manually constructed schemas is usually very low, making it impossible to generalize them to new scenarios.

For the purpose of automatically inducing event types and argument roles from raw text, researchers have investigated algorithms based on ad-hoc clustering [4,14,26,29] and probabilistic generative methods [3,8,23,34] to discover a set of event types and argument roles. Generally, these approaches employ manually designed representations and rely on strong statistical assumptions. Some studies remove those restrictions using techniques such as transfer learning [7,16], GAN [33] and VAE [15] without the need to explicitly derive patterns for new event types to extract events. However, these methods still require significant annotation of a known set of types.

Previous researchers have also utilized unsupervised learning methods to alleviate the cost pressure of data annotation, such as reference [12] and reference [14] uses the semantic information of event triggers to cluster event triggers and represent the event types through event trigger clustering. More recently, reference [27] used the pre-training model to obtain a semantic feature of event triggers and their object heads and then projected the two semantic features onto the same spherical vector space for clustering. Reference [19] mentioned high consistency of word cluster represents significant topic features, and clusters of event triggers possessing significant topic and high semantic similarity can help to identify specific event types, whereas the above method mainly focuses on the semantic information of the event triggers to generalize the event types and lacks the mining of the hidden event types of the documents themselves, resulting in low consistency of the clustered event trigger clusters.

To address the above problems, this paper introduces a novel open-domain event type induction framework. Unlike the traditional event type induction framework, this paper classifies event types into main types and subtypes and divides the task into two steps: generation of event main types and clustering of event subtypes. In the process of generating event master types, in order to better mine the hidden event types in the text, the model will feed more feature information, i.e., co-occurrence and semantic information of event triggers, to the generation module in the framework to improve the coherence of event trigger clusters. Then clustering based on the semantic information of event triggers is performed to get clusters of event triggers that will be represented as event subtypes. Compared to the event type induction model that focuses on semantic information, the event type induction model in this paper greatly improves the coherence of event trigger clusters with less loss of semantic similarity. The main contributions of the work are summarized as follows:

A neural variational inference model for generating event types that combines semantic information of event triggers with text-trigger co-occurrence fully exploits the event types in the hidden space, thereby improving the coherence of event trigger clustering.

The framework exploits the potential relationship between event type hierarchies and event trigger clusters to induce more consistent and high semantic similarity in event trigger clusters to represent high-quality event types.

This paper conducted extensive experiments on real-world datasets and demonstrated that our model can induce high-quality event types.

The structure of this paper is as follows: Section 2 gives an overview of related work. Section 3 introduces the ETGen-Clus framework. Section 4 details the experimental setup. Section 5 reports the experimental results and provides a thorough analysis. Section 6 concludes the research.

2. Related work

2.1. Event schema induction

The seminal work in event schema induction is the coreference chains [4] for template induction. Currently, mainstream methods include probability-based generative models [3,8,23,34] that jointly model predicates and arguments, as well as ad-hoc clustering algorithms [14,26,29]. Reference [20] adopts an entity-centric approach to event schema induction, where entities are grouped into semantic slots and the predicate for entity clusters is identified in a post-processing step. These methods rely on discrete, handcrafted features and impose strong statistical assumptions. For event type induction, reference [15]design a semi-supervised vector quantization variational autoencoder framework that utilizes knowledge learned from existing annotations to discover new event types. Reference [27] reduces the complexity of feature generation using pre-trained language models and relaxes strict statistical assumptions through latent spherical space clustering. These methods did not take into account the hierarchical relationship between event types.

2.2. Event extraction

Event extraction is considered a fundamental task in the field of information extraction [5]; it assumes a pre-defined event schema and attempts to identify event triggers and their associated arguments. Existing event extraction approaches have exploited two levels: sentence-level [21] and document-level [9]. Some also utilize RNNs [17], CNNs [6] and GANs [33] for extracting new event types. Recently, for zero-shot event extraction, reference [16] proposed transfer learning by applying a model trained on seen event types to unseen ones. Reference [35] used deep learning models to represent the semantics of labels and then developed an encoding model that aligns unrelated contexts with their respective definitions. This facilitates the mapping of candidate event mentions and definitions into a common embedding space, resulting in superior accuracy and efficiency in extracting arbitrarily defined event types.

3. Framework

As shown in Fig. 1, ETGen-Clus consists of three parts: 1) extraction of salient event triggers using dependency-based syntactic structures; 2) generation of latent space event main types; and 3) event trigger clustering based on semantic features to induce event subtypes.

Fig. 1.

Architecture overview. (Circles are features of event triggers obtained from the BERT pre-training model, squares are bag-of-words (BoW) representations of event triggers, and triangles are event triggers, where different colours represent semantically different.)

3.1. Event trigger identification

Similar to previous research [14,27], this paper employs a lexicality-based approach to identify candidate event triggers. Specifically, given a sentence, this paper leverages a dependency parser named Spacy to extract relevant linguistic features such as tokens, lemmas, and part-of-speech (POS) tags. Due to Spacy’s superior speed and accuracy compared to other natural language processing tools and considering the relatively large corpus used in this paper, utilizing the Spacy toolkit is expected to significantly enhance efficiency and yield favorable outcomes. Subsequently, this paper selects all non-auxiliary verb tokens as our candidate verb tokens from the input sentence. However, indiscriminately considering all verb tokens as candidate event triggers may lead to certain drawbacks, including an overabundance of overly general or information-poor event triggers as well as excessively specific and difficult-to-generalize event triggers. To mitigate these issues, this paper adopts a saliency-based approach to identify more informative verb tokens that satisfy two criteria: first, candidate event triggers should exhibit high frequency within the corpus, and second, the candidate event trigger frequency should not be excessive when compared to large general-domain background corpora. Use saliency scores [27] ranked in the top 80% as candidate event triggers. The saliency score calculation formula is as follows: $\begin{matrix} (1) & S_{score} (w) = (1 + log {(f (w))}^{2}) log (\frac{N_{-} bs}{bsf (w)}) \end{matrix}$ where $S_{score} (w)$ is saliency score of word, $f (w)$ is the frequency of word w, and $bsf (w)$ is the number of times the word w occurs in the background sentences out of the total number of background sentences. $N_{-} bs$ is the number of background sentences.

3.2. Event main type generative model

Fig. 2.

Plate notations for models (T – event triggers; d – documents; the observed variables are denoted by gray circles, while hidden variables are represented by white circles.)

Table 1

Notation used in this paper

Symbol	Description
C	A corpus
d	Number of documents
V	Number of event triggers
K	Number of event main types
n	Dimension of feature
$f_{t}$	Feature of event trigger
$h_{d}$	Frequency of each event trigger in a document
$f_{d}$	Feature of contextual
$f_{e}$	Feature of event main type
e	Latent variable of event type
α	Normal distribution parameter of e
β	Normal distribution parameter of $f_{e}$
θ	Multinomial distribution parameter of trigger

Given a corpus $C = {d_{1}, d_{2}, \dots, d_{n}}$ , where $d_{i}$ is a document, sample a latent event main type e for each document $d_{i}$ from a global normal distribution parameterized by α. Simultaneously, the model obtains a semantic feature $f_{t} \in R^{n}$ for the event trigger from a contextual encoder and computes the weighted average to obtain a contextual feature $f_{d} \in R^{n}$ . Moreover, this paper assumes that feature $f_{e} \in R^{n}$ , representing an event main type, follows a multivariate normal distribution with a diagonal covariance matrix. Denote all parameters for the K different multivariate normal distributions for $f_{e}$ as β ∈ $R^{K \times 2 n}$ , where K represents the number of event main types and n represents the dimension of $f_{e}$ . In the event type-trigger distribution, treat the probability matrix θ ∈ $R^{K \times V}$ as parameters subject to row-wise simplex constraints to ensure their validity, where V denotes the event trigger vocabulary size. This paper named this model ETGen. The plate notation of a generative model is shown in Fig. 2. The notation used in this paper is summarized in Table 1.

Pre-trained language models, in particular BERT [10] and GPTs [24], have demonstrated impressive capabilities in various natural language processing tasks by providing rich informational representations of the language. In the framework, this paper uses the pre-trained BERT model to extract semantic features from event triggers. Specifically, given the pre-training procedure of BERT, which encompasses the incorporation of masking language modeling and next-sentence prediction, the necessity arises to employ sentences as input data for the purpose of capturing semantic information at the level of individual words. Accordingly, this paper utilizes documents as BERT inputs, with the aim of generating specific semantic embeddings for each event trigger within their respective sentences.

The joint probability of a document is $\begin{matrix} (2) & p_{α, β, θ} (d) = p_{α} (e) \times p_{θ} (t | e) \times p_{β} (f_{e} | e) \end{matrix}$

The main goals of the model are twofold. First, the model needs to learn the optimal parameters. Second, seek to perform inference to estimate the posterior distribution of the latent variable e, given a document d. To accomplish both tasks, the model uses a neural inference network to obtain the variational parameters and implements the amortized variational inference method [28]. Following reference [28], the model collapses the discrete latent variable e to obtain an Evidence Lower Bound (ELBO) [18] of the log marginal likelihood. $\begin{aligned} log p_{α, β, θ} (d) \\ = log \int_{e} [p_{θ} (t | e) p_{β} (f_{e} | e)] p_{α} (e) d e \\ ⩾ {ELBO}_{d} (α, β, θ, ω) \\ (3) & = E_{q_{ω} (e)} log p_{β, θ} (d | e) - D_{KL} [q_{ω} (e) ‖ p_{α} (e)] \end{aligned}$ where $D_{KL} [q_{ω} (e) ‖ p_{α} (e)]$ represents the Kullback–Leibler (KL) divergence between the variational posterior $q_{ω}$ and the prior $p_{α}$ . In order to address the computational complexities involved in calculating the KL divergence between different distributional classes and to take advantage of reparameterization techniques applicable to normal distributions, this paper is devoted to parameterizing the variational posterior $q_{ω} (e)$ using learnable parameters ω so as to represent it as a normal distribution. These learnable parameters are estimated by utilizing a neural inference network. The inference network, shown in Fig. 3, takes the BoW representation of event triggers $h_{d}$ (which represents the frequency of each event trigger in a document) and contextual features $f_{d}$ as inputs, computes the mean vector μ and variance vector $σ^{2}$ of $q_{ω} (e)$ , and performs sampling of latent event type vectors. These latent event type vectors are used to reconstruct the BoW representation of event triggers and to construct the event main type feature $f_{e}$ .

3.2.1. Inference

Fig. 3.

High-level schema of the architecture for our inference network.

In Equation (3), the first component denotes the reconstruction error. In addition, the KL divergence term in this paper is computed in closed form. To maximize the ELBO, this paper uses the ADAM optimizer. This paper also alleviates the problem of component collapse by using a high moment weight (>0.8) and learning rate (in [0.001, 0.1]) in the ADAM optimizer while applying batch normalization and dropout techniques. These strategies are followed by [28]. After model training, this paper assigns an event main type feature to each event trigger feature by maximizing likelihood. $\begin{aligned} p_{β, θ} (t | e) & \propto p_{β, θ} (t, f_{e}) \\ (4) & = p_{θ} (t | e) \times p_{β} (f_{e} | t) \end{aligned}$

Algorithm 1

ETGen

The parameter learning is shown in Algorithm 1, where α, β and θ are model parameters. The notation $NN (\cdot; \cdot)$ denotes a neural network that takes inputs and parameters.

3.2.2. Event main type generation

Upon convergence of the generative model train, given a corpus C, first utilize the learned parameters α to sample K event main types of logarithmic probabilities. Then, for each document d in the corpus, employ the parameter θ to sample K event main types of logarithmic probabilities corresponding to each event trigger in the text. Simultaneously, by utilizing the parameter β, obtain the logarithmic probabilities of each event trigger’s semantic feature $f_{t}$ under the main type feature $f_{e}$ of each event. Subsequently, the sum of the aforementioned three logarithmic probabilities yields $t_{K} \in R^{V \times K}$ , where V represents the number of event triggers in the sampled text and K refers to the number of event main types. Finally, subject $t_{K}$ to softmax processing, thereby assigning each event trigger to the event main type with the highest probability. Figure 4 shows diagrams of the event main types generated by the model.

The generating function for each event main type corresponding to each event trigger is defined as follows: $\begin{array}{c} (5) & t_{K} = LogS (P_{α} (e)) + LogS (P_{θ_{d}} (t)) + LogS (P_{β_{e}} (f_{t})) \\ (6) & {Event}_{-} MType (t_{K}) = softmax (t_{K}) \end{array}$ where $P_{α} (e)$ denotes the probability distribution of K event main types conditioned on the parameter α. $P_{θ_{d}} (t)$ represents the conditional probability distribution of event triggers within each event main type for every document, given the parameter θ. $P_{β_{e}} (f_{t})$ corresponds to the maximum logarithmic probability of the semantic feature of each event trigger among multiple normal distributions for each event main type feature under the parameter β, where $LogS$ denotes the natural logarithm of the softmax function. $t_{K}$ is K logarithmic probability for each event trigger in every text, with each logarithmic probability associated with an event main type. Finally, ${Event}_{-} MType (t_{K})$ is the probability distribution obtained by applying the softmax function to $t_{K}$ .

Fig. 4.

The schematic diagram of event main types generated by our model, where each box represents an event main type and gray circles indicate the event trigger.

3.3. Event subtype induction

In the generated event trigger clusters, each event trigger exhibits a certain degree of semantic similarity and high coherence, representing the main type of event. Subsequently, based on the viewpoint that event triggers of event subtypes possess highly similar semantic features, this module uses the semantic characteristics of event triggers as input to perform further clustering for each event main type in order to obtain subtypes of events. Three classical clustering algorithms, namely Kmeans, Points To Identify The Clustering Structure (OPTICS) [1] and Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN) [2], will be applied as the final module of the framework for comparison. To reduce the sparsity of event trigger features and computational costs, this paper utilizes the Uniform Manifold Approximation and Projection (UMAP) [22] algorithm for dimensionality reduction. UMAP was chosen over T-distributed Stochastic Neighbor Embedding (t-SNE) [13] because it is known to better preserve both local and global features of high-dimensional data in lower projection dimensions.

Fig. 5.

The schematic diagram of event subtypes generated by clustering the event triggers.

By implementing further clustering algorithms, this module is able to locate areas of semantic density among event triggers, which represent subtypes of events. As shown in Fig. 5, the event main type is commit, and after clustering, each event trigger is clustered into its respective cluster, with purple representing the hurt event, blue representing the justice event, and orange representing the hearing event. The clustering results vary across different clustering algorithms. Figure 6 displays the visualization results of event trigger clusters after applying both Kmeans and OPTICS clustering algorithms based on semantic features. It can be observed that the distance-based Kmeans algorithm can only determine the number of event subtypes using a pre-defined K value, forcing all event triggers into one category, while the density-based OPTICS clustering algorithm can identify more densely populated clusters of event triggers in the semantic space.

In summary, the cluster method of this module involved utilizing UMAP for dimension reduction on event trigger semantic feature vectors to make them less sparse and computationally efficient, followed by using classical clustering to identify the dense areas of event trigger semantic features. Finally, the framework effectively identified fine-grained event subtypes by locating clusters of semantically similar event triggers within dense regions of each event main type.

Fig. 6.

UAMP visualization for Kmeans and OPTICS.

4. Experimental setup

4.1. Datasets

Table 2
Comparison with existing datasets

Dataset Documents Sentences Tokens

MAVEN 4,480 49,873 1,276k

GNB 12,985 134,237 2,245k

Dataset	Documents	Sentences	Tokens
MAVEN	4,480	49,873	1,276k
GNB	12,985	134,237	2,245k

This paper performs experiments on two datasets: MAssive eVENt(MAVEN) and GNBusiness-All(GNB). MAVEN was collected and constructed by reference [31]. To ensure consistency, this paper used identical pre-processing procedures to extract event triggers for both datasets. The introduction to these corpora is as follows:

MAVEN is a large-scale general-domain event detection dataset that utilizes English Wikipedia as the data source, consisting of 4,480 documents, 49,873 sentences, and 1,276k tokens. The content of the dataset covers a wide range of subject areas, including history, science, culture, technology, society, and sports, among others. Each article aims to provide comprehensive information on a particular topic, including its history, background, related concepts, important figures and events, and developmental trends, among other aspects.

GNB was obtained by reference [20] by crawling Google’s business news sites.1

News demonstration: https://news.google.com.

The dataset only contains the headlines and opening paragraphs of each news text and contains a total of 12,985 news reports, comprising 134,237 sentences and 2,245k tokens. The majority of event types in this dataset are related to business topics.

A comparison of MAVEN and GNB is shown in Table 2.

Table 3

Hyper-parameters setting

Name	Value
Feature dimension	768
Fully connected layer size	100
Activation function	Softplus
Learning rate	0.002
Dropout rate	0.02
Batch size	128

4.2. Experiment environment

This paper runs all experiments on a desktop computer with the Intel Core i7-13900HX CPU, one NVIDIA GeForce RTX 4060 GPU, 32 GB of RAM, and the PyTorch deep learning platform. Most of the hyper-parameters in our generative model are shown in Table 3. The number of event main types in an experiment is chosen according to experiments of event type discovery.

4.3. Evaluation metrics

4.3.1. Event type coherence

Several qualitative metrics have been proposed to assess topic coherence. Reference [25] showed a framework in which various existing topic coherence measures are combined together. $C_{P}$ is the recommended coherence metric in [25]. This measure uses Fitelson’s coherence, which evaluates every word in the context of all subsets that can be constructed from the remaining words. Experiments show that this approach is well suited to evaluating the results of the event types in the model.

Formally, the event coherence $C_{P} (e)$ of an event is calculated by using its top-N event triggers as $\begin{array}{c} (7) & C_{P} (e) = \frac{\sum_{i = 1}^{N} \sum_{j = 1}^{2^{N - 1} - 1} m_{f} (w_{i}, S {(i)}_{j})}{N \cdot (2^{N - 1} - 1)} \\ (8) & m_{f} (w_{i}, S {(i)}_{j}) = \frac{P (W_{i} | S {(i)}_{j}) - P (W_{i} | \neg S {(i)}_{j})}{P (W_{i} | S {(i)}_{j}) + P (W_{i} | \neg S {(i)}_{j})} \end{array}$ where $S (i)$ is the set of all subsets without the word $w_{i}$ , $P (W_{i} | S {(i)}_{j})$ and $P (W_{i} | \neg S {(i)}_{j})$ are estimated based on the conditional probability of $w_{i}$ and every single word set.

4.3.2. Event trigger semantic similarity

Due to the semantic similarity of triggers within an event type, this paper adopts semantic similarity as another measure of event type quality. This paper defines the cosine similarity of the top 10 event triggers under each event cluster. Results close to 1 indicate better similarity of event types; results close to −1 indicate poor similarity. $\begin{array}{c} (9) & C_{Sim} (e) = \frac{2}{N^{2} - N} \sum_{i = 2}^{N} \sum_{j = 1}^{i - 1} {Sim}_{cos} (w_{i}, w_{j}) \\ (10) & {Sim}_{cos} ({\vec{w}}_{i}, {\vec{w}}_{j}) = \frac{{\vec{w}}_{i} \cdot {\vec{w}}_{j}}{‖ {\vec{w}}_{i} ‖ ‖ {\vec{w}}_{j} ‖} \end{array}$ where ${\vec{w}}_{i}$ is the semantic feature vector of event triggers and ${Sim}_{cos} ({\vec{w}}_{i}, {\vec{w}}_{j})$ is the similarity between ${\vec{w}}_{i}$ and ${\vec{w}}_{j}$ .

4.3.3. Event type quality

This paper defines the overall metric for the quality of an event type as the weighted sum of its event type coherence and event trigger semantic similarity. $\begin{matrix} (11) & Quality (e) = λ * C_{P} (e) + (1 - λ) * C_{Sim} (e) \end{matrix}$ where λ is the weight coefficient. This paper set λ to 0.5 in experiment.

4.4. Compared methods

BERT-Kmeans: The classical clustering algorithm that works in the Euclidean feature space takes event trigger semantic representations and groups all candidate event triggers into clusters. In this paper, experiments are conducted at K values of [10:100], where K is the number of event types.

OPTICS: OPTICS is a density-based clustering algorithm that extracts clustering structures by constructing the reachability graph among input data points. Its core distance and minimum sample size parameters are determined based on the optimal quality score of event types.

HDBSCAN: HDBSCAN is also a density-based clustering algorithm that utilizes the density information of points to adaptively determine the clustering structure. Its main hyperparameter, minimum cluster size, is set according to the optimal event type quality score.

DEC: Reference [32] builds a self-taught loss for a deep clustering model. This paper followed the steps outlined in the original paper, which involved using semantic embeddings of event triggers as input data. This paper then reduced the dimensionality of these embeddings to a hidden space and performed clustering on the resulting reduced features.

ETypeClus [27]: The ETyeClus is another method based on deep neural networks. It extracts pairs of predicates and object heads from the corpus and disambiguates the sense of the predicate using a verb semantic dictionary. Finally, it obtains event types by jointly embedding and clustering the pairs in a latent spherical space. Unlike our work, which does not solely cluster predicates, ETyeClus also includes object heads as clustering objects. To enable comparison with our approach, this paper only considers predicates and follows the default data preprocessing and clustering steps outlined in the original paper.

ProdLDA [28] The ProdLDA is an autoencoding variational Bayesian-based topic model for the modeling and analysis of textual data. Its input is the co-occurrence frequency of event trigger words for each text, and the output is the set of event triggers for K topics. In this paper, the value of K is experimented with at [10:100], and the optimal event quality score will be obtained as the final result.

ETGen-Clus: The framework consists of a generative model that incorporates the co-occurrence information of event triggers and their semantic information, as well as a clustering module. Specifically, this paper employed Kmeans, HDBSCAN, and OPTICS as clustering modules.

5. Experimental results

5.1. Event type discovery

Fig. 7.

Averaged event type quality scores of the two datasets with different numbers of event main types K.

In order to determine the optimal number of event main types, this paper first applies a generative model to MAVEN and GNB to discover event main type number K by grid search in [5; 100] with the step equal to 5. Figure 7 shows the averaged event type quality score of the two datasets. For the average event type quality score, the optimal number of event types for MAVEN is 10 and 20 for GNB. This paper chose $K = 10$ and $K = 20$ as the number of event main types on the datasets MAVEN and GNB, respectively. Subsequently, in order to identify the event main types represented by each event trigger cluster, the proposed framework employs the OPTICS algorithm for further clustering to unveil event subtypes. Concretely, in this paper, the OPTICS algorithm is configured with the parameter $\min_cluster_size$ ranging from 10 to 50, with a step size of 5. The determination of the final number of clusters is predicated on the highest quality score attributed to the event types. The event type quality score, as employed in this paper, is computed as the weighted average of coherence and semantic similarity measures between event triggers. This composite score serves as a more comprehensive and unbiased yardstick for evaluation or decision-making, facilitating a well-informed selection of the optimal number of event types.

Some clustering examples are presented in Table 4 of this paper. From the observations, it can be noted that event trigger clusters with higher event quality scores are more likely to determine event types compared to clusters with lower scores. This finding indirectly suggests that the metric proposed in this paper is effective in evaluating the quality of event trigger clusters.

Table 4

Clustering of event triggers in the top five and bottom two event quality scores found on the MAVEN dataset. Event types have been manually annotated, and italics indicate subtypes of events. The event main type is indicated above the subtypes

Event type	Triggers	Quality
Violence Killing	Kill, murder, wound, shoot, stab, injure, maim, assassinate, hurt, gun	0.639
Scenario Catastrophe	Topple, damage, wreck, blow, collapse, tear, knock, fell, litter, wash	0.591
Violence Besieging	Overwhelm, secure, besiege, capture, infiltrate, invest, seize, recapture, attack, subdue	0.586
Scenario Competition	Play, defeat, score, win, hold, compete, retain, spare, represent, participate	0.472
Legality Legal rulings	Jail, imprison, incarcerate, try, extradite, pardon, await, release, disappear, free	0.452
-	Reassert, centralize, reaffirm, live, lean, reconstitute, monopolize, formalize, caliphate, empower	−0.0118
-	Presage, precipitate, unleash, exacerbate, catalyze, foment, intertwine, unfold, elicit, engender	−0.1214

Table 5

The induction results for event types, with each method being run five times, and the average results for each metric

Dataset	Method	Event type number	Coherence	Similarity	Quality
MAVEN	BERT-Kmeans	40	0.106	0.587	0.347
	HDBSCAN	42	0.131	0.588	0.359
	OPTICS	45	0.159	0.555	0.357
	DEC	50	0.131	0.594	0.363
	ETypeClus	50	0.146	0.572	0.361
	ProdLDA	50	0.346	0.392	0.369
	ETGen-Clus	50	0.194	0.591	0.392
GNB	BERT-Kmeans	40	0.036	0.467	0.251
	HDBSCAN	40	0.063	0.525	0.294
	OPTICS	47	0.089	0.544	0.316
	DEC	40	0.034	0.561	0.297
	ETypeClus	40	0.043	0.599	0.321
	ProdLDA	50	0.203	0.363	0.283
	ETGen-Clus	40	0.114	0.546	0.331

5.2. Comparison with other method

Table 5 presents the results of all methods on two datasets. This paper used 3 metrics to evaluate the event trigger clustering results for each dataset, resulting in 6 metrics for both datasets combined. This paper determined the number of event types induced by each method based on their event type quality scores. Event type induction methods are divided into three groups: induction based on semantic information, induction based on co-occurrence information, and induction in the ETGen-Clus framework.

As shown in Table 5, the results of all the methods on the two datasets. In this paper, coherence, semantic similarity, and event type quality scores are used to evaluate the clustering results of event triggers on each dataset. Compared to the baseline model, the model used in this paper achieves higher event type quality scores of 6.23% and 3.11% on the two datasets, respectively, and since the first six models in the table perform the clustering induction task for the semantic features of event triggers, they all have higher semantic similarity. However, their coherence is reduced compared to the ProdLDA models. This is mainly due to the fact that ProdLDA is a topic model that performs the event type induction task using information about the co-occurrence of event triggers in documents. Although ProdLDA performs better in obtaining the coherence of the set of event triggers, it performs poorly in terms of the semantic similarity of event triggers. In contrast, the ETGen-Clus achieves improvements in both coherence and semantic similarity aspects of the event trigger induction by incorporating co-occurrence and semantic information. Particularly in terms of coherence, ETGen-Clus outperforms models that induce event types based solely on semantic information, with increases of 22.01% and 28.08% on the two datasets, respectively. After considering these two metrics together, this paper’s model outperforms the existing baseline model.

As shown in Table 6, among the clustering modules used in the framework, the use of OPTICS and HDBSCAN resulted in the best performance, indicating that density-based clustering models can better group significant event triggers together.

Table 6
Results of event type induction obtained by selecting different clustering algorithms

Dataset Method Event type number Coherence Similarity Quality

MAVEN ETGen-Kmeans 50 0.144 0.539 0.341

ETGen-HDBSCAN 45 0.187 0.582 0.384

ETGen-OPTICS 55 0.194 0.591 0.392

GNB ETGen-Kmeans 40 0.081 0.503 0.292

ETGen-HDBSCAN 42 0.102 0.541 0.321

ETGen-OPTICS 40 0.114 0.546 0.331

Dataset	Method	Event type number	Coherence	Similarity	Quality
MAVEN	ETGen-Kmeans	50	0.144	0.539	0.341
ETGen-HDBSCAN	45	0.187	0.582	0.384
ETGen-OPTICS	55	0.194	0.591	0.392
GNB	ETGen-Kmeans	40	0.081	0.503	0.292
ETGen-HDBSCAN	42	0.102	0.541	0.321
ETGen-OPTICS	40	0.114	0.546	0.331

For the GNB dataset, all of the models showed a decrease in coherence metric scores. Each text in GNB consists of a news headline and first paragraph, which is relatively shorter in content compared to the MAVEN dataset and has a narrower range of event types covered, limiting the utilization of co-occurrence information of event trigger words. Therefore, except for the similarity metric, all other metrics showed a decrease.

Fig. 8.

Comparison between ETypeClus and ETGen-Clus.

Fig. 9.

Event type hierarchical relationship structure.

5.3. Mining latent event types analysis

For the event type induction model based on semantic information, only the semantic information of event triggers is used as input, lacking the assistance of co-occurrence information of event triggers, and the information mining of potential event types is insufficient. In contrast, the generative model in this paper is based on the neural variant autoencoder, fuses semantic information and co-occurrence information of event triggers to mine event type latent variables, and provides highly coherent clusters of event triggers for the semantic segmentation of event triggers in the next step. As shown in Fig. 8, in the range of the number of event types [10:100], the generative model proposed in this paper that fuses the semantic information of event triggers and co-occurrence information improves the event quality scores compared with those obtained from the semantic information-based event triggers induction task represented by ETypeClus, indicating that the model in this paper has been effective in the event trigger implicit event type information mining.

5.4. Event type hierarchical relationship analysis

The event type can be defined as a class or concept and can be established hierarchically through parent-child relationships or inclusion relationships. As shown in Fig. 9, Legal can be considered an event type and can be further categorized into subtypes such as Violent Crime, Theft Crime, Judicial, etc. thus forming a hierarchical structure. This model the event type hierarchy by using event trigger clusters to represent event types, which enables different properties to be exhibited by the event trigger clusters of main types and subtypes. Specifically, for an article describing a Legal event type, it may include event types such as Violent Crime, Theft Crime and Judicial. In the research task of this paper, this paper considers that in the cluster of event triggers that denote legal event types, there will be event trigger words that denote event types such as Violent Crime, Theft Crime and Judicial. Therefore, the event triggers in the event trigger clusters representing legal event types are more likely to co-occur in the same article. On the other hand, since the Violent Crime, Theft Crime and Judicial event types are more specific compared to the Legal event type, the semantic similarity between them will be higher for event triggers in the Violent Crime, Theft Crime and Judicial event type clusters.

As shown in Table 7, the generative module fully integrates co-occurrence information and semantic information of event triggers, resulting in better coherence of generated event triggers that conform to the properties of event main types. Similarly, in the event trigger word clustering module, the coherence of event triggers is reduced, but the similarity is significantly increased, conforming to the properties of event subtypes. Therefore, the framework fully utilizes the hierarchical relationship between event types to induce high-quality event types.

Table 7
The results of event type induction on two datasets by two modules

Dataset Module Coherence Similarity Quality

MAVEN Event main type generation 0.227 0.455 0.341

Event subtype induction 0.194 0.591 0.392

GNB Event main type generation 0.181 0.398 0.289

Event subtype induction 0.114 0.546 0.331

Dataset	Module	Coherence	Similarity	Quality
MAVEN	Event main type generation	0.227	0.455	0.341
Event subtype induction	0.194	0.591	0.392
GNB	Event main type generation	0.181	0.398	0.289
Event subtype induction	0.114	0.546	0.331

6. Conclusion and future work

This paper proposes an event type induction framework oriented towards event type hierarchies. Experimental results show that the ETGen-Clus framework outperforms event type induction models that only utilize semantic or co-occurrence information for event triggers. Specifically, this paper analyzes the relationship between event type hierarchy and event trigger clusters and divides the induction task into two steps, namely, generating event trigger clusters with higher coherence that represent the main types of events and inducting event trigger clusters with higher semantic similarity that represent the subclasses of events. The framework emphasizes the importance of event type hierarchy in event type induction. Experiments on two datasets demonstrate that ETGen-Clus is able to infer salient event types that are consistent with human judgment. In the future, we will explore the corresponding argument roles for each event type, analyze the entities on which each of them depends in the original text by clusters of event triggers denoting the event type, and finally obtain the argument roles under a specific event type to form a complete event schema.

Footnotes

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) under grants (No.72204261, No.72371245).

References

Ankerst,

M.M.

Breunig,

H.P.

Kriegel and

Sander, OPTICS: Ordering points to identify the clustering structure, ACM Sigmod record 28(2) (1999), 49–60. doi:10.1145/304181.304187.

R.J.

Campello,

Moulavi and

Sander, Density-based clustering based on hierarchical density estimates, Lecture Notes in Computer Science 7819(1) (2013), 173–184.

Chambers, Event schema induction with a probabilistic entity-driven model, in: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 2013, pp. 1797–1807.

Chambers and

Jurafsky, Template-based information extraction without the templates, in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, 2011, pp. 976–986.

M.H.

Chen,

H.M.

Zhang,

Ning,

M.L.

Li,

Ji,

McKeown and

Roth, Event-centric natural language processing, in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Tutorial Abstracts, 2021, pp. 6–14. doi:10.18653/v1/2021.acl-tutorials.2.

Y.B.

Chen,

L.H.

Xu,

Liu,

D.J.

Zen and

Zhao, Event extraction via dynamic multi-pooling convolutional neural networks, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015, pp. 167–176.

Y.F.

Chen, Multiple-level biomedical event trigger recognition with transfer learning, BMC Bioinformatics 20(1) (2019), 1–15. doi:10.1186/s12859-019-3030-z.

J.C.K.

Cheung,

Poon and

Vanderwende, Probabilistic frame induction, Computer Science (2013).

Chu and

Yang, Chinese document-level emergency event extraction dataset and corresponding methods, APPLIED SCIENCES-BASEL 13(12) (2023), 7015. doi:10.3390/app13127015.

10.

Devlin,

M.W.

Chang,

Lee and

Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv:1810.04805.

11.

Ding,

Z.R.

Li,

Z.Y.

Liu,

H.T.

Zheng and

Z.B.

Lin, Event detection with trigger-aware lattice neural network, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019, pp. 347–356. doi:10.18653/v1/D19-1033.

12.

Ding,

Qin and

Liu, Building Chinese event type paradigm based on trigger clustering, in: Proceedings of the Sixth International Joint Conference on Natural Language Processing, 2013, pp. 311–319.

13.

Hanke and

Knees, Visualizing data using t-SNE, Journal of machine learning research 9(11) (2008).

14.

L.F.

Huang,

Cassidy,

X.C.

Feng,

Ji,

Voss,

J.W.

Han and

Sil, Liberal event extraction and event schema induction, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, 2016, pp. 258–268.

15.

L.F.

Huang and

Ji, Semi-supervised new event type induction and event detection, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 718–724. doi:10.18653/v1/2020.emnlp-main.53.

16.

L.F.

Huang,

Ji,

Cho,

Dagan,

Riedel and

C.R.

Voss, Zero-shot transfer learning for event extraction, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 2160–2170. doi:10.18653/v1/P18-1201.

17.

Y.L.

Jin,

J.J.

Ye,

L.Q.

Shen,

Xiong,

L.L.

Fan and

Q.F.

Zangu, Hierarchical attention neural network for event types to improve event detection, Sensors 22(11) (2022), 4202. doi:10.3390/s22114202.

18.

D.P.

Kingma and

Welling, Auto-encoding variational Bayes, Statistics 1050 (2013), 1.

19.

J.H.

Lau,

Newman and

Baldwin, Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality, in: Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, 2014, pp. 530–539. doi:10.3115/v1/E14-1056.

20.

Liu,

H.Y.

Huang and

Zhang, Open domain event extraction using neural latent variable models, in: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, pp. 2860–2871. doi:10.18653/v1/P19-1276.

21.

S.D.

Lu,

Zhao,

Li and

Guo, Explainable document-level event extraction via back-tracing to sentence-level event clues, Knowledge-Based Systems 248 (2016). doi:10.1016/j.knosys.2022.108715.

22.

McInnes,

Healy and

Melville, Umap: Uniform manifold approximation and projection for dimension reduction, Statistics (2018).

23.

K.H.

Nguyen,

Tannier,

Ferret and

Besançon, Generative event schema induction with entity disambiguation, in: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, 2015, pp. 188–197.

24.

Radford,

Wu,

Child,

Luan,

Amodei and

Sutskever, Language models are unsupervised multitask learners, OpenAI blog 1(8) (2019), 9.

25.

Röder,

Both and

Hinneburg, Exploring the space of topic coherence measures, in: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, 2015, pp. 399–408. doi:10.1145/2684822.2685324.

26.

Sha,

S.J.

Li,

B.B.

Chang and

Z.F.

Sui, Joint learning templates and slots for event schema induction, in: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 428–434.

27.

J.M.

Shen,

Y.Y.

Zhang,

Ji and

J.W.

Han, Corpus-based open-domain event type induction, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, pp. 5427–5440. doi:10.18653/v1/2021.emnlp-main.441.

28.

Srivastava and

Sutton, Autoencoding variational inference for topic models, Statistics 1050 (2017), 4.

29.

Ustalov,

S.P.

Ponzetto,

Kutuzov,

Panchenko and

Biemann, Unsupervised semantic frame induction using triclustering, in: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 55–62. doi:10.18653/v1/P18-2010.

30.

Wadden,

Wennberg,

Luan and

Hajishirzi, Entity, relation, and event extraction with contextualized span representations, 2019, arXiv:1909.03546.

31.

X.Z.

Wang,

Z.Q.

Wang,

Han,

W.Y.

Jiang,

Han,

Z.Y.

Liu,

J.Z.

Li,

Y.K.

Lin and

Zhou, MAVEN: A massive general domain event detection dataset, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 1652–1671. doi:10.18653/v1/2020.emnlp-main.129.

32.

J.Y.

Xie,

Girshick and

Farhadi, Unsupervised deep embedding for clustering analysis, in: Proceedings of the 33rd International Conference on International Conference on Machine Learning, 2016, pp. 478–487. doi:10.5555/3045390.3045442.

33.

J.C.

Yan,

L.H.

Chen,

Y.C.

Yu,

H.X.

Xua,

Q.C.

Gao,

K.P.

Cao and

J.H.

Chen, EmergEventMine: End-to-end Chinese emergency event extraction using a deep adversarial network, ISPRS International Journal of Geo-Information 11(6) (2022), 345. doi:10.3390/ijgi11060345.

34.

Yuan,

Ren,

W.Q.

He,

Zhang,

X.H.

Geng,

L.F.

Huang,

Ji,

C.H.

Lin and

Han, Open-schema event profiling for massive news corpora, in: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, 2018, pp. 587–596. doi:10.1145/3269206.3271674.

35.

H.M.

Zhang,

W.L.

Yao and

Yu, Efficient zero-shot event extraction with context-definition alignment, in: Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 7169–7179. doi:10.18653/v1/2022.findings-emnlp.531.

Event type induction using latent variables with hierarchical relationship analysis

Abstract

Keywords

1. Introduction

2. Related work

2.1. Event schema induction

2.2. Event extraction

3. Framework

3.2. Event main type generative model

4.1. Datasets

Table 2 Comparison with existing datasets Dataset Documents Sentences Tokens MAVEN 4,480 49,873 1,276k GNB 12,985 134,237 2,245k

4.3. Evaluation metrics

4.3.1. Event type coherence

4.3.2. Event trigger semantic similarity

4.3.3. Event type quality

4.4. Compared methods

5. Experimental results

5.1. Event type discovery

5.4. Event type hierarchical relationship analysis

Footnotes

Acknowledgements

References

Table 2
Comparison with existing datasets

Dataset Documents Sentences Tokens

MAVEN 4,480 49,873 1,276k

GNB 12,985 134,237 2,245k