A hybrid deep neural network model for query intent classification

Abstract

Query intents describe user information needs for searching on the web. How to capture the query intents is a crucial research topic in information retrieval. Search engine users always employ insufficient or unclear words as queries, thus making query intents ambiguous and uncertain to be interpreted by search engines. Query intent classification can deal with the problem by clarifying user queries and interpreting information needs for improving user satisfaction. Two main challenges have been addressed to classify query intents: one is how to effectively represent short and ambiguous queries; the other is how to generate a set of appropriate categories for matching diverse queries. In the paper, we propose a hybrid deep neural network model for query intent classification to meet the challenges. Our model adopts two state-of-the-art neural network models to comprehensively represent queries as feature vectors. We then employ query logs to automatically generate intermediate categories for fine-grained query intent clarification. Experimental results show that our method can outperform other baseline models, and effectively improve the performance in query intent classification.

Keywords

Information retrieval query intent classification query representation deep neural network model machine learning

1. Introduction

With the increasing amount of information on the Internet, search engines become indispensable tools in our daily life. Information retrieval (IR), as the core technologies of search engines, seeks to comprehend user information needs and provide relevant documents for given queries. However, submitted queries for search engines may ambiguously describe the real user intents, thus hurting retrieval performance. For example, the intents for searching the query ‘apple’ may be associated with the fruit apple or associated with electronic products produced by the Apple Company. To better interpret queries, it becomes necessary to distinguish different query intents before retrieving and ranking candidate websites or documents.

To tackle the problem, query intent classification has become a crucial research issue. Query intent classification aims to map queries with predefined query categories so as to eliminate the ambiguity and uncertainty of original queries [1]. Although query intent classification is essentially a text classification problem, it has certain unique characteristics, which makes this task more challenging. Specifically, the average length of queries, compared to documents, tends to be very short. Therefore, existing methods for document classification can hardly be directly used for query intent classification. Moreover, there is a lack of labeled data for training the classifiers, because it is difficult to annotate the real query intents without enough contexts. To this end, it remains two research issues on query intent classification: effective representations of queries and automatic label strategy on query intents.

Existing research on query intent classification has addressed these two issues and carried out by directly using supervised machine learning methods [2 –7]. Most studies have transformed the problem to a document classification problem through pseudo-relevance feedback. Pseudo-relevance feedback assumes that top-ranked documents from initial retrieval are highly correlated with the original query. The top-ranked documents can therefore be treated as feedback to interpret user queries for mining the real query intents. The shortcoming for these studies lies in their incapability to cope with constantly changing categorizations of query intents due to the diverse information needs of users.

To capture the constantly changing intents, recent studies have proposed to adopt two-stage classification methods to improve the performance [8, 9]. In these studies, supervised methods were used to train classifiers for mapping queries to intermediate categories, and feedback documents were used to map queries in the intermediate categories with the target categories. These two-stage methods can deal with constantly changing categories of query intents by introducing intermediate categories, thus improving the classification accuracy. These studies motivate us to capture query intents in two stages for accurately understanding information needs of users. However, current studies still suffer from the difficulty in automatically generating the intermediate categories to reduce manual labor that is time and cost expensive.

To this end, we propose a hybrid deep neural network model for query intent classification. Our model leverages the state-of-the-art recursive and recurrent neural network models to implicitly construct a two-stage query intent classifier for query representations. To reduce manual costs of labeling, we introduce implicit relevance feedback based on query logs to automatically capture real query intents of users, which can also deal with the constantly changing categorizations of query intents. Experimental results show that our model outperforms baseline models in query intent classifications.

We summarize the contributions of our work as follows.

1) We propose a novel two-stage framework for query intent classification based on hybrid deep neural network models and query representations. In the first stage, we represent queries using word embedding-based features and syntactic features, and then encode the query representations using the recurrent and recursive neural networks, respectively, for mapping queries to intermediate categories. In the second stage, we concatenate the learned query representations and intermediate classification results as inputs, and use a multi-layer perceptron for classification in target categories.

2) We propose to generate intermediate categories of query intents based on implicit relevance feedback from query logs, which largely reduces manual labor in annotating query intents for intermediate classification. Our method adopts Open Directory Project (ODP) to automatically match the queries to their corresponding clicked web pages from query logs, and treats the categories of web pages from ODP as the intermediate categories of queries for effective classification of query intents.

3) We conduct experiments on a retrieval dataset, and experimental results show that our method outperforms the baseline methods in query intent classification. In the experiments, we use AOL query logs and ODP to generate intermediate categories, and classify queries based on two proposed hybrid neural networks. We then use TREC queries for the final stage classification based on Wikipedia. Experimental results demonstrate the effectiveness of the proposed method.

The rest of the paper is organized as follows. Section 2 introduces the related research work. Section 3 details the general framework of the proposed method. Section 4 introduces the query term feature extraction, and Section 5 provides details on the classification models. Section 6 reports the experimental results with in-depth analysis and discussions. Section 7 concludes this paper and provides future directions.

2. Related work

Query classification, as one of the most crucial issues in information retrieval, has attracted much attention in recent years [10 –13]. Early research has attempted to enrich the feature representations of queries [8 , 15]. For example, Shen et al. [8] won the 2005 KDD Cup for query enrichment by training document classifiers based on the search results. They extracted document features based on the titles, snippets and pages of documents for document classification, and then mapped the document taxonomy to the target taxonomy of queries. The quality of extracted features affects classification performance. Since queries usually contain only a few key words, how to generate effective features remains a great challenge.

Some studies have exploited external resources, such as the query logs, for generating effective query features. For example, Ashkan et al. [16] used clickthrough logs, query specific information and search results to capture commercial query intents. Radlinski et al. [17] identified the popular meanings of web queries using search logs and user click behaviors for more complete and user-oriented intents. Law et al. [18] designed a new online game to collect data on user query intents for reducing manual labor in annotating the intents for supervised training. Jiang et al. [19] mined query intents from query logs from a multi-dimensional perspective involving the URL, session and terms of queries. Results-oriented and topic-oriented frameworks were also proposed and investigated in their study. Sakar et al. [12] accurately predicted the online shopping intents of visitors using multilayer perceptron and LSTM recurrent neural networks. Yao et al. [20] developed a convolutional neural network-based ranker by integrating query-specific preferences with learning-to-rank methods. Singh et al. [11] presented a novel query expansion method based on fuzzy logic to select effective expansion terms from feedback documents. Ding et al. [21] propose an efficient query processing optimization approach based on extreme learning machine in ComMapReduce framework. Puigcerver et al. [10] tackled the problem of out-of-vocabulary query keywords using word similarity. These studies have implied that external resources are valuable in identifying query intents of users. Meanwhile, limited labeled data in query intent mining have posed another big challenge in improving the performance. Various methods have been proposed to meet the challenge. For instance, Li et al. [22] proposed two semi-supervised methods to augment labeled queries for query classification, in which clickthrough data were treated as graphs to enrich query features. Ayadi et al. [23] defined a list of generic and specific medical query features and exploited them in an association rule mining technique to discover correlations between query features and image retrieval models. Kim et al. [24] applied the matrix sketching algorithm to improve the scalability of query intent classification with limited labeled data. Abid et al. [13] provided a comprehensive review on search results diversification techniques, and pointed out user query intents are important for detecting subtopics of resultant documents. These studies motivate us to reduce manual labor in intent annotation using feedback from query logs.

In addition, query intent classification has also been addressed in different tasks. For example, Barua et al. [25] proposed to use suggested queries from search engines as external knowledge sources for named entity classification. Jiang et al. [26] addressed the session-based query auto-completion in terms of intent classification. They adopted learning to rank models to predict user intents and an ensemble model to combine the proposed features and conventional approaches. Khudabukhsh et al. [27] designed query based triggers to learn query classification models to detect self-harm intents in modern search systems. Ali et al. [28] highlighted the retrieval effectiveness of search engines in terms of the precision and recall based on case studies. Shi et al. [29] proposed using a deep long-short-term-memory based feature mapping to learn feature representations in query classification. Fariha et al. [30] developed a system for semantic similarity-aware query intent discovery. Cai et al. [31] proposed a CNN-LSTM model to predict user intents from online health communities. Glater et al. [32] proposed a novel framework for learning semantic query annotations suitable to the target intents of individual queries, and demonstrated the effectiveness of their approach for various target intents in different lengths and difficulty levels.

In this study, we propose a two-stage neural network model to further enhance the performance of query intent classification. Our model represents queries with abundant features and automatically matches query intents to target categories based on query logs through implicit feedback.

3. General framework

In this section, we introduce the general framework of the proposed model. Our framework classifies query intents in two stages for accurately mining user intents. The goal of the first stage is to classify the original queries into intermediate categories. The intermediate categorization is automatically generated to augment latent useful information in user queries. We exploit query logs as external resources in this stage for more accurate classification. We then generate comprehensive representations of queries, and integrate the representations with the intermediate categories in the second stage. The goal of the second stage is to map the queries from intermediate categories to target categories. We illustrate the general framework of our method in Fig. 1.

Fig. 1

General framework of the proposed method.

In the figure, we first transform the classification task in the first stage as a sequence classification problem. We take a given query as a sequence of query terms. Given the entire sequence of all the query terms, we compute the probabilistic distribution over intermediate categories for the first-stage classification. We then integrate the intermediate categories with query representations in the second stage to obtain the probabilistic distribution over target categories. In the next sections, we will introduce query term features, classification models and the labeling strategy on query intents, respectively.

4. Query representations

We transform each original query into a feature representation as the inputs of our framework. Since queries are submitted by users with consecutive terms, we treat query terms as a sequence, and extract features at query term level instead of the entire query. We extract three types of features, including word representation-based features, term weighting-based features and syntax-based features.

Word representation-based features are generated based on the neural network language models. We use the skip-gram model by word2vec [33] in our experiments, and train the model using the retrieval dataset. These features capture the information of terms based on their distribution in large corpus, encoding abundant semantics of terms in the context. Based on the word representations, each query term is represented as a feature vector. Distances between two vectors can reflect the similarity of two terms.

Term weighting-based features consider the importance of different terms in query context. We adopt the term re-weighting method [8] to learn the term weights. We then normalize the term weights as term features.

Syntax-based features are based on the part-of-speech (POS) information and the syntactic dependency trees of queries. For the POS based features, we represent the POS of query terms as integers, and treat them as features. For the syntactic dependency tree based features, we use the depth of each query term node in the tree as one feature. The tree-based feature can model the dependency relationship of different query terms for discriminating the usefulness of terms. We use the NLTK tool to obtain the POS and dependency trees of queries in our implementation.

We list all the defined features on query terms in Table 1 for easily understanding our query representations. We represent each query as a feature vector based on the defined term features for further processing. We believe that information on query intents have been comprehensively encoded in the feature vectors for further classification.

Table 1
Feature Definitions on Query Terms

Features Notations Dimensionality

Word representation vec _q d

Term weighting weight _q 1

Part-of-speech pos _q 1

Depths of nodes depth _q 1

Feature vector vector_q = (vec_q, weight_q, pos_q, depth_q) m = d + 3

Features	Notations	Dimensionality
Word representation	vec _q	d
Term weighting	weight _q	1
Part-of-speech	pos _q	1
Depths of nodes	depth _q	1
Feature vector	vector_q = (vec_q, weight_q, pos_q, depth_q)	m = d + 3

5. Query Intent Classification Based On Hybrid Deep Neural Networks

In this section, we introduce our hybrid deep neural network model in detail. The basic structure of our model has been illustrated in Fig. 1. In our framework, we adopt recurrent neural network and recursive neural network as the classification models in the first stage, respectively. We model the sequence of query terms as probabilistic distributions over intermediate categories. We generate the intermediate categories based on implicit relevance feedback from query logs. In the second stage, we employ the multiple layer perceptron to obtain the query intents in the target categories based on Wikipedia.

5.1. Intermediate categorization oriented mapping

The first stage of our model is designed to map the original queries to intermediate categories. The inputs of this stage are representations of query terms, and the outputs are the intermediate categories of queries. We adopt neural network models to map the inputs to the outputs for accurate classification. Next, we introduce our query intent labeling strategy and neural network models, respectively.

5.1.1 Query intent labeling based on query logs

We generate intermediate categories of query intents based on implicit relevance feedback from query logs. Query logs have been widely used to optimize the search results without active feedback from users. Query logs contain abundant search information, such as user queries and corresponding clicks, which can reflect the query intents.

To capture query intents from query logs, we regard clicked URLs of given queries as implicit relevance feedback. Implicit relevance feedback assumes the clicked URLs are relevant to queries, and can be used to interpret the queries. Based on this idea, we use the clicked URLs to build a query-click graph. A toy example of the query-click graph is illustrated in Fig. 2. The clicked URLs are assumed to be high correlated with the query intents. Therefore, we can use the categories of URLs to indicate the categories of query intents in our study.

Fig. 2

Construction of Query-click Graphs.

Like query intent categories, websites with a certain URL can be divided into different categories based on their contents, topics and other features. These categories for websites map a given URL to a category indicating characteristics of the URL. Therefore, we can use the categories of the clicked URLs to implicitly determine the category of query intents.

Specifically, we generate the intermediate categories of intents based on Open Directory Project (ODP). ODP is one of the largest publicly available categories of websites. ODP divides websites into 15 primary categories and more than one million secondary categories based on the topics and disciplines of websites covering more than 4 million websites. We adopt the secondary categories as the intermediate categories of query intents, which includes 219 different categories. The secondary categories are more focused than the primary categories, which may contribute more to intent classification. We list the primary and secondary categories in ODP in Table 2.

Table 2

Primary and counts of secondary categorizations in ODP

Primary	Secondary	Primary	Secondary	Primary	Secondary
Arts	26	Business	16	Computers	6
Games	18	Health	5	Home	16
Kids &Teens	13	News	14	Recreation	32
Reference	4	Regional	10	Science	12
Shopping	4	Society	30	Sports	13

By analyzing the URLs in secondary categories, we find that exact matching can only work for a small number of URLs. Therefore, we adopt a backtracking method to match URLs to certain categories. The method takes a clicked URL as inputs and judges whether it can match any secondary category. If matching successfully, we assign the category for the URL. Otherwise, we trim the URL to its parent and judge again until exact matching is achieved. Based on the algorithms, we match the clicked URLs to their corresponding secondary categories. In our experiment, we use AOL query logs as the datasets. By analyzing the data, we find that 49.56% URLs can be matched, and 41.37% of queries can be matched to at least one secondary category. Statistics of the matching results is shown in Table 3.

Table 3

The matching results in query logs

Items	Total Number	Matched items	Ratios of matching
Clicked URLs	19,442,629	9,635,757	49.56%
Queries	10,154,742	4,201,016	41.37%

Formally, for all the queries in query logs, we obtain a set of clicked URLs together with the corresponding clicks for each URL. Based on the ODP-based categorization, we match all queries with intermediate categories, and count the number of times of each matched secondary category. Finally, we use maximum likelihood estimation to convert the vectors of matching categories into probabilistic distribution over query intents as follows. $y_{q_{i}} = \frac{r_{q_{i}}}{\sum_{j}^{K} r_{q_{j}}}$ (1) where K is the number of categories for all the queries. y_{q
_i} is the probability of q_i in each category. We take the distribution as labels on query intents. It should be noted that other categorizations of URLs can also be applied in our framework, and different categorizations of URLs will lead to different categorizations of intermediate query intents.

To classify query representations into the intermediate categories, we adopt the recurrent neural network model and the recursive neural network model. These two networks have been proved effective in various text mining tasks. We introduce the structures of them below.

5.1.2 Recurrent neural network

The recurrent neural network is designed to deal with sequence classification problems [2–7 , 34]. We use the network to traverse all the query terms based on their sequence in the query, and makes predictions based on historical information from query terms along the sequence. Formally, the model is defined as follows. $a_{i}^{I H} = f (Θ^{I H} [a_{i - 1}^{I H}, v_{q_{i}}] + b^{I H})$ (2) where v_{q
_i} is the feature vector of the i^th query term. $a_{i}^{IH} \in R^{N}$ is the outputs of the i^th unit. Θ^IH ∈ R^N×(N+M) is the parameter matrix of hidden units. b^IH ∈ R^N is the bias. f is the nonlinear activation function. In our experiments, we use tanh as the activation function. The vector representations of queries are constant during the model training. Since recurrent neural network is designed to tackle sequential problems in time series, we specify the start and end time before training. We specify the outputs of the start time as zero vector, namely, $a_{0}^{IH} = 0$ . We assign a signifier # END as the end time, namely, v_{q
_|Q|+1} = 1. The outputs at the end time are the representations of the entire query, $v_{| Q |} = a_{| Q | + 1}^{IH}$ , which are then taken as the inputs of Softmax for classification.

We provide an example of the recurrent neural network in Fig. 3. In the figure, we remove the superscript ’IH’ for simplicity. We input the query ’where buy an apple’ in the example. Our model computes the outputs of hidden layer as ${a_{1}^{IH}, a_{2}^{IH}, a_{3}^{IH}, a_{4}^{IH}, a_{5}^{IH}}$ to obtain the final representation of the entire query $v_{Q} = a_{5}^{IH}$ and the probabilistic distribution z^l on intermediate categories.

Fig. 3

Example of the recurrent neural network.

5.1.3 Recursive neural network

The recursive neural network can also be used to tackle the problem of sequence classification [34]. The main difference between recursive neural network and recurrent neural network lies in that recursive neural network traverses all the query terms based on any given sequence, in which all the query terms are taken as leaf nodes of a recursive tree. Therefore, recursive neural network is more flexible in dealing with the input terms, but may ignore sequential information of the original query compared to recurrent neural network. We learn the recursive neural network based on syntactic dependence binary tree of each query, which captures the syntactic relationships among query terms. Long queries with complex syntactic structures may particularly be addressed by the recursive neural network to capture the dependence relationship among terms. We define the structure of the used recursive neural network as follows. $a^{IH} = f (Θ^{IH} [n_{l}, n_{r}] + b^{IH})$ (3) where n_l and n_r is the vector representations of left node and right node in a binary branch of the tree. The vector representations of queries are constant during the model training.

We provide an example of the recursive neural network in Fig. 4. In the figure, we also remove the superscript ‘IH’ for simplicity. We input the query ‘where buy an apple’ in the example. Our model computes the outputs of ‘where buy’ and ’an apple’ as $a_{1}^{IH}$ and $a_{2}^{IH}$ to obtain the final representation of the entire query $v_{Q} = a_{3}^{IH}$ and the probabilistic distribution z^l on intermediate categories.

Fig. 4

Example of the Recursive neural network.

We use a Softmax layer as the outputs of both models, which outputs probabilistic distributions on intermediate categories. The probability of a query Q to i^th class can be computed as follows. $z_{i}^{I} = p (c_{i}^{I} | Q) = \frac{\exp (a_{i}^{IO})}{\sum_{j} \exp (a_{j}^{IO})}$ (4) $a^{IO} = f (Θ^{IO} v_{Q} + b^{IO})$ (5) where Θ^IO is the parameter matrix of the output layer and b^IO is the bias vector.

5.2. Target categorization oriented matching

In the second stage of our model, we map the intermediate categories of query intents to the target categories. The target categories are determined based on Wikipedia [35], which includes 14 primary categories. To make the our model more robust, we take both the outputs of the first stage and the query representations as inputs of the second stage. We employ a multi-layer perceptron as the classifier in this stage, which contains one hidden layer and one Softmax layer. In our preliminary experiments, we have examined the effectiveness of the proposed network with more hidden layers. However, it brings more time cost on model training, but yields no significant improvement on the classification performance. Therefore, we use only one hidden layer in the perceptron. The inputs of the perceptron are the concatenation of the original query representation and the distribution on intermediate categories. The dimensionality of the inputs equals to the sum of the dimensionality of the query representation and the number of intermediate categories. We illustrate the mapping from intermediate categories to the target categories in Fig. 5. The hidden layer and the Softmax layer can be formalized as follows. $a^{TH} = f (Θ^{TH} [z^{I}, v_{Q}] + b^{TH})$ (6) $z^{T} = softmax (Θ^{TO} a^{TH} + b^{TO})$ (7)

Fig. 5

Softmax model for classification.

where z^I represents the distribution on intermediate categories. z^T represents the distribution on target categories.

Overall, our framework adopts the recurrent neural network and the recursive neural network, respectively, for query intent mining. We name these two models as HDNN-Recurrent and HDNN-Recursive, respectively.

6. Experiments

6.1. Experimental settings

In this section, we conduct sufficient experiments to evaluate our method. We first introduce the dataset used in two stages, respectively. The first stage uses AOL query logs and ODP to generate intermediate categories on query intents. We use 4 million queries as the training set, 0.1 million queries as the testing set, and remaining queries as the validation set. The dimensionality of outputs for this stage is set to be 219, which equals to the number of secondary categories of ODP.

For the second stage, we use the target categories based on Wikipedia [35], which includes 14 primary categories. We use 500 queries from ad-hoc track of Text REtrieval Conference (TREC), and label the queries manually by three experts. The final category of each query is determined by voting. The dimensionality of target categories is 14. We perform five-fold cross validations to achieve the average performance of all the queries. These amount of data used in these two stages are sufficient to evaluate the proposed method in this work. Since the annotations on query intents cost much manual labor, we will explore effective ways for automatic intent annotation in future, which would facilitate the model comparisons on larger datasets.

We use the skip-gram model [36] to obtain word representations based on Wikipedia datasets in 100 dimension. We use Keras framework with Theano as backend to implement the proposed deep learning models. At the model training, we adopt cross entropy as the cost function, and AdaDelta as the optimizer with early stop for dynamic adjustment of the learning rate. We set the initial learning rate as 1.0. The regularization parameter λ is empirically set to be 0.001. The size of batch is 10000, and the number of epoches is 500. Based on the difference of our two models in the first stage, we denote our two models as HDNN-Recurrent and HDNN-Recursive, respectively.

6.2. Baseline models and evaluation metrics

We choose four state-of-the-art models as baselines in our experiments, and adopt Precision, Recall, F1 and Accuracy as evaluation metrics to evaluate the performance of classification. We introduce the baseline models as follows.

(1) BridgeQC is a classic two-stage classification method for query intent classification [9]. In its first stage, we use ODP to map query to an intermediate category, and in the second stage, it adopts exact matching and SVM classifier to obtain the target query intents. A variation of the method has won the query intent classification task of KDDCUP [37].

(2) Vec-SVM takes the average of all the representations of query terms as the query representation, and use support vector machines (SVM) to train a classifier mapping queries to target categories directly. We take this method as a baseline to examine the effectiveness of two stage classification for query intent classification. (3) WVec-SVM is the weighted version of Vec-SVM, which takes weighted average of all the representations of query terms as the query representation. In our experiments, term weights are obtained based on the methods proposed in [8].

(4) HDNN-SVM is a modified version of our method using SVM as the classifier in the second stage. We use this method to examine the effectiveness of neural network based model in our method.

The baseline methods are implemented under the same framework as the proposed method for fair comparison. The support vector machines are trained with a linear kernel using l₂ penalty and a square hinge loss with C = 1, because relatively good performance can be achieved under this settings on the validation set. Since the proposed framework is easy to implement, other state-of-the-art models can also be compared with the proposed models in similar experimental settings. We will also explore other optimization strategies in our framework to further improve the classification performance.

6.3. Results and analysis

We report the experimental results in Table 4. As mentioned above, we categorize query intents based on Wikipedia. We perform five-fold cross validations for the Vec-SVM and WVec-SVM methods, and perform two-stage classifications for other methods using ODP-based intermediate categories with five-fold cross validations in the second stage.

Table 4
Classification Performance Comparisons

Methods Precision Recall F1 Accuracy Computational Cost (FLOPs)

Vec-SVM 0.674 0.502 0.553 0.547 1.110¹⁹

WVec-SVM 0.702 0.540 0.611 0.608 1.510¹⁹

BridgeQC 0.814 0.792 0.803 0.795 2.310¹⁹

HDNN-SVM 0.880 0.771 0.795 0.782 1.810¹⁹

HDNN-Recursive 0.888 0.870 0.875 0.875 1.710¹⁹

HDNN-Recurrent 0.922 0.907 0.911 0.909 1.910¹⁹

Methods	Precision	Recall	F1	Accuracy	Computational Cost (FLOPs)
Vec-SVM	0.674	0.502	0.553	0.547	1.1*10¹⁹
WVec-SVM	0.702	0.540	0.611	0.608	1.5*10¹⁹
BridgeQC	0.814	0.792	0.803	0.795	2.3*10¹⁹
HDNN-SVM	0.880	0.771	0.795	0.782	1.8*10¹⁹
HDNN-Recursive	0.888	0.870	0.875	0.875	1.7*10¹⁹
HDNN-Recurrent	0.922	0.907	0.911	0.909	1.9*10¹⁹

From the table, we observe that WVec-SVM outperforms Vec-SVM as one-stage classification methods, which indicates the usefulness of term weighting in query intent classification. Two-stage classification methods achieve better performance than one-stage methods. This finding implies that query representations purely based on summations of representations of query terms gain limited capability to interpret the query intents. The second stage contributes much to overcoming the lack in training instances, thus improving classification performance. We estimate the computational cost based on the number of floating point operations (FLOPs) used to train a model by multiplying the training time, the number of GPUs used, and an estimate of the sustained single-precision floating-point capacity of each GPU, which follows the way used in [38]. The comparison shows that the computational cost of our model is comparable to other baseline models.

Among all the two-stage methods, HDNN-Recurrent achieves the best performance, and gains 13.4% and 9.0% improvements over BridgeQC. HDNN-Recurrent achieves better performance than HDNN-Recursive. This indicates that recurrent neural network in the first stage is more useful to represent the queries. Sequential modeling on query terms yields more improvement in query representations than using the syntactic dependence trees.

To further analyze the effectiveness of our method, we plot the ROC curves of the compared models in Fig. 6, and report the classification performance in the first stage classification in Table 5. Figure 6 plots true positives versus false positives retained by setting different cutoffs. The curves show that the proposed method can yield better performance compared with other baseline models. HDNN-Recurrent outperforms other models under different false positive rates. The trends of performance of these models are consistent with that reported in Table 4. The results indicate that the proposed models can improve the classification accuracy of query intents, and the recurrent network based model is more effective.

Fig. 6

ROC curves for the compared models.

Table 5

Classification Performance for the First Stage

Methods	Precision	Recall	F1	Accuracy
BridgeQC	0.744	0.612	0.672	0.658
HDNN-Recurrsive	0.804	0.690	0.742	0.738
HDNN-Recurrent	0.857	0.722	0.783	0.769

Table 5 shows that the proposed methods outperform BridgeQC in terms of all the evaluation metrics. This eventually affects the overall classification performance. The results also imply that intent labeling based implicit relevance feedback is effective in building intermediate categories on query intents.

7. Conclusions and future work

In the paper, we propose a hybrid deep neural network model for query intent classifications. The model contains two stages: the first stage adopts ODP to generate intermediate category, and employs recurrent and recursive neural networks for query representations; the second stage adopts Wikipedia-based category as target category, and employs a multilayer perceptron to train the classifiers. Experimental results show that our method outperforms state-of-the-art baseline models and improves the performance of query intent classification. Experimental results demonstrate that ODP-based query intent labels based on query logs improve the performance of classification, and hybrid deep neural networks are effective in representing queries. We will carry out our future work to further optimize our method using relevance feedback to improve the classification effectiveness. We will also explore effective ways for automatic query intent annotation, which would facilitate further model comparisons on larger datasets and model optimization on other domain-specific tasks.

Copyright

Authors submitting a manuscript do so in the understanding that they have read and agreed to the terms of the IOS Press Author Copyright Agreement posted in the ‘Authors Corner’ on www.iospress.nl.

Footnotes

Acknowledgments

This work is partially supported by grant from the Ministry of Education Humanities and Social Science Project (No.19YJCZH199), the Natural Science Foundation of China (No. 61632011, 61572102, 61602078, 61572098), China Postdoctoral Science Foundation (No. 2018M641691), the Fundamental Research Funds for the Central Universities (No. DUT18ZD102), the National Key Research Development Program of China (No. 2016YFB1001103).

References

Z.T.

Kardkovacs ,

Tikk and

Bansaghi , The ferrety algorithm for the KDD cup 2005 problem, ACM SIGKDD Explorations Newsletter 7(2) (2005), 111–116.

J.D.J.

Rubio ,

Lughofer ,

J.A.

Medacampana ,

L.A.

Paramocarranza ,

J.F.

Novoa and

Pacheco , Neural network updating via argument kalman filter for modeling of takagi-sugeno fuzzy models, Journal of Intelligent and Fuzzy Systems 35(2) (2018), 2585–2596.

Meng ,

Shi and

Yao , An inequality approach for evaluating decision making units with a fuzzy output, Journal of Intelligent and Fuzzy Systems 34(1) (2018), 459–465.

J.D.J.

Rubio , SOFMLS: Online self-organizing fuzzy modified least-squares network, IEEE Transactions on Fuzzy Systems 17(6) (2009), 1296–1309.

Zhang and

Han , State estimation for static neural networks with time-varying delays based on an improved reciprocally convex inequality, IEEE Transactions on Neural Networks 29(4) (2018), 1376–1381.

J.D.J.

Rubio , Interpolation neural network model of a manufactured wind turbine, Neural Computing and Applications 28(8) (2017), 2017–2028.

Jiang ,

Liang ,

Feng ,

Fan ,

Pei ,

Xue and

Guan , Text classification based on deep belief network and softmax regression, Neural Computing and Applications 29(1) (2018), 61–70.

Shen ,

J.-T.

Sun ,

Yang and

Chen , Building bridges for web query classification, In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2006, pp. 131–138.

J.-T.

Sun , Q2C@UST: Our winning solution to query classification in KDD Cup 2005, ACM SIGKDD Explorations Newsletter 7 (2005), 100–110.

10.

Puigcerver ,

A.H.

Toselli and

Vidal , Querying out-of-vocabulary words in lexicon-based keyword spotting, Neural Computing and Applications 28(9) (2017), 1–10.

11.

Singh and

Sharan , A new fuzzy logic-based query expansion model for efficient information retrieval using relevance feedback approach, Neural Computing and Applications 28(9) (2017), 2557–2580.

12.

Okan Sakar ,

Olcay Polat ,

Katircioglu and

Kastro , Real-time prediction of online shoppers’ purchasing intention using multilayer perceptron and LSTM recurrent neural networks, Neural Computing and Applications (12) (2018), 1–16.

13.

Abid ,

Hussain ,

Abid ,

Ahmad ,

M.S.

Farooq ,

S.A.

Khan ,

Y.D.

Khan ,

M.A.

Naeem and

Sabir , A survey on search results diversification techniques, Neural Computing and Applications 27(5) (2016), 1207–1229.

14.

S.M.

Beitzel ,

E.C.

Jensen ,

Chowdhury and

Frieder , Varying approaches to topical web query classification, In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2007, pp. 783–784.

15.

A.Z.

Broder ,

Fontoura ,

Gabrilovich ,

Joshi ,

Josifovski and

Zhang , Robust classification of rare queries using web knowledge, In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2007, pp. 231–238.

16.

Ashkan ,

C.L.A.

Clarke ,

Agichtein and

Guo , Classifying and characterizing query intent, In European Conference on Information Retrieval, Springer, 2009, pp. 578–586.

17.

Radlinski ,

Szummer and

Craswell , Inferring query intent from reformulations and clicks, In Proceedings of the 19th International Conference on World Wide Web, ACM, 2010, pp. 1171–1172.

18.

Law ,

Mityagin and

Chickering , Intentions: A game for classifying search query intent, In CHI’09 Extended Abstracts on Human Factors in Computing Systems, ACM, 2009, pp. 3805–3810.

19.

Jiang ,

K.W.-T.

Leung and

Ng , Query intent mining with multiple dimensions of web search data, World Wide Web 19(3) (2016), 475–497.

20.

Yao ,

Liu and

Geng , Query-specific optimal convolutional neural ranker, Neural Computing and Applications (2017), 1–10.

21.

Ding ,

Xin and

Wang , An efficient query processing optimization based on elm in the cloud, Neural Computing and Applications 27(1) (2016), 35–44.

22.

Li ,

Wang ,

Shen and

Acero , Learning with click graph for query intent classification, ACM Transactions on Information Systems 28(3) (2010), (12.

23.

Ayadi ,

Torjmenkhemakhem ,

Daoud ,

J.X.

Huang and

M.B.

Jemaa , Mining correlations between medically dependent features and image retrieval models for query classification, Journal of the Association for Information Science and Technology 68(5) (2017), 1323–1334.

24.

Y.-B.

Kim ,

Stratos and

Sarikaya , Scalable semi-supervised query classification using matrix sketching, In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, volume 2, 2016, pp. 8–13.

25.

Barua and

Patel , Named entity classification using search engine’s query suggestions, In European Conference on Information Retrieval, Springer, 2017, pp. 612–618.

26.

J.-Y.

Jiang and

P.-J.

Cheng , Classifying user search intents for query auto-completion, In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, ACM, 2016, pp. 49–58.

27.

A.R.

KhudaBukhsh ,

P.N.

Bennett and

R.W.

White , Building effective query classifiers: A case study in self-harm intent detection, In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, ACM, 2015, pp. 1735–1738.

28.

Ali and

Gul , Search engine effectiveness using query classification: A study, Online Information Review 40(4) (2016), 515–528.

29.

Shi ,

Yao ,

Tian and

Jiang , Deep LSTM based feature mapping for query classification, In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016, pp. 1501–1511.

30.

Fariha ,

S.M.

Sarwar and

Meliou , SQuID: Semantic similarity-aware query intent discovery, In Proceedings of the 2018 International Conference on Management of Data, ACM, 2018, pp. 1745–1748.

31.

Cai ,

Zhu ,

Ji ,

Hao ,

Yan and

Liu , An CNN-LSTM attention approach to understanding user query intent from online health communities, In IEEE International Conference on Data Mining Workshops (ICDMW), IEEE, 2017, pp. 430–437.

32.

Glater ,

R.L.T.

Santos and

Ziviani , Intentaware semantic query annotation, In Proceedings of the 40^th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2017, pp. 485–494.

33.

Mikolov ,

Sutskever ,

Chen ,

G.S.

Corrado and

Dean , Distributed representations of words and phrases and their compositionality, In Advances in Neural Information Processing Systems, 2013, pp. 3111–3119.

34.

Schmidhuber , Deep learning in neural networks: An overview, Neural Networks 61 (2015), 85–117.

35.

Zheng and

Callan , Learning to reweight terms with distributed representations, In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, 2015, pp. 575–584.

36.

Hu ,

Wang ,

Lochovsky ,

J.-T.

Sun and

Chen , Understanding user’s query intent with wikipedia, In Proceedings of the 18th International Conference on World Wide Web, ACM, 2009, pp. 471–480.

37.

D.S.

Vogel ,

Bickel ,

Haider ,

Schimpfky ,

Siemen ,

Bridges and

Scheffer , Classifying search engine queries using the web as background knowledge, ACM SIGKDD Explorations Newsletter 7(2) (2005), 117–122.

38.

Vaswani ,

Shazeer ,

Parmar ,

Uszkoreit ,

Jones ,

A.N.

Gomez ,

Kaiser and

Polosukhin , Attention is all you need, In Neural Information Processing Systems, 2017, pp. 5998–6008.

A hybrid deep neural network model for query intent classification

Abstract

Keywords

1. Introduction

2. Related work

3. General framework

Table 1 Feature Definitions on Query Terms Features Notations Dimensionality Word representation vec q d Term weighting weight q 1 Part-of-speech pos q 1 Depths of nodes depth q 1 Feature vector vector q = (vec q , weight q , pos q , depth q ) m = d + 3

5.1. Intermediate categorization oriented mapping

5.1.1 Query intent labeling based on query logs

6.1. Experimental settings

6.2. Baseline models and evaluation metrics

6.3. Results and analysis

Copyright

Footnotes

Acknowledgments

References

Table 1
Feature Definitions on Query Terms

Features Notations Dimensionality

Word representation vec _q d

Term weighting weight _q 1

Part-of-speech pos _q 1

Depths of nodes depth _q 1

Feature vector vector_q = (vec_q, weight_q, pos_q, depth_q) m = d + 3