Intention mining: A deep learning-based approach for smart devices

2.1. Effectiveness of wearable devices

A vast majority of studies focus on evaluating the accuracy or reliability of wearable devices for patients. For instance, notable studies [30,31] have evaluated the accuracy of Apple Watch, Fitbit Charge HR, Samsung Gear S, and Mio Alpha, to measure heart rate. Another significant study [23] has evaluated and compared the accuracy of two wearable devices to track the activities among patients with Parkinson’s disease. Furthermore, an evaluation of the effectiveness of these devices has been conducted in various other healthcare settings, such as Aerobic exercises [15], monitoring ICU patients [22], and high-risk surgical patients [4].

It is observed that several studies have been conducted using Fitbit, which is one of the most widely used activity monitoring devices in the market [9]. For instance, an investigation has highlighted that Fitbit reasonably meets the validity and reliability standards, and therefore can objectively be used by primary care physicians to monitor patients’ physical activities [9]. Furthermore, similar to other devices, studies have been conducted using Fitbit in different settings, such as during treadmill walking [27], depressed alcohol-dependent women [1], physical activities of women [5], and in post-cardiac surgery population [8].

2.2. Adoption of wearable devices

Several studies [18,20,25] have examined the adoption of wearable devices. For instance, a recent study has explored the contributing and inhibiting factors that influence the adoption of wearable devices using Google Glass and Sony Smartwatch 3, as representative devices. Some studies, such as [18,25], have investigated the adaption of wearable devices including Fitbit, from an end-user perspective. However, these studies have the following limitations: (a) They either focus on evaluating the users’ acceptance on the conceptual level, rather than their perception of the actual usage of the devices [20]. (b) They either empirically evaluate the tendency of users to recommend the technology to other users [28], or acceptability of these technologies [7], rather than relying on the actual comments of users that are available in natural language text. Therefore, none of these quantitative evaluations can be used to develop deeper insights into users’ intention.

4.1. The protocol

The inputs to our innovative protocol are online-portals of two widely-used retailers, whereas the output is our i-Recommend corpus. As a part of the protocol, we scrapped the users’ feedback from the retailers’ webpages, extracted natural language text, and tokenized it to generate a large corpus of sentences. Given that the manual evaluation of such a large corpus is a cumbersome task, an innovative approach was employed to identify a realistic subset of sentences that can be evaluated manually. An overview of the protocol is depicted in Fig. 2.

Starting from the large pool of sentences, firstly, each sentence was tokenized to generate unigrams and the frequency of each unigram was computed from the sentence corpus. Two researchers independently screened the unigrams to identify the ones that may represent a recommendation. Subsequently, all the sentences that contained the identified unigrams were extracted for the manual evaluation. Furthermore, those unigrams whose surrounding words could potentially be used as a recommendation were also identified. These unigrams are referred to as ‘candidate unigrams’.

Secondly, the bigrams that include candidate unigrams were generated from the sentence corpus and the frequency of each bigram was computed. Subsequently, two researchers independently screened the bigrams to identify the ones that represents a recommendation, as well as candidate bigrams. The screened bigrams, as well as the candidate bigrams, were used to identify the set of sentences that should be manually evaluated for a recommendation.

Finally, a set of concrete guidelines were developed to annotate each sentence as ‘recommendation’ or ‘not recommendation’. Accordingly, the generated guidelines were used to manually annotate the realistic subset of sentences, which resulted in the development of the first-ever i-Recommend corpus. We contend that the generated benchmark can be used for learning and subsequently predicting users’ intention to ‘recommend’ or ‘not recommend’ a device. The details of the protocol are as follows:

4.2. Generating the i-Recommend corpus

For generating the i-Recommend corpus, we scrapped the user comments from the web portals of two widely used retailers, Amazon and Google Express. The primary reasons for the choice of the retailers were their world-wide presence, large customer base, and the market value of these corporations. For example, the Amazon portal is widely acknowledged as the most valuable global retailer [29] and the most valuable public company in the US, having the largest market of smart devices [2]. Furthermore, recent data show that half of the web shoppers go to Amazon for searching products and 50% of the US consumers use it for holidays shopping [3].

The scrapping of comments was limited to three widely-used and state-of-the-art wearable devices, Apple Watch Series 3, Samsung Galaxy Smart Watch, and Fitbit Alta HR. The reason for selection of these watches stems from the reputation, as well as the market value of the products. For instance, according to the IDC, Apple shipped over 10.4 million watches in the fourth quarter of 2018 [16]. Furthermore, IDC announced that Fitbit and Samsung Watches are among the top five best selling wearable devices for the year 2019 [16]. Additionally, a leading business platform (statista.com) has shown that the top three vendors, in terms of the market share of smartwatch units shipped during the second quarter of 2019, are Apple Inc., Samsung, and Fitbit Inc. [24].

A Python script was used to scrap the web and store the source content in a text file. Subsequently, data cleansing was performed using another script that identified the junk snippets, such as html tags and non-English text. Keeping in view the ethical aspects, we only extracted the real-world and publicly available data that is written as natural language text to avoid any proprietary claims. Also, the comments posted from March 1, 2017 to August 18, 2019, were extracted.

Our screening of the cleaned data revealed the diversity and complexity of comments. Some comments were composed of a single sentence and few words, whereas, other comments were composed of over one dozen sentences. The single sentence comments posed less complexity as it was relatively easy to determine their ‘recommendation’ or ‘not-recommendation’ status. On the contrary, the long comments were more complex as it was not always possible to determine whether the overall intention of the user is to ‘recommend’ or ‘not recommend’ the device. To address this complexity, all the comments were broken down into individual sentences, where punctuations were considered as sentence boundaries. Accordingly, we generated a large collection of 101,907 sentences. The details of the dataset are presented in Table 1.

As discussed in the preceding section, to reduce the number of sentences to a manageable level, firstly, we tokenized each sentence into words and computed the frequency of each unigram. In total, there were 10,693 unigrams, whose frequency varied between 1 and 22,280. Two researchers manually screened each unigram having a frequency greater than or equal to 5 to identify the ones that represent a recommendation. Note that the variations of spellings, abbreviations, or synonyms of recommendation were included in the recommendation. Consequently, we identified three types of unigrams: (i) that represents an intention-to-recommend are categorized Type A, (ii) the may be used in a sentence with intention-to-recommendation are categorized as Type B, and (iii) the remaining unigrams are categorized as Type C. Using the unigrams of Type B, we identified the bigrams by pairing the preceding and subsequent tokens.

Secondly, using the bigrams, that were generated using unigrams of Type B, we computed their frequencies. As a result, 6389 bigrams were identified, whose frequency varied between 1 and 510. Like the screening of unigrams, two researchers manually screened bigram to determine the type of bigram i.e. Type A and B. After that, all the sentences with the unigrams of Type A, bigrams of Type A and B (total of 156 n-grams), were extracted for manual screening and further development of i-Recommend corpus. A key benefit of employing this annotative approach is that instead of annotating 101,907 sentences, we merely annotated 7947 sentences. The details of the sentence corpus are presented in Table 2. The table includes the number of sentences extracted using 156 n-grams, the source of the sentences, as well as the target products of the users’ comments.

A pre-requisite to the unambiguous and consistent annotation of sentences is a set of clearly defined guidelines. As established in several existing studies [19,24], higher inter-annotator-agreement between two researchers, while working independently, is a key indicator for such an unambiguous and consistent annotation [13]. To that end, we randomly selected a sample of sentences (10% sample) from each data source, Amazon and Google Express. Subsequently, two researchers independently screened the sample and developed an initial set of guidelines, which were discussed by both researchers. Subsequently, a new random sample of 10% sentences was selected and two researchers annotated the same sample independently using the defined guidelines. The inter-annotator agreement between the two researchers was computed using the Kappa statistic [13].

Based on the Kappa statistic score, confusion matrix, and synthesis of the inconsistent annotations, the initial set of guidelines were refined. The improvements involved adding new guidelines and refining existing guidelines. An example of each type of improvement is as follows:

As a result of the two iterations, a high inter-annotator agreement of over 0.76 was achieved. The finalized guidelines that are used in the rest of the study are presented in Table 3. In essence, a sentence will be declared as a recommendation if and only if it explicitly gives a recommendation in favor of or against purchasing the product. If the person shows intention to buy another product, gift the product to another person, or spreads awareness about buying or not buying the product is also a recommendation. On the contrary, a sentence that merely praises the product, its specific feature, the quality of service delivered by the retailer or the manufacturing, is not a recommendation. Furthermore, if the person expresses that he/she received it as a gift, or expresses his dissatisfaction with the product, it is marked as not recommendation.

4.4. Specifications of corpus

Table 4 summarizes the specifications of the i-Recommend corpus. It can be observed from the table that the corpus is composed of 2586 positive examples and 5364 negative examples. Given the higher significance of the two types of examples for learning and prediction, the detailed distribution of these examples is presented in Table 5. It can be observed from the table that the corpus includes positive examples from both retailers, as well as for each product.

For a further comparison of the data collected from the two retailers, we extracted their vocabularies and frequency distribution of each token. We removed punctuations and remove stop words, and developed a word-cloud of each source. Figure 3 shows the word-cloud of the top 70 words from the Amazon corpus, whereas, Fig. 4 shows the word-cloud of the top 70 words from the Google Express corpus.

It can be observed from the two figures that there is a substantial difference between the two word-clouds. For example, the most visible common words in the two clouds include, watch, Fitbit, one, great, day, and use. Whereas, the most visible words that can be found in Amazon but not in Google Express are heart rate, feature, time, and charge. Similarly, there are several words that are found in Google Express sentences but not in Amazon. It includes, highly recommended, phone, bought, recommend, good, day, and buy. The differences in the vocabularies of two retailers justify the choice of the data sources for comprehensiveness.

OPEN IN VIEWER

Fig. 3.

Word-cloud of Amazon sentences.

OPEN IN VIEWER

Fig. 4.

Word-cloud of Google sentences.

5.1. Dataset

For the experimentation, we have used our newly developed i-Recommend corpus which is composed of 7947 sentences, including 2586 positive examples and 5364 negative examples. It is important to highlight that we have not used the very large source corpus of 101,907 sentences due to a very high imbalance ratio of 1:39. The reason for avoiding imbalance dataset is that it impedes the performance of machine learning techniques, as the imbalanced dataset does not offer equal opportunities for learning and prediction of both classes. Also, to cater the impact of imbalanced classes in the existing dataset (7947 sentences), we have used Pre-cision, Recall and F1 score as evaluation measures.

5.2. Evaluation measures

To evaluate the effectiveness of deep learning techniques for their ability to distinguish between two types of sentences, we have adopted three measures Precision, Recall, and F1 score. These measures are widely used in the information interval, as well as the natural language processing domain, to evaluate the effectiveness of supervised learning techniques.

To eliminate the bias that may have been induced due to the choice of training and testing sample, we have performed 10-folds cross-validation. For each fold, we computed Precision, Recall, and F1 scores, and subsequently computed the average score of the 10-folds.

For a formal specification of the evaluation measures, let S m be the complete set of sentences in our i-Recommend corpus that is used for experimentation. Such that S m = S r m ∪ S n m , where S r m is the set of sentences that have the intention-to-recommend, S r m is the set of sentences that do not have the intention-to-recommend, in the human benchmark. Furthermore, consider S t be the complete set of sentences tagged by a deep learning technique t, such that S t = S r t ∪ S n t , where S r t is the set of sentences that are predicted by a technique t having intention-to-recommend, and S r t is the set of sentences that do not have the intention-to-recommend, as predicted by the technique.

In the context of this study, Precision is the fraction of comments that are correctly classified by a deep learning technique, whereas, recall is the fraction of comments that should have been correctly classified by the technique. Formally, the Precision and Recall are defined as follows: Precision = | S r t ∩ S r m | | S r t | Recall = | S r t ∩ S r m | | S r m | Thus, the F1 score is the harmonic mean of Precision and Recall, which is formally defined as follows: P = 2 ∗ Precision ∗ Recall Precision + Recall

5.3. Word embeddings

Word embeddings are features learning techniques in natural language processing in which words are typically mapped to vectors of real numbers [21]. It is widely acknowledged that these representations have played a pivotal role in making major breakthroughs for various NLP tasks [6]. Recognizing the importance of these representations, leading corporations, such as Facebook Inc. and Google, as well as a leading research group (NLP group at Stanford University) have developed techniques for generating word embeddings. Both corporations and the research group have generated embeddings using billions of tokens and made them publicly available for research and development. Furthermore, they have made their techniques publicly available for generating new embeddings.

As shown in Table 6, this study has used five types of word embeddings, which include three types of existing word embeddings and two types of word embeddings that have been generated for this study. The existing word embeddings are as follows: (a) Facebook released word embeddings that are generated using fastText technique, (b) Google released word embeddings that are generated using Word2vec, and (c) the word embeddings available in Keras, an established API for generating embeddings.

In addition to these three existing embeddings, we have also generated two types of word embeddings. These embeddings are generated using two established techniques, Word2vec and fastText, and our seed corpus that is composed of over one million tokens and 101,907 sentences. For generating the embeddings we used the context of size 5 and vectors of 300 dimensions.

For the experiments, we used training to the testing ratio of 70:30, meaning that 70% of the randomly selected sentences were used for training and the remaining 30% sentencing were used for testing. Furthermore, we repeated the experiments ten times, formally called 10-folds cross-validation, to neutralize any bias in the dataset. The results presented in this paper are an average of the ten iterations that we have performed. The experiments were performed using state-of-the-art deep learning techniques Convolutional Neural Networks (CNN) and a specific variant of Recurrent Neural Networks (RNN) i.e. Long Short-Term Memory (LSTM).

For RNN, the context window of 5 words and 100 dimensions were used, whereas, for CNN the context window of 5 words and 300 dimensions were used. Additionally, a dropout layer of 10%, Relu activation function was used in the hidden layer, and Softmax activation was used in the output layer. The other details of the settings are presented in Table 7.

OPEN IN VIEWER

Fig. 5.

Comparison of precision and recall scores.