Evaluating a Bilingual Text-Mining System With a Taxonomy of Key Words and Hierarchical Visualization for Understanding Learner-Generated Text

Abstract

This study evaluated a bilingual text-mining system, which incorporated a bilingual taxonomy of key words and provided hierarchical visualization, for understanding learner-generated text in the learning management systems through automatic identification and counting of matching key words. A class of 27 in-service teachers studied a course “e-Learning in primary mathematics” was asked to reflect “what is e-Learning” before and after the course. Their concept of “e-Learning” was investigated by counting the matching key words using the text-mining system and a content analysis of learner-generated text using a rubric, respectively. The correlations of the results using these two methods were .823 and .840 in the preteaching and postteaching reflections. This text-mining system has the potential as a supporting tool for teachers to gain a general understanding of learner-generated text using the hierarchical visualization for supporting pedagogical decision-making, which can be applied to massive open online courses with a large enrolment of learners.

Keywords

bilingual text-mining system bilingual taxonomy of key words evaluation hierarchical visualization learner-generated text

The use of online discussion forums of learning management systems (LMSs) and social networks in virtual education environments not only enhances learners’ collaboration and discussion but also facilitates them to reflect their understanding of key concepts of domain knowledge. The growth of these learning environments well document what is actually happening in the learning process; this needs effective methods to process and comprehend these textual data (Gašević, Dawson, & Siemens, 2015). In addition to extracting statistics about learners’ participation and engagement from these learning environments, topic modeling based on semantic and linguistic methods has been adopted to analyze the quality of learners’ collaboration using their textual data, such as identifying the structure and topics of online-threaded discussions and supporting collaborative writing processes by providing visualization of discovered topics as feedback to learners (Southavilay, Yacef, Reimann, & Calvo, 2013). Since topic modeling enables teachers to discover the topics discussed by learners, various text-mining techniques have been applied to assess the conceptual knowledge of learners (Ming & Ming, 2015; Sherin, 2012; Williams & D’Mello, 2010).

The studies above incorporate the techniques of topic modeling in identifying learners’ conceptual understanding without using expert content knowledge. This study focuses on exploring learners’ conceptual understanding of a domain and provides a domain-specific model from learner-generated text about their degree of proficiency in a domain. Some previous studies have presented the construction of conceptual models for identifying learners’ domain concepts covered by their text in English (Dzikovska, Steinhauser, Farrow, Moore, & Campbell, 2014; Pérez-Marín & Pascual-Nieto, 2010), in which a conceptual model may be specified by a concept map, conceptual diagram, hierarchical model, or other knowledge representation formats. In these models, each concept is represented by a set of predefined answers or references.

Despite a variety of text-mining techniques have been presented for understanding learner-generated text, the majority of previous work has addressed the analysis of English text. Recently, text-mining techniques have been extended to Chinese text such as topic discovery (Zhao, Qin, Liu, & Tang, 2016), concept mining (Zhou & Wang, 2010), and domain ontology construction (Liu, He, Lim, & Wang, 2014). In Hong Kong, learners are allowed to choose the language, either English or Chinese, that they are able to express themselves well in the learning process. This poses challenges in analyzing these text data using existing text-mining techniques. Bilingual text mining for understanding domain concepts covered by learners’ text mixing of English and Chinese becomes a research agenda in this context.

Background of Study

Text-Mining Applications in Learner-Generated Text

The term text mining or knowledge discovery from text was first mentioned by Feldman and Dagan (1995) for the analysis of text supported by machine. There is an emerging trend of text mining which is to understand educational practices and research issues related to text generated from learners (Romero & Ventura, 2010). Data mining tools are usually designed to deal with structured data from databases. In contrast, text mining is focused on finding and extracting useful and interesting information from unstructured text. With the advancement of natural language processing techniques, the algorithms of text mining have been employed to extract and analyze learning-related text from online discussion forums for educational purposes such as extracting opinions in e-Learning systems (Song, Lin, & Yang, 2007), facilitating the automatic coding process for the study of a discussion forum (Lin, Hsieh, & Chuang, 2009), and developing a visual analytics system for students to overview their contributions in online discourse (Teplovs, Fujita, & Vatrapu, 2011).

In recent years, there are studies focusing on the understanding of learners’ conceptual models with learner-generated text. Learner-generated text in this study refers to text generated from discussions and self-reflections in the learning process using features such as discussion forums in LMSs. Pérez-Marín and Pascual-Nieto (2010) studied the automatic generation of learners’ conceptual models using a hierarchical structure of domain knowledge with learner-generated text. Before constructing this conceptual model for each learner, domain knowledge and the related concepts were defined by the teacher, in which a set of questions per concept and the corresponding correct answers or references were provided by the teacher. Lárusson and White (2012) developed an analysis method using a lexical reference to enable teachers to visualize writings of learners about the originality of words generated from learners. In this analysis, a set of key concepts covered by the lecture was manually input as the query terms to determine the originality of a learner’s blog post in an introductory course of Computer Science. Dzikovska et al. (2014) developed an adaptive feedback system supporting the analysis of learner-generated text to address the challenge of teaching conceptual topics in Physics. The system compared the objects and relations of topics in the answers of learners with the predefined reference answers to come up with feedback about the correctness, contradictory parts, missing chunks, and irrelevant fragments of answers. These studies indicate that the learners’ conceptual models distilled from learner-generated text match well with the manual model initially worked out by the teachers.

Multilingual Text Mining

The above techniques generally focus on the processing of English text from online discussions to understand and improve pedagogical practices. Notwithstanding, text mining on multilingual text data has been increasingly investigated and becoming a new trend in the field of text mining (Gupta & Lehal, 2009). Early studies on processing text corpora in different languages have focused on word matching across their languages. For example, Fung (1995) proposed a context heterogeneity measure between words and their translations to compile bilingual lexicon entries from a nonparallel English-Chinese corpus. In addition, statistical approaches are adopted to analyze bilingual text for word matching (e.g., Smadja, McKeown, & Hatzivassiloglou, 1996). Information extraction is one of the major functions in the field of text mining. Enormous efforts have been paid on tackling this issue for multilingual text. Riloff, Schafer, and Yarowsky (2002) presented the integration of cross-language projection into an information extraction system to learn the extraction rules for a new language automatically. Some studies have addressed the development of lexicon or key word extraction from text in different languages. Chu, Nakazawa, and Kurohashi (2014) proposed an approach to bilingual lexicon extraction from comparable corpora based on topical and contextual knowledge on Chinese-English and Japanese-English Wikipedia while Huang, Chen, and Yang (2015) presented a method to extract key words in a language with the help of the other by estimating preferences for topical key words and combining language-specific word statistics.

Recently, identifying underlying topics from text in different languages has become an active research area. Techniques derived from latent Dirichlet allocation (LDA) were commonly proposed to extract bilingual topics from unaligned corpora in two languages such as using term matching across languages (Boyd-Graber & Blei, 2009). Besides, a multilingual topic model with multilevel hyperpriors was proposed to capture the variation in topics across multilingual documents (Krstovski, Smith, & Kurtz, 2016). By detecting a set of common topics from multilingual nonparallel text data and finding the differences in perspectives on these topics across languages, a statistical model was developed to identify the cross-cultural differences (Gutiérrez, Shutova, Lichtenstein, de Melo, & Gilardi, 2016).

On the one hand, the above multilingual text-mining applications have focused on the development of automated techniques for information extraction, lexicon and key word extraction, and topic discovery from multilingual text. On the other hand, there are an increasing number of studies committed to show learners’ conceptual understanding through examining learner-generated text (Pérez-Marín & Pascual-Nieto, 2010; Lárusson & White, 2012; Dzikovska et al., 2014). However, the text-mining techniques attempting to understand conceptual model of learners are limited to text generated in a single language such as English only. In this regard, this study addressed the research gap to explore the use of a conceptual model together with the techniques of multilingual text mining to understand domain concepts by learner-generated text in LMS in multilingual context focusing on the use of both English and Chinese.

Hierarchical Visualization

Users are impeded to conveniently understand the content and structure of a huge amount of data in complex form. In this regard, techniques of visual representation have been studied to enable data to be represented in a manageable overview and allow a structure of visualization at a desired level of detail (de Oliveira & Levkowitz, 2003). For this purpose, Elmqvist and Fekete (2010) presented a model using hierarchical aggregation for information visualization to improve overview and scalability of large-scale visualization. Such hierarchically aggregated visualization provides users an overview of the data with multiscale representations to retrieve details of data at various levels on-demand (Shneiderman, 1996). Bostock, Ogievetsky, and Heer (2011) proposed the use of D3 (data-driven-documents) to build this hierarchical visualization through the web platform. Making use of scalable vector graphics, D3’s standardized representation enables not only better expressiveness and accessibility but also performance improvement using web browsers.

The Design of a Bilingual Text-Mining System With Hierarchical Visualization

This study focused on the application of a bilingual text-mining system with hierarchical visualization support for teachers to understand learner-generated text in the learning process. It involves text mining using a bilingual taxonomy of key words on a topic to identify and count the matching key words from their text. The results of the bilingual text mining are displayed in layers using a hierarchical structure for visualization of matching key words. The bilingual taxonomy of key words is set up initially by the teacher from a theoretical perspective. It is enriched by looking up relevant words and terms used by learners as well as key words discovered by LDA modeling.

Bilingual Text-Mining Design

Concept hierarchy is important for any domain knowledge because it allows learners to formulate relations in an abstract and concise way for facilitating the development, refinement, and understanding of knowledge (Cimiano, Hotho, & Staab, 2005). With a focus on exploring learners’ conceptual understanding, this study adopted a conceptual model using a hierarchical structure of a bilingual taxonomy on a domain to understand learner-generated text. Such bilingual taxonomy includes a set of concepts; and each concept consists of a number of subconcepts in order to capture domain knowledge with various granularities. This parent–child relationship provides an effective way to represent and organize concepts of a domain. For example, Figure 1 shows the hierarchical structure of a bilingual taxonomy on a domain with n concepts and each concept comprises of a number of subconcepts, where there are p and q subconcepts in the first and the nth concepts, respectively.

Figure 1.

The hierarchical structure of a bilingual taxonomy on a domain.

Setup of a bilingual taxonomy of key words

In Hong Kong, learners commonly use a mixture of English and Chinese in written communications, and therefore learner-generated text accumulated in LMSs usually includes English and Chinese words and terms. A bilingual taxonomy of key words is therefore adopted in this study to include a number of bilingual key words for each concept and subconcept for matching with text generated by the learners in order to comprehend their understanding of a domain. Referring to the work by Daems, Erkens, Malzahn, and Hoppe (2014) and Pérez-Marín & Pascual-Nieto (2010), the bilingual taxonomy adopted in this study sets a number of specific words and terms characterizing each concept and subconcept. When learners are asked to discuss their understanding of a domain, some of these key words will be covered in their elaborations of the related concepts and subconcepts of the domain. In other words, learners’ understanding of each concept and subconcept of the domain may be represented by the related key words used in their text. Since the concepts and subconcepts are structured hierarchically by this bilingual taxonomy, learners’ understanding of the domain can be overviewed and their understanding on any part and layer of the bilingual taxonomy can be identified. In this study, the bilingual taxonomy of key words is initially designed by the teacher through reviewing the related literature of the domain and the relevant curriculum.

Enrichment of bilingual taxonomy of key words by manual search and the support from LDA modeling

Although a bilingual taxonomy of key words is built according to teachers’ review of related literature and curriculum of the domain, learners are highly probable to use key words other than the ones listed in the bilingual taxonomy to express their understanding of the concepts and related subconcepts. Teachers therefore need to enrich the key words in the bilingual taxonomy by two ways: to extract key words directly by sampling a proportion of the learner-generated text in related LMSs and to use key words discovered by LDA modeling. LDA is a generative probabilistic model to discover latent topics in a document collection and was developed for topic discovery (Blei, Ng, & Jordan, 2003). The top words of latent topics from the LDA model may be employed to complement the original key words designed by the teachers. This study follows the recommendations by Blei et al. (2003) and Yin and Wang (2014) to preprocess the input text by part-of-speech tagging using the Stanford CoreNLP toolkit (Manning et al., 2014) in order to extract nouns, verbs, and adjectives from the input text irrespective of English and Chinese. The condensed text containing these three types of words is then processed by LDA to reveal the latent topics of the input text, where each topic is described by a probability distribution of words. With the reference to the words with high probabilities for each topic, the teachers can finalize the bilingual taxonomy of key words, which includes synonyms in both English and Chinese to accommodate the languages used in learner-generated text.

English and Chinese Text Processing

The input English text is first divided into a sequence of words, symbols, and other meaningful elements known as tokens by the Stanford CoreNLP toolkit (Manning et al., 2014). The Stanford Parser (Klein & Manning, 2003) is adopted for English word segmentation in this study. The process of chunking the sentence is then conducted by grouping the parser leaves with the same parent as a term. In the final step, lemmatization is performed so that the lemma of each word can be determined. Then, the words with the same lemma are grouped as a single item to improve the quality of word indexing.

Chinese text segmentation is the most important part in Chinese text processing since a Chinese term consists of several single words but without space between the words. For example, “短片” is considered as a sequence of two Chinese words, which means “video clip” rather than as individual words of “短” and “片,” which mean “short” and “slice,” respectively. This study adopted the conditional random field (CRF)-Lex Chinese word segmenter (Chang, Galley, & Manning, 2008) to split Chinese text into sequences of terms and identify the part of speech of these resulting terms, in which there are 423,224 distinct entries in the external lexicons to improve segmentation consistency of the CRF model.

Hierarchical Visualization Design

A hierarchical structure of a bilingual taxonomy is adopted for understanding the domain concepts covered by learner-generated text; text-mining results which contain those matching key words with counting figures are recorded in a hierarchical structural list. Hierarchical visualization in this study refers to the provision of hierarchically aggregated results for visualization through web browsers in which users can overview the data with multiscale representations to retrieve details of data at various parts and levels on-demand. These features enable users to understand the hierarchical results obtained from text mining. The method for hierarchical aggregation presented by Elmqvist and Fekete (2010) is adopted in this study to facilitate the reporting and the easy comprehension of the subordinate or membership relations among the matching key words obtained from the text-mining results. D3 (Bostock et al., 2011) is then employed to enable users to visualize the text-mining results with a large-scale structure interactively through web browsers.

This study adopted the technique presented by Teoh and Ma (2002) for a ringed circular layout of nodes to visualize the hierarchical results from text mining; and a click on the visualization can manage the viewing through the different hierarchical layers. In such visualization model, each concept or subconcept is represented by a blue ringed circular layout with balls inside. The blue color scheme of getting deeper in each layer is iteratively in line with more layers of subconcepts (for details, see the part “Results and Discussion”). Each ball denotes a set of bilingual key words with similar scope. The ball is orange in color if at least one of these key words matches with words or terms mentioned by learners and is gray in color otherwise. The size of an orange ball is proportional to the matching counts of these key words. The matching of learner-generated text with the key words of any concept/subconcept can be studied by viewing the number of orange balls together with the sizes of the orange balls which reveal, to certain extend, learners’ conceptual understanding of the domain.

The Use of Bilingual Text-Mining System to Examine Learner-Generated Text

Figure 2 shows users’ view on the bilingual text-mining process with three parts, namely, text extraction, using the bilingual text-mining system and hierarchical visualization. Text data are generated in LMSs such as text produced from learners in discussion forums of these platforms. In order to conduct text mining, learner-generated text is needed to be extracted from these platforms. In this study, a Moodle plug-in is designed for teachers to extract the text data from discussion forums for text mining. The bilingual text-mining system of this study consists of a text-mining interface, text-mining server, and database. The web-based text-mining interface is developed by PHP 5.5. There are five webpages in this user interface, namely, “template download,” “manage key word,” “import text,” “text mining,” and “result” (see Figure 3). This user interface is designed to facilitate the preparation of text data for mining which has been extracted from LMSs. The interface also has functions for the design and input of bilingual taxonomy of key words, the administration of the text-mining process, and the postprocessing of text mining for generating hierarchical visualization results.

Figure 2.

Users’ view on text extraction, using the bilingual text-mining system and hierarchical visualization of the bilingual text-mining process.

Figure 3.

The “Result” webpage of the user interface of the text-mining system.

Through the web-based interface, users can instruct the server to perform text mining through the Internet without installing the required programs on users’ own computers. The text-mining interface enables the matching of the words and terms in learner-generated text with the ones in the bilingual taxonomy and therefore is able to identify and count the matching key words. The database is built using MySQL 5.6 to store text-mining data including learner-generated text, bilingual taxonomy of key words, and text-mining results, so that users are enabled to access these text-mining data through the network. The hierarchical results produced from text mining are saved in text file format which can be viewed and downloaded from the “Result” webpage. In order to assist users to browse the text-mining results conveniently, they are enabled to produce their online visualization of the hierarchical results interactively through web browsers with the text-mining interface of the system (for details, see Figure 5).

Evaluating the Bilingual Text-Mining System With Hierarchical Visualization

Method

Two evaluation studies were conducted in order to validate the usefulness of the bilingual text-mining system with hierarchical visualization. One study was to evaluate learners’ domain concepts by counting the number of matching key words using the text-mining system in the preteaching and postteaching reflections. The other study was to conduct a content analysis of the two reflections. In content analysis, researchers are required to code and quantify the findings of a research topic in order to extract information from the qualitative data sets (Newby, 2014). In this study, two researchers who were familiar with the rubric designed for this study were assigned as the coders to independently code and score learners’ reflections and then discussed discrepancies on their scoring results in order to obtain a consensus on the final scores. The Cronbach’s α reliability coefficient between the two coders was .828.

Participants

The evaluation involved 27 in-service teachers who joined a 5-week full-day intensive professional development course on “e-Learning in primary mathematics.” These in-service teachers had 13 years of teaching experience on average (SD = 7.265 years), and they were majoring in the teaching of mathematics. By a statistical power analysis, the effect sizes in the text mining and content analysis were 1.480 and 2.679, respectively, considered as very large according to Sawilowsky (2009). The sample size with 27 participants is further justified to be sufficient for high-power statistical tests (α = .05 and power = .8) in the text mining and content analysis.

Bilingual taxonomy of key words on e-Learning for counting matching words

To know the course learners’ understanding of the concepts of “e-Learning” in school education, they were asked to reflect “what is e-Learning” before and after the teaching of the course, namely, preteaching reflection and postteaching reflection, respectively, using the discussion forum of Moodle. Since learners were allowed to reflect both in English and Chinese, a bilingual taxonomy of key words on e-Learning with synonyms in both English and Chinese were designed. In this study, the key words were first drafted by this course teacher based on an e-Learning framework conforming to the current practice and researches of e-Learning in school education as well as relevant words and terms used by another class with 33 learners of the same course in the previous semester. These relevant words and terms were identified through manual search on relevant words and terms covered by the text from the previous class as well as topic modeling by LDA on the text from the previous class. Having drafted the bilingual taxonomy of key words, these key words were enriched by relevant words and terms used by this class of learners through manual search and topic modeling based on the learner text from this class. Thus, the key word enrichment process can be iterated to refine the bilingual taxonomy of key words if more text from other cohorts of same course was available. Table 1 shows the enriched key words which were found manually from learner-generated text in their reflections by this class of learners and they are grouped according to their scopes specified by the bilingual taxonomy.

Table 1.

Enriched Key Words Which Were Found Manually From Learner-Generated Text in Their Reflections in This Study.

Subconcepts	Key words
E-resources	1. “電子課本” (electronic textbook)
	2. “電子教材” (electronic teaching resource)
	3. “網上資源” (online resource)/“圖片”/“圖像” (image/photo/photograph/picture)/“影片” (video clip)
Digital ways of communication	4. “搜尋” (search)
Digital ways of collecting data	5. “批改” (mark)
Digital technology	6. “平板電腦” (tablet)
	7. “Padlet”/“YouTube”
	8. “上網” (Internet access)
	9. “數碼通訊” (digital communication)
	10. “數據收集” (data collection)/“上載” (upload)
Principles/models/theories/strategies	11. “以學習者為中心” (student centered)
Principles/models/theories/strategies	12. “探究式學習” (inquiry-based learning)
Activities	13. “通訊” (communication)
Activities	14. “回應” (response)
Pedagogical decision-making/support	15. “調整”/“調節” (adjust)
Digital pedagogy	16. “教學策略” (pedagogical strategy)/“教學模式” (pedagogical model)
	17. “數據處理” (data collection)/“數據分析” (data analysis)
	18. “數據表達” (data presentation)/“visualization tool”

LDA models with two to five topics were attempted for the support of key word enrichment based on learners’ preteaching and postteaching reflections from this class. For each LDA model, the highest probabilities of 20 words/terms for each latent topic were identified. Table 2 shows the top 20 words/terms from a 4-topic LDA model, where English words and Chinese terms were identified by the first topic and the remaining topics, respectively. Some Chinese terms in Topic 2 covered the use of digital technology for learning such as “電子工具” (electronic tool), “資訊科技” (information technology), “模式” (model), and “討論” (discussion) while some Chinese terms in Topic 3 associated with the collection of learning data for pedagogical supports such as “數據” (data), “教學法” (pedagogy), “收集” (collect), “評估” (assessment), and “分析” (analysis). The Chinese terms in Topic 4 mainly focused on using e-resources and digital ways of communication to facilitate learning activities including “活動” (activity), “網上資料” (online resources), “回饋” (feedback), “分享” (sharing), “平台” (platform), and “軟件” (software). Since the latent topics revealed by the 4-topic model were the most closely related to the subconcepts specified by the bilingual taxonomy among the attempted LDA models, the 4-topic model for the learner text from this class was adopted, and the highlighted words/terms in Table 2 were selected by the teacher to enrich the designed key words in this study. The consolidated words/terms include “electronic communication”, “電子工具” (electronic tool), “資訊科技” (information technology), and “數碼溝通” (digital communication). With these additional key words, the bilingual taxonomy of key words on e-Learning was finalized for this study and shown in Figure 4. The bilingual key words were placed in the innermost layer of the hierarchical structure and those key words with similar scope in learning were grouped together so that there were three layers in this bilingual taxonomy, in which the first, the second, and the third layers contained 2 concepts, 8 subconcepts, and 29 sets of bilingual key words, respectively.

Table 2.

The Top 20 Words/Terms From a 4-Topic LDA Model on Learners’ Preteaching and Postteaching Reflections.

Topic 1	Topic 2	Topic 3	Topic 4
Learning	學習	學習	利用
Student	學生	電子	電子
Learn	電子	教學	時間
E-learning	工具	溝通	老師
Teacher	進行	數碼	活動
Digital	可以	數據	學習者
Use	教學	資源	網上
Device	教師	教學法	資料
Tool	科技	工具	回饋
Process	不同	數學	分享
Pupil	課堂	課堂	促進
E-resource	資訊	知識	進行
Electronic	提供	收集	平台
Communication	模式	科技	教室
Pedagogy	增加	包括	作出
Facilitate	提升	運用	軟件
Way	配合	評估	空間
Concept	輔助	老師	預習
Technology	內容	認知	好處
Teaching	討論	分析	電腦

Note. LDA = latent Dirichlet allocation. The highlighted words/terms were selected to enrich the designed key words in this study.

Figure 5.

The hierarchical visualization of text-mining results of learners’ pre-teaching and post-teaching reflections of their understanding of e-Learning.

Rubric for content analysis of learner-generated text

The learners’ reflections were also evaluated by a content analysis to measure their understanding of e-Learning. In this regard, a rubric based on the e-Learning framework was developed for researchers to assign scores and quantify the quality of learners’ reflections on their understanding of the concept of “e-Learning” (for details, see rubric in the Appendix). Learners’ understanding in the “technology” concept was assessed by Components A and C of the rubric while the other two components were used to measure their understanding in the “pedagogy” concept.

Results and Discussion

After completing the process of text mining of the learner-generated text from preteaching and postteaching reflections, the matching key words were identified, counted, and reported using hierarchical visualizations (for details, see Figure 5), where a set of bilingual key words with similar scope in learning was represented by an orange ball if at least one of these key words was matched with learner-generated text and a gray ball otherwise. Note that each ball was labeled by the first key word of its corresponding set of key words (see Figure 4). Learners’ learning progress could be studied using this visualization tool. As Figure 5 shows, there were 15 sets of key words covered by the preteaching reflections while there were 27 sets of key words covered by the postteaching reflections. Also, the counting of the matching key words has increased from 107 to 410 after the teaching of the course. These figures indicated that learners had better understanding of e-Learning after studying the course because more key words of e-Learning were recognized, and these key words were used more frequently to elaborate their understanding of e-Learning in the postteaching reflections.

Figure 4.

The finalized bilingual taxonomy of key words on e-Learning for this study.

Since the hierarchical aggregation of matching key words was available through web browsers, the learning progress of any concept and subconcept of e-Learning could be examined visually based on the change of color and size of corresponding key word balls in the two reflections. For example, there were significant changes of the number and size of orange balls in the subconcept “digital ways of collecting data” of the technology concept including “text mining,” “summative data,” and “formative data,” as shown in the extended parts of Figure 5. This learning outcome was consistent with the emphasis of this concept in the course. The visualization also shows that some key word balls in “e-resources” and “digital ways of communication” subconcepts remained in gray color after the teaching of the course.

Further, the counts of matching key words provided a means for teachers to investigate learners’ understanding of any concepts and subconcepts of e-Learning before and after the teaching of the course using the proportions of matching counts at various levels of the bilingual taxonomy, as summarized in Table 3. At the first level, the proportion of matching counts in the pedagogy concept was

41 / 107 = 0.383

in the preteaching reflection. Before the teaching of the course, learners generally thought that e-Learning was the use of digital technology to facilitate learning. Since both technological and pedagogical issues for the implementation of e-Learning in school education have been explored in the course, learners had deeper understanding of the pedagogy concept and this proportion therefore was increased to

218 / 410 = 0.532

after the teaching of the course.

Table 3.

The Counts of Matching Key Words and Proportions of Matching Counts at Various Levels of the Bilingual Taxonomy in the Preteaching and Postteaching Reflections.

	Count of matching key words		Proportion of matching counts
Bilingual taxonomy	Preteaching reflection	Postteaching reflection	Preteaching reflection	Postteaching reflection
• e-Learning	107	410	–	–
• Technology	66	192	0.617	.468
• E-resources	4	53	0.061	.276
• Electronic textbook	1	0	0.250	.000
• Open courseware	0	11	0.000	.208
• Curriculum-related electronic resource	0	9	0.000	.170
• Online resource	3	33	0.750	.623
• Digital ways of communication	6	22	0.091	.115
• Classroom communication	0	0	0.000	.000
• Learning management system	6	21	1.000	.955
• Social network	0	1	0.000	.045
• Web page	0	0	0.000	.000
• Digital ways of collecting data	2	22	0.030	.115
• Instant data	2	4	1.000	.182
• Formative data	0	1	0.000	.045
• Summative data	0	2	0.000	.091
• Text mining	0	15	0.000	.682
• Digital technology	54	95	0.818	.495
• Digital device	31	21	0.574	.221
• Software	5	14	0.093	.147
• Digital technology	16	34	0.296	.358
• E-communication	0	18	0.000	.189
• Data collection	2	8	0.037	.084
• Pedagogy	41	218	0.383	.532
• Principles/models/ theories/strategies	9	27	0.220	.124
• Student-centered learning	0	1	0.000	.037
• Collaborative learning	7	20	0.778	.741
• Constructivist learning	0	3	0.000	.111
• Blended learning	2	3	0.222	.111
• Activities	24	130	0.585	.596
• Reflection	1	9	0.042	.069
• Discussion	16	86	0.667	.662
• Feedback	7	24	0.292	.185
• Scaffolding	0	11	0.000	.085
• Pedagogical decision-making/support	0	10	0.000	.046
• Data analytics	0	2	–	.200
• Adjust	0	8	–	.800
• Digital pedagogy	8	51	0.195	.234
• Pedagogy	3	37	0.375	.725
• Activities	5	11	0.625	.216
• Data processing	0	3	0.000	.059

Also, the proportions of matching counts at the second level were adopted to examine learners’ understanding of the technology and pedagogy concepts. As for the examination of learners’ technology concepts, for example, the proportion of matching counts in “digital technology” was $54 / 66 = 0.818$ among the counts related to the technology concept in the preteaching reflection. Digital technology included some common words and terms associated with the use of digital technology, which was not specifically designed for educational use, such as digital device, software, e-communication, and data collection (see Figure 4). The high proportion of matching counts in digital technology might indicate that learners’ understanding of general use, not educational use specifically, of such digital technology. After the teaching of the course, learners had deepen their knowledge of educational use of digital technology and gave a decrease in this proportion to $95 / 192 = 0.495$ ; while the proportions of matching counts in other technology-related subconcepts including “e-resources,” “digital ways of communication,” and “digital ways of collecting data” were increased to .276, .115, and .115, respectively. As for the examination of learners’ pedagogy concepts, it was found that learners had not paid enough attention to the “pedagogical decision-making/support” subconcept since this proportion was still at a low level ( $10 / 218 = 0.046$ ) after the teaching of the course.

Likewise, learners’ understanding of the subconcepts of e-Learning could be examined using this measure at the third level. For example, “collaborative learning” was the most popular key word set of the “principles/models/theories/strategies” subconcept mentioned by learners in the postteaching reflection with the proportion of matching counts equal to $20 / 27 = 0.741$ .

These examples demonstrate that the counts of matching key words found by bilingual text mining on learner-generated text enable teachers to make pedagogical decision with informed judgment based on the strengths and weaknesses of learners demonstrated in the learning process.

Table 4 summarizes the counts and scores of the learners in the preteaching and postteaching reflections in these two evaluation studies. Table 5 summarizes the statistical results of learners’ preteaching and postteaching reflections measured in these two evaluation studies including their understanding of the technology and pedagogy concepts of e-Learning. The statistical comparison of learners’ scores on preteaching and postteaching reflections measured by these two methods indicated a consistent result that learners had a significant progress in their conceptual understanding of e-Learning in school education.

Table 4.

The Counts and Scores of the Learners in the Preteaching and Postteaching Reflections by Counting Matching Key Word Using Text-Mining and Content Analysis With a Rubric, Respectively.

	Count of matching key words using text mining		Score obtained by content analysis with a rubric
Learner	Preteaching reflection	Postteaching reflection	Preteaching reflection	Postteaching reflection
1	15	29	6	12
2	6	27	3	11
3	5	8	2	6
4	2	14	2	7
5	5	19	3	10
6	3	13	2	7
7	4	28	2	10
8	2	17	2	7
9	2	13	2	7
10	2	14	2	6
11	9	9	4	6
12	0	4	1	4
13	3	22	2	8
14	1	15	1	7
15	9	5	2	3
16	1	23	1	8
17	5	18	3	7
18	2	12	3	8
19	3	12	3	9
20	9	14	5	8
21	3	15	3	9
22	3	14	2	9
23	5	10	2	8
24	2	31	2	10
25	3	15	2	8
26	3	7	2	5
27	0	2	2	4
Mean (SD)	3.963 (3.311)	15.185 (7.596)	2.444 (1.121)	7.556 (2.136)

Table 5.

Statistical Results of Learners’ Reflections Measured by Text-Mining and Content Analysis Methods.

		Preteaching reflection		Postteaching reflection
Method	Concept	Mean	SD	Mean	SD	t Test
Text mining	Technology	2.444	2.391	7.111	4.701	4.828***
	Pedagogy	1.519	1.695	8.074	3.802	8.997***
	Total	3.963	3.311	15.185	7.596	7.691***
Content analysis	Technology	1.519	0.802	4.630	0.967	16.597***
	Pedagogy	0.926	0.474	2.926	1.466	7.649***
	Total	2.444	1.121	7.556	2.136	13.918***

***

p < .001.

The Pearson correlation coefficient was adopted in this study to measure the relationship between the results obtained from text-mining and content analysis methods. This correlation analysis showed that there was a strong correlation between the results measured by text-mining and content analysis methods in the postteaching reflection (r = .840, p < .001) and the preteaching reflection (r = .823, p < .001). Such correlation results validated the usefulness of the text-mining method in understanding learners’ reflections. On top of using content analysis that involves a labor-intensive and time-consuming process, the bilingual taxonomy of key words provides an alternative and instant feedback about learner-generated text and therefore has potential as an alternative and less labor-intensive tool to monitor learners’ learning progress in online courses. This will be especially useful to online courses with large enrolment of participants such as massive open online courses (MOOCs), in which it is not easy for teachers to have an understanding of learner-generated text without text-mining support.

Summary and Future Work

This study reports the evaluation of a bilingual text-mining system, which is capable of understanding the domain concepts covered by learner-generated text accumulated in LMSs either in English, Chinese, or a mixture of English and Chinese in order to support pedagogical practices in the bilingual learning and teaching environment. With a bilingual taxonomy of key words on the concerned domain, the text-mining system identifies and counts the matching key words covered by their text automatically. In this study, a tailor-made Moodle plug-in is developed. It can be redeveloped to work with other LMSs to facilitate the extraction of text for mining. An online web-based text-mining interface was designed to facilitate the process of text mining and result reporting, which include the preparation of text data and bilingual taxonomy of key words as well as the processing and reporting of text-mining results through hierarchical visualization.

Summary

A class of 27 in-service teachers studying a course about e-Learning in primary mathematics was involved to evaluate the usefulness of the text-mining system. Having designed the bilingual taxonomy of key words on e-Learning, learners’ reflections on the concept of “e-Learning” before and after the course were processed by the text-mining system in order to identify and count the matching key words of learner-generated text in Moodle’s forum. The text-mining results of both reflections were reported through hierarchical visualization and employed to determine the proportions of matching counts at various levels of the bilingual taxonomy for understanding the learning progress of learners. The same sets of learners’ reflections were evaluated again by content analysis using a rubric to measure their understanding of e-Learning. The correlation analysis indicated that there was a strong correlation between the results using these two evaluation methods with r = .823 and r = .840 in the preteaching and postteaching reflections, respectively. Both evaluation methods indicated that learners gained from the teaching and had a much better understanding of e-Learning in school education.

Contributions

This bilingual text-mining system contributes to the text-mining technique to conduct an automatic analysis of online learner-generated text in English and Chinese for an efficient understanding of learners’ domain knowledge level. With the bilingual taxonomy of key words, this bilingual text-mining system can provide instant feedback on counting matching key words in learner-generated text based on the conceptual model of domain knowledge. With the hierarchical visualization of matching key words, this bilingual text-mining system can support teachers to easily gain a general understanding of learner-generated text through visualizing the proportions of these matching counts at various levels of a bilingual taxonomy for online courses. The bilingual text-mining system therefore is a supporting tool potential for teachers to improve pedagogical practices based on learners’ learning data.

Limitations and future work

This study had two limitations. The first limitation was related to research design—the need to design and then enrich the bilingual taxonomy of key words on the related domain before using this bilingual text-mining system. The second limitation was related to data collection—the small sample size for evaluating this bilingual text-mining system. There are two incentives to address these limitations for increasing the usage of this text-mining system in future: making the repeated use of the bilingual taxonomy of key words across different cohorts of the same course and applying this text-mining system to online courses with high enrolment of learners such as MOOCs. In this context, the designed key words can become more stable if the key words are enriched by working with more than one cohort of learners. In addition to extracting key words by sampling a proportion of the text generated by learners in related LMSs or MOOCs, teachers are recommended to enrich key words in the bilingual taxonomy using a LDA model to discover key words from learner-generated text and other algorithms to support the building of a domain taxonomy and its domain vocabulary like excerpting Wikipedia’s pages and their underlying structure of categories for the initial knowledge base (Daems et al., 2014). The support from these automatic processes is especially important for these online courses because it is generally impractical for teachers to manually search key words used by learners when the number of learners is large. Hence, further research on the usefulness of this bilingual text-mining system in MOOCs environment will be explored.

Appendix: A Rubric for Evaluating Learners’ Understanding of E-Learning

Dimension Score Description

Component A Use of digital resources and digital ways of communication 4 The answer covers explicitly both the aspect of using digital resources for learning and the aspect of digital ways of communication for learning includes wordings like “using digital resources … for learning” and “using communication tools … to support learning.”

3 The answer touches upon implicitly using digital resources or electronic devices to support learning and communication and mentions examples of digital resources or digital ways of communication to support learning and teaching, includes wordings like “using ICT … to support learning and enhance communication.”

2 The answer covers examples on both using digital resources AND using digital ways of communication for learning” includes example such as “using Internet and e-books to support learning.”

1 The answer covers only either one example of digital resources OR digital ways of communication for learning with examples such as “using e-books for learning.”

0 The answer does not cover any wordings related to using digital resources for learning or using digital ways of communication for learning.

Component B Applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning 4 The answer covers the ways of using digital resources AND using digital ways of communication to facilitate the applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning.

3 The answer covers the ways of using either digital resources OR digital ways of communication to facilitate the applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning.

2 The answer covers examples of using digital resources AND digital ways of communication to support learning and teaching without elaboration.

1 The answer covers only examples of using either digital resources OR digital ways of communication to support learning and teaching without elaboration.

0 The answer does not cover any ways or examples of learning and teaching supported by using digital resources and digital ways of communication.

Component C Use of digital ways to collect evidence of learning progress 2 The answer covers the discussion of using digital ways to collect evidence of learning progress.

1 The answer covers examples of using digital ways to collect evidence of learning progress.

0 The answer does not cover any uses of digital ways or tools to collect evidence of learning progress.

Component D Use of digital ways/means to support the analysis of learning data such as learning analytics to facilitate pedagogical decision-making 2 The answer covers the discussion of using digital ways to analyze learning-related data such as learning analytics to facilitate pedagogical decision-making.

1 The answer covers examples of using digital ways to analyze learning-related data such as learning analytics to facilitate pedagogical decision-making.

0 The answer does not mention anything related to the use of digital ways or tools to analyze learning-related data.

Note. ICT = Information and Communication Technology.

Dimension	Score	Description
Component A Use of digital resources and digital ways of communication	4	The answer covers explicitly both the aspect of using digital resources for learning and the aspect of digital ways of communication for learning includes wordings like “using digital resources … for learning” and “using communication tools … to support learning.”
3	The answer touches upon implicitly using digital resources or electronic devices to support learning and communication and mentions examples of digital resources or digital ways of communication to support learning and teaching, includes wordings like “using ICT … to support learning and enhance communication.”
2	The answer covers examples on both using digital resources AND using digital ways of communication for learning” includes example such as “using Internet and e-books to support learning.”
1	The answer covers only either one example of digital resources OR digital ways of communication for learning with examples such as “using e-books for learning.”
0	The answer does not cover any wordings related to using digital resources for learning or using digital ways of communication for learning.
Component B Applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning	4	The answer covers the ways of using digital resources AND using digital ways of communication to facilitate the applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning.
3	The answer covers the ways of using either digital resources OR digital ways of communication to facilitate the applications of principles/models/theories in pedagogical design and practices for active, constructive, and interactive ways of learning.
2	The answer covers examples of using digital resources AND digital ways of communication to support learning and teaching without elaboration.
1	The answer covers only examples of using either digital resources OR digital ways of communication to support learning and teaching without elaboration.
0	The answer does not cover any ways or examples of learning and teaching supported by using digital resources and digital ways of communication.
Component C Use of digital ways to collect evidence of learning progress	2	The answer covers the discussion of using digital ways to collect evidence of learning progress.
1	The answer covers examples of using digital ways to collect evidence of learning progress.
0	The answer does not cover any uses of digital ways or tools to collect evidence of learning progress.
Component D Use of digital ways/means to support the analysis of learning data such as learning analytics to facilitate pedagogical decision-making	2	The answer covers the discussion of using digital ways to analyze learning-related data such as learning analytics to facilitate pedagogical decision-making.
1	The answer covers examples of using digital ways to analyze learning-related data such as learning analytics to facilitate pedagogical decision-making.
0	The answer does not mention anything related to the use of digital ways or tools to analyze learning-related data.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded by the Hong Kong UGC (University Grants Committee) TDG (Teaching Development Grant) (Ref: HKIED7/T&L/12-15).

Author Biographies

Siu Cheung Kong is a professor of the Department of Mathematics and Information Technology and Director of Centre for Learning, Teaching and Technology at The Education University of Hong Kong. His research interests include pedagogy in the digital classroom, policy on technology-transformed education, and the professional development of teacher for learner-centered learning in seamless learning environments, IT in mathematics education, programming for computational thinking development, and computational thinking education.

Ping Li received his PhD from The Chinese University of Hong Kong. He is currently a lecturer at The Education University of Hong Kong. His research interests include image/video stylization, learning analytics, big data visualization, and creative media.

Yanjie Song is an assistant professor at the Department of Mathematics and Information Technology, The Education University of Hong Kong. Her research interests mainly focus on e-learning, mobile learning, learning analytics, and innovative pedagogies.

References

Blei

D. M.

A. Y.

Jordan

M. I.

(2003) Latent Dirichlet allocation. The Journal of Machine Learning Research 3: 993–1022.

Bostock

Ogievetsky

Heer

(2011) D3: Data-driven documents. IEEE Transactions on Visualization and Computer Graphics 17(12): 2301–2309.

Boyd-Graber

Blei

D. M.

(2009, June) Multilingual topic models for unaligned text. In: Bilmes

A. Y.

(eds) Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, Canada: AUAI Press, pp. 75–82.

Chang

P.-C.

Galley

Manning

C. D.

(2008, June) Optimizing Chinese word segmentation for machine translation performance. In: Callison-Burch

Koehn

Monz

Schroeder

Fordyce

C. S.

(eds) Proceedings of the Workshop on Statistical Machine Translation, Stroudsburg, PA: Association for Computational Linguistics, pp. 224–232.

Chu

Nakazawa

Kurohashi

(2014) Iterative bilingual lexicon extraction from comparable corpora with topical and contextual knowledge. In: Gelbukh

(ed.) Computational Linguistics and Intelligent Text Processing, Berlin, Heidelberg: Springer, pp. 296–309.

Cimiano

Hotho

Staab

(2005) Learning concept hierarchies from text corpora using formal concept analysis. Journal of Artiﬁcial Intelligence Research 24: 305–339.

Daems

Erkens

Malzahn

Hoppe

H. U.

(2014) Using content analysis and domain ontologies to check learners’ understanding of science concepts. Journal of Computers in Education 1(2–3): 113–131.

de Oliveira

M. F.

Levkowitz

(2003) From visual data exploration to visual data mining: A survey. IEEE Transactions on Visualization and Computer Graphics 9(3): 378–394.

Dzikovska

Steinhauser

Farrow

Moore

Campbell

(2014) BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. International Journal of Artificial Intelligence in Education 24(3): 284–332.

10.

Elmqvist

Fekete

J.-D.

(2010) Hierarchical aggregation for information visualization: Overview, techniques, and design guidelines. IEEE Transactions on Visualization and Computer Graphics 16(3): 439–454.

11.

Feldman

Dagan

(1995, August) Knowledge discovery in textual databases (KDT). In: Fayyad

U. M.

Uthurusamy

(eds) Proceedings of the First International Conference on Knowledge Discovery and Data Mining, Montreal, Canada: Association for the Advancement of Artificial Intelligence, pp. 112–117.

12.

Fung

(1995, June) Compiling bilingual lexicon entries from a non-parallel English-Chinese corpus. In: Yarowsky

Church

(eds) Proceedings of the Third Workshop on Very Large Corpora, Cambridge: Massachusetts Institute of Technology, pp. 173–183.

13.

Gašević

Dawson

Siemens

(2015) Let’s not forget: Learning analytics are about learning. TechTrends 59(1): 64–71.

14.

Gupta

Lehal

G. S.

(2009) A survey of text-mining techniques and applications. Journal of Emerging Technologies in Web Intelligence 1(1): 60–76.

15.

Gutiérrez

E. D.

Shutova

Lichtenstein

de Melo

Gilardi

(2016) Detecting cross-cultural differences using a multilingual topic model. Transactions of the Association for Computational Linguistics 4: 47–60.

16.

Huang

C. C.

Chen

M. H.

Yang

P. C.

(2015, July) Bilingual key word extraction and its educational application. In: Chen

H.-H.

Tseng

Y.-H.

Matsumoto

Wong

L. H.

(eds) Proceedings of 2nd Workshop on Natural Language Processing Techniques for Educational Applications, Beijing, China: Association for Computational Linguistics and Asian Federation of Natural Language Processing, pp. 43–48.

17.

Klein

Manning

C. D.

(2003, July) Accurate unlexicalized parsing. In: Hinrichs

E. W.

Roth

(eds) Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, Stroudsburg, PA: Association for Computational Linguistics, pp. 423–430.

18.

Krstovski

Smith

D. A.

Kurtz

M. J.

(2016, June) Online multilingual topic models with multi-level hyperpriors. In: Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA: Association for Computational Linguistics, pp. 454–459.

19.

Lárusson

J. A.

White

(2012, April) Monitoring student progress through their written point of originality. In: Shum

S. B.

Gasevic

Ferguson

(eds) Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, New York, NY: Association for Computing Machinery, pp. 212–221.

20.

Lin

F. R.

Hsieh

L. S.

Chuang

F. T.

(2009) Discovering genres of online discussion threads via text-mining. Computers & Education 52(2): 481–495.

21.

Liu

J. N. K.

Y. L.

Lim

E. H.

Wang

X. Z.

(2014) Domain ontology graph model and its application in Chinese text classification. Neural Computing and Applications 24(3–4): 779–798.

22.

Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., & McClosky, D. (2014, June). The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 55–60). Baltimore, Maryland: Association for Computational Linguistics.

23.

Ming

N. C.

Ming

V. L.

(2015) Visualizing and assessing knowledge from unstructured student writing. Technology, Instruction, Cognition & Learning 10(1): 27–44.

24.

Newby

(2010) Research methods for education, Harlow, England: Pearson Education.

25.

Pérez-Marín

Pascual-Nieto

(2010) Showing automatically generated students’ conceptual models to students and teachers. International Journal of Artificial Intelligence in Education 20(1): 47–72.

26.

Riloff

Schafer

Yarowsky

(2002, February) Inducing information extraction systems for new languages via cross-language projection. In: Proceedings of the 19th International Conference on Computational Linguistics—Volume 1, Stroudsburg, PA: Association for Computational Linguistics, pp. 1–7.

27.

Romero

Ventura

(2010) Educational data mining: A review of the state of the art. IEEE Transaction on Systems, Man, and Cybernetics, Part C: Applications and Reviews 40(6): 601–618.

28.

Sawilowsky

S. S.

(2009) New effect size rules of thumb. Journal of Modern Applied Statistical Methods 8(2): 597–599.

29.

Sherin

(2012, April) Using computational methods to discover student science conceptions in interview data. In: Shum

S. B.

Gasevic

Ferguson

(eds) Proceedings of the 2nd International Conference on Learning Analytics and Knowledge, New York, NY: Association for Computing Machinery, pp. 188–197.

30.

Shneiderman

(1996, September) The eyes have it: A task by data type taxonomy for information visualizations. Proceedings of the IEEE Symposium on Visual Languages, Los Alamitos, CA: IEEE, pp. 336–343.

31.

Smadja

McKeown

K. R.

Hatzivassiloglou

(1996) Translating collocations for bilingual lexicons: A statistical approach. Computational Linguistics 22(1): 1–38.

32.

Song

Lin

Yang

(2007, September) Opinion mining in e-learning system. In: Li

Xiang

Jin

Cao

(eds) Proceedings of International Conference on Network and Parallel Computing Workshops, Washington, DC: IEEE, pp. 788–792.

33.

Southavilay

Yacef

Reimann

Calvo

R. A.

(2013) Analysis of collaborative writing processes using revision maps and probabilistic topic models. In: Suthers

Verbert

Duval

Ochoa

(eds) In Proceedings of 3rd ACM International Conference on Learning Analytics and Knowledge, New York, NY: Association for Computing Machinery, pp. 38–47.

34.

Teoh

S. T.

K.-L.

(2002, August) RINGS: A technique for visualizing large hierarchies. In: Kobourov

S. G.

Goodrich

M. T.

(eds) Proceedings of the Revised Papers from the 10th International Symposium on Graph Drawing, London, England: Springer-Verlag, pp. 268–275.

35.

Teplovs

Fujita

Vatrapu

(2011, February) Generating predictive models of learner community dynamics. In: Long

Siemens

Conole

Gasevic

(eds) Proceedings of the 1st International Conference on Learning Analytics and Knowledge, New York, NY: Association for Computing Machinery, pp. 147–152.

36.

Williams

D’Mello

(2010, June) Predicting student knowledge level from domain-independent function and content words. In: Aleven

Kay

Mostow

(eds) Proceedings of the 10th International Conference on Intelligent Tutoring Systems, Pittsburgh, PA: Springer-Verlag, pp. 62–71.

37.

Yin

Wang

(2014, August) A Dirichlet multinomial mixture model-based approach for short text clustering. In: Macskassy

Perlich

(eds) Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY: Association for Computing Machinery, pp. 233–242.

38.

Zhao

Qin

Liu

Tang

(2016) Social sentiment sensor: A visualization system for topic detection and topic sentiment analysis on microblog. Multimedia Tools and Applications 75(15): 8843–8860.

39.

Zhou, J., & Wang, S. (2010). Concept mining and inner relationship discovery from text. In J. Zhang (Ed.), New Advances in Machine Learning (pp. 305–327). Rijeka, Croatia: InTech.