Incorporation of Collaborative Problem Solving and Cognitive Tools to Improve Higher Cognitive Processing in Online Discussion Environments

Abstract

During collaborative learning in online learning communities, teachers usually guide their students through the learning process by means of discussion-based didactics. According to relevant research, an uncontrolled, nonintrusive discussion environment is usually insufficient for promoting higher cognitive processing (HCP). To address this insufficiency, we adopted a collaborative problem solving approach as a teaching strategy to tutor students in online discussion activities using concept maps as a cognitive tool and using Facebook for communication within the learning community. We examined the system using quantitative content analysis and lag sequential analysis to verify the feasibility of the system for improving HCP. The results of this study indicate that the system is capable of guiding cognition and improving HCP. However, the results with respect to improvement must be evaluated after taking into account the appropriateness and difficulty of the questions that were posed to the learners. Moreover, by incorporating a scaffolding function as a teaching strategy and through the application of cognitive tools, learners were better able to concentrate on the learning activities.

Keywords

online discussion collaborative problem solving concept map Facebook lag sequential analysis scaffolding

Introduction

In online learning communities, online collaborative learning is an approach that facilitates the transformation of knowledge for learners (Boud, Cohen, & Sampson, 2014; Fong & Slotta, 2018). An online peer-assisted learning platform provides the students with an opportunity to employ a range of strategies to cooperate, instruct, supervise, evaluate, learn from, and give feedback to their peers. These interactions may help them sharpen interpersonal skills, achieve academic success, improve cognitive functions, and acquire meta-cognitive knowledge in online learning communities (Ouyang & Scharber, 2017; Pifarre & Cobos, 2010; Topping, 2010). From studies related to community of practice (CoP), we also learned that technology could enhance social interactions. By applying this approach on teaching and learning, it can also increase the students' knowledge construction through social interaction (Smith, Hayes, & Shea, 2017; Tseng & Kuo, 2014). In the past, when building an online learning community, most researchers tended to specifically create an innovative online learning environment that required learners to log in during or after class to access the online learning community. However, according to studies by Mazman and Usluel (2010) and Wu, Hou, Hwang, and Liu (2013), an online learning community should be an extension of the learner's most favored social community. In this sense, it is advisable that students be allowed to select a learning community in which they feel the most comfortable rather than being placed in specific learning communities by an instructor. Many researchers today have conducted studies of Facebook-based online learning communities and found that using Facebook as a learning environment for students is beneficial and promotes student learning (Lampe, Wohn, Vitak, Ellison, & Wash, 2011; Raspopovic, Cvetanovic, Medan, & Ljubojevic, 2017). For example, in a study designed with an experimental group (interactive Facebook instructional method) and a control group (non-Facebook instructional method), Wang, Lin, Yu, and Wu (2013) found that students in the experimental group, which used Facebook for instruction, achieved better scores and exhibited a higher level of student involvement and student satisfaction.

In online learning communities, discussion didactics are frequently used for collaborative learning to develop students' cognitive skills and critical thinking (Anderson & Krathwohl, 2001; Anderson, Rourke, Garrison, & Archer, 2001). For example, Jin and Jeong (2013) investigated relationship of the Bloom cognitive levels of learning and types of postings (i.e., argument, critique, evidence, and explanation), the results pointed out that besides knowledge cognition level, other five cognition level achieved to a certain extent among the four types of postings. In addition, higher cognition levels were most likely to be exhibited in critique and argument postings. Thus, to achieve these goals, teachers usually incorporate some interactive teaching strategies such as peer assessment (Formanek, Wenger, Buxner, Impey, & Sonam, 2017), role play (Howes & Cruz, 2009), peer tutoring (De Backer, Van Keer, & Valcke, 2016), problem-solving-based activities, and project-based activities (Koh, Herring, & Hew, 2010; Şendag & Odabaşi, 2009).

Online discussion forums help increase participants' learning performance, and recent studies have indicated that some other instructional strategies may also increase the cognitive skills of students (Broadbent & Poon, 2015; Hou, Wang, Lin, & Chang, 2015). The point here is how students can develop their behaviors when they are engaged in a higher level cognitive process. Although the incorporation of online discussion activities in online learning communities resulted in improved learning results, recent studies have suggested that it was necessary to enhance HCP behaviors when HCP teaching strategies were applied in conjunction with online discussions. During the peer assessment of online discussion activities conducted in a project-based learning environment, Hou, Chang, and Sung (2007) found that students failed to demonstrate advanced cognitive thinking in discussions. Hou, Chang, and Sung (2008) explored the patterns of asynchronous online discussions of college students using problem-based discussion activities as a teaching strategy. The results revealed a widespread tendency to not review the solutions proposed for solving advanced problems. De Wever, Schellens, Keer, and Valcke (2008) had their students play assigned roles and properly conduct peer discussions. Their study revealed that while role play did inspire a variety of thinking patterns, the students' abilities to engage in high-level discussions did not improve. During a project-based asynchronous online discussion, Hou (2010) conducted a progressive time-lag sequential analysis (LSA) and discovered that the longer the discussion, the more likely it was that the discussion's behavioral pattern would take on significant differences between students. Nevertheless, the HCP behaviors of students failed to improve. Furthermore, Wu, Hou, and Hwang (2012) noted that the discussions often were significantly lacking in advanced levels of cognition and were teeming with continuous off-topic discussions.

It is evident from the aforementioned studies that online discussions do improve students' cognitive processing (Hou, 2010; De Wever et al., 2008; Wu et al., 2012). However, these studies also suggested that, at times, it may be difficult for students to engage HCP in an online activity without guidance or intervention from a teacher (Hou et al., 2007; Wu et al., 2013). Compared to lower cognitive processing (e.g., to remember, understand and apply), HCP (e.g., to analyze, evaluate and create) can be even more helpful for students in acquiring knowledge (Hou & Wu, 2011; Kim, Park, Jang, & Nam, 2017), even though it proves to be a more complicated process (Apino & Retnawati, 2017).

Given the abovementioned concerns, some authors of recent studies have advised that task design, for example, guidance from a teacher or teaching assistant or the application of a teaching strategy that can induce higher level thinking in students, should be incorporated in online discussion learning activities (De Wever, Schellens, Van Keer, & Valcke, 2008; Simon & Maloney, 2007). For example, Jalil, McFarlane, Ismail, and Krauss (2008) used tasks of various types including scaffolding, performance feedback, and cognition structuring in their online discussions. Jorczak (2009) probed into the influence of the features of tasks on higher order learning during discussion activities including task scenarios and result criteria. Accordingly, students can better structure their discussions according to the task design of the teaching activities and, with the guidance of a teacher or a teaching assistant, have a better understanding of the necessary steps to follow. However, such guidance can be perceived by teachers as an additional burden on them.

In addition to recommending that task design should be incorporated in online discussion teaching activities, some researchers have explored how to exploit various cognitive tools, such as functional modules that help the development of knowledge and concepts during discussion, to improve the learning results from online discussions. For example, Stahl (2006) discussed group referencing in a text chat room using Virtual Math Teams Project as a cognitive tool and determined students how to create, structure, support, and assess an online chat-based collaborative community. It was analyzing the forms of group cognition that emerge from the use of shared cognitive tools with specific functionalities. The results showed that the cognitive tools—Virtual Math Teams Project can be supporting group alignment, intentionality, and cognition in online collaborative mathematics communities. O'Donnell, Dansereau, and Hall (2002) conducted a 12-year literature review of knowledge representation and knowledge mapping and concluded that knowledge mapping could help learners recall more central ideas when they learned from the knowledge map than learning from the text. In addition, knowledge maps also appear to amplify the benefits associated with scripted cooperation. Learning from maps is enhanced by active processing strategies such as summarization or annotation and by designing maps according to gestalt principles of organization. The improvement of online discussion quality with system tools was discussed in certain studies that were not focused on improving cognitive processing as their main purpose but rather were designed for a single subject or purpose. Thus, because such system tools are not universal, they cannot be adopted by all teachers.

Based on the earlier discussion, this study uses online learning communities, online discussion, and collaborative problem solving (CPS) as theoretical foundations, thus developing an integrated system where instructional scaffolding provides varied teaching strategies and guided mechanisms. Accordingly, the primary purpose of this study was to develop an online discussion system based on Facebook groups that can be customized by teachers to improve HCP of students. With this system, teachers can manage and design an online discussion system to improve students' HCP.

To verify whether the system could improve HCP and to understand the system's influence on cognition in online discussions, we conducted a quantitative content analysis (QCA) of the discussion content and a time-LSA of its operational behavioral patterns (e.g., Lehmann-Willenbrock, Allen, & Kauffeld, 2013; Wu, Chen, & Hou, 2015). LSA is used for determining the coding category of a continuous series of coding categories through an interactive process and for visualizing behavioral patterns. Currently, LSA has been applied in behavioral pattern analyses of online discussions by studies (Chiu & Khoo, 2005; Hou, 2012; Hou & Wu, 2011; Jeong, 2003). The secondary purpose of this study was to verify the system using QCA and LSA to determine whether it could promote the generation of HCP behaviors and determine the frequency and mode of such behaviors in learners who used the system in online discussions.

Theoretical Basis and System Development

The system of this study was mainly a scaffolding design (Korhonen, Ruhalahti, & Veermans, 2018). Currently, some studies have indicated that teachers' scaffolding (Kilic, 2018), guidance from parents (Lin & Liu, 2012), and learning from peers (Agrawal, Nandanwar, & Musti, 2017) can all be seen as guided strategies. In addition, they can also be guided in the construction of their programs through computer scaffolding for program construction (Lye & Koh, 2014). Among such a wide variety of guided strategies, the strategy known as “computer scaffolding for program construction” is most convenient and widely applicable.

We intend to help students engaged in online discussions in a Facebook group, using CPS as the framework for scaffolding. CPS, a teaching strategy proposed by Nelson (1999), is widely used by researchers and educators for task design in online discussion teaching activities (Guimarães, Antunes, García, & Fernandes, 2013; Pollastri, Epstein, Heath, & Ablon, 2013; Santangelo, 2009). The CPS approach encourages learners to learn from practice and attaches importance to the collaborative learning environment and problem-solving competence. Therefore, we adopted the CPS approach for learning strategy design in this study.

This system was designed to reduce teachers' workload in instructional activities in the hope of increasing students' higher cognitive abilities. In other words, the duties of a teacher are to set procedures for class discussions in accordance with learning units and curriculum objectives. This system, which is designed by our team, provides teachers with a scenario-related problem designed in accordance with learning elements and course objectives. Students are administered a pretest and then grouped according to the learning objectives. Once the students have joined a discussion group, the teacher announces the learning progress of the CPS approach in eight steps based on their discussion status, and the content released from the server will be visible to the Facebook group. Students then engage in the discussions based on the released information. The eight steps consist of (a) preparing task statements and learning objectives, (b) preparing a group list, (c) defining the problem, (d) electing a leader, (e) collaborative exploration and concept formation, (f) developing problems and solutions, (g) sharing ideas and reflections, and (h) preparing a summary and feedback.

Figure 1 illustrates the system design. When students engage in the group discussions, the discussions in each group are independent of the other discussions. In addition, all discussion content and system operation behaviors are recorded in chronological order in a background database to facilitate further post factum analysis.

Figure 1.

System flowchart.

One of the applications frequently used by researchers and educators to improve cognition using system tools is the concept map, which represents the structure of content knowledge. A concept map consists of concept nodes and interconcept relationship links wherein the conjunctions between two concept nodes constitute a proposition and the concepts are presented in a concept map as hierarchical relations. A concept map serves as an auxiliary tool for members of a constructivist learning activity in the learning of raw content and in collaborative learning (Novak & Gowin, 1984). Some studies have noted that concept maps are beneficial to learning behaviors (Roessger, Daley, & Hafez, 2018; Sung & Hwang, 2018; Wang, Cheng, Chen, Mercer, & Kirschner, 2017). For example, Nesbit and Adesope (2006) found that using the concept map can help in knowledge retention and transfer through meta-analysis reviews and experimental and quasi-experimental studies. Wu, Chen, and Hou (2016) also noted that the use of concept map in the online discussion can strengthen higher level cognitive processing and guide learners back to the discussion topic. In addition, Lee (2012) also noted that through exploring the process of students' the concept map drawing by LSA can further diagnose their learning causality. Therefore, a concept map was used as a cognitive tool in this study. The intent was that teachers would use the concept map to display the cognition scaffolding of the learning unit as a way to guide students in their learning.

The application of conceptual underpinnings occurred in Step 5 of the CPS, that is, collaborative exploration and concept formation, and it was essential that the teacher initially establish the learning premise of the concept map to facilitate the formation of global concepts through the combination of guidance and online discussions. Once the teacher had established the settings, students in the groups were required to answer the question(s) in the concept map. This process provides students with clues to facilitate their thinking, their discussions, and the formation of relevant concepts. Once the students completed the discussions, they were required to answer the assigned questions to verify the accuracy of the concepts they had developed. The answering process was executed under specific guidance to promote student discussion and to provide hints for online information collection.

Teachers may inform their students about enrolling in the program once they have set the procedures and curriculum content. Then, teachers will be able to divide up these students into groups for them to partake in all kinds of activities. When they go through the process, teachers may set up the maximum execution time for each activity according to the level of difficulty. This way, the entire class can move to the next step within a set timeframe.

This study was designed to create an authoring tool (Wu, 2015), which is an online learning community environment based on a social community, that is, Facebook. In this study, the CPS approach was adopted as a teaching strategy, and the concept map was applied as a cognitive tool. Accordingly, the teacher sets various concept maps in advance according to the various subjects being taught and uses the CPS approach to guide the students as they solve problems through online discussions.

Research Design

Participants and Procedure

Convenience sampling is adopted in this study. We recruited 68 grade-seven student participants from two high school classes. All the participants were taught by the same teacher and were randomly assigned to four groups. Once the teacher had prepared the questions and the guidance concept maps, a task was designed. Then, each group of students was asked to collaboratively complete the task. Each group was to discuss the questions and provide solutions to the questions in their respective Facebook groups using the CPS approach and the concept maps. The procedure was as follows (Figure 2):

Introduction to the discussion environment: The students were assigned to the designated Facebook groups according to the grouping determined prior to the class. The procedures for the discussion activities were then explained to the students.

Introduction to the discussion tasks: A CPS-based discussion was adopted as the teaching strategy for this study, and the teacher prepared discussion topics appropriate to the students' backgrounds. For example, one description was as follows:

Jianyu is a high school student who usually enjoys interacting with his classmates on the Internet. One day, Jianyu received an email titled “Good News for Everyone!!” This email was forwarded to him by Baobao, one of his good friends. Without hesitation, Jianyu opened the mail and found that it was an advertisement for an Internet shopping mall. The mail contained a hyperlink, which redirected him to a web page where he was requested to enter his name, date of birth, and telephone number to be eligible for a lottery. Please respond to the following two questions: What methods can you use to determine the authenticity of Internet activities such as this? What can you do to avoid cyber fraud?

Figure 2.

Procedure flowchart.

Discussion: The following eight weeks were organized as follows: Week 1 was dedicated to task statements and learning objectives (CPS-1), grouping (CPS-2), problem definition (CPS-3) and leader election (CPS-4). Weeks 2 and 3 were devoted to collaborative exploration and concept formation (CPS-5). Weeks 4 and 5 were spent identifying problems and solutions (CPS-6). Weeks 6 and 7 were devoted to the exchange of ideas and reflections (CPS-7). Week 8 was dedicated to summarization and feedback (CPS-8). During the discussions, the students in a group discussed the questions according to the system settings in their Facebook groups. The groups were not able to see the discussions of other groups. After completing the discussions, each group was required to propose solutions to the identified problems.

Pooling and analysis: At the end of the activity, all discussion content and system operation logs were downloaded for follow-up data analysis. We will introduce encodings of the discussions in the next section.

Coding Scheme

To identify the interaction patterns of the students' online discussions in the system, the discussion content was coded in accordance with the cognitive processing dimension of the revised Bloom's taxonomy by Anderson and Krathwohl (2001) and Anderson (2006) (modified from Bloom, 1956, 1971), which also facilitated the follow-up QCA and LSA. This particular coding scheme, which is frequently adopted for analyzing cognitive skill levels in various learning activities, divides cognitive processing activities into six categories, that is, to remember, understand, apply, analyze, evaluate, and create, as shown in Table 1. This coding scheme has been widely adopted in the analysis of online discussion content (Johnson, 2008; Lin, Hou, Wu, & Chang, 2014; Morueta, López, Gómez, & Harris, 2016; Rabbany, ElAtia, Takaffoli, & Zaïane, 2014).

Table 1.

Cognitive Processing Dimensions in the Revised Bloom's Taxonomy.

Code	Dimension	Examples of the cognitive processes involved
	Example of discussion content
B1	Remember: can the student recall or remember the information?	Define, duplicate, list, memorize, recall, repeat, reproduce, state
	It is dangerous to open a suspicious email.
B2	Understand: can the student explain ideas or concepts?	Classify, describe, discuss, explain, identify, locate, recognize, report, select, translate, paraphrase
	Some emails use very attractive subject lines or even add hyperlinks to catch your attention.
B3	Apply: can the student use the information in a new way?	Choose, demonstrate, dramatize, employ, illustrate, interpret, operate, schedule, sketch, solve, use, write
	The most common online scams include . . .
B4	Analyze: can the student distinguish between the different parts?	Appraise, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test
	Email scams and phishing are different in some ways.
B5	Evaluate: can the student justify a stand or decision?	Appraise, argue, defend, judge, select, support, value, evaluate
	I think some clues can be used to identify the authenticity of online activities
B6	Create: can the student create a new product or perspective?	Assemble, construct, create, design, develop, formulate, write
	We can observe the Web address, or look for online evaluations (about any promotional events or discount) in the first place.
B7	Off-topic	Discussions irrelevant to knowledge construction
	What is the assignment for science tomorrow?

The codes for all system logs and records are explained in Table 2.

Table 2.

Explanations of Codes for System Logs and Records.

Click-on permission	Behavioral event	Description of event
All Members	List (LT)	Access home page
All Members	Error (ER)	System operation error
All Members	Log in (LI)	Login on foreground web page
All Members	Log out (LO)	Logout on foreground web page
All Members	Access (AC)	Access content of activity steps menu
Group Leader	Add answer (AN)	Submit a new answer: Steps 6 to 8 of CPS
Group Leader	Answer editor (AE)	Edit a submitted answer
All Members	Concept map (CM)	Start Step 5 of CPS approach
All Members	Concept map node (CN)	Click on a sub-concept
All Members	Concept map bingo (CB)	Correctly answer a sub-concept question
All Members	Concept map wrong (CW)	Incorrectly answer a sub-concept question
All Members	Concept map hint (CH)	Access a concept map hint
All Members	Concept map close (CC)	Shut down sub-concept window

Note. CPS = collaborative problem solving.

Data Collection and Processing

The students discussed and completed the assigned tasks in their Facebook groups. During this process, the discussion content and system operation history of each group was recorded to facilitate the post factum analysis.

During the discussion, all discussion content was sorted based on the discussion message units, which may consist of several sentences and/or paragraphs, and was coded in accordance with the codes for cognitive processing (Table 1). All the coding was performed by trained coders. To ensure the consistency of scoring results between different coders, a sample of 50% of the discussion content was submitted to another coder for coding. An analysis of QCA, that is, the frequency of codes and distribution proportion, and an LSA were conducted of the discussion and system operation logs. This step was performed by subjecting the abovementioned codes to temporal sequencing and calculations of the frequency transition matrix of the codes. Individual sequences of continuity significance were then deduced using a series of sequence matrix operations (Bakeman & Gottman, 1997) from which a sequential transfer diagram was plotted to determine the behavioral sequence patterns of the cognitive processing and knowledge construction.

In addition to these quantitative content and behavioral analyses, the participants were interviewed to obtain additional qualitative data and thereby improve the overall validity of the study.

Results and Discussion

We developed a Facebook-based online learning community that integrated the CPS approach and the concept map mechanism. This system allowed the teacher to establish concept maps that incorporated a guidance function aligned with the subject content and to use the CPS approach to guide students as they solved problems through online discussions. To better understand whether this system can increase HCP, this study explores operation behaviors and cognitive processing using QCA and LSA. Furthermore, we will examine the behaviors involved in generating HCP and try to verify them from the conception of operation behaviors.

Quantitative Analysis of Operation Behaviors and Cognitive Processing

A total of 8,484 discussion content entries and system logs are discussed in this study. These data included 3,826 discussion messages (45.10%) and 4,658 entries in system logs (54.90%). The results of the analyses are presented in Table 3.

Table 3.

Quantitative Analysis Table.

Behavioral event	Frequency	System %	Total %
List (LT)	73	1.57	0.86
Error (ER)	3	0.06	0.04
Log in (LI)	249	5.35	2.93
Log out (LO)	99	2.13	1.17
Access (AC)	1,269	27.24	14.96
Add answer (AN)	112	2.40	1.32
Answer editor (AE)	208	4.47	2.45
Concept map (CM)	1,514	32.50	17.85
Concept map node (CN)	974	20.91	11.48
Concept map bingo (CB)	137	2.94	1.61
Concept map wrong (CW)	10	0.21	0.12
Concept map hint (CH)	0	0.00	0.00
Concept map close (CC)	10	0.21	0.12
System operation	4,658	Discussion %	54.90
Remember (B1)	1,231	32.17	14.51
Understand (B2)	1,865	48.75	21.98
Apply (B3)	8	0.21	0.09
Analyze (B4)	29	0.76	0.34
Evaluate (B5)	74	1.93	0.87
Create (B6)	28	0.73	0.33
Off-topic (B7)	591	15.45	6.97
Discussion content	3,826		45.10
Total	8,484

It can be inferred from the results that the three most frequent system operation behaviors were concept map (CM) (N = 1514, 32.50%), access (AC) (N = 1269, 27.24%), and concept map node (CN) (974, 20.91%). Because all the students, after logging on to the system, must perform an access (AC) to answer questions or prepare concept maps, it is natural that the frequency of access (AC) operations is high. Concept map (CM) and concept map node (CN) are both operations required for concept mapping. Compared with the concept map (CM) and concept map node (CN) operations, concept map wrong (CW) (N = 10, 0.21%), concept map close (CC) (N = 10, 0.21%), and concept map hint (CH) (N = 0, 0.00%) behaviors were less frequently observed. In fact, none of the learners accessed a concept map hint (CH). Judging from these operation behaviors, it is inferred that the students spent a significant amount of time managing the concept map when operating the system, which is attributable to the system design. However, finding no concept map hint (CH) behavior suggests that the students never used the hint function even when their answers were wrong. The follow-up interviews revealed that the students believed they could answer the questions by searching on the Internet, and as a result, they did not use the hint function. From comparing the phenomenon of this behavior in the study with the study (Wu et al., 2016) of concept map behavior recorded by system operation, the advantage of the concept map with given prompts is to provide a structured learning map while a free concept map drawing provides better exploring space. Both of which have significance on teaching. However, from the concept map of this study, when designing guiding questions, teachers must consider the difficulty of the questions and the design of the task and questions to thereby encouraging students to answers the questions with the help of the hint function rather than by searching on the Internet.

Regarding cognitive processes behavior, the results showed that the interrater Kappa reliability coefficients was .73 (P < .001) between the two coders, showing a very high consistency. The cognitive processes observed most frequently were those designated understand (B2) (N = 1865, 48.75%), remember (B1) (N = 1231, 32.17%), and off-topic (B7) (N = 591, 15.45%), indicating that most of the students accessed the understand (B2) and remember (B1) aspects of cognitive processing during the CPS-based discussion activities. This result is consistent with those of most studies in the field (Lin et al., 2014; Wu et al., 2012). In many studies, the frequency of remember (B1) operations is high. In comparison, it is evident that this system can also improve the understand (B2) aspect of cognitive processing. The proportion of off-topic (B7) discussions was relatively low in this study compared to that of other relevant studies (Wu et al., 2012, 2015). Our follow-up interviews revealed that all the students were focused during the group discussions because the system requires the leader of each group to answer the questions assigned by the teacher (i.e., Steps 6–8 of the CPS approach) at the end of each discussion stage. Conversely, to analyze (B4), evaluate (B5) and create (B6) require HCP. Although these last three cognitive behaviors were identified in this study, they were observed at low frequencies, which may imply that the students were more focused on understanding (B2) and remembering (B1) during the discussion.

It can be inferred from the results of the QCA that the frequencies of such operation behaviors were affected by the workflow design of this study in that the system requires the leader of each group to answer the assigned questions on behalf of the other members of the group. In this system, every student can access the concept map operation. However, judging from the frequencies of all concept map behaviors, it is advisable that when the teacher designs a task, the guiding questions should take into account the difficulty of the questions and should encourage students to answer the questions with the help of the hint function rather than searching for answers on the Internet (Ho, Harris, Kumar, & Velan, 2018; Wu, Hwang, Milrad, Ke, & Huang, 2012). This practice aligns with the cognitive processing results. With respect to cyber fraud, most students resort to Internet searches to find the answers, which results in a lack of identified HCP behaviors, and accordingly, in the future, this tendency should be considered when selecting the subject and designing the tasks.

Operation Behaviors and Behavioral Patterns of Cognitive Processing

When LSA is used to identify behavioral patterns, a behavioral transfer diagram is plotted using an adjusted residuals table. This study involved both operation behaviors and discussion-based cognitive processing behaviors, and adjusted residuals tables and behavioral transfer diagrams are presented in Tables 4 and 5 and Figures 3 and 4, respectively. In the adjusted residuals tables, the columns present the categories of starting behaviors, and the rows present the behaviors that occurred immediately after the behaviors in the column. When Z value exceeding 1.96 indicates that the sequence displays continuity significance.

Figure 3.

Behavioral transfer diagram of system logs.

Figure 4.

Behavioral transfer diagram of cognitive processing.

Table 4.

Adjusted Residuals Table of System Logs.

	LT	ER	LI	LO	AC	AN	AE	CM	CN	CB	CW	CC
LT	15.27*	−0.21	4.31*	12.8*	0.95	−1.29	−1.76	−4.76	−3.55	−1.43	−0.39	−0.39
ER	−0.21	−0.04	4.74*	3.82*	−0.88	−0.26	−0.36	−0.96	−0.77	−0.29	−0.08	−0.08
LI	6.91*	7.3*	4.61*	24.1*	9.24*	−2.39	−3.25	−8.83	−6.92	−2.64	−0.71	−0.71
LO	2.91*	−0.25	35.26*	0	−4.48	−1.5	−2.05	−5.54	−4.44	−1.66	−0.45	−0.45
AC	1.87	−0.88	−0.93	−3.09	20.23*	3.55*	2.91*	−13.56	−3.74	−5.64	−1.61	−1.61
AN	−1.29	−0.26	−2.39	−1.5	3.36*	38.44*	−1.72	−5.9	−4.73	−1.77	−0.48	−0.48
AE	−1.76	−0.36	−3.25	−2.05	3.05*	−2.18	41.53*	−8.06	−6.45	−2.41	−0.65	−0.65
CM	−3.91	−0.96	−6.65	−4.09	−15.62	−5.9	−8.06	33.18*	−6.04	−6.53	−1.76	−1.76
CN	−3.02	−0.77	−3.78	−3.99	−3.61	−4.73	−6.45	−17.18	26.25*	20.16*	5.69*	−1.41
CB	−1.43	−0.29	−1.88	−1.66	−5.97	−1.77	−2.41	−2.05	−4.84	−0.42	−0.53	−0.53
CW	−0.39	−0.08	−0.71	−0.45	−1.61	−0.48	−0.65	−1.76	−1.41	−0.53	−0.14	69.91*
CC	−0.39	−0.08	−0.71	−0.45	−1.61	−0.48	−0.65	−1.76	5.69*	−0.53	−0.14	−0.14

Note. *p<0.5.

Table 5.

Adjusted Residuals Table of Cognitive Processing.

	B1	B2	B3	B4	B5	B6	B7
B1	18.8*	−11.15	0.9	−0.75	−0.76	−3	−6.43
B2	−11.37	16.38*	0.06	−0.82	−1.83	0.11	−10.06
B3	0.9	−0.45	−0.13	−0.25	−0.39	−0.24	−0.21
B4	−1.74	0.25	−0.25	10.22*	0.59	−0.46	−0.69
B5	−2.21	0.69	−0.39	−0.75	10.54*	4.71*	−2.49
B6	−2.33	−2.06	−0.24	1.72	12.89*	19.47*	−2.08
B7	−6.51	−10.42	−1.11	0.26	−2.19	−1.59	28.36*

Note. *p<0.5.

The behavioral patterns of the system logs shown in Figure 3 shows that when the students were operating the system, the behavioral sequences of significance can be divided into three types, that is, (a) logging in (LI), logging out (LO) and accessing (AC); (b) adding and modifying answers (Steps 6–8) in CPS activities; and (c) clicking on concept maps. The operation procedure follows the original design concept. In other words, the system tutors the students. However, as stated in the previous section, no concept map hint behavior occurred. The LSA data indicate that when the students incorrectly answered a question, there was a continuous significant sequence behavior change from concept map wrong (CW) to concept map close (CC), suggesting that the students did not respond appropriately to the hint function. From the interviews conducted after the activity, it seems that students tended to have another round of discussions or to find the answer to the question before writing their answers on the answer sheet. Therefore, future systems should be designed such that when a student incorrectly answers a question, the system offers a hint, and the student is then allowed to answer the question again. However, if the student still cannot provide the correct answer, the system would display a message that allows the student to search for the information online. In addition, the results are consistent with Wu et al. (2016), that when students are using the system and having discussions, they will be focusing on only system operations or online discussions at one time.

The behavioral patterns of cognitive processing shown in Figure 4 indicate that there were higher cognitive behaviors, that is, analyze (B4), evaluate (B5), and create (B6)), although the frequencies were not high. There were significant sequences between the evaluate (B5) and create (B6) behaviors, that is, evaluate (B5)→create (B6) and create (B6)→evaluate (B5). This phenomenon has rarely been reported in previous studies. Comparing a few studies of online discussion studies by LSA (e.g., Lin et al., 2014; Wu et al., 2012), these studies do not reveal significance on evaluate (B5) and create (B6)'s behavior patterns interaction. In other words, the content of high-level cognitive discussions is affirmed in the design of this study, probably because teachers employ concept maps in the scaffolding-based system and allow students to engage in activities according to a rundown of the activity, thus stimulating more higher level discussions. In addition, the comparison of the concept map research conducted through LSA also showed that concept map can indeed guide students to engage in more focused and higher level conversations (Wu et al., 2016). However, the frequency of HCP is low, which may be attributable to the questions designed by the teacher. Thus, it is recommended that teachers pay attention to the design of the questions in follow-up exploration of HCP.

Conclusions and Suggestions

To address students' lack of HCP in online discussions, we reviewed relevant studies and scaffolding mechanism and developed a Facebook group-based online discussion system using a combination of CPS and concept maps. With this system, teachers can plan for and improve students' HCP. We researched and examined the system using QCA and LSA to verify the feasibility of the system and its ability to improve HCP, to understand the cognitive processing during online discussions, and to offer recommendations for future implementation.

We infer from the results that, first, from LSA of operation behaviors, the system operations are affected by the designed system workflow. In other words, the system has a cognition guidance function (Wei, He, Chen, Zhou, & Tang, 2017). Thus, the appropriate teaching strategies or cognitive tools may be incorporated into the systems developed for future studies to acquire a cognition guidance function, which is similar to the function of completing online learning sheets when visiting museums (e.g., Hou et al., 2014). A well-designed mechanism may considerably lower the workload of instructors in an online learning environment. Second, from operation behaviors of the concept, it is necessary for teachers or researchers to consider the appropriateness and difficulty of questions for students when designing questions for discussion. In addition, students' prior knowledge and the cognitive design of instructional strategies should also be taken into consideration.

Third, this study explores how “hints” can be used when scaffolding instruction is carried out (Saleh et al., 2018) in hopes that students are able to collect data and discuss over online platforms via Q&A to form concepts of the field of knowledge. It is advised that a hint function be incorporated into the design of the system to encourage students to answer questions themselves rather than searching for answers on the Internet. Fourth, the incorporation of appropriate teaching strategies and the application of relevant cognitive tools may improve higher cognitive behaviors. However, these steps would require that the questions align with objectives and strategies. In other words, in addition to the guidance from the system, questions that demand and deserve exploration and thinking will also affect students' discussions and their higher cognitive behaviors. Fifth, by incorporating teaching strategies that encompass scaffolding functions and applying the appropriate cognitive tools, it is possible to reduce the occurrence of off-topic discussions. Thus, students will be better able to focus on the learning activities (Wu et al., 2016).

Overall, the design of this online discussion mechanism can really develop students' HCP. This mechanism is more effective than most online learning activities and serves as useful reference for many researchers to consider the pros and cons of online learning communities.

Finally, a system that incorporates an appropriate teaching strategy and uses cognitive tools for improving HCP was developed based on the literature and then verified. Because this study was a single-case study design, it is advised that future researchers incorporate various teaching strategies and cognitive tools into their teaching systems to explore in-depth the benefits of this method. In addition, though the fact that this study is based on an online learning environment with QCA and LSA, and it can compensate for the deficiencies of quantitative research. Qualitative analysis is recommended to apply for more in-depth analysis, including the discussions between study field participants (teachers, students, and researchers) and students, as well as the interaction relationship of Bloom's taxonomy between different categories.

Footnotes

Article Notes

References

Anderson

L. W.

Krathwohl

D. R.

(2001) A Taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives, New York, NY: Addison Wesley Longman, Inc.

Anderson

Rourke

Garrison

D. R.

Archer

(2001) Assessing teaching presence in a computer conference context. Journal of Asynchronous Learning Networks 5(2): 1–17.

Anderson, L. (2006). Revised Bloom's Taxonomy. Paper presented at North Carolina Career and Technical Education Curriculum Development Training, Raleigh, NC.

Agrawal R., Nandanwar, S., & Musti, N. M. (2017). Grouping students for maximizing learning from peers. In International Conference on Educational Data Mining, 156–161.

Apino, E., & Retnawati, H. (2017, February). Developing instructional design to improve mathematical higher order thinking skills of students. In Journal of Physics: Conference Series (Vol. 812, No. 1, p. 012100). Bristol, England: IOP Publishing.

Bakeman

Gottman

J. M.

(1997) Observing interaction: An introduction to sequential analysis, 2nd. Cambridge, England: Cambridge University Press.

Bloom

B. S.

(1956) Taxonomy of educational objectives. Vol. 1: Cognitive domain, New York, NY: McKay, pp. 20–24.

Bloom

B. S.

(1971) Handbook on formative and summative evaluation of student learning, New York, NY: McGraw-Hill.

Boud

Cohen

Sampson

(2014) Peer learning in higher education: Learning from and with each other, Abingdon, England: Routledge.

10.

Broadbent

Poon

W. L.

(2015) Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. The Internet and Higher Education 27: 1–13.

11.

Chiu

M. M.

Khoo

(2005) A new method for analyzing sequential processes: Dynamic multilevel analysis. Small Group Research 36(5): 600–631.

12.

De Wever

Schellens

Van Keer

Valcke

(2008) Structuring asynchronous discussion groups by introducing roles: Do students act in line with assigned roles?. Small Group Research 39(6): 770–794.

13.

De Backer

Van Keer

Valcke

(2016) Eliciting reciprocal peer-tutoring groups' metacognitive regulation through structuring and problematizing scaffolds. The Journal of Experimental Education 84(4): 804–828.

14.

Formanek

Wenger

M. C.

Buxner

S. R.

Impey

C. D.

Sonam

(2017) Insights about large-scale online peer assessment from an analysis of an astronomy MOOC. Computers & Education 113: 243–262.

15.

Fong

Slotta

J. D.

(2018) Supporting communities of learners in the elementary classroom: The common knowledge learning environment. Instructional Science 46(4): 533–561.

16.

Guimarães

Antunes

D. R.

García

L. S.

Fernandes

(2013) Collaborative consensus and knowledge creation: Computer-mediated methodology for sign language studies. In: M. D. Lytras, D. Ruan, R. D. tennyson, P. O. De Pablos, F. J. García Peñalvo, & L. Rusu (Ed.), Information Systems, E-learning, and Knowledge Management Research, Berlin, Germany: Springer, pp. 278–292.

17.

V. W.

Harris

P. G.

Kumar

R. K.

Velan

G. M.

(2018) Knowledge maps: A tool for online assessment with automated feedback. Medical Education Online 23(1): 1457394.

18.

Hou

H. T.

Wang

S. M.

Lin

P. C.

Chang

K. E.

(2015) Exploring the learner's knowledge construction and cognitive patterns of different asynchronous platforms: Comparison of an online discussion forum and Facebook. Innovations in Education and Teaching International 52(6): 610–620.

19.

Hou

H. T.

S. Y.

Lin

P. C.

Sung

Y. T.

Lin

J. W.

Chang

K. E.

(2014) A blended mobile learning environment for museum learning. Educational Technology & Society 17(2): 207–218.

20.

Hou

H. T.

Chang

K. E.

Sung

Y. T.

(2007) An analysis of peer assessment online discussions within a course that uses project–based learning. Interactive Learning Environments 15(3): 237–251.

21.

Hou

H. T.

Chang

K. E.

Sung

Y. T.

(2008) Analysis of problem–solving based online asynchronous discussion pattern. Educational Technology & Society 11(1): 17–28.

22.

Hou

H. T.

(2010) Exploring the behavioural patterns in project–based learning with online discussion: Quantitative content analysis and progressive sequential analysis. Turkish Online Journal of Educational Technology 9(3): 52–60.

23.

Hou

H. T.

S. Y.

(2011) Analyzing the social knowledge construction behavioral patterns of an online synchronous collaborative discussion instructional activity using an instant messaging tool: A case study. Computers & Education 57: 1459–1468.

24.

Hou

H. T.

(2012) Analyzing the learning process of an online role-playing discussion activity. Educational Technology & Society 15(1): 211–222.

25.

Howes

E. V.

Cruz

B. C.

(2009) Role–playing in science education: An effective strategy for developing multiple perspectives. Journal of Elementary Science Education 21(3): 33–46.

26.

Johnson

(2008) The relative learning benefits of synchronous and asynchronous text–based discussion. British Journal of Educational Technology 39(1): 166–169.

27.

Jeong

A. C.

(2003) The sequential analysis of group interaction and critical thinking in online threaded discussions. American Journal of Distance Education 17(1): 25–43.

28.

Jalil

H. A.

McFarlane

Ismail

I. A.

Krauss

S. E.

(2008) Assisted performance in different task types of online discussion. European Journal of Scientific Research 22(3): 329–339.

29.

Jorczak, R. L. (2009). The effects of task characteristics on online discussion. In C. O'Malley, D. Suthers, P. Reimann, & A. Dimitracopoulou (Eds.), Proceedings of the 9th international conference on computer supported collaborative learning, 1, 586–595.

30.

Jin

Jeong

(2013) Learning achieved in structured online debates: Levels of learning and types of postings. Instructional Science 41(6): 1141–1152.

31.

Koh

J. H. L.

Herring

S. C.

Hew

K. F.

(2010) Project–based learning and student knowledge construction during asynchronous online discussion. The Internet and Higher Education 13(4): 284–291.

32.

Kilic

(2018) Pre-service mathematics teachers' noticing skills and scaffolding practices. International Journal of Science and Mathematics Education 16(2): 377–400.

33.

Kim

J. E.

Park

Jang

Nam

(2017) Exploring flipped classroom effects on second language learners' cognitive processing. Foreign Language Annals 50(2): 260–284.

34.

Korhonen

A. M.

Ruhalahti

Veermans

(2018) The online learning process and scaffolding in student teachers' personal learning environments. Education and Information Technologies 24: 755–779.

35.

Lehmann-Willenbrock

Allen

J. A.

Kauffeld

(2013) A sequential analysis of procedural meeting communication: How teams facilitate their meetings. Journal of Applied Communication Research 41(4): 365–388.

36.

Lin

P. C.

Hou

H. T.

S. Y.

Chang

K. E.

(2014) Exploring college students' cognitive processing patterns during a collaborative problem-solving teaching activity integrating Facebook discussion and simulation tools. The Internet and Higher Education 22: 51–56.

37.

Lee, W. J. (2012). Exploring Students' Mapping Behaviors and Interactive Discourses in a Case Diagnosis Problem: Sequential Analysis of Collaborative Causal Map Drawing Processes. ProQuest LLC. 789 East Eisenhower Parkway, PO Box 1346, Ann Arbor, MI 48106.

38.

Lampe

Wohn

D. Y.

Vitak

Ellison

N. B.

Wash

(2011) Student use of Facebook for organizing collaborative classroom activities. International Journal of Computer-Supported Collaborative Learning 6(3): 329–347.

39.

Lin

J. M. C.

Liu

S. F.

(2012) An investigation into parent-child collaboration in learning computer programming. Journal of Educational Technology & Society 15(1): 162–173.

40.

Lye

S. Y.

Koh

J. H. L.

(2014) Review on teaching and learning of computational thinking through programming: What is next for K-12?. Computers in Human Behavior 41: 51–61.

41.

Mazman

S. G.

Usluel

Y. K.

(2010) Modeling educational usage of Facebook. Computers & Education 55(2): 444–453.

42.

Morueta

R. T.

López

P. M.

Gómez

Á. H.

Harris

V. W.

(2016) Exploring social and cognitive presences in communities of inquiry to perform higher cognitive tasks. The Internet and Higher Education 31: 122–131.

43.

Novak

J. D.

Gowin

D. B.

(1984) Learning how to learn, New York, NY: Cambridge University Press.

44.

Nelson

L. M.

(1999) Collaborative problem solving. In: Reigeluth

C. M.

(ed) Instructional-design theories and models: A new paradigm of instructional theory, Mahwah, NJ: Erlbaum Associates.

45.

Nesbit

J. C.

Adesope

O. O.

(2006) Learning with concept and knowledge maps: A meta-analysis. Review of Educational Research 76: 413–448.

46.

O'Donnell

A. M.

Dansereau

D. F.

Hall

R. H.

(2002) Knowledge maps as scaffolds for cognitive processing. Educational Psychology Review 14: 71–86.

47.

Ouyang

Scharber

(2017) The influences of an experienced instructor's discussion design and facilitation on an online learning community development: A social network analysis study. The Internet and Higher Education 35: 34–47.

48.

Pollastri

A. R.

Epstein

L. D.

Heath

G. H.

Ablon

J. S.

(2013) The collaborative problem solving approach: Outcomes across settings. Harvard Review of Psychiatry 21(4): 188–199.

49.

Pifarre

Cobos

(2010) Promoting metacognitive skills through peer scaffolding in a CSCL environment. International Journal of Computer-Supported Collaborative Learning 5(2): 237–253.

50.

Rabbany

ElAtia

Takaffoli

Zaïane

O. R.

(2014) Collaborative learning of students in online discussion forums: A social network analysis perspective. In: Educational data mining, Cham, Switzerland: Springer International Publishing, pp. 441–466.

51.

Raspopovic

Cvetanovic

Medan

Ljubojevic

(2017) The effects of integrating social learning environment with online learning. The International Review of Research in Open and Distributed Learning 18(1): 141–160.

52.

Roessger

K. M.

Daley

B. J.

Hafez

D. A.

(2018) Effects of teaching concept mapping using practice, feedback, and relational framing. Learning and Instruction 54: 11–21.

53.

Şendag

Odabaşi

H. F.

(2009) Effects of an online problem based learning course on content knowledge acquisition and critical thinking skills. Computers & Education 53: 132–141.

54.

Simon

Maloney

(2007) Activities for promoting small-group discussion and argumentation. School Science Review 88(324): 49–57.

55.

Stahl

(2006) Supporting group cognition in an online math community: A cognitive tool for small-group referencing in text chat. Journal of Educational Computing Research 35(2): 103–122.

56.

Santangelo

(2009) Collaborative problem solving effectively implemented, but not sustained: A case for aligning the sun, the moon, and the stars. Exceptional Children 75(2): 185–209.

57.

Smith

S. U.

Hayes

Shea

(2017) A critical review of the use of Wenger's Community of Practice (CoP) theoretical framework in online and blended learning research, 2000-2014. Online Learning 21(1): 209–237.

58.

Saleh, A., Hmelo-Silver, C. E., Chen, Y., Shanahan, K., Rowe, J., & Lester, J. (2018). Scaffolding peer facilitation in computer-supported problem-based learning environments. In J. Kay & R. Luckin (Eds.), Rethinking learning in the digital age: Making the learning sciences count, 13th International Conference of the Learning Sciences (ICLS) 2018 (Vol. 3). London, England: International Society of the Learning Sciences.

59.

Sung

H. Y.

Hwang

G. J.

(2018) Facilitating effective digital game-based learning behaviors and learning performances of students based on a collaborative knowledge construction strategy. Interactive Learning Environments 26(1): 118–134.

60.

Topping

K. J.

(2010) Methodological quandaries in studying process and outcomes in peer assessment. Learning and Instruction 20(4): 339–343.

61.

Tseng

F. C.

Kuo

F. Y.

(2014) A study of social participation and knowledge sharing in the teachers' online professional community of practice. Computers & Education 72: 37–47.

62.

Wang

Lin

C. F. C.

W. C. W.

(2013) Meaningful engagement in Facebook learning environments: Merging social and academic lives. Turkish Online Journal of Distance Education 14(1): 302–322.

63.

Wang

Cheng

Chen

Mercer

Kirschner

P. A.

(2017) The use of web-based collaborative concept mapping to support group learning and interaction in an online environment. The Internet and Higher Education 34: 28–40.

64.

Wei

Chen

Zhou

Tang

(2017) Collaborative filtering and deep learning based recommendation system for cold start items. Expert Systems with Applications 69: 29–39.

65.

P. H.

Hwang

G. J.

Milrad

H. R.

Huang

Y. M.

(2012) An innovative concept map approach for improving students' learning performance with an instant feedback mechanism. British Journal of Educational Technology 43(2): 217–232.

66.

S. Y.

Hou

H. T.

Hwang

W. Y.

(2012) Exploring students' cognitive dimensions and behavioral patterns during a synchronous peer assessment discussion activity using Instant Messaging. The Asia-Pacific Education Researcher 21(3): 442–453.

67.

S. Y.

Hou

H. T.

Hwang

W. Y.

Liu

E. Z. F.

(2013) Analysis of learning behavior in problem solving-based and project-based discussion activities within the seamless online learning integrated discussion (SOLID) system. Journal of Educational Computing Research 49(1): 61–82.

68.

S. Y.

Chen

S. Y.

Hou

H. T.

(2015) A study of users' reactions to a mixed online discussion model: A lag sequential analysis approach. International Journal of Human-Computer Interaction 31(3): 180–192.

69.

Wu, S. Y. (2015). A possible mechanism for enhancing the advanced knowledge construction in online learning communities. The International Conference on Education in Mathematics, Science & Technology (ICEMST), Turkey, Antalya.

70.

S. Y.

Chen

S. Y.

Hou

H. T.

(2016) Exploring the interactive patterns of concept map-based online discussion: A sequential analysis of users' operations, cognitive processing, and knowledge construction. Interactive Learning Environments 24(8): 1778–1794.