Abstract
Data collected from an educational experiment were analyzed to investigate the effectiveness of an instructional reform referred to as the inverted classroom (learning activities traditionally taking place inside the classroom now take place outside the classroom and vice versa) in undergraduate statistics education. Students were taught the same statistics course in the inverted classroom (135 students) and the traditional classroom (135 students) over one semester. A small sample HLM (hierarchical linear modeling) approach was used to account for the data structure of students nested within 12 sections of the course and small sample sizes at both student and section levels. The treatment effects were adjusted for individual background characteristics and high school performances. There were positive treatment effects of the inverted classroom across all three outcome measures of coursework, attendance, and final grade. After adjustment, the inverted classroom would increase scores in coursework, attendance, and final grade by 3.47, 2.14, and 1.18 percentage points respectively, representing a limited success of the inverted classroom.
Keywords
Introduction
There has been an ever-increasing emphasis on improving students’ ability to reason statistically, because the modern society is characterized as the age of information in which people must become statistically literate not only for the society to survive economically but also for the individuals to survive personally (e.g., Raffaghelli et al., 2020; Sharma, 2017). Towards this noble goal, curricular and instructional reforms have kept emerging at all levels of statistics education, with a common focus on processing quantitative information via collecting and interpreting data as well as drawing inferences and making predictions (e.g., Common Core State Standards Initiative, 2024; Watson & Smith, 2022). Some of those educational reforms are very popular as a response to the persistent difficulty of understanding statistical reasoning among students at every educational level.
How effective are those educational reforms? The lack of credible answers in the research literature is troublesome. There are very few direct connections between practice and research (e.g., Kula & Koçer, 2020). The present study was a response to this lack. Using data from a recent educational experiment, the effectiveness of the inverted classroom was examined in the present study. The inverted classroom is the instructional practice in which learning activities that traditionally take place inside the classroom now take place outside the classroom, meanwhile learning activities that traditionally take place outside the classroom now take place inside the classroom. The inverted classroom has become a worldwide innovative instructional reform in mathematics education (Cevikbas & Kaiser, 2020). Empirical studies are needed to provide insights on the extent to which the inverted classroom is a solid educational measure.
Review of literature
Statistical challenge
Inappropriate statistical reasoning is both widespread across all areas of statistics and persistent across all levels of education (Setiawan & Sukoco, 2021). For example, Huck (2015) discussed 52 statistical misconceptions concerning data and statistical summaries alone that are common in undergraduate and graduate courses in statistics, research methods, and quantitative analysis. At the undergraduate level, there has long been a call for improving the teaching and learning of undergraduate statistics courses (e.g., Legaki et al., 2020; Sanchez, 2023). Karkelanova (2019) discussed in detail the challenges facing undergraduate statistics education. The cognitive challenge speaks to the fact that students see statistics as being difficult to understand (e.g., counterintuitive) and being impossible to do well (e.g., too complex); the affective challenge speaks to the negative dispositions of students toward statistics (e.g., negative attitudes, negative beliefs, and high anxieties); and the pedagogical challenge speaks to the fact that statistics is traditionally taught as mathematical science with a strong emphasis on theoretical coherence and practical computation.
Educational solution
To overcome the challenges aforementioned, the traditional approach, still popular (and useful), is to seek an interface among curriculum, instruction, and technology to change the focus of undergraduate statistics education from emphasizing the “what” to emphasizing both the “how” and the “why” (see Johannssen et al., 2021; Sabbag et al., 2018). Specifically, Karkelanova (2019) argued for a meaningful shift in both curriculum and instruction in undergraduate statistics education. One strategy that can facilitate the shift is to focus on the development of critical and fundamental statistical concepts in undergraduate statistics courses. Under this strategy, the key theme is to carry out a sequence of both carefully and purposefully designed learning activities aimed to increase the understanding of undergraduate students on those critical and fundamental statistical concepts (see also Bromage et al., 2021). The inverted classroom thus comes into the picture as a potentially effective way to implement this strategy.
Inverted classroom
It may be helpful to compare the typical inverted classroom with the typical traditional classroom as a way to introduce the inverted classroom. Students in the inverted classroom watch video lectures before they attend class. This learning activity traditionally takes place inside the classroom, but it now becomes homework (assignments) outside the classroom. When students come to class, they complete a series of learning activities designed to engage them in discovery learning of critical and fundamental statistical concepts that they have already studied by watching video lectures. Traditionally, this learning activity is for students to do independently after class as homework (assignments), but it now becomes the interactions of students with the instructor and with one another in class. In contrast, in the traditional classroom, students come to class to listen to a lecture. During the lecture, students usually have opportunities to interact with the instructor (i.e., asking the instructor questions and answering questions from the instructor). After class, learning activities (homework) are assigned to students, aimed at reviewing and practicing the content covered during the lecture.
The inverted classroom transforms the instructional role of the instructors; they are still the source of knowledge and skills but are no longer the center of instructional activities (Roehling & Bredow, 2021). To implement the inverted classroom, instructors are “forced” to create an inquiry-based, discovery-oriented instructional environment where the center of learning shifts from instructors to students and the learning experience is tailored to facilitate the understanding of all students (Ben-Zvi et al., 2018; Rock et al., 2016). The inverted classroom is especially popular in undergraduate STEM (science, technology, engineering, and mathematics) education where the reform brings hope to overcome the persistent difficulties in the teaching and learning of STEM courses (Overmyer, 2014).
There are at least three advantages of the inverted classroom over the traditional classroom (Morgan et al., 2015). First, the inverted classroom can increase time for instructional interactions between the instructor and students without sacrificing curricular contents. Second, in the inverted classroom, the instructor can present the same curriculum in different formats of instruction so as to engage students of different learning styles. Finally, the inverted classroom can train students to become a self-regulated learner for life, given that self-regulated learning is regarded as the highest level of individual intellectual pursuit. In much more detail, Fulton (2012) adequately provided a classic summary of strong points of the inverted classroom. Students learn at their own pace; inverting homework (assignments) into the classroom gives instructors a better understanding of the learning difficulties and learning styles among students; instructors are given the opportunity to customize and update curricular contents around the clock; instructors gain more classroom time to focus on important curricular contents; the inverted classroom promotes interest, engagement, and achievement among students; learning theories support the conception of the inverted classroom; there are promising opportunities in the inverted classroom for instructors to flexibly apply technology to curriculum and instruction; the inverted classroom creates more time for students to engage in authentic research under the guidance of instructors; students gain more time to operate scientific equipment that must remain in the classroom; the inverted classroom makes it easier for students who are absent from class to catch up; there is increased time for students to keep reasoning (thinking) because it is now a required task both inside and outside the classroom; the inverted classroom promotes students to get actively involved in the learning process; and finally many students come to enjoy the inverted classroom (so as to become motivated towards and engaged in the learning process) (see also Day, 2018).
Nonetheless, it may come as a surprise to many educators that the empirical base for the effectiveness of the inverted classroom is rather thin, with supportive evidence for the reform coming mainly from social media which is not sufficient to be labeled as empirical evidence (i.e., research-based evidence) (e.g., Goodwin & Miller, 2013; Zainuddin et al., 2019). For example, the Flipping with Kirch blog (http://flippingwithkirch.blogspot.ca) has been actively supporting the reform of the inverted classroom on the basis of individual classroom instructional experiences. The lack of research-based evidence in the literature adequately justified the purpose of the present study aimed to generate empirical insights into the effectiveness of the inverted classroom.
Method
Context
The University of Kentucky, a southern state in the United States, started to reform its general education program in 2005 and put in place a new program in 2009. The overarching theme of the new program is to reform the base of curriculum and instruction from cultural actions built upon logical opinions to cultural actions built upon evidence-based reasoning. As a result, reasoning (thinking) takes the central stage. Among a set of four primary learning outcomes is quantitative reasoning, demanding students to demonstrate knowledge (understanding) and skills (abilities) in terms of effective applications of quantitative reasoning methods to solve the real-world problems. Undergraduate statistics education has been tasked to fulfill this primary learning outcome.
As part of the effort, created in 2010, STA 210 is a conceptual statistics course, algebra-based with an emphasis on statistical concepts rather than mathematical manipulations. Statistics courses like this belong to the genre often labeled as “statistics for poets” in the United States, because they require much more reading and writing about conceptual statistical ideas and principles than do traditional statistics courses (of the same level). STA 210 adopts a curricular approach of less is more to focus in detail on the understanding of just three topics or modules (human inference, confidence interval, and hypothesis testing). There is no required textbook for the course, and students work with a workbook entitled Beyond the numbers: Student-centered activities for learning statistical reasoning (Rayens, 2014). This workbook contains hands-on daily exercises, large projects, and capstone projects. Since 2010, STA 210 has been taught in the inverted classroom with lectures traditionally happening inside the classroom being moved to outside the classroom to make room for interactive learning activities. To facilitate the implementation of the inverted classroom, there are a total of 18 videos available on YouTube for all students in the course. To evaluate the effectiveness of the inverted classroom, an educational experiment was planned for the fall semester of 2014 (see Karkelanova, 2019).
Data
Data for the present study came from Karkelanova (2019) in which data were collected during the fall semester of 2014 from students taught in two different instructional methods in STA 210. Designed as an educational experiment, one instructional method was the inverted classroom, the experiment condition (EXP) and the other instructional method was the traditional classroom, the control condition (CON). In the literature review earlier, the example of the typical inverted classroom precisely illustrated the instructional practice in EXP, and the example of the typical traditional classroom precisely illustrated the instructional practice in CON (see the section of Inverted Classroom). There were 135 students in EXP and 135 students in CON for a total sample size of 270 for the present study.
To make data from EXP and CON comparable for statistical analysis, six EXP sections and six CON sections were offered to students simply as STA 210 with 12 sections. Students did not know prior to enrollment whether they would take the course in EXP or CON (note that the University Registrar confirmed that teaching styles do not need to be communicated to students prior to enrollment). So, students were not randomly but blindly assigned to EXP and CON. The same instructor taught students in EXP and CON in similar classrooms and during similar daytime. Each group (EXP or CON) met twice each week for two classes of one hour each and a third time in an hour-long section-based recitation for a total of three class hours each week. Of course, students in EXP learned through the inverted classroom, and students in CON learned in the traditional classroom. Both groups shared the same course materials, course assignments, and major projects throughout the entire semester. In addition, assignments, projects, tests, and exams were the same for both groups and were given to students at similar times. All of the above measures functioned mainly to control the learning environment in which students in EXP and CON took the course.
Measures
The outcomes in the present study were measures of coursework, attendance, and final (course) grade of students in both EXP and CON. Coursework was an aggregated measure (in percentage) of assignments (both inside and outside class) and major projects. Attendance measured (in percentage) students’ level of participation in the course. Final grade was an aggregated measure (in percentage) of tests and exams. In STA 210, multiple-choice questions and open-ended questions are used to evaluate various aspects of critical statistical reasoning identified in the course (see earlier discussion on STA 210). Under the guidance of a faculty advisory committee, tests and exams in the course have been revisited many times for improvement, with the current ones considered both valid and reliable. Students are given study guides to prepare prior to taking those tests and exams.
The predictors in the present study captured students’ individual background characteristics and high school performances. Although the blind assignment of students to EXP and CON could help achieve randomization to a certain degree, predictors descriptive of students would go further to compensate the lack of true random assignment (i.e., they would help adjust for potential selection bias). Individual background characteristics included gender (male or female), age (continuous), race (White or Non-White), and financial aid (yes or no) as an indicator for socioeconomic status (SES). High school performances included high school GPA (grade point average) and ACT (mathematics score), functioning as measures of prior academic ability.
Analyses
As mentioned earlier, there were 12 sections in STA 210 (six under EXP and six under CON), and learning activities occurred within each section. Active interactions among students in EXP were encouraged to promote learning in each section over a period of one semester. As a result, the assumption on the independence of observations became questionable. Research scenarios such as this clearly call for an HLM (hierarchical linear modeling) approach to data analysis (e.g., Raudenbush & Bryk, 2002). HLM, also referred to as multilevel modeling, works with data with a certain hierarchical structure such as students nested within sections in the present study. The challenge in the present study was the small sample size at the student level (i.e., 270 students) and especially at the section level (i.e., 12 sections). Such small sample sizes would make a straightforward application of HLM a questionable statistical practice (see Maas & Hox, 2005). Consequently, the present study employed a small sample HLM approach to data analysis. Based on the recommendation in Hox and McNeish (2020), the present study adopted the Bayesian perspective to carry out small sample HLM analyses. Stata 17 platform was used to perform those Bayesian analyses.
One small sample HLM analysis was performed on each outcome measure (coursework, attendance, and final grade). This univariate strategy indicated the research intention to examine each outcome in initial, specific detail to lay the foundation for any potential multivariate strategy. For each analysis, the main simulation was carried out using the adaptive Metropolis-Hastings (MH) Markov chain Monte Carlo (MCMC) method. Stata 17 recommends 12,500 iterations of MCMC with a burn-in period of 2,500 and a MCMC sample size of 10,000 to ensure adequate accuracy of estimated parameters. Oftentimes, 10,000 iterations are considered adequate to produce accurate results (accurate to about 1 decimal place). Burn-in is the analytical practice of removing a number of iterations (2,500 in this case) when MCMC starts to run, also referred to as the burn-in period (where results do not count). This term comes from electronics manufacturing in which parts tend to fail in the initial runs. Burn-in is then performed to remove bad products so as to build more reliable ones to ensure quality. So, 10,000 + 2,500 = 12,500 MCMC iterations are recommended by Stata 17. The MCMC sample size speaks to the fact that MCMC draws samples from a probability distribution repeatedly, but those samples are not independent. Effective sample size (ESS) is often used to assess the convergence of sums of MCMC samples (in a heuristic way). Basically, ESS estimates the rate between dependent samples and independent samples (e.g., 1,000 correlated MCMC samples may be equivalent to about 100 independent samples). With the above working principles, the Bayesian method of small sample HLM was performed to address the research questions in the present study.
Results
Table 1 presents descriptive information on the outcomes and predictors for students in both EXP and CON. Although descriptive information was mostly comparable between EXP and CON, the use of those predictors for adjustment for selection bias appeared to be necessary (e.g., in terms of gender with a large gap). All of those predictors were present as adjustment for each of the three small sample HLM analyses that were the focus of the interpretation here.
Descriptive information on students in inverted classroom and traditional classroom.
Descriptive information on students in inverted classroom and traditional classroom.
Note. SD = standard deviation. Financial aid is used as a measure of socioeconomic status.
Table 2 presents the treatment effects of the inverted classroom (compared with the traditional classroom) on coursework. In total, three models were run. The null model contained coursework (score), the outcome measure, without anything else. The absolute model introduced the treatment dummy (the inverted classroom versus the traditional classroom) with the absence of predictors, and the treatment dummy thus estimated the absolute treatment effects of the inverted classroom on coursework. The relative model added predictors in addition to the treatment dummy, and thus the treatment dummy in the presence of predictors estimated the relative treatment effects of the inverted classroom on coursework (i.e., with control over the effects of the predictors).
Analytical results of small sample HLM on effects of inverted classroom on coursework.
Note. The null model contains the outcome measure (coursework score) only. The absolute model contains the treatment dummy with the absence of predictors. The relative model contains the treatment dummy with the presence of predictors.
Because probability theory is the foundation of Bayesian statistics, the interpretation of Bayesian results often avoids the traditional (frequentist) notions of p value and statistical significance. Although this avoidance remains still a topic of debate, Bayesian results contain enough information (by themselves) for a good understanding of what has happened and how well it has happened. The interpretation of the Bayesian results in the present study would follow such a logical approach.
In Table 2, the absolute treatment effects were 2.77 percentage points, indicating that the inverted classroom would increase the coursework score by 2.77 percentage points. The relative treatment effects were 3.47 percentage points, indicating that, after controlling over the predictors (gender, age, race, financial aid as a measure of SES, high school GPA, and ACT), the inverted classroom would increase the coursework score by 3.47 percentage points. Because there were three models, model comparison was made using DIC (deviance information criterion). For DIC, the smaller, the better. The relative model, therefore, was identified as the best model. For the relative model, the MCSE (Monte Carlo standard error) for the posterior mean was estimated as .05. MCSE is defined as the standard deviation of the chains divided by their ESS, measuring the accuracy of the simulation (Bayesian) results (i.e., how serious the estimation noise is) (see Kruschke, 2014). For MCSE, the closer to zero, the better the estimate. With the MCSE = .05, the treatment effects were well estimated (i.e., in an accurate way). Finally, the treatment effects can also be appreciated from the change in the median associated with the treatment dummy. After controlling over the predictors, the inverted classroom would increase the median of EXP (over CON) in coursework score by 3.47.
Tables 3 and 4 can be interpreted in the same fashion. Table 3 presents the treatment effects of the inverted classroom (compared with the traditional classroom) on attendance (score). The relative model was identified as the best model (i.e., smallest DIC). After controlling over the predictors (gender, age, race, financial aid as a measure of SES, high school GPA, and ACT), the inverted classroom would increase the attendance score by 2.14 percentage points. With the MCSE = .07, the treatment effects were estimated accurately. The associated median increase was 2.09. Table 4 presents the treatment effects of the inverted classroom (compared with the traditional classroom) on final grade (score). Again, the relative model was identified as the best model (i.e., smallest DIC). After controlling over the predictors, the inverted classroom would increase the final grade score by 1.18 percentage points. With the MCSE = .08, the treatment effects were also estimated accurately. The associated median increase was 1.17.
Analytical results of small sample HLM on effects of inverted classroom on attendance.
Note. The null model contains the outcome measure (attendance score) only. The absolute model contains the treatment dummy with the absence of predictors. The relative model contains the treatment dummy with the presence of predictors.
Analytical results of small sample HLM on effects of inverted classroom on final grade.
Note. The null model contains the outcome measure (final grade score) only. The absolute model contains the treatment dummy with the absence of predictors. The relative model contains the treatment dummy with the presence of predictors.
Summary of findings
The present study attempted to adopt a small sample HLM approach to data analysis to sufficiently capture the complexity of the data in Karkelanova (2019). The treatment effects of the inverted classroom (EXP) were compared with the treatment effects of the traditional classroom (CON) concerning three outcome measures of coursework, attendance, and final grade in one undergraduate statistics course. The treatment effects were adjusted for students’ individual background characteristics (gender, age, race, and financial aid as a measure of SES) and high school performances (high school GPA and ACT) to reduce selection bias.
There were positive treatment effects of the inverted classroom (compared with the traditional classroom) across all three outcome measures of coursework, attendance, and final grade. After controlling over the predictors (gender, age, race, financial aid as a measure of SES, high school GPA, and ACT), the inverted classroom would increase coursework, attendance, and final grade scores by 3.47, 2.14, and 1.18 percentage points respectively. Although the treatment effects were positive on each and every outcome measure, the magnitude was at most moderate, with a range from 1.18 to 3.47 percentage points. As a result, the inverted classroom brought about moderate benefits to students in EXP (compared with students in CON). The (relatively) strongest treatment effects occurred in coursework, and the (relatively) weakest treatment effects occurred in final grade.
What can be learned?
Perhaps the best way to describe or characterize the inverted classroom as examined in the present study is its limited success. The findings that students in the inverted classroom did better than students in the traditional classroom do lay the foundation for the effectiveness of the inverted classroom. The present study added an important piece of empirical evidence so rare in the literature at this time. The positive evidence is also comprehensive across all three measures of learning outcomes (coursework, attendance, and final grade). To some extent, it is reasonable to consider the inverted classroom an effective instructional reform to strengthen the understating of critical and fundamental statistical concepts in undergraduate statistics education, arguably with room for much more dramatic improvement.
The expression used earlier, limited success, appears to be appropriate. The word, limited, refers to the fact that the magnitude of the treatment effects was not as dramatic as the present study came to hope on the learning outcome measures. If the improvement of 3.47 percentage points on coursework can be considered reasonable, then the improvement of 1.18 percentage points on final grade does dampen the excitement. Nonetheless, there is something to be learned here regarding the effectiveness of the inverted classroom. In the context of undergraduate statistics education, it appears that the inverted classroom would be most effective in improving coursework and, to a little less degree, attendance; meanwhile, the inverted classroom would be least effective at this time in improving final grade.
Interestingly, the current literature contains a meta-analysis summarizing the effects of the inverted classroom in undergraduate mathematics education, reporting a positive Cohen's d of .494 (p < .001) on mathematics achievement (Sopamena et al., 2023). This effect size falls into the small effect category (.2 to .5) and so indicates limited success of the inverted classroom in undergraduate mathematics education like in the present study. Overall, drawing support from this meta-analysis concerning the inverted classroom in undergraduate mathematics education, confidence increases for the present study in suggesting that after adopting the inverted classroom as an instructional reform in undergraduate statistics education, reasonable improvement can be expected in learning outcome measures.
What can be offered?
With limited success, there may need to be more effort in course development to fully exercise the reform idea of the inverted classroom for the purpose of larger improvement in multiple learning outcome measures. For example, if design and implementation of the inverted classroom can be considered the core components for a successful application of the inverted classroom, apparently more needs to be done there. The present study may function to bring out the complicity of the inverted classroom (see Karkelanova, 2019). One key issue is that undergraduate statistics courses are organizationally complex in that a number of parties are involved in the teaching and learning process, including instructors, teaching assistants, and students. As discussed earlier, in the present study, the three hours of weekly instructional activities included two classes of one hour each in which the instructor interacts with students in a large classroom plus one class of one hour long in which a teaching assistant works with students within each section of the course (referred to as section-based recitation). For the inverted classroom to be dramatically successful, the coordination and cooperation between instructors and teaching assistants are critical.
Obviously, both instructors and teaching assistants play a very important role in undergraduate statistics education. Stonebridge Colleges (2022) featured a good blog article that effectively illustrates the unique contribution of teaching assistants (even though not specifically about teaching assistants for undergraduate statistics education). Oftentimes across universities in the United States (and very likely worldwide), teaching assistants in undergraduate statistics education are graduate students in statistics (e.g., doctoral students). They are carefully selected to show great statistical knowledge and skills. For them to take part in the inverted classroom, some pedagogical training (e.g., on collaborative learning) is indeed provided, often in the form of a workshop. Is that enough training for teaching assistants to effectively work with students in the inverted classroom? Maybe a series of workshops carefully designed to let teaching assistants adequately capture the theory and practice of collaborative learning are needed (or maybe even a formal graduate course on the theory and practice of collaborative learning). Perhaps there may need to be more accountability measures to teaching assistants beyond simply student evaluation of the recitation. Meaningful improvement may also call for the involvement of faculty members to fully train and work with teaching assistants (e.g., observing and discussing the teaching in recitation of each teaching assistant). Overall, the pedagogical quality of teaching assistants, together with the close coordination and cooperation between them and instructors, may hold a critical key for more effective application of the inverted classroom.
Strength and weakness
From the perspective of an educational experiment, two methodological points are worthy of making. Can blind assignment (see discussion earlier) be used in the place of random assignment? Conceptually no, of course. A comparison between the absolute model (blind assignment without predictors) and the relative model (with predictors to adjust for selection bias) may illustrate (see Tables 2–4). For example, the treatment effects were 2.77 percentage points in the absolute model but 3.47 percentage points in the relative model in terms of coursework (Table 2). Although blind assignment may be considered an improvement over a direct use of volunteers, a good adjustment with predictors for selection bias (resembling random assignment) is necessary. In addition, the outcome measures (coursework, attendance, and final grade) were a strong point in Karkelanova (2019). Each measure captures a variety of learning activities (aspects) that make it realistic and meaningful, yet does not increase the measurement complexity (i.e., no psychometric analysis necessary).
In contrast, the age of the data was a weakness, with the educational experiment conducted in the fall of 2014 (see Karkelanova, 2019). The undergraduate population attending the University of Kentucky has remained stable in terms of the demographic and academic background characteristics, even during the COVID19 years. Although there is not any strong reason to believe that the results would be rather different if the same educational experiment were to be conducted today, the age of the data does indicate that the present study would provide more of a historical perspective concerning the inverted classroom. Nonetheless, the methodological undertaking in the present study opens doors indeed for upcoming waves of educational experiments to produce newer empirical evidence on the effectiveness of the inverted classroom in a much more credible way.
Concluding remark
Improving undergraduate statistics education is a common educational pursuit of many countries in the world because difficulties in teaching and learning undergraduate statistics are not unique to a certain country. The present study offered a piece of important empirical evidence on the effectiveness of the inverted classroom as it continues to expand its popularity as an instructional reform across the globe. It revealed the limited success of the inverted (student-centered) classroom over the traditional (instructor-centered) classroom. To make the success grow larger, potential improvements in the practice of the inverted classroom are offered not from empirical findings but from classroom experiences. Hopefully, new progress on both course development and empirical research will keep coming to provide a fuller picture on how to make the inverted classroom a dramatically effective instructional method for educational benefits beyond undergraduate statistics education.
Footnotes
Contributorship
Xin Ma conducted data analysis and drafted the manuscript. Jing Zhang conducted literature review and drafted the manuscript. Both authors read and approved the final manuscript. Petya Karkelanova performed data management and participated in data analysis. Michael C. Osborne provided important ideas and comments for the research. Anushka Karkelanova provided data and important materials for the construction of the manuscript.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Informed consent
This study was a secondary data analysis, utilizing existing data from a doctoral dissertation research (Karkelanova, 2019) approved by the Institutional Review Board of the University of Kentucky (protocol code: 14-0975-P4K and date of approval: 2 November 2016). Informed consent was obtained from all subjects involved in the doctoral dissertation research.
