The Use of Case Study Design in Learning Management System Research: A Label of Convenience?

Abstract

Case study design is often characterized as a poorly applied and understood approach to conduct research into information systems. This literature review examined 28 empirical studies of Learning Management Systems (LMS) that were self-declared as case studies. This paper’s main aim was to establish whether these studies actually applied case study techniques or simply utilized the term “case study” as a label of convenience. Alarmingly, more than one-third of the examined articles contained little or no justification for adopting a case study design and did not meet the established criteria for identification as a case study as outlined in this review. The lack of an appropriate discussion on the characteristics and applicability of the case study approach to empirical discovery brings into question the robustness of a study’s stated methodology. It is not sufficient to assume that the label “case study” by itself will portray a universal understanding of the research design of an empirical LMS study. This paper proposes that a robust description of what constitutes a case study in a LMS context will assist in ensuring that future studies on LMS (or other information systems) research that intend to adopt a case study approach, adequately present appropriate justifications for adopting this methodology. It is hoped that our definition and description of LMS case study methodology will serve to inspire further reflection on the characteristics of this robust research approach that might lead to a set of guidelines for applying this methodology to information systems research.

Keywords

case study methodology classification research design research method

Introduction

There are many views on the characteristics of case studies and how they should be employed to investigate a research problem. Case study research seeks to answer the “what,” “where” and “how” questions of a research problem, much like the questions formulated in constructing an experiment (Kyburz-Graber, 2004). Both qualitative and quantitative methods can be employed within a case study framework. The preferred paradigm for the case study methodology is realism (Perry, 1998). A realist approach assumes that knowledge is acquired through the researcher’s senses and is aptly suited to the task of examining phenomena within identifiable constraints. This raises the question of how to define a case? Gerring (2004) describes a case as being made up of several relevant dimensions constructed from single or multiple observations from which an understanding of a larger class of observed phenomena can be extracted. The design of a case study can include a single case or multiple cases. The advantage of multiple cases is that the identification of commonalities between cases strengthens the validity of emerging theory from the individual case studies (Tellis, 1997b). Cases can be categorized in three ways: (1) exploratory case studies where exploration of the phenomena is of interest to the researcher and needs to be discovered; (2) descriptive case studies that lead to the development of a narrative of the phenomena with reference to extant literature; and (3) explanatory case studies that ask the “why” questions about the observed phenomena which may lead to the establishment of theories (Zainal, 2007). Another interesting consideration is the motivation to study a case. Stake (1995) presents a motivational dichotomy which he calls intrinsic and instrumental. Intrinsic cases are those which are pursued solely to develop a deeper understanding of a phenomenon or situation by itself, whereas an instrumental case study involves investigating a particular case to develop a broader appreciation or understanding of more general problems or issues (Stake, 1995). Undertaking case study research is not an easy endeavor, and researchers should possess a basic toolkit of skills before undertaking case study research. Yin (1994) includes questioning, listening, adaptability, background knowledge of the phenomenon investigated, and an unbiased attitude as essential skills for researchers. The main strengths of developing theory from case study research are: (1) the likelihood of generating a novel theory; (2) emergent theory is likely to be testable with constructs that can be easily measured; and (3) the resultant theory has a high chance of being empirically valid because the theory building process is intimately tied with the evidence used to construct it (Eisenhardt, 1989).

In conducting case study research, triangulation is an important technique to apply to the research design. The concept, derived from navigational triangulation where a location was determined by the intersection of two points with the observer’s position, is used in research to examine issues or arguments from different perspectives (Simons, 2009). Tellis (1997a) confirms the importance of four types of triangulation: (1) data source triangulation which means that researchers look for similar patterns or themes in multiple types of data; (2) investigator triangulation when more than one researcher examines the same phenomenon; (3) theory triangulation that applies different investigator perspectives to the same phenomenon; and (4) methodological triangulation when different approaches and techniques are applied to the same phenomenon. The main objective of triangulation is to enhance the validity of case study findings by providing two or more sources of evidence that confirm emerging explanations for observed phenomena (Heale & Forbes, 2013).

Not all researchers agree that case study methodology is a robust research approach: “Case studies often lack academic rigor and are, as such, regarded as inferior to more rigorous methods where there are more specific guidelines for collecting and analyzing data. These criticisms stress that there is a need to be very explicit about the choices one makes and the need to justify them” (Meyer, 2001, p. 348). More recently, there have been claims that case studies are characterized by poor generalizability, take too long to conduct, produce large amounts of unreadable documents, and are often confused with case study teaching strategies used in disciplines like business (Gaikwad, 2017). There have also been claims that the case study methodology lacks the rigor of other approaches and cannot be considered a scientific method. Flyvbjerg (2006) identifies (and debunks) five misunderstandings of case studies that may impugn the selection of a case study methodology for an intended study: (1) theoretical knowledge has greater value than practical knowledge; (2) case studies cannot be generalized beyond a single case; (3) the effectiveness of case studies is in hypothesis generation, not hypothesis testing; (4) case studies have a tendency to validate a researcher’s preconceived ideas about a study; and (5) specific case studies are difficult to summarize and synthesize into general propositions and theories. However, Flyvbjerg is a proponent of the value of case study research in certain circumstances, particularly when used in conjunction with studies that involve large random samples of entire populations. He concludes with the comment that “In this connection, it is worth repeating the insight of Thomas Kuhn that a discipline without a large number of thoroughly executed case studies is a discipline without systematic production of exemplars, and that a discipline without exemplars is an ineffective one. In social science more good case studies could help remedy this situation” (p. 27).

Given the oft-cited “pros and cons” of case study methodologies, this paper seeks to investigate whether research projects related to the use of educational technology that purport to use case study methodologies, actually do so or merely use the term case study as a general descriptor for the research activity. To narrow the focus to a manageable area of educational technology, empirical studies of Learning Management Systems (LMSs) were examined to find out if the methodologies employed were consistent with established characteristics of a case study methodology. Turnbull et al. (2019) provide the following definition of LMSs: “Learning management systems can be defined as web-based software platforms that provide an interactive online learning environment and automate the administration, organization, delivery, and reporting of educational content and learner outcomes” (p. 1). In this COVID-19 environment, interest in LMSs has increased significantly as universities and schools continue to adapt their course offerings to online delivery (Huang et al., 2020; Moorhouse, 2020; Raza et al., 2020). Therefore, it is logical to assume that more research in this area will be forthcoming in the future. To better inform future studies in LMSs on the appropriateness of case study methodologies in research designs, this literature review examines the methodologies of prior “self-declared” empirical LMS research to determine which paper’s methodologies exhibit characteristics consistent with case study design, and what explanations, if any, are provided to justify methodological choices.

This paper is structured as follows. First, the approach taken to review the literature is outlined, including the databases used, the rationale for their selection, and explicit inclusion and exclusion criteria adopted for article selection. Next, the findings are presented with tabular overviews of three categories of studies based on the degree to which they adhered to popular principles of case study design. This is followed by a discussion section that prescribes a definition for case study methodology within the context of LMS research but with applicability to other information systems too. This paper’s conclusion emphasizes the need for research projects that explicitly declare the use of case study methodology and adequately address how their research designs align with the characteristics of case study practices.

Method

The approach taken in this review of relevant LMS empirical studies is to adopt a narrative approach to reporting the use of case study design in LMS research, while applying some of the rigor of systematic review methodologies to the selection and inclusion of relevant peer-reviewed studies. There is a precedence for this hybrid approach to conducting literature reviews in information systems-related fields, such as social media (Best et al., 2014; Chugh & Ruhi, 2018; Sarmiento et al., 2018), online learning (McCutcheon et al., 2015; Sinclair et al., 2016) and human-computer interactions (Boyle et al., 2012; Fleischer, 2012). The result is a more easily digestible overview of important issues and constructs that have appeal to both an academic and professional audience.

A pilot search was conducted using Google Scholar to test possible search terms and their synonyms. By process of trial and error, a search string was developed that was likely to capture the most relevant results. The search string applied to the databases in this study was: (“Learning Management System” OR LMS) AND “Case Study” AND (Impact OR effect OR improve* OR benefit) AND (learner OR student).

The main limiters applied to this search string were the abstract field, full text, and peer-reviewed. The year of publication was constrained to the range: 2008 to 2019. The abstract field limiter was deemed necessary to maximize the chance that each extracted paper dealt explicitly with LMSs, used a case study methodology, and was related to learning effectiveness in some way.

The following data sources were used in the search: Scopus, EBSCOhost, and Gale. The databases accessed by these systems covered a wide range of topics that had a high chance of including peer-reviewed articles on LMS research. The application of the search string to each data source yielded 30 Scopus articles, 34 papers from EBSCOhost, and two items from Gale. After removing duplicate entries and discounting results that did not include an accessible PDF file, the final tally of included papers was 28. These studies are presented in the following section.

Results

Table 1 provides a summary of the 28 studies in this review.

Table 1.

Overview of Case Studies.

Author (Year)	Study Aims	Case Study Design Described
Alenezi (2017)	This paper compares the use of mobile technologies to personal computers in aiding student learning. The research design is characterized by the use of surveys, which were administered to students and faculty (male and female). The data were analyzed using quantitative techniques (t-tests) to establish any differences in responses between groups. There was no evidence that a case study design was employed.	No
Altunoglu (2017)	This study investigated the perceptions of students’ first-time use of a LMS. The students were enrolled in Anadolu University’s (Turkey) open university system as distance students. The methodology employed in the research was described as “interpretative qualitative” case study design. The use of an interpretive paradigm to form the study was discussed, but no mention of the incorporation of this framework within a case study approach was mentioned. Data were largely collected via five focus groups in five cities.	Yes
Asamoah (2019)	The study looked at the type and quality of learner support services provided to postgraduate students engaged in hybrid learning using LMSs. The use of a case study design was clearly stated in the abstract and elucidated in the methodology section. The researcher explains that a case study methodology was selected because it allows the analysis of the conditions within which a phenomenon is situated, adding meaning to the characteristics of the studied phenomenon.	Yes
Beckmann and Mahanty (2016)	The study was a 5-year endeavor to examine the evolution of an online role play that postgraduate students participate in to explore the issues of land rights negotiations. The role of LMSs in this study was critically examined in the context of a transition from face-to-face to an online environment. The methodology employed was described as design-based research.	No
Bosco and Rodriguez-Gómez (2011)	The research examined an online course in geography with the aim to understand how online teaching contributed to innovative practices within this course. The paper discusses the case of the geography course from a broader research design that was multi-case in nature and included two other subject cases (veterinary studies and mathematics). The paper justifies the choice of multiple case exploratory study approach for the three distinct subject areas, and case study methodology was the overarching framework for other qualitative methods used in the study. The geography course was considered to have the most relevance to online learning due to its existing integration with LMS technologies. Within the case study framework, exploratory interviews and documents and artifacts from the LMS were the primary data sources.	Yes
Cantabella et al. (2019)	This paper analyzed the use of LMS analytics gleaned from large LMS databases, as a means to describe student behavior within the LMS. The study focused on tables containing information about users, user sessions, and user events. Quantitative techniques were used to analyze data to determine student LMS usage patterns and characteristics.	No
Choi et al. (2018)	This research analyzed the use of clicker data (information obtained from interactive applications accessible via mobile devices) to predict students at risk. This data was compared with traditional LMS analytics and other strategies to determine overall effectiveness. A quantitative approach to the analysis of the data were taken by comparing student-response data with exam results. No discussion of case study methodology was undertaken.	No
Chowdhurya et al. (2015)	This paper presented a methodology to explore the question of whether media choice has an influence on the learning efficacy of students in an education program. It does not present any findings as it was not a completed study. The proposed study’s aims are to examine how students will analyze case studies required in their 1st year of study, and whether the media they choose has any impact on effectiveness.	No
Company et al. (2017)	This study analyzed the use of Web-based rubrics that can be adapted or expanded for assessments of CAD assignments as required. The rubrics are created outside of the existing LMS. The new rubrics were tested using two experiments: one where students were asked to complete an assignment using traditional spreadsheet rubrics, and another where students used the new adaptable rubrics.	No
Derntl (2010)	The study was presented as a case study of student use of blogs in a reflective journal for a course in computer science. Descriptive statistics were used to analyze the blog entries (into an LMS) along with surveys which were administered after the blogging activity.	No
Feuer (2011)	This Canadian study examined the impact of employing social media elements in a course on social science writing and information literacy. Participants were divided into two groups: a control group, where students were taught without social media integration, and a group where students were taught the same course structured to facilitate social media interaction.	Yes
Hasan (2019)	This study has the broad aim of investigating the efficacy of LMSs at a Saudi university from the viewpoints and experiences of the institution’s LMS users. It would appear that the researcher of this study has used the term “case study” as an adjective to describe the institution that is the focus of her research, rather than relating the methodology to the research approach. In fact, the methodology section makes no attempt to justify a rationale for the methods used in the study and does not even mention the term “case study.”	No
Hernández-García et al. (2016)	The study analyses the use of Gephi, a social network learning analytics tool, in an introductory business administration course at the university level. The study is described as a case study, although no mention is made of this methodology. The data for the study were extracted from the course LMS and restructured for analysis in Gephi. The data analysis output from Gephi was presented as a combination of statistical data and visual graphs and charts. In essence, the study was an overview of the capabilities of the software.	No
Langley et al. (2008)	The study investigated the behavior of students when submitting assignments in a LMS, with regard to the viewing of other student’s submissions, and when this occurred. The approach taken was to survey students about their behavior and reconcile this data with access records extracted from the LMS. The study was described as an explorative case study in the abstract and identified as a case study in the conclusion.	Yes
Lata and Luhach (2014)	This paper examined whether collaborative learning delivered through a Moodle-based LMS assisted students of a speech writing module that was part of a public speaking course. Students were separated into three groups, and their performance was tracked over 6 months using data extracted from the LMS and researcher observations.	No
Liu (2009)	This study investigated the use of a mobile language learning environment called HELLO to enhance the English learning capability of students. The system was deployed on PDAs. An experimental approach to the study was taken where one group of students were taught English with traditional printed materials, and the other group used the HELLO system.	No
Martin and Noakes (2012)	The paper is a study of the deployment of a course called: “Education in a global society” via a LMS to an international cohort of students. The paper provided a rich array of examples of student participation and feedback on course effectiveness. There did not appear to be a rigid methodology to data extraction and analysis: rather, information was analyzed as it became available throughout the course. There were two iterations of the course that represented separate phenomena in this study.	No
McGuinness and Fulton (2019)	An investigation into how students engage with e-tutorials and e-learning, in general, was the central focus of this study. The researchers clearly explained their research design with detailed explanations of data collection and analysis techniques which were principally based on the use of survey instruments. They included a concise justification for selecting a case study approach: the study was based on student experiences of online learning in a real-world situation.	Yes
Özdağoğlu et al. (2019)	Future directions for LMS analytical data use was the main aim of this study. The methodological framework of this study included a description of the data sources from a Moodle platform and how these were extracted. There was neither an attempt to situate the research approach within case study design parameters, nor any mention of the relevance of case study characteristics to the study. The paper also lacked a discussion of alternate methodologies that could have been considered, and why the established approach was eventually chosen.	No
Paľová et al. (2018)	This study investigated how LMSs were used by a technology company collaborating with a Slovakian university, with a view to establishing some principles for facilitating the involvement of students in participating in practical, research-based problem solving for corporations. The study purports to use a case-study design, and the authors highlight their chosen methodology as adopting the characteristics of a case study. However, no effort is made to explain how the case study design fits into their chosen methodology or why this was the best approach to take.	No
Schwartz-Bechet (2012)	The study sought to determine if a Masters of Arts design course promoted personalized learning through a LMS platform. The research design involved both qualitative and quantitative analysis from multiple data sources, including surveys, course content review and grades. Triangulation of these data sources was deemed an important feature of the methodology.	No
Segrelles et al. (2017)	This study investigated the use of virtualized environments on the cloud to develop group skills via project-based learning (PBL). The methodology adopted was to compare 28 groups of students who used virtualized computer environments to develop project-based learning activities, with 30 groups who used the existing LMS, Sakai.	No
Sheeba and Krishnan (2019)	This research project attempted to establish a model that could be used to construct learner profiles based on static and dynamic information of LMS users stored in web log files. The term “case study” only appeared in the abstract, and no attempt was made in the study’s methodology to relate the research approach to the characteristics of case studies.	No
Soon (2011)	This paper explored the challenges and barriers in e-learning and m-learning to group assignment collaboration in a course on information management in organizations. The study explained the choice of case study methodology adequately. The choice of case study methodology is justified because it is a real-world situation where the boundaries between phenomenon and context are unclear.	Yes
Tay and Low (2017)	The study investigated the key factors enabling the transformation of printed resources to e-learning resources. The methodology outlined is a qualitative exploratory case study approach. The methodology section specified the type of case study as an in-depth single case study that followed standard approaches to case definition, case selection, data collection and case conclusion.	Yes
Teräs and Kartoğlu (2018)	This study described the outcomes of a pilot study to implement an online professional development program in vaccine management, called e-VVM. The methodology section described the case study as a classic case study where the complex phenomenon is the learning experience within the e-VVM course.	Yes
Thurab-Nkhosi (2018)	This paper investigated the education policy implications for university managers and other stakeholders involved in the implementation of transitional arrangements for courses to be delivered in a blended-learning environment. The methodology section clearly states: “This is a qualitative case study, focusing on the stage of blended learning implementation and the perceptions of deans and administrative officers at a specific higher education institution.” While the scope of the study has been clearly constrained to a single institution, no attempt was made to discuss the merits of adopting case study methodology in this instance.	No
Ye (2014)	The study reported the findings of an online education course designed to give educators an experience of online learning from a student perspective. Data were collected via Blackboard’s analytics, and surveys were administered to students with open and closed-ended questions. SPSS was used to analyze quantitative data while emerging themes were explored with ATLAS.ti qualitative data analysis software.	No

A distinction needs to be made here between a study that encapsulates the output of a research effort within boundaries that are meant to define a phenomenon under investigation, and a study that actually uses case study methodology. Alpi and Evans (2019) propose that research articles that include detailed clarifications of the data collection methods, clarification of the role of the investigator, and contextualized analysis of the results are more likely to have the characteristics of case studies whereas the absence of these qualities relegates the publication to the status of a report. The papers that did not include a description or discussion of case study design were further examined to determine whether they exhibited the characteristics of a case study and accordingly could be classified as such.

In order to do this, every study that included a description or discussion of case study research design was labeled “Yes” and those that did not were labeled “No,” as shown in the third column of Table 1.

The studies presented in Table 1 that did not contain a description of how case studies were applied (classification: “NO”) were further examined to see if they could be regarded as case studies in their own rights. To be considered as a candidate case study, the paper had to contain an adequate methodology section with clear explanations of how the study was conducted. Papers that met this condition were then further scrutinized against the following three case study inclusion criteria: the unit of study was adequately identified; the boundaries of the case(s) were identified and explained, and triangulation techniques were applied to each study. This led to the segmentation of the 28 studies into three categories, as shown in Table 2.

Table 2.

Case Study Categories.

Category	Explanation
Case study methodology explained	These studies included an explicit explanation of why the case study design was chosen and included a discussion on the merits and appropriateness of case study design. They refer to items in column three of Table 1 with a “YES” designation.
Case study not explained but meets criteria	These studies were deemed to have met the three criteria for the classification of a case study. However, there was no discussion of the merits or applicability of case study methodology.
Non-Case Studies	These studies did not meet the three criteria for a case study and were classified as non-case studies.

Figure 1 displays all 28 studies as segmented according to the classification in Table 2.

Figure 1.

LMS empirical studies and their case study classification.

Non-Case Studies

As indicated in Figure 1, there were 11 papers in this review that did not meet the criteria for inclusion as case studies. One of the most important reasons for rejecting a study was the lack of a clearly laid out methodology section as was the case with the studies by Cantabella et al. (2019), Lata and Luhach (2014), Segrelles et al. (2017), Paľová et al. (2018), and Company et al. (2017). The study on online role play by postgraduate students was considered not to meet the criteria of a case study because it was not a study of actual phenomena but rather a description of a controlled event (Beckmann & Mahanty, 2016). Chowdhurya et al. (2015)’s proposed study utilized case studies as a learning tool but did not identify case design as their preferred methodology. Liu (2009)’s paper described their study as “a series of controlled experiments” (p. 519) with no reference to case study design. Özdağoğlu et al. (2019), Hasan (2019) and Sheeba and Krishnan (2019)’s papers made an attempt to describe data collection and analysis procedures but failed to situate these in any chosen methodology.

Case Study Methodology Defined or Explained

Of the 28 empirical studies in this review, only nine explicitly identified case studies as an approach that drove their research design. It is interesting to note how each study incorporated case study characteristics into their research designs. One dimension that was considered was the number of cases to be included in the study. Tay and Low (2017) specified that their research design was based on a single case structure that followed standard procedures for conducting such a study. Given the intent of the study, i.e. an examination of the transformation of written learning materials into electronic form, this design is appropriate for this single phenomenon. While not using the word single to describe the research design, Teräs and Kartoğlu (2018)’s examination of authentic learning in vaccine management is also depicted as a single case study by reference to the research design as being based on a “complex phenomenon” (p. 21). Reference to the bounded phenomenon as a characteristic of case studies that lead to the selection of this methodology was also made in the study of learner support services for postgraduate students (Asamoah, 2019). This is also supported by McGuinness and Fulton (2019) who additionally highlight the focus of case study methodology on contemporary phenomena in realistic contexts as a useful attribute to integrate into research designs. By contrast, multiple case study design was used to describe the research design of the study into geography, veterinary science and geography courses offered online (Bosco & Rodriguez-Gómez, 2011). Clearly, the three distinctive subject areas are good candidates for a multiple case design.

Exploratory case study design was another descriptor used to specify research design. The methodology section of Soon (2011)’s study on E-Learning and M-Learning justified this approach by claiming that the phenomenon to be observed had an unfamiliar context but readily definable boundaries. Feuer (2011, p. 14) described their research design as an “exploratory, descriptive, quantitative case study.” This description appropriately identified the studied phenomenon as an exploratory case utilizing quantitative methods in a descriptive context. Langley et al. (2008) also make use of the term “exploratory case study” to describe the research design of their study into 1st-year student behavior in an open assignment submission regime. However, the term is only used in the abstract and is not expanded upon in the methodology section of the study. The final description of case study design identified in this paper is an “interpretive qualitative case study design” (Altunoglu, 2017, p. 99). This description aligns with the purpose of their study: to investigate student perceptions of first-time use of a LMS.

Case Studies Meeting Classification Criteria That Did Not Include Discussions on Case Study Design

The remaining eight papers did not discuss case study design in the methodology sections, but they exhibited characteristics consistent with the three inclusion criteria. Table 3 lists these studies and how they meet the inclusion criteria:

Table 3.

Studies That Were Classified as Case Studies Without Discussing Case Study Design.

Study	Unit of Study	Case Boundaries and Constraints	Triangulation
Alenezi (2017)	Northern Borders University full-time students.	Constrained to an examination of the student preferences for e-Learning and M-Learning.	Survey method used both paper-based and electronic approaches.
Choi et al. (2018)	Students enrolled in the 2016 and 2017 bachelor of business administration course.	The study was limited to 1st-year students. Suggestions for future work included an evidence-based research approach to validate findings in this case.	Clicker data were supplemented by other data sources such as questionnaires.
Derntl (2010)	The unit of study was a module on software architecture used by a university based in Vienna.	The study was constrained to a single undergraduate course lab course in computer science. The conclusion indicates that the study cannot be generalized beyond the confines of the case.	Blogging statistics were supplemented with survey data.
Hernández-García et al. (2016)	The unit of study was identified as an introduction to financial information with a specific time frame (Sept 2013 to Jan 2014).	The constraints of the study were emphasized in the conclusion as a snapshot of data from Gephi and not a complete overview of the course.	Multiple analytical tools were employed to illustrate how data could be presented.
Martin and Noakes (2012)	The unit of study was adequately defined as a global education course delivered via a LMS and could be considered a multiple case study because there were two cohorts.	The tools used to facilitate cooperative learning were limited to Blackboard and Wimba. The study was also declared as an 8-week descriptive case study of one particular course.	Multiple data sources were used in the study.
Schwartz-Bechet (2012)	The study consisted of 70 students in a 2011 offering of a Master of Art program.	As the study was conducted in a single university and only one program, the findings could not be generalized to other institutions and academic content areas.	Triangulation was mentioned in the methodology as a technique applied to data collection.
Thurab-Nkhosi (2018)	The unit of study was a Caribbean university LMS and the administrators responsible for its operation.	The study was constrained to a single university, and participants were restricted to Dean-level administrators.	Moodle analytical data and interview data were analyzed in the study.
Ye (2014)	The unit of study was defined as the 24 teacher candidates enrolled in an online course in teacher education.	Only one teacher education course was examined by this study, limiting the capacity to generalize findings beyond this constraint.	Multiple methods were used including surveys and LMS data.

Discussion

Clear descriptions of the type of case study employed in research projects add credibility to methodological descriptions. Of the nine papers that discussed the use of case study design, the terms used to describe the case study approach include: “interpretive, qualitative”; “multiple”; “explorative” (twice); “exploratory, descriptive, quantitative”; “qualitative case study approach”; and “classic.” Descriptive words add credibility to the choice of case study design and enhance the methodology section of the study. This is particularly so when the description is aided by a discussion of the merits of case study design, as was the case with Teräs and Kartoğlu (2018), Tay and Low (2017) and Soon (2011). The methodology sections of the 11 papers that did not explicitly address the use of case study design, but exhibited the characteristics used in this review to classify a study as a case study design, could have been enhanced by a discussion of the merits and limitations of case study application. In fact, failing to explain the choice and application of a research design as distinct from research methods can lead to poor evaluation of designs (De Vaus, 2001). Creating a methodology section that considers the research design separately from the data collection and analysis methods employed would also assist in defining the context and parameters of a study.

The eight papers that did not meet the criteria of a case study as set out in this review appear to have applied the label of “case study” to the underlying research without due consideration for the characteristics and suitability of case study design. The use of the term "case study" as a label of convenience rather than a rigorous research design can be misleading to readers who have an appreciation for the robustness and characteristics of the approach. As Swanborn (2010) puts it: “Writing and speaking about case studies, one experiences some frustration: the label ‘case studies’ seems to be used for many purposes” (p. 10).

A central cause of the misuse of the term “case study” may lie with the lack of consensus on the definition of a case study. Thomas (2011) outlines some of the discipline-specific perspectives of case study design. These include (1) sociology, education and psychology’s interpretivist view, (2) a regard for the identification of variables within a case by business and political disciplines, and (3) the medical and law fraternity’s use of case study design to highlight both new and typical phenomena. However, the lack of consensus among researchers on a unified definition of the case study approach is not an excuse for failing to clarify its application to a particular study. Given this lack of clarity, a more robust definition of case study characteristics in the context of LMS research could be beneficial to future studies. As a first attempt, this paper proposes a definition of a case study in the context of LMS research which also has applicability to information systems in general:

A LMS case study is an investigation into single or multiple instances of observable complex phenomena that can be constrained by clearly identifiable boundaries. Case study approaches to LMS research involve a careful dissection of all bounded phenomena to identify discrete units of analysis. Examples of units of analysis could include LMS system components, LMS subject-matter content, the LMS user community, and host institutional policies and procedures. LMS case study research does not dictate the use of specific data collection or analysis methods but instead focusses on defining a workable framework for the conduct of a research project.

At the very least, a paper’s methodology section should attempt to situate the data collection and analytical methods of a declared case study approach within a discussion of case study theory.

Conclusion

The term “case study” is often used as a label of convenience in LMS research, rather than a descriptor of a rigorous approach to research design. Designing studies within a case study framework is a well-established approach to conducting research in diverse areas such as medicine, law, and engineering. Nevertheless, in the specific field of LMSs, this term is often an inappropriate label for the methodologies employed in empirical studies. This review found that 11 of the 28 examined studies claimed to adopt a case study approach without justifying how this framed the methodology of the study. Moreover, from further examination, these studies did not appear to exhibit the characteristics usually attributed to case study design which include as a minimum the recognition that a proposed study is a contemporary phenomenon existing within a real-world context (Yin, 2018). Of the remaining 17 studies, the methodology sections of eight papers demonstrated that there was sufficient justification for accepting that they could be considered as case studies. However, there was no explicit discussion of why this approach was chosen against competing methodological options. The remaining nine papers provided clear justifications for the choice of a case study framework, evidenced by appropriate references to established literature.

Moving forward, this paper proposes that a clear description of the characteristics of a case study as applied to empirical research into LMSs would be of assistance to future research projects in this field. A possible definition of what might constitute a LMS case study is clearly laid out in the discussion section of this paper. However, given this study’s limitations to LMSs and English language publications, much more needs to be done in this area—particularly if this paper’s proposed definition of LMS-related case study research is to have wider applicability to other information systems-related projects. Hopefully, this study will inspire other researchers to reflect on what constitutes a convincing case study research project.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

ORCID iD

Darren Turnbull

Jo Luck

References

Alenezi

(2017). Checking on preferential choices of e-learning & mlearning: A case study of northern border university, Saudi Arabia. International Journal of Emerging Technologies in Learning, 12(5), 98–116. https://doi.org/10.3991/ijet.v12i05.6790

Alpi

K. M.

Evans

J. J.

(2019). Distinguishing case study as a research method from case reports as a publication type. Journal of the Medical Library Association, 107(1), 1. https://doi.org/https://doi.org/10.5195/JMLA.2019.615

Altunoglu

(2017). Initial perceptions of open higher education students with learner management systems. Turkish Online Journal of Distance Education, 18(3), 96–104. https://doi.org/https://doi.org/10.17718/tojde.328939

Asamoah

M. K.

(2019). Learner support services for postgraduate students: A qualitative approach. E-Learning and Digital Media, 16(5), 367–392. https://doi.org/10.1177/2042753019860613

Beckmann

E. A.

Mahanty

(2016). The evolution and evaluation of an online role play through design-based research. Australasian Journal of Educational Technology, 32(5), 35–47. https://doi.org/10.14742/ajet.1957

Best

Manktelow

Taylor

(2014). Online communication, social media and adolescent wellbeing: A systematic narrative review. Children and Youth Services Review, 41, 27–36. https://doi.org/10.1016/j.childyouth.2014.03.001

Bosco

Rodriguez-Gómez

(2011). Virtual university teaching: Contributions to innovation in higher education. The case of online geography at the universitat autonoma de Barcelona. Innovations in Education & Teaching International, 48(1), 13–23. https://doi.org/10.1080/14703297.2010.543772

Boyle

E. A.

Connolly

T. M.

Hainey

Boyle

J. M.

(2012). Engagement in digital entertainment games: A systematic review. Computers in Human Behavior, 28(3), 771–780. https://doi.org/10.1016/j.chb.2011.11.020

Cantabella

Martínez-España

Ayuso

Yáñez

J. A.

Muñoz

(2019). Analysis of student behavior in learning management systems through a big data framework. Future Generation Computer Systems, 90, 262–272. https://doi.org/10.1016/j.future.2018.08.003

10.

Choi

S. P. M.

Lam

S. S.

K. C.

Wong

B. T. M.

(2018). Learning analytics at low cost: At-risk student prediction with clicker data and systematic proactive interventions. Educational Technology and Society, 21(2), 273–290.

11.

Chowdhurya

A. A.

Jayasuriya

Eschenbach

E. A.

(2015). Work in progress: Model design to measure the efficacy of students learning preferences—Does media matter? Lecture Notes in Electrical Engineering, 312, 407–412. https://doi.org/10.1007/978-3-319-06764-3_50

12.

Chugh

Ruhi

(2018). Social media in higher education: A literature review of Facebook. Education and Information Technologies, 23(2), 605–616. https://doi.org/10.1007/s10639-017-9621-2

13.

Company

Contero

Otey

Camba

J. D.

Agost

M.-J.

Perez-Lopez

(2017). Web-based system for adaptable rubrics: Case study on CAD assessment. Educational Technology & Society, 20(3), 24–41.

14.

Derntl

(2010). Revealing student blogging activities using RSS feeds and LMS logs. International Journal of Distance Education Technologies, 8(3), 16–30. https://doi.org/https://doi.org/10.4018/jdet.2010070102

15.

De Vaus

D. A.

(Ed.). (2001). What is research design? In Research design in social research (pp. 1–16). Sage.

16.

Eisenhardt

K. M.

(1989). Building theories from case study research (Special forum on theory building). Academy of Management Review, 14(4), 532–550. https://doi.org/10.5465/AMR.1989.4308385

17.

Feuer

(2011). Is social software really a “killer app” in the education of net generation students? Findings from a case study. Library Hi Tech News, 28(7), 14–17. https://doi.org/10.1108/07419051111184043

18.

Fleischer

(2012). What is our current understanding of one-to-one computer projects: A systematic narrative research review. Educational Research Review, 7(2), 107–122. https://doi.org/10.1016/j.edurev.2011.11.004

19.

Flyvbjerg

(2006). Five misunderstandings about case-study research. Qualitative Inquiry, 12(2), 219–245. https://doi.org/10.1177/1077800405284363

20.

Gaikwad

(2017). Including rigor and artistry in case study as a strategic qualitative methodology. The Qualitative Report, 22(13), 3431–3446.

21.

Gerring

(2004). What is a case study and what is it good for? American Political Science Review, 98(2), 341–354. https://doi.org/10.1017/S0003055404001182

22.

Hasan

(2019). The usefulness and usability of Moodle LMS as employed by Zarqa university in Jordan. Journal of Information Systems & Technology Management, 16. https://doi.org/10.4301/S1807-1775201916009

23.

Heale

Forbes

(2013). Understanding triangulation in research. Evidence-Based Nursing, 16(4), 98–98. https://doi.org/10.1136/eb-2013-101494

24.

Hernández-García

González-González

Jiménez Zarco

A. I.

Chaparro-Peláez

(2016). Visualizations of online course interactions for social network learning analytics. International Journal of Emerging Technologies in Learning, 11(7), 6–15. https://doi.org/10.3991/ijet.v11i07.5889

25.

Huang

Liu

Tlili

Yang

Wang

(2020). Handbook on facilitating flexible learning during educational disruption: The Chinese experience in maintaining undisrupted learning in COVID-19 outbreak. Smart Learning Institute of Beijing Normal University.

26.

Kyburz-Graber

(2004). Does case-study methodology lack rigour? The need for quality criteria for sound case-study research, as illustrated by a recent case in secondary and higher education. Environmental Education Research, 10(1), 53–65. https://doi.org/https://doi.org/10.1080/1350462032000173706

27.

Langley

Ronen

Shachar

S. b.

(2008). Open online assignment submission: First year students’ behavior and views. Issues in Informing Science & Information Technology, 5, 297–310. https://doi.org/https://doi.org/10.28945/1011

28.

Lata

Luhach

(2014). An exploratory investigation of online forum mediated collaborative learning of speech writing: A case study. Journal on English Language Teaching, 4(1), 18–28. https://doi.org/https://doi.org/10.26634/jelt.4.1.2639

29.

Liu

T. Y.

(2009). A context-aware ubiquitous learning environment for language listening and speaking. Journal of Computer Assisted Learning, 25(6), 515–527. https://doi.org/10.1111/j.1365-2729.2009.00329.x

30.

Martin

Noakes

(2012). Fostering a web 2.0 ethos in a traditional e-learning environment. Electronic Journal of e-Learning, 10(3), 284–292.

31.

McCutcheon

Lohan

Traynor

Martin

(2015). A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. Journal of Advanced Nursing, 71(2), 255–270. https://doi.org/10.1111/jan.12509

32.

McGuinness

Fulton

(2019). Digital literacy in higher education: A case study of student engagement with e-tutorials using blended learning. Journal of Information Technology Education: Innovations in Practice, 18, 1–28. https://doi.org/https://doi.org/10.28945/4190

33.

Meyer

C. B.

(2001). A case in case study methodology. Field Methods, 13(4), 329–352. https://doi.org/https://doi.org/10.1177/1525822X0101300402

34.

Moorhouse

B. L.

(2020). Adaptations to a face-to-face initial teacher education course ‘forced’ online due to the COVID-19 pandemic. Journal of Education for Teaching, 46(4), 609–611. https://doi.org/10.1080/02607476.2020.1755205

35.

Özdağoğlu

Öztaş

G. Z.

Çağliyangil

(2019). An application framework for mining online learning processes through event-logs. Business Process Management Journal, 25(5), 860–886. https://doi.org/10.1108/bpmj-10-2017-0279

36.

Paľová

Czaja

Vejačka

(2018). Innovative approach to education improvement via enterprise-education collaboration. Quality Innovation Prosperity, 22(3), 68–82. https://doi.org/10.12776/qip.v22i3.1171

37.

Perry

(1998). Processes of a case study methodology for postgraduate research in marketing. European Journal of Marketing, 32(9/10), 785–802. https://doi.org/10.1108/03090569810232237

38.

Raza

S. A.

Qazi

Khan

K. A.

Salam

(2020). Social isolation and acceptance of the learning management system (LMS) in the time of COVID-19 pandemic: An expansion of the UTAUT model. Journal of Educational Computing Research, 1-26. https://doi.org/10.1177/0735633120960421

39.

Sarmiento

I. G.

Olson

Yeo

Chen

Y. A.

Toma

C. L.

Brown

B. B.

Bellmore

Mares

M.-L.

(2018). How does social media use relate to adolescents’ internalising symptoms? Conclusions from a systematic narrative review. Adolescent Research Review, 5, 381–404. https://doi.org/https://doi.org/10.1007/s40894-018-0095-2

40.

Schwartz-Bechet

(2012). Can course design in an online MAT program promote personalised learning through E-teaching and E-learning practices? International Journal of Advanced Corporate Learning, 5(4), 38–41. https://doi.org/10.3991/ijac.v5i4.2200

41.

Segrelles

J. D.

Martinez

Castilla

Moltó

(2017). Virtualized computational environments on the cloud to foster group skills through PBL: A case study in architecture. Computers and Education, 108, 131–144. https://doi.org/10.1016/j.compedu.2017.02.001

42.

Sheeba

Krishnan

(2019). Fuzzy integrated ontology model of dynamic learner profiling. International Journal of Innovative Technology and Exploring Engineering, 8(12), 1549–1558. https://doi.org/10.35940/ijitee.L3119.1081219

43.

Simons

(2009). Case study research in practice. Sage.

44.

Sinclair

P. M.

Kable

Levett-Jones

Booth

(2016). The effectiveness of internet-based e-learning on clinician behaviour and patient outcomes: A systematic review. International Journal of Nursing Studies, 57, 70–81. https://doi.org/https://doi.org/10.1016/j.ijnurstu.2016.01.011

45.

Soon

(2011). E-Learning and M-learning: Challenges and barriers in distance education group assignment collaboration. International Journal of Mobile and Blended Learning, 3(3), 43–58. https://doi.org/10.4018/jmbl.2011070104

46.

Stake

R. E.

(1995). The art of case study research. Sage.

47.

Swanborn

P. G.

(2010). Case study research: What, why and how? (pp. 1–23). Sage.

48.

Tay

H. L.

Low

S. W. K.

(2017). Digitalisation of learning resources in a HEI—A lean management perspective. International Journal of Productivity and Performance Management, 66(5), 680–694. https://doi.org/10.1108/IJPPM-09-2016-0193

49.

Tellis

W. M.

(1997a). Application of a case study methodology. The Qualitative Report, 3(3), 1–19.

50.

Tellis

W. M.

(1997b). Introduction to case study. The Qualitative Report, 3(2), 1–14. https://doi.org/https://doi.org/10.1108/10650749710187617

51.

Teräs

Kartoğlu

Ü.

(2018). Authentic learning with technology for professional development in vaccine management. Australasian Journal of Educational Technology, 34(3), 15–29. https://doi.org/https://doi.org/10.14742/ajet.2815

52.

Thomas

(2011). A typology for the case study in social science following a review of definition, discourse, and structure. Qualitative Inquiry, 17(6), 511–521. https://doi.org/10.1177/1077800411409884

53.

Thurab-Nkhosi

(2018). Implementing a blended/online learning policy on a face-to-face campus: Perspectives of administrators and implications for change. Journal of Learning for Development, 5(2), 133–147.

54.

Turnbull

Chugh

Luck

(2019). Learning management systems: An overview. In Tatnall

(Ed.), Encyclopedia of education and information technologies. Springer Nature. https://doi:10.1007/978-3-319-600130_248-1

55.

(2014). Universal design for learning in an online teacher education course: Enhancing learners’ confidence to teach online. Journal of Online Learning & Teaching, 10(2), 283–298.

56.

Yin

(1994). Conducting case studies: Preparing for data collection. Sage.

57.

Yin

(2018). Case study research and applications: Design and methods (6th ed.). Sage.

58.

Zainal

(2007). Case study as a research method. Jurnal Kemanusiaan, 9(1), 1–6.