Abstract
Using various types of research methods yields breadth and depth in knowledge creation. Replying to calls for more research employing diverse research methods, this study examines research method trends, rigor, and emerging research methods within the field of human resource development (HRD). We created a research method coding scheme to capture the rigor of empirical research among empirical articles published in three specifically selected HRD journals from 2016–2023. Out of 488 selected studies, quantitative (n = 269) and qualitative (n = 185) methodologies were dominantly used in empirical studies with only 7.0% being mixed methods research (n = 34). The study findings provide implications and reflections on (1) the current status of published scholarly research; and (2) the research practices in the field. To move the field of HRD forward as a mature, effective, and evolving discipline, it is imperative to incorporate multiple approaches, including interdisciplinary investigation of emerging research methods, and development of clear guidelines for the rigor of research methods.
Keywords
One of the critical mechanisms of the development of scientific knowledge is through innovative ideas and epistemological endeavors. Paavola and Hakkarainen (2005) emphasized a need for an emergent epistemological approach to knowledge creation, which helps to understand the nature of knowledge in the current society. Using various types of research methods following rigorous methodological standards yields different yet complementary types of knowledge that account for various social practices with breadth and depth (Han et al., 2022). Moreover, research rigor in different types of research methods is pivotal to yield quality of knowledge production and scholarship in the field of human resource development (HRD) (Cho et al., 2022; Harrison et al., 2020; Legate et al., 2023; Reio, 2010).
In a series of editorials and special issues within the Academy of Human Resource Development sponsored HRD peer reviewed journals and related well respected HRD journals, there has been a continuous call for more research applying diverse and innovative research methods such as theory-building research method articles (Reio, 2010), innovative research (Cho et al., 2022; Hatcher & Guerdat, 2008), and emergent research methods (Reio, 2009). For over 10 years, major HRD studies have called for more research utilizing innovative and emergent research methods (Reio, 2009, 2010). Since then, some studies have explored the trends of research in the field such as definitions of HRD (e.g., Hamlin & Stewart, 2011; Han et al., 2017), HRD research topics and themes (e.g., Han et al., 2017; Yoo et al., 2018; Yoon & Chae, 2022), key trends and challenges in HRD (e.g., McGuire & Cseh, 2006), and citation networks on who is citing whom (e.g., Jo et al., 2009).
Although some studies examined HRD research topics and themes, limited studies have comprehensively attempted to explore trends and rigor of research methods employed in the HRD field. It is not difficult to find such studies in other fields such as educational technology (e.g., Reeves & Oh, 2017; Zhu et al., 2018), management (e.g., Wilson et al., 2014), health professions education (e.g., Han et al., 2022), and information management (e.g., Dwivedi et al., 2011). These studies provide critical insight on how research methods have shaped the collective set of knowledge in their fields through longitudinal or cross-sectional studies.
Ever since Onwuegbuzie and Corrigan’s (2014) editorial urging for more mixed methods research, there has not been a study holistically analyzing the research methods employed in HRD studies. This study was designed to fill the void and examined all eight years of empirical research published in three major journals in HRD: two journals sponsored by Academy of Human Resource Development, Human Resource Development Quarterly (HRDQ) and Human Resource Development International (HRDI), and one leading journal in the field of HRD, European Journal of Training and Development (EJTD). As there has been a limited investigation of overall research method trends in the field of HRD, this study aimed to understand the degree of diversity and rigor of research methods employed in the field of HRD. To achieve the purpose of the study, the authors conducted a review of literature on research methods and created a coding scheme used as a checklist to analyze the research method rigor in the selected articles. The result of the analysis is presented in this study. We expect this study on the landscape of research methods across three major journals in HRD to provide insights to researchers and practitioners regarding the need for more diverse perspectives in addressing important issues in the field of HRD. Furthermore, we expect this coding scheme to be used as a guideline, rubric, or checklist for scholars and practitioners in the field when conducting empirical research. The following research questions guided this inquiry. RQ1. What are the research methods employed in empirical studies published in HRD journals from 2016 to 2023? RQ1-1. How diverse are the research methods employed in the select articles? RQ2. What is the degree of rigor in the research design of the select articles from 2016 to 2020? RQ3. What are the types of emerging research in the field of human resource development from 2016 to 2023?
In the sections that follow, we presented a review of literature on the topic of research rigor across the major paradigms of research.
Literature Review
Epistemological exploration has long been a critical topic in social science research (Alkin, 2004). Adopting the reductionist logic of inquiry from natural science research, earlier social science researchers viewed an experimental research design as a validated and trusted methodology to discover truth and knowledge. Noting the complex and different nature of the social sciences, researchers adopted different paradigms and started the methodological debate, which resulted in the inclusion of qualitative inquiry from constructivism or postmodernism and mixed methods from the pluralism of paradigms. The diversity in social sciences research resulted in going beyond the philosophical underpinnings of epistemology involving practical considerations specific to each research method. As different epistemological approaches pursue diverse aspects of truth, it is critical to facilitate and seek various methodologies to build collective and comprehensive systems of knowledge in the field (Han et al., 2022; Nimon, 2016).
The rigor of research involves many different research elements, such as a research topic, rationale, conceptual framework, and writing quality, however, the scope of this paper is focused primarily on research method rigor. Many researchers have emphasized the importance of reporting detailed methodological processes and taking study context into consideration when designing an empirical study (American Educational Research Association [AERA], 2006). According to the American Educational Research Association’s guidelines in reporting on empirical social science research, empirical research should be “warranted” and “transparent” with sufficient information on the research methodological procedure and processes enough for other scholars to understand and replicate the research (AERA, 2006). Each research paradigm has different methodological rigor categories, which we will review in-depth in the following sections.
Rigor of Quantitative Research
As an applied discipline, evidence-based practice is one of the critical components to construct identity in the field of HRD. Validity of evidence is directly related to quality or rigor of research to justify the causality of variables in quantitative research (Gubbins & Rousseau, 2015; Nimon & Astakhova, 2015). Building on Gubbins and Rousseau’s (2015) general hierarchy of evidence, which is a hierarchy of research design, Nimon and Astakhova (2015) highlighted the top four research designs that may enhance the rigor of quantitative HRD research. Such designs supported meta-analytic reviews, a retrospective approach to experimental, quasi-experimental, pre-experimental designs, a continuum of mediated designs, and choice of mediators.
From a quantitative methodological standpoint, establishing causality is the ultimate goal and the most fundamental criteria to assess rigor of quantitative research (Gubbins & Rousseau, 2015; Nimon & Astakhova, 2015). However, in the current society, organizational structures and processes are evolving rapidly in more complicated ways. Thus, exploring the sophisticated relationships among research variables yields meaningful results in HRD theory building and helps in the development of practical strategies with a more accurate prediction of the HRD phenomenon. Along with this consideration, the use of the most advanced methodological approaches rather than simple analytical procedures significantly contributes to enhance the rigor of HRD research (Song & Lim, 2015). For example, the use of mediation analysis using structural equation modeling is increasing in social sciences disciplines (Long, 2014), as it analyzes the individual and holistic relationships among the variables (Hair et al., 2010).
In terms of the rigor of quantitative research, other important points include reporting the methods, results, and interpreting the results. Maula and Stam (2019) provided detailed ideas regarding methodological issues to enhance transparency in reporting and appropriate interpretation of results in entrepreneurship research. Most of these ideas are also applicable and have particular importance to HRD research. First, previous research on HRD research trends (Kim, 2004) reported that HRD research heavily depended on the survey method, thus ensuring validity and reliability of measurement is the most essential foundation of the research. With consideration for this issue, common method bias is a critical threat to the rigor of quantitative research. Next, considering the increased diversity of HRD research topics reflecting rapid and dynamic changes in HRD field (Shirmohammadi et al., 2020; Yoon & Chae, 2022), detailed information about sources of data is important to understand the context of the research and then the advantages and limitations of the results. Third, applying appropriate analytical tools, which align with research questions and research context, has received growing attention among researchers (Legate et al., 2023). As statistical analysis methods and tools are rapidly evolving in quantitative research, many methodology scholars urge social sciences researchers to continuously learn new research methods and apply these to the examination of relevant HRD theories and strategies (Legate et al., 2023; Song & Lim, 2015).
Rigor of Qualitative Research
While advocating the importance of quality and credibility of a qualitative study, many researchers have argued that qualitative studies need different quality criteria from quantitative studies that represent different epistemological processes (Cornelissen, 2017; Creswell & Poth, 2018; Lynham & Lincoln, 2011; Seale, 1999). Although HRD is a quantitative research dominant field, qualitative research has increased in confidence and expertise in recent years (Anderson, 2017). In recent editorials and special issues, many studies have called for more in-depth qualitative research and innovative qualitative research methods in HRD (i.e., Anderson, 2017; Cho et al., 2022).
Although reliability and validity are common rigor standards of quantitative studies, it is notable that they are not adequate criteria to capture the quality of qualitative studies. Academics have also long debated how to achieve rigor in qualitative research among diverse contemporary qualitative research methodologies (Grodal et al., 2021; Levitt et al., 2017; Lincoln & Lynham, 2011; Storberg-Walker, 2003). Given a number of contradictory perspectives across multiple qualitative traditions in evaluating qualitative research, Leviett et al. (2017) proposed the concept of methodological integrity as an overarching criterion of qualitative research rigor. Methodological integrity consists of two composite processes, fidelity to the subject matter and utility in achieving research goals. The two processes resemble utility-relevance and rigor-relevance (empirical rigor and trustworthiness), which Lynham (2002) highlighted as criteria for evaluating theories.
Though it is in the context of theory building in HRD, Lincoln and Lynham (2011) proposed five new criteria to assess rigor of theory from an interpretive perspective, which can be extended to evaluate the rigor of qualitative inquiry: (1) compellingness (the ability to prompt action), (2) saturation (explanations and examples are exhaustively sampled), (3) prompt to action (providing understanding for practice), (4) fittingness (being situated in a local context), and (5) transferability and transportability (to carry knowledge from one context to inform other contexts, or the applicability to varying populations). Other scholars have paid more attention to particular aspects of qualitative research according to their methodological traditions and/or their fields of discipline. For example, Collins and Stockton (2018) highlighted the central role of theory in research rigor by suggesting an evaluative quadrant for judging the appropriate use of theory and method. Grodal et al. (2021) specified more detailed steps in categorizing data to enhance rigor in data analysis. Moravcsik (2013) insisted that active citation enables a researcher to dramatically enhance transparency, particularly in qualitative research.
Lastly, Maxwell (2004), and Bennett and McWhorter (2016) emphasized the value of qualitative study in handling causal mechanisms and processes from a realistic approach to causality. We introduced some, but not all, of the various viewpoints on the rigor of qualitative research. Debates among scholars from diverse methodological traditions may require researchers to harness the unique values of qualitative research in exploring explanations and knowledge. However, addressing this issue in greater detail goes beyond the purpose of this study. Although the relative focuses may be different according to methodological traditions, rigor indicators introduced previously seemed to share some common ground. Despite those debates, it was also not unusual that scholars across various disciplines, including HRD, suggested a series of criteria to elicit a shared understanding of rigor in quality research, which may have roots in the classical criteria conceptualized by Guba and Lincoln (1982) and encompass the debated indicators in general (Anderson, 2017; Cypress, 2017; Johnson et al., 2020; Lub, 2015; O’Brien et al., 2014).
Guba and Lincoln (1982) proposed trustworthiness as a quality criterion to ensure credibility and rigor of qualitative studies. Trustworthiness involves four subdomains: (1) Credibility (i.e., “Are the study findings credible?”), (2) transferability (i.e., “Can the study findings be applicable to other contexts? Does the study provide enough thick description such as research setting and context?”), (3) dependability (i.e. “Does the research process go stable considering all unpredictable changes?”), and (4) confirmability (i.e. “Are the research findings confirmable by other researchers?”).
Some methods, for instance, prolonged engagement, persistent observation, peer debriefing, triangulation (data sources, perspectives, or theories), adequate diverse materials, and member checks are utilized to improve credibility of a qualitative study (Guba & Lincoln, 1982). To ensure transferability, researchers should adopt theoretical and purposeful sampling and provide enough information about the study context (Anderson, 2017; Seal, 1999). Rocco (2003) asserts that when discussing the data collection process, stating “interviews were conducted” is not sufficient. Qualitative studies should report a particular method of data collection, details of the process (i.e., pilot test, expert review, peer review), data collection schedules, frequency, and interview types (i.e., focus groups, life story, survey). Triangulation may be utilized to establish dependability and confirmability of a qualitative study (Guba & Lincoln, 1982).
Additionally, auditing methodological steps and researchers’ practicing reflexivity were also discussed as possible ways to improve trustworthiness in qualitative research. Auditing methodological steps can benefit data analysis as qualitative data analysis is generally characterized as subjective, iterative, and non-standardized (Lester et al., 2020). In fact, there are numerous ways to analyze qualitative data and it becomes important for researchers to provide sufficient information about what and how data analysis was conducted in published studies. Guba and Lincoln (1982) noted that the criteria cannot guarantee the trustworthiness of research but can work as guidelines to help qualitative studies to develop as a credible scientific method of inquiry.
Rigor of Mixed Methods Research
Mixed methods research (MMR) is not just a simple combination of research methods. Thus, mixed methods researchers need to understand the logic of data integration, justification, and crucial evaluation, as well as including a discussion about incorporating multiple methods (Ellaway, 2020). In an editorial for HRDQ, Onwuegbuzie and Corrigan (2014) noted that the prevalence rate of MMR is not more than 13% because quality studies are not submitted and thus not accepted in HRDQ. While conducting a mixed-method design study can help enrich a study in breadth and depth, the way the data are merged is also important (Harrison et al., 2020). Onwuegbuzie and Corrigan (2014) outlined five steps in reporting MMR with rigor: comprehensive, systematic, evaluative, defensible, and transparent. Gibson (2017) introduced four main values of MMR which can increase the rigor of MMR: (1) elaboration, (2) generalization, (3) data integration, and (4) interpretations. Following Harrison et al. (2020), this study analyzes MMR from the perspective of handling and mixing data from qualitative and quantitative sources such as reporting data integration methods and providing rationale for employing mixed methods in their study. In summary, the literature review on the rigor criteria for each research method provided a comprehensive conceptual framework for this current study.
Methods
We reviewed empirical research published in the field of HRD over a period of five years in selected HRD journals and analyzed the corpus of articles using a content analysis method with a coding scheme that we created.
Data Collection
To ensure that we captured the recent research trends in the field of HRD, the researchers implemented a purposive and cluster sampling method utilizing three selection criteria: (1) published in major journals in the field of HRD, (2) empirical studies, and (3) published from 2016 to 2023 (8 years in total).
First, we analyzed all empirical articles (a total of 488 studies) published in three major journals in the field of HRD. Among the four journals sponsored by the Academy of Human Resource Development, Human Resource Development Quarterly (HRDQ) and Human Resource Development International (HRDI) were chosen because of their focus on publishing empirical studies. Human Resource Development Review was excluded from our study because it predominantly publishes theory, literature review and conceptual papers. The Advances in Human Resource Development journal was also excluded because the journal was operated through special issues with invited editors during our analysis timeframe (2016–2020). As the articles focused on a specific topic, the diversity of topics were less comparable to other journals that do not primarily operate through special issues. Although the journal accepts empirical studies, when we reviewed the abstracts of the published studies, many were predominantly theoretical or conceptual. Previous research has also valued the AHRD sponsored journals and has included these in their studies (Tkachenko et al., 2022). As part of our efforts to include studies published in other HRD journals, we purposefully selected European Journal of Training and Development (EJTD), one of the major HRD journals that publishes 9 issues per year, alongside two AHRD sponsored journals. Although there are other journals that discuss HRD topics, we have selected three journals, HRDQ, HRDI, and EJTD as a purposive and cluster sample of HRD journals, which meets the scope of our study.
Second, we selected empirical studies, which exclude non-empirical studies such as literature reviews, theory papers, conceptual papers, editorials, and opinion pieces. The first author conducted preliminary selection for empirical studies and organized by journal, publication year, and method used for the empirical study (i.e., quantitative method, qualitative method, and mixed methods). On the journal websites, we reviewed the title and abstract, skimmed through each of the articles and identified 271 studies published from 2016 to 2020 (five years in total), 217 studies published from 2021 to 2023 (three years in total), with a total of 488 studies in recent eight years (2016–2023). When we implemented our study in 2021, initially we conducted our in-depth preliminary analysis of five years (2016–2020) of studies, which includes details of research rigor (e.g., organization size, type, population). Then we added three years more (2021–2023) to provide the most recent holistic landscape of the research methods employed in the field of HRD. The expansion of our data did not change the research method trends in our initial findings, such as specific patterns or trends with the research methods employed in the selected years. We purposefully selected the years starting from 2016 as Onwuegbuzie and Corrigan’s (2014) study conducted a similar study analyzing research methods, specifically on mixed methods research trends until 2014 in HRDQ. Our study is not only an extension of the previous study but also expanded it including all three research methods (i.e., qualitative, quantitative, and mixed methods research) and three major journals in the field (i.e., HRDQ, HRDI, and EJTD). In addition, we did not use any sampling methods in identifying studies for review. Rather, we analyzed all empirical studies published in the selected three journals within the 8-year timeframe. Therefore, we identified five years of studies as our data saturation point and analyzed empirical studies published between 2016 to 2020 and additionally analyzed three years of studies (2021–2023) to provide the most recent landscape of research methods in the field.
Development of Coding Scheme and Data Analysis
We developed the coding scheme using both deductive and inductive approaches. Initially, we used existing literature on research methods and developed our first version of the coding scheme (e.g., Cohen et al., 2017; Creswell & Poth, 2018; Fraenkel et al., 2015; Johnson, 2001; Patten & Newhart, 2018). After conducting four rounds of pilot data analysis, we used an inductive approach to elaborate the coding scheme. Coding discrepancies were discussed respectively after we finished our pilot coding four times. We piloted the coding scheme by analyzing all empirical studies published in 2020 in the selected journals. Interrater agreement was achieved over 80% between and across three coders. During our weekly meetings, we discussed the coding process, reviewed the coding discrepancies, discussed about the rationale of coding until we achieved full consensus. Given the rigorous procedures of creating the coding scheme, we believe it is applicable for the analysis of current and future studies published in the field of HRD.
Coding Scheme for Analyzing Qualitative, Quantitative, and Mixed-Methods Research.
We analyzed the data using descriptive statistics (i.e. frequency) provided by SPSS and Microsoft Excel. After identifying missing coding, we went back to the data to address the missing data.
Results
Based on the three main research questions, the study findings are described below. The research questions include: (1) what are the research methods employed in empirical studies in HRD journals; (2) what is the degree of rigor in the research design of the select articles; and (3) what are the types of emerging research conducted in the field of human resource development. Our results are presented below to address each research question.
RQ1. What Are the Research Methods Employed in Empirical Studies Published in HRD Journals from 2016 to 2023? How Diverse Are the Research Methods Employed in the Select Articles?
Out of 488 empirical papers in three selected journals, quantitative (n = 269) and qualitative (n = 185) methodologies were predominantly used. As shown in Figure 1, among published empirical research from 2016 to 2023 in the three major journals selected, only 7.0% are MMR (34 among 488 studies). There was no other type of research method that did not fall into three paradigms: quantitative, qualitative, and mixed methods. Empirical studies in three journals by year.
Methods of Quantitative and Qualitative Study Type by Journal and Year.
There was a small variation in studies that employed qualitative research methods. More than half of the studies (56.8%) did not provide information on what specific qualitative research approach the study adopted. Among 185 articles that reported a qualitative study approach, most studies utilized case study (23.2%), which was followed by phenomenology (5.4%), narrative (4.3%), grounded theory (2.2%), and ethnography (2.2%). These patterns are similar across the journals and publication years. Four ethnographic studies were found since 2019 across all journals in HRDQ, HRDI and EJTD.
We investigate the current state of MMR in terms of research design dominance and sequence of inquiry. Given a small number of MMR in our sample (34 in total), it is difficult to report that there is a predominance of MMR design incorporated into HRD research. Seven studies applied triangulation design followed by exploration (n = 6), explanation (n = 3), and embedded design (n = 3). There were nine studies categorized as not specific, four studies classified as other (i.e., a quasi-experimental study, a case study, a hybrid Delphi study, and a mixture of triangulation and exploration design), and two were survey studies using qualitative and quantitative questions in a questionnaire. For dominance and sequence of inquiry, interestingly, quantitative dominant (n = 9), qualitative dominant (n = 11), and equal status (n = 10) was nearly equally distributed among 34 select studies excluding four studies that did not state the dominance of inquiry. The analysis results indicate that there is no preference toward the use of a dominant method in MMR. In the sequence of inquiry, qualitative to quantitative (n = 17) was the most dominant as half of the select MMR studies used the sequence. Quantitative to qualitative (n = 8) and simultaneous (n = 4) were used in less than half of the MMR studies. Four studies did not report on a specific sequence used to collect data and two studies were difficult to classify into the traditional categories, as data was collected at multiple points of time.
RQ2. What is the Degree of Rigor in the Research Design of the Select Articles from 2016 to 2020?
Quantitative Studies
The rigor of quantitative studies was analyzed based on population, context, sampling, data collections, data analysis, and measurement validity and reliability (see Table 1).
Population, Context, and Sampling
It is noteworthy that studies did not provide enough information to understand the context in which each was conducted. Organization size was reported in only 32.4% (n = 47). Among these, 15 studies reported the specific number of the organization, 21 studies used a small/medium/large category, and 11 studies used their own category. A majority of studies (n = 98, 67.6%) collected data from multiple sites. However, only 44 studies among them specified the number of sites, which varied from 2 to 2000. 2 to 5 sites (n = 18, 40.9%) were most frequently reported. 23.4% of studies (n = 34) were conducted in a single site. Four studies did not specify whether the study used multiple sites or a single site. Nine studies used other data sources including a single training academy, state education board, archive data, meta-analysis, and the MTurk database. 42 studies (29%) did not report the type of organization.
Despite the fact that statistical inference is the foundation of quantitative study, 71.7% (n = 104) of the studies used non-probability sampling. Convenience sampling (n = 57, 39.3%) and purposeful sampling (n = 21, 14.5%) were common in quantitative studies. On the other hand, quota sampling that systematically controlled detailed characteristics of the sample took up only 1.4% (n = 2) of the studies. A high proportion of convenience or purposeful sampling methods implies that generalizability of the results is confined within the context (i.e. the characteristics of each sample) and the results should be cautiously and tentatively recommended to practice in other contexts (Reio, 2016).
The number of participants ranged widely, from 24 to 92,262. While 201-400 participants was the most popular range, studies with less than 200 participants and studies with over 1000 occupied a similar proportion, 20% and 15.9% respectively.
The job positions of participants were precisely reported in 63 studies (43.4%). Whereas 18 studies (12.4%) were partially reported. The remainder described some characteristics of the participants or did not mention the job position of participants at all. Subordinates were 24 cases (16.6%) and manager/directors, and executive/owner/CEO occupied 36 cases (24.8%) and 11 cases (7.6%) respectively. An unexpected finding was that studies represented the perspectives of managerial level employees (e.g., managers, leaders) more than subordinates. Besides typical for-profit organization contexts, faculty, students, and others constituted 11 cases (7.6%), 11 cases (7.6%), and 10 cases (6.9%) respectively among the empirical studies.
The context of Asia took up 37.9% (n = 55) followed by Europe (n = 31, 21.4%) and North America (n = 27, 18.6%). International (multiple countries) occupied 7.6% (n = 11). 6.2% (n = 9) studies did not specify the country in which the study was conducted.
Duration of Observation and Data Sources
SEM is known as a statistical method to test causal relationships among variables; however, the causal relationship can be established only when the study is conducted with a longitudinal design. The study results indicated that only 19.3% (n = 28) of the studies collected data at multiple points of time. Moreover, very few studies 6.9% (n = 10) took time over one year for data collection. For data sources, most of the studies used single methods (n = 136, 93.8%). 130 studies (89.7%) used surveys followed by archives (n = 14, 9.7%), other (n = 3, 2.1%), or observed data (n = 2, 1.4%) for data collection. Among studies categorized as “other,” two studies were meta-analysis studies.
Data Analysis
Methods of Data Analysis by Journal and Year (n = 145).
1 = Group Difference, 2 = Regression analysis, 3 = SEM, 4 = Model validation & prediction, 5 = Multilevel modeling, 6 = Other (Descriptive, correlations, multiple, etc.).
For unit of analysis, six studies analyzed dyadic relationships (n = 6, 4.1%) and multiple hierarchy (i.e., both individual and organization level) (n = 11, 7.6%). The rest of the studies analyzed data at the individual level (n = 118, 81.4%). Next, we analyzed research models used in the top three data analysis methods. Among 44 mediation analysis studies, SEM (n = 41, 93.2%) was a typical approach. Only 15 studies tested a moderation model and studies used either regression analysis (n = 8, 53.3%) or SEM (n = 7, 46.7%). SEM was also used for more complex model testing involving both mediating and moderating relationships (n = 8, 5.5%). Among 59 studies that used SEM, only 19 studies specified control variables. Most of the HLM analyses were applied to hierarchical structure (n = 16, 80.0%) and only three studies used HLM for testing a time lag model.
Measurement Validity and Reliability
As an indicator of methodological rigor, 123 studies (84.8%) reported a reliability coefficient for measures, typically Cronbach’s alpha. On the other hand, 88 studies (58.6% of all the quantitative studies) tested validation of measures used in each study. It was typical to report the measurement validation process prior to SEM analysis results, but that was not the case for other analysis methods. Although most of the studies (130 cases, 89.7%) relied on survey data, only 58 studies conducted common method variance control processes. Moreover, a majority of the studies (n = 40) simply relied on a statistical control method for common method bias testing. Less than 11% (n = 15) of the studies practically controlled the common method variance using multiple data sources.
Qualitative Studies
Rigor of qualitative studies can be observed in a detailed description of context, participants, duration of observation, data analysis, and trustworthiness.
Description of Study Context
More than half of the qualitative studies were conducted mainly in Europe (31%) and Asia (20%). The remaining qualitative studies were conducted in North America (18%), international locations or multiple countries (9%), Africa (8%), Oceania (4%), and South America (3%). However, some studies did not specify the location of the study (7%). These overall patterns had some variation by journals and years. Most qualitative articles published at HRDQ were conducted in Europe (37.0%) and North America (22.2%), while HRDI published papers were conducted mostly in Asia (30%.3%) and Europe (27.3%). EJTD published studies mainly in Europe (30.0%), followed by Asia (20.0%) and North America (20.0%). Studies in Asia have increased from 10.5% in 2016 to 35.3% in 2020, while articles in North America had a fluctuating pattern.
Most qualitative studies (65%) did not provide a study population regarding participants’ organizational size. Twenty out of 35 studies provided specific numbers of employees of the organizations, among which 15 studies were situated in an organization whose size was more than 500 employees. Eight studies ranged from 50 to 499 or medium. Eleven studies recruited participants from small organizations with less than 50 employees. While this pattern is consistent among journals, an increased tendency was found in EJTD (72.5%). However, the recent publications provided population information regarding an organizational size (47.1% in 2020) more than five years ago (15.8% in 2016).
Some studies (28%) did not report the organization type information. Twenty-five studies (25%) were conducted at private or for-profit organizations, while 14 studies (14%) were with public/government/non-profit organizations. Twenty studies (20%) were conducted in both private and public organizations. Organizations’ business areas showed a similar pattern. Twenty-one studies (21%) did not specify the organization’s business. Qualitative studies were rarely conducted in manufacturing (5.1%). Many studies were conducted in multiple mixed business sectors (25%). Some studies (16%) were conducted in service sectors (non-manufacturing), educational institutions (10%), and governmental institutions (9%).
Description of Participants
Twenty-six out of 100 studies did not report the participants’ job positions. Among those who reported the participants’ job positions, 47 articles recruited participants from multiple job positions. Study participants’ job positions included managers (51%), subordinates (29%), executive/CEO (21%), students (3%), faculty (4%), and other (15%).
More than half of the studies (59%) recruited participants from multiple sites. However, 22 studies out of 59 with multiple sites did not report the specific number of sampling sites. Thirty studies recruited participants from a single location.
As expected, the sampling method of qualitative studies was a nonprobability method (91%). While few in number, some studies (8%) did not specify what sampling methods were utilized. Regarding nonprobability sampling types, purposeful sampling was the most prevalent sampling type (47%), which was followed by convenience sampling (15%), mixed (13.0%), snowball sampling (7%), voluntary sampling (2%), and other (2%). Some studies did not specify the sampling type (5%).
Duration of Observation and Data Sources
Most qualitative studies (80%) collected data at one single point in time. Only 15 studies collected data at multiple points in time. More than half of these did not report a concrete timeline. All six studies that provided a data collection timeline were within one year regarding data collection. These patterns were consistent across the journals and publication years. There was no qualitative study that investigated a topic longitudinally over a year.
Most qualitative studies (74%) used only one data source, while some studies (25%) utilized multiple data sources. Interviews were the most frequently used data collection method (90%), followed by document analysis (22%) and observations (17%). These patterns were consistent across the journals and publication years. Qualitative case studies were the most popular qualitative methodology in the data. Ten (34.5%) out of 29 case studies utilized only a single data source, such as interviews.
Data Analysis
Most qualitative studies analyzed data using either content analysis (31%), thematic analysis (29%), or other (14%). Other included autoethnography analysis, discourse analysis, narrative analysis, pattern matching, and general descriptions. More than half of the studies (52%) started with open coding, an inductive data analysis technique. Some studies (24%) utilized both open coding and a priori coding processes. A few studies (10%) adopted a deductive approach using a priori coding. The open coding results reveal that only 21 studies were conducted utilizing axial coding, a relationship analysis. Six studies did not specify data analysis approaches. These patterns were consistent across the journals and publication years.
Trustworthiness
Qualitative studies used either triangulation (35%), member checking (8%), or both (17%) to improve the trustworthiness of the methodology. However, a good number of studies (37%) did not report any process for trustworthiness. These patterns were consistent across the journals. More efforts for trustworthiness have been observed in recent publications. For example, 50% of the studies published in 2017 did not report on trustworthiness, which has changed so that only 17.6% of the papers failed to report on trustworthiness in 2020.
Mixed Methods Research
Analyzing MMR with a unified coding scheme for a small sample (26 studies) was a challenge as it may differ based on design (i.e., triangulation, embedded, explanation, exploration design); dominance of inquiry (i.e., quantitative dominant, qualitative dominant, equal); sequence (i.e., qualitative to quantitative, quantitative to qualitative, simultaneous); and data integration (i.e., merging, embedding, connecting). This study heavily relies on the rigor of qualitative and quantitative research. For the scope of this study, we analyzed two aspects of MMR rigor including: (1) data integration/merging; and (2) discussion of the value of using mixed methods, which are the core rigor criteria for MMR (Gibson, 2017; Harrison et al., 2020).
Data Integration/Merging
Data integration refers to how the qualitative and quantitative results are brought together in a mixed methods study (Creswell & Clark, 2017). The analysis shows that connecting the data (11) and merging the data (10) were the majority integration methods used for the select studies. Two studies used the embed method, two were unclear, and one used another method that was not included in Creswell and Clark (2017).
Discussion of the Value of Conducting MMR
Having discussed the primary elements of rigorous MMR, we will now discuss the advanced elements that may contribute to an article’s mixed methods rigor. The first advanced element involves the discussion of the aims and purpose for conducting mixed methods research. The other is an integrated discussion of both qualitative and quantitative results. While 11 studies out of 26 provided a rationale for using MMR for their research, only 8 out of 26 studies provided an integrated discussion regarding the study findings from both qualitative and quantitative data.
RQ3. What Are the Types of Emerging Research Conducted in the Field of Human Resource Development from 2016 to 2023?
All empirical papers were categorized into one of three categories: qualitative, quantitative, and mixed methods research. There was no paper that used a novel methodology that did not follow this categorization. Within each category, further investigation was conducted to identify emerging research methods.
There were only a handful of studies that adopted emerging or non-mainstream qualitative research methods, which included autoethnography (Black & Warhurst, 2019; Sisco et al., 2022), grounded theory (Barlett et al., 2021), and discourse analysis studies (Kirrane et al., 2017; Mizzi & O’Brien-Klewchuk, 2016). As an emerging quantitative study method, there was a text mining paper (Fahrenbach et al., 2020) that utilized natural language processing processes. In a field study, a diary method was utilized on a weekly basis (Bellini et al., 2021). Also, a quasi-experimental study implemented latent class analysis to assess interventions (Pandya, 2023).
Akdere and colleagues incorporated biometric data in their mixed methods research (Akdere et al., 2022, 2023). A critical incident technique was utilized to develop and validate a novel index of occupational strengths (Moore et al., 2022). Most studies did not provide any explicit evidence or a statement of their methodological rigor, such as trustworthiness, validity, and reliability. Transparency regarding the rigor of emerging or non-mainstream methodologies remains unclear.
Discussion
This study found relatively balanced proportions of quantitative and qualitative studies in HRD research among the 488 empirical studies reported in the three selected HRD journals although there was a small number of mixed methods studies. Given the different epistemologies and research focuses of the analyzed studies, this study may contribute to expand and deepen our understanding on HRD research theory and practice. The study findings also provide implications and reflections on the current status of HRD research practice and publications. We will discuss implications for practice and research in details based on each research method below.
Quantitative Research
Beyond exploring the relationships among research variables, establishing causality among them is the ultimate criteria for quantitative research (Nimon & Astakhova, 2015). However, this study revealed that methodologies of HRD studies relied heavily on statistical methods to explain the causal relationships, which did not adequately meet the conditions to establish causality. Establishing causality in quantitative research requires rigorous design, analysis, and interpretation. While it is challenging to establish causality definitively, several measures can help strengthen causal claims. Establishing causality in quantitative research includes measures such as experimental design, control groups, randomization, manipulation of the independent variable, covariate control, and replication. We discovered that HRD researchers have been keeping up to date with the recent development of statistical methods including SEM and HLM. For example, this study found that SEM became a standard statistical approach for mediation analysis, which does not align with Kim’s (2004) findings. Interestingly, the rate of studies using HLM analysis reached its peak (24.3% of quantitative studies) in 2019. Despite the increased use of advanced statistical methods, it is noteworthy that it does not guarantee that there is causality among variables. In fact, the results of this study indicated that the majority of HRD research still conducted cross-sectional single-respondent survey studies, which is a critical limitation of HRD research. This phenomenon is not unique to HRD research and is a longstanding issue in social science research. For example, similar results were reported in the entrepreneurship business field (Maula & Stam, 2019) and creativity studies and gifted education research (Long, 2014). Maula and Stam (2019) also reported a trend that multi-wave surveys and use of multiple informants in data collection are becoming the new standard in quantitative studies to address the limitations of single point self-reported survey studies. In particular, topics related to performance and change, the key phenomenon in the HRD field, would be properly addressed based on longitudinal data and objective evidence such as other’s ratings or organizational level data. This study found that 19.3% of HRD studies collected longitudinal data from 2016 to 2020, which was a notably higher rate than 2% reported in Long’s (2014) review on creativity studies. On the other hand, overreliance on self-reported data seems to be more problematic in HRD research, since only 10.3% of studies used multiple respondents (i.e., both from employee and supervisor) for data collection.
Regarding longitudinal analyses, this study found only a small portion of studies in HRD kept temporal lags over one year. Appropriate temporal lags are important to ensuring the pattern and the stability of relationships over time. Therefore, HRD scholars may need to explicitly inquire into the assumptions surrounding the design of longitudinal studies such as appropriate temporal lags and the number of times to measure as well as the issues of attrition and missing data.
In quantitative studies, the information regarding research setting and the context seemed to be overlooked. This is a critical threat to transparency in reporting and generalizability of the results (Maula & Stam, 2019). Many studies did not report enough information to judge the suitability of the data set such as organizational types and sizes and participants’ job positions. Studies that briefly explained the research setting were ones that used archival data, studied in educational institutions, and obtained research data from private companies that provide a diverse pool of participants matching the population of the researcher’s needs. However, very few studies provided information about the background of the participants. Given the increase of online data collection and online platforms in recruiting research participants (e.g., MTurk), it is natural to expect that HRD studies will use more large size databases. Like convenience sampling, it is very plausible that online databases will be constructed using voluntary sampling and this could aggravate the selection bias in HRD research. Thus, it is necessary to establish criteria to report the background information of participants and research settings. The sampling criteria and procedures, associated measures, and the characteristics of missing data and the implications of these information in terms of research questions need to be clarified with more detailed accounts.
Qualitative Research
These findings provide several implications for qualitative research. First, there is an alarming lack of clarification of qualitative research approaches in published qualitative research papers in HRD. The current study found that half of the qualitative studies did not explicitly provide a specific approach that the authors adopted to investigate the topic. Lester et al. (2020) also reported a similar finding that 19 out of 59 qualitative research papers published in HRDQ from 1990 to 2019 did not report a specific qualitative research approach. They further argued for the importance of methodological transparency in reporting a qualitative study, where research rigor can be estimated. Qualitative research has a wide range of inquiry approaches including case studies, phenomenology, ethnography, narrative, discourse analysis, and conversation analysis, based on the purpose of inquiry and epistemological stance (Creswell & Poth, 2018). Each approach has guidelines regarding what constitutes each approach and what specific research design researchers should consider and focus on. For example, ethnography is a qualitative research approach to pursue an understanding of shared culture, patterns of behavior, belief, and language of a group of people that developed over time. It often entails prolonged field work including observations, interviews, and gathering artifacts (Creswell & Poth, 2018). A lack of clarification of approaches is problematic. It could cause a poor understanding of why the paper uses a certain type of qualitative research and how the methodological processes become valid and appropriate for the qualitative epistemology.
Second, a research context is a critical element in understanding how we shape knowledge in the field (Reio, 2021). Specific information about the participants and their contexts in qualitative research is crucial for transparency regarding the logic of the inquiry, trustworthiness, and transferability (AERA, 2006; Guba & Lincoln, 1982). Moreover, the limited range of research population reported in publications creates room for reflections on the knowledge production process in the field regarding whose voices are mostly heard in qualitative studies. Except for HRDI, most published qualitative papers were from Western countries, including North America and Europe. While championing multiple realities and ontologies, the current HRD knowledge generated from qualitative studies is obviously shaped by limited contexts, which calls for changes. It may be associated with a structural problem in which most journals are based in Western countries, and editorial members are trained and working in those areas. Those who are in non-Western countries may be less familiar with the subtle nuances of publication-quality criteria for qualitative studies. A recent innovative program, called International Developmental Workshop for “Writing for Publication,” initiated by HRDR and HRDI, reflects this need for diverse voices in publications and aims to facilitate publications by non-native English speakers from around the world.
When looking at a more granular level, the current study found that most qualitative studies did not provide the organizational context of the participants. Most papers did not clarify how big the participants’ organizations were, or what types of business or service their organizations had. Among those which were known, most participants were from large for-profit organizations and hold a manager position. This homogeneous group of study participants is also a concern as their perspectives and narratives represent the narratives of the field that shape the knowledge in HRD. It would undoubtedly be different to be a manager at a not-for-profit organization with 20 employees than to be one for a for-profit with 10,000 employees. Moreover, only two studies investigated the perspectives of the unemployed, who are part of human resources but have rarely been part of HRD research populations.
Another reflection is about the source of data. Most qualitative studies in HRD literature conducted interviews as the only primary data source within one year. Participants’ perspectives and values shared in an interview format were the sources of the knowledge in the literature. Few studies adopted comprehensive and diverse data sources, including field observations, conversations, artifacts, documents, etc., for multiple years to acquire a longitudinal understanding of the research topic. It is worth reconsidering the credibility of a collective set of knowledge in the field generated predominantly from cross-sectional interviews. There is a different set of knowledge that we could acquire from different sources of data, such as prolonged observations, conversations, and artifacts.
Methodological diversity and openness to emerging inquiry methods have been recognized as essential to advance the breadth and depth of knowledge in the HRD field (Nimon, 2016). However, the current study found limited diversity in qualitative research methods. The absence of emerging qualitative methodologies helps us reflect and ask what the absence of some methodologies means to the field. For example, we have found no conversation analysis study and only one autoethnography study published in the three selected HRD journals for the last five years. It may be the case of a vicious cycle where a lack of understanding of rigor of other qualitative research approaches among researchers and reviewers results in few publications, which hinders any future attempts for other methodologies.
Mixed Methods Research
MMR studies represented only 9.6% (26 out of 271) of the empirical articles published in the past five years from 2016–2020 and 7.0% (34 out of 488) from 2016–2023 in the three selected journals. This information reveals that the number of MMR have in fact decreased in recent three years. Before discussing anything about MMR, it is evident that MMR is not sufficient and thus more quality MMR needs to be conducted in HRD to increase the number of MMR. In HRDQ’s editorial in 2014, Onwuegbuzie and Corrigan have urged for more quality MMR in the field of HRD. Their analysis reveal that from 2000 to mid 2014, 20 MMR (13% of 230 total articles) was published in HRDQ.
Comparing the ratio of MMR in total empirical research provides a rough overview of the percentage of published MMR in HRD empirical research although it is difficult to compare these two studies equally, given that different timeframes (8 years vs. 13.25 years), published journals (HRDQ, HRDI, EJTD vs. only HRDQ), and the total number of empirical research (488 vs. 230). It is alarming that there has been a decrease of MMR research published since then. The study results also show that less than half of the studies have provided a justification for selecting MMR for their research and only limited studies reported an integrated discussion from the findings of the various data collected.
Overall Implications for Theory and Practice
In summary, the current study provides evidence for concern that HRD journal editorials have raised to improve methodological rigor, focusing on the quality of evidence in reporting research studies (Nimon & Astakhova, 2015; Reio, 2021). To move the field of HRD forward as a mature, effective, and evolving discipline, it is imperative to improve the breadth and depth of epistemological diversity and rigor in research methods. By reviewing the recent eight years of empirical research in three major selected journals, this study provides integrated insights that inform diverse stakeholders including researchers, practitioners, and journal reviewers on how we can achieve research diversity in the field of HRD.
First, we found the encouraging phenomenon of continuous advancement in HRD theory building. For example, in quantitative research, more researchers have tried to use advanced mediation analytical approaches such as SEM and HLM, which may enhance the quality of research and generate sound theory building (Song & Lim, 2015). It has been known that some other disciplines tend to perceive qualitative research as ‘second class’ research resulting in the dominance of quantitative research (Tong et al., 2007). However, a balanced rate of qualitative and qualitative research implies that researchers in HRD communities acknowledge the unique value of qualitative research in creating knowledge. We believe this phenomenon implies that HRD communities have made ceaseless efforts to enhance overall diversity and quality of HRD research, which compares favorably with other related disciplines such as psychology and management and organizational studies. Moving forward, HRD scholars need to focus on methodological pluralism embracing MMR and examining higher quality MMR that is designed and reports dominance and sequence of inquiry, and rationale of conducting MMR.
Beyond the advantages of a current research trend, there were several common disadvantages across the three types of research methods (i.e., quantitative, qualitative, and mixed methods) to overcome. First, lack of diversity in data collection and data analysis were quite evident. Overreliance on cross-sectional single respondents were problematic both in quantitative (i.e., survey study) and qualitative (i.e., interview) research methods, which can systemically skew results and hamper reliable and valid theory building (i.e., establishing and explaining causality) (Reio, 2010). Given that this issue has not been improved since Song and Lim (2015) criticized the similar issue in their review on mediating analysis research in recent ten years, HRD communities need to be more open-minded to try various alternative data sources.
The analytical approach was another source of biased knowledge creation in HRD. SEM, content analysis, and thematic analysis were the most prominent in quantitative and qualitative research respectively. These data analysis approaches may be advanced and valid in the given context of an individual study. However, each methodological approach involves principled unique decision-making processes to structure individual experience and/or social phenomenon (Seale, 1999). Seale (1999) argued that rather than conducting uniform procedures, the development of one’s own research style, which leverages a diverse series of principled procedures, makes a research community flourish. Emphasis on diversity in research methods is not limited to the qualitative paradigm; researchers with a quantitative orientation also recommended the use of multiple analytical procedures to explain complicated research frameworks (MacKinnon, 2012; Song & Lim, 2015). Nimon (2016) also encouraged researchers to spend less time on convenience research and take more risk in answering the questions that really matter. Pursuing diversity in methodological approaches may be of key importance to determine rigor and maturity in HRD research. Leaders in the HRD research community should encourage future researchers to implement divergent research frameworks and methodological approaches.
Another disadvantage, lack of contextual information, has an important implication particularly for practice as well as theory building (Rocco, 2003). This issue relates to thinking about ‘what works’ questions, which generate effective solutions in a given context. As Nimon (2016) asserts, it is not unusual that an optimally designed study does not make a difference in the given field. A detailed description of research context allows the reader to get a sense of the practical meaning and impact of research findings and judge the generalizability and replicability of the research. Continuous discussions in the form of professional conference workshops or webinars would help raise awareness for the reporting of details regarding the study context, data collection processes, and analysis methods, as well as employing diverse research methods in the field of HRD. Additionally, academic programs should focus more on teaching students the requirements of research methods rigor and reporting.
The coding scheme (see Table 1) can be used as a guideline or checklist tool to assist individual researchers during research design and manuscript preparation, editors and reviewers in the manuscript review process, readers when appraising study findings, and instructors when developing a curriculum on research methods. It may be especially helpful for graduate students and early career researchers to use as a checklist or guide when writing empirical papers. Overall, this study will contribute to increase the degree of awareness about unappreciated gaps in the diversity and quality of HRD research over time. Ultimately, we anticipate that this research will open opportunities for a solid research conversation to reach methodological pluralism in the field of HRD.
Limitations and Suggestions for Future Studies
There are several limitations associated with this study regarding the selection criteria for empirical research included in the study and the items in the coding scheme. First, we purposefully included only those articles published in three specific HRD journals within eight- and five-year period. Eight years of research from 2016 to 2023 were analyzed to provide a holistic landscape of the research methods employed in three selected HRD journals. A more in-depth analysis of research rigor was conducted for research from 2016 to 2020 (5 years in total). This selection criteria may have yielded a sampling bias that limits the generalizability of the research findings within the selected range of field and time. Future studies may consider extending the study to explore the research trends in a longer period. This may be conducted as a historical tracing of the evolution of the HRD field. We limited the timeframe to eight (2016–2023) and five years (2016–2020) because we examined all studies published in three journals. Other studies may look into individual journals and examine the landscape of the research methods utilized in each of the select journals. Other major journals in HRD-related content may be included in the analysis in future studies as well. Second, items in our coding scheme were intended as indicators for methodological rigor rather than as a comprehensive list of elements to construct each type of research method. For example, a clear description of researcher reflexivity is one of the essential elements to improve the transparency of qualitative research studies; however, we did not include author subjectivity statements in our coding scheme. We expect the coding scheme to develop over time through the HRD communities’ continuous discussions and efforts to enrich methodological rigor and diversity. Third, MMR was difficult to fit into a unified coding scheme and therefore, this study heavily relies on the analysis of qualitative and quantitative studies. The number of published MMR was limited as well. Continuing Onwuegbuzie and Corrigan’s (2014) work, future studies may conduct in-depth analysis on the MMR conducted in the field of HRD.
Rigorous and abundant knowledge creation in the HRD discipline can be achieved through multiple approaches such as: (1) including interdisciplinary investigation of emerging research methods; and (2) developing clear guidelines for rigor of research methods especially for emerging methods. Journals should welcome studies that adapt less traditional research methods and encourage such papers in special issues. Also, one of the reasons for lacking detailed information about the study context, population, data collection process, and analysis may be because of the word-limit or page-limit set by journals, forcing authors to trim down the length of the paper.
While some guidelines have been provided for quantitative studies, few papers investigated the status and rigor of qualitative research and MMR in the field of HRD. We call for future studies and ongoing discussions to improve the methodological diversity and rigor in the field of HRD. In particular, as there is a lack of quality MMR published in HRD journals, future studies may dive deeper into addressing how to increase rigor in MMR in the field of HRD. Due to rapid societal changes and advancements in technology, we anticipate more HRD studies on emerging topics and interdisciplinary topics articulating more diverse research methods. The coding scheme was not developed as a stagnant checklist. Rather, we expect opportunities to revise, expand, and further develop it as new research methods proliferate.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
Correction (September 2024):
Article updated to correct the citation “Tkachenki et al., 2022” to “Tkachenko et al., 2022”.
Author Biographies