Implementing Quality Criteria in Designing and Conducting a Sequential QUAN → QUAL Mixed Methods Study of Student Engagement With Learning Applied Research Methods Online

Abstract

In spite of recent methodological developments related to quality assurance in mixed methods research, practical examples of how to implement quality criteria in designing and conducting sequential QUAN → QUAL mixed methods studies to ensure the process is systematic and rigorous remain scarce. This article discusses a three-step procedure for securing the quality of the meta-inferences generated from sequential employment of quantitative and qualitative methods and offers several validation strategies specific to a sequential QUAN → QUAL mixed methods design: applying a systematic process for selecting participants for qualitative follow-up, elaborating on unexpected quantitative results, and observing interaction between qualitative and quantitative study strands. The discussed procedures are illustrated using a mixed methods study of graduate student engagement in learning applied research methods online.

Keywords

mixed methods research sequential design validity inferences

In spite of advances in the adoption and utilization of mixed methods research (Creswell, 2010; Ivankova & Kawamura, 2010), quality assurance or validity in mixed methods studies has remained one of the provocative methodological issues and most debatable topics in mixed methods field in recent decade. Concerns about drawing quality conclusions or inferences in mixed methods studies have been listed among major controversies in the two editions of the Sage Handbook of Mixed Methods in Social & Behavioral Research (Tashakkori & Teddlie, 2003, 2010). In addition, a number of mixed methods authors have emphasized the complexity of mixed methods validity stemming from different conceptualizations of quality in mixed methods research (Creswell, 2010; Maxwell & Mittapalli, 2010; O’Cathain, 2010; Onwuegbuzie & Johnson, 2006). Ensuring quality in mixed methods studies can be especially challenging because of the intended integration of quantitative and qualitative results to produce credible meta-inferences. Teddlie and Tashakkori (2009) defined “meta-inference” as “a conclusion generated through an integration of the inferences that have been obtained from the results of the QUAL and QUAN strands of a MM study” (p. 152). As such, integration of the inferences derived deductively and inductively is a critical stage in a mixed methods study process and researchers should adhere to rigorous standards for assessing inference quality to ensure their credibility and validity.

Despite this critical need, there are no accepted criteria for appraising the methodological quality of mixed methods research and the produced meta-inferences (Creswell & Plano Clark, 2011; Leech, Dellinger, Brannagan, & Tanaka, 2010; O’Cathain, 2010). Moreover, social science researchers have varied opinions on what constitutes quality in quantitative, qualitative, and mixed methods research (Bryman, Becker, & Semptik, 2008). Evaluation standards and the choice of strategies are often influenced by researchers’ philosophical views and epistemological practices that dictate the terminology, definitions, and interpretations of validity in quantitative and qualitative inquiry and, consequently, the assessment of the quality of the integrated meta-inferences in mixed methods research (Dellinger & Leech, 2007; Greene, 2007; Maxwell & Mittapalli, 2010).

Special considerations should also be given to quality assurance and validity related to types of mixed methods designs and how the quantitative and qualitative methods are integrated (Creswell & Plano Clark, 2011). Different chronology of quantitative and qualitative data collection and analysis procedures in concurrent and sequential mixed methods designs, the order of quantitative and qualitative strands in sequential designs (quantitative or qualitative first), and the point of methods integration or interface (Morse & Niehaus, 2009) call for special strategies aimed at ensuring inference quality and minimizing threats to validity caused by unique mixed methods design features. Specifically, in sequential mixed methods designs, in which one study strand builds on another, the quality of the inferences produced in one study strand may markedly affect the quality of the inferences generated in another strand (Creswell & Plano Clark, 2011; Teddlie & Tashakkori, 2009). Ultimately, the quality of the meta-inferences from the overall study may be affected.

However, available conceptual discussions about quality assurance and validity in mixed methods research outweigh practical recommendations and examples on how to implement quality criteria in designing and conducting mixed methods studies to ensure the process is systematic and rigorous. Providing researchers with methodological guidance on how to assess and establish quality during a mixed methods study design and implementation may enhance a study’s validity and ensure the credibility of the inferences resulting from the application of mixed methods. This article contributes to the discussion about quality assurance in mixed methods research and focuses on the methodological steps and strategies that can be used to achieve credible inferences and alleviate validity threats related to a sequential QUAN → QUAL mixed methods design and its implementation. The discussed procedures are illustrated using a sequential QUAN → QUAL mixed methods study of graduate student engagement in learning applied research methods online.

Quality Assurance and Validity in Mixed Methods Research

Quality Assurance and Validity of Meta-Inferences

Quality assurance and validity in mixed methods research have long been the focus of methodological discussions by mixed methods scholars. In their seminal Handbook on Mixed Methods in the Behavioral & Social Sciences, Tashakkori and Teddlie (2003) introduced the term inference quality as a criterion to judge the validity and transferability of the generated meta-inferences in a mixed methods study. Inference quality is defined as the accuracy with which researchers draw inductively and deductively derived conclusions from a mixed methods study, characterized by meaningful integration of quantitative and qualitative methods (Tashakkori & Creswell, 2007). Tashakkori and Teddlie (2003) suggested judging about inference quality using two sets of standards: design quality—standards to evaluate methodological rigor of a mixed methods study, and interpretative rigor—standards to evaluate accuracy of the generated conclusions. Later, Teddlie and Tashakkori (2009) introduced the concept of integrative framework for inference quality in mixed methods research. The framework includes 10 aspects of quality related to a mixed methods study design (suitability to answer the research question, fidelity of the study procedures and methodological rigor, consistency of all research aspects of the study, and adequacy of analytic procedures) and interpretive rigor of the meta-inferences produced as overall study outcomes (consistency with findings, theory, and previous research, with the study purpose, with inferences from each study strand, with other possible interpretations by scholars and study participants, and distinctiveness of credible conclusions). The integrative framework is focused on helping researchers first reduce any inconsistencies by assessing quantitative and qualitative inferences generated in separate study strands using respective quality criteria and standards and then assess the degree to which the meta-inferences resulting from the entire study are credible.

Alternatively, Onwuegbuzie and Johnson (2006) proposed a legitimation model—a typology of nine types of legitimation or validity in mixed methods research (sample integration, inside-outside, weakness minimization, sequential, conversion, paradigmatic mixing, commensurability, multiple validities, and political) that permeate all research aspects of a mixed methods study from a researcher’s philosophical stance to a study design and implementation. The authors viewed legitimation as a continuous process of evaluating the study design, procedures for data collection and analysis, and interpretation of results. The advantage of the legitimation model is that it connects establishing validity in a mixed methods study with stages in the mixed methods research process, thus equipping researchers with conceptual knowledge of potential legitimation threats in the process of a mixed methods study implementation.

Leech and colleagues (Dellinger & Leech, 2007; Leech et al., 2010) proposed a framework for viewing mixed methods validity within a general notion of construct validity that was analogous with construct validity in quantitative research and encompassed all validity evidence in a study. Their validation framework includes four elements (foundational, inferential consistency, utilization, and consequential) that embrace all phases of the research process from review of the literature, through the study design, implementation, and evaluation, to the generation and application of inferences. Recently, O’Cathain (2010) suggested a comprehensive framework to assess quality in mixed methods research based on Teddlie and Tashakkori’s (2009) integrative framework for inference quality and contributions from other mixed methods researchers. This framework attempts to incorporate different conceptualizations of validity in mixed methods research and provides researchers with common language and guidance on how to access the quality of meta-inferences. The framework consists of eight domains that correspond to the stages of study design and implementation and include quality of planning, design, data, interpretative rigor, inference transferability, reporting, synthesizability, and utility.

In spite of different conceptual approaches to quality assurance in mixed methods research, many authors agree that the quality of respective quantitative and qualitative findings directly affects the quality of the meta-inferences drawn from an overall mixed methods study (Bryman et al., 2008; Creswell & Plano Clark, 2011; Dellinger & Leech, 2007; Greene, 2007; Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009). For example, Greene (2007) suggested differentiating between the quality of the method used (quantitative and qualitative) and the related data, and the quality of the mixed methods inferences or interpretations. Teddlie and Tashakkori (2009) argued that because mixed methods involves collection and analysis of both quantitative and qualitative data, three sets of validity checks should be employed to assess the quality of the generated inferences: (a) evaluating the inferences made on the basis of quantitative data using quantitative standards, (b) evaluating the inferences made on the basis of qualitative data using qualitative standards, and (c) assessing the degree to which the meta-inferences made on the basis of these two sets of inferences are credible. Importantly, failure to observe appropriate validity standards for a respective quantitative or qualitative approach used in a study strand might lead to a chain of erroneous conclusions resulting in less credible study outcomes. Consequently, mixed methods researchers are faced with the need to address validity issues in all quantitative and qualitative strands to enhance the quality of the overall study results.

Strategies for establishing reliability and validity of the quantitative data and results and credibility and trustworthiness of the qualitative findings are discussed in several mixed methods related books and articles (Creswell & Plano Clark, 2007; Dellinger & Leech, 2007; Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009), as well as numerous research methods texts. It is recommended that researchers apply these validity checks while collecting and analyzing data for the quantitative and qualitative strands in a mixed methods study. Dellinger and Leech (2007) argued that accruing evidence of validity during each study strand helps support data meanings and generate meaningful conclusions.

Quality Assurance and Validity Issues Related to Mixed Methods Research Designs

Along with discussions about quality assurance in mixed methods research, issues related to validity have been identified as pertinent to specific features of mixed methods designs (Creswell, 2010). In their integrative framework for inference quality, Teddlie and Tashakkori (2009) underscored the importance of design quality that refers to “the degree to which the investigator has selected and implemented the most appropriate procedures for answering the research questions” (p. 302). Teddlie and Tashakkori (2009) suggested that criteria of quality related to design issues should include the indicators related to (a) design suitability or appropriateness for answering the research questions, (b) design fidelity or adequacy of all study procedures, (c) within-design consistency of all components and study strands, and (d) analytic adequacy of data analysis procedures for answering the research questions.

Similarly, Onwuegbuzie and Johnson (2006) connected four of the nine inference legitimation types to a mixed methods study design—sample integration, inside-outside, sequential, and conversion. Sample integration is the extent to which the relationship between quantitative and qualitative sampling procedures affects the quality of the meta-inferences. Inside-outside legitimation refers to the accuracy and appropriateness of a researcher’s utilization of insider and observer stances in qualitative and quantitative strands of the study. Sequential legitimation deals with the degree a specific sequence of the study strands can affect the quality of the meta-inferences. Conversion legitimation relates to the extent to which the data converted from one type to another (e.g., quantitizing qualitative data or qualitizing quantitative data) produces quality meta-inferences. Each of these legitimation types can serve as a quality evaluation criterion during a mixed methods study design and implementation.

Creswell and Plano Clark (2007, 2011) linked mixed methods study design and implementation characteristics and the purposes of data integration with specific threats that can compromise the validity of resulting meta-inferences. For example, at the level of data collection, threats can be caused by sample selection and size, research bias, faulty procedures, and wrong data sources in quantitative and qualitative study strands. At the level of data analysis, threats to validity may arise because of inadequate quantitative and qualitative data representation, inappropriate statistical analytical techniques, and choice of weak results for qualitative follow-up. At the level of data interpretation, quality of inferences may be threatened by not addressing contradictions or divergent findings, attending to one set of results at the expense of the other, switching the order of results interpretation in sequential designs, using the wrong integration strategy, and not discussing the results in the context of the mixed methods research questions. Creswell and Plano Clark (2007, 2011) emphasized the importance of observing these validity threats in a mixed methods study design and implementation, and offered solutions or strategies for minimizing validity threats at each design and implementation level. The strategies are aimed at helping researchers avoid inconsistent and wrongful conclusions caused by inadequate integration of quantitative and qualitative components within a mixed methods study.

Quality Assurance and Validity Issues in a Sequential QUAN → QUAL Mixed Methods Design

Drawing quality meta-inferences in sequential mixed methods designs, in which one study strand builds on another, may be particularly challenging because of a cumulative effect of inferences generated in each consecutive study strand. Failure to produce quality inferences in a preceding study strand may significantly affect the quality of the inferences produced in the next strand (Creswell & Plano Clark, 2011; Morse & Niehaus, 2009; Teddlie & Tashakkori, 2009). In addition to general design specifications related to sampling, data collection, and analysis, it is important to consider how the study strands are connected in sequential mixed methods designs (Creswell, Plano Clark, & Garrett, 2008; Ivankova, Creswell, & Stick, 2006). The decisions about what research aspects to emphasize at the connecting stage and what methodological strategies to use to proceed from one study strand to another may either compromise or increase the overall mixed methods study validity. Teddlie and Tashakkori (2009) underscored within-design consistency aimed at preserving the cohesiveness and the logical flow of the study strands as one of the inference quality aspects in sequential mixed methods designs.

In a sequential QUAN → QUAL mixed methods design, consisting of an initial quantitative strand and a subsequent qualitative strand, meta-inferences are developed based on initial quantitative examination of the research problem and subsequent more in-depth exploration of the quantitative results using qualitative methods (Creswell & Plano Clark, 2011; Morse & Niehaus, 2009; Teddlie & Tashakkori, 2009). The study strands typically are connected while choosing participants for qualitative follow-up interviews and/or observations to better understand the results from the initial statistical tests (Ivankova et al., 2006). Quantitative results also can inform the development of qualitative data collection protocols and shape the emergent qualitative research subquestions.

Potential validity threats in this type of design may be caused by selecting inappropriate individuals or sample sizes for the quantitative and qualitative data collection, selecting wrong individuals to follow-up with to help explain significant results, choosing weak quantitative results to follow-up qualitatively, interpreting the two sets of results in reverse order, and not adequately relating the stages or projects reciprocally in a multiphase study to each other (Creswell & Plano Clark, 2011). Oftentimes an emergent nature of a sequential QUAN → QUAL mixed methods design (Morse & Niehaus, 2009) may add obstacles to establishing the overall study quality, because the design of the subsequent study strand depends on the results from the preceding strand, for example, following up on unexpected findings.

Besides the general strategies to overcome threats to validity related to procedural issues outlined by Creswell and Plano Clark (2011), practical examples of how to implement quality criteria in designing and conducting a sequential QUAN → QUAL mixed methods design remain scarce. For example, based on a study of doctoral students’ persistence in a distance learning program, Ivankova et al. (2006) illustrated how a systematic selection of participants for a follow-up qualitative multiple-case study using a typical response to the initial quantitative survey can enhance the quality of the study conclusions. Using summed mean scores and the standard error of the mean, the researchers identified respondents from each of four distinct groups with the mean scores within one standard error of the mean. A maximal variation sampling strategy then was used to identify one participant per group bearing different dimensions of the same demographic characteristics. Applying this systematic case selection procedure for further in-depth exploration of the factors affecting doctoral students’ persistence in an online program allowed the researchers to solicit more representative views and develop more meaningful and credible meta-inferences grounded in the quantitative and qualitative inferences.

Papadimitriou, Ivankova, and Hurtado (in press) discussed the use of eight validity checks derived from mixed methods literature to alleviate potential validity threats to emergent meta-inferences prompted by the sequential implementation of the quantitative and qualitative components in a QUAN → QUAL mixed methods design. These validity checks relate to methodological procedures specifically focusing on the systematic use of methods, validation strategies, follow-up sample selection, choice of results for follow-up, interaction of the study components, and formulation of the meta-inferences.

This article adds to a limited body of practical illustrations of how quality criteria can be applied in designing and conducting a sequential QUAN → QUAL mixed methods study. The article presents a three-step process of securing the quality of the meta-inferences generated from the sequential employment of quantitative and qualitative methods in a study. It also provides strategies that can be utilized to achieve credible inferences and decrease potential validity threats related to a sequential QUAN → QUAL study design and implementation issues. A mixed methods study of graduate student engagement in learning applied research methods online is used to illustrate the methodological procedures and steps in the validation process.

Mixed Methods Study of Student Engagement With Learning Applied Research Methods Online

The illustrative study was conducted to understand how graduate students learn applied research methods online. In spite of the fact that many postsecondary institutions offer a number of online courses in research methods, limited knowledge exists of how students engage in learning applied research methods online, what teaching strategies they perceive as being effective, and how they value and apply such knowledge. Reportedly, an online learning environment provides students with new learning opportunities and facilitates maximum involvement by all participants (Beaudoin, 2006; Hofmann, 2002; Ivankova & Stick, 2007). At the same time, it creates challenges caused by the asynchronous nature of online courses, technological issues, and personally related factors including students’ learning style preferences (Kember, 1995). Additional challenges are imposed by the applied nature of research methods courses that are aimed at developing practical research skills and often require an instructor’s physical presence (Ivankova, 2010).

The purpose of this sequential QUAN → QUAL mixed methods study was to explore from students’ perspectives how they engaged in learning applied research methods in an online environment using quantitative survey and qualitative interviews with graduate (master’s and doctoral) students who successfully completed one or more online research methods courses from a research university in the southeastern United States. The study was guided by the following overarching or mixed research question (Plano Clark & Badiee, 2010): How do students engage in learning applied research methods online? Research questions that were addressed in the quantitative phase and then further elaborated on in the qualitative phase included the following: What teaching and learning strategies do students perceive as being effective in learning applied research methods online? How do students value and apply knowledge acquired online?

The study consisted of two sequential strands: a quantitative survey strand followed by a qualitative individual interview strand. The goal was to explore the problem both quantitatively and qualitatively to gain a deeper insight into the issue while ensuring the meta-inferences made were valid and well justified (Greene & Caracelli, 1997). The numeric data collected in the first, quantitative strand provided a general picture of how students engaged in learning applied research methods online and what teaching and learning strategies they perceived as being effective. Qualitative follow-up interviews with a subset of purposefully selected survey respondents sought the explanation of the quantitative trends and helped elaborate on the somewhat unexpected statistical results. In the final stage of the study, the quantitative and qualitative findings from the two strands were integrated to create meta-inferences with the purpose of providing more complete and insightful answers to the research questions. Figure 1 presents a visual diagram of the study design and flow of the procedures. Capitalized letters indicate the priority of the quantitative method in the study (Creswell & Plano Clark, 2011). A reverse arrow leading from the qualitative data analysis stage back to the quantitative data analysis stage and a forward arrow leading from the quantitative data analysis stage to the qualitative data collection stage indicate interaction between these study strands discussed further in the article.

Figure 1.

Study design and procedures visual model.

Online Research Methods Courses

Four semester-based online courses—Introduction to Qualitative Research, Advanced Qualitative Research, Survey Research Methods, and Mixed Methods Research—were developed to meet the distance students’ needs in learning applied research methods. Each of these three credit hour courses enrolls master’s and doctoral students and is taught via a Blackboard Learn platform once a year. These courses blend online methodological discussions with application of the studied methods to either a research design proposal for a dissertation or for a funded research study (Survey Research Methods and Mixed Methods Research) or a group or individual small scale research study (Introduction to Qualitative Research and Advanced Qualitative Research). The course contents evolve logically and follow the sequential steps inherent in designing and conducting a study within a specific research approach. The study procedures and findings are presented below.

Strand I: Quantitative

The goal of the first, quantitative study strand was to identify teaching and learning strategies students perceived as being effective in learning applied research methods online and the perceived value and applicability of acquired knowledge.

Data Collection and Analysis

Students’ perspectives were explored by surveying 141 current and former students who had successfully completed at least one of the four online graduate research methods courses, described above, between spring 2005 and fall 2009, and taught by the principal investigator. One hundred and nineteen students (84.5%) responded to the survey administered online via Survey Monkey. The questionnaire scales and items were developed based on the themes and categories revealed in the literature on student engagement in online courses, items from Community of Inquiry Questionnaire (Arbaugh et al., 2008), and the instructor’s personal experience of teaching applied research methods online. In addition to 14 demographic questions, the questionnaire featured 10 Likert-type scales and five categorical items, addressing comfort level with an online learning environment, teaching and learning strategies, and evaluation and application of gained knowledge, and four open-ended questions seeking more in-depth information about students’ online experiences. The survey data were analyzed using descriptive statistics for trends and patterns of the respondents’ views, analysis of variances for differences among student categories, and multiple regression with forward selection to identify the best predictors of perceived effectiveness of learning research methods online based on two measures (single item score vs. composite score), gained knowledge and its application based on two teaching strategies (instructors’ presence and course design), and five learning strategies (reading course materials, writing module responses, discussing, applying course content, and time management). SPSS version 19 was employed. Table 1 presents a list of quantitative variables and measures used in the analysis and corresponding survey questions.

Table 1.

Strand I Variables and Measures.

Variables	Measures	Survey Questions
Dependent variables (DV)
DV1: Effectiveness of learning research methods online (2 measures)	Likert-type 5-point scale (Strongly disagree to Strongly agree) Single item Composite variable	Research methods can be learned as effectively online as in a face-to-face course. Learning online gives more time to process the information. Discussing course concepts in online threaded discussions helps understand them better. The requirement to support responses by course readings enhances learning. Providing written scholarly responses in answer to module questions enhances learning. Sharing different project components with other course participants and instructor enhances learning. Being able to receive immediate feedback from other course participants and instructor enhances learning. Providing constructive feedback to other course participants’ research project components enhances learning. Learning at ones’ own pace but within the specified time frame enhances learning. Being exposed to different course participants’ perspectives on the same question enhances learning. Being able to see application of the research method to different course participants’ projects across disciplines enhances learning.
DV2: Knowledge about research methods	Excellent/good/average/poor/very poor	How would you rate your knowledge about research methods after taking your most recent online research methods course?
DV3: Ability to apply research procedures	Excellent/good/average/poor/very poor	How would you rate your ability to apply research procedures learned in your most recent online research methods course?
Independent variables (IV)
Teaching strategies
IV1: Course design	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	The course design is efficient for learning research methods online. The interactive nature of the online course enhanced my learning. The right balance between methodological discussions and application of the method to a research project enhanced my learning. The course Blackboard layout is efficient for learning research methods online. Clear course objectives enhanced my learning. Clear and detailed course syllabus enhanced my learning. The structure and content of the course assignments enhanced my learning. The sequence of the course assignments enhanced my learning. The applied nature of research project related assignments enhanced my learning. The applied nature of the course research project enhanced my learning. Clear guidelines for learning activities enhanced my learning. Clear due dates/time frames for learning activities enhanced my learning. Clear grading guidelines on the course assignments and research project enhanced my learning. The requirement for active participation in online threaded discussions enhanced my learning. The fact that I can read instructor’s feedback to other course participants’ research project application assignments enhanced my learning. The requirement to evaluate other course participants’ pre-final research project drafts enhanced my learning. Critiquing and discussing exemplar published research studies enhanced my learning.
IV2: Instructor’s presence	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	The way instructor helped facilitate course discussions enhanced my learning. The way instructor provided regular feedback to research project application assignments enhanced my learning. The way instructor helped in guiding the class toward understanding and applying course concepts enhanced my learning. The way instructor provided timely responses to students’ questions enhanced my learning. The way instructor helped keep course participants engaged and participating in productive discussions enhanced my learning. The way instructor helped keep students on task enhanced my learning. The way instructor helped clarify issues emerged in discussion enhanced my learning. The way instructor encouraged course participants to explore new application of the method enhanced my learning. The way instructor monitored the course enhanced my learning. The way instructor reinforced the development of a sense of community among course participants enhanced my learning.
Learning strategies
IV3: Reading course materials	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	When reading for the course I make up questions to help focus my reading. When reading for the course I use only provided course materials. When reading for the course I discuss the information with someone outside my online class. I treat the course materials as a starting point and try to develop my own ideas about the information discussed. When reading about research procedures I refer to provided research examples. When reading about research procedures I search for more research examples.
IV4: Writing module responses	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	When writing responses in answer to module questions I use only course readings. When writing responses in answer to module questions I search Internet for additional resources. When writing responses in answer to module questions I first read all the assigned readings and then choose a question to respond to. When writing responses in answer to module questions I first read other course participants’ postings and then choose a question to respond to. When writing responses in answer to module questions I try to support my viewpoint citing the course readings. When writing responses in answer to module questions I try to critically reflect on the issue discussed.
IV5: Discussing	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	When responding to other course participants’ postings I use only course readings. When responding to other course participants’ postings I search Internet for additional resources. Before responding to another course participant’s posting I take time to reflect on its content. I try to respond to more course participants than specified in the course requirements. I read all the assignment responses posted by other course participants. I try to sustain online discussions on module questions. I try to always respond to other course participants’ and Instructor’s comments on my postings.
IV6: Applying course content	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	When reading about a specific research procedure I try to remember its application in research studies I read. When reading about a specific research procedure I reflect on how I will apply it to my project. When reflecting about application of research procedures to my project I evaluate how it will fit the research problem I want to explore. When reflecting about application of research procedures to my project I revisit the study purpose and research questions. When writing about application of research procedures to my project I try to model it to provided research examples. When writing about application of a specific research procedure to my project I first try to see how it was applied in other course participants’ research projects.
IV7: Time management	Likert-type 5-point scale (Strongly disagree to Strongly agree) Composite variable	I manage my study time for the course well. I make sure that I keep up with the weekly readings and assignments for the course. I can prioritize learning online and other responsibilities well. I can balance reading and writing well. I can balance participating in online threaded discussions and other course assignments well. Logging into the course at least every other day helps me stay with the course. I don’t wait till the last day of the module to respond to fellow-students’ postings.

Results

The study participants ranged between 25 and 61 years of age with the mean of around 40. The majority were female (73%); 65% were White and 33% Black. Seventeen percent were students from other U.S. and international universities. Half of the respondents completed only the first course in the sequence (Introduction to Qualitative Research), whereas the rest of the respondents either completed one course at a more advanced level or two or more courses. Eighteen percent of the participants completed two or more courses. The majority of the students (76%) were enrolled in doctoral programs at the time of taking the online courses.

The majority of the respondents (88%) indicated they were comfortable learning in an online environment; 89% rated knowledge of research methods acquired online as good and excellent, and 84% positively rated the ability to apply their acquired knowledge to research projects. Students who completed two or more online courses tended to be more comfortable learning research methods online, F(2, 116) = 3.674, p = .028). Students who completed three online research methods courses tended to perceive learning applied research methods in an online environment more effective than students who completed one or two online courses, F(2, 116) = 3.291, p = .041. There was a statistically significant difference in claimed learned knowledge about research methods acquired via an online format between students who completed one online course versus those who completed two or more courses, F(2, 116) = 5.162, p = .007. Similarly, students who completed two or more research methods courses online perceived that they could apply acquired knowledge better than did students who completed only one course, F(2, 116) = 7.960, p = .001.

The results of multiple regression indicated that among teaching strategies only course design was deemed to be an important factor in predicting students’ perceived effectiveness of learning research methods online: course design accounted for 64% of variance, R² = .638, F(1, 117) = 205.96, p = .000. Among learning strategies only personal time management and immediate or subsequent application of course content in relevant contexts emerged as important factors in predicting students’ perceived effectiveness of learning research methods online: Application of course materials and time management accounted for 30% of variance, R² = .301, F(2, 116) = 205.01, p = .000.

Strand II: Qualitative

The goal of the follow-up, qualitative study strand was to elaborate on the survey quantitative results in more depth and obtain more detailed understanding of how students engaged in learning applied research methods online and how they applied their acquired knowledge.

Data Collection and Analysis

Individual open-ended phone interviews were conducted with 13 purposefully selected individuals, who responded to the survey in Strand I. Interviews were conducted by the research assistant to decrease a potential bias and social desirability in responses. Interview protocol was grounded in the content of the survey items and consisted of 12 questions and corresponding elaborative probes that sought further understanding of the role of each teaching and learning strategy in students’ learning applied research methods online (see Table 2 for a list of interview questions).

Table 2.

Strand II Interview Questions.

Interview questions	Corresponding probes
1. Please tell me about yourself.	What program are you in or have graduated from? What are your degree objectives? If graduated, tell me about what you are doing now. What online courses in research methods did you take while studying at this university? What other online courses did you take while studying in this program? Why did you choose to take a research methods course online?
2. Please describe how you learned research methods in an online course.	What reading/writing/ discussion/application strategies did you use to learn research methods online? How did you use these strategies? What strategies do you think were the most effective and why? What strategies do you think were the least effective and why?
3. What course components did you find most effective to learn research methods online and why?	How useful was the content of the course? How useful were the assignments you were asked to complete during the course? What assignments do you think were the most effective and why? What assignments do you think were the least effective and why? How useful was the project/project proposal that you had to complete in the course? What course components were the easiest to complete in an online environment? What course components were the most difficult to complete in an online environment?
4. Please describe how the design of the course affected your learning of research methods online.	What do you think about the layout of the online course? What specific features of the course design helped you learn research methods online? What specific features of the course design impede you from learning research methods online effectively? How would you want the course to be structured?
5. Please describe the role of the course instructor in your learning research methods online.	How do you see the role of the instructor in a research methods online course? What instructor role would you prefer and why: facilitator or lecturer? How would you describe instructor’s presence in the online course?
6. Please describe how you managed time while taking research methods online course.	How did you balance the work in the course and other responsibilities (family, work, and other courses)? How did you balance reading and writing assignments? How did you decide when and for how long to participate in online discussions? How did you decide whose posting to respond to? What specific strategies did you use of organizing yourself in this online course? How different were these strategies from the ones you use when taking a face-to-face class? What strategies were the most effective and why? What strategies were the least effective and why?
7. What was the most important factor that helped you learn research methods in an online course and why?	How did this factor help you learn research methods in an online course?
8. What was the least important factor that helped you learn research methods in an online course and why?	How did this factor impede your learning research methods in an online course?
9. Do you believe an online community of learners emerged in the course? What do you think was its role in learning research methods online?	How do you define online community of learners? How was the online community created? How do you envision an online community in the research methods course? How different would it be from the community of learners that emerge in face-to-face courses? How did the online community help you learn research methods online?
10. How do you value the knowledge of research methods acquired in the online course?	How would you rate the knowledge of research methods acquired online? What aspect of this knowledge do you value most? How do you know that you learned?
11. How do you apply the knowledge of research methods acquired in the online course?	How did you use knowledge received in the online research methods course? What research projects did you apply this knowledge to? How do you plan to apply this knowledge in the future? How comfortable are you applying knowledge received in the online research methods course?
12. Is there anything else that you would like to add about learning research methods online?

The telephone interviews ranged from 30 to 40 minutes and were recorded and transcribed verbatim. Follow-up email requests to elaborate on or explain certain responses were sent to each participant after careful reading of each interview transcript by the principal investigator and the research assistant. Both the principal investigator and the research assistant kept reflective notes for triangulation purposes during the initial reading of the transcripts and data analysis. Interview transcripts and email responses were analyzed using standard thematic analysis procedures to generate codes, themes, and subthemes. The analysis employed a constant comparison method (Glaser & Strauss, 1967) and followed a zigzag process of data collection and analysis to achieve saturation of identified themes and subthemes (Creswell, 2012). Nvivo 9 software was used to assist with data management and analysis.

Findings

Students pursuing doctoral degrees in various major areas of study showed more engagement with learning research methods online with the purpose of being able to apply them to their dissertation studies than did the students enrolled in programs leading to a Master’s Degree or Education Specialist. The latter students generally completed just the first introductory qualitative course as a requirement for graduation. Doctoral students were more committed to learning research wanting to become competent in different research approaches to be able to complete a dissertation study. They felt more prepared for the rigor of research methods courses because of previous course work and were more comfortable learning research methods via online instruction. A study participant observed,

One of the things that helped me out, when I started taking research methods courses online, I had already taken several research methods courses in the classroom, so I had already developed my way to learn research methods and my way to approach it and whatever strategies worked for me.

The number and nature of learning strategies also varied across the interviewed participants depending on the degree pursued and number of prior online courses taken. Doctoral students discussed using more varied strategies that helped them master studied methodological concepts and procedures. A doctoral student stated,

To summarize the strategies that best helped me learn research methods online were (a) reading the course literature in depth, (b) answering questions that helped me understand design issues and skills that applied to my research, and (c) working with classmates that had similar research interests.

Though all interviewed participants described the knowledge of research methods acquired online as valuable and useful, doctoral students emphasized the application of course content to their doctoral and future funded research projects. Students pursuing Master’s and Educational Specialist Degrees discussed how they would apply such knowledge in their professional lives.

Among learning strategies, time management was confirmed as a major factor in learning research methods online and that fact was similar to other online courses that required discipline and organization on the part of students. Doctoral students seemed to place more emphasis on this strategy and perceived effectiveness of learning research methods online through their ability to stay focused and organized. A doctoral student explained,

I am a planner and set time aside at the beginning of the course. I planned how much time I would need to complete the questions and projects. I had daily and weekly goals of what I wanted to finish.

At the same time, doctoral students dominated the quantitative sample, which might have accounted for the statistical significance of time management compared to other learning strategy variables. Other strategies that were identified through the interviews included setting priorities, balancing, working ahead, trying not to fall behind, keeping calendars, and meeting deadlines.

The ability to apply the mastered concepts and procedures to class projects, either in the form of a mini-research study, or a research proposal was another important learning strategy. All interviewed participants stressed the value of practical application of the acquired skills culminating in the feeling of being comfortable understanding and conducting research, or using the skills in the workplace. A study participant observed,

As I did my own personal research I found it very useful, but as I am now reading other research papers or projects it makes it much more understandable and I can see what the other person is going through, and what they are trying to accomplish in their presentation of their material based on their research and their description of their research methodology.

Similarly, course design was viewed as an essential factor in effective learning of research methods online. All students viewed the design and layout of the courses as an effective pedagogical tool aimed at enhancing their ability to learn. A student observed, “I guess the layout of the course, I guess, first of all, it tells you your process of learning.” The participants highlighted the importance of logical sequence of topics, balance between discussion and project application assignments, and the ability to view and comment on other students’ work. They often referred to the course components as “building blocks” that allowed them to build knowledge about research and develop research skills in a logical manner. In the participants’ views, the online courses that were well-organized and balanced, provided scaffolding of information, and allowed students making connections between other courses and their professional activities were deemed to be effective and facilitated the learning process.

In the first, quantitative study strand, the instructor’s presence was not found to be a statistically significant predictor of effectiveness of learning research methods online. The follow-up interviews revealed that students viewed the instructor’s role in an online course differently than in a traditional classroom. Importantly, the instructor was perceived as being responsible for an effective online course design and timely delivery of information, and thus was blended with the course design. A study participant summarized, “The instructor is the course.” Students tended to view the instructor as a facilitator of learning in a structured online environment via elaborated course design, reading resources, choice of assignments, course monitoring, and continuous feedback to application assignments related to research projects. The instructor also was deemed to be responsible for keeping students accountable by providing regular, substantive, and timely feedback. A student explained,

She was very much a facilitator. . . .She laid out the requirement well in the syllabus at the beginning of the class, everybody knew what was expected, and gave the assignments, and then she would give encouragement and feedback as we went along through the course to help us refine our understanding, which was very good.

The interviewees emphasized that differences in the learning styles were important to consider in online learning; the latter affected students’ ability and willingness to learn and influenced the transfer of the responsibility for learning in an online environment from instructor to students. Such views accounted for some negative views on learning research methods online expressed by some survey respondents in Strand I. A study participant stated, “I think, really, to fully take advantage of learning research methods online, I think a lot of it does still come down to the way that the individual learns.” Interestingly, the shift from instructor’s teaching to student learning that occurs online was differently appreciated by students from different degrees and majors. For some students pursuing a Master’s or Education Specialist Degree, and who completed only one required online course, and who were not interested in pursuing careers in research, the learner-centered approach was associated with extra work and sometimes unnecessary learning activities. Conversely, doctoral students welcomed the rigor of online coursework and the emphasis on the development of research skills through weekly project-related assignments. They also valued the opportunity to learn maximally from other students in the course that an interactive online environment offered. A doctoral student explained, “In our online community we had someone from England and someone from Nebraska. So this is different than what you would have in a face-to-face class. It makes for a diverse group to work with and learn from.”

Implementing Quality Criteria in Designing and Conducting the Study

A three-step process of securing the quality of the meta-inferences generated from the sequential employment of quantitative and qualitative methods in the study was used. First, consistent with the recommendations from mixed methods literature for securing the quality of the meta-inferences in a mixed methods study (Bryman et al., 2008; Creswell & Plano Clark, 2011; Dellinger & Leech, 2007; Greene, 2007; Onwuegbuzie & Johnson, 2006; Teddlie & Tashakkori, 2009), separate procedures were used to assess the reliability and validity of the quantitative data and results and the credibility and trustworthiness of the qualitative data and findings. Next, to ensure the meta-inferences based on the integrated conclusions from the quantitative and qualitative findings were credible, additional strategies specific to a sequential QUAN→QUAL mixed methods design were employed: applying a systematic procedure for selecting participants for qualitative follow-up, elaborating on unexpected quantitative results, and observing interaction between qualitative and quantitative study strands.

Reliability and Validity of Quantitative Data and Results

Reliability and validity of the survey data was assessed using regular psychometric procedures. Content validity of the self-developed survey instrument was established through an expert panel consisting of instructors teaching online and research methods at different schools within the university (Nunnally & Bernstein, 1994). The survey instrument was pilot tested on 15 former graduate students who took at least one of the applied research methods online courses discussed above. After the survey administration, Cronbach’s alpha values were used to estimate internal consistency reliability of the survey scales and items (Thorndike & Thorndike-Christ, 2011). Cronbach’s alpha coefficients for 10 scales of Likert-type fixed-response items ranged from .581 for the Writing scale to .961for the Instructor’s Presence scale with 95% of the coefficients at .70 or greater. Exploratory factor analysis was not performed because of the small sample size in the study.

Credibility and Trustworthiness of Qualitative Data and Findings

Per Creswell (2007) recommendation, several strategies were used to ensure credibility and trustworthiness of the qualitative data and findings. Triangulation of data sources (interviews, follow-up emails, and researchers’ notes at the initial reading of the transcripts and during data analysis) was used to secure corroborating evidence in the data. Both the principal investigator and the research assistant analyzed the interview and follow-up email data to ensure consistency in coding and thematic categorization. Intercoder agreement was assessed at 94% on themes and 89% on codes (Miles & Huberman, 1994). The procedure involved first independent coding of the transcripts and then comparing and discussing codes and themes until agreement was reached. Member checking was conducted to ensure accuracy of the recorded data and presentation of participants’ views (Lincoln & Guba, 1985). Each participant received an extended summary of the interview with the request to verify its accuracy. Although most confirmed the accuracy of the recording, some participants used these summaries to further clarify their views and the meaning of some comments made during the interview.

Validation Strategies Specific to a Sequential QUAN → QUAL Mixed Methods Design

The following strategies grounded in the QUAN → QUAL design features were used to ensure the integrated conclusions from sequentially generated quantitative and qualitative findings were credible: applying a systematic procedure for selecting participants for qualitative follow-up, elaborating on unexpected quantitative results, and observing interaction between qualitative and quantitative study strands informing different aspects of quantitative and qualitative data collection and analysis process.

Selecting Participants for Qualitative Follow-Up

The participants for follow-up qualitative interviews were selected from those who completed the survey in Strand I using a systematic procedure. Creswell and Plano Clark (2011) argued that because the purpose of the follow-up qualitative strand in this design is to explain the initial quantitative results, same respondents should participate in both study strands. Selecting new participants may result in divergent views and cause inconsistencies in the inferences derived from the analysis of the quantitative and qualitative data (Teddlie & Tashakkori, 2009). To systematically select the participants a matrix of all respondents’ scores on three dependent variables (perceived learning effectiveness [two measures], perceived acquired knowledge, and knowledge application) was developed. The goal was to look for consistency of students’ responses on these measures and identify contradictory patterns. For instance, a student may have scored high or low on all four measures meaning that a student either highly rated learning effectiveness, acquired knowledge, and application of knowledge of research methods mastered online, or conversely, did not benefit from this mode of learning, and assigned low scores to all measures. In another instance, a student highly rated acquired knowledge and knowledge application, but rated learning effectiveness as average.

All survey respondents were categorized by the number and combination of courses taken. Using Explore procedure in SPSS version 19, typical, extreme, and unique cases were identified within each category of respondents based on consistency of their scores on four dependent measures. A maximal variation procedure was used to further select participants who varied on demographic variables (degree pursued, major, home institution) when more than one case in a certain category was available. This procedure yielded 13 participants who represented different combination of demographic characteristics and views on their experiences with learning applied research methods online. This approach to sampling allowed capturing variations in the sample and ensured that the chosen participants were the “best informants” who could provide the most accurate information about the experiences tested in Strand I (Patton, 2001). All selected individuals agreed to participate in follow-up interviews. Table 3 presents characteristics and scores on four dependent measures for 13 selected participants with assigned anonymous names.

Table 3.

Strand II Participants.

Name	Courses taken	Degree pursued	Major	University	Effectiveness (1)^a	Effectiveness (2)	Knowledge	Application
Typical case
Rachel^b	1—Intro Qualitative	Education Specialist	Educational Leadership	Home	4	4.8	4	4
Julie	1—Intro Qualitative	Masters	School Counseling	Home	4	4.6	4	4
Tina	1—Intro Qualitative	Education Specialist	Curriculum & Instruction	Home	5	4.9	5	5
Ralph	1—Advanced Qualitative	Doctorate	Educational Leadership	Home	4	4.6	5	5
Teresa	1—Mixed Methods	Doctorate	Educational Leadership	Home	5	5.0	5	5
Cathy	2—Survey, Mixed Methods	Doctorate	Health Behavior	Home	5	4.0	5	5
Jill	2—Intro Qualitative, Survey	Doctorate	Health Education/Promotion	Home	4	3.8	4	4
Robert	2—Intro Qualitative, Mixed Methods	Doctorate	Educational Leadership	Other	5	4.8	4	5
Laura	3—Intro Qualitative, Survey, Mixed Methods	Doctorate	Educational Leadership	Other	5	5.0	4	5
Extreme case
Tammi	1—Intro Qualitative	Doctorate	Curriculum & Instruction	Home	1	2.2	3	3
Unique (inconsistent response) case
Martha	1—Intro Qualitative	Doctorate	Health Education/Promotion	Home	1	3.6	4	4
Cheryl	1—Mixed Methods	Doctorate	Health Behavior	Home	3	3.4	5	5
Craig	2—Intro Qualitative, Survey	Doctorate	Health Education/Promotion	Home	1	4.2	4	5

See Table 1 for variable measurement scales.

Assigned anonymous names.

Elaborating on Unexpected Quantitative Results

Another purpose of a sequential QUAN → QUAL mixed methods design is to seek explanation of unexpected results obtained in the first, quantitative study strand (Morse & Niehaus, 2009). According to Creswell and Plano Clark (2011), choosing weak quantitative results to follow-up on using qualitative methods constitutes a potential validity threat and can compromise the meta-inferences generated from the two study strands. To ensure correct interpretations of the quantitative survey results, the researcher chose to explore the reasons for inconsistent and extreme scores on four dependent measures (perceived learning effectiveness, acquired knowledge, and knowledge application), as well as surprising or unexpected results via individual interviews with systematically selected survey respondents, as discussed in the section above.

For example, it was surprising that several participants inconsistently rated knowledge of research methods acquired online and the ability to apply this knowledge as high, while at the same time rated effectiveness of learning research methods online as average (see Table 3). Also there were inconsistencies in the scores on two measures of perceived learning effectiveness for some of these respondents. Qualitative interviews helped understand the inconsistencies in these ratings. Interviews revealed that these participants were not satisfied with the degree and quality of other students’ participation in online discussions, thus contributing to their negative online learning experiences. For the participants who completed the Introduction into Qualitative Research course, additional negative perceptions were associated with low commitment of some course participants to a team research project and the resulting burden of having to do extra work to complete the project on time. Negative team project experience was the only explanation of negative perceptions of learning research methods online for one extreme case participant.

Another unexpected statistical finding that prompted further exploration was the fact that the instructor’s presence was found to be a nonsignificant predictor of learning effectiveness in Strand I. This nonsignificant result contradicted previous research showing the important role that is attributed to the instructor’s presence in student learning process, particularly at the undergraduate level. Admittedly, these were graduate-level courses. The follow-up interviews revealed that students highly regarded the instructor’s facilitator role in the online course, but the instructor’s presence was blended with the course design and delivery. At the same time, the learner-centered approach in online teaching and learning was differently appreciated by students from different degrees and majors. Thus, carefully selecting the results from Strand I to follow-up qualitatively helped ensure within-design consistency that Teddlie and Tashakkori (2009) referred to as one of the inference quality aspects in sequential mixed methods designs.

Observing Interaction Between Qualitative and Quantitative Study Strands (QUAN ↔ QUAL)

In spite of obvious linearity of a sequential QUAN → QUAL mixed methods design, research practice shows that two sequential phases may interact in a nonlinear fashion. In addition to elaborating on initial quantitative results, qualitative findings may reveal the need for additional statistical examination of the quantitative data, which, in turn, may help better understand these qualitative results. Greene (2007) referred to such a design as integrated mixed methods design, while Teddlie and Tashakkori (2009) described it under the family of fully integrated mixed designs, in which methods integration occurs in an interactive manner at all stages of the study. The interactive process of the quantitative and qualitative study components informing each other in sequential mixed methods designs can be viewed as a means of ensuring consistency between the study strands and help achieve interpretative rigor (Teddlie & Tashakkori, 2009) of the study conclusions, thus further improving the quality of the meta-inferences generated by integrating quantitative and qualitative results.

In the illustrative study, the ongoing analysis of the interview data revealed the need for conducting additional statistical analysis of the survey data to confirm or disconfirm emergent qualitative findings. Alternatively, the results from additional statistical analysis informed further qualitative data collection and prompted several new interview questions to enhance further understanding of participants’ experiences and enriching interpretations from both data sets. For example, students’ comfort level with learning in an online environment initially was analyzed using descriptive statistics, showing the dominant majority of the respondents (88%) were comfortable learning online. Although this quantitative indicator was initially considered sufficient for the study purposes, after conducting and analyzing one third of interviews in Strand II, it became apparent that participants who had taken more research methods courses online had more positive comments about their experiences and tended to discuss them in more detail. To seek corroborating evidence, this naturalistic observation was tested by comparing groups of respondents who took one online course versus two or three courses. Analysis of variance with the Tukey post hoc test revealed that students who took two or more online courses were more comfortable learning online, F(2, 116) = 3.674, p = .028). Thus, the results from the statistical analysis confirmed naturalistic observations that were emerging from the analysis of the qualitative data.

Similarly, during Strand I, the statistical analysis focused only on the predictive power of overall effectiveness of learning research methods online. The contribution of individual online course components, for example, assignments, readings, threaded discussions, research project, and so on, was not assessed deemed less reliable because of a small study sample size. During the interviews, however, the participants talked more about different course components and their role in mastering applied research methods online than about overall learning effectiveness. This observation prompted revisiting the survey data and conducting correlation analysis to test for the relationship between individual course components and three dependent variables (perceived learning effectiveness [two measures], perceived acquired knowledge, and knowledge application). The results indicated that there was a low and moderate, however a statistically significant correlation between learning effectiveness and all 17 course components on the first measure, and 15 course components on the second measure (p < .01; see Table 4). There was also a significant correlation between most of the course components and acquired knowledge, and knowledge application (p < .05) with the exception of required texts, evaluation of other students’ research projects, and producing reading summaries and book reports. The results from the correlation analyses were used to develop additional interview questions and probes that targeted the role of each course component in learning research methods online in subsequent interviews with other participants and follow-up email clarifications.

Table 4.

Strand I Correlations of Course Components With Dependent Variables.

Course components	Effectiveness of learning research methods online (Measure 1)	Effectiveness of learning research methods online (Measure 2)	Knowledge about research methods	Ability to apply research procedures
Course topic module structure	.321**	.154	.304**	.363**
	.000	.095	.001	.000
	119	119	119	119
Content related assignments	.399**	.298**	.316**	.305**
	.000	.001	.000	.001
	119	119	119	119
Project related assignments	.577**	.333**	.318**	.380**
	.000	.000	.000	.000
	119	119	119	119
Required texts	.289**	.219*	.161	.148
	.001	.017	.080	.107
	119	119	119	119
Additional readings	.349**	.328**	.197*	.272**
	.000	.000	.032	.003
	119	119	119	119
Online threaded discussions	.650**	.443**	.184*	.313**
	.000	.000	.045	.001
	119	119	119	119
Fellow-students’ written feedback	.446**	.276**	.157	.235*
	.000	.002	.088	.010
	119	119	119	119
Instructor’s written feedback	.422**	.261**	.299**	.309**
	.000	.004	.001	.001
	119	119	119	119
Research project or project proposal	.530**	.259**	.308**	.284**
	.000	.004	.001	.002
	119	119	119	119
Research project guidelines	.557**	.291**	.380**	.343**
	.000	.001	.000	.000
	119	119	119	119
Research project grading rubric	.568**	.305**	.240**	.267**
	.000	.001	.009	.003
	119	119	119	119
Examples of other students’ research projects	.424**	.254**	.245**	.203*
	.000	.005	.007	.027
	119	119	119	119
Evaluation of students’ research projects	.617**	.366**	.177	.160
	.000	.000	.055	.082
	119	119	119	119
Research article critique	.391**	.250**	.226*	.191*
	.000	.006	.014	.037
	119	119	119	119
Locating research studies on the topic of interest	.365**	.313**	.187*	.170
	.000	.001	.042	.065
	119	119	119	119
Reading summary or book reports	.206*	.156	−.019	.069
	.024	.090	.836	.459
	119	119	119	119
Summarizing discussion on a module question	.323**	.200*	.211*	.290**
	.000	.030	.021	.001
	119	119	119	119

Note. The values in the three rows for each variable indicate the Pearson correlation, significance (two-tailed), and N, respectively.

Correlation is significant at the 0.05 level (2-tailed). **Correlation is significant at the 0.01 level (2-tailed).

Finally, in Strand I, time management along with application of course materials were found to be the only successful learning strategies that explained 30% of variance in students’ perceived effectiveness of learning research methods online. During the interviews in Strand II, time management strategies emerged as having varied perceived effect across participants from different degrees and programs (e.g., “staying on top of things,”“working ahead,” etc.). Doctoral students from the programs that required more research methods courses tended to have more elaborate and effective time management techniques. This qualitative finding prompted to perform a subsequent correlation analysis to check for the relationship between time management and a number of face-to-face research methods courses taken (r = .37, p = .42), prior experience with learning online (r = .83, p < .05), and hours per week spent on most recent online research methods course (r = .85, p < .05). Although time management was not significantly related to prior research methods course work, it was significantly correlated with prior online experience and time spent on the course. Based on this information, follow-up emails were sent to all Strand II participants asking them to explain how specific time management strategies revealed during their respective interviews enhanced their learning of applied research methods online, how those strategies were used in prior online courses, and how differently these strategies were applied in research methods courses. In this way, additional statistical analysis helped enhance further understanding of the qualitative findings and enrich their interpretations, thus providing additional validity checks to secure credible meta-inferences.

Conclusions

This article adds to the discussion about quality assurance in mixed methods research. Specifically, it deals with the issues related to implementing quality criteria in designing and conducting sequential QUAN → QUAL mixed methods design studies, in which meta-inferences are developed based on initial quantitative examination of the research problem and subsequent more in-depth exploration of the quantitative results using qualitative methods. An illustrative study of graduate student engagement with learning applied research methods online was used to empirically augment the discussion and to illustrate the presented procedures.

The article discussed a three-step process of securing the quality of the meta-inferences generated from the sequential employment of quantitative and qualitative methods in the study and offered several validation strategies specific to a sequential QUAN → QUAL mixed methods design: applying a systematic process for selecting participants for qualitative follow-up, elaborating on unexpected quantitative results, and observing interaction between qualitative and quantitative study strands. It is believed that the employed procedures helped secure the credibility of the integrated inferences from both study strands and supported the overall conclusions from the study: (a) course design is a prerequisite for effective online learning and should be connected with practical research experience and students’ interests; (b) instructors should be prepared to give the responsibility for learning to students while students should assume the role of active learners, which, in turn, requires a change in philosophy of and attitudes to learning; (c) students should be kept accountable for their learning and cultivate a team approach to learning and research; (d) accumulation of successful online learning experience generates a perception of increased effectiveness of learning applied research methods online; and (e) being able to apply research skills acquired online signifies effectiveness of learning applied research methods online.

Certain limitations inherent to the study design and implementation should be considered in the interpretation of the study conclusions and related methodological observations. The four online research methods courses were taught in one program and by one instructor, the principal investigator for this study. Convenience sampling was used in the quantitative study phase, which limits generalizability of the study findings. A self-developed survey instrument was used; although its reliability was relatively high, construct validity through factor analysis remains to be established. Recall bias may have slanted the results, as there was a time lag for those students who completed the courses earlier; besides, some study participants completed more than one research methods course from the same instructor both via online and conventional formats. Although prior experiences with online learning and previous coursework in research methods were assessed, these factors were not controlled for statistically. Finally, students who completed a doctoral degree could better assess their ability to apply research skills than master’s or education specialist students.

In spite of these limitations, it is believed that methodological observations and suggested steps and strategies aimed at ensuring the quality of the meta-inferences in a sequential QUAN → QUAL mixed methods design are reliable and useful. Further exploration of the ways to ensure quality in mixed methods research in general and mixed methods designs in particular is necessary, taking into account methodological conventions of specific mixed methods designs and paying particular attention to the integration of methods consistent with the purposes of each design. Identifying effective validation strategies grounded in mixed methods design and implementation features will give practicing researchers tools to answer the following question with confidence, “How do we know that our inferences are credible or believable, and not merely a function of our imagination?” (Teddlie & Tashakkori, 2010, p. 28).

Footnotes

Acknowledgements

I would like to extend my thanks to Dr. Marcia O’Neal for her assistance with the quantitative data collection for the illustrative study and to Dr. Corilyn Ott for her assistance with the qualitative study phase.

Author’s Note

The paper based on this article was presented as part of the symposium “Ensuring Quality in Mixed Methods Research in Higher Education: From Theory to Practice” at the annual meeting of the American Educational Research Association, Vancouver, British Columbia, Canada, April 13-17, 2012.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was sponsored by the UAB Office of Provost through Scholarship of Teaching Grant Program in February 2009.

References

Arbaugh

J. B.

Cleveland-Innes

Diaz

S. R.

Garrison

D. R.

Ice

Richardson

J. C.

Swan

K. P.

(2008). Developing a community of inquiry instrument: Testing a measure of the Community of Inquiry framework using a multi-institutional sample. Internet and Higher Education, 11, 133-136.

Beaudoin

M. F.

(Ed.). (2006). Perspectives on higher education in the digital age. New York, NY: Nova Science.

Bryman

Becker

Semptik

(2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11, 261-276.

Creswell

J. W.

(2007). Qualitative inquiry and research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: Sage.

Creswell

J. W.

(2010). Mapping the developing landscape of mixed methods research. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 45-68). Thousand Oaks, CA: Sage.

Creswell

J. W.

(2012). Educational research: Planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Upper Saddle River, NJ: Merrill Prentice Hall.

Creswell

J. W.

Plano Clark

V. L.

(2007). Designing and conducting mixed methods research. Thousand Oaks, CA: Sage.

Creswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.

Creswell

J. W.

Plano Clark

V. L.

Garrett

(2008). Methodological issues in conducting mixed methods research designs. In Bergman

M. M.

(Ed.), Advances in mixed methods research: Theories and applications (pp. 66-83). London, England: Sage.

10.

Dellinger

A. B.

Leech

N. L.

(2007). Toward a unified validation framework in mixed methods research. Journal of Mixed Methods Research, 1(4), 309-332.

11.

Glaser

Strauss

(1967). The discovery of grounded theory. Chicago, IL: Aldine.

12.

Greene

J. C.

(2007). Mixed methods in social inquiry. San Francisco, CA: Jossey-Bass.

13.

Greene

J. C.

Caracelli

V. J.

(Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse programs (New Directions for Evaluation, Vol. 74). San Francisco, CA: Jossey-Bass.

14.

Hofmann

D. W.

(2002). Internet-based distance learning in higher education. Tech Directions, 62(1), 28-32.

15.

Ivankova

N. V.

(2010). Teaching and learning mixed methods research in computer-mediated environment: Educational gains and challenges. International Journal of Multiple Research Approaches, 4(1), 49-65.

16.

Ivankova

N. V.

Creswell

J. W.

Stick

(2006). Using mixed methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3-20.

17.

Ivankova

Kawamura

(2010). Emerging trends in the utilization of integrated designs in social, behavioral, and health sciences. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 581-611). Thousand Oaks, CA: Sage.

18.

Ivankova

Stick

(2007). Online higher education programs: New challenges, new frontiers, new learning, and new scholarship. In Wright

Miller

(Eds.), Training higher education policy makers and leaders: A graduate program perspective (pp. 75-96). Charlotte, NC: Information Age.

19.

Kember

(1995). Open learning courses for adults: A model of student progress. Englewood Cliffs, NJ: Educational Technology.

20.

Leech

N. L.

Dellinger

A. B.

Brannagan

K. B.

Tanaka

(2010). Evaluating mixed research studies: A mixed methods approach. Journal of Mixed Methods Research, 4(1), 17-31.

21.

Lincoln

Y. S.

Guba

E. G.

(1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

22.

Maxwell

J. A.

Mittapalli

(2010). Realism as a stance for mixed methods research. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 145-167). Thousand Oaks, CA: Sage.

23.

Miles

M. B.

Huberman

A. M.

(1994). Qualitative data analysis: A sourcebook (2nd ed.). Thousand Oaks, CA: Sage.

24.

Morse

J. M.

Niehaus

(2009). Mixed method design: Principles and procedures. Walnut Creek, CA: Left Coast Press.

25.

Nunnally

J. C.

Bernstein

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

26.

O’Cathain

(2010). Assessing the quality of mixed methods research: Toward a comprehensive framework. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in the social & behavioral research (2nd ed., pp. 531-555). Thousand Oaks, CA: Sage.

27.

Onwuegbuzie

A. J.

Johnson

R. B.

(2006). The validity issue in mixed research. Research in the Schools, 13(1), 48-63.

28.

Papadimitriou

Ivankova

Hurtado

(in press). Addressing challenges of conducting quality mixed methods studies in higher education. In: Huisman

Tight

(Eds.), Theory and method in higher education research. Bingley, England: Emerald Group.

29.

Patton

M. Q.

(2001). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

30.

Plano Clark

Badiee

(2010). Research questions in mixed methods research. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 275-304). Thousand Oaks, CA: Sage.

31.

Tashakkori

Creswell

(2007). The new era of mixed methods. Journal of Mixed Methods Research, 1(1), 3-8.

32.

Tashakkori

Teddlie

(Eds.). (2003). Handbook on mixed methods in the behavioral and social sciences. Thousand Oaks, CA: Sage.

33.

Tashakkori

Teddlie

(Eds.). (2010). Sage handbook of mixed methods in social & behavioral research (2nd ed.). Thousand Oaks, CA: Sage.

34.

Teddlie

Tashakkori

(2009). Foundations of mixed methods research: Integrating quantitative and qualitative approaches in the social and behavioral sciences. Thousand Oaks, CA: Sage.

35.

Teddlie

Tashakkori

(2010). Overview of contemporary issues in mixed methods research. In Tashakkori

Teddlie

(Eds.), Sage handbook of mixed methods in social & behavioral research (2nd ed., pp. 1-41). Thousand Oaks, CA: Sage Publications.

36.

Thorndike

R. M.

Thorndike-Christ

T. N.

(2011). Measurement and evaluation in psychology and education (8th ed.). Upper Saddle River, NJ: Prentice Hall.