Abstract
Introduction
Designing and conducting an intervention study is an arduous task for students that requires strong teaching and mentoring practices from psychology instructors.
Statement of the Problem
Most research methods textbooks simply describe how an intervention is incorporated into a research study design (Johnson & Christensen, 2016), but rarely discuss steps on how to plan and develop an intervention study, leaving instructors without a guide to mentor students throughout the process.
Literature Review
Some research methods education instructors incorporate teaching techniques such as problem-based learning, “learning by doing,” or by incorporating online modules into their research methods curricula (Gurung, R. A. R., & Stoa, R. (2020). A national survey of teaching and learning research methods: Important concepts and faculty and student perspectives. Teaching of Psychology,
Teaching Implications
This article provides a summary of best practices from the literature and from our experiences in planning, developing, and implementing rigorous intervention studies utilizing a checklist teaching-tool intended for undergraduate- and graduate-level psychology instructors. Additionally, teaching tips on incorporating hands-on intervention activities and projects into coursework are provided (Butler, B. M., & Morrow, J. A. (2021). Developing and implementing an intervention study: Strategies for mentoring students throughout the research process supplementary material. Figshare).
Conclusion
This checklist can enhance the teaching and mentorship of students to ensure successful intervention studies of their own.
Introduction
Developing an intervention study (see Chaves et al., 2017 and Hargrave & Sénéchal, 2000 for examples of interventions) can be both an arduous and daunting task for a seasoned researcher, especially for an undergraduate or graduate student. An intervention study involves the development, implementation, and evaluation of a program designed to improve various parts of a program participant’s life (Johnson & Christensen, 2016). Oftentimes, the program involves multiple stages that participants progress through, which provides practical resources for participants to implement into practice. For example, we recently developed an intervention study designed to increase the perceived English academic writing skills of international graduate students through three interactive online modules. Participants were also encouraged to share their own resources and tips when writing academically throughout the modules with other participants. Through surveys and interviews, we measured participants’ perceived knowledge and confidence changes in their academic writing skills from before and after the intervention.
Although 97% of United States undergraduate psychology programs required a research methods course for degree completion in 2014 that explore topics such as interventions, only 65% of such courses included a lab component (Norcross et al., 2016). These statistics may indicate that a significant percentage of undergraduate students are not provided the opportunity to apply their knowledge in a hands-on environment about the research process (Gurung et al., 2016), despite many undergraduates who report that hands-on learning in a research methods course enhances their interest in research and their perceived level of preparation to conduct their own research (Roberts & Allen, 2012, 2013).
This literature on undergraduate research methods indicates that psychology undergraduate students may not fully understand the complexity of the process required to conduct an intervention-based project, especially if this is not the focus of any hands-on project requirement. Undergraduate students’ potential lack of a full understanding of interventions is especially troubling for psychology graduate programs, as incoming students may not have the fundamental knowledge of intervention planning and execution needed to conduct their own intervention studies. To address this pedagogical gap, this article will offer several tips for psychology undergraduate and graduate instructors and mentors (e.g., research methods course instructors, undergraduate honors thesis/graduate thesis, and dissertation advisors) to teach and mentor students how to plan, develop, and implement a successful intervention study. Additionally, a checklist for conducting a rigorous intervention study is provided to guide both undergraduate and graduate psychology students and their mentors throughout the intervention development process (Butler & Morrow, 2021; see article’s supplementary material on Figshare). These tips and teaching materials are based on best practices from the literature (e.g., An et al., 2006; Berg et al., 2014; Centers for Disease, 2019; Curry & Nunez-Smith, 2015; Hintz et al., 2014; McBride, 2016; See et al., 2015; Van Zyl et al., 2019; Woodside et al., 2015) as well as lessons learned from our own experiences as researchers recorded in an electronic diary of the successes and stumbling blocks we encountered when developing and implementing an intervention.
Research Methods Education for Undergraduate and Graduate Students
There are a limited number of articles that specifically discuss the learning objectives of research methods undergraduate or graduate courses in the behavioral and social sciences. A recent study (Gurung & Stoa, 2020) surveyed faculty to assess what content areas were most important to include in a research methods course. Gurung and Stoa (2020) found that the most important topic rated by faculty, out of their list of 25 topics, was the experimental approach, which is a foundation of many intervention studies. Stoa et al. (2020) emphasized in their study investigating students’ perceptions of challenges faced in research methods courses that the concepts of validity and quasi-experimental design were most difficult for students, concepts that are crucial to the understanding of intervention studies.
Even more troublesome is the lack of literature that aligns specific research methods’ learning objectives to effective teaching and mentoring strategies (e.g., Earley, 2014; Roberts & Allen, 2012; 2013; Wagner et al., 2011). This literature gap indicates inconsistency between the pedagogy and curricula of undergraduate and graduate psychology research methods courses. However, in a synthesis of what research methods education literature exists, some instructors incorporate teaching techniques such as problem-based learning, “learning by doing” through experiential learning or online learning modules into their research methods curricula (Earley, 2014; Gurung & Stoa, 2020). However, these projects vary from hands-on activities which promote deeper learning through individual/group projects where students collect their own data to secondary data analysis projects where students are not involved in the research planning process (Earley, 2014; Gurung et al., 2016).
This research indicates that additional focus is needed on promoting students’ procedural knowledge, or the understanding of how to plan and perform research methods, through the expansion of hands-on activities incorporated into research methods courses. One content area this misalignment is especially apparent in is with intervention studies. Most undergraduate and graduate research methods textbooks simply describe how an intervention is incorporated into a research study design (e.g., pre/post quasi-experimental design, randomized controlled trials; Johnson & Christensen, 2016), but rarely discuss the steps on how to plan an intervention study, how to develop the intervention materials, and methods of gathering feedback from experts to ensure the intervention materials are of high quality (Gertler et al., 2016; Johnson & Christensen, 2016; Maruyama & Ryan, 2014).
Instead, we challenge instructors to promote deeper student learning of intervention processes through “learning by doing” or experiential learning (Earley, 2014). In this pedagogical framework, students can work (in groups, if undergraduate students, or independently, if advanced undergraduate or graduate students) to plan, develop, and conduct an intervention of their own. Moreover, focusing on the mastery of students’ skillsets with conducting an intervention provides students with practical research design methods for “real-world settings” as opposed to a true experimental design that is difficult to conduct outside of a lab (Johnson & Christensen, 2016). For students that may never conduct an intervention study outside of a classroom, it is still imperative that they adequately understand the components that comprise a rigorous intervention, as many students will be consumers of intervention research in their future careers and need to adequately critique the rigor of the research designs utilized in published studies.
To promote deeper learning of intervention designs, we encourage instructors to have students utilize the intervention checklist (see article’s supplementary material on Figshare) for students to plan, develop, and implement an intervention study of their own (Butler & Morrow, 2021). The checklist is intended for the mentorship of advanced undergraduate students enrolled in a year-long course or project, such as a senior-level capstone course or throughout their undergraduate honors thesis (aspects of the checklist can be modified to appropriately fit undergraduate-level coursework), or for the mentorship of graduate students in a research methods course or throughout their thesis/dissertation. The checklist can also be modified to fit the time length of an undergraduate or graduate semester course (see Teaching Implications for additional details). In order to ensure that undergraduate and graduate students are provided guidance throughout the process of planning, developing, and implementing a rigorous intervention study of their own, the following sections will discuss teaching and mentoring tips for undergraduate and graduate instructors and advisors.
Best Practices in Planning Rigorous Interventions
When planning an intervention, students may communicate to their instructors or mentors that they have a novel idea for an intervention. Advise students to not “reinvent the wheel” and encourage them to perform a literature review on their topic of interest. Chances are there is at least one evidenced-based intervention study that is like their idea that they can model their intervention off of (Centers for Disease, 2019). Additionally, encourage students to perform a second literature review on best practices in the literature that relate to the specific intervention topic (Berg et al., 2014; Hintz et al., 2014), which allows for students to better understand the current findings in the field to incorporate into the intervention design and any gaps in knowledge that the intervention might seek to explore (McBride, 2016).
During this planning period, it is extremely helpful for students to have access to established professional contacts to reach out to at least three content experts to plan and review the intervention content (Curry & Nunez-Smith, 2015) and to obtain feedback from at least three individuals in the target audience, which are from the population that is the focus of the intervention (McBride, 2016), to better understand their needs and values. Mentors should assist students with seeking out and communicating with these contacts as well as encouraging students to develop their own professional contacts by developing a LinkedIn or a professional Twitter account and becoming active in their professional organizations’ social media platforms.
We would also advise instructors and mentors to guide their students to engage in rich conversations with at least three experts in the intervention topic area (Curry & Nunez-Smith, 2015) to indicate a potential “gap in policy, practice and research” (McBride, 2016, p. 15) that the intervention may seek to explore. Encourage students to develop talking points to structure these conversations, such as “What resources are currently available to address [the target audience]’s needs?” which will increase the richness of the conversations. Advise students to ask about the target audience’s typical schedules to ensure appropriate timing of the intervention to minimize disruptions to participants (e.g., planning intervention around students’ school schedule; McBride, 2016; See et al., 2015).
During conversations with content experts, coach students to gain the experts’ support for the intervention to start planning out several successful recruitment strategies (Curry & Nunez-Smith, 2015; See et al., 2015). From our experience, at least one recruitment method will fall through, so it is critical for students have several points of contact with the target audience. However, modifications to these conversations held, such as having students only reach out to one content expert instead of three to five experts, may be appropriate to ensure the project can be completed in the given timespan. For the mentorship of undergraduate students, this step of contacting content experts can be altered (e.g., having the instructor or teaching assistant lead this effort) or removed entirely, if deemed appropriate.
Once students have reached out to content experts and modified materials based on their suggestions, encourage students to reach out to members of both the intervention target audience and the secondary target group, or the population implementing the intervention (e.g., teachers), to incorporate their feedback into the design and content of the intervention to ensure its relevance to the population (Curry & Nunez-Smith, 2015; McBride, 2016; Nagy & Fawcett, 2019). If students have trouble directly contacting the target audience, then advise students to use indirect outlets such as social media or listservs targeted toward the population (McBride, 2016).
As students plan their intervention study, develop a contract between you and the student that details out the steps and timeline for implementing the intervention. Meet with students to review their timeline to detect any unrealistic plans that may not be feasible to conduct given the timeframe for the study. We discovered through our research that what looks like a small intervention on paper is quite large in reality; therefore, beginning with a manageable intervention and adding additional elements if necessary may be a wise strategy when mentoring students. Spend time during this planning phase to investigate if the intervention needs to be reviewed by an institutional review board (IRB) to ensure ethical treatment of participants, which is conducted if the intervention is considered research that will be shared beyond a classroom presentation. If needed, guide the student to include the steps taken to have the study reviewed by the IRB into the list of tasks to do before implementing the intervention.
Best Practices in Developing Rigorous Interventions
From the feedback gathered during conversations with the content experts and target audience members, in conjunction with information gathered from the literature reviews performed during the planning stage, students should have a draft of the content and methodology of the intervention completed. For the data collection instruments (e.g., surveys) in the intervention study, advise students to utilize materials found in the literature, if available, especially if they have been previously validated (See et al., 2015). During the development stage of the intervention, we found it useful to conduct a review with three to five content experts of the newly developed intervention to obtain feedback and suggestions for content modification. Additionally, we found it helpful to contact three to five assessment experts to provide feedback on drafts of the data collection tools to be used in the intervention. If students are working in a limited timeframe on the intervention, it may be appropriate to remove this step of contacting content and assessment experts. Alternatively, if students are collaborating in a group for an intervention project, have students volunteer or assign students to specific roles during the intervention planning process. Each group can have a “content expert liaison,” an “assessment expert liaison,” and a “target population liaison”. Splitting tasks ensures that students clearly understand what is expected of them and may minimize group conflict.
One aspect of the development stage of an intervention that almost all of the literature reviewed agreed upon (An et al., 2006; Berg et al., 2014; Curry & Nunez-Smith, 2015; Hintz et al., 2014; McBride, 2016; Van Zyl et al., 2019; Woodside et al., 2015) is to conduct a pilot-test study, which is a small-scale study (typically with three to five participants from the target audience) to test if all processes in the intervention occur as expected (Woodside et al., 2015). Meet with students to discuss how the pilot study went, which may indicate the feasibility of conducting a full-scale intervention and any logistical issues that may occur during the full-scale study (Woodside et al., 2015). Guide students to incorporate the feedback from the pilot study into full intervention study (An et al., 2006; Berg et al., 2014; Hintz et al., 2014), in addition to information gathered from content and assessment experts. If students have a brief timeframe to complete an intervention project (e.g., course project), it is recommended that conducting the pilot study is the last portion of the assignment. However, if this project is being completed as part of a student’s thesis or dissertation, then the implementation step, which is described in the next section, should be completed as well.
Best Practices in Implementing Rigorous Interventions
During the implementation stage, it is recommended to advise students to develop an intervention manual that details how the intervention is disseminated (and modified, if applicable) to ensure standardization. Also, in the manual, information on how data are collected should be described by the students in detail. Encourage students to include in the manual a protocol for potential problems that may occur throughout the study, such as steps that should be taken if technology fails during the intervention or a “Plan B” if unexpected events occur during data collection. Discussing with students their protocols and potential decisions for unplanned events can help minimize the potential impact of delay in the intervention study later (McBride, 2016).
During an intervention study, we found that communication with the participants was heightened compared to the amount of contact that we typically have with our non-intervention research participants. This heightened contact may be due to the many components that participants often complete throughout the intervention and thus requires several initial contacts with participants in addition to follow-up reminders about tasks to complete. Therefore, it is important to guide students to schedule into the project ample time to contact participants via a variety of methods that best fit the population of interest. Due to the many aspects that an intervention project often requires, we also found it extremely important to diligently keep track of which participants have completed what stages of the intervention/data collection tools through the use of a project database. Require students to keep detailed records to track which participants are exposed to and complete the sections of the intervention/data collection.
It is also imperative to have students ask participants throughout the intervention and at the conclusion of the intervention what they thought went well and what did not go well in the intervention (e.g., formative assessment). Advise students to reflect on participants’ evaluations of the intervention to suggest future improvements to similar interventions (Berg et al., 2014; Hintz et al., 2014; Johnson & Christensen, 2016; Nagy & Fawcett, 2019). Additionally, we would encourage students to constantly reflect on their experiences throughout the intervention process in a reflection journal or project diary (Johnson & Christensen, 2016). Requiring students to submit these diaries to their instructor/mentor periodically throughout the project encourages students to keep working on their reflections. Advise students to reflect on their diaries to suggest future modifications to their intervention project at the completion of their research.
Teaching Implications
Various ways for psychology instructors to successfully structure rigorous intervention projects into a year-long course, such as in an undergraduate senior-level capstone course or a graduate-level research methods course, are provided below. Adaptations of the intervention checklist (Butler & Morrow, 2021) to use in a semester-long course are also discussed below. Additionally, the learning objectives covered in such activities are provided to promote an increase in hands-on activities related to interventions included in undergraduate and graduate psychology courses (Gurung et al., 2016).
Structuring Intervention Activities into Courses
When integrating the project of planning, developing, and implementing an intervention activity into a multi-semester course, it is important to consider how each component followed in the project aligns with the course’s learning objectives. The “intervention planning” portion of the assignment can align with course learning objectives that promote students’ abilities to apply literature to the development of research methods. This portion enhances students’ interviewing and qualitative analysis skillsets when they gather feedback from content experts and individuals from the target audience. This planning stage of the assignment should take students about 1 month to complete.
The “intervention development” portion of the assignment will align with course learning objectives regarding students’ analysis and synthesis of research results, as students will be expected in this step to evaluate data collected from expert and assessment reviews, along with the pilot-test data, to modify their intervention. The development stage of the assignment should take students between one to 2 months to complete, depending on how many experts are contacted. Finally, the “intervention implementation” stage will align with course learning objectives that promote students’ understanding of fidelity in research by students developing an intervention manual and keeping strong records of participants’ progress throughout the intervention. This stage is expected to take students between one to 4 months to complete, depending on the intervention’s complexity.
To modify this assignment to a semester-long course, students can develop a research proposal on how they would plan their own intervention study using the intervention checklist guidelines (Butler & Morrow, 2021). After the assignment is completed, holding a class discussion on areas of anticipated difficulty in the intervention process may lead to strengthening students’ awareness of the barriers to success that are encountered in an intervention study. Alternatively, students enrolled in a semester-long course can critique published manuscripts in their area of interest (e.g., mental health counseling) that focus on evaluating the effectiveness of a specific intervention. Using the intervention checklist, students can critique the manuscript and describe steps in the intervention process that were either not described in the manuscript or not conducted. This assignment may encourage students to critically think about the intervention studies they are consumers of in their future careers (see article’s supplementary material for detailed assignment descriptions and student resource guides; Butler & Morrow, 2021).
Grading Intervention Activities
It is recommended that instructors who incorporate an intervention project into their course, whether in a semester-long or multi-semester course, provide grades and constructive feedback to students during multiple points, such as after student completion of each of the three main categories of the intervention checklist (Butler & Morrow, 2021). This allows students to incorporate instructor feedback into improving their final product. Instructors should develop rubrics that align with the course learning objectives which detail the grading structure of each portion of the student’s intervention project to clearly disseminate to students the expectations of the assignment.
It is advised that this project is utilized as a “final exam” for a course and that the project holds a substantial weight toward students’ final course grade (e.g., 30%–50%) due to the size and scope of the assignment. For group projects, students’ grades should be a combination of an overall group score, an individual score assigned by the other group members, and an individual score by the instructor to hold all group members accountable. Overall, these teaching and mentoring tips, informed by best practices from the literature and our own experiences, may provide undergraduate and graduate students the guidance to ensuring successful planning, developing, and conducting of intervention studies of their own.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by The University of Tennessee, Knoxville - Graduate School (Student/Faculty Research Awards).
