Modernizing Training in Psychotherapy Competencies With Adaptive Learning Systems: Proof of Concept

Abstract

Purpose:

This proof-of-concept study assessed the feasibility, acceptability, and effectiveness of an intelligent tutoring system (ITS) as a classroom adjunct to improve training bachelor of social work (BSW) students in client engagement strategies.

Methods:

We codeveloped the ITS with 11 undergraduate students and pilot tested it with six BSW students enrolled in a class on telephone-based cognitive behavioral therapy (tCBT). Student competencies in tCBT were assessed by expert review of role-plays. We also examined time spent using ITS and relation with competency.

Results:

The majority of students (81.8%) in Wave 1 and all of the students who submitted role-plays in Wave 2 passed the clinical skills role-play. Students advancing through the ITS more quickly had better tCBT competency ratings than those progressing more slowly.

Discussion:

One of the most challenging aspects of training is how to competently deliver evidence-based practices. ITS has the potential to streamline and scale such training.

Keywords

eLearning adaptive training intelligent tutoring psychotherapy

Ethnic minorities (Agency for Healthcare Research and Quality, 2019), older adults (Kessler et al., 2015), and people living in rural areas (Mott et al., 2015) are at greatest risk of not receiving evidence-based psychosocial intervention (EBPI), despite their preference for this treatment (Dwight-Johnson et al., 2013; Givens et al., 2007; McHugh et al., 2013; Mohlman, 2012; Raue et al., 2011; Raue, Schulberg, et al., 2019). At fault are several instrumental barriers to care such as transportation barriers, distance, time, and the stigma that many cultural groups have about seeking care (Arnaez et al., 2020; Brenes et al., 2015; Fortney et al., 2011; Kreps, 2017; Pepin et al., 2009). Of all the barriers that influence accessibility of EBPI in low-resourced communities, the most problematic are supply-side barriers—that is, too few EBPI-prepared clinicians living in these areas (Health Resources and Services Administration, Substance Abuse and Mental Health Services Administration, & Office of Policy, Planning, and Innovation, 2016). This low supply is further compounded by a lack of linguistically and culturally responsive clinicians to provide services to an increasingly diverse U.S. population. Due to inadequate numbers of EBPI-prepared clinicians and serious geographic maldistribution, many parts of the United States experience a severe shortage of mental health providers, particularly low-income and rural areas (Thomas et al., 2009). Addressing these supply-side barriers has reached critical importance.

Training Challenges

According to the Institute of Medicine (2015), training programs have historically underprepared the mental health workforce to deliver EBPIs. Rather, clinicians often learn EBPIs in the field as health care organizations move toward the medical home model and adopt integrated care programs like Collaborative Care (Archer et al., 2012). This imposes considerable limitations on the time dedicated to training as well as the effort involved in learning new interventions that were not designed for busy practices (Coleman et al., 2017). Most high-quality programs use educational scaffolding methods (Jessee & Tanner, 2016) such as Bloom’s Revised Taxonomy (Siaki et al., 2014). Learning objectives build on six cognitive processes in the taxonomy: remembering information; analyzing, understanding, applying, then evaluating this information; and finally, creativity synthesizing this knowledge. Teaching methods are then tailored to these levels (e.g., didactics for knowledge transfer, simulated case role-play for knowledge application, mentored case review for analytic skill; Clark, 2011; Clark et al., 2009; Wesson et al., 2013). Although such educational best practices are effective, they result in very expensive training programs that many U.S.-based schools and clinical sites cannot support (Roundfield & Lang, 2017; Vasiliadis et al., 2015). The Veterans Health Administration and the University of Washington (UW) Advancing Integrated Mental Health Solutions Center use such models, with less intensity; however, the success of such modified models has been variable (W. Cross et al., 2007). The scalability of these programs is limited by expert time to conduct training activities (W. Cross, Moore, & Ockerby, 2010; W. Cross & Wyman, 2006; W. F. Cross et al., 2014), clinician time away from work to engage in training activities, training expense, and variable training quality across supervisors and trainers (W. Cross, Cerulli, et al., 2010; W. F. Cross et al., 2014).

Intelligent Tutoring Systems (ITSs)

Innovations in instructional technology and adaptive learning have the potential to transform EBPI training by decreasing training time and improving competency in a scalable way. Adaptive learning is an educational method that uses adaptive computerized algorithms to tailor the presentation of educational material to trainees’ learning needs based on initial assessments, experience, and in-training task performance. Development and application of this technology encompass diverse disciplines including computer science, education, psychology, and neuroscience. These programs arose from the observation that tailored education is difficult to deploy and scale using traditional, nonadaptive training approaches. ITSs are a type of adaptive learning program relevant to EBPI training. ITS are computerized programs used as classroom adjuncts that support student learning as they guide trainees through simulated cases, decisional dilemmas, and other higher order cognitive tasks and provide immediate and customized instruction and feedback to the trainee (Belland et al., 2017; Kron et al., 2017; Schaffer et al., 2017). ITS emulate one-on-one tutoring and are particularly beneficial in cases where there are too few instructors available. Recent research shows that these systems have the capacity to train a wide range of novice learners in high-level problem-solving and deep content knowledge (Belland et al., 2017). Moreover, training times to reach competency can be significantly reduced. For example, children who use adaptive training are able to learn complex algebraic concepts (seventh-grade content) in 1.5 hr and could complete algebraic solutions with nearly 100% accuracy (Liu et al., 2015). Similar results for ITS systems are found in various fields of medicine (Blechner & Sordo, 2005; Crowley et al., 2005; Gonzalez et al., 2012; Ong et al., 2005) and behavior change counseling for substance use (de Ruijter et al., 2018; Hayes-Roth et al., 2010) but have yet to be applied to mental health.

Social workers are the most numerous of all mental health professionals (Heisler, 2018), and overall demand for social workers is expected to grow in the next decade to meet an increased need for health care and social services (Bureau of Labor Statistics, 2020). They are essential in primary care and other service settings, where they help clients and consumers set and achieve personal goals, and do so in an empathic, culturally sensitive way. Social workers also need strong person-centered communication skills and the ability to structure meetings and sessions. Furthermore, growing attention is focused on broadening the mental health workforce shortage. Although not yet a standard practice in BSW programs, one way to address such a shortage is to train bachelor-level social workers and case managers in low-intensity EBPIs (see Choi et al., 2020; Clark, 2011; Richards et al., 2016). Thus, attention to training bachelor-level social work students in engagement skills and client-centered strategies is imperative. We created an ITS that can be used to strengthen the core competencies of client and consumer engagement and for training in any social work encounter when working with people with common social stressors and mental health disorders. The system is focused on competent delivery of common communication strategies, in particular, the nonspecific elements of care. The purpose of this proof of concept study was to (1) codevelop the content and user interface of the ITS and (2) conduct preliminary assessment of acceptability, feasibility, and effectiveness of the ITS on bachelor-level student competency in delivering a low-intensity intervention for depression.

Method

Participants

Participants in this study consisted of two different waves of students pursuing their bachelor’s degree. The first wave was comprised of undergraduate students working toward their bachelor’s degree in psychology (n = 10) or social work (n = 1) at the UW who enrolled in a four-credit upper level elective course during the summer 2018 quarter. See Table 1 for student characteristics. The second wave of students consisted of six bachelor of social work (BSW) students in their senior year at Heritage University (HU). HU is a designated Hispanic- and Native American–Serving Institution in the Yakima Valley of rural central Washington. Students were recruited during the spring of their junior year via a presentation about a training program to deliver manualized telephone-based cognitive behavioral therapy (tCBT). tCBT is an acceptable and effective intervention for various populations experiencing depression, including low-income rural Latino adults, and has been effectively delivered by social work students with little clinical experience (Aisenberg et al., 2012; Dwight-Johnson et al., 2004, 2011). Six Latino BSW students applied and were accepted into this specialized program in fall 2019.

Table 1.

Student Participant Demographics.

Demographics	Wave 1(N = 11)	Wave 2(N = 6)
Age, M (SD) years	23.80 (5.71)	33.00 (8.46)
Gender, n (%)
Female	7 (63.6)	5 (83.3)
Male	4 (36.4)	1 (16.7)
Ethnicity, n (%)
Hispanic/Latino	0 (0)	6 (100.0)
Race, n (%)
Asian/Asian American	2 (18.2)	0 (0)
Black/African American	0 (0)	0 (0)
White	7 (63.6)	1 (16.7)
Multiracial	2 (18.2)	0 (0)
Other ^a	0 (0)	5 (83.3)
Major area of study, n (%)
Social work	1 (9.1)	6 (100.0)
Psychology	10 (90.9)	0 (0)
Other	0 (0)	0 (0)
Class rank, n (%)
Senior	8 (72.7)	6 (100.0)
Junior	2 (18.2)	0 (0)
Other	1 (9.1)	0 (0)

Note. First wave consisted of upper level undergraduates enrolled in an elective psychology and social work course at the University of Washington; second wave consisted of senior BSW students enrolled in an elective psychotherapy skills course at Heritage University.

^a All students reporting “Other” for racial identity specified ethnicity for race (i.e., Hispanic/Latino).

Procedure

Courses

The Wave 1 course was team-taught by B.N.R., P.A.A., and P.J.R. and consisted of nine weekly in-person class meetings, each 2 hr in length. The course focused on basic clinical interviewing skills and a brief behavioral intervention called patient activation (Raue, Sirey, et al., 2019; Raue et al., 2019), consisting of strategies drawn from formal behavioral activation. Students typically spent the first hour in didactic classroom instruction and discussion and the second hour using the ITS and role-playing simulated clients played by either fellow students or instructors. In this wave, we had students who use the ITS as part of classroom activities over 7 weeks so that we could rapidly collect data and feedback on ITS usability and utility.

The Wave 2 course consisted of five, face-to-face half-day classes taught by EA. Each class consisted of didactic instruction and discussion as well as role-play practice of engagement and tCBT skills. The dyad role-play allowed students to be both practitioner and client for each of the eight sessions of the tCBT treatment manual. Students were assigned to use the ITS asynchronously (i.e., as homework) between each in-person class session. Wave 2 students received a US$750 stipend for participation in the specialized curriculum and training program and were incentivized US$50 for each interview. Students in both waves conducted role-plays outside of classroom time and submitted recordings for fidelity review (see Assessment of Clinical Skills subsection). This study was approved by the institutional review board at the UW, and all students provided informed consent for study procedures.

Materials and Measures

Assessment of clinical skills

Fidelity to patient activation was assessed over the final 3 weeks of class in Wave 1. Beginning with the seventh class, students submitted role-plays of a full first session and the beginning of a follow-up session as part of their homework. Students were randomly partnered with other students each week for role-plays to ensure a variety of pairings. Each student took turns delivering the intervention and playing a simulated client. They recorded the role-plays and submitted them to course instructors for assessment. To rate student skill acquisition, study investigators used a standardized fidelity tool that evaluated adherence to the patient activation protocol across seven specific skill domains and a final global score. Scores ranged from 0 (very poor) to 5 (very good); global scores of 3 (satisfactory) were considered passing.

Students in Wave 2 were paired up after the second class session, and they conducted weekly role-plays of the eight manualized tCBT sessions with their partners outside of class time. Each student took a turn role-playing both the interventionist and a simulated client for each session. Students received biweekly group supervision to review the role-plays from either EA or a master’s-level licensed clinical social worker. Role-plays were audio recorded for rating by an external reviewer, who assessed role-play Sessions 2, 4, and 8 for fidelity to tCBT. Fidelity, including process and communication skills, was rated on a scale from 0 (not passing) to 2 (excellent). A global score of >1 was considered passing.

ITS

The ITS for this study was created by a team of engineers and experts in training baccalaureate-level practitioners in EBPIs. The program is an educational adjunct that focuses on helping students develop critical thinking skills and applying new knowledge. This is accomplished by presenting case scenarios that illustrate clinical difficulties in EBPI delivery followed by a series of potential (correct and incorrect) ways that a clinician may respond (see Figure 1).

Figure 1.

Sample intelligent tutoring system scenario with student response and system feedback.

ITS content was identified by interviewing four local experts from social work and clinical psychology and nine (n = 2 MSW/Licensed Clinical Social Worker, n = 7 PhD) national experts. Experts were selected for their experience training lay workers, including bachelor-level providers, in psychotherapy. These semistructured qualitative interviews focused on EBPI domains that were most challenging for trainees to master and areas where trainees needed additional support. Human-centered design methodology informed the codevelopment of ITS content and user interface, which were iteratively refined based on expert and user feedback three times over 1 year, including feedback incorporated from the first wave of students. The beta version presented herein was tested with the HU cohort (Wave 2). By testing the ITS in two different groups of students, we were able to observe use patterns and difficulties with the initial version of the ITS, collect user feedback about the interface, and incorporate improvements to the design of the system before testing it on a new group of students.

The training content and format of the ITS beta version focused on (a) therapeutic alliance skills, including empathy, reflective listening, and cultural responsiveness skills specific to local Mexican American culture typical of the Yakima Valley; (b) structuring the session skills, including agenda setting and redirection; and (c) goal setting and action planning skills, including assigning and reviewing homework. Each therapeutic skill was broken down into the skill components. For example, reflection (a therapeutic alliance skill) includes these components: (a) repeating the client’s feeling or experience; (b) restating the client’s contextual issues, problems, or stressors; and (c) drawing a connection between the life circumstance and the client’s emotional experience. Then, we created a series of scenarios and question types based on Bloom’s Revised Taxonomy of cognitive processes (see Table 2). HU students were first prompted to remember the components of the skill being tutored. Next, they were presented with a case scenario and selected all the therapist responses that do, or do not, typify the skill. This pushed the trainee to extend their learning beyond remembering, as above, to understand the concepts through recognition and classification. For the third set of learning challenges, they applied the information they have learned by identifying the components of the skill demonstrated in the therapist response. Then, students analyzed multiple therapist responses to a scenario and identified those best demonstrating a specific skill. In the fifth type of challenge, trainees evaluated a therapist response to a client scenario by highlighting various components of the response, evaluating whether the skill was demonstrated, and if so, identifying the corresponding concept being demonstrated. Finally, trainees were asked to put the information together in a novel way. They created new therapist responses by selecting statements that demonstrated a skill. The adaptive algorithm individually advanced each student through this sequence of challenges to become progressively more complicated. Consecutive correct answers triggered advancement to a more complex challenge.

Table 2.

ITS Beta Version: Six Types of Learning Challenges Illustrated With Example Skills, Scenarios, and Reponses.

Module	Sample Skill	Bloom’s Taxonomy	Question Type	Sample Case and Scenario	Sample Response Choices ^a
Therapeutic alliance
	Empathy	Remember	Which of these concepts demonstrates [skill]?Which of these concepts does not demonstrate [skill]? Select all that apply.	Not applicable	Express that a client’s emotions or experiences are normal, expected, or understandable.Give advice on how to view the problem or what to do about it.Communicate concern for the client’s feeling or experiences.Communicate negative judgment about a client’s feelings.
	Reflection	Understand	Choose all statements that demonstrate [skill].Choose all statements that do not demonstrate [skill]. Select all of the potential therapist responses that apply.	Juan is a 42-year-old man who is unmarried and does not have children. His doctor recommended that he see you after a work-related injury for some help adjusting back to his daily activities.He tells you: “I’ve been frustrated and upset lately. I feel achy and I’m in pain much of the day. It’s hard for me just to get out of bed in the morning.”	You’re not feeling like yourself, which is making it hard to take care of your responsibilities. Since your accident, you have understandably been in a lot of pain. This is really affecting how you feel. Coping with your pain has made it hard to do much of anything. How do you end up getting out of bed? Sounds like you really wish you had someone around to take care of you. What did you used to do in the morning, before you were injured?
Structuring the session
	Agenda setting	Apply	Why does this demonstrate [skill]?Why doesn’t this demonstrate [skill]? Select the concept(s) that apply to the therapist statement.	Maria is a 28-year-old married woman who recently immigrated to Washington from Mexico. She is referred to you by her primary care doctor, who is concerned that she has symptoms of depression.At the start of the third session with Maria, she jumps right in and starts discussing a fight that her son was in at school.	It definitely sounds like we should discuss that. But first, let’s take a step back and set an agenda so we can be sure we cover everything we want to today. Asks client if they feel they have improved since the start of the program.Brings up an agenda as first topic of the meeting and provides a brief overview of what is included on the agenda.Asks unfocused, open-ended question about the past week.
	Redirection	Analyze	Select the therapist response that shows [concept of skill].Select the therapist response that do not show [concept of skill]. For example, Select the therapist response that suggests returning to the task at hand or prioritizing the current concern as more important.	David is a 62-year-old man who has been unemployed for the last year after experiencing a work-related injury. His primary care doctor recommends he see you.With about 10 min left in session, you and David are setting up the homework for the upcoming week, when he starts talking about something upsetting in the news.	So, you saw news of a mugging near your house—that’s scary! Can we finish setting an action plan and then troubleshoot any difficulties that might come up?I bet that’s unsettling. Tell me what happened!Sorry, but we won’t have time today to talk about this news that bothered you.
Goal setting and action planning
	Assigning homework	Evaluate	For each part of the therapist response, define why the statement demonstrates [skill].Students are shown a therapist’s response. Selected parts of the response are highlighted, for which the student selects whether the skill (e.g., assigning homework) is demonstrated, selects the dependent concept.	Lucia is a 19-year-old woman who is new to the clinic.She wants to work on feeling more accomplished and less overwhelmed with her work problems.	You mentioned that it might be helpful to prioritize your work tasks and focus on one at a time. Let’s talk about how you might do that at the start of every workday. I also want us to consider what might get in the way of this plan. Sample concept response options:Sets specific goals and action plans in collaboration with the client.Ensures the client will be successful by predicting obstacles and planning realistic strategies to overcome them.Picks an unrealistic goal.
	Reviewing homework	Create	Select therapist statements below to create a response demonstrating [skill].Students select from a list of potential statements to create a compound therapist statement demonstrating a skill (e.g., reviewing homework).	Miguel is a 55-year-old mechanic who came to see you because his wife is worried that he has been acting differently.A few sessions into your work together, Miguel says: “I know we’ve met three times already, but I’m not feeling any better. I don’t see how doing these action plans are supposed to help me.”	I’m glad you’re bringing this up.Recovery can take time. I suggest trying again this week.Sometimes, it can take a bit of time for mood to improve, but you are getting into the habit of adding things to your life that will enrich it. Over time, as you do more, your mood will improve more.What we planned is probably not the right activity for you.Do you think we’re planning things that aren’t important or helpful to you?We can talk about how to help you feel more successful making the changes that you’d like to see.One thing we could do is look back at your list of enjoyable and meaningful activities. Then, we can make sure you are picking activities that will be better mood boosters for you.Let’s just try one more week and see how it goes any better.

Note. These six types of problems were repeated for each of the seven skills in ITS. We use six of the skills here to illustrate this progression across a selected sample of cases and scenarios. ITS = intelligent tutoring system.

^a Therapist statements are given in italics.

ITS usability and utility

HU students participated in individual exit interviews with E.C.F near the end of the fall course. The semistructured interview guide was designed by E.C.F, a user experience researcher. Open-ended questions about ITS use behavior (when, where, how often, and for how long?) and the system’s interface were designed to uncover difficulties, areas for future improvement, and opportunities for increased engagement. Students rated how engaging they found the ITS on a scale of 1 (not at all) to 5 (extremely); similarly, they rated how successful the ITS was as a tutor for core therapeutic elements (1 = not at all to 5 = extremely) and were asked to elaborate on a score of 3 or lower. Next, they discussed how well they grasped each of the seven ITS skills and whether there were any that were particularly difficult to understand. We then showed them each of the six ITS learning challenges and asked probed their experience and suggestions to improve clarity or helpfulness. Finally, we showed them a mock-up of a redesign of the ITS that was in development (in part based on the previous student codesign workshop) to get feedback and to validate our design decisions.

Data Analysis

Qualitative data were analyzed using inductive thematic analysis (Braun & Clarke, 2006); responses to scaled questions were collated using descriptive statistics. The ITS collected quantitative data reflecting ITS engagement and performance during the trial and was analyzed for trends over time. Descriptive statistics were summarized. We examined a sequence of automatically generated ITS performance graphs for possible trends; visually discovered hypotheses were subsequently analyzed. Association between content advancement speed per time spent using ITS and competency ratings was assessed with bivariate Pearson’s correlation.

Results

Wave 1

Student competencies

Half (54.5%) of the students were rated as competent in delivering the intervention by the seventh class. With further practice and feedback, the majority of students (81.8%, n = 9) were rated as competent in delivering the intervention by the end of the course (nine class sessions).

Wave 2

Student competencies

Although all students completed the entire ITS, each progressed at a significantly different pace (range of total ITS time from 160 min to 350 min) and different cadence (single sessions ranged from 10 min to > 1 hr). Figure 2 displays student advancement through the ITS.

Figure 2.

Wave 2 student progression in ITS over time. Note. Lines depict the cumulative ITS progress for each of the six students in Wave 2. Highlight a (steepest slope) depicts the student who completed the ITS training the fastest (total of 160 min spent using ITS). The student depicted in Highlight b took 350 min to complete the content (slowest progression through the ITS curriculum). Highlight c illustrates one example of a student struggling (“flatlining,” or plateauing), which is followed by a “lightbulb” moment where a concept appeared to suddenly start making sense. This was followed by a rapid advance through related content (illustrated by a near-vertical progression).

Two students who were paired together did not complete any role-plays and thus were not reviewed for competencies. All four students who completed role-plays achieved at least passing scores (M = 1.5, SD = 0.58) on the eighth session. Eleven of the 12 role-play sessions reviewed (91.7%) were rated as passing or excellent (M = 1.33, SD = 0.65). The sample with completed role-plays (n = 4) was too small to draw any inferences about statistical significance. However, we were able to identify useful patterns between ITS performance and performance on tCBT role-plays that will drive future ITS optimization and analysis. First, learning velocity was represented as the inverse of total ITS time. We found that the best tCBT performers demonstrated better learning velocity (progress per unit of time spent on the ITS; r = .97), as early as one third of the way through the ITS. Second, we found no correlation (r = .13) between the length of individual ITS training sessions and performance with tCBT role-play performance. It is likely that the idiosyncratic ways that students engage with the ITS (in short spurts or in long sessions) do not impact learning outcomes. Third, when analyzing individual performance on more complex portions of the ITS, students with greater tCBT competency (based on role-play performance) struggled in the same places that lower performing students did but for a shorter period of time (see Figure 2, Highlights a and b).

ITS usability and utility

Wave 2 students rated the ITS as fairly engaging (M = 3.3, SD = 0.52, range = 3–4) and moderately helpful as a tutor (M = 3.3, SD = 1.19, range = 2–5). Thematic analysis of the student feedback revealed consensus that the ITS provided clinically relevant material not explicitly covered in other BSW coursework. Students also shared that the ITS was a welcome opportunity to practice and hone their clinical skills before using them in real-world client sessions. Additionally, students were satisfied with the presentation of mock client scenarios and found the cultural examples relevant to their practicum placements. Students provided very positive feedback about design elements including liking the system’s lighthearted feedback (e.g., a confetti display to celebrate successes; a message saying, “Don’t sweat it” as feedback on errors) and appreciating the ability to review content. Finally, suggestions that will be considered in a future iteration of the ITS included wanting more thorough explanations of incorrect answers and functionality to allow students to repeat completed skills for additional practice.

Discussion and Applications to Practice

Bachelor-level social work training programs commonly teach client-centered skills for effective communication, goal setting, and action planning. Given the need to broaden the mental health workforce and the increased popularity in stepped care interventions (Maehder et al., 2019), such programs are also well positioned to enhance treatment access by developing essential clinical skills in BSW students. While most BSW programs lack a mental health focus in their curriculum, based on preliminary findings and experience with Wave 2 students, there is a promise that BSW students with proper training and supervision can develop the skills necessary to effectively and competently deliver EBPIs, particularly with rural and traditionally underserved populations, where access to licensed clinicians is very limited. However, there remains a considerable gap between what we know to be effective treatments and what is actually accessible in routine clinical care, driven in part by the cost and time of high-quality EBPI training. We provide preliminary proof of concept to suggest that a computerized ITS may supplement traditional curricula to deepen and accelerate skill acquisition of core competencies in the processes of engagement and service delivery in the helping relationship.

Consistent with prior examination of CBT fidelity among graduate-level practitioners (Waltman et al., 2017), expert interviews that informed our content development revealed training difficulties in agenda setting and session pacing, structuring and reviewing client goals, and interpersonal effectiveness, in addition to need for cultural responsiveness. These are important communication and EBPI competencies that the ITS can support to augment traditional social work and other training. We applied these findings to codevelop and refine the ITS, resulting in a beta version with seven skill modules and enhanced user interface that was tested with BSW students in rural area with a large Latino community. The adaptations resulted in adequate acceptability, usability, and educational utility as reported by students and as tested by role-play assessment using external review. The ITS content was specifically created to support quality delivery of EBPIs by nonexperts. Indeed, all BSW students who submitted role-plays for evaluation during beta testing demonstrated competent tCBT delivery. While the sample size was small, results of this study nonetheless provide a promising signal that EBPI skills can be taught to BSW students in a relatively short time (a five-session university training program) and reinforced by the ITS for skill development.

Another interesting, albeit preliminary, finding is the potential of the ITS to identify both areas of struggle and students who may need additional assistance relatively early in the training. Variability in progression speed through ITS may serve as an early indicator of future EBPI competency. In addition, observing overall performance on early ITS “bottleneck skills”—those skills in which students uniformly struggle—seems to predict far transfer of skills, so it may be a useful indicator as to which students need more support, especially in new or tough situations.

An important aspect of our codevelopment was ongoing refinement of the ITS content in order to develop the most relevant skills for client and consumer engagement. For example, empathy has a foundational role in mental health and social service interactions (Greeno et al., 2017). Prior research has also supported cultural responsiveness as a critical component of competent and effective practice (Horevitz et al., 2013) and demonstrated that targeted training can improve these skills (Neely-Barnes et al., 2019). The profession of social work’s commitment to culturally responsive practice is affirmed by the standards of the National Association of Social Workers (2003) and the Council on Social Work Education (CSWE, 2015). We explicitly tied our ITS training content to these practice competencies including (a) developing advanced skills in individual intervention and service delivery, (b) emphasizing client-centered and culturally responsive care, and (c) promulgating practice-informed research and research-informed practice. For the purposes of our work, we focused on therapeutic strategies commonly used in behavioral interventions versus techniques specific to one intervention (e.g., cognitive restructuring, activity monitoring). Furthermore, we adapted the ITS to allow the beta version to operate asynchronously such that the students could use the platform for homework, thus maximizing live class time for didactic content, discussion, skill demonstration, practice with peers, and feedback from the instructor(s).

While promising, our findings must be interpreted in light of study limitations. Our alpha testing and beta testing were cohort studies that lacked a “learning-as-usual” control group. The first wave of students was largely comprised of undergraduate students majoring in psychology; we attempted to recruit an equal representation of social work students to codesign the ITS but were unable to identify many of these students available for a summer course. However, our purpose was to obtain initial reactions to ITS from students who were naive to clinical encounters; as such, our first-wave students were instrumental in codesigning a usable version of the ITS that was later deployed with BSW students (Wave 2). Moreover, the students self-selected to enroll in these advanced elective programs and may not be representative of typical undergraduate learners. Nonetheless, our testing across diverse learner groups in two institutions lends strength to our findings. Although our testing was conducted with small groups of students learning two different EBPIs, these sample sizes (e.g., as small as five to seven users) are sufficient to encounter usability issues (Nielsen & Landauer, 1993). Finally, although we found substantial improvement in student competency over time (54%–90%), a truly randomized trial is needed to clarify the contribution of the ITS to this improvement. Future trials with larger samples are necessary to ascertain learner outcomes attributed to the ITS.

Efficiently investing resources to develop EBPI competency during social work education could positively impact the profession and real-world care, preparing a workforce pipeline who can already demonstrate proficiency and bypass the expensive and time-consuming need to train (or retrain) on the job (Miller & Moyers, 2016). Our ITS was designed to supplement classroom instruction, not replace training. We acknowledge that supervision of field experience is the signature pedagogy of social work, intended to connect and deepen the theoretical foundations from classroom learning with practice in real-world settings (CSWE, 2015). However, the availability and quality of supervision are fraught with similar challenges as other trainings, including an overtaxed workforce (Gushwa & Harriman, 2019). Considering the cost and other access challenges across training, our innovative instructional platform warrants additional testing and refinement.

Results from our beta testing phase will continue to inform adaptations to enhance training. ITS will improve both automatically, based on machine learning of performance outcomes, and through analytics-driven content refinement. The automated refinement depends on a large number of users. This use data will train the ITS progression so that we can more accurately determine the optimal moment to advance trainees to new skills. The content refinement occurs at three levels: intra-problem scaffolding adaptation (assisting a user with just-in-time feedback), inter-problem adaptation (increasing difficulty for each unique user), and engagement and retention adjustments to ensure trainees’ perseverance through the ITS. All three levels are informed by data analytics of user performance, which in turn identify key weaknesses that should be improved by changing the ITS interaction or content. Rapid changes to address these bottlenecks will be deployed so that we can observe relative improvement through analytics of subsequent users. Although our sample size was too small to develop this algorithm, it serves as proof of concept for full-scale deployment. This final adapted ITS will be used in a future test with a large sample of BSW students.

Technology-enabled platforms like ITS hold great promise to expand access to competency-based training while reducing cost and time burdens associated with such skill acquisition. This potential may be most pronounced in those very settings with the greatest need—that is, underresourced and underserved communities in which BSW programs are particularly positioned to serve. Our study extends innovations from computer science to social work training and demonstrates proof of concept to further develop and test our computerized instructional platform to improve EBPI competency.

Footnotes

Authors’ Note

The content is solely the responsibility of the authors and does not necessarily represent the official views of the funders.

Acknowledgments

We thank Kelly Thompson, Leah Lucid, and Noah Triplett for their instrumental support as teaching assistants for courses conducted at the UW. Donovan “Nova” Rivera assisted with early phases of (intelligent tutoring system) ITS review and with transcription and coding of one of our student focus groups. Dmitri Danilov and Ronald “Ric” Gray implemented the ITS tested herein. We thank Alejandra Lopez for her support of user interviews and improvements to the user interface design. Finally, we thank our study partners, Mary O’Brien from the Yakima Valley Farm Workers Clinic and Corey Hodge and Miguel Juarez from Heritage University, for their contributions to the recruitment of students as well as planning and teaching of the HU curriculum and training program.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This publication was supported in part by the National Institute of Mental Health (grants P50MH115837, T32MH073553, and R34MH111849) and intramural funding through the Population Health Initiative at the University of Washington.

ORCID iD

Brenna N. Renn

References

Agency for Healthcare Research and Quality. (2019). 2018 National healthcare quality and disparities report. AHRQ Pub. No. 19-0070-EF. https://www.ahrq.gov/research/findings/nhqrdr/nhqdr18/index.html

Aisenberg

Dwight-Johnson

O’Brien

Ludman

E. J.

Golinelli

(2012). Building a community-academic partnership: Implementing a community-based trial of telephone cognitive behavioral therapy for rural Latinos. Depression Research and Treatment. https://doi.org/10.1155/2012/257858

Archer

Bower

Gilbody

Lovell

Richards

Gask

Dickens

Coventry

(2012). Collaborative care for depression and anxiety problems. Cochrane Database of Systematic Reviews, 10, CD006525. https://doi.org/10.1002/14651858.CD006525.pub2

Arnaez

J. M.

Krendl

A. C.

McCormick

B. P.

Chen

Chomistek

A. K.

(2020). The association of depression stigma with barriers to seeking mental health care: A cross-sectional analysis. Journal of Mental Health, 29(2), 182–190. https://doi.org/10.1080/09638237.2019.1644494

Belland

B. R.

Walker

A. E.

Kim

N. J.

Lefler

(2017). Synthesizing results from empirical research on computer-based scaffolding in STEM education: A meta-analysis. Review of Educational Research, 87(2), 309–344. https://doi.org/10.3102/0034654316670999

Blechner

M. D.

Sordo

(2005). Feasibility of an intelligent tutoring system for laboratory medicine. In American Medical Informatics Association (AMIA) annual symposium proceedings (p. 895). American Medical Informatics Association.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Brenes

G. A.

Danhauer

S. C.

Lyles

M. F.

Hogan

P. E.

Miller

M. E.

(2015). Barriers to mental health treatment in rural older adults. American Journal of Geriatric Psychiatry, 23(11), 1172–1178. https://doi.org/10.1016/j.jagp.2015.06.002

Bureau of Labor Statistics, U.S. Department of Labor. (2020). Social workers. In Occupational outlook handbook (Vol. 2020). https://www.bls.gov/ooh/community-and-social-service/social-workers.htm

10.

Choi

N. G.

Marti

C. N.

Wilson

N. L.

Chen

G. J.

Sirrianni

Hegel

M. T.

Bruce

M. L.

Kunik

M. E.

(2020). Effect of telehealth treatment by lay counselors vs by clinicians on depressive symptoms among older adults who are homebound: A randomized clinical trial. JAMA Network Open, 3(8), e2015648. https://doi.org/10.1001/jamanetworkopen.2020.15648

11.

Clark

D. M.

(2011). Implementing NICE guidelines for the psychological treatment of depression and anxiety disorders: The IAPT experience. International Review of Psychiatry, 23(4), 318–327. https://doi.org/10.3109/09540261.2011.606803

12.

Clark

D. M.

Layard

Smithies

Richards

D. A.

Suckling

Wright

(2009). Improving access to psychological therapy: Initial evaluation of two UK demonstration sites. Behaviour Research and Therapy, 47(11), 910–920. https://doi.org/10.1016/j.brat.2009.07.010

13.

Coleman

K. J.

Hemmila

Valenti

M. D.

Smith

Quarrell

Ruona

L. K.

Brandenfels

Hann

Hinnenkamp

Parra

M. D.

Monkman

Vos

Rossom

R. C.

(2017). Understanding the experience of care managers and relationship with patient outcomes: The COMPASS initiative. General Hospital Psychiatry, 44, 86–90. https://doi.org/10.1016/j.genhosppsych.2016.03.003

14.

Council on Social Work Education. (2015). Educational policy and accreditation standards. https://www.cswe.org/Accreditation

15.

Cross

Cerulli

Richards

Herrmann

(2010). Predicting dissemination of a disaster mental health “Train-the-Trainer” program. Disaster Medicine and Public Health Preparedness, 4(4), 339–343. https://doi.org/10.1001/dmp.2010.6

16.

Cross

Matthieu

M. M.

Cerel

Knox

K. L.

(2007). Proximate outcomes of gatekeeper training for suicide prevention in the workplace. Suicide and Life-Threatening Behavior, 37(6), 659–670. https://doi.org/10.1521/suli.2007.37.6.659

17.

Cross

Moore

Ockerby

(2010). Clinical supervision of general nurses in a busy medical ward of a teaching hospital. Contemporary Nurse, 35(2), 245–253. https://doi.org/10.5172/conu.2010.35.2.245

18.

Cross

Wyman

P. A.

(2006). Training and motivational factors as predictors of job satisfaction and anticipated job retention among implementers of a school-based prevention program. The Journal of Primary Prevention, 27(2), 195–215. https://doi.org/10.1007/s10935-005-0018-4

19.

Cross

W. F.

Pisani

A. R.

Schmeelk-Cone

Xia

McMahon

Munfakh

J. L.

Gould

M. S.

(2014). Measuring trainer fidelity in the transfer of suicide prevention training. Crisis, 35(3), 202–212. https://doi.org/10.1027/0227-5910/a000253

20.

Crowley

R. S.

Tseytlin

Jukic

(2005). ReportTutor—An intelligent tutoring system that uses a natural language interface. In American Medical Informatics Association (AMIA) annual symposium proceedings (pp. 171–175). American Medical Informatics Association.

21.

de Ruijter

Candel

Smit

E. S.

de Vries

Hoving

(2018). The effectiveness of a computer-tailored e-learning program for practice nurses to improve their adherence to smoking cessation counseling guidelines: Randomized controlled trial. Journal of Medical Internet Research, 20(5), e193. https://doi.org/10.2196/jmir.9276

22.

Dwight-Johnson

Aisenberg

Golinelli

Hong

O’Brien

Ludman

(2011). Telephone-based cognitive-behavioral therapy for Latino patients living in rural areas: A randomized pilot study. Psychiatric Services, 62(8), 936–942. https://doi.org/10.1176/ps.62.8.pss6208_0936

23.

Dwight-Johnson

Apesoa-Varano

Hay

Unutzer

Hinton

(2013). Depression treatment preferences of older White and Mexican origin men. General Hospital Psychiatry, 35(1), 59–65. https://doi.org/10.1016/j.genhosppsych.2012.08.003

24.

Dwight-Johnson

Lagomasino

I. T.

Aisenberg

Hay

(2004). Using conjoint analysis to assess depression treatment preferences among low-income Latinos. Psychiatric Services, 55(8), 934–936. https://doi.org/10.1176/appi.ps.55.8.934

25.

Fortney

J. C.

Burgess

J. F.

Bosworth

H. B.

Booth

B. M.

Kaboli

P. J.

(2011). A re-conceptualization of access for 21st century healthcare. Journal of General Internal Medicine, 26(Supp. 2), 639–647. https://doi.org/10.1007/s11606-011-1806-6

26.

Givens

J. L.

Houston

T. K.

Van Voorhees

B. W.

Ford

D. E.

Cooper

L. A.

(2007). Ethnicity and preferences for depression treatment. General Hospital Psychiatry, 29(3), 182–191. https://doi.org/10.1016/j.genhosppsych.2006.11.002

27.

Gonzalez

Blobel

Lopez

D. M.

(2012). Adaptive intelligent systems for pHealth—An architectural approach. Studies in Health Technology and Informatics, 177, 170–175.

28.

Greeno

E. J.

Ting

Pecukonis

Hodorowicz

Wade

(2017). The role of empathy in training social work students in motivational interviewing. Social Work Education, 36(7), 794–808. https://doi.org/10.1080/02615479.2017.1346071

29.

Gushwa

Harriman

(2019). Paddling against the tide: Contemporary challenges in field education. Clinical Social Work Journal, 47(1), 17–22.

30.

Hayes-Roth

Saker

Amano

(2010). Automating individualized coaching and authentic role-play practice for brief intervention training. Methods of Information in Medicine, 49(4), 406–411. https://doi.org/10.3414/ME9311

31.

Health Resources and Services Administration, National Center for Health Workforce Analysis, Substance Abuse and Mental Health Services Administration, & Office of Policy, Planning, and Innovation. (2016). National projections of supply and demand for behavioral health practitioners: 2013-2025. https://bhw.hrsa.gov/sites/default/files/bhw/health-workforce-analysis/research/projections/behavioral-health2013-2025.pdf

32.

Heisler

E. J.

(2018). The mental health workforce: A primer. Congressional Research Service. https://fas.org/sgp/crs/misc/R43255.pdf

33.

Horevitz

Lawson

Chow

J. C.

(2013). Examining cultural competence in health care: Implications for social workers. Health & Social Work, 38(3), 135–145. https://doi.org/10.1093/hsw/hlt015

34.

Institute of Medicine. (2015). Psychosocial interventions for mental and substance use disorders: A framework for establishing evidence-based standards. The National Academies Press. https://www.nap.edu/catalog/19013/psychosocial-interventions-for-mental-and-substance-use-disorders-a-framework

35.

Jessee

M. A.

Tanner

C. A.

(2016). Pursuing improvement in clinical reasoning: Development of the Clinical Coaching Interactions Inventory. Journal of Nursing Education, 55(9), 495–504. https://doi.org/10.3928/01484834-20160816-03

36.

Kessler

E.-M.

Agines

Bowen

C. E.

(2015). Attitudes towards seeking mental health services among older adults: Personal and contextual correlates. Aging and Mental Health, 19(2), 182–191. https://doi.org/10.1080/13607863.2014.920300

37.

Kreps

G. L.

(2017). Stigma and the reluctance to address mental health issues in minority communities. Journal of Family Strengths, 17(1), Article 3. https://digitalcommons.library.tmc.edu/cgi/viewcontent.cgi?referer=https://www.google.com/&httpsredir=1&article=1340&context=jfs

38.

Kron

F. W.

Fetters

M. D.

Scerbo

M. W.

White

C. B.

Lypson

M. L.

Padilla

M. A.

Gliva-McConvey

G. A.

Belfore

L. A.

, II West

Wallace

A. M.

Guetterman

T. C.

Schleicher

L. S.

Kennedy

R. A.

Mangrulkar

R. S.

Cleary

J. F.

Marsella

S. C.

Becker

D. M.

(2017). Using a computer simulation for teaching communication skills: A blinded multisite mixed methods randomized controlled trial. Patient Education and Counseling, 100(4), 748–759. https://doi.org/10.1016/j.pec.2016.10.024

39.

Liu

Y. E.

Ballweber

O’rourke

Butler

Thummaphan

Popović

(2015). Large-scale educational campaigns. ACM Transactions on Computer-Human Interaction, 22(2). https://doi.org/10.1145/2699760

40.

Maehder

Löwe

Härter

Heddaeus

Scherer

Weigel

(2019). Management of comorbid mental and somatic disorders in stepped care approaches in primary care: A systematic review. Family Practice, 36(1), 38–52. https://doi.org/10.1093/fampra/cmy122

41.

McHugh

R. K.

Whitton

S. W.

Peckham

A. D.

Welge

J. A.

Otto

M. W.

(2013). Patient preference for psychological vs. pharmacological treatment of psychiatric disorders: A meta-analytic review. The Journal of Clinical Psychiatry, 74(6), 595–602. https://doi.org/10.4088/JCP.12r07757

42.

Miller

W. R.

Moyers

T. B.

(2016). Asking better questions about clinical skills training. Addiction, 111(7), 1151–1152. https://doi.org/10.1111/add.13095

43.

Mohlman

(2012). A community based survey of older adults’ preferences for treatment of anxiety. Psychology and Aging, 27(4), 1182–1190. https://doi.org/10.1037/a0023126

44.

Mott

J. M.

Grubbs

K. M.

Sansgiry

Fortney

J. C.

Cully

J. A.

(2015). Psychotherapy utilization among rural and urban veterans from 2007 to 2010. Journal of Rural Health, 31(3), 235–243. https://doi.org/10.1111/jrh.12099

45.

National Association of Social Workers. (2003). Standards and indicators for cultural competence in social work practice. https://www.socialworkers.org/Practice/Practice-Standards-Guidelines

46.

Neely-Barnes

S. L.

Kirk

B. G.

Elswick

S. E.

Taylor

L. C.

Delavega

Pettet

F. L.

Landry

(2019). Training second-year MSW students to improve cultural and linguistic competence. Research on Social Work Practice, 30(4), 392–398. https://doi.org/10.1177_1049731519874397

47.

Nielsen

Landauer

T. K.

(1993). A mathematical model of the finding of usability problems. INTERACT ‘93 and CHI ‘93 Conference on Human Factors in Computing Systems, 206–213. https://doi.org/10.1145/169059.169166

48.

Ong

L. S.

Vijayan

Koh

C. S.

Lai

C. C.

Lim

C. W.

Loke

W. F.

Low

S. H.

Tang

K. Y.

Wong

F. L.

Yong

K. L.

(2005). An intelligent tutoring system for trauma management (Trauma-Teach): A preliminary report. Annals of the Academy of Medicine in Singapore, 34(8), 499–504.

49.

Pepin

Segal

D. L.

Coolidge

F. L.

(2009). Intrinsic and extrinsic barriers to mental health care among community-dwelling younger and older adults. Aging and Mental Health, 13(5), 769–777. https://doi.org/10.1080/13607860902918231

50.

Raue

P. J.

Dawson

Hoeft

Russo

Ferguson

Green

Petersky

Kaplan

. (2019). Acceptability of a lay-delivered intervention for depression in senior centers. Aging and Mental Health. Advance online publication. https://doi.org/10.1080/13607863.2019.1698515

51.

Raue

P. J.

Schulberg

H. C.

Bruce

M. L.

Banerjee

Artis

Espejo

Catalan

Romero

(2019). Effectiveness of shared decision-making for elderly depressed minority primary care patients. The American Journal of Geriatric Psychiatry, 27(8), 883–893. https://doi.org/10.1016/j.jagp.2019.02.016

52.

Raue

P. J.

Sirey

J. A.

Dawson

Berman

Bruce

M. L.

(2019). Lay-delivered behavioral activation for depressed senior center clients: Pilot RCT. International Journal of Geriatric Psychiatry, 34, 1715–1723. https://doi.org/10.1002/gps.5186

53.

Raue

P. J.

Weinberger

Sirey

Meyers

Bruce

(2011). Preferences for depression treatment among elderly home health care patients. Psychiatric Services, 62(5), 532–537.

54.

Richards

D. A.

Ekers

McMillan

Taylor

R. S.

Byford

Warren

F. C.

Barrett

Farrand

P. A.

Gilbody

Kuyken

O’Mahen

Watkins

E. R.

Wright

K. A.

Hollon

S. D.

Reed

Rhodes

Fletcher

Finning

(2016). Cost and outcome of behavioural activation versus cognitive behavioural therapy for depression (COBRA): A randomised, controlled, non-inferiority trial. Lancet, 388(10047), 871–880. https://doi.org/10.1016/S0140-6736(16)31140-0

55.

Roundfield

K. D.

Lang

J. M.

(2017). Costs to community mental health agencies to sustain an evidence-based practice. Psychiatric Services, 68(9), 876–882. https://doi.org/10.1176/appi.ps.201600193

56.

Schaffer

H. E.

Young

K. R.

Ligon

E. W.

Chapman

D. D.

(2017). Automating individualized formative feedback in large classes based on a directed concept graph. Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00260

57.

Siaki

L. A.

Lentino

C. V

Mark

D. D.

Hopkins-Chadwick

D. L.

(2014). Achieving army nursing evidence-based practice competencies through a civilian-military nurse partnership. The United States Army Medical Department Journal, 2014, 42–51.

58.

Thomas

K. C.

Ellis

A. R.

Konrad

T. R.

Holzer

C. E.

Morrissey

J. P.

(2009). County-level estimates of mental health professional shortage in the United States. Psychiatric Services, 60(10), 1323–1328. https://doi.org/10.1176/appi.ps.60.10.1323

59.

Vasiliadis

H. M.

Lesage

Latimer

Seguin

(2015). Implementing suicide prevention programs: Costs and potential life years saved in Canada. The Journal of Mental Health Policy and Economics, 18(3), 147–155.

60.

Waltman

Hall

B. C.

McFarr

L. M.

Beck

A. T.

Creed

T. A.

(2017). In-session stuck points and pitfalls of community clinicians learning CBT: Qualitative investigation. Cognitive and Behavioral Practice, 24(2), 256–267.

61.

Wesson

M. I.

Whybrow

Greenberg

Gould

(2013). Clinical supervision in a UK military department of community mental health. Journal of the Royal Army Medical Corps, 159(4), 291–293. https://doi.org/10.1136/jramc-2013-000079