The impact of guided versus supportive coaching on mental health app engagement and clinical outcomes

Abstract

Although mobile mental health apps have the unique potential to increase access to care, evidence reveals engagement is low unless coupled with coaching. However, most coaching protocols are limited in their scalability. This study assesses how human support and guidance from a Digital Navigator (DN), a scalable coach, can impact mental health app engagement and effectiveness on anxiety and depressive symptoms. This study aims to detach components of coaching, specifically personalized recommendations versus general support, to inform scalability of coaching models for mental health apps. 156 participants were split into the DN Guide versus DN Support groups for the 6-week study. Both groups utilized the mindLAMP app for the duration of the study and had equal time with the DN, but the Guide group received personalized app recommendations. The Guide group completed significantly more activities than the Support group. 34% (49/139) of all participants saw a 25% decrease in PHQ-9 scores and 38% (53/141) saw a 25% decrease in GAD-7 scores. These findings show mental health apps, especially when supported by DNs, can reduce depression and anxiety symptoms when coupled with coaching, suggesting a feasible path for large-scale deployment.

Keywords

digital navigator smartphones mobile apps mental health

Introduction

Mental health outcomes for nearly all populations around the world continue to decline, with youth especially impacted. Rates of depression and anxiety have only increased in light of COVID-19 and related economic challenges. The resulting need for more mental health services has accelerated interest in telehealth, especially smartphone apps that are capable of offering scalable and accessible care. However, clinical experience with smartphone apps has been disappointing as many mental health apps have suffered from low rates of uptake and engagement,¹ precluding any clinical benefit. This study explores the challenge of engagement by investigating which model of coaching can support mental health app use and by assessing the feasibility of training and deploying Digital Navigators. A Digital Navigator is a member of the care team that utilizes their specific expertise in digital health to assist patients with technical support, troubleshooting, app customization to improve engagement, and clinical integration.

The need for scalable and accessible coaching models in mobile mental health care is crucial for realizing the potential of mental health apps. With rates of engagement with non-human supported mental health apps often approaching nearly zero percent after just 2 weeks, there has been a recent focus on how human support can assist.² Given that the potential of smartphone apps in increasing access to care rests in their scalability, it is critical that coaching models be equally scalable. A recent review of coaching to support mental health apps by our team found that much of the training for coaches is not shared or that coaches themselves are advanced mental health professionals (e.g., graduate students or licensed psychologists).³ A related 2022 review found significant heterogeneity in how coaches were deployed to support mental health technologies and was also unable to report on the impact of coaching on outcomes.⁴ Without sharing details on training so that it is scalable or creating programs accessible to non-professionals, it is likely that coaching to support mental health apps will not help realize the full potential of transformative increased access to care through mobile mental health.

To strive for scalable coaching, our team has developed a coaching model to train a new or existing member of the care team called a Digital Navigator. An individual’s care team may be comprised of, but is not limited to, physicians, clinicians, social workers, specialists, and administrators. We have established a 10-h Digital Navigator curriculum to train such individuals or new members to take on this innovative role. The training covers five core themes: (1) basic smartphone functions, (2) basic smartphone troubleshooting, (3) selecting and setting up smartphone apps, (4) sharing feedback around data and app use with patients, and (5) supporting engagement through motivational techniques.⁵ This training is scalable in nature for several reasons. First, the training can be done hybrid (5 h in-person and 5 h online), which allows those who cannot commit to a full day of training to still be able to receive certification. Additionally, the curriculum can be tailored to fit the needs of the community and team receiving the training. Since the training is broken into core themes, any aspects that do not pertain to those receiving the training can be easily omitted. Lastly, the curriculum is publicly available for other groups to adapt and lead. The curriculum has since been adopted by outside groups in 2022 and won a digital health equity award from the National Digital Inclusion Alliance. While we have employed the role in our own clinic to help support the integration of mobile health into routine clinical care,⁶ to date we have not formally assessed its role in supporting app use outside of clinical care.

Thus, in this study, we aimed to assess how support from a trained Digital Navigator could enable people to engage with a mental health app to reduce symptoms of anxiety and depression. We utilized the mindLAMP app that our team has developed and shares as open-source software. MindLAMP features a suite of CBT-related exercises or interventions that we have deployed in prior studies. However, we have found that engagement with these interventions has been limited previously⁷ and therefore sought to assess how human support through the Digital Navigator may increase it.

In planning this research, we understood that engagement with the app itself is not an ideal metric to assess the impact of the Digital Navigator. This is because prior research has shown that while some degree of engagement is necessary for any app to be effective, the relationship between engagement and outcomes is not simple.^8–11 One potential new metric developed by our team called the Digital Working Alliance Inventory (D-WAI) seeks to assess the alliance between a person and the app (i.e., not the human).¹² Prior research with the D-WAI suggests that it may be a valuable marker for assessing app engagement and impact¹³ and, thus, we chose to employ it in this study as an exploratory endpoint.

Finally, in deploying Digital Navigators, we aimed to separate components of coaching to inform scalability, specifically we investigated the impact of using all skills acquired from the training versus a limited set. The objective was to understand if sharing feedback around data/app use and supporting engagement through motivational techniques are necessary for individuals to engage with and benefit from the app. Feedback refers to discussion of CBT exercises utilized by the individual with the Digital Navigator to assist in tailoring and selecting upcoming activities. We hypothesized that the group receiving only core support (i.e. did not receive personalized app recommendations and notifications), not guidance, from the digital navigators, would have overall lower engagement and less symptomatic improvement with the app. As a pilot and feasibility study, we sought to assess if either form of digital navigator support would improve outcomes around depression or anxiety and provide the necessary data to power a clinical trial.

Methods

Participants

This study included individuals experiencing anxiety and other forms of mental illness who were recruited online (researchmatch.org) between July 2021 to February 2022. The participants were eligible if they were 18 years or older, owned a smartphone compatible with the study app, and had a minimum score of 5 (mild anxiety) on the Generalized Anxiety Disorder-7 (GAD-7)¹⁴ survey. Participants were excluded if they scored below a five on the GAD-7 questionnaire or were unable to provide informed consent. There were no exclusions for scores on other metrics, including depression scales such as the Patient Health Questionnaire-9 (PHQ-9), as we sought to assess the impact of this program on people with anxiety. Any PHQ-9 score indicating a risk of suicide was discussed with the study PI and followed up with a risk assessment of the participant.

Procedures

Measures

The 6-week study involved participants meeting virtually with a Digital Navigator at the beginning of the study, at week 2, and at week 6. After each of these three study visits, participants completed a set of Redcap surveys including the Patient Health Questionnaire-9 (PHQ-9),¹⁵ Generalized Anxiety Disorder-7 (GAD-7), Pittsburgh Sleep Quality Index (PSQI),¹⁶ Perceived Stress Scale (PSS-10),¹⁷ the Behavior and Symptom Identification Scale (BASIS),¹⁸ the Social Interaction Anxiety Scale (SIAS),¹⁹ the UCLA Loneliness Scale,²⁰ the Flourishing Scale,²¹ the System Usability Scale (SUS),²² and the D-WAI¹² The D-WAI is used to assess engagement as it is designed to measure the “alliance” between the user and the app. All measures were self-reported by participants. Measures like the PSQI, PSS-10, BASIS, and SIAS were not analyzed in this study but included to provide common metrics used in other mindLAMP studies which will enable future analysis.

Digital navigators

Research assistants with mental health experience through their role served as Digital Navigators and underwent training per our published protocol.⁵ The training was completed in person at Beth Israel Deaconess Medical Center. All participants provided written informed consent.

Participants were placed into the Digital Navigator Guide versus Digital Navigator Support conditions in a 3 to 1 ratio. This ratio was determined based on prior research which has shown that app engagement without guidance leads to minimal app engagement.¹ Our primary interest was the Guide condition, but we wanted to gather pilot data on the Support condition in the hopes that the Support condition would be a more scalable version of the Digital Navigator. Both groups had equal access and time, approximately 20 min per visit, with the Digital Navigator, but during study visits the Guide group received personalized recommendations and support around customizing mindLAMP to their needs. For example, Digital Navigator Guide may help a participant explore their response to a CBT exercise the prior week and suggest a related exercise to help build on those gains. Digital Navigators in the Guide group would inquire about the individual’s impression of that week’s activity, how it impacted their mental health if at all, and if they would like to build on the skills they learned or if they would prefer to try new ones. In the Support group, a Digital Navigator would check on how the participant was using the app, but not offer suggestions or recommendations for future use.

nindLAMP

Both groups had full access to mindLAMP, with modules including Thought Patterns (beginner and advanced), Mindfulness (beginner and advanced), Journaling, Distraction Games, Gratitude Journaling, Behavioral Activation, and Strengths. The Digital Navigator Guide group received notifications from the app directing them to complete Daily and Weekly Surveys, as well as weekly module activities co-selected with and tailored by their Digital Navigator Guide. However, the Digital Navigator Support group received generic notifications to remind them to engage with the app. Both groups also had access to their data through the ‘portal’ tab in mindLAMP, but only the Digital Navigator Guide group utilized this data to inform future module selection.

Compensation in both groups was the same. Participants were mailed a reloadable debit card. After each visit, participants received $25 for a total of $75 received upon completion of the study. This compensation was paid regardless of app engagement.

Analysis

First, we evaluated engagement. In this study, we used both the number of activities completed and D-WAI scores as a measure of engagement with the app. Despite weaknesses, the number of activities completed remains one of the most utilized metrics to report on engagement.⁸ First, two-tailed independent t-tests were performed to compare the activity counts between the Digital Navigator Guide and Digital Navigator Support groups using the scipy.stats module.²³ In addition to total activity counts, we also compared the number of Weekly Surveys, Daily Surveys, and other activities.

Second, we focused on anxiety and depression improvement as an outcome. We used a threshold of 25% symptom improvement. As many studies of coached technology platforms classify any clinical improvement as an outcome^24,25 we also report on this. Participants with initial scores of less than 5 on the GAD-7 were excluded from the GAD-7 analysis and participants with initial PHQ-9 scores less than 5 were excluded from the PHQ-9 analysis as there was no initial depression or anxiety for improvement to be measured. We also compared the change in PHQ-9 and GAD-7 scores from week 0 to week 6 between the Digital Navigator Guide and Digital Navigator Support groups.

Third, we wanted to investigate the relationship between engagement, alliance, and improvement. Thus, we looked at the correlations between the average SUS and D-WAI question scores and activity counts. The Hochberg method was used to correct the p-values for multiple comparisons using the statsmodels.stats.multitest.multipletests module in Python.²⁶ Two-tailed independent t-tests were used to compare average D-WAI scores for participants that did or did not improve on PHQ-9 and GAD-7. For both PHQ-9 and GAD-7, as above, improvement was defined as a 25% decrease in total score from week 0 to week 6.

Results

Of the 156 participants that completed the study, 119 were part of the Digital Navigator Guide group (Age: 34.9 ± 12.6) and 37 (Age: 35.8 ± 12.2) were part of the Digital Navigator Support group. Participant demographics are reported in Table 1.

Table 1.

Demographic information for participants in both the Digital Navigator Guide and Digital Navigator Support groups.

Race	Female	Male	Non-binary	Prefer not to say/other	Total
Asian	9	0	0	1	10
African American	10	3	0	0	13
Latinx	5	2	2	1	10
Native Hawaiian/Pacific Islander	0	1	0	0	1
White	89	17	4	2	112
Other	7	3	0	0	10
Total	120	26	6	4	156

An analysis of engagement between groups revealed that the Digital Navigator Guide group completed more module activities overall than the Digital Navigator Support group (p < .001), as shown in Figure 1. However, the Support group completed significantly more activities in their first week than the Guided group (p < .001, Guide: 103.8 ± 63.4; Support: 43.5 ± 52.7). The activities were broken down into three subsets: Weekly Survey, Daily Survey, and other activities which consisted of module-related activities. Module-related activities included Thought Patterns (beginner and advanced), Mindfulness (beginner and advanced), Journaling, Distraction Games, Gratitude Journaling, Behavioral Activation, and Strengths. In all three subsets, the Digital Navigator Guide group completed more activities (p < .001). The D-WAI scores were not different between the Guide and Support groups (p = .30, Guide: 22.1 $\pm$ 4.7; Support 21.2 $\pm$ 3.2).

Figure 1.

Comparison between number of activities completed by the Digital Navigator Guide versus the Digital Navigator Support group. (a) Total number of activities. (b) Number of weekly surveys. (c) Number of daily surveys. (b) Number of other activities completed.

To measure improvements in depression and anxiety symptoms, only participants with at least mild depression or anxiety, namely a PHQ-9 score or GAD-7 score of greater than 5, at session 1 were included in the following findings as participants without depression or anxiety at the beginning of the study could not improve. Based on this criteria, the total number of participants included in the PHQ-9 analysis was 139 (105 Guided, 34 Support), while the total for GAD-7 was 141 (106 Guided, 35 Support). After 6 weeks of utilizing the mobile mental health app, 34% (49/139) of participants saw a 25% decrease in PHQ-9 scores and 38% (53/141) saw a 25% decrease in GAD-7 scores. The number of participants with any decrease in score were 60% (84/139) for PHQ-9 and 62% (87/139) for GAD-7, listed in Table 2. There was no significant difference in raw depression (p = .92) or anxiety (p = .76) score change between the Digital Navigator Guide and Digital Navigator Support groups (Figure 2).

Table 2.

The number of participants that improved in the Digital Navigator Guide and Digital Navigator Support groups. Improvement is either a 25% or greater decrease in score or any decrease in score (>0%). All participants included in this table had a minimum starting score of at least 5.

		Guided	Support	Total
PHQ-9	Score decrease (≥25%)	36 (34%)	13 (38%)	49 (35%)
	Score did not decrease (≥25%)	69 (66%)	21 (62%)	90 (65%)
	Score decrease (>0%)	61 (58%)	23 (68%)	84 (60%)
	Score did not decrease (>0%)	44 (42%)	11 (32%)	55 (40%)
GAD-7	Score decrease (≥25%)	37 (35%)	16 (46%)	53 (38%)
	Score did not decrease (≥25%)	69 (65%)	19 (54%)	88 (62%)
	Score decrease (>0%)	66 (62%)	21 (60%)	87 (62%)
	Score did not decrease (>0%)	40 (38%)	14 (40%)	54 (38%)

Figure 2.

Raw score difference across the duration of the study comparing the Digital Navigator Guide and Digital Navigator Support groups. Negative numbers indicate improvement. (a) Change in PHQ-9 score. (b) Change in GAD-7 score.

High scores on two SUS survey questions (‘I think that I would like to use [mindLAMP] frequently’ and ‘I found the functions of [mindLAMP] well integrated’) were significantly correlated (p < .05) with number of activities completed on the mobile app for all participants, as shown in Table 2. Activity counts were also significantly correlated with total D-WAI score and all individual D-WAI questions except ‘the app is easy to use and operate.’ (Table 3).

Table 3.

Significant positive correlations (p < .05) between SUS and D-WAI questions and number of activities completed.

Survey question	Correlation coefficient	p-value (corrected)
(SUS) I think that I would like to use the recommended app frequently	0.274	0.0005
(SUS) I found the functions of the app well integrated [into your daily life]	0.216	0.0069
D-WAI total score	0.227	0.0046
(D-WAI) I trust the app to guide me towards my personal goals	0.179	0.0250
(D-WAI) I believe the app tasks will help me to address my problems	0.213	0.0077
(D-WAI) the app encourages me to accomplish tasks and make progress	0.263	0.0009
(D-WAI) I agree that the tasks within the app are important for my goals	0.204	0.0106
(D-WAI) the app supports me to overcome challenges	0.222	0.0053

Lastly, individuals that saw a score decrease by 25% or more had higher average D-WAI scores than those that did not see a score decrease for both PHQ-9 and GAD-7 (p = .0075; p = .0059), as seen in Table 4. All individual D-WAI questions were significantly different across the groups with a 25% improvement in PHQ-9 scores versus those that did not improve. For those that experienced a 25% improvement in GAD-7 scores versus those that did not improve, all individual D-WAI questions were significantly different except for ‘the app is easy to use and operate.’

Table 4.

D-WAI questions that are significantly different across the groups that did and did not improve by 25%). p-values greater than 0.05 are not shown.

Survey question	PHQ-9 (25%)	GAD-7 (25%)
Survey question	p-value	p-value
D-WAI total score	0.0013	0.0005
(D-WAI) I trust the app to guide me towards my personal goals	0.0177	0.0011
(D-WAI) I believe the app tasks will help me to address my problems	0.0085	0.0002
(D-WAI) the app encourages me to accomplish tasks and make progress	0.0023	0.0072
(D-WAI) I agree that the tasks within the app are important for my goals	0.0112	0.0045
(D-WAI) the app is easy to use and operate	0.0048
(D-WAI) the app supports me to overcome challenges	0.0025	8.8e − 5

Discussion

This study investigated the impact of a 6-weeks mobile mental health app intervention, mindLAMP, in conjunction with a coach known as a Digital Navigator, on mood and anxiety symptoms. By exploring the impact of human support on outcomes, we found that both the Guided and Supported Digital Navigator groups benefited from the study with approximately 1/3 of all participants having a 25% reduction in anxiety and depression scores. Given the scalable nature of this intervention, any improvement in symptoms is notable as the potential to deploy at a population level today is present. While the Guided group completed more activities, both groups reported high and similar working alliance scores with the app. This suggests a possible mechanism of action for the interaction between the app and Digital Navigators for future exploration.

An interesting finding concerning participant app engagement was that the support group participants appeared to binge, or over engage in, activities in the first week. Binging on the app refers to indulging in many or all of the app offerings in excess within a short period of time. This excess usage early may lead to a drop in app engagement over time perhaps because of the e-attainment effect or boredom with the app.^27–29 This resulted in a higher app engagement in the first week compared to that of the guide group. Despite this, the guide group ended up completing more activities overall suggesting that the extra guidance facilitated not only increased app engagement, but also more consistent app use over time. A potential contributor to the increased engagement for the guided group is the presence of personalized notifications reminding participants to complete the activities assigned by the Digital Navigator. Furthermore, participants in the guided group reviewed their data with their Digital Navigator and received personalized activity recommendations. Both may have contributed to the steadier level of engagement seen in the guided group.

As there was no difference in depression and anxiety outcomes between the two groups, the results seem to indicate that binging did not have a straightforward effect on clinical outcomes. However, our study was not designed to elucidate the relationship between binging and other app engagement patterns on outcomes, so further investigation must be done.

Our analysis of changes in depression and anxiety symptoms on a weekly basis revealed that participant response to modules was heterogeneous, see appendix A. One possible explanation, not well captured in the data but often shared with Digital Navigators, is the confounding role of life events outside the study. For example, one participant mentioned that three family members passed away unexpectedly between week 2 and week 6 of the study. While she found the modules helpful, her depression and anxiety scores both rose given her situation. The use of the app for the prevention of worsening symptoms versus direct improvement represents another area for future exploration.

Although there were no overall differences in D-WAI scores between the Digital Navigator Guide group and the Digital Navigator Support group, findings suggest that a strong alliance was formed in both groups. Therefore, this likely did not impact outcomes. Our results across both groups suggest that the D-WAI was more so correlated with activity completion than the SUS. One interpretation is that most participants found the app overall usable and thus the SUS did not capture unique information from participants. Other studies have shown that the D-WAI is associated with app engagement and our results add further support.¹³ These results also suggest an interesting theory regarding engagement in that the relationship between app use and clinical outcomes may be moderated or even mediated by alliance. While our results suggest a possible mediator, our study was not designed to investigate this effect. However, such a theory would explain the many prior studies that have not found a direct relationship between engagement and outcomes.^30,31

The lack of differences between the Guided and Support conditions suggests that the guided help, data review, and module assignment from the Digital Navigator did not impact outcomes around depression or anxiety. The success of the Support condition is important as it highlights how a simplified version of the role may result in nearly equivalent clinical benefits for patients. This has the potential to make the Digital Navigator curriculum more scalable than before, because it suggests that the training can be simplified, which would reduce the amount of time to train Navigators and ultimately makes offering the entire program more scalable. While Digital Navigators are of course not as scalable as self-help apps, the near-zero percent engagement with these self-help apps¹ underscores the importance of this and related roles. Future studies assessing alliance with both coaches and apps can help better separate the therapeutic role of each in helping apps be more effective.

The limitations in our work also suggest areas for future research. Given that any interaction with a Digital Navigator or coach may potentially offer therapeutic benefits, adding a control group with no human support would help in understanding our results. Our participants were recruited online and not assessed in person, though this limitation is a growing reality as patients seek care online during the COVID-19 pandemic. Several participants in the study had at least mild anxiety (score of 5-9 on the GAD-7 screening survey), but by the time participants were contacted, scheduled, consented, and enrolled, their GAD-7 scores at the initial visit were below 5. Thus, these participants had less room for improvement from the start to the end of the study. Additionally, participants in the two groups were unmatched, so future studies should aim to match the participants to control for confounding variables. While study payments were the same across both groups, it is possible these payments may have impacted outcomes if they changed anxiety and mood. As a pilot study, we did not capture follow-up data on the longer-term impact of this program. These results, however, may be used to power a clinical trial and explore the questions we were unable to do so here.

Conclusion

There is an ongoing need for accessible mental health care, especially due to the adverse impact the COVID-19 pandemic has had on mental health. Our results around using an already widely used and open-source app, coupled with a scalable coaching model, present a ready-to-deploy and actionable means to offer immediate and scalable support today. While we have discussed mental health applications, there is ongoing research in related coaching models to support health and wellness across the world.³² Future research efforts through clinical trials will continue to refine the optimal role of coaching but should not preclude using smartphone apps to provide services to patients today.

Footnotes

Acknowledgements

This work was supported by a grant from the Argosy Foundation. The authors would like to thank Jenny Melcher, Tanvi Lakhtakia, Luke Scheuer, Ryan Hays, Ryan D’Mello, Ashley Meyer, and Suraj Patel for hosting visits for study participants.

Authors’ note

All authors have read and approved of the submitted manuscript.

Author contributions

JT was involved in the conceptualization, design, and writing of the manuscript. EC, SC, and DC were involved in the design, data collection and curation, analysis, and writing.

Declaration of conflicting interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: JT has had research support from Otsuka in the past 3 years. JT has also co-founded a mental health app company called Precision Mental Wellness.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Argosy Foundation.

Ethical Statement

ORCID iDs

Sarah M. Chang

John Torous

Appendix

References

Baumel

Muench

Edan

, et al. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25; 21(9): e14567.

Szinay

Jones

Chadborn

, et al. Influences on the Uptake of and Engagement With Health and Well-Being Smartphone Apps: Systematic Review. J Med Internet Res 2020; 22(5): e17572.

Meyer

Wisniewski

Torous

. Coaching to Support Mental Health Apps: Exploratory Narrative Review. JMIR Human Factors 2022 Mar 8; 9(1): e28301.

Bernstein

Weingarden

Wolfe

, et al. Human Support in App-Based Cognitive Behavioral Therapies for Emotional Disorders: Scoping Review. J Med Internet Res 2022 Apr 8; 24(4): e33307.

Wisniewski

Gorrindo

Rauseo-Ricupero

, et al. The role of digital navigators in promoting clinical care and technology integration into practice. Digit Biomark 2020; 4(1): 119–135.

Rauseo-Ricupero

Henson

Agate-Mays

, et al. Case studies from the digital clinic: integrating digital phenotyping and clinical practice into today’s world. Int Rev Psychiatr 2021 May 19; 33(4): 394–403.

Melcher

Patel

Scheuer

, et al. Assessing engagement features in an observational study of mental health apps in college students. Psychiatr Res 2022 Feb 23; 310: 114470.

Bradway

Gabarron

Johansen

, et al. Methods and measures used to evaluate patient-operated mobile health interventions: Scoping literature review. JMIR mHealth and uHealth 2020; 8(4): e16814.

Jacob

Sezgin

Sanchez-Vazquez

, et al. Sociotechnical Factors Affecting Patients’ Adoption of Mobile Health Tools: Systematic Literature Review and Narrative Synthesis. JMIR mHealth and uHealth 2022 May 5; 10(5): e36284.

10.

Mohr

Kwasny

Meyerhoff

, et al. The effect of depression and anxiety symptom severity on clinical outcomes and app use in digital mental health treatments: Meta-regression of three trials. Behav Res Ther 2021 Sep 24; 147: 103972.

11.

Molloy

Anderson

. Engagement with mobile health interventions for depression: A systematic review. Internet Interventions 2021 Sep 11; 26: 100454.

12.

Henson

Wisniewski

Hollis

, et al. Digital mental health apps and the therapeutic alliance: initial review. BJPsych open 2019 Jan; 5(1): e15.

13.

Goldberg

Baldwin

Riordan

, et al. Alliance with an unguided smartphone app: validation of the digital working alliance inventory. Assessment 2021 May 18; 29: 1331–1345.

14.

Spitzer

Kroenke

Williams

JBW

, et al. Generalized Anxiety Disorder 7 [Internet]. Washington, DC: American Psychological Association, 2011 [cited 2022 Aug 31]. Available from. http://doi.apa.org/getdoi.cfm?doi=10.1037/t02591-000

15.

Kroenke

Spitzer

Williams

JBW

. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med 2001 Sep; 16(9): 606–613.

16.

Buysse

Reynolds

Monk

, et al. The Pittsburgh sleep quality index: A new instrument for psychiatric practice and research. Psychiatr Res 1989 May; 28(2): 193–213.

17.

Cohen

Kamarck

Mermelstein

. A Global Measure of Perceived Stress. J Health Soc Behav 1983 Dec; 24(4): 385.

18.

Cameron

Cunningham

Crawford

, et al. Psychometric properties of the BASIS-24© (Behaviour and Symptom Identification Scale–Revised) Mental Health Outcome Measure. Int J Psychiatr Clin Pract 2007 Jan; 11(1): 36–43.

19.

Mattick

Clarke

. Development and validation of measures of social phobia scrutiny fear and social interaction. Behav Res Ther 1998 Apr; 36(4): 455–470.

20.

Russell

Peplau

Ferguson

. Developing a Measure of Loneliness. J Pers Assess 1978 Jun; 42(3): 290–294.

21.

Diener

Wirtz

Biswas-Diener

, et al. New Measures of Well-Being. In: Diener

(ed) Assessing Well-Being. Netherlands: Springer Netherlands, 2009, vol. 39, pp. 247–266. DOI: 10.1007/978-90-481-2354-4_12.

22.

Brooke

. SUS: A quick and dirty usability scale. In Usability Evaluation In Industry. Boca Raton, FL: CRC Press, 1995 Nov 30, p. 189.

23.

Virtanen

Gommers

Oliphant

, et al. SciPy 1.0: fundamental algorithms for scientific computing in Python. Nat Methods 2020 Mar; 17(3): 261–272.

24.

Richards

Duffy

Burke

, et al. Supported internet-delivered cognitive behavior treatment for adults with severe depressive symptoms: a secondary analysis. JMIR Mental Health 2018 Oct 2; 5(4): e10204.

25.

Wilhelm

Weingarden

Greenberg

, et al. Efficacy of App-Based Cognitive Behavioral Therapy for Body Dysmorphic Disorder with Coach Support: Initial Randomized Controlled Clinical Trial. Psychother Psychosom 2022; 91(4): 267–275.

26.

Seabold

Perktold

. Statsmodels: Econometric and statistical modeling with python. In Proc of the 9th Python in Science Conf 2010 Jun 28; 57: 61.

27.

Wang

Liang

, et al. Identifying major impact factors affecting the continuance intention of mHealth: a systematic review and multi-subgroup meta-analysis. NPJ Digital Medicine 2022 Sep 15; 5(1): 145.

28.

Balaskas

Schueller

Cox

, et al. Understanding users’ perspectives on mobile apps for anxiety management. Frontiers in Digital Health 2022; 4: 854263.

29.

Kaveladze

Wasil

Bunyi

, et al. User experience, engagement, and popularity in mental health apps: secondary analysis of app analytics and expert app reviews. JMIR human factors 2022 Jan 31; 9(1): e30766.

30.

Kaveladze

Wasil

Bunyi

, et al. User Experience, Engagement, and Popularity in Mental Health Apps: Secondary Analysis of App Analytics and Expert App Reviews. JMIR human factors 2022 Jan 31; 9(1): e30766.

31.

Torous

Michalak

O’Brien

. Digital health and engagement—looking behind the measures and methods. JAMA Netw Open 2020 Jul 1; 3(7): e2010918.

32.

Tessarolo

Petsani

Conotter

, et al. Developing ambient assisted living technologies exploiting potential of user-centred co-creation and agile methodology: The CAPTAIN project experience. J Ambient Intell Hum Comput. 2022; 13: 1–16.