Alliance With an Unguided Smartphone App: Validation of the Digital Working Alliance Inventory

Participants and Procedure

Study 1 involved a cross-sectional online survey. Participants for Study 1 were recruited through Prolific (www.Prolific.co). Similar to functionality available through Amazon’s Mechanical Turk, Prolific is an online participant recruitment platform. Documented advantages of Prolific include access to a pool of more diverse participants who are both less dishonest (i.e., less likely to cheat in order to gain a bonus payment) and more naïve (i.e., unfamiliar with commonly used measures) than Mechanical Turk (Peer et al., 2017). Participants in Study 1 were drawn from a larger study investigating the utilization of meditation in the U. S. population. Eligible participants for the larger study were adults (≥18 years old) living in the United States. The current analyses were not part of the preregistered aims of the larger study (https://osf.io/4h86s/?view_only=0e5d7ad85f87468ea40e047b3cf7c795). We include data from participants who passed two attention check items (“please select the leftmost response,” “I have been randomly selecting responses on this survey”), reported having used a smartphone-based meditation app, and completed the DWAI in relation to their most used meditation app (n = 290). Consistent with the general population (Wasil et al., 2020), the two most commonly used apps were Headspace (31.0%) and Calm (28.6%). Both apps include instruction in various meditation techniques (e.g., focusing attention on the breath, scanning attention through the body, generating feelings of care for others) with content focused on increasing well-being and attention regulation as well decreasing stress, anxiety, and sleep disruption. Participants in Study 1 were on average 39.93 years old (SD = 14.77); 56.2% (n = 163) were female, 42.1% (n = 122) male, and 1.7% (n = 5) gender nonbinary; 70.3% (n = 204) were non-Hispanic White, 12.4% (n = 36) Black, 7.9% (n = 23) Latinx, 6.2% (n = 18) Asian, 2.8% (n = 8) multiracial, and 0.3% (n = 1) Native American; 60.0% (n = 174) had completed college; 59.0% (n = 171) had an annual income ≤$50,000. The survey was administered between November 22nd and December 4th, 2020.

Participants for Study 2 were drawn from the intervention arm of a randomized controlled trial testing a smartphone-based meditation app—the Healthy Minds Program (HMP; for a description of the app and its underlying model of well-being, see Dahl et al., 2020; Goldberg, Imhoff-Smith, et al., 2020) in comparison with a waitlist control (https://osf.io/eqgt7). The HMP app includes four modules with practices aimed at developing skills supportive of well-being. These include practices to strengthen mindfulness and stabilize attention (Awareness), cultivate qualities like appreciation and kindness that contribute to healthy relationships with oneself and others (Connection), support self-inquiry (Insight), and enhance meaning in life by clarifying and applying core values and self-transcendent motivations (Purpose). The current analyses do not overlap with the primary aims of the larger trial, although we did hypothesize in separate preregistrations that the DWAI would be associated with changes in outcomes (https://osf.io/85kya/?view_only=cef24dc17c784a9790e388d5b5814f1d) and adherence (https://osf.io/swerk/?view_only=b31cd287334b491d806d0c63f3a583fb). Results from the full set of preregistered analyses may be published elsewhere. Eligible participants for the larger trial were adults (≥18 years old) currently employed by a school district within the state of Wisconsin. Additional exclusion criteria included extensive meditation experience (i.e., retreat experience, weekly practice for ≥1 year, daily practice for ≥6 months), past use of the HMP app, and lack of access to a device capable of running the HMP app. Outcomes were assessed at baseline and posttreatment (4-weeks postbaseline). The DWAI was administered one, two, three, and 4-weeks postbaseline for those in the HMP condition. The current study includes those participants randomized to the HMP condition who completed the DWAI at least once (n = 314). Participants in Study 2 were on average 42.66 years old (SD = 11.00); 84.7% (n = 266) were female, 11.5% (n = 36) male, and 0.3% (n = 1) gender nonbinary, and 3.5% (n = 11) of unknown gender; 86.0% (n = 270) were non-Hispanic White, 2.6% (n = 8) Black, 0.3% (n = 1) Latinx, 1.3% (n = 4) Asian, 4.1% (n = 13) multiracial, 0.3% (n = 1) Native American, and 5.4% (n = 17) of unknown race/ethnicity; 86.0% (n = 270) had completed college; 15.3% (n = 48) had an annual income ≤$50,000. Recruitment for Study 2 occurred between June 18th and September 7th, 2020.

Measures

Analyses

As the DWAI has not previously been evaluated, we first sought to establish the measure’s factor structure. We first conducted exploratory factor analysis using data from Study 1. Based on high intercorrelations between alliance dimensions observed in previous studies (e.g., Tracey & Kokotovic, 1989), models with an oblique (promax) rotation were run using the “factanal” function in R (R Core Team, 2018). We aimed to determine the ideal number of factors that provided simple structure (i.e., items loading highly on only one factor; Thompson, 2004). In addition, we examined a scree plot and associated eigenvalues derived using the “eigen” function in R and conducted a parallel analysis which is a preferred method for determining the number of factors to retain (Thompson, 2004) using the “fa.parallel” function in the “psych” package in R (Revelle, 2020).

Having determined the appropriate number of factors and the corresponding item factor loadings, we conducted a series of CFAs using DWAI data from Study 2. The primary purpose of these analyses was to see if the structure from Study 1 replicated in Study 2. Additionally, because Study 2 involves longitudinal data, we could evaluate whether items were related to one another over time above and beyond the correlation among the latent factors over time. Model 1 was a longitudinal CFA that used the preferred factor structure from Study 1 at each of the four DWAI administrations (Weeks 1, 2, 3, and 4 of the study). We included a covariance between the latent variables across time. Thus, Model 1 assumes that the only relationship between items both within- and between-time points was at the latent-variable level. Model 2 added a residual correlation between the same item across time points (e.g., Item 1 at Week 1 correlated with Item 1 at Week 2; Item 2 at Week 1 with Item 2 at Week 2; and so on). Model 2 constrained the correlations to be equal across time on a per item basis (i.e., six total correlations, one for each item). Model 3 was identical to Model 2 except that correlations were freely estimated. Finally, Model 4 included an autoregression correlation structure, wherein correlations decay as the time points get further apart.

Overall fit of the CFA models was evaluated based on the comparative fit index (CFI) and root mean square error of approximation (RMSEA) using standard indices (i.e., CFI ≥ .95, RMSEA ≤ .06; Thompson, 2004). Fit was compared across models using the Akaike information criterion and the Bayesian information criterion. CFA was conducted using Mplus Version 8.1 (Muthén & Muthén, 2017) and the “MplusAutomation” package in R (Hallquist & Wiley, 2018).

Reliability was assessed in two ways. In both Study 1 and Study 2, we calculated internal consistency reliability based on Cronbach’s α. As Study 2 involved repeated DWAI assessments, we calculated test–retest reliability between weeks as the intraclass correlation coefficient (ICC1; single raters, absolute agreement) using the “ICC” function in the “psych” package (Revelle, 2020).

Validity was assessed in several different ways. Discriminant validity was evaluated by examining associations between DWAI scores with theoretically unrelated constructs. These included social desirability (Study 1), baseline psychological distress (Study 1 and Study 2), and preference for waitlist (Study 2). Convergent validity was evaluated by examining associations between DWAI scores with theoretically related constructs. These included perceived app effectiveness (Study 1) and preference for the HMP condition (Study 2). Finally, tests of predictive validity examined associations between DWAI scores with app utilization and changes in psychological distress. Utilization was assessed by self-report in Study 1 and obtained objectively through the HMP app in Study 2. All validity tests relied on correlation coefficients (i.e., Pearson’s r, which simplifies to a point biserial correlation when including a dichotomous variable like regular app use). Linear regression models were used to examine DWAI scores as a predictor of posttreatment psychological distress controlling for pretreatment psychological distress in Study 2. A subsequent linear regression model added preference for the HMP condition as a predictor of posttreatment psychological distress to assess incremental validity. In order to evaluate whether the strength of the alliance-outcome association varied depending on when alliance was assessed, we fit models in Mplus with these time-specific associations constrained to be equivalent or unique (i.e., free to vary across DWAI assessment time points). Fit was compared using a log-likelihood ratio test. An exploratory analysis described below evaluated whether the predictive validity of the DWAI varied based on psychological distress at baseline (i.e., test of moderation), to allow comparisons with the broader alliance in psychotherapy literature which has focused on clinical samples (Horvath et al., 2011). With the exception of test–retest reliability and predictive validity tests, all models using Study 2 DWAI scores focused on baseline DWAI scores.

Exploratory and Confirmatory Factor Analysis

Factor loadings for one-, two-, and three-factor solutions with an oblique (promax) rotation are presented in Table 4. As can be seen, all items loaded (≥0.43) on the single factor in the one-factor model. In the two-factor solution, multiple items (Items 2 and 4) demonstrated cross-loadings (i.e., loadings ≥0.30 on multiple factors). Cross-loading was observed in the three-factor solution along with a single-item factor. A single-factor solution was also supported by examination of the scree plot which indicated the presence of only one component showing an eigenvalue >1 (Supplemental Materials Figure 1, available online) and through parallel analysis (Supplemental Materials Figure 2, available online). Thus, we concluded that a one-factor solution was preferred.

OPEN IN VIEWER

Table 4.

Results of Exploratory Factor Analysis With Oblique (Promax) Rotation in Study 1.

Item	One factor	Two factor		Three factor
	Factor 1	Factor 1	Factor 2	Factor 1	Factor 2	Factor 3
1	0.864	0.117	0.845	0.227	0.403	0.285
2	0.861	0.485	0.412	0.124	0.010	0.894
3	0.784	0.810	0.005	0.541	0.372	−0.085
4	0.845	0.568	0.310	0.017	0.928	−0.012
5	0.434	0.493	−0.045	0.517	−0.057	−0.002
6	0.807	0.814	0.025	0.877	−0.063	0.059

Note. Values indicate factor loadings. n = 290.

Next, we conducted CFA using DWAI data from Study 2. As shown in Table 5, four models were estimated and compared. Both Model 2 and Model 3 showed near acceptable fit based on the CFI and TLI (≥.95) and RMSEAs (≤.060; Thompson, 2004). As Model 2 and Model 3 are nested, a formal model comparison was conducted. This indicated superior fit for Model 3, χ²(30) = 48.69, p = .017. However, the simpler Model 2 which constrained correlated residuals to be equal across time points fit the data best based on Akaike information criterion and Bayesian information criterion. Thus, Model 2 was considered the final model. Factor loadings for both models are reported in Table 6, with residual correlations in Supplemental Materials Table 1 (available online). As expected, all items loaded highly on the single factor (loadings ≥0.49). Loadings were nearly identical across Model 2 and Model 3.

OPEN IN VIEWER

Table 5.

Model Fit for Confirmatory Factor Analysis With Repeated Digital Working Alliance Inventory Assessments in Study 2.

Model	# Param	χ2	df	CFI	TLI	AIC	BIC	RMSEA	90% CILB	90% CIUB
Model 1	78	1139.415	246	0.846	0.827	15338.712	15631.165	0.108	0.101	0.114
Model 2	84	453.779	240	0.963	0.958	14665.076	14980.025	0.053	0.046	0.061
Model 3	114	405.091	210	0.966	0.956	14676.388	15103.819	0.054	0.046	0.062
Model 4	66	837.234	258	0.900	0.893	15012.531	15259.991	0.085	0.078	0.091

Note. Model 1 = independent factor structure across time points; Model 2 = correlated residuals constrained to be equal across time points; Model 3 = correlated residuals not constrained; Model 4 = correlated residuals with autoregressive (AR1) covariance structure; df = degrees of freedom; # Param = number of parameters; CFI = comparative fit index; TLI = Tucker–Lewis index; AIC = Akaike information criterion; BIC = Bayesian information criterion; RMSEA = root mean square error of approximation; 90% CI = 90% confidence interval lower (LB) and upper (UB) bounds.

OPEN IN VIEWER

Table 6.

Standardized Factor Loadings for Final Confirmatory Factor Analysis Models.

Time	Item	Model 2		Model 3
		Estimate	SE	Estimate	SE
Week 1	1	0.782	0.026	0.797	0.025
Week 1	2	0.801	0.024	0.791	0.025
Week 1	3	0.774	0.026	0.794	0.025
Week 1	4	0.867	0.019	0.869	0.019
Week 1	5	0.492	0.041	0.496	0.041
Week 1	6	0.790	0.025	0.797	0.025
Week 2	1	0.858	0.018	0.871	0.018
Week 2	2	0.859	0.018	0.859	0.018
Week 2	3	0.834	0.019	0.824	0.021
Week 2	4	0.895	0.015	0.895	0.015
Week 2	5	0.618	0.034	0.611	0.035
Week 2	6	0.835	0.020	0.836	0.021
Week 3	1	0.887	0.015	0.876	0.017
Week 3	2	0.868	0.018	0.882	0.017
Week 3	3	0.808	0.022	0.805	0.023
Week 3	4	0.856	0.019	0.853	0.019
Week 3	5	0.571	0.037	0.565	0.037
Week 3	6	0.801	0.024	0.796	0.025
Week 4	1	0.889	0.014	0.879	0.016
Week 4	2	0.887	0.014	0.886	0.015
Week 4	3	0.837	0.018	0.833	0.019
Week 4	4	0.901	0.013	0.902	0.014
Week 4	5	0.607	0.033	0.601	0.034
Week 4	6	0.818	0.021	0.819	0.021

Note. Results reported for Model 2 which included correlated residuals constrained to be equal across time points and Model 3 which allowed residuals to vary across time points. SE = standard error.

ps < .001 for all factor loadings.

Reliability

Internal consistency reliability was high in Study 1 (α = .90) and at all four time points the DWAI was assessed in Study 2 (αs = .88, .92, .91, and .92, for Weeks 1, 2, 3, and 4, respectively). ¹ Test–retest reliability ranged from ICCs = .55 to .68 in Study 2 (Table 3; ps < .001). Between Week 1 and Week 2, test–retest reliability was ICC = .65.

Validity

Predictive

We examined changes in psychological distress in Study 2 and app utilization in both Study 1 and Study 2 as assessments of predictive validity. The DWAI was associated with greater likelihood of regular app use (weekly or more frequent use) in Study 1 (r = .42, p < .001). Average days of HMP use in Study 2 was 11.92 over the 4 weeks of the study (range = 0-29; Table 1). All DWAI assessments (i.e., Weeks 1 to 4) were associated with greater HMP app use (rs = .17-.22, p < .01). In support of incremental validity, the associations persisted when controlling for preference for the HMP app condition at baseline (rs = .16-.23, ps < .050).

The key test of the value of the DWAI is arguably provided by its ability to predict changes in psychological distress over the course of an unguided smartphone-based intervention. As shown in Table 7, early DWAI scores (Weeks 1 and 2) did not predict post-treatment distress when controlling for baseline distress (βs = −.05 and −.08, respectively, ps > .050). In contrast, higher DWAI scores assessed in Weeks 3 and 4 were associated with lower posttreatment distress (βs = −.17 and −.13, respectively, ps < .01). These effects remained statistically significant when adjusting for multiple tests (i.e., four DWAI scores) using Benjamini and Hochberg’s (1995) false discovery rate (p_FDR) adjustment method. In support of incremental validity, results were essentially unchanged when controlling for preference for the HMP app condition at baseline (Table 7). Constraining the association between DWAI and posttest distress (controlling for pretest distress) to be uniform across DWAI assessment time points showed poorer fit than the unconstrained model in which these associations could vary freely, χ²(3) = 8.90, p = .031. This suggests that the strength of the alliance-outcome association was not uniform across all weeks of Study 2.

OPEN IN VIEWER

Table 7.

Results of Linear Regression Models Predicting Posttreatment Distress.

Model	DWAI Week	β	SE β	95% CILB	95% CIUB	p	p FDR
Unadjusted	Week 1	−0.050	0.048	−0.145	0.045	.301	.301
Unadjusted	Week 2	−0.082	0.050	−0.181	0.018	.107	.143
Unadjusted	Week 3	−0.174	0.049	−0.270	−0.077	<.001	<.001
Unadjusted	Week 4	−0.131	0.046	−0.221	−0.041	.004	.008
Adjusted	Week 1	−0.046	0.050	−0.144	0.052	.358	.358
Adjusted	Week 2	−0.080	0.051	−0.180	0.020	.117	.156
Adjusted	Week 3	−0.178	0.050	−0.277	−0.079	<.001	<.001
Adjusted	Week 4	−0.136	0.047	−0.228	−0.043	.004	.008

Note. β are standardized regression coefficients for Digital Working Alliance Inventory (DWAI) scores assessed at Weeks 1, 2, 3 or 4 as predictors of posttreatment distress controlling for pretreatment distress. Adjusted models added baseline preference for the Healthy Minds Program (HMP) app condition as a covariate to assess incremental validity. SE = standard error; 95% CI = 95% confidence interval lower (LB) and upper (UB) bound; p = p value for DWAI coefficient; pFDR = p value adjusted for multiple tests using Benjamini and Hochberg’s (1995) method.

As the alliance-outcome literature is primarily based on clinical samples (i.e., those seeking psychotherapy; Horvath et al., 2011) and the current trial in Study 2 was conducted in the general population (i.e., school district employees), we conducted a set of exploratory analyses examining whether the strength of the alliance-outcome association varied by baseline psychological distress. Specifically, we added a DWAI by baseline distress interaction term to the linear regression models predicting posttreatment distress from DWAI scores controlling for baseline distress. The interaction term was not significant for DWAI scores assessed at Weeks 1, 2, or 3 (Βs = −0.016 to −0.0024, p > .050). However, the interaction between Week 4 DWAI scores and baseline distress was significant (Β = −0.021, p = .010, p_FDR = .041). As shown in Figure 2, the association between Week 4 DWAI scores and change in psychological distress (residualized change scores) is stronger (more negative) for those with higher baseline distress. As a point of comparison, we examined the Week 4 alliance-outcome association when the sample was restricted to those with PROMIS Depression or PROMIS Anxiety scores above the moderate or higher clinical cutoff (i.e., T ≥ 60; Choi et al., 2014; HealthMeasures, n.d.). In this subsample (n = 150), the standardized regression coefficient for Week 4 DWAI scores predicting post-treatment psychological distress was β = −0.24, 95% confidence interval [−0.38, −0.09], p = .002.

OPEN IN VIEWER

Figure 2.

Week 4 Digital Working Alliance Inventory (DWAI) scores predicting pre–post residualized change in psychological distress in Study 2.

Limitations and Future Directions

The promising psychometric characteristics aside, it is worth asking whether the construct measured by the DWAI is the same as that measured by the WAI and other alliance measures in traditional, face-to-face psychotherapy. DWAI assessed later in Study 2 indeed predicted changes in distress. However, in our view, additional conceptual and empirical work is needed to more fully characterize what precisely working alliance with a smartphone app means. How does the inherently relational nature of the alliance in psychotherapy translate to a relationship with technology? Unguided smartphone apps necessarily lack the interpersonal back-and-forth characteristic of traditional psychotherapy. Key therapeutic processes associated with the alliance (e.g., alliance rupture and repair; Eubanks et al., 2018) are presumably impossible in the absence of interaction. At once, apps can be designed to create a sense of connection between the user with the content and perhaps also with the app creators (e.g., by having a guide or narrator who leads users through app content). Thus, participants may indeed experience a genuine interpersonal connection of sorts.

Clarifying the conceptual overlap and divergence between the DWAI and alliance in psychotherapy is essential to avoiding the nominal fallacy of believing we have explained, measured, or understood the alliance with a smartphone app simply calling a measure “alliance” and showing it predicts outcome. This topic could be addressed in future qualitative studies seeking to more richly describe the relational elements of smartphone app-based interventions. Such efforts may naturally depart from traditional conceptualizations of the alliance in psychotherapy, and may find, for example, that core elements of Bordin’s (1979) model (e.g., bond) are less relevant or manifest differently in app-based interventions. Thus, development of future measures designed to assess alliance in mHealth need not use the WAI as a starting point. It will likewise be crucial to evaluate the degree to which the DWAI or other measures purported to assess alliance with technology demonstrate discriminant and incremental validity in relation to conceptually distinct constructs such as treatment satisfaction and treatment expectancy (Kirsch et al., 2018; Tetzlaff et al., 2005). The extremely high correlation between DWAI scores with perceived app effectiveness in Study 1 (r = .75) highlights the possibility that assessment of the DWAI is strongly linked or perhaps confounded with other constructs drawn from the nomological network of alliance.

It would also be valuable to take up in future research the decades-old debate regarding the causal direction of the alliance-outcome association (DeRubeis & Feeley, 1990; Tang & DeRubeis, 1999). Our study, like those included in Flückiger et al.’s (2018) meta-analysis, merely shows an alliance-outcome association. While it could be worthwhile examining the ways in which alliance and outcome interrelate longitudinally (e.g., using cross-lagged panel models; Falkenström et al., 2013), mHealth technology may allow a more satisfying method for addressing this question. It may be logistically feasible and perhaps ethically permissible to manipulate alliance within the context of an unguided smartphone app. For example, one may randomly assign individuals to low or high alliance versions of a smartphone app (e.g., with or without content designed specifically to increase alliance). The HMP app investigated in Study 2 includes content designed to recreate a “guided” experience even within the unguided app format (e.g., recorded meditation instructions in which a narrator speaks directly to app users; Goldberg, Imhoff-Smith, et al., 2020). If outcomes are assessed more intensively, one could readily examine the impact of shorter term alliance manipulations (e.g., receiving meditation practice instructions designed to augment a sense of connection with the app guide vs. instructions lacking such content) on more proximal measures (e.g., mood, app utilization).

As the alliance showed consistent relationships with usage even when alliance was assessed early in the intervention, it may serve as a valuable predictor of treatment dropout and disengagement. mHealth interventions show notoriously low retention (Christensen et al., 2009; Eysenbach, 2005; Linardon & Fuller-Tyszkiewicz, 2020; Pratap et al., 2020). Alliance scores might therefore be used as a “canary,” alerting of potential disengagement when there may still be time to reengage users. Future studies could test this possibility using alliance scores to implement adaptive trial designs (e.g., sequential multiple assignment randomized trials; Collins et al., 2007). An individual reporting low alliance scores could receive an alliance-boosting intervention component (e.g., encouraging text message, motivation enhancing content) which ultimately could be built into highly responsive mHealth interventions.

This study has several important limitations that are worth noting. First, we evaluated only one of several potential alliance measures that have been proposed for use in an mHealth context. It is entirely possible that other measures (e.g., Berry et al., 2018; Herrero et al., 2020; Miragall et al., 2015) may have performed as well or better than the DWAI. Second, the DWAI only includes six items, which may have limited our ability to reliably detect more complex factor structures. Third, we did not include measures of some key constructs that will be important to examine to further evaluate discriminant validity (e.g., system usability). Fourth, although Study 1 included a fairly diverse sample in terms of gender, race/ethnicity, age, education, and income, Study 2 was more racially/ethnically homogenous and predominantly female. The fact that race/ethnicity was associated with DWAI scores in Study 1 highlights the possibility that this construct may vary across demographic groups. Thus, future studies in highly diverse sample are warranted. Fifth, both studies focused exclusively on smartphone-based meditation apps. While meditation apps are representative of the majority of mental health app use (Wasil et al., 2020), future studies should examine DWAI or other alliance measure within other kinds of smartphone apps.