A Factor Analytic Investigation of the Person-in-Recovery and Provider Versions of the Revised Recovery Self-Assessment (RSA-R)

Abstract

Recovery is understood as living a life with hope, purpose, autonomy, productivity, and community engagement despite a mental illness. The aim of this study was to provide further information on the psychometric properties of the Person-in-Recovery and Provider versions of the Revised Recovery Self-Assessment (RSA-R), a widely used measure of recovery orientation. Data from 654 individuals were analyzed, 519 of whom were treatment providers (63.6% female), while 135 were inpatients (10.4% female) of a Canadian tertiary-level psychiatric hospital. Confirmatory and exploratory techniques were used to investigate the factor structure of both versions of the instrument. Results of the confirmatory factor analyses showed that none of the four theoretically plausible models fit the data well. Principal component analyses could not replicate the structure obtained by the scale developers either and instead resulted in a five-component solution for the Provider and a four-component solution for the Person-in-Recovery version. When considering the results of a parallel analysis, the number of components to retain dropped to two for the Provider version and one for the Person-in-Recovery version. We can conclude that the RSA-R requires further revision to become a psychometrically sound instrument for assessing recovery-oriented practices in an inpatient mental health-care setting.

Keywords

recovery orientation staff perception patient perception assessment confirmatory factor analysis

Although no universal consensus exists on the exact definition of recovery in the mental health field, most professionals suggest that recovery means living a life with hope, meaning and purpose, self-management and autonomy, personal productivity, and community engagement despite living with mental illness (Williams, Almeida, & Knyahnytska, 2015). Recovery perspectives have been adopted by a growing number of mental health service providers across Western societies; however, the change of the mental health-care system from a paternalistic, illness-oriented perspective to a collaborative and autonomy enhancing one is a major shift in service delivery (American Association of Community Psychiatry, 2003), which requires constant empirical monitoring to help the process to move forward.

In recent years, several instruments have been developed to assess recovery orientation in mental health service delivery, the Recovery Self-Assessment (RSA; O’Connell, Tondora, Croog, Evans, & Davidson, 2005) being one of the most widely used. The RSA was designed to gauge the degree to which programs and entire organizations implement recovery-oriented mental health services regardless of psychiatric diagnosis. It is a self-administered tool covering elements of recovery philosophy to help identify strengths and target areas for improvement for health-care providers attempting to offer recovery-oriented services. Despite the widespread employment of this tool in both the research and clinical realm, little effort has been devoted to the thorough investigation of its psychometric properties. Therefore, the aim of the present study was to provide further information on the factor structure of the latest, revised version of this instrument (RSA-R; O’Connell, 2007a, 2007b) examining both the Person-in-Recovery and Provider versions.

Method

Sample and Procedure

The findings of this investigation form a part of a larger project, known as the “Therapeutic Climate Study,” examining the adoption process of recovery philosophy at the Waypoint Centre for Mental Health Care, a partly forensic, specialized psychiatric hospital in Canada for the inpatient treatment of severe mental illness. Informed consent was obtained from each participant, and the protocol for the study has been approved by the research and ethics board of the hospital (HPRA #14.01.01). Currently, data from three waves of data collection for the Therapeutic Climate Study are available (Spring 2014, Fall 2014, and Spring 2015), resulting in a total sample size of 1,447.

For the purposes of the present investigation, only the responses of those participants who were inpatients (n = 135, 88.1% male) or treatment providers (n = 519, 33.3% male) were considered for inclusion (N = 654). Outpatients in the sample were excluded, as the sample size (n = 36) in their case was too low to allow separate analysis, while collapsing them with the inpatient sample would have caused incongruence due to the substantially different treatment setting. Managers were also excluded, as the RSA-R has a distinct version for this employee group, but the present organization does not employ managers in a volume that could allow reliable factor analytic investigations of the CEOs and Directors version of the scale. Finally, employees without direct client contact (e.g., financing) and anyone who had more missing data than is considered allowable in the guidelines provided by the RSA-R developers (O’Connell, 2007c) were excluded from the study. Further details on the study sample are described in Table 1.

Table 1.

Characteristics of the Two Subsamples of the Study.

Treatment Providers (n = 519)		Persons in Recovery (n = 135)
Sex		Sex
Male	173 (33.3)	Male	119 (88.1)
Female	330 (63.6)	Female	14 (10.4)
Missing data	16 (3.1)	Missing data	2 (1.5)
Age^a		Age^a
18 years or younger	1 (0.2)	18 years or younger	—
18–24 years	16 (3.1)	18–24 years	16 (11.9)
25–44 years	212 (40.8)	25–44 years	68 (50.4)
45 years or older	278 (53.6)	45 years or older	48 (35.6)
Missing data	12 (2.3)	Missing data	3 (2.2)
Role		Program^b
Allied health	137 (26.4)	Program A	8 (5.9)
Nurse	264 (50.9)	Program B	8 (5.9)
Medical staff	19 (3.7)	Program C	27 (20.0)
Nonclinical staff^c	88 (17.0)	Program D	24 (17.8)
		Program E	20 (14.8)
		Program F	14 (10.4)
Missing data	11 (2.1)	Program G	13 (9.6)
Years worked in mental health care		Program H	18 (13.3)
1 year or less	24 (4.7)	Other program	3 (2.2)
1–5 years	111 (21.4)	Forensic status
5 years or more	371 (71.5)	Nonforensic	41 (30.4)
Missing data	13 (2.5)	General forensic	20 (14.8)
		Secure forensic	73 (54.1)
		Missing data	1 (0.7)

^aData on age were collected only as categorical information to increase perceived anonymity of the survey. ^bProgram A: A program providing specialized, short-term crisis intervention, and rapid stabilization; Program B: High-security forensic psychiatry program for males with a dual diagnosis of serious mental illness and developmental disorder; Program C: High-security forensic psychiatry program for males with active psychosis, typically diagnosed with schizophrenia, bipolar disorder, treatment-resistant symptoms, and concurrent substance abuse; Program D: A high-security forensic psychiatry program for the long-term care of males with a major mental illness and a personality disorder as well as a step-down program for others transitioning to lower security; Program E: A medium-to-low secure forensic psychiatry program for males and females; Program F: A high-security forensic psychiatry program for males who have come in contact with the law, and require court-ordered assessments to determine fitness to stand trial, and if they may be not criminally responsible due to mental illness; Program G: A voluntary residential program for people with co-occurring substance use disorder and another mental illness; and Program H: Psychosocial rehabilitation program for persons with a serious mental illness.

^cFrom a recovery philosophy perspective, each employee having direct client contact can and does have a role in the recovery process or its failure. This way, the inclusion of security staff (who are providing support to allow patients to travel around the building to attend group therapy, and recreation, and who are coming into contact with patients in the most challenging situations, such as in moments when restraints and/or seclusion is necessary) or other auxiliary personnel (such as dietary staff) is justifiable when investigating the level of recovery orientation in a given organization.

Staff were invited to participate in the study through e-mails, posters, and announcements in the weekly hospital newsletter, asking them to anonymously complete the survey via SurveyMonkey. Hard copies of the survey were also available, and in some instances, the investigators were invited to present the study and collect hard copy responses (anonymously) at staff meetings. Patients were made aware of the study by posters displayed on their units, and individuals were also invited by their program’s psychometrist to complete hard copies (with assistance if required). A voucher for a free coffee/tea at the hospital café was given to all participants as an incentive. Participants were asked to answer the questions based on their last 3 months at the institution and within the context of the program where they primarily worked or received services/treatment.

Measure

The Person-in-Recovery and the Provider versions¹ of the RSA-R were used in the present study (O’Connell, 2007a, 2007b; O’Connell et al., 2005). Both versions include 32 items with minimal differences in wording reflecting patient or treatment provider role (e.g., “staff believe in the ability of program participants to recover” vs. “staff believe that I can recover.”). Response options include a 5-point Likert-type scale ranging from 1 (strongly disagree) to 5 (strongly agree) with two auxiliary options (“not applicable” and “do not know”) that are to be coded as missing data. The instrument consists of the following six domains (total number of items, number of items with missing data allowed, Cronbach’s α in the present sample for the Person-in-Recovery version, and Cronbach’s α for the Provider version): Life Goals (11, 3, 0.91, and 0.90), Involvement (5, 1, 0.88, and 0.88), Diversity of Treatment Options (5, 1, 0.83, and 0.84), Choice (5, 1, 0.78, and 0.75), Individually Tailored Services (4, 1, 0.83, and 0.76), and Inviting Space (2, 0, 0.67, and 0.63).

The Life Goals domain of the instrument refers to the extent to which staff helps with the development and attainment of life goals defined by the person in recovery. Involvement refers to the degree to which staff and patients perceive that patients are involved in their treatment and in program/organization-level decisions. Diversity of Treatment Options domain of the instrument refers to the extent to which the institution provides a variety of treatment options and helps with involvement in nonmental health activities. The Choice Subscale refers to the degree to which staff and patients feel that choices are available to patients in their treatment and whether their choices are respected. The domain of Individually Tailored Services refers to the perception that clinical services are tailored to individual needs, culture, and preferences. Finally, the Inviting Space Subscale, a special feature of the revised version of the instrument assesses whether persons in recovery feel invited and dignified by the physical surroundings and staff of the treatment program.

Statistical Analyses

All analyses were performed separately for the Person-in-Recovery and Provider versions. Confirmatory factor analyses were carried out using Mplus Version 7.1, while all other analyses were conducted using SPSS Version 23. First, in order to investigate the fit of the data with theory-driven factor structures, four competing confirmatory factor analytic models were tested (Figure 1). Model 1 was a single-factor solution with one general “recovery orientation” factor responsible for all 32 item responses. Model 2 was a first-order, six-factor solution representing the six correlating factors of recovery orientation. Model 3 was a second-order solution where, in addition to the six first-order factors, a second-order recovery orientation factor was incorporated. Finally, Model 4 was a bifactor model (Reise, Moore, & Haviland, 2010; Reise, Scheines, Widaman, & Haviland, 2013), which—contrary to traditional second-order models—allows all items to load directly onto a general factor (recovery orientation) and a domain-specific factor (e.g., Diversity of Treatment Choices or Life Goals) simultaneously.

Figure 1.

Sematic graphical representation of the factor analytic models in the study. Number of items and factors does not reflect the true characteristics of the instrument examined to facilitate visibility. A1–A2, B1–B2, C1–C2, and D1–D2 = items; F1–F4 = first-order factors; H = higher/second-order factor; S1–S4 = specific factors in a bifactor model; and G = global factor in a bifactor model.

Considering the ordinal nature of the observed variables, the robust maximum likelihood estimation was used (Li, 2016). Model fit was evaluated based on the χ² test (nonsignificant results indicating adequate fit), the Tucker–Lewis index and the comparative fit index (values greater than .95 suggest good fit for both indexes), the root mean square error of approximation (values below .05 indicate good fit), the standardized root mean square residual (ideally below .05), and sample size–adjusted Bayesian information criterion (without absolute cutoff, lower values indicate better fit; Hooper, Coughlan, & Mullen, 2008).

After the confirmatory factor analyses, a set of principal component analyses were also run. Since subscales of the RSA-R were reported to correlate with each other (Salyers, Tsai, & Stultz, 2007) and because this method does not require the extracted factors to be independent, the oblimin rotation was chosen when conducting the principal component analyses. To determine the number of components to retain, parallel analyses (Horn, 1965) were also conducted in both subsamples. This technique is a Monte Carlo–based simulation method that compares the eigenvalues from the study sample with those obtained from a random data set. A component is to be retained if its eigenvalue exceeds the 1 derived from the 95th percentile of the random sample. Random eigenvalues for the comparisons were generated using a web-based application (Patil, Singh, Mishra, & Donavan, 2007).

Results

Results of the confirmatory factor analytic investigations showed that none of the theoretically plausible models fit the data well: No fit indices indicated good fit for any of the models in either population (Table 2). Standardized coefficients greater than 1 for the relationships among latent variables were also observed in Models 2 and 3 indicating suboptimal model specification. Standardized factor loadings from each model are available as Online Supplementary Material to this article on the publisher’s website.

Table 2.

Competing Factor Analytic Models of the Person-in-Recovery and Provider Versions of the Revised Recovery Self-Assessment.

Model Tested		χ²	df	p	TLI	CFI	RMSEA	90% CI for RMSEA	SRMR	SSA BIC
Provider version	Model 1	2,018.9	464	<.001	.791	.804	.080	[.077, .084]	.061	42,633.9
	Model 2	1,737.8	449	<.001	.821	.838	.074	[.071, .078]	.059	42,310.3
	Model 3	1,819.8	458	<.001	.814	.829	.076	[.072, .079]	.060	42,390.5
	Model 4	1,556.1	432	<.001	.838	.859	.071	[.067, .075]	.053	42,105.0
Person-in-Recovery version	Model 1	913.7	464	<.001	.790	.804	.085	[.077, .093]	.071	11,957.0
	Model 2	849.1	449	<.001	.807	.825	.081	[.073, .090]	.069	11,901.0
	Model 3	855.2	458	<.001	.812	.827	.080	[.072, .088]	.069	11,892.8
	Model 4	This complex model could not be reliably estimated in this subsample

Note. TLI = Tucker–Lewis index; CFI = comparative fit index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual; SSA BIC = sample size–adjusted Bayesian information criterion.

To explore what data-driven factor structure would be more appropriate for the instrument, a separate principal component analysis was conducted for each subsample. The Bartlett’s test of sphericity was significant (p < .001) and the Kaiser–Mayer–Olkin measure of sampling adequacy indicated acceptable sampling in both subsamples (.91 for providers and .96 for persons in recovery). In the provider sample, five components emerged with eigenvalues greater than 1, while in the patient sample, four components were extracted according to the same criterion. When considering the results of the parallel analysis, number of components to retain dropped to two for the Provider version and one for the Person-in-Recovery version. Detailed results for the complete set of principal component analyses are available as Online Supplementary Material to this article on the publisher’s website.

Discussion

The aim of this study was to investigate the psychometric properties of the RSA-R—a tool developed to assess the extent to which a mental health facility delivers recovery-oriented services—with advanced statistical procedures. To the best of our knowledge, this is the first study employing confirmatory factor analytic techniques to investigate the factor structure of this instrument.

The findings of the present study were not able to confirm the psychometric adequacy of the questionnaire. Although, similar to previous studies (O’Connell et al., 2005; Salyers et al., 2007), some traditional scale characteristics were acceptable in the present sample (e.g., Cronbach’s αs of the subscales, corrected item total correlations of the items—only partly reported in the main text but available in detail as Online Supplementary Material to this article), the confirmatory factor analyses showed that neither of the theoretically feasible structures fit the data well for either the Provider or the Person-in-Recovery versions. Principal component analyses could not replicate the structure obtained by the scale developers and produced unclear item-to-factor assignments.

Potential reasons for the discrepancy between the results of the developers and the present study are differences between the studies in terms of treatment setting (combined analysis of inpatient and outpatient service providers vs. focus on inpatients only) and analytic strategy (pooled vs. stratified examinations of the Provider and Person-in-Recovery versions of the instrument). Findings of the present study, however, are in line with those of a recent study, investigating a shortened Person-in-Recovery version of the instrument in an outpatient sample with advanced statistical methods (Barbic, Kidd, Davidson, McKenzie, & O’Connell, 2015). Authors of this study reported unfavorable characteristics for item composition (items cover only the lower continuum of the recovery construct as hope but not the upper end as realization of goals, for instance), scoring options (respondents unable to distinguish among the three middle response options), and wording (items containing multiple elements of a concept simultaneously resulting in potential confusion in the responder).

Strength and limitations of the present study should also be noted. Main strengths are the simultaneous examination of the treatment Provider and Person-in-Recovery versions of the instrument, the use of advanced statistical procedures, and the employment of a statistical software efficiently handling missing data. Limitations include the relatively low sample size for the patient subsample and the uncertainty whether patients were capable of fully understanding the content of each item (some patients in the institution suffer from temporary or permanent cognitive deficits, an aspect not assessed in the survey). However, the fact that the results were very similar to those obtained in the substantially larger staff sample suggests that these limitations of the patient sample were less likely to affect the main conclusions of this investigation.

We can conclude that the findings of the present study indicate that despite its good face validity, acceptable psychometric properties according to traditional terms, and extensive use in program evaluation, the RSA-R requires further development. Revising the response options, rewording complex items causing confusion, and completing the item pool with recovery-related contents not covered by current items are all possible options to increase the psychometric quality of the instrument.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

The online supplementary material available at .

Note

References

American Association of Community Psychiatry. (2003). AACP guidelines for recovery oriented services. Retrieved from http://www.communitypsychiatry.org/assets/docs/publications/clinical_and_administrative_tools_guidelines/ROSGuidelines.pdf

Barbic

S. P.

Kidd

S. A.

Davidson

McKenzie

O’Connell

M. J.

(2015). Validation of the brief version of the Recovery Self-Assessment (RSA-B) using Rasch measurement theory. Psychiatric Rehabilitation Journal, 38, 349–358. doi:10.1037/prj0000139

Hooper

Coughlan

Mullen

M. R.

(2008). Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6, 53–60.

Horn

J. L.

(1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30, 179–185. doi:10.1007/BF02289447

C.-H.

(2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48, 936–949.

O’Connell

(2007a). RSA-R: Person in recovery version. Retrieved from http://medicine.yale.edu/psychiatry/prch/tools/RSA_Person_in_Recovery_tcm800-204208_tcm800-284-32.pdf

O’Connell

(2007b). RSA-R: Provider version. Retrieved from http://medicine.yale.edu/psychiatry/prch/tools/RSA_Provider_tcm800-204210_tcm800-284-32.pdf

O’Connell

(2007c). Scoring instructions for the RSA and RSA-R. Retrieved from http://medicine.yale.edu/psychiatry/prch/tools/RSA_Factors_File2_tcm800-204214_tcm800-284-32.pdf

O’Connell

Tondora

Croog

Evans

Davidson

(2005). From rhetoric to routine: Assessing perceptions of recovery-oriented practices in a state mental health and addiction system. Psychiatric Rehabilitation Journal, 28, 378–386. doi:10.2975/28.2005.378.386

10.

Patil

V. H.

Singh

S. N.

Mishra

Donavan

D. T.

(2007). Parallel analysis engine to aid determining number of factors to retain. Retrieved from http://smishra.faculty.ku.edu/parallelengine.htm

11.

Reise

S. P.

Moore

T. M.

Haviland

M. G.

(2010). Bifactor models and rotations: exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92, 544–559. doi:10.1080/00223891.2010.496477

12.

Reise

S. P.

Scheines

Widaman

K. F.

Haviland

M. G.

(2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73, 5–26. doi:10.1177/0013164412449831

13.

Salyers

M. P.

Tsai

Stultz

T. A.

(2007). Measuring recovery orientation in a hospital setting. Psychiatric Rehabilitation Journal, 31, 131–137. doi:10.2975/31.2.2007.131.137

14.

Williams

C. C.

Almeida

Knyahnytska

(2015). Towards a biopsychosociopolitical frame for recovery in the context of mental illness. British Journal of Social Work, 45, i9–i26. doi:10.1093/bjsw/bcv100

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.55 MB