Examining the Application of the Opening Minds Survey in the Community Health Centre Setting

Abstract

French

Objective:

Stigma has been identified as a complex and problematic issue. It acts as a major barrier to accessing care and can exacerbate the experience of a health condition, particularly for clients with mental illness and substance use issues. Scales designed to assess stigmatising attitudes towards those with mental illness and substance use problems among health care providers are necessary to evaluate programs designed to reduce that stigma. The goal of this study was to evaluate the internal reliability and external validity of the Opening Minds Survey for Health Care Providers (OMS-HC).

Methods:

The current study examined the use of the OMS-HC in assessing stigma held by Community Health Centre (CHC) staff towards clients with mental and/or substance use problems. Participants represented staff from 6 CHCs in the Greater Toronto Area (n = 190).

Results:

The OMS-HC was found to have acceptable internal reliability for the 15-item version of the scale (α = 0.766) and mixed reliability for its subscales (α = 0.792-0.673). Confirmatory factor analysis showed good absolute (root mean square error of approximation = 0.013) and relative fit (Tucker-Lewis index = 0.996) for the current data. The OMS-HC was also shown to correlate with a series of scales commonly used in stigma research.

Conclusions:

After testing for internal validity and comparing the OMS-HC to other commonly used scales for assessing stigma and attitudes concerning recovery, the scale was found to be appropriate for the CHC setting and may be advantageous over the use of multiple scales.

Keywords

stigma psychometrics opening minds

The stigma of mental illness presents an important challenge to health care providers. Research has shown that those who experience stigma as a result of a mental illness or substance use problems are less likely to access health care for these conditions and are less likely to engage fully with treatment programs.^1,2 Stigma impedes care at different levels. First, stigma held by a health care professional towards patients with a mental illness may negatively affect the service that the health care professional provides. For example, stigmatising attitudes held by health care professionals towards those with mental illnesses or substance use issues can lead them to misinterpret physical health concerns as side effects of a mental illness in a process referred to as ‘diagnostic overshadowing’.³ Feelings of distrust and social distance between the health care worker and patient have also been shown to negatively affect health care provision.^4
–6

Stigma also negatively affects health care through the experience of self-stigma on the part of the patient. The internalisation of stigma conveyed by health professionals can discourage a client from seeking treatment for mental health and substance use problems.^7,8 For example, in a 2013 study, Tucker et al.⁹ found that feelings of self-stigma were inversely related to both intentions of seeking help and attitudes towards professional psychological help. Self-stigma can also result in low self-esteem on the part of the patients, affecting their ability to recover from or manage a mental illness or substance use problem. This presents a serious problem to any recovery program as belief in one’s ability to successfully complete a treatment program is a strong predictor of program adherence.¹⁰ Feelings of self-stigma have also been shown to negatively affect mental health patients’ quality of life, which in turn has important consequences for exacerbating current conditions and developing new ones.¹¹

Observing the effectiveness of the Opening Minds Survey for Health Care Providers (OMS-HC) in the Community Health Centre (CHC) context has important implications for health care provision in Canada. CHCs exist for the expressed goal of meeting the health care needs of citizens who face barriers to health care. CHCs are designed to provide multiple types of primary health care, community health care, and social support programs.¹² Addressing stigma at CHCs is of particular concern as these locations provide health care to the most marginalised groups.¹³ Stigma has been shown to have particularly negative effects on treatment seeking of marginalised groups such as racialized populations,¹⁴ economically disadvantaged groups,¹⁵ and veterans.¹⁶

The OMS-HC was developed to determine the degree of stigma held by health providers towards those with mental illnesses.^17,18 The goal of the scale is to determine the efficacy and effectiveness of antistigma programs in efforts to diminish stigma’s impact on health care provision. The OMS-HC was developed for use in the Canadian context and was designed to use relatively few items to examine a number of factors related to stigma,¹⁹ making its use practical in the routine assessment of antistigma interventions. These factors include attitudes about disclosure of mental illness, attitudes towards those who have an issue related to mental illness, and desired social distance from those with mental health problems. The OMS-HC has been applied to a variety of research topics, including borderline personality disorder²⁰ and online educational content for physicians.²¹ In addition to use in a research setting, the OMS-HC may also be useful in an educational context and in program evaluations. As noted by Kassam et al.,¹⁹ the OMS-HC requires greater external validation, and to date, no studies have compared the performance of the OMS-HC with other validated stigma scales.

The current study seeks to address this gap by pursuing 3 research goals: 1) examining the internal consistency of the OMS-HC and its subscales through the use of Cronbach’s α, 2) confirming the validity of the factor structure suggested by Modgill et al.¹⁷ through the use of confirmatory factor analysis, and 3) determining its external validity by testing the correlations of the OMC-HC with several other scales that measure stigma and attitudes surrounding recovery of those experiencing mental health and/or substance use issues.

Methods

Sample

The current study is based on the baseline results of a randomised cluster trial aimed at reducing stigma towards clients experiencing issues related to mental health or substance use in a primary health care setting. Data were collected from staff members over the age of 18 years at 6 CHCs located in the Greater Toronto Area. The collection period spanned over 3 months in 2014. Inclusion criteria required that participating staff members had direct contact with clients through their CHC duties in the past year. Self-administered questionnaires were offered to all current staff and completed on a voluntary basis. Of the 489 staff employed by the CHCs at the time of the study, 190 completed the survey, giving a response rate of 38.9%. The data reported represent baseline data for a 3-year study designed to test the effectiveness of antistigma intervention at CHCs. Demographic and CHC roles of the staff are described in table 1.

Measures

The first section of the survey included a battery of scales that measure stigma and recovery of people with mental health and/or substance use problems. The scale of interest to the current study is the 15-item OMS-HC.¹⁹ This scale has a possible range of 15 to 75, with higher scores indicating negative views of clients with mental health or substance issues, and takes approximately 4 minutes to complete. Several other scales designed to measure stigma towards mental health care clients were included to assess the external validity of the OMS-HC. These scales include the 16-item Mental Illness Clinician’s Attitudes Scale (MICA),²² which ranges from scores of 16 to 96, with higher scores indicating more negative views of clients with mental health or substance issues; the Modified Bogardus Social Distance Scale for both schizophrenia and heroin dependence,⁴ which has a score ranging from 6 to 24, with higher scores indicating greater feelings of social distance; the Recovery Assessment Scale (RAS) for both mental illness and addictions,²³ which ranges from 13 to 117, with higher scores indicating negative views about the recovery potential of clients with mental health or substance issues; the Recovery Self-Assessment (RSA-R) Provider Version,²⁴ which ranges from 32 to 160, with higher scores indicating better recovery-oriented practices at the participants’ places of work; and the Marlowe-Crowne Social Desirability Scale short form (MCSDS) to control for social desirability response bias, which ranges from 0 to 13, with higher scores indicating lower possible social desirability response bias.²⁵ The survey also includes participant demographics, the respondents’ role at their CHC, and personal experience with mental health and substance use problems.

Table 1.

Descriptive Statistics.

Variable	n (%)
Sex
Male	29 (15.3)
Female	160 (84.2)
Age, y
13-18	2 (1.1)
19-25	10 (5.3)
26-34	56 (29.5)
35-44	48 (25.3)
45-54	39 (20.5)
55-64	30 (15.8)
65 and over	2 (1.1)
Work role at Community Health Centre
Primary health care team	45 (23.7)
Allied health care team	37 (19.5)
Community health team	34 (17.9)
Administrative	31 (16.3)
Maintenance or housekeeping	1 (0.5)
Other	39 (20.5)
Additional training in mental health or addictions
No	124 (65.3)
Yes	61 (32.1)

Analysis

To account of missing data on the various scales examined, the study mean scores were taken of all answered questions in each scale, which were then multiplied by the number of items in that scale if that respondent gave valid answers to at least 80% of the scale. Respondents who answered less than 80% of a scale were removed from analysis of that scale. Internal consistency of the OMS-HC and its subscales was examined using Cronbach’s α. This study uses the values for acceptability proposed by Gliem and Gliem,²⁶ who suggested that an α value of >0.7 reflects acceptable internal consistency. Confirmatory factor analysis of the factor structure proposed by Modgill et al.¹⁷ was also performed. Associations of the OMS-HC, its subscales, and other scales related to stigma and recovery from mental health and addiction problems were assessed using Pearson’s correlations. Descriptive and correlational analyses were conducted using SPSS 20 (SPSS, Inc., an IBM Company, Chicago, IL). Confirmatory factor analysis was performed using the lavaan package for the R statistical computing environment.²⁷

Results

Table 2 describes the properties of the scales used in the correlation analysis. Most scales showed acceptable reliability. However, the RAS for mental illness and the MCSDS showed poor reliability. While the poor reliability of the MCSDS might be expected as it is a control for social desirability bias, the relatively low reliability of the RAS for mental illness is unexpected, especially considering the good reliability of the RAS for addiction, which has very similar wording. No scales showed significant skewness or kurtosis.

Table 2.

Descriptive Statistics for Study Scales.

	n	Minimum	Maximum	Mean	Standard Deviation	Cronbach’s α
OMS-HC	187	15	51	30.38	6.72	0.77
MICA	188	19	59	35.01	7.85	0.71
Bogardus: Schizophrenia	189	6	22	11.64	3.18	0.72
Bogardus: Heroin	189	6	24	14.39	3.48	0.72
RAS: Mental illness	189	13	92	49.86	14.21	0.50
RAS: Addiction	186	13	95	53.43	14.33	0.89
RSA-R	162	53	160	128.48	18.19	0.80
MCSDS	180	2	13	8.97	2.57	0.43

MCSDS, Marlowe-Crowne Social Desirability Scale; MICA, Mental Illness Clinician’s Attitudes Scale; OMS-HC, Opening Minds Survey for Health Care Providers; RAS, Recovery Assessment Scale; RSA-R, Recovery Self-Assessment Provider Version.

Internal Consistency

Examination of the internal consistency showed acceptable consistency for the complete 15-item OMS-HC and for each of its subscales. Table 3 shows that the entire scale showed a Cronbach’s α score of 0.77. The Cronbach’s α for 3 subscales (i.e., attitudes towards those with mental illness, attitudes towards the disclosure of mental illness, and the social distance from those with mental illness) showed αs of 0.79, 0.67, and 0.72, respectively. The interitem correlation for the scale ranged from a maximum of 0.49 to a minimum of –0.11 with a mean of 0.15. Table 3 also shows that the 15-item OMS-HC showed acceptable internal consistency across all occupation categories with the exception of the administrative staff.

Table 3.

Comparison of Internal Consistency across Teams in the Community Health Centre (Cronbach’s α).

	OMS-HC	Attitudes	Disclosure	Social Distance
Overall sample (n = 187)	0.766	0.792	0.673	0.720
Primary health care (n = 44)	0.788	0.719	0.693	0.798
Allied health care (n = 36)	0.758	0.602	0.718	0.644
Community health care (n = 34)	0.808	0.655	0.691	0.706
Administration (n = 30)	0.682	0.909	0.318	0.673
Other (n = 39)	0.802	0.679	0.769	0.683

OMS-HC, Opening Minds Survey for Health Care Providers.

Confirmatory factor analysis was conducted on the subscales to test their appropriateness as identified by Modgill et al.¹⁷ A diagonally weighted least squares estimator was used to adjust for the ordinal nature of the Likert scale questions used in the OMS-HC. The 3-factor model showed a χ² of 89.904 (P = 0.394). The results showed good fit on the Tucker-Lewis index (0.996), the comparative fit index (0.996), and the root mean square error of approximation (0.013). The standardised root mean square residual shows acceptable fit at 0.070²⁸). Table 4 lists the standardised estimates for the confirmatory factor analysis.

Table 4.

Item Loadings from Confirmatory Factor Analysis.

Factor	Standardized Estimate
Factor 1: Attitudes of health care providers towards people with mental illness
I am more comfortable helping a person who has a physical illness than I am helping a person who has a mental illness. (1 of 20)	0.539
Despite my professional beliefs, I have negative reactions towards people who have mental illness. (12 of 20)	0.325
There is little I can do to help people with mental illness. (13 of 20)	0.508
More than half of people with mental illness don’t try hard enough to get better. (14 of 20)	0.469
Health care providers do not need to be advocates for people with mental illness. (18 of 20)	0.263
I struggle to feel compassion for a person with a mental illness. (20 of 20)	0.446
Factor 2: Disclosure/help seeking
If I were under treatment for a mental illness I would not disclose this to any of my colleagues. (4 of 20)	0.338
I would see myself as weak if I had a mental illness and could not fix it myself. (6 of 20)	0.573
I would be reluctant to seek help if I had a mental illness. (7 of 20)	0.519
If I had a mental illness, I would tell my friends. (10r of 20)	0.461
Factor 3: Social distance
If a colleague with whom I work told me they had a managed mental illness, I would be as willing to work with him/her. (3r of 20)	0.439
Employers should hire a person with a managed mental illness if he/she is the best person for the job. (8r of 20)	0.382
I would still go to a physician if I knew that the physician had been treated for a mental illness. (9r of 20)	0.709
I would not want a person with a mental illness, even if it were appropriately managed, to work with children. (17 of 20)	0.591
I would not mind if a person with a mental illness lived next door to me. (19r of 20)	0.55
Covariances
Factor 1–factor 2	0.526
Factor 1–factor 3	0.814
Factor 2–factor 3	0.388

Note. r = reverse coded.

Table 5 displays a series of Pearson’s correlations that were run to compare the OMS-HC and its subscales to other commonly used scales that measure stigma and attitudes towards patient recovery among health care providers. First, the OMS-HC scores are strongly related to MICA scores with a correlation of 0.746 (P < 0.001). This is to be expected as several of the items used in the OMS-HC are adapted from the MICA.¹⁹ A strong correlation is also observed between the OMS-HC and the modified Bogardus social distance scales for schizophrenia and heroin dependence at 0.565 and 0.461, respectively. Both correlations were significant at the P < 0.001 level. The OMS-HC also correlates positively with the RAS for mental illness and addiction at 0.179 and 0.191, respectively, with both correlations being significant at the P < 0.01 level. However, these correlations are fairly weak. The OMS-HC does not show a significant correlation with the RSA-R scale. The OMS-HC also does not correlate with the MCSDS. The MCSDS was included as a control for social desirability response bias, suggesting the above relationships are not likely to be affected by such bias.

Table 5.

Correlations between Opening Minds Survey for Health Care Providers and Other Scales.

Study Scale	Pearson Correlation	P Value	n
MICA	0.746	0.000	186
Bogardus: Schizophrenia	0.565	0.000	186
Bogardus: Heroin dependence	0.461	0.000	186
RAS: Mental illness	0.179	0.009	186
RAS: Addiction	0.191	0.003	183
RSA-R	–0.073	0.202	160
MCSDS	0.100	0.515	187

MCSDS, Marlowe-Crowne Social Desirability Scale; MICA, Mental Illness Clinician’s Attitudes Scale; RAS, Recovery Assessment Scale; RSA-R, Recovery Self-Assessment Provider Version.

Correlations between subscales of the OMS-HC and scales designed to measure similar constructs were also examined. First, the attitudes subscale of the OMS-HC was compared to the “knowledge of mental illness” and “views of health/social care field and mental illness” subscales of the MICA. The OMS-HC attitudes subscale showed a moderate positive correlation with both MICA subscales. They show a score of 0.561 (P < 0.001) for the MICA knowledge subscale and 0.524 (P < 0.001) for the MICA attitudes subscale.²⁹ The OMS-HC disclosure subscale was also compared to the disclosure subscale found in the MICA. These 2 disclosure subscales showed a positive correlation of 0.585 (P < 0.001).

The social distance subscale of the OMS-HC was compared to the 2 Bogardus social distance scales. The OMS-HC social distance subscale showed a significant positive correlation with the Bogardus social distance scales for schizophrenia and heroin dependence scales at 0.497 and 0.350, respectively (P < 0.001 for both correlations).

Discussion

The first goal of the current study was to examine the internal reliability of the OMS-HC and its subscales when used with staff from CHC. The tests showed acceptable reliability for the 15-item scale as well as for the 3 subscales. The disclosure subscale showed the lowest consistency score at α = 0.67, which is below what is typically considered the acceptable range. The confirmatory factor analysis indicates that the factor structure identified by Modgill et al.¹⁷ demonstrated an acceptable fit to the current data, supporting the use of the subscales and their application to the CHC setting. Determining that the OMS-HC is an appropriate tool for the study of CHC staff is valuable, as these health centres have significant contact with marginalised populations. This has important implications for health care provision because marginalised populations have disproportionately high rates of both mental health and substance use problems, and reducing stigma is an important step in increasing access to health care.^30,31 This means that there is much to be gained by having valid and reliable measures of stigma among health care workers in the CHC setting. The OMS-HC’s ability to perform these tasks is useful not only in a research context but also for development and evaluation of educational programs aimed at addressing stigma within multiple primary health care settings.

The current study also sought to compare the performance of the OMS-HC to other survey tools commonly used to measure stigma towards those with mental health and substance use problems among health care workers and their attitudes and behavioural intentions regarding client recovery from such conditions. It was found that the OMS-HC showed significant positive correlations with the MICA, both Bogardus social distance scales, and both RAS. The strongest correlation observed was between MICA and OMS-HC, which is to be expected as both scales seek to measure health care providers’ behavioural intent towards clients with mental health and/or substance use issues. In addition, the OMS-HC contains several items from the MICA. However, the OMS-HC showed better internal consistency than the MICA, suggesting that if they measure similar concepts, the OMS-HC may be preferable. The OMS-HC showed moderate correlations with the Bogardus social distance scales. The lower correlation is expected as social distance is one part of the more complex concept of stigma that the OMS-HC tries to capture. The significant yet weaker correlations with the RAS are also expected. This study observed no significant correlation between the OMS-HC and the RSA-R. The RSA-R seeks to measure the extent to which health care providers see their health care setting as meeting the principles of recovery-oriented care. The lack of correlation between these scales suggests that attitudes towards clients have relatively little association with a more objective appraisal of policies and practices of a health care setting.

As noted by Modgill et al.,¹⁷ there is some concern that the self-report nature of the OMS-HC could lead to underestimating levels of stigma. They suggest that a social desirability bias might lead participants to artificially lower their scores. However, no significant correlation with the MCSDS scale was observed. While the lack of correlation between the 2 scales does suggest that OMS-HC scores are not significantly affected by social desirability bias in this sample, the low internal validity of the MCSDS does mean that further investigation is required.

The results of the current study also support the use of the factor structure of the OMS-HC. Each of the subscales proposed by Modgill et al.¹⁷ showed acceptable internal validity as indicated by the Cronbach α statistic. In addition, each of the subscales correlated significantly with other survey tools and subscales designed to detect similar constructs. The social distance subscale correlated significantly with both Bogardus social distance scales, the attitudes subscale correlated significantly with the MICA knowledge and views subscales, and the disclosure subscale correlated significantly with the MICA disclosure subscale. These relationships help bolster the external validity of the subscales of the OMS-HC. The validity and reliability of the subscales of the OMS-HC as shown through these correlations and the confirmatory factor analysis suggest that using the OMS-HC could be used in place of several different scales to capture similar information. In addition, in relation to the Bogardus social distance scale for heroin dependence and the RAS addictions scale, the correlations suggest that the OMS-HC could be extended to evaluating health care programs targeted towards clients with poor mental health and substance use problems.

Limitations

One important limitation of the current study is the small sample size. With just 190 participants, it is difficult to compare the consistency of the OMS-HC and its subscales across the different demographics and CHC role-related variables. While this small sample size did affect the range of analyses that were possible for the current study, the sample size was sufficient for the analyses that are conducted above. Similarly, the response rate for the current study is low at approximately 40%. This rate is low compared to earlier work on the scale performed by Modgill et al.,¹⁷ who achieved a rate of 56.7%. Information of the characteristics of those who did not complete the survey was not available to the research team, making it difficult to speculate on the levels of stigma among nonrespondents. This low rate of participation makes nonresponse bias a possible problem, and interpretation of the results should be made with this in mind. The ability of the OMS-HC to detect meaningful change in attitudes in a trial setting has not been properly addressed in the current use of the scale. Future research is needed to determine whether the OMS-HC is appropriate for this purpose.

Conclusion

Addressing stigma towards those with mental health and addictions problems is an efficient and cost-effective way to increase access to health care for those individuals.³² In addition, CHCs are designed in such a way that they receive a high proportion of clients who experience mental health or addictions problems. The findings of the current research suggest that the OMS-HC is an appropriate tool for measuring stigma towards clients in the CHC setting. The findings of the current study also demonstrate the external validity of the OMS-HC through its comparison to other scales that are widely used in the evaluation of client care in mental health. The well-fitting factor structure of the scale and correlation of the subscales with other measures of social distance, disclosure, and attitudes towards clients experiencing mental illness or substance use problems suggest that the OMS-HC could be used in place of multiple scales in future research.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The authors received support from the Canadian Institutes of Health research.

References

Corrigan

. How stigma interferes with mental health care. Am Psychol. 2004;59(7):614–625.

Thornicroft

. Stigma and discrimination limit access to mental health care. Epidemiol Psychiatr Soc. 2008;17(1):14–19.

Jopp

Keys

. Diagnostic overshadowing reviewed and reconsidered. Am J Ment Retard. 2001;106(5):416–433.

Link

Cullen

Frank

. The social rejection of former mental patients: understanding why labels matter. Am J Sociol. 1987;92(6):1461–1500.

Bentz

Hollister

Kherlopian

. Attitudes of social distance and social responsibility for mental illness: a comparison of teachers and the general public. Psychology Sch. 1970;7(2):198–203.

Tsao

CIP

Tummala

Roberts

. Stigma in mental health care. Acad Psychiatry. 2008;32(2):70–72.

Al-Krenawi

Graham

Dean

. Cross-national study of attitudes towards seeking professional help: Jordan, United Arab Emirates (UAE) and Arabs in Israel. Int J Soc Psychiatry. 2004;50(2):102–114.

Hoge

Castro

Messer

. Combat duty in Iraq and Afghanistan, mental health problems, and barriers to care. N Engl J Med. 2004;351(1):13–22.

Tucker

Hammer

Vogel

. Disentangling self-stigma: are mental illness and help-seeking self-stigmas different? J Couns Psychol. 2013;60(4):520–531.

10.

Verhaeghe

Bracke

Bruynooghe

. Stigmatization and self-esteem of persons in recovery from mental illness: the role of peer support. Int J Soc Psychiatry. 2008;54(3):206–218.

11.

Tang

. Quality of life and self-stigma in individuals with schizophrenia. Psychiatr Q. 2012;83(4):497–507.

12.

Albrecht

. Community health centres in Canada. Leadersh Health Serv. 1998;11(1):5–10.

13.

Greening

Wild

Caine

. Expanding a Canadian community health centre. Int J Health Care Qual Assur. 2014;27(3):231–243.

14.

Brown

Conner

Copeland

. Depression stigma, race, and treatment seeking behavior and attitudes. J Commun Psychology. 2010;38(3):350–368.

15.

Ssebunnya

Kigozi

Lund

. Stakeholder perceptions of mental health stigma and poverty in Uganda. BMC Int Health Hum Rights. 2009;9(5):1–9.

16.

Mittal

Drummond

Blevins

. Stigma associated with PTSD. Psychiatr Rehabil J. 2013;36(2):86–92.

17.

Modgill

Patten

Knaak

. Opening Minds Stigma Scale for Health Care Providers (OMS-HC): examination of psychometric properties and responsiveness. BMC Psychiatry. 2014;14(120):1–10.

18.

Patten

Remillard

Phillips

. Effectiveness of contact-based education for reducing mental illness-related stigma in pharmacy students. BMC Med Educ. 2012;12(120):1–9.

19.

Kassam

Papish

Modgill

. The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: the Opening Minds Scale for Health Care Providers (OMS-HC). BMC Psychiatry. 2012;12(62):1–12.

20.

Knaak

Szeto

Fitch

. Stigma towards borderline personality disorder: effectiveness and generalizability of an anti-stigma program for healthcare providers using a pre-post randomized design. Borderline Personal Disord Emot Dysregul. 2015;2(9):1–8.

21.

Knaak

Ungar

Patten

. Seeing is believing: biological information may reduce mental health stigma amongst physicians. Aust N Z J Psychiatry. 2015;49(8):751–752.

22.

Kassam

Glozier

Leese

. Development and responsiveness of a scale to measure clinicians’ attitudes to people with mental illness (medical student version). Acta Psychiatr Scand. 2010;122(2):153–161.

23.

Corrigan

Salzer

Ralph

. Examining the factor structure of the recovery assessment scale. Schizophr Bull. 2004;30(4):1035–1041.

24.

O’Connell

Tondora

Croog

. From rhetoric to routine: assessing perceptions of recovery-oriented practices in a state mental health and addiction system. Psychiatr Rehabil J. 2005;28(4):378–386.

25.

Reynolds

. Development of reliable and valid short forms of the Marlowe-Crowne Social Desirability Scale. J Clin Psychology. 1982;38(1):119–125.

26.

Gliem

. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales. Paper presented at Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education; 2003 October; Columbus, OH.

27.

Rosseel

. lavaan: an R package for structural equation modeling. J Stat Softw. 2012;48(2):1–36.

28.

Hooper

Coughlan

Mullen

. Structural equation modelling: guidelines for determining model fit. Electron J Bus Res. 2008;6(1):53–60.

29.

Healey

. Statistics: a tool for social research. Toronto (ONT): Nelson Education; 2014.

30.

Kreek

. Extreme marginalization: addiction and other mental health disorders, stigma, and imprisonment. Ann N Y Acad Sci. 2011;1231(1):65–72.

31.

Thornicroft

Rose

Kassam

. Discrimination in health care against people with mental illness. Int Rev Psychiatry. 2007;19(2):113–122.

32.

Sharac

Mccrone

Clement

. The economic impact of mental health stigma and discrimination: a systematic review. Epidemiol Psichiatr Soc. 2010;19(3):223–232.