Initial Validation of the Mental Health Provider Stigma Inventory

Abstract

Objective:

To conduct an initial validation of the mental health provider stigma inventory (MHPSI). The MHPSI assesses stigma within the service provider–client relationship on three domains—namely, attitudes, behaviors, and coworker influence.

Methods:

Initial validation of the MHPSI was conducted with a sample of 212 mental health employees and graduate student counseling interns. Scale conceptualization and development, content validation, and construct validation measures are discussed.

Results:

Preliminary psychometrics indicated good model fit for a three-factor multidimensional scale. Reliabilities were strong (coefficient αs ranged from .83 to .93) and the global stratified α was .95. Evidence of construct validity supported the hypotheses about the accuracy of underlying constructs. Conclusions: The MHPSI appears to be a reliable and valid measure of mental health provider stigma. As social workers comprise the front lines of mental health service provision, the MHPSI may be a valuable tool for guiding agency trainings and social work education.

Keywords

stigma mental health scale development psychometric study

Erving Goffman (1963, p. 3) defines stigma as “an attribute that is deeply discrediting” while acknowledging that “a language of relationships, not attributes, is really needed.” That is, what is stigmatized and by whom are social constructions—products of the relationship between some attribute and a related stereotype (Goffman, 1963). In this model, stigmas are grounded in physical characteristics (e.g., disability), perceived character deficits (e.g., mental disorder, addiction, and homosexuality), or the so-called tribal stigmas of race, nation, and religion (Goffman, 1963). Based on the social interaction theory, stigmas have been found to influence identity development and social relationships, resulting in social exclusion (Link & Phelan, 2006). Expanding on Goffman’s definition, Link and Phelan (2001) broaden the stigma concept to emphasize the importance of structural context and social power. They define stigma as the cooccurrence of its components—labeling, stereotyping, out-grouping, and discrimination—and further indicate that power must be exercised for stigmatization to occur. The renewed focus on structure and power suggests that stigmas may be an important determinant of imbalanced power relationships, especially those which occur between stigmatized individuals and the professionals who serve them in the mental health care system.

Although the majority of the literature focuses on primary stigma, or the direct experience of stigmatization by an affected person or group, a related stigma construct also deserves attention. Courtesy stigma, or stigma by association, is experienced by the “normal” associates of stigmatized individuals. This type of stigma affects the friends, family, and professional associates of the fully stigmatized (Goffman, 1963). The courtesy stigma construct has been explored among the friends and the family of persons with disabilities (Birenbaum, 1970; Green, 2003), autism (Gray, 2002), genetic disorders (Turner, Biesecker, Leib, Biesecker, & Peters, 2007), and mental disorders (Corrigan & Miller, 2004). However, only a small minority of studies investigate the impact of courtesy stigma on the service providers who interact with these heavily stigmatized groups (Phillips, Benoit, Hallgrimsdottir, & Vallance, 2012; Sadow, Ryder, & Webster, 2002; Snyder, Omoto, & Crain, 1999; Verhaeghe & Bracke, 2012). The literature suggests that providers are susceptible to social network censure (e.g., “How could you work with someone like that?”) and may manage this censure by creating emotional distance between themselves and clients (Snyder et al., 1999).

To illustrate, Phillips, Benoit, Hallgrimsdottir, and Vallance (2012) conducted a mixed methods study of 17 employees in a nonprofit organization providing health care to sex workers in Canada. The majority of employees in the study reported experiencing courtesy stigma from friends, family members, and allied service professionals in other agencies. This perception of stigma was amplified for those employees who were themselves former sex workers. Phillips and colleagues (2012) suggest that, in keeping with Goffman’s (1963) theory, the anxiety of courtesy stigma was managed through impression management tactics and professionalization narratives. That is, workers created emotional and logistic boundaries by not discussing work issues with friends and family and by accentuating the differences between themselves (as credentialed workers) and their clients. Thus, for service providers, the perception of courtesy stigma exaggerated the power differential inherent in the service relationship and led to the out-grouping of clients. In this study, the experience and management of courtesy stigma was associated with negative outcomes for both the worker and her client including poor worker satisfaction, high rates of employee turnover, and organizational conflict and strained relationships with other service providers in the community (Phillips et al., 2012).

A focus on courtesy stigma alone, however, may obscure the unique contributions of social and organizational factors to the service provider–client relationship in the field of interest. As noted earlier, courtesy stigma research suggests that human service providers are susceptible to external social network censure as their friends and family members struggle to understand how and why the provider would choose to work with such persons as sex workers or mental health clients (Phillips et al., 2012; Sadow et al., 2002; Snyder et al., 1999). As such, many human service professions can be broadly conceptualized under the heading of “dirty work” because of the worker’s voluntary affiliation to fully stigmatized groups. In general, dirty work occupations are those that are considered to have physical, social, or moral taint (Hughes, 1958). Dirty work deals with the more unsavory aspects of social living, from physically dirty tasks (e.g., emptying bedpans and preparing bodies for burial) to tasks assumed by some to be socially or morally dirty (e.g., working with mental health clients, welfare recipients, child abusers, prisoners, or persons living with HIV/AIDS). Hughes (1962) highlights the double bind inherent in dirty work, that is, society requires this work to be done, but the workers themselves are disowned and disavowed. Thus, dirty work occupations often have low social prestige, even though the work itself is generally not perceived to be unimportant or trivial, just distasteful or disgusting (Ashforth & Kreiner, 1999).

Organizational behavior research posits that individuals seek a positive sense of self and that occupational role is a pivotal factor in how self-identity is created and maintained (Ashforth & Kreiner, 1999; Ashforth, Kreiner, Clark, & Fugate, 2007). Therefore, initial hypotheses speculated that persons in dirty work occupations would have relatively low occupational self-esteem and weak work group cultures. However, qualitative research indicates quite the opposite (Ashforth & Kreiner, 1999; Ashforth et al., 2007; Bolton, 2005; Cahill, 1999; Tracy, 2004). Ashforth and Kreiner (1999) posit that strong occupational and work group cultures emerge in dirty work professions, which in turn foster strong group membership bonds. Thus, workers in stigmatized occupations are highly bonded to each other, and their bonds may influence their behavior within the workplace (Ashforth et al., 2007). In this way, group membership is thought to trump both institutional and individual factors in identity development and stigma management (Ashforth & Kreiner, 1999; Ashforth et al., 2007).

When taken together, human service providers in dirty work occupations may create a positive sense of both the self and the professional group by engaging in provider stigma—the labeling, stereotyping, out-grouping, and discrimination of clients (Baines, Evans, & Neysmith, 1992; Sadow et al., 2002). The tendency for workers to accentuate their professional status, create emotional boundaries between themselves and clients, and forge strong allegiances with coworkers may contribute to this stigmatization. Provider stigma, therefore, is hypothesized to be a product of primary stigma and the perception of courtesy stigma within the context of coworker influence.

Although primary stigma, courtesy stigma, and the issue of dirty work are well explored in the literature, the specific issues inherent to provider stigma are less clear. Qualitative and mixed methods work suggests that provider stigma is an issue among mental health providers (Cohen & Cohen, 1984; Covarrubias & Han, 2011; Delaney, 2012; Hanzawa et al., 2012; Horsfall, Cleary, & Hunt, 2010; Kassam, Papish, Modgill, & Patten, 2012), medical doctors treating mental health clients in general practice (Wallace, 2012), and HIV/AIDS service providers (Rutledge, Whyte, Abell, Brown, & Cesnales, 2011). Provider stigma may negatively influence client outcomes and service delivery, especially if providers base their behaviors on stereotyped information about other clients sharing similar characteristics (e.g., diagnostic label and chronicity of condition; Ahmedani, 2011; Hugo, 2001; Lyons, Hopley, & Horrocks, 2009; & Horrocks, 2009; McDaid, 2008; Nordt, Rossler & Lauber, 2006; & Lauber, 2006; Wahl, 1999).

A recently developed measure of mental health provider stigma, the Opening Minds Scale for Health Care Providers (OMS-HC; Kassam et al., 2012), measures the stigmatizing attitudes of professionals and student interns who provide services to mental health clients (e.g., nurses, psychologists, physicians, psychiatrists, occupational therapists, and social workers). Although the scale demonstrated strong initial psychometrics, it was only designed to assess provider attitudes toward mental health disorders and mental health clients. The scale does not measure stigmatizing behaviors toward clients or the influence of coworker relationships. The only identified provider stigma measurement tool that does measure both attitudes and behaviors is the HIV/AIDS Provider Stigma Inventory (HAPSI; Rutledge et al., 2011). However, the HAPSI was designed to measure stigma among service professionals working with people living with HIV/AIDS and is not readily generalizable to other human service providers. A preliminary psychometric evaluation of the HAPSI showed strong reliability coefficients. Initial validation results from the OMC-HC and the HAPSI highlight the utility of provider stigma scales in a continuing education or intervention context. Therefore, a measure of provider stigma capable of assessing attitudes, behaviors, and coworker influence among mental health providers was designed to fill the gap in the mental health literature.

Method

Scale Conceptualization and Development

The mental health provider stigma inventory (MHPSI) was designed to assess the presence and magnitude of stigma within the service provider–client relationship. Using Link and Phelan’s (2001) conceptual framework, provider stigma was defined as the labeling, stereotyping, out-grouping, and discrimination of clients by mental health service providers. The operational definitions of these four constructs were adapted from Rutledge, Whyte, Abell, Brown, and Cesnales (2011) psychometric validation study of the HAPSI. Labeling was defined as using pejorative names for clients when thinking or talking about them. Stereotyping was defined as judging clients based on the personal characteristics or qualities. Out-grouping was defined as the reinforcement of difference between providers and clients. And, discrimination was defined as rejecting clients based on their personal characteristics or qualities.

Initially, these four constructs were dimensionally represented within overarching concepts of felt, enacted, and courtesy stigma. These represented, respectively, providers’ stigmatizing thoughts or emotions toward clients that were not expressed overtly, those that were expressed overtly, and those that were experienced by the providers themselves due to their associations with mental health clients. The developers’ original intent was to include these components of provider stigma in a way that encompassed the broad conceptualization of the term while keeping the focus on felt, enacted, and courtesy domains.

A preliminary 120-item version of the MHPSI (40 items in each of the following three domains: felt, enacted, and courtesy stigma) was piloted among an expert panel of eight members. The ultimate goal was to create a measure with under 50 items to reduce the burden for service provider respondents. Preliminary items were generated by the first author and workshopped with the second and third authors to ensure coverage of the felt, enacted, and courtesy stigma domains. Although the 120 original items cannot be reproduced here, they do not substantively differ in tone or content from the 41 pilot-tested items. For example, the items “I use negative names to describe clients when describing them to coworkers,” and “I talk about clients to my coworkers in scornful terms” were both discarded, in favor of the retained item, “I talk about clients to my coworkers in dismissive terms.” Multiple items with varying wording were provided to panelists in an attempt to achieve true construct saturation and match the stigma constructs to the practice environment in a way that was both logical and emotionally resonant.

Content Validation

Members of the expert panel were asked to assess the level of fit between construct definitions and proposed items using a 5-point Likert-type scale. Additionally, panelists were asked to provide qualitative feedback about item content, overall conceptualization, the appropriateness of scale language, and the format. Four of the panelists were doctoral students in sociology or social work, all with a research interest in stigma. The fifth panelist was an assistant professor of social work, chosen for his methodological expertise, and the remaining three panelists were master’s level social workers providing mental health services in the field.

From a quantitative perspective, the 120-item, 3-domain MHPSI fared very well. Panelists endorsed the majority of items, that is, 79.17% (n = 95) had a mean above 4 and 14.17% (n = 17) had a mean between 3.5 and 3.99. Only 6.67% of the items (n = 8) had a mean score below 3.49. However, although panelists reported that items reflected the overarching constructs, qualitative feedback indicated systemic problems with the conceptualization and structure of the scale itself.

Qualitative analysis was conducted by the first two authors. Comments were read, cataloged, and grouped into patterns independently and then brought together for discussion and comparison. The revisions described below were unanimously endorsed. Overall, panelists questioned the distinction between felt stigma and enacted stigma and suggested that these two sections be reconceptualized as attitudes and behaviors. Although several panelists noted that the workplace environment was a key feature of stigma, the courtesy stigma domain was deemed unsuccessful at adequately measuring peer influence in the work setting. Therefore, items were pulled from the original felt and enacted stigma domains to comprise the attitude and behavior domains of the final scale, respectively. The courtesy stigma domain of the original measure underwent extensive revision to more adequately and accurately capture coworker influence. After reconceptualization, qualitative feedback was solicited from three of the original eight panelists. Feedback indicated improved clarity, construct congruence, and utility.

Conceptually, the revised MHPSI divided the provider stigma construct into the following three components: attitudes, behaviors, and coworker influence. Attitudes were defined as the thoughts, feelings, and beliefs respondents have about mental health clients. Behaviors were measured as both actions toward clients and actions with coworkers in relation to clients. Coworker influence was defined as influence exerted by the workplace peer group, encouraging providers to change their attitudes, values, or behaviors in order to conform to group norms.

Dimensional representation of labeling, stereotyping, out-grouping, and discrimination was retained within each of the three new overarching domains. It was decided that distributing this content was preferable to creating explicit domains based on the constructs of labeling, stereotyping, out-grouping, and discrimination. This strategy was deemed necessary to preserve the authors’ intention to create a scale that would allow for an assessment of workplace context and the influence of workplace relationships on the respondent’s behaviors.

Sampling

Next, the pilot validation was conducted with a purposive, nonprobability sample of employees and graduate student interns providing mental health counseling services in a variety of contexts. This study was reviewed by and received ethics clearance for human subjects from the Institutional Review Board of a major public university in the southeastern United States. Mental health service providers were recruited from two public psychiatric hospitals, one of which espouses a recovery orientation. Both hospitals serve a broad range of clients, offering emergency and chronic care, forensic services, detoxification and inpatient substance abuse treatment, and outpatient mental health and substance abuse counseling. Employees at a university counseling center were also targeted for participation. Additionally, students enrolled in six graduate programs in counseling fields (art therapy, clinical psychology, marriage and family therapy, mental health counseling, music therapy, and social work) at a large state university in the southeast were invited to participate. All of the named programs require graduate students to complete an internship, field placement, or to log practice hours in the community. Finally, a snowball sampling strategy was employed with community mental health providers personally known to the researchers. This strategy was deemed necessary because mental health services are often provided outside of a traditional institutional or agency context (e.g., hospitals, prisons, and schools) and a variety of persons provide mental health services (e.g., addiction professionals, nurses, psychologists, and social workers) both formally and informally. The foundation of this sampling strategy was adapted from Kassam, Papish, Modgill, and Patten’s (2012) initial validation study of the OMS-HC among a broad range of mental health providers.

Construct Validation Measures

Evidence of convergent construct validity was assessed for each MHPSI subscale, namely, Attitudes, Behaviors, and Coworker Influence. Given the rarity of studies on provider stigma, the therapeutic relationship was chosen for construct validation, as the quality of the therapeutic relationship should be indirectly related to stigmatizing attitudes. A negative relationship between the two was hypothesized—as stigmatizing attitudes increased, the quality of the therapeutic relationship was expected to decrease. To examine construct validity for the Attitudes subscale, the clinician version of the Scale to Assess Therapeutic Relationships in Community Mental Health Care (STAR-C; McGuire-Snieckus, McCabe, Catty, Hansson, & Priebe, 2007) was included with the data collection instrument. The STAR-C asks respondents to assess the quality of the provider–client relationship and measures that relationship as an amalgam of both attitudes and behaviors.

The STAR-C demonstrated strong initial psychometrics when tested with 266 community mental health patients (all with a diagnosed psychotic illness) and their 120 psychiatric clinicians in England and Sweden (McGuire-Snieckus et al., 2007). Although the STAR-C reflects an explicitly medical perspective (e.g., use of the term patient vs. client) and was designed to assess the relationship between a psychiatrist and one specific patient, it was hypothesized that the scale would retain utility as a construct validity tool. The majority of other identified therapeutic relationship scales measure the quality of the relationship solely from the perspective of the client (e.g., Helping Relationship Inventory; Poulin & Young, 1997).

To explore the construct validity of the Behaviors subscale, the Depersonalization subscale of the Maslach Burnout Inventory (MBI; Maslach & Jackson, 1981) was also included with the data collection instrument. Depersonalization refers to the tendency of stressful working conditions to result in service providers’ alienation from service recipients (Maslach & Jackson, 1981). It was hypothesized that there would be a positive relationship between the Behaviors subscale and depersonalization—as stigmatizing behaviors increased, we expected depersonalization to increase. The Depersonalization subscale is a small portion of the 28-item MBI. However, because each item in the MBI is rated twice (first for frequency and then for strength), including the entire MBI was felt to unnecessarily increase respondent burden. Additionally, in initial psychometric validation among 1,025 adult human service providers (e.g., counselors, nurses, social workers, and teachers), each subscale was found to have strong independent validity and reliability (Maslach & Jackson, 1981).

A single-item validity indicator was used to examine the construct validity of the Coworker Influence subscale. This item read, “When I’m at work, I feel pressure to change my attitudes, values, or behaviors in order to conform to group norms.” A validated, relevant scale assessing peer pressure, conformity, or groupthink in the workplace could not be identified. It was hypothesized that scores on the Coworker Influence subscale would be positively correlated with scores on the single-item validity indicator—that is, as respondents indicated that their coworker’s opinions about a client would influence their own attitudes or behaviors toward that client, we expected they would also report increased pressure to conform to group norms in the workplace.

Finally, evidence of discriminant construct validity was explored using the demographic variables of race and gender. It was hypothesized that race and gender would not be significantly correlated to any of the three subscales or the MHPSI instrument as a whole.

Data Collection Instrument

The complete data collection instrument was available to participants electronically hosted on a secure server by Qualtrics. The link did not collect Internet protocol addresses or any other identifying information. The online consent form comprised the first page of the data collection instrument, and participants had to provide informed consent (via selecting a box) before the survey was accessed. The data collection instrument consisted of, in this order, the 41-item MHPSI, a single-item validity indicator, the 12-item STAR-C, the 10-item Depersonalization scale of the MBI, and a 10-item demographics section, which included items regarding gender, age, and race as well as educational attainment, professional identity (e.g., art therapist, social worker, and student intern), workplace characteristics, and time served in the field. Participants were asked to respond to items on the MHPSI and to the single-item validity indicator using a 7-point semantic differential, where 1 = completely disagree and 7 = completely agree. Both the STAR-C and the MBI were rated and scored using published guidelines.

Data Management and Analytic Strategy

MHPSI survey data (n = 219) were imported from Qualtrics directly into SPSS, Version 20, for analysis. The items were distributed as follows: 14 items on the Attitudes subscale, 18 items on the Behaviors subscale, and 8 items on the Coworker Influence subscale (see Table 1). Seven cases were removed prior to the analysis because they did not meet inclusion criteria for the study (i.e., participant reported their number of years working in the mental health field as 0).

Table 1.

The 41-Item Piloted Version of the MHPSI.

The first domain of the MHPSI concerns service provider attitudes about mental health clients. Please read each item below, then choose the number, where 1 = completely disagree and 7 = completely agree
Attitudes	1. It’s okay to refer to clients by their diagnostic label(s). 2. My good decisions in life have protected me from having problems like my clients have. 3. Clients behave like other people with the same diagnostic label(s). 4. Clients with chronic diagnostic labels should not make their own decisions. 5. It’s useless to help people with certain diagnostic labels. 6. Clients will not follow through on recommendations or instructions. 7. Once a person becomes a mental health client, they will never get better. 8. It’s okay to remind clients that the staff is in charge. 9. Clients are incapable of change. 10. I could never have the same problems as clients. 11. Clients do not deserve the care/services I provide. 12. Clients are crazy. 13. Once a client has a chronic diagnosis, there’s no point in working hard to help them. 14. I am the expert in my relationships with clients.
The second domain of the MHPSI concerns service provider behaviors. Please read each item below, then circle the number, where 1 = completely disagree and 7 = completely agree
Behaviors	15. I tell clients that they cause their own problems. 16. I purposely don’t follow through on referrals for clients. 17. I tell clients that they don’t have the right to make decisions about their care. 18. I refer to clients by their diagnostic label(s). 19. I intentionally treat adult clients like children. 20. I validate clients’ stories, even when I suspect they are not accurate.^a 21. I tell clients that they’ve made a mess of their lives. 22. I make fun of clients for the things they believe. 23. I tell clients that I am the expert. 24. I talk about clients to my coworkers in dismissive terms. 25. I tell coworkers not to bother helping clients with certain diagnostic labels. 26. I tell coworkers I don’t approve of clients’ lifestyles. 27. I make fun of clients when talking about them to coworkers. 28. When a client and coworker are having a disagreement, I side with my coworker. 29. I tell coworkers that all clients with the same diagnosis are basically the same. 30. It bothers me when coworkers make fun of clients.^a 31. I tell coworkers that clients will never change. 32. I refer to clients by their diagnostic label(s) when discussing them with coworkers. 33. My coworkers talk about clients in disrespectful terms.
The third domain of the MHPSI addresses coworker influence in the workplace. Please read each item below, then choose the number, where 1 = completely disagree and 7 = completely agree
Coworker Influence	34. If my coworkers talked about a client in disrespectful terms, I would be more likely to use ugly or hurtful language when discussing them myself. 35. If my coworkers treated an adult client like a child, I would be more likely to restrict that client from making decisions about their care. 36. If my coworkers talked about how a client was incapable of change, I would be more likely to give up on that client. 37. If my coworkers talked about how they were smarter or more rational than clients, I would be more likely to think clients should do what I say because I’m “the expert”. 38. If my coworkers made fun of clients, I would be more likely to do so too. 39. If my coworkers talked about the bad decisions a client made, I would be more likely to lower my expectations for that client. 40. If my coworkers talked about how clients were incapable of change, I would be more likely to give up on trying to help them. 41. If my coworkers told me a client was a liar, I would be more likely to ignore that client if they alleged mistreatment from the staff.

Note. MHPSI = mental health provider stigma inventory. ^aReverse item.

Descriptive statistics of the 41-item MHPSI were run on the remaining sample of participants (n = 212) who met all eligibility criteria. Twelve kurtotic (kurtosis > 10.0) items were removed from the item pool, that is, 4 items from the Attitudes subscale (Items 5, 9, 11, and 13) and 8 items from the Behaviors subscale (Items 16, 17, 19, 21, 22, 25, 29, and 32). Remaining items were within acceptable ranges for skewness (<3), making further parametric statistical testing appropriate (Abell, Springer, & Kamata, 2009). Missing value analyses on the remaining 29 items showed that 19 items had missing values, with 11 items showing missingness at or below 2.4%. The remaining 8 items showed 5.7–6.1% missingness. Little’s missing completely at random (MCAR) test was not significant (χ² = 603.952, degrees of freedom [df] = 590, p = .336), indicating that missing data were MCAR. Therefore, missing values were replaced using the expectation maximization (EM) algorithm in order to maximize sample size (Dempster, Laird, & Rubin, 1977). Missing values were not replaced on the STAR-C, MBI, or the single-item validity indicator.

The three hypothesized subscales were then evaluated for reliability, factor structure, and construct validity. Correlations between items were investigated to identify items with duplicate content. Internal consistency was assessed using Cronbach’s coefficient α (Cronbach & Meehl, 1955), stratified α, and standard error of measurement (SEM). Factor structure was verified using a confirmatory factor analysis (CFA). Finally, construct validity was assessed using Pearson’s correlations.

Results

Demographics and Sample Characteristics

As detailed in Table 2, participants were 85.9% female and 76.2% Caucasian, which was expected based on the demographic characteristics of workers and students in the helping professions. Participants had a mean age of 36.1 (SD = 11.8) and ranged in age from 21 to 67. The majority of participants had completed a bachelor’s (n = 85; 44.0%) or a master’s degree (n = 80; 41.5%) and worked mostly in outpatient settings (n = 125; 65.4%). Specifically, the majority of participants worked in a community or social service agency (n = 73; 38.2%), psychiatric hospital (n = 33; 17.3%), or a counseling center (n = 23; 12.0%). Participants reported a mean of 6.7 years working in the mental health field (SD = 7.6) with a range of 0.2 to 40 years. Workers identified most often as social workers (n = 51; 26.7%), nurses (n = 17; 8.9%), and psychologists (n = 12; 6.3%). Student interns were most often enrolled in social work (n = 43; 58.9%) and clinical psychology (n = 11; 15.1%) graduate programs.

Table 2.

Demographics of the Sample.

	Frequency	%	Mean (SD)	Range
Gender (n = 192)
Female	165	85.9
Male	27	14.1
Race (n = 193)
Caucasian	147	76.2
African American	19	9.8
Hispanic	9	4.7
Asian	5	2.6
Multiracial/other	13	6.7
Age (n = 188)			36.1 (11.8)	21–67
Education (n = 193)
Associates	6	3.1
Bachelors	85	44.0
Masters	80	41.5
PhD	19	9.0
Specialist/other	3	1.6
Workplace (n = 191)
Community agency	73	38.2
Psychiatric Hospital	33	17.3
Counseling center	23	12.0
Hospital	14	7.3
Private practice	14	7.3
Substance abuse facility	11	5.8
School	8	4.2
Nursing home	5	2.6
Other	10	5.3
Years in mental health (n = 187)			6.7 (7.6)	0.2–40
Job title (n = 191)
Student intern	73	38.2
Social worker	51	26.7
Nurse	17	8.9
Psychologist	12	6.3
Mental health counselor	9	4.7
Marriage and family therapist	6	3.1
Art therapist	4	2.1
Music therapist	4	2.1
Other	15	7.8

Initial Reliability and Item Redundancy

To assess initial reliability of the three subscales and the instrument as a whole, correlation matrices, Cronbach’s α coefficients, and α-if-item-deleted statistics were examined. First, a bivariate correlation matrix of the 29-item MHPSI (after 12 items were removed due to extreme kurtosis as detailed earlier) revealed that two pairs of items were highly correlated (r > .8). One item each from the Attitudes subscale (Item 1) and the Coworker Influence subscale (Item 40) was removed after item content was deemed redundant. Likewise, Item 20 was removed from the Behaviors subscale as it correlated negatively with more than 90% of the remaining items.

Then, internal scale consistency on the remaining 26 items was evaluated for each of the three hypothesized subscales and for the global scale using Cronbach’s α coefficients. Reliability coefficients are provided for both the initial and the final models in Table 3. Initial model subscale reliabilities were strong, that is, 9-item Attitudes (not including Item 28, added later as detailed subsequently) α = .842, 10-item Behaviors α = .843, and 7-item Coworker Influence α = .936. The global Cronbach’s α was .938. Initial model α-if-item-deleted indices were then reviewed by subscale for the remaining 26 items to gauge whether removing additional items would improve internal consistency. Item 33 was removed from the Behaviors subscale based on these analyses. After removing this item, factor analyses were conducted with the remaining 25 MHPSI items. The relocation of Item 28 from behaviors to attitudes depicted in final model α-if-item-deleted columns of Table 3 was based on those analyses, as described subsequently. Values are not provided for items that did not move forward into the final model.

Table 3.

Initial and Final Model Reliability: Internal Consistency by Subscale.

Attitudes			Behaviors			Coworker Influence
Original Item #	Initial Model α-if-Item Deleted	Final Model α-if-Item Deleted	Original Item #	Initial Model α-if-Item Deleted	Final model α-if-Item Deleted	Original Item #	Initial Model α-if-Item Deleted	Final Model α-if-Item Deleted
2	.824	.816	15	.833	.847	34	.926	.926
3	.833	.823	18	.817	.824	35	.930	.930
4	.824	.815	23	.828	.844	36	.925	.925
6	.819	.807	24	.813	.825	37	.926	.926
7	.825	.813	26	.819	.832	38	.930	.930
8	.827	.824	27	.813	.820	39	.924	.924
10	.822	.810	28	.832		41	.924	.924
12	.831	.819	30	.847	.864^a
14	.824		32	.822	.832
28		.824	33	.858^a
α	.842	.834	α	.843	.854	α	.936	.936
SEM		.346	SEM		.365	SEM		.291
Global stratified α = .95
Global SEM = .197

Note. SEM = standard error of measurement. ^aRemoving this item would improve subscale reliability.

Factorial Validation: Structural Equation Modeling

A CFA of the predicted model was conducted using LISREL, Version 9.1 (see Table 4). Twenty-five observed and three latent variables were entered into the path analysis. The initial model returned a significant χ² value of 1,030.057 (p < .01), which supported rejecting the hypothesis that the model was correct. However, the χ² statistic is directly affected by the sample size, with larger samples returning larger (and therefore statistically significant) values (Abell et al., 2009). Therefore, the χ²/df ratio is suggested as a better gauge of fit, as this statistic accounts for sample size. Unfortunately, the χ²/df ratio for the initial model was 3.77, above recommendations that better model fit is demonstrated by ratio of less than 3, and ideal model fit characterized by a ratio of less than 2 (Abell et al., 2009). Other fit indices were examined, including the comparative fit index (CFI), the Tucker–Lewis index (TLI), the root mean square of approximation (RMSEA), and the standardized root mean square residual (SRMR). Good model fit on these additional indices is suggested by established critical values. The critical value for the CFI and TLI is .90, with values above .95 representing excellent fit; the RMSEA must be at or below .08 to demonstrate acceptable fit, with values below .05 preferred; and for the SRMR, a value of less than .10 is acceptable (Hu & Bentler, 1999; Kline, 2005).

Table 4.

Confirmatory Factor Analyses: Initial and Respecified Model.

Scale Model	χ²/df	CFI	TLI	RMSEA	SRMR
Initial	3.79	0.93	0.93	0.115	0.077
Respecified^a	2.81	0.96	0.95	0.092	0.067

Note. CFI = comparative fit index; TLI = Tucker–Lewis index; RMSEA = root mean square error of approximation; SRMR = standardized root mean square residual. ^aPermitting 14 error covariances, removing 1 item from the Attitudes subscale, and changing the path of 1 item from the Behaviors subscale to the Attitudes subscale.

The initial model solution returned promising values on the CFI, TLI, and SRMR and an unacceptable value on the RMSEA. Additionally, LISREL generated a warning that the model was “not positive definite” due to evidence of multicollinearity between variables. To address multicollinearity, Item 14 from the Attitudes scale was removed because it returned high error covariances (>25), with multiple items on the Behaviors and Coworker Influences subscales. Additionally, modification indices suggested changing the path of Item 28 from the Behaviors subscale to the Attitudes subscale. After consideration of item content and the operational definitions of the subscales, this change was deemed reasonable and Item 28 was moved to the Attitudes subscale. Thus, the respecified model included 24 observed variables and 3 latent variables distributed as follows: 9 items on the Attitudes subscale, 8 items on the Behaviors subscale, and 7 items on the Coworker Influence subscale. Additionally, the respecified model allowed errors to correlate between items on the same subscale when the output suggested they would reduce the model χ² value by greater than or equal to 10. The respecified model converged with no further errors.

The 24-item respecified model returned a χ²/df ratio of 2.81 and improved fit indices overall, while still returning a value on the RMSEA higher than the critical value of .08. Specifically, the respecified values on the CFI, TLI, and SRMR indices suggested excellent model fit. Although mixed findings may indicate that a three-factor model is not appropriate for the MHPSI, interfactor correlation and multicollinearity were anticipated in the scale development phase. The MHPSI was designed to provide information, which contextualizes the attitudes and behaviors of mental health provider respondents within the framework of organizational behavior and coworker relationships. Therefore, we decided to move forward with the respecified 24-item model and examine internal consistency, measurement error, and correlations between respecified subscales.

First, α-if-item-deleted indices were reevaluated on the final model 24-item model solution (see Table 3). Final αs for each of the proposed subscales exceeded .80: Attitudes (α = .83), Behaviors (α = .85), and Coworker Influence (α = .94). These analyses suggested removing Item 30 from the Behaviors subscale to improve subscale reliability. However, when the CFA was rerun without Item 30, psychometric indices indicated substantially poorer model fit overall. Thus, this item was retained in the final version of the MHPSI. Cronbach’s α for the MHPSI as a whole exceeded .90 (α = .94). As Cronbach’s α may underestimate the reliability of a multidimensional scale (Abell et al., 2009), stratified α was calculated to compensate for this bias. As expected, the global stratified α was slightly higher (α = .95).

Then, SEM statistics were calculated for each subscale in the final model, that is, Attitudes = .35, Behaviors = .37, and Coworker Influence = .29. The SEM for the global scale (using the stratified α reported earlier) was .20. These values were slightly above Springer, Abell, and Hudson’s (2002) recommended cut point of .30 for the Attitudes and Behaviors subscales (5% of the possible range of scores on a 7-point scale), suggesting some deviation between the observed score and the true score. However, SEM may be affected by the homogeneity of responses to a specific grouping of items and are not necessarily indicative of measurement deficiencies (Pike & Hudson, 1998). That is, the slightly elevated SEM values reported here may be attributed to the nonprobability sampling strategy and relatively modest sample size of the pilot study.

Finally, correlation matrices between subscales were examined (see Table 5). Significant correlations suggest that scores on each item as well as scores on each of the subscales were interdependent. This was not unexpected as it was hypothesized that the MHPSI would yield a meaningful global score, with higher scores suggesting higher levels of stigmatizing attitudes and behaviors within the workplace context. Means and standard deviations for all items included on the final version of the scale are presented in Table 6.

Table 5.

Correlation Matrix by Subscale.

	Behaviors	Coworker Influence
Attitudes	.698**	.643**
Behaviors		.693**

**p < .01.

Table 6.

Item and Subscale Mean Scores and Standard Deviations.

Attitudes			Behaviors			Coworker Influence
Original Item #	M	SD	Original Item #	M	SD	Original Item #	M	SD
2	2.70	1.490	15	1.71	1.071	34	1.94	1.378
3	2.84	1.509	18	1.62	1.212	35	1.56	1.091
4	1.84	1.259	23	1.64	1.293	36	1.74	1.231
6	2.12	1.277	24	1.83	1.258	37	1.50	1.122
7	1.40	0.905	26	1.76	1.269	38	1.96	1.528
8	2.60	1.726	27	1.73	1.334	39	2.19	1.547
10	1.45	0.965	30	2.34	1.825	41	2.26	1.499
12	1.36	0.862	32	1.94	1.469
28	2.51	1.400
Subscale	2.09	0.850		1.82	0.955		1.88	1.152

Construct Validation

Convergent construct validation was evaluated by testing the MHPSI against theoretically related concepts from the literature. It was hypothesized that each of the MHPSI subscales would individually correlate with either a standardized measure or a single-item validity indicator, when no relevant standardized scale could be identified (see Table 7). Evidence of construct validity was established as all of the relationships tested were statistically significant and conformed to directional hypotheses.

Table 7.

Convergent Construct Validation.

Scale	Indicator	N	r	r ²
Attitudes	STAR-C	195	−.424**	.180
Behaviors	MBI-Depersonalization subscale, frequency	195	.535**	.286
	MBI-Depersonalization subscale, strength	191	.464**	.215
Coworker Influence	SII: When I’m at work, I feel pressure to change my attitudes, values, or behaviors in order to conform to group norms	200	.417**	.174
		Mean		.214

Note. STAR-C = Scale to Assess Therapeutic Relationships in Community Mental Health Care; MBI = Maslach Burnout Inventory; SII = single-item validity indicator.

**p < .01.

The STAR-C performed well in this sample (n = 195; α = .82) and, as hypothesized, had a negative relationship with the Attitudes subscale (r = −.42). Variance in responses on the STAR-C explained 18% of the variance on the Attitudes subscale. The frequency portion of the MBI Depersonalization subscale performed acceptably (n = 195; α = .77) as did the strength portion (n = 191; α =.79). Also as hypothesized, there was a positive correlation between the Behaviors subscale of the MHPSI and both the frequency of depersonalization (r = .53) and the strength of depersonalization (r = .46) toward clients within the workplace context. Variance in responses on the frequency and strength portions of the Depersonalization subscale of the MBI explained 29% and 22%, respectively, of the variance on the Behaviors subscale of the MHPSI. Finally, a significant positive correlation was identified between the Coworker Influence subscale and a single-item validity indicator (n = 200; r = .42). Variance in this item explained 17% of the variance in the Coworker Influence subscale. Although effect sizes were relatively modest (M = .21, with a range of .17–.29), all hypothesized convergent relationships were tested and found to be statistically significant using Pearson’s correlation coefficients.

Discriminant construct validity was tested by evaluating all three subscales and the global MHPSI with respondent race and gender (see Table 8). As hypothesized, race and gender were not significantly correlated with the Attitudes, Behaviors, or Coworker Influence subscales. Likewise, race and gender were not significantly correlated with the MHPSI as a whole.

Table 8.

Discriminant Construct Validation.

Scale	Indicator	r	r ²	p
Attitudes	Gender	.004	.000	.961
	Race	−.083	.007	.252
Behaviors	Gender	−.110	.012	.129
	Race	−.055	.003	.446
Coworker Influence	Gender	−.050	.003	.494
	Race	−.027	.001	.705
Global	Gender	−.059	.003	.420
	Race	−.062	.004	.393
	Mean		.004

Discussion and Implications for Social Work Practice

Stigmatization of mental health clients is an important social problem. Although primary and courtesy stigmas are well explored in the literature, the incidence, prevalence, and magnitude of mental health provider stigma is less clear. The MHPSI was designed to address this gap in the literature. Overall, the MHPSI achieved strong psychometrics in this initial validation study. Content validation allowed the original model to be crafted into a briefer measure that resonated with expert panelists working in the field. The Attitudes, Behaviors, and Coworker Influence subscales demonstrated strong psychometric coefficients and a CFA provided confirmation of the hypothesized factor structure. Likewise, the evidence for construct validity suggests that each subscale successfully measured the constructs they were intended to measure.

The final 24-item MHPSI is anticipated to take 10 minutes for respondents to complete and is relatively easy to score. Only one reverse item (initial Item 30) was retained in the final version of the MHPSI. After reverse coding this item, scoring requires generating the mean of responses on each subscale. Higher means on the Attitudes and Behaviors subscales are indicative of higher rates of providers’ stigmatizing attitudes and behaviors, respectively. Higher means on the Coworker Influence subscale suggest that the providers’ behaviors toward clients are more strongly influenced by the stigmatizing behaviors of coworkers. A global score is obtained by calculating the mean of all three subscale scores. Higher global scores on the MHPSI reflect a greater tendency to stigmatize mental health clients. It is suggested that all four scores be interpreted together to provide the most salient information about mental health provider stigma in the full organizational context. The final version of the MHPSI is available in Table 9, with the reverse item noted.

Table 9.

The Final Version of the MHPSI With Respondent Instructions.

The first domain of the MHPSI concerns service provider attitudes about mental health clients. Please read each item below, then choose the number, where 1 = completely disagree and 7 = completely agree
Attitudes	1. My good decisions in life have protected me from having problems like my clients have.	1 2 3 4 5 6 7
	2. Clients behave like other people with the same diagnostic label(s).	1 2 3 4 5 6 7
	3. Clients with chronic diagnostic labels should not make their own decisions.	1 2 3 4 5 6 7
	4. Clients will not follow through on recommendations or instructions.	1 2 3 4 5 6 7
	5. Once a person becomes a mental health client, they will never get better.	1 2 3 4 5 6 7
	6. It’s okay to remind clients that the staff is in charge.	1 2 3 4 5 6 7
	7. I could never have the same problems as clients.	1 2 3 4 5 6 7
	8. Clients are crazy.	1 2 3 4 5 6 7
	9. When a client and coworker are having a disagreement, I side with my coworker.	1 2 3 4 5 6 7
The second domain of the MHPSI concerns service provider behaviors. Please read each item below, then choose the number, where 1 = completely disagree and 7 = completely agree
Behaviors	10. I tell clients that they cause their own problems.	1 2 3 4 5 6 7
	11. I refer to clients by their diagnostic label(s).	1 2 3 4 5 6 7
	12. I tell clients that I am the expert.	1 2 3 4 5 6 7
	13. I talk about clients to my coworkers in dismissive terms.	1 2 3 4 5 6 7
	14. I tell coworkers I don’t approve of clients’ lifestyles.	1 2 3 4 5 6 7
	15. I make fun of clients when talking about them to coworkers.	1 2 3 4 5 6 7
	16. It bothers me when coworkers make fun of clients.^a	1 2 3 4 5 6 7
	17. I refer to clients by their diagnostic label(s) when discussing them with coworkers.	1 2 3 4 5 6 7
The third domain of the MHPSI addresses coworker influence in the workplace. Please read each item below, then choose the number, where 1 = completely disagree and 7 = completely agree
Coworker Influence	18. If my coworkers talked about a client in disrespectful terms, I would be more likely to use ugly or hurtful language when discussing them myself.	1 2 3 4 5 6 7
	19. If my coworkers treated an adult client like a child, I would be more likely to restrict that client from making decisions about their care.	1 2 3 4 5 6 7
	20. If my coworkers talked about how a client was incapable of change, I would be more likely to give up on that client.	1 2 3 4 5 6 7
	21. If my coworkers talked about how they were smarter or more rational than clients, I would be more likely to think clients should do what I say because I’m “the expert.”	1 2 3 4 5 6 7
	22. If my coworkers made fun of clients, I would be more likely to do so too.	1 2 3 4 5 6 7
	23. If my coworkers talked about the bad decisions a client made, I would be more likely to lower my expectations for that client.	1 2 3 4 5 6 7
	24. If my coworkers told me a client was a liar, I would be more likely to ignore that client if they alleged mistreatment from the staff.	1 2 3 4 5 6 7

Note. MHPSI = mental health provider stigma inventory. ^aReverse item.

Measuring the presence of mental health provider stigma is important to the field of social work, as 60–70% of mental health services are provided by social workers nationally (Proctor, 2004). Likewise, social workers often comprise the front lines of acute psychiatric care, providing initial assessment and triage services in a variety of emergency settings (Hall, Amodeo, Shaffer, & Vander Bilt, 2000). In these settings, clients are often seen with a family member or loved one and the psychosocial assessment provides environmental context for the presenting mental health symptoms. Therefore, this first contact has implications for both micro-level service provision and mezzo-level psychoeducation (Ahmedani, 2011). Social workers have an opportunity to combat stigma in direct practice while also educating family members about the powerful negative influence of mental health stigma. The National Association of Social Workers (1999) charges the profession to uphold the core values of client rights and self-determination, empowerment, self-realization, and the dignity and worth of the person all within a broader social justice framework. To this end, social workers are well positioned to uphold the rights of vulnerable persons in the community while educating others about inclusive and empowering care.

Ahmedani (2011) suggests that mental health provider stigma, especially among social workers, may be a facet of burnout or complicated by countertransference issues. Only through a process of honest self-reflection can workers acknowledge the subtle biases and internalized stigmas that may be unconsciously affecting their practice. Therefore, the MHPSI is designed to serve as a catalyst to this process, shedding light on uncomfortable attitudes and behaviors which may exist despite professional training. Mindful and reflective practice is hypothesized as the key to reducing stigma within the service provider–client relationship (Ahmedani, 2011).

Like other provider stigma measures (Kassam et al., 2012; Rutledge et al., 2011), the MHPSI may be a valuable tool for professional training, continuing education, and workplace assessment. Results on the scale may help researchers or program administrators develop targeted workshops to address quality improvement or quality assurance in specific workplace or agency. Additionally, the MHPSI may also serve as a useful outcome measure for anti-stigma interventions or trainings conducted with professionals in the field. The final version of the MHPSI has a Flesch reading ease score of 73.5 and a Flesch–Kincaid grade level of 6.5, making it an accessible tool for the diverse range of persons who provide mental health services both formally and informally in a variety of contexts.

Finally, the authors would like to emphasize that the MHPSI is not intended for use as a punitive tool—to distinguish “good” mental health providers from “bad” ones. Rather, we feel that the real utility of the MHPSI is that it can potentially provide an understanding of the influence of organizational environment factors on providers’ stigmatizing attitudes and behaviors. Creating awareness of these nuanced interactive patterns and interrupting potentially harmful interactions can be the focus of workplace trainings or interventions.

Additionally, the authors assert that the MHPSI is not meant to place a provider stigma label on members of strong work group cultures. Provider stigma is not inherent in professionalization narratives or strong occupational subcultures. In fact, the emotional benefits of forming boundaries between workers and clients are an uncontroversial feature of the basic curricula in many human service professions (e.g., Zastrow, 2010). Professional boundaries, in this context, are associated with self-care and managing burnout in the emotionally draining work of providing care and services to challenging populations (Zastrow, 2010). Working with mental health clients often entails listening to stories filled with horror, abuse, trauma, and despair. When these facts are added to issues of confidentiality, it makes sense that workers would turn to each other, rather than to friends and family, for emotional support. The MHPSI was not designed to criticize the tendency for workers to come together in solidarity. Instead, we posit that provider stigma occurs when primary stigma, courtesy stigma, and organizational behavior negatively influence the care and services that are provided to the clients.

Limitations and Areas for Future Research

Generalizations beyond this initial validation study should be approached with caution. This study is limited by the use of a relative modest purposive sample of both working professionals and graduate student interns. Unfortunately, the necessity of using a snowball sample to expand reach means that there is no way for us to evaluate what percentage of potential participants joined the study in response to a suggestion by others. It may be that individuals who chose not to complete the MHPSI differ from study participants in unknown ways. The justification for the utility of our obtained sample, however, is supported in part by participant demographics which place the mean participant age at 36.1 (SD = 11.8) and the average number of years working in the field at 6.7 (SD = 7.6). These descriptive statistics suggest that the validation study reached a broad range of mental health service providers at varying stages in their careers. Additionally, the inclusion of participants who provide mental health counseling services both formally (e.g., social workers and psychologists) and informally (e.g., nurses) allows for analysis of the scale’s utility beyond one specific discipline (Kassam et al., 2012).

The issue of social desirability also limits interpretations of the current findings. Although the scale was conducted online in an anonymous and secure format, social desirability factors may have influenced participant responses in unforeseen ways. Social desirability may also have impacted attrition from the study, as participants were more likely to drop out of the study as they clicked through the four screens that comprised the entirety of the MHPSI scale. Although missingness on any item did not exceed 6.1%, there was a higher percentage of missing values on the final 9 items (the fourth screen) of the MHPSI online data collection instrument. Ideally, making the final version of the MHPSI shorter (reducing from 41 to 24 total items) will alleviate respondent burden. However, we acknowledge that responses may be skewed if participants dropped out of the study because they acknowledged the presence of stigmatizing attitudes or behaviors within their own practice.

This concern is underlined by the low overall rates of stigma reported by the sample. Descriptively, mean scores on all items were below expected rates. Item mean scores ranged from a low of 1.36 (SD = .862) to a high of 2.85 (SD = 1.509). Subscale mean scores ranged from a low of 1.82 (SD = .955) on the Behaviors subscale to a high of 2.09 (SD = .850) on the Attitudes subscale. It is possible that the low levels of overall reported stigma are responsible for the extreme kurtosis identified in a subset of items during preliminary analyses. Thus, initial item reduction may have been influenced by sample characteristics more than item utility.

Additional validation of the MHPSI is needed to establish both its psychometric properties and its utility as an assessment tool for mental health professionals in the field. Strong initial psychometrics, however, suggest that replication among a more seasoned and diverse sample of workers is warranted. Replications comparing electronic versus paper-and-pencil responses may also be useful to see if attrition was due to the electronic format of the MHPSI data collection instrument or is attributable to other factors.

Footnotes

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

References

Abell

Springer

D. W.

Kamata

(2009). Developing and validating rapid assessment instruments. Oxford, England: Oxford University Press.

Ahmedani

B. K.

(2011). Mental health stigma: Society, individuals, and the profession. Journal of Social Work Values and Ethics, 8. Retrieved from http://www.jswvearchives.com/fall11/fall114.pdf

Ashforth

B. E.

Kreiner

G. E.

(1999). “How can you do it?”: Dirty work and the challenge of constructing a positive identity. Academy of Management Review, 24, 413–434.

Ashforth

B. E.

Kreiner

G. E.

Clark

M. A.

Fugate

(2007). Normalizing dirty work: Managerial tactics for countering occupational taint. Academy of Management Journal, 50, 149–174. doi:10.5465/AMJ.2007.24162092

Baines

Evans

Neysmith

(1992). Confronting women’s caring: Challenges for practice and policy. Affilia, 7, 21–44.

Birenbaum

(1970). On managing a courtesy stigma. Journal of Health and Social Behavior, 11, 196–206. doi:10.2307/2948301

Bolton

S. C.

(2005). Women’s work, dirty work: The gynaecology nurse as “other.” Gender , Work and Organization, 12, 169–186.

Cahill

S. E.

(1999). The boundaries of professionalization: The case of North American funeral direction. Symbolic Interaction, 22, 105–119.

Cohen

(1984). The clinician’s illusion. Archives of General Psychiatry, 41, 1178–1182.

10.

Corrigan

Miller

F. E.

(2004). Shame, blame and contamination: A review of the impact of mental illness stigma on family members. Journal of Mental Health, 13, 537–548.

11.

Covarrubias

Han

(2011). Mental health stigma about serious mental illness among MSW students: Social contact and attitude. Social Work, 56, 317–325.

12.

Cronbach

L. J.

Meehl

P. E.

(1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.

13.

Delaney

K. R.

(2012). Psychiatric mental health nurses: Stigma issues we fail to see. Archives of Psychiatric Nursing, 26, 333–335.

14.

Dempster

A. P.

Laird

N. M.

Rubin

D. B.

(1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistics Society, 39, 1–38.

15.

Goffman

(1963). Stigma: Notes on the management of spoiled identity. Englewood Cliffs, NJ: Prentice Hall.

16.

Gray

D. E.

(2002) ‘Everybody just freezes. Everybody is just embarrassed’: Felt and enacted stigma among parents of children with high functioning autism. Sociology of Health & Illness, 24, 734–739.

17.

Green

S. E.

(2003) ‘What do you mean “what’s wrong with her”?’: Stigma and the lives of families of children with disabilities. Social Science & Medicine, 57, 1361–1374.

18.

Hall

M. N.

Amodeo

Shaffer

H. J.

Vander Bilt

(2000). Social workers employed in substance abuse treatment agencies: A training needs assessment. Social Work, 45, 141–155.

19.

Hanzawa

Nosaki

Yatabe

Nagai

Tanaka

Nakane

(2012). Study of understanding the internalized stigma of schizophrenia in psychiatric nurses in Japan. Psychiatry and Clinical Neurosciences, 66, 113–120. doi:10.1111/j.1440-1819.2011.02307.x

20.

Horsfall

Cleary

Hunt

G. E.

(2010). Stigma in mental health: Clients and professionals. Issues in Mental Health Nursing, 31, 450–455.

21.

L.-T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.

22.

Hughes

E. C.

(1958). Men and their work. Glencoe, IL: Free Press.

23.

Hughes

E. C.

(1962). Good people and dirty work. Social Problems, 10, 3–10.

24.

Hugo

(2001). Mental health professionals’ attitudes towards people who have experienced a mental health disorder. Journal of Psychiatric & Mental Health Nursing, 8, 419–425.

25.

Kassam

Papish

Modgill

Patten

(2012). The development and psychometric properties of a new scale to measure mental illness related stigma by health care providers: The opening minds scale for health care providers (OMS-HC). BMC Psychiatry, 12, 62. doi:10.1186/1471-244X-12-62

26.

Kline

R. B.

(2005). Principles and practice of structural equation modeling (2nd ed.). New York, NY: Guilford Press.

27.

Link

B. G.

Phelan

J. C.

(2001). Conceptualizing stigma. Annual Review of Sociology, 27, 363–385.

28.

Link

B. G.

Phelan

J. C.

(2006). Stigma and its public health implication. Lancet, 367, 528–529.

29.

Lyons

Hopley

Horrocks

(2009). A decade of stigma and discrimination in mental health: Plus ca change, plus c’est la meme chose (the more things change, the more they stay the same). Journal of Psychiatric & Mental Health Nursing, 16, 501–507.

30.

Maslach

Jackson

S. T.

(1981). The measurement and experience of burnout. Journal of Occupational Behaviour, 2, 99–113.

31.

McDaid

(2008). Countering the stigmatisation and discrimination of people with mental health problems in Europe. Luxembourg: European Commission.

32.

McGuire-Snieckus

McCabe

Catty

Hansson

Priebe

(2007). A new scale to assess the therapeutic relationship in community mental health care: STAR. Psychological Medicine, 37, 85–95. doi:10.1017/S0033291706009299

33.

National Association of Social Workers. (1999). Code of ethics of the National Association of Social Workers. Washington, DC: NASW Press.

34.

Nordt

Rossler

Lauber

(2006). Attitudes of mental health professionals toward people with schizophrenia and major depression. Schizophrenia Bulletin, 32, 709–714.

35.

Phillips

Benoit

Hallgrimsdottir

Vallance

(2012). Courtesy stigma: A hidden health concern among front-line service providers to sex workers. Sociology of Health & Illness, 34, 681–696. doi:10.1111/j.1467-9566.2011.01410.x

36.

Pike

C. K.

Hudson

W. W.

(1998). Reliability and measurement error in the presence of homogeneity. Journal of Social Service Research, 24, 149–163.

37.

Poulin

Young

(1997). Development of a helping relationship inventory for social work practice. Research on Social Work Practice, 7, 463–489. doi:10.1177/104973159700700403

38.

Proctor

E. K.

(2004). Research to inform mental health practice: Social work’s contribution. Social Work Research, 28, 195–197.

39.

Rutledge

S. E.

Whyte

Abell

Brown

K. M.

Cesnales

N. I.

(2011). Measuring stigma among health care and social service providers: The HIV/AIDS provider stigma inventory. AIDS Patient Care and STDs, 25, 673–682.

40.

Sadow

Ryder

Webster

(2002). Is education of health professionals encouraging stigma towards the mentally ill? Journal of Mental Health, 11, 651–665.

41.

Snyder

Omoto

A. M.

Crain

A. L.

(1999). Punished for their good deeds: Stigmatization of AIDS volunteers. American Behavioral Scientist, 42, 1193–211.

42.

Springer

D. W.

Abell

Hudson

W. W.

(2002). Creating and validating rapid assessment instruments for practice and research: Part 1. Research on Social Work Practice, 12, 408–439. doi:10.1177/1049731502012003005

43.

Tracy

S. J.

(2004). The construction of correctional officers: Layers of emotionality behind bars. Qualitative Inquiry, 10, 509–533.

44.

Turner

Biesecker

Leib

Biesecker

Peters

K. F.

(2007). Parenting children with proteus syndrome: Experiences with, and adaptation to, courtesy stigma. American Journal of Medical Genetics, 143A, 2089–2097.

45.

Verhaeghe

Bracke

(2012). Associative stigma among mental health professionals: Implications for professional and service user well-being. Journal of Health and Social Behavior, 53, 17–32. doi:10.1177/0022146512439453

46.

Wahl

O. F.

(1999). Mental health consumers’ experience of stigma. Schizophrenia Bulletin, 25, 467–478.

47.

Wallace

J. E.

(2012). Mental health and stigma in the medical profession. Health, 16, 3–18. doi:10.1177/1363459310371080

48.

Zastrow

(2010). Introduction to social work and social welfare: Empowering people (10th ed.). Belmont, CA: Brooks/Cole.