Do Specialty Programs for Justice-Involved People With Mental Illness Exacerbate Stigma?

Abstract

Specialty mental health probation caseloads have shown promise in reducing recidivism for justice-involved people with mental illness. However, assignment to these caseloads may be stigmatizing due to labeling effects. We examined (1) whether assignment to specialty probation versus traditional probation is associated with greater internalized stigma among clients and (2) whether probation officers are the source of some of this stigmatization. As part of a multisite longitudinal study, 138 specialty probation clients and 148 similar clients from traditional probation rated their internalized stigma of mental illness, and officers rated their attitudes toward each of their supervisees. Specialty probation clients experienced more internalized stigma (d = .61) than traditional clients. Although both specialty and traditional officers held stigmatizing attitudes toward clients, only traditional officers’ attitudes were associated with clients’ internalized stigma. Probation officers from both types of agency may benefit from antistigma interventions to effectively work with clients with mental illness.

Keywords

offenders with mental illness probation officers stigma

The past two decades have witnessed a proliferation in policy recommendations aimed at addressing the disproportionate number of people with serious mental illness in the criminal justice system and the poor outcomes typically experienced by this group. Among the 6.6 million people supervised by the U.S. criminal justice system as of 2016 (Kaeble & Cowhig, 2018), approximately 31% of women and 15% of men had a disorder such as major depression, bipolar disorder, or schizophrenia, rates that are 2 to 3 times higher than the general population for women and men, respectively (Fazel & Danesh, 2002; Steadman et al., 2009). Justice-involved people with mental illness face barriers to rehabilitation and are more likely to return to the criminal justice system compared to their nondisordered counterparts (Cloyes et al., 2009; Messina et al., 2004; Skeem et al., 2014). Given the broad scope and serious consequences of this problem, policymakers have recommended the development of specialty programs at all stages of criminal justice processing, including forensic community treatment, police crisis intervention teams, behavioral health courts, and specialty mental health probation caseloads (Council of State Governments, 2002; Epperson et al., 2014; Skeem et al., 2011).

Specialty mental health probation caseloads are a relatively well-studied example of such programs. More than 100 probation agencies across the United States have developed specialty mental health programs (Lurigio et al., 2012; Skeem et al., 2006), which typically utilize small, homogeneous caseloads (composed only of clients with mental illness) supervised by officers with training in mental health who integrate criminal justice and social service resources and use problem-solving strategies, rather than sanction threats, in response to noncompliance (Skeem et al., 2006; see also Eno Louden et al., 2008). Two rigorous quasi-experimental evaluations of specialty probation have yielded promising results. For example, one study found that clients with mental illness placed on specialty caseloads were about half as likely to be arrested during a 2-year follow-up period (Skeem et al., 2017; see also Wolff et al., 2014). Along with mental health courts, specialty probation caseloads offer the most promising evidence for improving clinical and criminal justice outcomes for justice-involved people with mental illness (Epperson et al., 2014; Skeem et al., 2011).

However, any program or intervention—even those delivered with good intentions and shown to have positive effects—can have unintended adverse effects (Barlow, 2010; Dimidjian & Hollon, 2010; Lilienfeld, 2007). For example, for adolescents, antidepressant medication can both ameliorate symptoms of depression and increase suicide risk (Dimidjian & Hollon, 2010). Identifying adverse effects of programs is arguably as important as identifying their positive effects (Lilienfeld, 2007). As described in the following sections, the present study examines whether stigma is an unintended negative effect of placement into specialty programming for justice-involved people with mental illness.

Increasing Internalized Stigma Via Program Assignment

The most fundamental way that specialty programming may be stigmatizing is through the simple act of identifying people with mental illness by assigning them to a separate justice program. Although stigma is multifaceted, its core involves the rejection of one group by the majority group due to negative attitudes regarding the “inferior” stigmatized group (Angermeyer & Matschinger, 1995; Corrigan & Cooper, 2005; Link et al., 1999; Link & Phelan, 1999; Major & O’Brien, 2005). Because justice-involved people with mental illness are members of multiple groups toward which the public has negative attitudes—they have a mental illness, they are involved in the justice system, and often they have co-occurring substance use disorders (Eno Louden & Skeem, 2013; Hartwell, 2004; LeBel, 2012; Winnick & Bodkin, 2008)—they likely experience some degree of stigma regardless of the type of programming they receive within the justice system. However, placement in a “mental health” court, specialty caseload, or other specialized program highlights a person’s mental illness—perhaps making this feature more salient to the person, their officer, their friends and family members, and others.

For example, clients on specialty probation are supervised by officers who are known by other officers (and potentially other probation clients) as supervising “mental health caseloads.” The process of signing in to meet with a mental health officer and then sitting in a waiting room with others may make it salient that an individual is categorically different from other probation clients. The mere presence of a label, in this case “client with mental illness,” is enough to evoke stigmatizing attitudes from others (Hinshaw & Stier, 2008).

Members of the stigmatized group can also incorporate these labels into their self-concept (see Link, 1987; Rosenfield, 2012). Sociological labeling theorists argue that labels are transmitted via socialization with others and shaped by personal experience (Link et al., 1987). Those with mental illness may bring to mind society’s stereotypes toward people with mental illness and applying society’s negative attitudes toward mental illness—including beliefs that people with mental illness are incompetent, unpredictable, and dangerous—to himself or herself (Markowitz, 2017). This is known as internalized stigma, or self-stigma. When individuals incorporate labels into their self-concept, this decreases self-esteem, self-efficacy, and overall quality of life (Link, 1987; Rosenfield, 2012; see also Corrigan & Kosyluk, 2013). These negative outcomes run counter to the goals of specialty programs for justice-involved clients.

Increasing Stigma Via Exposure to Professionals With Stigmatizing Attitudes

A second way that placement in a special program for justice-involved people with mental illness could exacerbate stigma is by affecting the extent to which clients are exposed to justice professionals with stigmatizing attitudes. In specialty probation caseloads, officers are different from traditional officers in meaningful ways. Specialty officers have more training in mental health issues and devote their supervision time exclusively to clients with mental illness (Skeem et al., 2006). Research on traditional probation officers suggests that these officers prefer greater social distance from clients with mental illness compared with nondisordered clients (Eno Louden, 2009). Social distance is one way of conceptualizing stigma and is measured as people’s (un)willingness to engage in a variety of social situations with a person with mental illness, such as living next door to them (Corrigan et al., 2001). Beyond desiring social distance from clients with mental illness, traditional probation officers also believe this group is risky and likely to be violent—key components of mental illness stigma (Eno Louden & Skeem, 2013; see also Eno Louden et al., 2018).

Specialty officers may hold more strongly stigmatizing attitudes than those held by traditional officers. This may seem counterintuitive, given that specialty officers often self-select into these caseloads because of their interest in or experience with mental illness (Skeem et al., 2006). However, specialty officers spend all of their supervision time interacting with clients with mental illness, and their more frequent and intensive contact with these clients could lead to having more stigmatizing attitudes than traditional officers, who supervise a diverse range of clients, have. Professionals who often work with individuals in crisis—responding to violence, self-harm, and other extreme behavior (Link et al., 1999)—tend to have negative experiences with individuals with mental illness (Bahora et al., 2008; Hinshaw & Stier, 2008). While traditional officers may respond to a diverse range of client crises, any crisis a specialty officer encounters by definition involves a client with mental illness. These negative experiences may be particularly frequent and salient for staff who work primarily with justice-involved clients with mental illness.

The presence of stigmatizing attitudes can be distinguished from the expression of those attitudes. Unless attitudes are expressed toward clients, they are unlikely to be detected—and perhaps internalized—by clients. Attitudes could be expressed in a variety of ways. Officers may treat clients with mental illness differently than they would a client without mental illness. This was found in an experimental study, and probation officers preferred more frequent contacts and were more likely to endorse forced treatment when they were presented with a hypothetical client with mental illness than if they were presented with a client without mental illness (see Eno Louden & Skeem, 2013). Furthermore, officers may talk down to clients with mental illness or treat them like they are incompetent or dangerous (see Skeem et al., 2003). As noted in a focus group by a probation client with mental illness:

My PO—I sometimes have the feeling he’s kind of looking down his nose at me—and then again I get that feeling from just about everybody at that office . . . one of them is chuckling to the other one . . . and nods his head over towards me and says, “You can tell when he’s lying cause his lips are moving” (Skeem et al., 2003, p. 444).

Officers may also communicate their stigmatizing attitudes via subtle verbal or behavioral messages that make people with mental illness feel ashamed, invalidated, or feared (Gonzales et al., 2015).

Although we hypothesize that specialty officers may hold more stigmatizing attitudes than traditional officers, specialty officers may better conceal these attitudes from their clients. Stigmatizing attitudes and their associated stereotypes lead to automatic cognitive (e.g., beliefs about dangerousness) and emotional responses (e.g., fear), but people who are motivated to do so can control these reactions (Dunton & Fazio, 2016; Overton & Medina, 2008). Stigmatizing attitudes toward people with mental illness are less acceptable among people trained in health-related professions compared to those trained in criminal justice fields (Weaver et al., 2018)—a key difference between specialty probation officers and traditional officers. As such, professionals who work primarily with clients with mental illness may be more motivated to conceal the stigmatizing attitudes they hold toward these clients. Given that people who hold stigmatizing attitudes but who are motivated to control their reactions to members of stigmatized groups can actually do so (Glaser & Knowles, 2008), clients on specialty mental health programs may be less affected by professionals’ attitudes.

The Present Study

The present study examines the stigma of mental illness associated with the assignment of people to specialty justice programming within the context of specialty mental health probation. To do so, we use data from a longitudinal matched trial that tested the effectiveness of specialty mental health probation compared to traditional probation. The primary study aims are to compare specialty and traditional agencies to test (1) whether clients on specialty mental health probation experience more internalized stigma than similarly situated clients on traditional probation and (2) whether specialty mental health officers hold more stigmatizing attitudes toward their clients than traditional officers do. To address these aims, we use design and statistical techniques to approximate a randomized controlled trial. At baseline (close to the time of probation placement), clients’ supervising officers rated the level of stigma they held toward each of their probation clients. Six months later, each probation client rated their own level of internalized stigma.

The secondary study aim is to assess the association between an officer’s stigmatizing attitudes and internalized stigma reported by their clients. We addressed this aim separately by site (i.e., within specialty probation and within traditional probation) given the possibility that specialty officers may be more motivated to conceal their stigmatizing attitudes toward clients than traditional officers, which could weaken the association between attitudes and internalized stigma.

Method

Probation clients with mental illness were recruited from a prototypic specialty mental health agency (see Skeem et al., 2006) and a traditional probation agency that was similar to the specialty agency in terms of jurisdiction size, urban location, county expenditures on mental health services, and demographic characteristics of its supervisee population. Clients from the traditional agency were matched to clients in the specialty agency on gender, age, ethnicity, length of time on probation, and index offense. Over the course of 1 year, data were collected at three time points from clients and their supervising officers (baseline assessment around 2 months of assignment to probation, 6-month follow-up, and 12-month follow-up). Clients were compensated for participation with cash, and officers were compensated with gift cards. Here, we describe the participants and measures used in the present analyses as well as the propensity weighting approach used to account for any pre-existing differences between the two groups of clients. Further details regarding the method of the larger study, including participant recruitment strategies, are presented in Skeem et al. (2017).

Participants

Probation Clients

One hundred eighty-three clients on mental health caseloads at the specialty site participated in the study (74% of those eligible). At this site, clients were eligible for assignment to a specialty caseload if they had a serious mental illness, which is generally a preexisting diagnosis discovered by the supervising officer (see Manchak et al., 2014). At the traditional site, 176 clients participated (57% of those recruited). These clients were primarily referred by their supervising officers (80%), where officers were asked to refer clients with a known psychiatric diagnosis. The remaining traditional clients were identified by the use of a validated mental health screening questionnaire, where the clients were recruited if they scored above a predetermined cutoff score (see Eno Louden et al., 2013). Characteristics of the two groups of clients are presented in Table 1. The clients at the two sites did not differ significantly from each other in regard to age, gender, ethnicity, age at first arrest, index offense (violent vs nonviolent), or likelihood of past psychiatric hospitalization.

Table 1:

Characteristics of Probation Clients

	Specialty (N = 138)	Traditional (N = 148)	Group difference
Variable	% or M (SD)		p value
Gender
Male	54.3%	59.5%	.41
Female	45.7%	40.5%
Ethnicity			.46
African American	53.6%	52.0%
White	33.3%	37.8%
Hispanic	9.4%	8.8%
Age (SD)	36.3 (10.3)	38.3 (10.9)	.11
Employment status			.48
Full-time	14.0%	10.3%
Part-time	16.2%	13.7%
Unemployed	69.9%	76.0%
Educational attainment			.25
One year or less of college	84.1%	76.2%
Over 1 year college-BS/BA	15.2%	22.4%
Some graduate/postgraduate	0.7%	1.4%
Criminal/childhood abuse history
Index offense			<.001
Person arrest	30.2%	42.1%
Property arrest	37.2%	19.5%
Drug arrest	16.3%	35.3%
Minor/other arrest	16.3%	3.0%
Number of lifetime arrests			.08
One time	11.9%	4.8%
2 times	11.1%	10.2%
≥3 times	77.0%	85.0%
Most serious crime			.001
Person	42.6%	65.1%
Property	24.3%	13.0%
Drug	27.2%	20.5%
Minor	5.9%	1.4%
Violence, prior 6 months	38.4%	28.4%	.11
Time on probation (months)	16.10 (15.41)	11.73 (10.20)	.006
Child abuse seriousness			.004
None	15.2%	31.8%
Bare hand only (no physical injury)	2.2%	4.1%
With an object (no physical injury)	63.8%	45.9%
Resulting in physical injury	18.8%	18.2%
Symptoms
PAI subscales
Anxiety	37.04 (13.09)	29.95 (12.57)	<.001
Paranoia	33.32 (9.16)	33.32 (11.70)	.99
Mania	33.04 (12.15)	31.96 (11.35)	.44
Schizophrenia	30.25 (12.08)	27.03 (12.19)	.03
Antisocial	26.66 (10.79)	27.08 (10.91)	.75
Aggression	24.23 (11.07)	23.17 (10.46)	.41
Alcohol	9.37 (8.00)	10.31 (7.85)	.32
Drug	13.73 (8.02)	15.16 (8.51)	.15
Psychiatric symptoms (CSI total)	30.34 (12.06)	26.61 (12.55)	.011
Researcher-rated psychiatric functioning (GAF)	46.03 (11.86)	53.94 (14.71)	<.001

Note. PAI = Personality Assessment Inventory subscales; CSI = Colorado Symptom Index; GAF = Global Assessment of Functioning.

The measure of internalized stigma was not administered to 73 clients in the larger study because the measure was added after the study began, so the total sample size for the present study was 286. There were no statistically significant differences between clients with and without the measure of internalized stigma on any of the demographic, clinical, or criminal variables listed above.

Officers

There were 15 specialty officers and 87 traditional officers (each officer supervised multiple clients in the study). Officers at the two sites were of similar ages (M = 44.9, SD = 12.5 at the specialty site and M = 46.0, SD = 10.8 at the traditional site), but the officers at the specialty site were more likely to be female, 80% versus 48.3%, χ²(1) = 5.17, p = .02, and White, 60.0% versus 10.3%, χ²(1) = 21.71, p < .001, compared to officers at the traditional site. Specialty officers were more likely to have education beyond a bachelor’s degree, 33.3% versus 11.5%, χ²(1) = 4.86, p = .03, but had been on the job for less time, M = 8.3, SD = 4.7 years versus M = 14.7, SD = 8.2, t(100) = 2.96, p = .004, d = 0.96, compared to their traditional counterparts.

Measures

Officers’ Social Distance

During the baseline officer survey, officers responded via paper-and-pencil questionnaire to a series of items assessing social distance. These items were used in both the 1996 and 2006 iterations of the General Social Survey, a biennial nationally representative survey of public opinion (Pescosolido et al., 2010). Social distance is often used as a measure of stigmatizing attitudes and is assessed by asking participants to rate their willingness to engage in a series of increasingly intimate social interactions with a target (Link et al., 1999). Officers were asked to rate specific clients using the instructions, “Please bring this probationer to mind. For each blank below, mentally insert that probationer’s name.” The questionnaire asks officers to consider how willing they would be to engage in a series of five social interactions with the client (e.g., move next door to, have the person marry into the family). Officers responded to each item using a 4-point scale where 1 = “definitely willing,” and 4 = “definitely not willing.” Responses were summed to create a social distance index, where higher scores indicate higher desire for social distance and thus more stigmatizing attitudes than lower scores. This measure demonstrated excellent internal consistency (Cronbach’s α = .89).

Clients’ Internalized Stigma

Clients completed the Internalized Stigma of Mental Illness (ISMI; Boyd Ritsher et al., 2003) scale during the 6-month interview. The ISMI is a 29-item measure that assesses internalized stigma using a 4-point scale (“Strongly agree” to “Strongly disagree”). Although the ISMI can be scored in a way that yields subscales, we used a total score in our analyses for simplicity. Prior studies suggest that the ISMI has good internal consistency (Cronbach’s α = .80–.92) and correlates with conceptually similar constructs, such as self-esteem (Boyd et al., 2014). Cronbach’s α for the ISMI total in the present study was .91.

Analytic Strategy

We addressed the primary aims of the study using regression models, in which we regressed the clients’ ISMI total scores (Aim 1) and officers’ social distance scores (Aim 2) on the site indicator (specialty vs traditional). Although a randomized controlled trial was not possible, we approximated this research design by attempting to account for any pre-existing differences between specialty and traditional client groups. Specifically, we computed propensity weights to indicate the probability of assignment to specialty rather than traditional probation based on the 21 covariates listed in Table 1—including demographic variables, socioeconomic status, criminal history, history of childhood abuse and substance abuse, and psychiatric symptoms (see Skeem et al., 2017, for details). Propensity weights are helpful when making comparisons between groups that are not equivalent, such as the higher level of psychopathology found in the specialty clients. The propensity weights ensure that differences in the constructs of interest are attributable to true differences by program type rather than an artifact of pre-existing differences between the clients. Covariates were selected a priori, and propensity weights were calculated via SuperLearner (van der Laan et al., 2007), a method that utilizes machine learning and cross-validation to create an optimal weighted combination of covariates (see Skeem et al., 2017 for details). SuperLearner is able to handle nonparametric data as well as outliers (van der Laan et al., 2007), making it particularly well-suited to account for the covariates in this study.

We also controlled for covariates at the officer level known to be related to stigma. Specifically, we controlled for officer gender and ethnicity as covariates given the relationship between these variables and stigma found in prior research (Corrigan & Watson, 2002; Parcesepe & Cabassa, 2013). In addition, we controlled for officers’ caseload size because, as noted earlier, this differed between the sites and could affect the relationship between the variables as the officers at the former had more clients than the officers at the latter and the amount of contact one has with clients with mental illness could affect attitudes toward this group.

Because the officers within each agency supervised multiple clients, clients were not independent of each other. There is likely dependence in clients’ responses as they are supervised by the same officer. To account for this clustering, we used cluster-robust standard errors in the analyses for all three aims (Rogers, 1994). Cluster-robust standard errors are another version of a sandwich estimator (similar to Huber–White standard errors; Huber, 1967; White, 1980), commonly used with a clustered sample. We chose this method for handling cluster dependence rather than using hierarchical linear modeling techniques because officer effects were not the main focus of the study, and we sought to take full advantage of the propensity weights, which were designed for use at the client level.

Results

Aim 1: Do Specialty Mental Health Probation Clients Experience Greater Internalized Stigma Than Traditional Clients?

Controlling for officers’ caseload size, gender, and ethnicity, probation clients experienced significantly higher levels of internalized stigma of mental illness at the specialty site than the traditional site, β = 8.45, SE^robust = 3.06, F(4, 88) = 3.75, p = .007. Specifically, after adjusting for potential confounders using propensity weights and controlling for officers’ caseload size, gender, and ethnicity, the average ISMI total scores at the traditional and specialty sites were 59.14 (SD = 12.94) and 67.60 (SD = 14.56), respectively. Clients at the specialty site scored an average 8.45 points higher (SE^robust =3.06). Cohen’s d, calculated using adjusted means accounting for propensity weights, was estimated at 0.61, a medium effect size. The intraclass correlation (ICC) was estimated at 10%, suggesting that a minority of variation in ISMI scores is at the officer level. In sum, specialty clients experience moderately greater stigma than traditional clients, and little of that difference is attributable to officers.

Aim 2: Do Specialty and Traditional Officers Differ in Their Attitudes Toward Their Clients With Mental Illness?

After adjusting for potential confounders using clients’ propensity weights and controlling for officers’ caseloads, gender, and ethnicity, officers’ social distance scores for clients at the specialty site did not significantly differ from those for clients at the traditional site, β = .89, SE^robust = 0.94, F(1, 82) = .90, p = .35. Specifically, officers’ average social distance score for clients at the traditional site was estimated at 16.21 (SD = 3.67), and officers scored clients at the specialty site an average of only 0.13 points higher (17.10, SD = 3.29).

ICC was estimated at 49%, suggesting that about half of the variation in social distance scores is at the officer level. This is not surprising given that officers generated social distance scores for their clients. Given this finding, we ran a supplemental regression analysis at the officer level. Here, we computed means of social distance ratings by officer and compared the two sites without using clients’ propensity weights as these account for variance by client rather than by officer. In this analysis, there were no significant differences in officers’ attitudes by site. This lends confidence to the basic finding that specialty and traditional officers hold similar levels of stigmatizing attitudes toward their clients.

Secondary Aim: Are Officers’ Stigmatizing Attitudes Associated With Clients’ Internalized Stigma Within Traditional or Specialty Agencies?

To examine whether officers’ social distance ratings are associated with clients’ internalized stigma, we examined the relationship between officers’ ratings of individual clients on their caseloads and those clients’ internalized stigma. For this analysis, we omitted 22 cases where the client was supervised by a different officer at 6 months than at baseline (resulting in n = 264).

At the traditional site, controlling for officers’ caseload size, gender, and ethnicity, officers’ attitudes toward their clients weakly predicted internalized stigma. Specifically, at the traditional site, for every unit increase in the client’s social distance measure, their ISMI score is estimated to increase by about 0.72 units—as summarized in Table 2. In contrast, at the specialty site, officers’ attitudes toward their clients did not significantly affect internalized stigma (controlling for officers’ caseload size, gender, and ethnicity).

Table 2:

Regression Coefficients Estimating Association Between Officers’ Social Distance Scores and Clients’ Internalized Stigma by Agency Type

Variable	Specialty siteN = 15	Traditional siteN = 87
Variable	Estimate (SE^robust)	Estimate (SE^robust)
Social distance	−0.02 (0.52)	0.72 (0.30)*
Caseload size	−0.14 (0.23)	−0.86 (0.88)
Officer gender	0.62 (3.65)	−3.27 (2.38)
Officer race	3.90 (4.06)	−3.52 (6.76)
Intercept (β₁)	65.26 (6.24)***	49.38 (6.16)***
R ²	.01	.06

p < .05. **p < .01. ***p < .005.

This relationship is depicted in Figure 1, where solid circles represent clients at the traditional site, and hollow circles represent clients at the specialty site. The relationships between probationer-level social distance score and ISMI composite score are represented by dashed and solid lines for specialty and traditional sites, respectively, adjusted for other variables in the model. As shown in Figure 1, the dotted line (specialty site) is essentially flat. To put the statistically significant positive slope of the solid line (traditional site) into perspective, for every standard deviation increase in the social distance score at the traditional site, the ISMI composite is expected to increase by about 0.20 standard deviations, indicating a small effect size. In sum, traditional officers’ stigmatizing attitudes toward their clients were weakly associated with clients’ levels of internalized stigma, but this was not the case in the specialty agency.

Figure 1:

Relationship Between Officer Social Distance Ratings and Client Internalized Stigma

Discussion

In this study, we examined the extent to which placement on traditional versus specialty mental health probation is associated with internalized stigma among clients and the extent to which this stigma may be related to officers’ stigmatizing attitudes toward clients with mental illness. Given the proliferation of specialty programs for justice-involved people with mental illness, it is important that policymakers and practitioners understand the full spectrum of these programs’ effects, both positive and negative. Our key findings may be organized into 3 points. First, compared to placement on traditional probation, placement on specialty mental health caseloads is associated with greater internalized stigma among clients with mental illness. Second, specialty and traditional officers have similarly high levels of stigmatizing attitudes toward the clients with mental illness on their caseloads. Finally, officers’ stigmatizing attitudes toward particular supervisees are associated with clients’ internalized stigma for traditional caseloads but not for specialty caseloads. Each of these findings will be discussed next.

Supervision on Specialty Mental Health Probation Is Associated With More Internalized Stigma Than Supervision on Traditional Probation

After accounting for potential confounders, specialty probation clients reported moderately greater internalized stigma than traditional clients. Internalized stigma, or self-stigma, results from an internalization of public stigma—the negative attitudes toward people with mental illness that are held by much of society (Rüsch et al., 2005). These negative attitudes are in part based on faulty assumptions about people with mental illness, such as that they are irresponsible, weak, unpredictable, and violent (Rüsch et al., 2005). There is evidence that merely labeling an individual as having a mental illness increases the stigmatizing attitudes toward him or her as measured by desire for social distance and perceived dangerousness (Angermeyer & Matschinger, 2005; Link et al., 1987, 1999). Although some scholars debate the link between labeling and public stigma (see Wright et al., 2011), there seems to be some agreement that individuals with mental illness experience harm from being labeled such in the form of lowered self-esteem, lowered self-efficacy, and reluctance to seek needed treatment (Rosenfield, 2012).

Although our quasi-experimental design cannot support causal inference, we used a propensity weighting approach to eliminate potential differences between clients at the specialty and traditional agencies to avoid a type I error. Overall, we found a moderate difference between clients’ levels of internalized stigma at the two sites. Because there is so little research on internalized stigma among justice-involved people with mental illness, it is difficult to draw comparisons between the present findings and those of prior research. Research examining relative stigma among forensic patients provides some context for our findings. Livingston and colleagues (2011) compared self-stigma among people with mental illness receiving forensic versus nonforensic mental health services in Canada (N = 91). For clients who were receiving mental health services, the authors found no significant increase in internalized stigma when clients were additionally labeled as “forensic” (or legal) cases. In the present study, we compared self-stigma among probation clients placed on mental health versus generalist caseloads. For clients who were on probation, we found higher rates of internalized stigma when clients were additionally labeled as “mentally ill.” The difference in these two studies’ findings may reflect the fact that serious mental illness is one of the most stigmatized conditions in society—more stigmatizing, perhaps, than even criminal justice involvement (Markowitz, 2017).

Specialty and Traditional Officers Have Similarly High Levels of Stigmatizing Attitudes

Specialty and traditional officers both endorsed a moderately high desire for social distance from their supervisees with mental illness; their mean item score of 3.30 indicated they are “definitely not willing” to engage in social contact with these supervisees. To place this into context, community resident respondents who rated vignettes of hypothetical non-justice-involved people with and without mental illness in the General Social Survey reported the greatest desire for social distance from the people portrayed as having schizophrenia (M = 2.75, SD = 0.59) and cocaine dependence (M = 3.20, SD = 0.57; Link et al., 1999). As such, officers’ desire for social distance from their supervisees was as high as community residents’ desire for social distance from the most stigmatized conditions in the General Social Survey.

There are multiple factors that could explain officers’ attitudes, all of which could be operating simultaneously. First, at the most basic level, stigmatizing attitudes are widely held by members of the public, and insensitive treatment of people with mental illness is still considered socially acceptable by many (e.g., use of the word “crazy,” media depictions of violence; Hinshaw & Stier, 2008; Wilson et al., 2000). Neither traditional nor specialty officers are immune to social norms. Second, for traditional officers, clients with mental illness are challenging cases whose needs are difficult to meet with the resources generally available within their agencies (Skeem et al., 2003). Traditional officers perceive these clients as being riskier and more likely to be violent than typical clients and, as a result, monitor these clients closely (Eno Louden et al., 2018; Eno Louden & Skeem, 2013; Skeem et al., 2003). Third, specialty officers spend a lot of time helping their supervisees navigate an often-frustrating mental health system (Skeem et al., 2006). Both types of officers witness the negative effects of mental illness when their clients experience crises, but the volume is higher for specialty officers because their caseload is exclusive to people with mental illness. In contrast, specialty officers spend more time with clients with mental illness and thus may be more likely to experience positive (and neutral) interactions with these clients. Further research is needed to determine the effect officers’ attitudes have on how they interact with and make decisions for their supervisees.

Officers’ Attitudes Toward Clients Predict Clients’ Internalized Stigma in Traditional (Not Specialty) Probation

Officers in both agencies reported stigmatizing attitudes toward their clients, but only the attitudes held by the traditional officers were associated with clients’ internalized stigma. In the traditional agency only, an officer’s attitude toward a specific client was associated with that client’s level of internalized stigma. We tentatively speculate that this difference between specialty and traditional officers reflects differences in motivation to control prejudice. People differ in the extent to which they are motivated to control prejudice toward members of stigmatized groups (Overton & Medina, 2008; West & Hewstone, 2012). People who have a strong motivation to control prejudice can suppress negative reactions to members of a stigmatized group even when holding negative attitudes toward that group. Justice professionals who choose to work primarily or exclusively with clients who have mental illnesses may be relatively motivated to control prejudice. For example, mental health care providers both hold stigmatizing attitudes toward their clients (see Rao et al., 2009; Schulze, 2009) and often support antistigma efforts (Schulze, 2009). This suggests it is possible to simultaneously have some degree of fear, disgust, or pity for one’s clients and understand (and seek to prevent) the negative effects that stigma has on people with mental illness. This same phenomenon could be operating among justice professionals who elect to work primarily with clients with mental illness. These professionals may censor actions (e.g., statements, facial expressions) that could divulge their negative attitudes, providing less indication of stigma for clients to internalize. It is important to underscore that this interpretation is speculative. In this study, we did not measure motivation to control prejudice. This study provides direction for future research that could examine whether and how professionals’ motivation to control prejudice relates to internalized stigma among clients with mental illness in specialty justice programs.

Limitations

As with any research, this study has limitations that should be considered. First, clients were not randomly assigned to specialty versus traditional supervision. Although ideal from a methodological standpoint, this type of research design was not feasible due to agency policies. Our use of propensity weights allowed us to account for any pre-existing differences between the two groups of clients, but it is possible that unobserved differences were not captured by the 21 variables that were controlled via this approach. Second, only one specialty and one traditional agency were included, so further research is needed to understand the extent to which our findings generalize to other agencies. Of note, the specialty agency where data were collected was chosen because it possessed the five key features of specialty mental health probation agencies identified by Skeem and colleagues (2006), and the traditional agency was similar to the specialty agency in important geographical and socioeconomic factors. Third, officers’ attitudes and clients’ internalized stigma were only measured at one point in time. Although the two were associated in one agency, the direction of influence is unclear and could reflect a third variable. In future research, it will be important to assess whether officers’ attitudes are associated with changes in clients’ internalized stigma, pre- and postspecialty program placement. As noted above, it will also be important to measure officers’ motivation to control prejudice in future research to assess whether and how this affects clients’ internalized stigma. Third, the measure of officer stigma we used had some limitations. For example, although social distance is often used to assess stigmatizing attitudes in members of the public, officers may view social distance as a component of professionalism. Furthermore, this measure is only able to measure attitudes that officers are aware of and can thus self-report (or conceal) and cannot measure implicit bias toward clients. In addition, although by asking officers to rate their attitudes toward a specific client allowed us to link officers’ attitudes toward that client with the client’s own internalized stigma, it did not allow us to disentangle stigmatizing attitudes toward people with mental illness from stigmatizing attitudes toward justice-involved people (see Eno Louden et al., 2018). Nevertheless, the findings of the present study provide a glimpse into probation clients’ experiences of stigma in specialty and traditional agencies and provide a useful starting point for future research.

Conclusions and Implications

Although our results must be replicated in an experiment that can fully support causal inference, they suggest that assigning clients to a justice program that is specially designated as a mental health program may heighten the salience of their mental illness and increase their internalized stigma. Although both specialty and traditional officers hold quite stigmatizing attitudes toward their supervisees, only traditional officers’ attitudes were associated with their clients’ experiences of internalized stigma, perhaps because these officers were less motivated to control the “leakage” of those negative attitudes than specialty officers.

Stigma is harmful—it lowers the quality of life for stigmatized people and reduces the likelihood that they will seek mental health treatment (Link, 1987; Rosenfield, 2012). In future research, it would be helpful to test whether relevant interventions reduce stigma among justice-involved clients with mental illness. First, interventions that target internalized stigma may be beneficial for clients of specialty probation and similar programs. Group-based interventions, such as Ending Self-Stigma and Narrative Enhancement and Cognitive Therapy, have shown promise in reducing self-stigma in other contexts (see Yanos et al., 2015). Second, professionals who work with clients with mental illness (particularly those in traditional agencies) may benefit from programs designed to reduce stigmatizing attitudes toward people with mental illness. Programs such as those designed for use with police officers and other professionals could be adapted for this purpose (see Merino et al., 2018; Mulay et al., 2016; Watson et al., 2017).

We do not believe our results speak against the implementation of special programming for justice-involved people with mental illness, including specialty probation. Many evidence-based interventions offer both positive and negative effects. What our findings do suggest is that stigma associated with placement in such programs must be ameliorated so that clients may benefit more fully from the positive clinical and criminal justice outcomes these programs offer.

Footnotes

Authors’ Note:

Portions of this research were presented at a conference of the American Psychology-Law Society and the North American Correctional and Criminal Justice Psychology Conference. We thank Sarah Manchak, Tracy Johnson, and Sarah Vidal for their significant contributions to data collection and project management, and John Monahan, PhD, Henry Steadman, PhD, and the other MacArthur Research Network members for their insightful input into the larger study. We also express our appreciation to the probation chiefs at our research sites: V. J. Lair, Theresa Williams, and David Davies. This research was funded by the John D. & Catherine T. MacArthur Research Network on Mandated Community Treatment.

ORCID iD

Jennifer Eno Louden

Jennifer Eno Louden is an associate professor of psychology at The University of Texas at El Paso. Her research primarily focuses on identifying best practices in the criminal justice system, particularly for vulnerable populations such as justice-involved people with mental illness.

Perman Gochyyev is a research statistician at the University of California, Berkeley. Perman received his PhD in Quantitative Methods and Evaluation from UC Berkeley in 2015. His research focuses on latent variable and multilevel modeling, survey methodology, and issues related to causal inference in behavioral statistics.

Jennifer L. Skeem is the Florence Krenz Mack Professor of Social Welfare and a Professor of Public Policy at the University of California, Berkeley. She is a psychologist who directs the Risk-Resilience Lab and writes and teaches about the intersection between behavioral science and the justice system.

References

Angermeyer

M. C.

Matschinger

(1995). Violent attacks on public figures by persons suffering from psychiatric disorders: Their effects on the social distance towards the mentally ill. European Archives of Psychiatry and Clinical Neuroscience, 245, 159–164. https://doi.org/10.1007/BF02193089

Angermeyer

M. C.

Matschinger

(2005). Labeling—stereotype—discrimination. An investigation of the stigma process. Social Psychiatry and Psychiatric Epidemiology, 40, 391–395. https://doi.org/10.1007/s00127-005-0903-4

Bahora

Hanafi

Chien

V. H.

Compton

M. T.

(2008). Preliminary evidence of effects of crisis intervention team training on self-efficacy and social distance. Administration and Policy in Mental Health and Mental Health Services Research, 35, 159–167. https://doi.org/10.1007/s10488-007-0153-8

Barlow

D. H.

(2010). Negative effects from psychological treatments: A perspective. American Psychologist, 65, 13–20. https://doi.org/10.1037/a0015643

Boyd

J. E.

Adler

E. P.

Otilingam

P. G.

Peters

(2014). Internalized Stigma of Mental Illness (ISMI) Scale: A multinational review. Comprehensive Psychiatry, 55, 221–231. https://doi.org/10.1016/j.comppsych.2013.06.005

Boyd Ritsher

Otilingam

P. G.

Grajales

(2003). Internalized stigma of mental illness: Psychometric properties of a new measure. Psychiatry Research, 121, 31–49. https://doi.org/10.1016/j.psychres.2003.08.008

Cloyes

Wong

Latimer

Abarca

(2009). Time to prison return for offenders with serious mental illness released from prison. Criminal Justice and Behavior, 37, 175–187. https://doi.org/10.1177/0093854809354370

Corrigan

P. W.

Cooper

A. E.

(2005). Mental illness and dangerousness: Fact or misperception, and implications for stigma. In Corrigan

P. W.

(Ed.), On the stigma of mental illness: Practical strategies for research and social change (pp. 165–180). American Psychological Association.

Corrigan

P. W.

Green

Lundin

Kubiak

M. A.

Penn

D. L.

(2001). Familiarity with and social distance from people who have serious mental illness. Psychiatric Services, 52, 953–958. https://doi.org/10.1176/appi.ps.52.7.953

10.

Corrigan

P. W.

Kosyluk

K. A.

(2013). Erasing the stigma: Where science meets advocacy. Basic & Applied Social Psychology, 1, 131–140. https://doi.org/10.1080/01973533.2012.746598

11.

Corrigan

P. W.

Watson

A. C.

(2002). The paradox of self-stigma and mental illness. Clinical Psychology: Science and Practice, 9, 35–53. https://doi.org/10.1093/clipsy.9.1.35

12.

Council of State Governments. (2002). Criminal justice/mental health consensus project. https://csgjusticecenter.org/mental-health-projects/report-of-the-consensus-project/

13.

Dimidjian

Hollon

S. D.

(2010). How would we know if psychotherapy were harmful? American Psychologist, 65, 21–33. https://doi.org/10.1037/a0017299

14.

Dunton

Fazio

(2016). An individual difference measure of motivation to control prejudiced reactions. Personality and Social Psychology Bulletin, 23, 316–326. https://doi.org/10.1177/0146167297233009

15.

Eno Louden

(2009). Effect of mental disorder and substance abuse stigma on probation officers’ case management decisions [Doctoral dissertation]. ProQuest Dissertations Publishing, University of California, Irvine.

16.

Eno Louden

Manchak

S. M.

Ricks

E. P.

Kennealy

(2018). The role of stigma toward mental illness in probation officers’ perceptions of risk and case management decisions. Criminal Justice and Behavior, 45, 573–588. https://doi.org/10.1177/0093854818756148

17.

Eno Louden

Skeem

J. L.

(2013). How do probation officers assess and manage recidivism and violence risk for probationers with mental disorder? An experimental investigation. Law and Human Behavior, 7, 22–34. https://doi.org/10.1037/h0093991

18.

Eno Louden

Skeem

J. L.

Blevins

(2013). Comparing the predictive utility of two screening tools for mental disorder among probationers. Psychological Assessment, 25, 405–415. https://doi.org/10.1037/a0031213

19.

Eno Louden

Skeem

J. L.

Camp

Christensen

(2008). Supervising probationers with mental disorder: How do agencies respond to violations? Criminal Justice and Behavior, 35, 832–847. https://doi.org/10.1177/0093854808319042

20.

Epperson

Wolff

Morgan

Fisher

Frueh

Huening

(2014). Envisioning the next generation of behavioral health and criminal justice interventions. International Journal of Law and Psychiatry, 37, 427–438. https://doi.org/10.1016/j.ijlp.2014.02.015

21.

Fazel

Danesh

(2002). Serious mental disorder in 23000 prisoners: A systematic review of 62 surveys. The Lancet, 359, 545–550. https://doi.org/10.1016/s0140-6736(02)07740-1

22.

Glaser

Knowles

(2008). Implicit motivation to control prejudice. Journal of Experimental Social Psychology, 44, 164–172. https://doi.org/10.1016/j.jesp.2007.01.002

23.

Gonzales

Davidoff

Nadal

Yanos

(2015). Microaggressions experienced by persons with mental illnesses: An exploratory study. Psychiatric Rehabilitation Journal, 38, 234–241. https://doi.org/10.1037/prj0000096

24.

Hartwell

(2004). Triple stigma: Persons with mental illness and substance abuse problems in the criminal justice system. Criminal Justice Policy Review, 15, 84–99. https://doi.org/10.1177/0887403403255064

25.

Hinshaw

S. P.

Stier

(2008). Stigma as related to mental disorders. Annual Review of Clinical Psychology, 4, 367–393. https://doi.org/10.1146/annurev.clinpsy.4.022007.141245

26.

Huber

P. J.

(1967). The behavior of maximum likelihood estimates under non-standard conditions. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1, 221–233.

27.

Kaeble

Cowhig

(2018). Correctional populations in the United States, 2016. Bureau of Justice Statistics, U.S. Department of Justice. https://www.bjs.gov/content/pub/pdf/cpus16.pdf

28.

LeBel

(2012). Invisible stripes? Formerly incarcerated persons’ perceptions of stigma. Deviant Behavior, 33, 89–107. https://doi.org/10.1080/01639625.2010.538365

29.

Lilienfeld

S. O.

(2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2, 53–70. https://doi.org/10.1111/j.1745-6916.2007.00029.x

30.

Link

B. G.

(1987). Understanding labeling effects in the area of mental disorders: An assessment of the effects of expectations of rejection. American Sociological Review, 52, 96–112. https://www-jstor-org-443.web.bisu.edu.cn/stable/2095395

31.

Link

B. G.

Cullen

F. T.

Frank

Wozniak

J. F.

(1987). The social rejection of former mental patients: Understanding why labels matter. American Journal of Sociology, 92, 1461–1500. https://doi.org/10.1086/228672

32.

Link

B. G.

Phelan

J. C.

(1999). Labeling and stigma. In Aneshensel

C. S.

Phelan

J. C.

(Eds.), Handbook of the sociology of mental health (pp. 481–494). Kluwer.

33.

Link

B. G.

Phelan

J. C.

Bresnahan

Steuve

Pescosolido

(1999). Public conceptions of mental illness: Labels, causes, dangerousness, and social distance. American Journal of Public Health, 89, 1328–1333. https://doi.org/10.2105/AJPH.89.9.1328

34.

Livingston

J. D.

Rossiter

K. R.

Verdun-Jones

S. N.

(2011). “Forensic” labelling: An empirical assessment of its effects on self-stigma for people with severe mental illness. Psychiatry Research, 188, 115–122. https://doi.org/10.1016/j.psychres.2011.01.018

35.

Lurigio

Epperson

Canada

Babchuk

(2012). Specialized probation programs for people with mental illnesses: A review of practices and research. Journal of Crime and Justice, 35, 317–326. https://doi.org/10.1080/0735648X.2012.682845

36.

Major

O’Brien

(2005). The social psychology of stigma. Annual Review of Psychology, 56, 393–421. https://doi.org/10.1146/annurev.psych.56.091103.070137

37.

Manchak

S. M.

Skeem

J. L.

Kennealy

P. J.

Eno Louden

(2014). High-fidelity specialty mental health probation improves officer practices, treatment access, and rule compliance. Law and Human Behavior, 38, 450–461. https://doi.org/10.1037/lhb0000076

38.

Markowitz

F. E.

(2017). Labeling theory and mental illness. In Farrington

D. P.

Murray

(Eds.), Labeling theory: Empirical tests (pp. 45–62). Routledge.

39.

Merino

Adams

Hall

W. J.

(2018). Implicit bias and mental health professionals: Priorities and directions for research. Psychiatric Services, 69, 723–725. https://doi.org/10.1176/appi.ps.201700294

40.

Messina

Burdon

Hagopian

Prendergast

(2004). One year return to custody rates among co-disordered offenders. Behavioral Sciences & the Law, 22, 503–518. https://doi.org/10.1002/bsl.600

41.

Mulay

A. L.

Vayshenker

West

M. L.

Kelly

(2016). Crisis Intervention Training and implicit stigma toward mental illness: Reducing bias among criminal justice personnel. International Journal of Forensic Mental Health, 15, 369–381. https://doi.org/10.1080/14999013.2016.1208308

42.

Overton

Medina

(2008). The stigma of mental illness. Journal of Counseling & Development, 86, 143–151. https://doi.org/10.1002/j.1556-6678.2008.tb00491.x

43.

Parcesepe

Cabassa

(2013). Public stigma of mental illness in the United States: A systematic literature review. Administration and Policy in Mental Health and Mental Health Services Research, 40(5), 384–399. https://doi.org/10.1007/s10488-012-0430-z

44.

Pescosolido

B. A.

Martin

J. K.

Long

J. S.

Medina

Phelan

Link

B. G.

(2010). “A disease like any other?” A decade of change in public reactions to schizophrenia, depression, and alcohol dependence. American Journal of Psychiatry, 167, 1321–1330. https://doi.org/10.1176/appi.ajp.2010.09121743

45.

Rao

Mahadevappa

Pillay

Sessay

Abraham

Luty

(2009). A study of stigmatized attitudes towards people with mental health problems among health professionals. Journal of Psychiatric and Mental Health Nursing, 16, 279–284. https://doi.org/10.1111/j.1365-2850.2008.01369.x

46.

Rogers

(1994). Regression standard errors in clustered samples. Stata Technical Bulletin, 3, 19–23.

47.

Rosenfield

(2012). Triple jeopardy? Mental health at the intersection of gender, race, and class. Social Science & Medicine, 74, 1791–1801. https://doi.org/10.1016/j.socscimed.2011.11.010

48.

Rüsch

Angermeyer

M. C.

Corrigan

P. W.

(2005). Mental illness stigma: Concepts, consequences, and initiatives to reduce stigma. European Psychiatry, 20, 529–539. https://doi.org/10.1016/j.eurpsy.2005.04.004

49.

Schulze

(2009). Stigma and mental health professionals: A review of the evidence on an intricate relationship. International Review of Psychiatry, 19, 137–155. https://doi.org/10.1080/09540260701278929

50.

Skeem

J. L.

Emke-Francis

Eno Louden

(2006). Probation, mental health, and mandated treatment: A national survey. Criminal Justice and Behavior, 33, 158–184. https://doi.org/10.1177/0093854805284420

51.

Skeem

J. L.

Encandela

Eno Louden

(2003). Perspectives on probation and mandated mental health treatment in specialized and traditional probation departments. Behavioral Sciences & the Law, 21, 429–458. https://doi.org/10.1002/bsl.547

52.

Skeem

J. L.

Manchak

S. M.

Montoya

(2017). Comparing public safety outcomes for traditional probation vs specialty mental health probation. JAMA Psychiatry, 74, 942–948. https://doi.org/10.1001/jamapsychiatry.2017.1384

53.

Skeem

J. L.

Manchak

S. M.

Peterson

J. K.

(2011). Correctional policy for offenders with mental illness: Creating a new paradigm for recidivism reduction. Law and Human Behavior, 35, 110–126. https://doi.org/10.1007/s10979-010-9223-7

54.

Skeem

J. L.

Winter

Kennealy

Eno Louden

Tatar

II . (2014). Offenders with mental illness have criminogenic needs, too: Toward recidivism reduction. Law and Human Behavior, 38, 212–224. https://doi.org/10.1037/lhb0000054

55.

Steadman

Osher

F. C. R.

Case

Samuels

(2009). Prevalence of serious mental illness among jail inmates. Psychiatric Services, 60, 761–765. https://doi.org/10.1176/ps.2009.60.6.761

56.

van der Laan

M. J.

Polley

E. C.

Hubbard

A. E.

(2007). Super learner. Statistical Applications in Genetics and Molecular Biology, 6, 1–21. https://doi.org/10.2202/1544-6115.1309

57.

Watson

Compton

Draine

(2017). The Crisis Intervention Team (CIT) model: An evidence-based policing practice? Behavioral Sciences & the Law, 35, 431–441. https://doi.org/10.1002/bsl.2304

58.

Weaver

Lee

Choi

Johnson

Clements

(2018). Offenders living with mental illness: How are they perceived by future professionals? Journal of Social Work, 19, 83–101. https://doi.org/10.1177/1468017318757383

59.

West

Hewstone

(2012). Relatively socially acceptable prejudice within and between societies. Journal of Community & Applied Social Psychology, 22, 269–282. https://doi.org/10.1002/casp.1112

60.

White

(1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–830.

61.

Wilson

Nairn

Coverdale

Panapa

(2000). How mental illness is portrayed in children’s television: A prospective study. British Journal of Psychiatry, 176, 440–443. https://doi.org/10.1192/bjp.176.5.440

62.

Winnick

Bodkin

(2008). Anticipated stigma and stigma management among those to be labeled “ex-con.” Deviant Behavior, 29, 295–333. https://doi.org/10.1080/01639620701588081

63.

Wolff

Epperson

Shi

Huening

Schumann

Sullivan

I. R.

(2014). Mental health specialized probation caseloads: Are they effective? International Journal of Law and Psychiatry, 37, 464–472. https://doi.org/10.1016/j.ijlp.2014.02.019

64.

Wright

Jorm

A. F.

Mackinnon

A. J.

(2011). Labeling of mental disorders and stigma in young people. Social Science & Medicine, 73, 498–506. https://doi.org/10.1016/j.socscimed.2011.06.015

65.

Yanos

P. T.

Lucksted

Drapalski

A. L.

Roe

Lysaker

(2015). Interventions targeting mental health self-stigma: A review and comparison. Psychiatric Rehabilitation Journal, 38, 171–178. https://doi.org/10.1037/prj0000100