Assessment and Modification of General Criminal Attitudes Among Men Who Have Sexually Offended

Abstract

General criminal attitudes have been well established as a dynamic risk factor for the origin, maintenance, and continuation of criminal behavior. Guided by the risk–need–responsivity (RNR) framework, this study examined self-reported change on a measure of general criminal attitudes in a sample of incarcerated men who participated in a sexual offense treatment program. Participants were administered the original version of the Criminal Sentiments Scale (CSS) and other measures at pretreatment and posttreatment and followed up in the community an average 14 years post-release. The results demonstrated that CSS total and subscale scores predicted general and violent recidivism, showed convergence with actuarial measures of criminogenic need, and had clinically meaningful associations with responsivity considerations. Pre–post changes on the CSS were associated with decreased general and violent recidivism controlling for pretreatment score and baseline risk. Implications for forensic assessment and correctional intervention are discussed.

Keywords

Criminal Sentiments Scale criminal attitudes change risk–need–responsivity sexual offense

The construct of criminal attitudes is integrated into much of general criminological theory, research, and practice. According to Andrews and Bonta (2010a), “attitudes are evaluative cognitions and feelings that organize the actor’s decision to act and behavior toward a person, thing, or action” (p. 234). When attitudes are favorable toward crime and/or hostile toward authority or convention, criminal conduct may result. Criminal attitudes assume a central role in the predominant sociological theories of Control (Hirschi, 1969) and Differential Association (Sutherland, 1947) as well as the General Personality and Social Psychological perspective (Andrews & Bonta, 1994). Indeed, meta-analytic reviews of the predictors of criminal recidivism (e.g., Gendreau et al., 1996; Olver, Stockdale, & Wormith, 2014; L. Simourd & Andrews, 1994) demonstrate criminal attitudes to be among the top predictors of adult and juvenile offense recidivism (rs = .18–.40). As such, the theoretical and empirical evidence of the value of criminal attitudes prompted Andrews et al. (2006) to regard them as one of the “Central Eight” risk factors. At a practical level, many of the mainstream cognitive type intervention programs such as Courage to Change, Thinking for a Change, Moral Reconation Therapy, Reasoning and Rehabilitation, and Cognitive Self-Change have specific modules related to attitudes and a limited number of standalone criminal attitude programs have also been developed (D. J. Simourd et al., 2016).

Efforts to measure criminal attitudes have most frequently taken the form of self-report measures such as the Measure of Criminal Attitudes and Associates (MCAA; Mills et al., 2002), Psychological Inventory of Criminal Thinking Styles (PICTS; Walters, 1995), Criminal Thinking Profile (CTP; Mitchell & Tafrate, 2012), the Criminal Sentiments Scale (CSS; Gendreau et al., 1979), and the Criminal Sentiments Scale–Modified (CSS-M; D. J. Simourd, 1997). The measures broadly encompass rationalizations, excuses, or blatant endorsements that legitimize criminal behavior or that are oppositional to criminal justice and authority. Moreover, given the ubiquity of criminal attitudes as a correlate of criminal behavior, many structured risk assessment instruments rated by frontline service providers include criminal attitude item content (e.g., Level of Service Inventory–Revised [LSI-R]; Andrews & Bonta, 1995). Extant research supports the construct validity and predictive properties of both self-report measures of criminal attitudes (Walters, 2006) and purpose-built dynamic risk measures containing criminal attitude content (e.g., Olver, Stockdale et al., 2014).

The Risk–Need–Responsivity Relevance of General Criminal Attitudes

The risk–need–responsivity (RNR) framework (Andrews & Bonta, 2010a) is arguably the dominant model for the assessment and treatment of correctional clientele. Research demonstrates that correctional programs subscribing to a successively greater number of its three principles yield progressively larger reductions in recidivism (Andrews & Bonta, 2010b).

The risk principle has two components (Bonta & Andrews, 2007): (a) that service intensity should be matched to the risk level of the individual, such that higher risk cases receive more services, and lower risk cases receive fewer services, and (b) recidivism can be reliably predicted such that high-risk clientele can be accurately discriminated from low-risk individuals on the basis of risk scores. The need principle asserts that dynamic risk factors (i.e., criminogenic needs) should be prioritized for treatment, given that problems in these domains predict increased recidivism and positive changes in these domains should predict decreased recidivism. Finally, the responsivity principle states that correctional services should be grounded in cognitive behavioral methods of change (general responsivity) and service delivery should be tailored to the unique characteristics of clientele (e.g., motivation, culture, learning style, cognitive capability) (specific responsivity).

The RNR model of assessment and treatment has relevance within the construct of criminal attitudes. Correctional clients with entrenched problems in this domain are at elevated risk, particularly given that criminal attitudes tend to underlie and correlate with other risk-need domains (Van de Ven, 2004; Wormith et al., 2007), thus warranting higher intensity services (risk principle). Problems in this need area similarly predict increased recidivism (Olver et al., 2014) while improvements predict decreased recidivism (Wormith, 1984) (need principle). Criminal attitudes may also manifest as hostility or resistance toward services (e.g., con code mentality, poor attitudes toward authority) or signal other vulnerabilities (e.g., lower IQ, weak motivation) that warrant adaptation of services to maximize client engagement and to minimize attrition (responsivity principle). In this vein, criminal attitudes are also predictive of correctional treatment attrition (Mitchell et al., 2013; Olver et al., 2011).

The Role and Relevance of Criminal Attitudes With Sexual Offending Populations

Much of the theory, research, and practice related to the concept of criminal attitudes has been based on nonspecialized correctional populations consisting predominately of male general offense samples. Some knowledge base has been developed among more specialized offending populations such as those convicted for sexual offenses. At a theoretical level, the techniques of neutralization (i.e., “justifications of deviant behavior”), articulated by Sykes and Matza (1957, p. 667) and Bandura’s (1986) disengagement of self-evaluative standards, have been integrated into the conceptualizations of sexual offending (Murphy, 1990). This framework has further informed the assessment and treatment of criminal attitudes with this population. In addition to general measures of criminal attitudes, specific measures of sexual offense attitudes have also been developed. This includes self-report instruments such as the Hanson Sex Attitudes Questionnaire (Hanson et al., 1994), the Bumby Rape and Molest scales (Bumby, 1996), and the Abel and Becker Cognition Scale (Abel et al., 1989). Dynamic sexual violence risk measures (e.g., Violence Risk Scale-Sexual Offense version, VRS-SO; Wong et al., 2003–2017) also incorporate criminal attitude content specific to sexual offending.

Empirically, both general and specific criminal attitudes predict recidivism in sexual offending samples. For instance, a meta-analysis of sexual offense-specific attitudes (e.g., rape myths, sexual entitlement, legitimization of adult-child sexual contact) among sexual offending samples, which included several of the aforementioned measures (Helmus et al., 2013), found that attitudes supportive of sexual offending predicted sexual recidivism (d = .22, k = 46, n = 13,782). Moreover, a meta-analysis of the Level of Service Inventory (LSI) measures, a family of general risk-need assessment tools, found that the general criminal attitude risk-need domain had a small in magnitude effect in the prediction of sexual recidivism (r_w = .10, k = 5, n = 2,389; Olver et al., 2014). Among the sexual offense subgroup of studies in this meta-analysis, ancillary analyses demonstrate the LSI general criminal attitudes domain to be a comparatively stronger predictor of violent (r_w = .15, k = 3, n = 2,175) and general (r_w = .26, k = 4, n = 2,279) recidivism. Measures of general criminal attitudes have also been found to correlate positively with other measures of risk and need in sexual offending samples (Mills & Kroner, 1997; Witte et al., 2006). As such, both general and specific criminal attitudes have risk relevance among individuals convicted of sexual offenses and are worthy of assessment and intervention as part of a broader sexual violence risk management regime.

Finally, criminal attitudes are a common element of sexual offense treatment programs, as documented by the Safer Society Survey (McGrath et al., 2010), with most programs having a module addressing criminal attitudes within the context of a larger comprehensive program. The extant literature has demonstrated that the RNR principles also apply to sexual offense treatment. In their meta-analysis of sexual offense treatment outcome, Hanson et al. (2009) found rates of sexual recidivism were lower for treated groups (10.9%) than untreated groups (19.2%) across 22 studies. As with the general correctional treatment literature, programs subscribing to all three RNR principles demonstrated the largest reductions in recidivism (odds ratio [OR] = 0.21), compared to those adhering to two (OR = 0.63), one (OR = 0.64), or none (OR = 1.17). In an updated meta-analysis of 44 sexual offense treatment outcome studies, Gannon et al. (2019) similarly found lower rates of sexual recidivism for treated (9.5%) versus untreated (14.1%) groups. Moreover, programs coded as promising (OR = 0.56) or most promising (OR = 0.57) in terms of adherence to RNR principles also yielded the most substantive reductions in sexual recidivism.

Criminal Attitude Change

Comparatively less research, however, has examined changes in criminal attitudes (i.e., from treatment or other change agents) and their associations with recidivism. A seminal study by Wormith (1984) articulated the processes of criminal attitudes as an underlying mechanism of criminal recidivism and for changes in criminal attitudes to be associated with decreased recidivism. The topic of applied criminal attitude research was virtually absent in the correctional literature at the time, which unfortunately persisted for some time. Several years ago, D. J. Simourd (1996) suggested the topic of criminal attitudes was the “silent partner” in crime, meaning it was of relevance in both academic and practical contexts yet mainly ignored. In the years to follow, only a small collection of studies has examined changes in criminal attitudes and associations with recidivism, particularly in sexual offending samples (e.g., Beggs & Grace, 2011; Olver, Nicholaichuk, & Wong, 2014; Woessner & Schwedler, 2014). These studies have examined various measures of criminal attitudes and have found varying associations with recidivism. To our knowledge, none have employed the CSS with this population.

Current Study and Rationale

Theory, research, and practice support the predictive accuracy and risk relevance of criminal attitudes for recidivism and their salience in the origin and maintenance of criminal behavior. This study sought to expand the existing knowledge of criminal attitudes (see Wormith, 1977, 1984; Wormith & Andrews, 1984) among specialized criminal justice client groups and address the need for further applied criminal attitudes research in a treated sexual offending sample. Informed by RNR principles, in a prospective examination of the psychometric properties of the CSS, we examine the role and relevance of criminal attitudes in a broadly high-risk sample of men who attended a high-intensity sexual violence treatment program. The findings of the study have potential implications for: (a) the role and relevance of general criminal attitudes for men who sexually offend, (b) the dynamic nature of risk, and (c) the potential utility of including a measure of general criminal attitudes to assess treatment progress vis-à-vis attitudinal change within a specialized program. The following hypotheses were proposed:

Hypothesis 1: CSS total and subscale scores would predict recidivism post-release, particularly more general outcomes (i.e., violent and general recidivism), illustrating that higher CSS scores correspond to increasing levels of risk and hence their risk relevance (risk principle).

Hypothesis 2: CSS total and subscale scores would show patterns of convergence with sexual violence measures of risk and need, demonstrating shared variance in the measurement of a common underlying risk construct (i.e., general recidivism risk), and thus, common criminogenic relevance (need principle).

Hypothesis 3: Changes in CSS total and subscale scores, denoting a reduction in criminal attitudes, would be associated with decreased recidivism, thus illustrating the risk relevance of changes in this need area (need principle).

Hypothesis 4: CSS total and subscale scores would be associated with variables that impact client response to service delivery (e.g., cognitive ability, education level). Specifically, such associations would reflect whether persons exhibiting high levels of criminal attitudes may also be more likely to have other vulnerabilities (e.g., lower cognitive ability) that may have implications for service delivery (e.g., adapting program content and style of delivery to the cognitive capabilities of clientele; responsivity principle).

Method

This study was conducted as part of an ongoing program of sexual violence risk assessment and change research that received ethical approval from the University of Saskatchewan’s Behavioural Research Ethics Board (#Beh-07-269) and operational approval from the Correctional Service of Canada (CSC).

Participants

The sample consisted of 363 males who were 35.1 years of age on average (SD = 10.1) and had 9.6 years education (SD = 2.7). Racial ethnicity and marital status information was available for approximately two thirds of the full sample. In all, 152 participants (58.2%) were Caucasian and 109 (41.8%) were of Indigenous ancestry. Furthermore, 121 participants (47.3%) were divorced/widowed, 96 (35.9%) were single, and 39 (15.2%) married or equivalent. Diagnostic information was available for approximately two thirds of cases, of whom 19.8% (47/237) had a major mental disorder (i.e., psychosis, mood disorder, anxiety disorder), 60.8% (144/237) any substance use disorder (SUD), 47.7% (114/239) antisocial personality disorder (ASPD), 67.8% (162/239) any personality disorder (PD), and 37.6% (89/237) were diagnosed with any paraphilia.

All participants had a current or prior conviction for a sexual offense and were serving a mean determinate sentence length of 5.8 years (SD = 3.2). Criminal history information was available for 93% of the sample (n = 338). Almost all participants (92% of the sample) had at least one prior conviction, with 45% having a prior conviction for a sex offense and 47.2%, a prior nonsexual violent conviction. Participants had an average of 11.7 (SD = 11.8) prior convictions. Nearly two thirds of men had at least one adult victim (62.9%, 161/256), while the remainder had exclusively child victims under the age of 14 years (37.1%, 95/256). We examined men with mixed and adult-only victim profiles as a composite group based on research (e.g., Olver & Wong, 2006) showing greater similarities in their psychological and criminogenic characteristics compared to men with only child victims.

Treatment Program

The Clearwater Program was a high-intensity sexual violence reduction program housed at a maximum security correctional mental health facility. The primary aim of the program was to offer treatment and stabilization services for high-risk, high-need Canadian federally sentenced men convicted of a sexual offense (most typically a contact offense). The Clearwater Program provided sexual offense–specific services of approximately 8 months duration and was cognitive behavioral in nature. Participants were referred because they were either deemed high risk for sexual violence or had personal or psychological characteristics that merited formal intervention in a coordinated program. The program included a combination of group and individual services targeting domains relevant to sexual violence risk including problems with sexual self-regulation and inappropriate interests, intimacy and relationship concerns, offense supportive attitudes, emotional regulation, healthy sexuality, and relapse prevention among other areas. Although the program began in 1983, content of the program has evolved with the rise of RNR and the developing knowledge base of “what works” in the assessment, intervention, and management of sexual offending (Olver & Wong, 2013). The program had consistently been staffed by a multidisciplinary treatment team that included psychiatric nurses, who would typically provide group and individual therapy, as well as occupational therapists, psychologists, social workers, psychiatrists, parole officers, and correctional officers. Indigenous Elders also played a prominent role in treatment and consultation, and cultural services (e.g., sweat lodges, smudging) due to the overrepresentation of Indigenous persons in Canadian corrections.

Measures

Cognitive Functioning

Three self-report administered paper-and-pencil measures of cognitive functioning were employed: (a) Symbol Digit Modality Test (SDMT; Smith, 1973), a timed measure of cognitive processing speed; (b) Raven’s Standard Progressive Matrices (Raven, 1938), a measure of nonverbal reasoning; and (c) Quick Test (Ammons & Ammons, 1962; see also Zagar et al., 2013), a brief measure of receptive vocabulary that provides an index of verbal intelligence. Collectively, these measures assessed cognitive domains of processing speed, nonverbal reasoning, and verbal ability.

CSS

The CSS (Gendreau et al., 1979) is a 41-item self-report measure of criminal attitudes. Items are endorsed on a 5-point Likert-type scale, ranging from “strongly disagree” to “strongly agree.” Item scores range from −2 to +2 and several items are reverse coded. The items are arranged into three subscales: (a) Law, Court, Police (LCP) comprising 25 items that reflect adversarial attitudes toward these three legal agents; (b) Tolerance toward Law Violations (TLV), comprising 10 items condoning criminal behaviors; and (c) Identification with Criminal Others (ICO), consisting of six items, with statements reflecting similarity or allegiance to individuals who break the law. The LCP scale is summed in the positive direction, such that higher scores represent more positive attitudes toward the law, courts, and police. By contrast, increasing scores on TLV and ICO represent more antisocial attitudes (i.e., favorable toward breaking the law and identifying with criminals). A CSS total score can be computed by summing the TLV and ICO subscales and subtracting this from the LCP subscale—higher total scores represent more prosocial attitudes, while lower scores represent more antisocial attitudes. Shields and Simourd (1991) developed the CSS-M, which employs a 3-point Likert-type scale, with no negative item weighting, and rewording the same 41 items so that a single summation of items can be used to generate a criminal attitudes score. Although the CSS-M is now more widely used than the CSS, the original CSS was used in this study because it was the version employed at the time of data collection.

Static-99R

Static-99R (Helmus et al., 2012) is an actuarial sexual offense risk tool comprising 10 static items (e.g., offense history, perpetrator, and victim demographics). Static-99R scores range from −3 to 12 with higher scores representing increased risk for sexual offending. Results of meta-analysis (Helmus et al., 2012) have demonstrated the Static-99R to be a strong predictor of sexual recidivism (5-year, area under the curve [AUC] = .72; 10-year, AUC = .71). In this study, the Static-99 was scored initially and later converted to Static-99R by recoding the revised age at release item. Thirty-five randomly selected double-coded cases generated acceptable interrater reliability (ICC_C,1 = .82) via single rater, consistency measure intraclass correlation (Olver et al., 2007).

VRS-SO

The VRS-SO (Wong et al., 2003–2017) is a clinician-rated sexual violence risk assessment and treatment planning tool. Comprising 7 static (historical) and 17 dynamic (changeable) items, each item is rated on a 4-point (0, 1, 2, 3) scale, with higher scores representing increased risk for sexual offending. Items with 2 or 3 ratings are considered criminogenic and targets for treatment or risk management. Factor analytic research supports an oblique three-factor model for the dynamic items: Sexual Deviance (e.g., deviant sexual preference, sexual compulsivity), Criminality (e.g., impulsivity, interpersonal aggression), and Treatment Responsivity (e.g., cognitive distortions, insight; Olver et al., 2007). Research from five international samples supports the predictive properties of the VRS-SO for sexual recidivism (Olver & Eher, 2020; Olver et al., 2018). Dynamic items are rated at multiple time points (e.g., pretreatment and posttreatment). A modified application of the Stages of Change Model (Prochaska et al., 1992) is used to track and measure changes on each dynamic item identified as criminogenic. Dynamic item change scores are then summed to generate a total change score representing the amount of risk reduction from treatment or other change agents. Thirty-five randomly selected double-coded VRS-SO protocols generate acceptable interrater reliability: Dynamic pretreatment, ICC_C,1 = .74; Dynamic posttreatment, ICC_C,1 = .79; Dynamic change, r = .68 (Olver et al., 2007).

Treatment Attrition

Treatment attrition was defined as noncompletion of the Clearwater program for any reason including administrative, client, or staff initiated (see Wormith & Olver, 2002). Treatment attrition was coded in a binary manner (1 = noncompletion, 0 = successful completion). This variable was coded through access to hardcopy treatment files; about two thirds of the sample (232/363 or 63.9%) had information available about whether the men had successfully completed the program.

Recidivism Variables

Recidivism data were obtained through a national criminal record database of official criminal charges and convictions, the Canadian Police Information Centre (CPIC). Recidivism was defined as any new criminal conviction incurred post-release and was coded in a binary manner (i.e., 1 = yes, 0 = no). The conviction date was also coded to permit survival analysis. Three outcome criteria were examined: sexual recidivism, violent recidivism, and general recidivism. Sexual recidivism included any new conviction for an offense that was sexually motivated, which included both contact (e.g., sexual assault, sexual interference) and noncontact (e.g., invitation to sexual touching, exposure) types. Offenses that were adjudicated by the court as a nonsexual crime (e.g., murder) but could be determined to be of a sexual nature (e.g., sexual homicide) were coded as sex recidivism. Violent recidivism was defined as any offense committed against the person (e.g., robbery, assault), including sex offenses. We employed a broad definition of violence to capture recidivistic behavior that involves any actual, potential, or threatened physical or psychological harm to a person, whether this be sexual or nonsexual in nature. Finally, general recidivism was defined as a new offense from any category. The general recidivism category is an omnibus category that is intended to capture any and all offending, be it violent or nonviolent, and is deliberately created to provide the broadest recidivism capture possible. Thus, the violent and general recidivism categories are not mutually exclusive as violent recidivism includes sexual reoffending and general recidivism includes recidivism of all types.

Procedure

The CSS was administered to all participants admitted to the facility shortly upon intake and then upon program completion as part of routine practice. All participants were voluntarily admitted to the facility and consented to participate in treatment in the Clearwater Program. Participants completed all self-report measures in group format within their treatment cohort. The present sample represents all CSS data that were available from these admission cohorts over years of the delivery of the program. A psychometrist typically administered the measures, supervised their completion, and was available to field any questions that may have arisen during test administration. Participants were not permitted to talk to each other during the testing. The psychometrist scored each of the measures, and these were cross checked and entered into a spreadsheet by the first author and trained student research assistants (RAs). Of note, both treatment completers and noncompleters completed the CSS at posttreatment upon discharge. The Static-99R and VRS-SO were rated from comprehensive institutional file information by trained RAs who had no ties to the CSS administration or access to CSS test scores. VRS-SOs were rated from pretreatment and posttreatment information on file; interrater reliability ratings of the VRS-SO and Static-99R were completed by the first author. All measures were rated blind to recidivism outcome. Recidivism and demographic data were collected by the first author.

Data Analytic Strategy

Analyses centered around examination of the psychological correlates and predictive properties of criminal attitudes as measured by the CSS as it pertains to the RNR model. This occurred in four phases. First, group comparisons were conducted of victim profile (i.e., adult victim vs. child victim) on the CSS subscale and total using one-way multivariate analysis of variance (MANOVA) with Tukey beta post hoc comparisons. This was done as a point of interest, given that research has demonstrated that men with adult victims tend to have more prominent general criminal attitudes than men with exclusively child victims (Mills et al., 2004). All remaining analyses, however, were conducted on the aggregate sample.

Second, correlates of CSS-measured criminal attitudes were examined via Pearson correlations with demographic and psychometric variables (per the responsivity principle) and the Static-99R and VRS-SO dynamic risk scores (per the need principle). If criminal attitudes as measured by the CSS are significantly correlated in the expected direction with risk measures, they would have shared risk variance, common criminogenic relevance, and therefore represent viable forensic treatment targets in line with the need principle. In addition, if CSS-measured criminal attitudes are correlated with cognitive ability, educational attainment, employment history, and treatment completion, they would represent a responsivity issue and thus signal the need to tailor programming to promote retention and maximize gain.

Third, the predictive accuracy of criminal attitudes as measured by CSS subscale and total scores were examined for the three recidivism outcomes, that is, to what extent do CSS scores discriminate recidivists from nonrecidivists. The AUC was used to examine predictive accuracy for recidivism and specifically in differentiating the higher risk cases from lower risk cases (per the risk principle). AUC values range from 0 to 1.0, representing the probability that a randomly selected recidivist will score higher on a given measure (e.g., CSS) than a randomly selected nonrecidivist. A general interpretative guideline is that values of .56, .64, and .71 represent small, medium, and large effect sizes, respectively (Rice & Harris, 2005). For prediction analyses, participants had variable follow-up times and thus Harrell’s C was computed using an SPSS macro, which computes a time-dependent AUC using the results from Cox regression survival analysis to control for individual differences in follow-up time.

Finally, pre-posttreatment changes on the CSS subscales and total scores, and the associations between positive changes in criminal attitudes and decreased recidivism across the three outcomes were examined. Participants with higher CSS scores have greater potential for larger pre–post change scores than participants with comparatively lower CSS scores (i.e., fewer criminal attitudes). To address this issue, residual change scores per Beggs and Grace (2011) were analyzed. In this procedure, change scores are regressed on the pretreatment score, and the residual (i.e., predicted CSS change score − actual change score) represents change in criminal attitudes unconstrained by the pretreatment score. We computed Cohen’s d to examine the magnitude of pre–post change, as well as AUCs and Harrell’s C for residualized change scores on recidivism to examine associations with outcome. (In this case, the outcome variable is reverse keyed for the TLV and ICO subscales so that positive AUC values represent the extent to which a randomly selected nonrecidivist made greater changes in criminal attitudes than a randomly selected recidivist.)

The analyses concluded with a series of Cox regressions to examine the association between changes in criminal attitudes (CSS score) with possible changes in recidivism across the three outcomes, after controlling for pretreatment CSS score, as well as individual differences on static and dynamic risk factors via the Static-99R and VRS-SO (pretreatment dynamic and change scores), respectively. The latter is a stringent and comprehensive test of the dynamic predictive validity of criminal attitudes as assessed by the CSS and the extent to which changes in these domains have risk relevance. As with all other analyses, we examine the predictors as continuous measures in the aggregate sample to preserve variance and maximize power.

Results

Sexual Offense Group Comparisons

Sexual offense group comparisons were conducted by subtype using one-way MANOVA with the magnitude of group differences quantified through standardized mean difference (Cohen’s d). As seen in Table 1, participants with adult victims had significantly worse LCP scores at posttreatment and significantly higher TLV, ICO, and CSS total scores (i.e., had greater criminal attitudes) measured at both pretreatment and posttreatment than participants with exclusively child victims. There were no group differences in terms of changes in criminal attitudes from pretreatment to posttreatment. In short, participants who committed contact sex offenses against adult victims tended to endorse more criminal attitudes than participants with child victims, including negative attitudes toward the law, court, and police, tolerance of law violations, and identifying with criminal associates.

Table 1:

Means and Standard Deviations of CSS Measures as a Function of Victim Profile

CSS measure	Adult victim		Child victim		Total sample		d
CSS measure	M	SD	M	SD	M	SD	d
LCP
Pre	90.9	15.0	93.9	15.3	91.4	15.0	–.20
Post	94.1	15.1	99.7	15.1	96.4	15.1	–.37**
Change	3.3	12.5	5.8	11.1	5.0	12.0	–.21
TLV
Pre	23.2	6.3	21.3	6.4	22.6	6.2	.30*
Post	21.4	6.2	19.5	6.8	20.4	6.0	.30*
Change	1.9	4.9	1.8	5.2	2.2	5.2	.02
ICO
Pre	16.1	3.2	14.5	3.3	15.6	3.2	.49***
Post	15.1	3.2	13.9	3.3	14.6	3.3	.37**
Change	1.0	3.6	0.6	3.4	0.9	3.4	.11
Total
Pre	51.6	21.9	58.0	22.7	53.2	21.9	–.28*
Post	57.7	22.5	66.2	22.3	61.3	22.1	–.38**
Change	6.1	17.6	8.2	16.4	8.1	17.4	–.12

Note. Adult victim group n = 161, child victim group n = 95, overall N = 363. CSS = Criminal Sentiments Scale; LCP = Law, Court, Police; TLV = Tolerance toward Law Violations; ICO = Identification of Criminal Others.

p < .05. **p < .01. ***p < .001.

Risk, Need, and Responsivity Correlates of CSS Scores

Table 2 reports bivariate associations between CSS scores with measures of risk and need. LCP, TLV, ICO, and total pretreatment and posttreatment scores had small to moderate correlations in the expected direction with commensurate VRS-SO dynamic, Criminality, and Treatment Responsivity scores. Change scores on the CSS components, however, were not significantly associated with changes on the VRS-SO dynamic score or its factor domains, which suggests limited overlap between risk change as assessed via self-reported criminal attitudes with that as assessed by the VRS-SO. Few associations between CSS scores and the Sexual Deviance domain attained significance, with the exception being positive LCP and CSS total scores (indicating fewer general criminal attitudes) associated with higher pretreatment scores on the factor. Static-99R scores had small to moderate associations in the expected direction with CSS subscale and total scores. In all, the pattern of bivariate associations demonstrated that general criminal attitudes measured by the CSS are associated with static and dynamic markers of sexual offense risk.

Table 2:

Risk and Need Correlates of CSS Scores

CSS measure	VRS-SO				Static-99R
CSS measure	Dynamic	Sexual deviance	Criminality	Treatment responsivity	Static-99R
LCP
Pre	–.27***	.12	–.39***	–.27***	–.21***
Post	–.19**	.19**	–.30***	–.22***	–.19**
Change	.10	.04	.06	.07	.01
TLV
Pre	.25***	–.12	.35***	.27***	.15*
Post	.27***	–.10	.35***	.30***	.15*
Change	.05	.00	.00	.08	.00
ICO
Pre	.32***	–.04	.41***	.24***	.22***
Post	.29***	–.04	.35***	.27***	.20**
Change	.04	–.02	.03	.07	.03
Total
Pre	–.30***	.12	–.42***	–.29***	–.21***
Post	–.24***	.16*	–.34***	–.26***	–.20**
Change	.09	.03	.05	.09	.01
VRS-SO and Static-99R
Pre M (SD)	28.2 (7.3)	8.1 (3.9)	9.7 (3.8)	6.8 (2.3)	4.7 (2.2)
Post M (SD)	24.5 (6.9)	7.1 (3.4)	8.5 (3.5)	5.6 (2.3)	—
Change M (SD)	3.7 (2.7)	1.0 (1.2)	1.2 (1.1)	1.2 (0.9)	—

Note. N = 233. CSS = Criminal Sentiments Scale; VRS-SO = Violence Risk Scale-Sexual Offense Version; LCP = Law, Court, Police; TLV = Tolerance toward Law Violations; ICO = Identification of Criminal Others.

p < .05. **p < .01. ***p < .001.

Table 3 reports the associations between responsivity variables with criminal attitudes as measured by CSS scores. Associations with cognitive functioning, demographic, and program variables are represented by correlation (r) while comparisons between diagnostic groups (i.e., diagnosis vs. no diagnosis) are represented by standardized mean difference (d). On the cognitive measures, participants scored at approximately average or slightly below average on each measure with considerable range within the sample: SDMT: M standard score = −0.63 (SD = 1.14; range −3.0 to 3.0); Ravens: M = 57.5th percentile (SD = 30.1; range 20th to 100th percentile); Quick Test: M = 38.8th percentile (SD = 27.6; range <1st to 99th percentile). The Clearwater Program also demonstrated a low rate of attrition with 12.1% (28/232) of men discharged prior to program completion, but conversely, 87.9% (204/232) of the men successfully completing the program.

Table 3:

Responsivity Correlates of CSS Scores

CSS measure	r						d
CSS measure	Raven’s	SDMT	Quick test	Education	Age	SOTP	MMD	Paraphilia	SUD	ASPD	PD
LCP
Pre	.33***	.18***	.40***	.28***	.23***	.16*	–.02	.02	.16	.33*	.28*
Post	.41***	.24***	.44***	.34***	.24***	.16*	.03	–.19	.18	.20	.24
Change	.09	.08	.06	.08	.02	.00	.07	–.27*	.03	–.16	–.04
TLV
Pre	–.32***	–.16**	–.35***	–.33***	–.26***	–.18*	–.01	.02	–.27*	–.21	–.24
Post	–.32***	–.23***	–.42***	–.35***	–.32***	–.24***	.14	.10	–.32*	–.22	–.26
Change	–.01	.07	.06	.02	.06	–.07	–.18	–.09	.03	.01	.01
ICO
Pre	–.34***	–.23***	–.35***	–.31***	–.21***	–.09	.15	.15	–.32*	–.29*	–.32*
Post	–.31***	–.19***	–.39***	–.35***	–.22***	–.23***	.24	.07	–.30*	–.31*	–.18
Change	–.01	.04	.04	.05	.01	.13*	–.07	.08	–.03	.00	–.14
Total
Pre	.37***	.20***	.42***	.33***	.26***	.17**	–.03	–.02	.23	.33*	.31*
Post	.41***	.26***	.47***	.38***	.28***	.20**	–.05	–.16	.25	.24	.26
Change	.06	.07	.06	.07	.03	.05	–.02	–.19	.03	–.11	–.05

Note. For the interpretation of correlations (r), negative values represent associations with increased criminal attitudes, while positive values represent associations with fewer criminal attitudes. For the interpretation of d values, negative (−) values denote higher scores (in SD units) by the diagnostic group, while positive values indicate higher scores (in SD units) by the comparison group without the diagnosis. ns as follows: Raven’s, SDMT, and education n = 359; Quick Test n = 340; age admission n = 286; SOTP n = 232; diagnostic variables ns = 237–239. CSS = Criminal Sentiments Scale; SDMT = symbol digit modality test; SOTP = sexual offense treatment program; MMD = major mental disorder; SUD = substance use disorder; ASPD = antisocial personality disorder; PD = personality disorder; LCP = Law, Court, Police; TLV = Tolerance toward Law Violations; ICO = Identification with Criminal Others.

p < .05. **p < .01. ***p < .001.

The pattern of responsivity associations demonstrated that criminal attitudes as measured by CSS subscale and total scores (both pre and post) had small to moderate magnitude correlations with each responsivity domain. Specifically, greater criminal attitude endorsement was associated with lower cognitive ability across all three domains (nonverbal reasoning, verbal ability, and processing speed), fewer years of education, younger age, and sexual offense treatment program (SOTP) noncompletion. The CSS change scores also tended to have weak and nonsignificant associations with each of these domains, with the one exception being lower levels of ICO change being significantly associated with higher rates of SOTP noncompletion.

For the diagnostic variables, SUD and ASPD diagnoses in particular were associated with increased endorsement of criminal attitudes on the CSS, particularly on the TLV and ICO subscales, while PD had some significant associations with pretreatment measures. Few meaningful associations were found between major mental disorder or paraphilia diagnoses with CSS scores. In all, endorsement of criminal attitudes was also associated with the presence of responsivity vulnerabilities that have implications for treatment and risk management.

Predictive Accuracy of CSS Scores for Recidivism

Participants were released between 1983 and 2009 (median 1998) and followed up an average 14.5 years (SD = 4.8) post release. Release outcome was available for 329 cases. A high proportion of participants incurred a recidivistic event during the follow-up period with 96 (29.2%) convicted of a new sexual offense, 143 (43.5%) any violent (including sexual) offense, and 186 (51.2%) any new offense. Predictive accuracy analyses of CSS total and subscale scores for sexual, violent, and general recidivism are reported in Table 4. Several trends are noteworthy. First, none of the pre- or post-CSS measures significantly predicted sexual recidivism, with AUC magnitudes ranging from small to little more than chance; univariate Cox regressions for pretreatment ICO and total scores attained significance. Second, pretreatment and posttreatment LCP, TLV, ICO, and total scores each significantly predicted violent and general recidivism with small in magnitude AUCs observed. Harrell’s C index values generated from Cox regression were of broadly the same magnitude as the AUCs. Thus, per the risk principle, more criminalized attitudes as measured by the CSS significantly discriminated violent and general recidivists from nonrecidivists.

Table 4:

Predictive Accuracy of CSS Measures for Recidivism

CSS measure	Sexual recidivism			Violent recidivism			General recidivism
CSS measure	AUC[95% CI]	C	e^B [95% CI]	AUC[95% CI]	C	e^B [95% CI]	AUC[95% CI]	C	e^B [95% CI]
LCP
Pre	.56 [0.49, 0.63]	.56	1.01 [1.00, 1.03]	.58* [0.51, 0.64]	.58	1.01*** [1.01, 1.03]	.61*** [0.55, 0.67]	.59	1.02*** [1.01, 1.03]
Post	.53 [0.46, 0.62]	.53	1.01 [0.99,1.02]	.59** [0.52, 0.65]	.59	1.02*** [1.01,1.03]	.62*** [0.56, 0.68]	.59	1.02*** [1.01,1.03]
Change	.50 [0.43, 0.58]	.50	1.00 [0.98, 1.02]	.55 [0.49, 0.61]	.56	0.98* [0.97, 0.999]	.56* [0.50, 0.63]	.55	0.98* [0.97, 0.997]
TLV
Pre	.52 [0.45, 0.59]	.53	1.02 [0.99, 1.05]	.56 [0.49, 0.62]	.57	1.04* [1.01, 1.06]	.62*** [0.56, 0.68]	.60	1.05*** [1.02, 1.08]
Post	.55 [0.49, 0.63]	.55	1.03 [1.00, 1.07]	.60** [0.54, 0.66]	.60	1.06*** [1.03, 1.09]	.65*** [0.59, 0.69]	.62	1.07*** [1.04, 1.09]
Change	.54 [0.47, 0.61]	.53	0.97 [0.92, 1.01]	.58** [0.51, 0.64]	.58	0.94*** [0.90, 0.97]	.59** [0.53, 0.65]	.57	0.94*** [0.91, 0.98]
ICO
Pre	.56 [0.49, 0.63]	.57	1.08* [1.01, 1.15]	.61*** [0.55, 0.67]	.60	1.11*** [1.05, 1.17]	.63*** [0.57, 0.69]	.59	1.11*** [1.06, 1.17]
Post	.55 [0.49, 0.62]	.55	1.06 [0.99, 1.12]	.60*** [0.54, 0.66]	.60	1.11*** [1.06, 1.17]	.63*** [0.56, 0.69]	.59	1.11*** [1.06, 1.16]
Change	.52 [0.45, 0.58]	.50	0.97 [0.91, 1.04]	.55 [0.49, 0.62]	.56	0.93** [0.88, 0.98]	.57* [0.50, 0.63]	.56	0.93** [0.89, 0.98]
CSS Total
Pre	.55 [0.49, 0.62]	.56	1.01* [1.00, 1.02]	.58*** [0.52, 0.65]	.59	1.01*** [1.01, 1.02]	.63*** [0.56, 0.69]	.60	1.02*** [1.01, 1.02]
Post	.54 [0.47, 0.61]	.54	1.01* [0.998, 1.02]	.60*** [0.54, 0.66]	.60	1.02*** [1.01, 1.02]	.64*** [0.58, 0.70]	.60	1.02*** [1.01, 1.02]
Change	.52 [0.45, 0.59]	.51	1.00 [0.99, 1.01]	.56* [0.50, 0.63]	.57	0.99** [0.98, 0.997]	.58* [0.52, 0.64]	.56	0.99** [0.98, 0.996]

Note. Change score associations with recidivism employ residualized change scores (i.e., controlling for pretreatment score). CSS = Criminal Sentiments Scale; AUC = area under the curve; CI = confidence interval; C = Harrell’s C index (time varying AUCs); e^B = hazard ratio; LCP = Law, Court, Police; TLV = Tolerance toward Law Violations; ICO = Identification with Criminal Others.

p < .05. **p < .01. ***p < .001.

Changes in Criminal Attitudes: Associations With Decreased Recidivism

Significant pre–post differences were found on each component of the CSS amounting to approximately one third of a standard deviation of change (all ps < .001): LCP d = .33, TLV d = .36, ICO d = .31, and CSS total d = .38. Table 4 also reports AUCs, univariate Cox regression hazard ratios, and Harrell’s C values for residualized change scores. The direction of AUCs for change scores were reversed so that they would be consistent with values from pretreatment and post-treatment scores; thus, positive AUCs and C values for change scores represent associations with decreased recidivism. CSS change scores were not meaningfully associated with decreased sexual recidivism; however, positive changes on each CSS component were significantly associated with general recidivism, with small in magnitude effects observed. Furthermore, changes in TLV and the total CSS score were significantly associated with decreased violent recidivism (AUC and C magnitudes indicating small effects), while the Cox regressions demonstrated significant small inverse associations between criminal attitude changes and decreased violent recidivism over time for all CSS scale components.

Finally, Cox regression survival analyses were conducted to examine the incremental predictive validity of criminal attitude change to possible reductions in recidivism controlling for baseline static and dynamic risk, via the Static-99R and VRS-SO pretreatment dynamic and change scores. As seen in Table 5, the results (Block 2, Models 1–4) largely mirrored the bivariate analyses. The three sets of risk measures consistently incrementally predicted sexual and violent recidivism, and Static-99R and VRS-SO change scores incrementally predicted general recidivism, after controlling for the CSS measures. None of the CSS change scores were significantly uniquely associated with reductions in sexual recidivism controlling for pretreatment scores and individual differences on static or dynamic risk factors. TLV and ICO change scores, however, were each significantly uniquely associated with decreased violent recidivism, after controlling for Static-99R, VRS-SO (pretreatment and change), and the respective CSS subscale’s pretreatment score. CSS pretreatment scores across the subscale domains and total score incrementally predicted general recidivism, although the CSS change scores did not.

Table 5:

Cox Regression Survival Analyses: Incremental Associations of Changes in CSS Measured Criminal Attitudes With Recidivism Controlling for Static and Dynamic Measures of Sexual Violence Risk

Cox regression Models 1–4	Sexual recidivism						Violent recidivism						General recidivism
Cox regression Models 1–4	B	SE	Wald	p	e ^B	95% CI	B	SE	Wald	p	e ^B	95% CI	B	SE	Wald	p	e ^B	95% CI
Block 1 (all analyses)
Static-99R	.174	.058	9.04	.003	1.19	[1.06, 1.33]	.149	.044	11.72	.001	1.16	[1.07, 1.26]	.136	.038	12.95	<.001	1.15	[1.06, 1.23]
Dynamic (pre)	.042	.018	5.70	.017	1.04	[1.01, 1.08]	.040	.014	8.27	.004	1.04	[1.01, 1.07]	.026	.013	4.37	.037	1.03	[1.002, 1.05]
Change	–.098	.042	5.37	.020	0.91	[0.83, 0.99]	–.139	.036	14.96	<.001	0.87	[0.81, 0.93]	–.093	.032	8.60	.003	0.91	[0.86, 0.97]
Block 2 (Model 1)
Static-99R	.176	.059	8.81	.003	1.19	[1.06, 1.34]	.127	.044	8.37	.004	1.14	[1.04, 1.24]	.109	.038	8.15	.004	1.12	[1.04, 1.20]
Dynamic (pre)	.039	.018	4.68	.030	1.04	[1.004, 1.08]	.038	.014	7.62	.006	1.04	[1.01, 1.07]	.021	.013	2.59	.108	1.02	[1.00, 1.05]
Change	–.103	.043	5.83	.016	0.90	[0.83, 0.98]	–.132	.036	13.59	<.001	0.88	[0.82, 0.94]	–.083	.032	6.67	.010	0.92	[0.86, 0.98]
LCP pre	–.003	.008	0.11	.739	1.00	[0.98, 1.01]	–.017	.007	6.86	.009	0.98	[0.97, 0.996]	–.019	.006	9.53	.002	0.98	[0.97, 0.99]
LCP change	.013	.010	1.64	.200	1.01	[0.99, 1.03]	–.010	.008	1.38	.239	0.99	[0.97, 1.01]	–.007	.008	0.82	.365	0.99	[0.98, 1.01]
Block 2 (Model 2)
Static-99R	.174	.058	9.04	.003	1.19	[1.06, 1.34]	.136	.0444	9.45	.002	1.15	[1.05, 1.25]	.117	.038	9.54	.002	1.13	[1.04, 1.21]
Dynamic (pre)	.043	.018	5.70	.019	1.04	[1.01, 1.08]	.034	.014	5.75	.017	1.03	[1.01, 1.06]	.014	.013	1.22	.270	1.01	[0.99, 1.04]
Change	–.099	.043	5.37	.021	0.91	[0.83, 0.99]	–.127	.036	12.37	<.001	0.88	[0.82, 0.95]	–.068	.032	4.49	.034	0.93	[0.88, 0.995]
TLV pre	–.002	.020	1.01	.903	1.00	[0.96, 1.04]	.044	.017	7.03	.008	1.05	[1.01, 1.08]	.060	.015	16.49	<.001	1.06	[1.03, 1.09]
TLV change	.001	.025	0.89	.964	1.00	[0.95, 1.05]	–.050	.022	5.33	.021	0.95	[0.91, 0.99]	–.036	.019	3.55	.060	0.97	[0.93, 1.00]
Block 2 (Model 3)
Static-99R	.174	.058	8.61	.003	1.19	[1.06, 1.34]	.125	.046	7.46	.006	1.13	[1.04, 1.24]	.121	.040	9.24	.002	1.13	[1.04, 1.22]
Dynamic (pre)	.041	.018	5.00	.025	1.04	[1.01, 1.08]	.030	.014	4.24	.040	1.03	[1.001, 1.06]	.011	.014	0.64	.424	1.01	[0.98, 1.04]
Change	–.100	.043	5.43	.020	0.91	[0.83, 0.98]	–.130	.036	13.16	<.001	0.88	[0.82, 0.94]	–.082	.032	6.47	.011	0.92	[0.87, 0.98]
ICO pre	–.011	.042	0.07	.790	1.01	[0.93, 1.10]	.120	.036	11.37	.001	1.13	[1.08, 1.24]	.122	.032	14.47	<.001	1.13	[1.06, 1.20]
ICO change	.024	.036	0.44	.505	1.03	[0.95, 1.10]	–.064	.032	4.02	.045	0.94	[0.88, 0.999]	–.044	.029	2.38	.123	0.96	[0.90, 1.01]
Block 2 (Model 4)
Static-99R	.177	.059	8.93	.003	1.19	[1.06, 1.34]	.126	.044	8.09	.004	1.13	[1.04, 1.24]	.108	.038	8.02	.005	1.11	[1.03, 1.20]
Dynamic (pre)	.040	.018	4.73	.030	1.04	[1.004, 1.08]	.035	.014	6.54	.011	1.04	[1.01, 1.07]	.016	.013	1.45	.228	1.02	[0.99, 1.04]
Change	–.102	.043	5.71	.017	0.90	[0.83, 0.98]	–.128	.036	12.74	<.001	0.88	[0.82, 0.94]	–.076	.032	5.52	.019	0.93	[0.87, 0.99]
CSS total pre	–.001	.005	0.07	.792	1.00	[0.99, 1.01]	–.013	.005	8.53	.003	0.99	[0.98, 1.00]	–.015	.004	13.89	<.001	0.99	[0.98, 0.99]
CSS total change	.008	.007	1.22	.270	1.01	[0.99, 1.02]	–.009	.006	2.34	.126	0.99	[0.98, 1.00]	–.005	.005	1.01	.315	1.00	[0.98, 1.01]

Note. N = 233. Significant p values in bold font. CSS = Criminal Sentiments Scale; CI = confidence interval; LCP = Law, Court, Police; TLV = Tolerance toward Law Violations; ICO = Identification with Criminal Others.

Discussion

This study sought to expand knowledge of the assessment and modification of general criminal attitudes among men who have sexually offended. This was done by way of examining a criminal attitude measure that has been the focus of previous seminal work (see Wormith, 1984) among nonspecialized populations. Specifically, the study examined the convergent validity and dynamic predictive properties of CSS scores in a large treated Canadian sample of men serving federal sentences for sexual offenses. This study contributes to what remains a limited knowledge base examining change on risk-relevant propensities and their associations with recidivism. It specifically offers some insights into service delivery for men convicted of sexual offenses.

Criminal Attitude Profiles and Prediction of Recidivism

Continued research in the area is worthwhile in light of extant findings demonstrating general criminal attitudes to be predictive of different recidivism outcomes, such as general, sexual, and violent recidivism (Olver et al., 2014), and hence, the prominence with which criminal attitudes are targeted in correctional programs (D. J. Simourd et al., 2016). The present study findings underscore the RNR relevance of the criminal attitudes construct, and the psychometric properties of the CSS for capturing this.

CSS total and subscale scores converged in conceptually meaningful ways with markers of criminogenic need, per the need principle, particularly more general domains such as the VRS-SO’s Criminality and Treatment Responsivity factors, consistent with past research showing convergence between the CSS measures with general static and dynamic risk measures, such as the LSI-R, General Statistical Information on Recidivism scale (G-SIR; Nuffield, 1982), and Factor 2 (i.e., chronic antisocial lifestyle features) of the Psychopathy Checklist–Revised (PCL-R; Hare, 1991, 2003; D. J. Simourd, 1997; Witte et al., 2006). The CSS did not have strong convergence with VRS-SO measured Sexual Deviance, and it had smaller although occasionally significant correlations with Static-99R scores—this likely reflects the generality of the CSS and such observed associations are likely attributable to the Static-99R’s general and nonsexual violence criminal history (see Mills & Kroner, 1997) and demographic item content.

CSS total and subscale scores had other important and meaningful patterns of association. Specifically, participants who endorsed a greater number of criminal attitudes also tended to be younger, less educated, have lower cognitive ability level across several domains of functioning, and were prone to discontinuing sexual offense–specific treatment. These associations have important relevance for the responsivity principle, that is, men with elevated CSS scores are not only more antisocial with a criminal attitude and value system, but they are also more likely to have other vulnerabilities that bode for difficulties in service delivery and engagement, unless important adaptations are made. To this end, changes on each of the CSS domains were not significantly associated with any of the responsivity markers (or need markers), indicating that participants documented pre–post changes in criminal attitude, via self-report, largely independent of their cognitive ability level. Arguably this could be a testament to the program itself in engaging lower cognitive functioning men and facilitating the development of treatment gains.

Furthermore, CSS total and subscale scores predicted recidivism outcomes consistent with findings on correctional samples elsewhere (D. J. Simourd & Olver, 2002; D. J. Simourd & Van de Ven, 1999; Witte et al., 2006; Wormith, 1984), and per the risk principle. Of note, this study features a considerably larger sample than previous evaluations as well as longer mean follow-up at more than 14 years. Perhaps the most direct comparison is Witte et al.’s (2006) evaluation of 72 participants (60 with post information) from the Clearwater Program followed up 54.5 months post-release. Although their sample overlaps with this study, the current investigation has a sample approximately 5 times larger, with longer follow-up, and different convergent measures and responsivity variables. In keeping with Witte et al. (2006), CSS total and subscale scores significantly predicted violent and general, but not sexual, recidivism. Mills and Kroner (1997), in a 16-month follow-up study, found that the CSS had weak predictive accuracy for general recidivism in their sexual and violent offending subsamples. Moreover, D. J. Simourd and van de Ven (1999), in their 389-day follow-up investigation, found the CSS-M total and TLV scores predicted rearrest and reincarceration in their general offending sample. Previous evaluations have not consistently found the ICO subscale, essentially the attitudinal equivalent of antisocial associates in the Psychology of Criminal Conduct model (Andrews & Bonta, 2010a), to be predictive of recidivism. An exception, and consistent with this study, Wormith (1984), in his classic evaluation of behavioral self-control versus recreational programming, found the ICO subscale to be predictive of 3-year general recidivism and for positive changes to be associated with decreased returns to custody.

Criminal Attitude Change and Associations With Recidivism

A unique feature of this study, in contrast to most previous CSS evaluations, was the examination of pre–post change from a well-established sexual offense–specific treatment program, and the association of changes in CSS-measured criminal attitudes to recidivism. The participants on average made approximately one third of a standard deviation of change on each of the CSS scales. Positive changes—in the direction of decreasing criminal attitudes—on each domain of the CSS were significantly associated with reductions in violent and general (but not sexual) recidivism, after controlling for pretreatment score. However, after imposing a stringent control for individual differences on static and dynamic risk factors (including changes in risk), positive changes on the TLV and ICO subscales remained significantly associated with decreased general violent recidivism. The findings are noteworthy on several grounds.

First, the findings demonstrate that changes on a self-report measure of criminal attitudes from an RNR-based SOTP have risk relevance. This has important implications for the use of structured criminal attitude measures, but also for the efficacy of a specialized risk reduction program to instill changes in a general risk-need domain, with positive implications for outcome. As Wormith (1984) aptly stated, such “results imply that complex offender change during incarceration may be more important to long-term success than the composite of offender attributes, upon either prison admission or discharge” (p. 607). The findings also indicate that CSS scores have unique risk-relevant variance, not already captured by a set of purpose-built sexual violence risk assessment tools. The TLV and ICO domains, respectively, represent attitudes favorable toward crime and antisocial peers; as such, changes in these domains had relevance for decreases in broader recidivism outcomes, but not specific outcomes such as sexual recidivism, which was best accounted for by the sexual violence risk measures. The results extend the findings of Wormith (1984) and D. J. Simourd et al. (2016), who found changes on the CSS-M, assessed pre-posttreatment from a criminal attitudes program in Alaska, to be associated with decreased general recidivism over a 140-day follow-up period.

Strengths, Limitations, and Future Directions

This study has strengths and limitations with implications for future research. One strength is the sample size and length of follow-up which provided adequate power (i.e., via increased recidivism base rates) to drive study analyses. Second, the study was a prospective examination of a measure of criminal attitudes and program outcomes, with the measure administered in real-life clinical practice and the participants followed up in real time. Given that most criminal attitude recidivism studies have examined criminal attitudes measured at only a single time point (and hence in a static manner), a third strength of the study was that the CSS was administered at multiple time points, enabling the examination of change. Fourth, the participants attended a well-established RNR-based sexual violence reduction program, and as such, the CSS changes measured pre-posttreatment arguably came from a credible change agent. Fifth, the administration of the criminal attitude measure was independent of the completion of other clinician-rated measures (e.g., VRS-SO, Static-99R); the risk measures were rated archivally blind to CSS ratings (as CSS protocols and test scores were not located on the men’s treatment files), increasing the rigor of the convergent validity analyses. Sixth, the risk measures were rated blind to recidivism outcome; although the risk measures were not rated for the purpose of the study, that they were completed blind to recidivism status avoids the problem of criterion contamination, which could otherwise inflate risk associations with outcome and artificially attenuate the associations between CSS scores and outcome when controlling for risk. Seventh, the inclusion of multiple measures facilitated rigorous examination of the change and predictive properties of the CSS. Finally, the criminal attitude dynamic prediction literature, particularly for sexual offending samples, is rather thin and this study contributes to filling this gap in the field.

There is clearly a need for further dynamic risk research. Although this study is an important step forward, there are limitations with implications for practice. There are few international replications of work of this nature or with specialized subpopulations (e.g., female correctional samples). This study featured a high risk, high need, treated sexual offense sample, and to increase generalizability it is recommended that future research replicate and extend this study to other samples, settings, and additional criminal attitude measures (e.g., CTP, MCAA, PICTS). Moreover, protective factors have been gaining increasing support in recent years, with research on measures such as the Structured Assessment of Protective Factors (SAPROF; de Vogel et al., 2009), demonstrating associations with decreased recidivism (de Vries Robbé et al., 2015) and improved positive community outcomes (Coupland & Olver, 2020). The SAPROF and its recently developed sexual offense–specific counterpart, the SAPROF-Sexual Offence version (SAPROF-SO; Willis et al., 2018), also contain item content pertaining to prosocial attitudes (e.g., attitude toward authority; attitude toward rules and regulations). Future research is recommended to examine the use of protective factors in forensic evaluations and their integration with other established structured forensic measures, including measures of criminal attitudes.

Furthermore, the present investigation was not a controlled treatment outcome study, with an independent control group; as such, it is possible that other factors aside from treatment contributed to the observed changes in the criminal attitude measure scores. However, it is reasonable to infer treatment to be one viable source of criminal attitude change, given that participants completed a risk reduction program that targeted criminogenic needs (including criminal attitudes). Future research is recommended to examine changes in other need domains, as well as to go beyond the pre–post design to include an untreated comparison control. Finally, while this study was a comprehensive examination of the psychometric properties of CSS scores, we did not have item-level data available that would have permitted a more nuanced examination of other key properties, such as internal consistency and factor structure.

Implications and Conclusions

In framing the context and rationale of our study, we noted that our findings have implications for: (a) the role and relevance of general criminal attitudes for men who sexually offend, (b) the dynamic nature of risk, and (c) the potential utility of including a measure of general criminal attitudes to assess treatment progress vis-à-vis attitudinal change within a specialized program. With respect to the first item, general criminal attitudes are endemic to correctional populations and theory and research demonstrate their relevance in the origin and continuation of crime; our findings reinforce this in a sample of men treated for sexual offending. While men who commit sexual offenses have attitudes and schema specific to sexual offending, general criminal attitudes may also be present. These have risk relevance and merit attention as part of sexual offense specific treatment or ancillary criminal attitudes programs (see examples listed previously). With respect to the second item, this study shows that criminal attitudes can and do change. Comprehensive sexual violence reduction programs address antisocial attitudes in many forms (e.g., eroticization of children, misogynistic attitudes, negative attitudes toward school and work, positive attitudes toward deviant sex); when attitudes change at a general and global level, this can bode favorably for broader release outcomes.

Finally, there may be a benefit to administering paper-and-pencil criminal attitude measures by offering a criminal attitude profile that may be informative for intervention planning (e.g., adjunct interventions to address general criminal attitudes). Information from these types of assessments may even augment formal appraisals of risk and need. Individuals with high baseline criminal attitudes, from a responsivity standpoint, are unlikely to be willing and motivated clients, and yet possess an entrenched value and attitude system that places them at risk for future antisocial behavior. One may not need a self-report criminal attitude measure to sense this (interacting with the client will likely make this apparent rather quickly), but progress on such measures and concordant changes behaviorally and attitudinally may be formally documented and informative. For instance, it may further elucidate the types of criminal attitudes held and the client’s willingness to disclose such attitudes in additional forums beyond a clinical interview. This multisource data may be particularly informative for men with adult or mixed sexual offense victim profiles who may be particularly inclined to harbor general criminal attitudes.

In closing, our findings support and extend the conclusions advanced by Wormith in his seminal 1984 work examining criminal attitude change as a function of behavioral self-control programming, where he presciently wrote,

Changes in attitude and personality played an important role in the offender’s success after release. Multiple regression analyses support the contention that recidivism is related to the amount of offender change during incarceration (Lukin, 1981). Indeed, it may be more predictive and more important to post-release success than either the offender’s characteristics upon prison admission or upon discharge. (p. 612)

A priori characteristics matter in terms of future prognosis and likelihood of return to prison, but as Wormith (1984) articulated, these are not a fait accompli; criminal attitudes can and do change. They are part of the framework for the origin and maintenance of criminal behavior; the results of meta-analysis substantiate this (Olver et al., 2014). But as dynamic entities acquired through the social learning process, what has been learned can be unlearned or perhaps more appropriately, relearned.

The field of correctional psychology has advanced considerably with the proliferation of criminal attitude intervention programs and assessment measures, in addition to the development of multimodal risk reduction programs and general and specialized risk-need measures that incorporate the criminal attitude construct. Attitude change alone while positive is insufficient, however, and unless accompanied by the development of prosocial behavioral skills, little can be expected in terms of criminal behavior reduction (Wormith, 1977). We anticipate the scientific and clinical advances made will stimulate continued research and practice toward the dynamic assessment, treatment, and management of criminal attitudes and behaviors to bolster prosocial functioning, aid reintegration, and promote safer and healthier communities, which includes the reduction of sexual violence.

Footnotes

Authors’ Note:

The views, opinions, and assumptions expressed in this paper are those of the authors and do not necessarily reflect the views or official positions of the University of Saskatchewan, the Saskatoon Police Service, or Correctional Service of Canada. Mark E. Olver is a coauthor of the Violence Risk Scale-Sexual Offense Version (VRS-SO) and receives remuneration from consultation and training services with the tool. We acknowledge our decades long friend and colleague, Dr. J. Stephen Wormith, for his mentorship, collaboration, and contributions toward the assessment and treatment of correctional clientele, including the measurement and modification of criminal attitudes. As Dr. Wormith reputably left no stone unturned in his approach to data analysis and scholarly inquiry, we strove to adopt a similar spirit in our interrogation of the substantive research questions explored in this article. We also thank Curtis Brad, Richard Coupland, and Tyson Kurtenbach for their assistance with data entry.

ORCID iD

Mark E. Olver

Mark E. Olver, PhD, is a full professor in the Department of Psychology, University of Saskatchewan, and a registered doctoral psychologist in the province of Saskatchewan. He conducts research and provides training on the assessment and treatment of high-risk violent and sexually violent person groups.

Keira C. Stockdale, PhD, is a registered doctoral psychologist in the province of Saskatchewan and an adjunct professor in the Department of Psychology at the University of Saskatchewan. In previous employment capacities, she has provided assessment and treatment services to justice involved adults and youth and is currently employed with the Saskatoon Police Service.

David J. Simourd, PhD, has been a licensed psychologist providing assessment and treatment services to offenders for more than 26 years. He has published articles, delivered training workshops, and served as a consultant on offender assessment and treatment to correctional agencies worldwide. He is a fellow of the Canadian Psychological Association. He can be reached at dave@acesink.com .

References

Abel

G. G.

Gore

D. K.

Holland

C. L.

Camp

Becker

J. V.

Rathner

(1989). The measurement of cognitive distortions of child molesters. Annals of Sex Research, 2, 135–153. https://doi.org/10.1177/107906328900200202

Ammons

R. B.

Ammons

C. H.

(1962). The Quick Test (QT): Provisional manual. Psychological Reports, 11, 111–162.

Andrews

D. A.

Bonta

(1994). The psychology of criminal conduct. Anderson.

Andrews

D. A.

Bonta

(1995). Level of Service Inventory-Revised (LSI-R): An offender assessment system. User’s guide. Multi-Health Systems.

Andrews

D. A.

Bonta

(2010a). The psychology of criminal conduct (5th ed.). LexisNexis.

Andrews

D. A.

Bonta

(2010b). Rehabilitating criminal justice policy and practice. Psychology, Public Policy, and Law, 16, 39–55. https://doi.org/10.1037/a0018362

Andrews

D. A.

Bonta

Wormith

J. S.

(2006). The recent past and near future of risk and/or needs assessment. Crime and Delinquency, 52, 7–27. https://doi.org/10.1177/0011128705281756

Bandura

(1986). Social foundations of thought and action: A social cognitive theory. Prentice Hall.

Beggs

S. M.

Grace

R. C.

(2011). Treatment gains for sexual offenders against children predicts reduced recidivism: A comparative validity study. Journal of Consulting and Clinical Psychology, 79, 182–192. https://doi.org/10.1037/a0022900

10.

Bonta

Andrews

D. A.

(2007). Risk-need-responsivity model for offender assessment and rehabilitation (User Report #2007-06). Public Safety Canada.

11.

Bumby

K. M.

(1996). Assessing the cognitive distortions of child molesters and rapists: Development and validation of the MOLEST and RAPE Scales. Sexual Abuse: A Journal of Research and Treatment, 8, 37–54. https://doi.org/10.1177/107906329600800105

12.

Coupland

R. B. A.

Olver

M. E.

(2020). Assessing protective factors in treated violent offenders: Associations with recidivism reduction and positive community outcomes. Psychological Assessment, 32, 493–508. https://doi.org/10.1037/pas0000807

13.

de Vogel

de Ruiter

Bouman

Y. H. A.

de Vries Robbé

. (2009). SAPROF: Guidelines for the assessment of protective factors for violence risk. Forum Educatief.

14.

de Vries Robbé

de Vogel

Douglas

K. S.

Nijman

H. L. I

. (2015). Changes in dynamic risk and protective factors for violence during inpatient forensic psychiatric treatment: Predicting reductions in postdischarge community recidivism. Law and Human Behavior, 39, 53–61. https://doi.org/10.1037/lhb0000089

15.

Gannon

T. A.

Olver

M. E.

Mallion

J. S.

James

(2019). Does specialized psychological treatment for offending reduce recidivism? A meta-analysis examining staff and program variables as predictors of treatment effectiveness. Clinical Psychology Review, 73, 1–18. https://doi.org/10.1016/j.cpr.2019.101752

16.

Gendreau

Grant

B. A.

Leipciger

Collins

(1979). Norms and recidivism rates for the MMPI and selected experimental scales on a Canadian delinquent sample. Canadian Journal of Behavioral Science, 11, 21–31. https://doi.org/10.1037/h0081569

17.

Gendreau

Little

Goggin

(1996). A meta-analysis of predictors of adult recidivism: What works! Criminology, 34, 401–433. https://doi.org/10.1111/j.1745-9125.1996.tb01220.x

18.

Hanson

R. K.

Bourgon

Helmus

Hodgson

(2009). The principles of effective correctional treatment also apply to sexual offenders: A meta-analysis. Criminal Justice and Behavior, 36, 865–891. https://doi.org/10.1177/0093854809338545

19.

Hanson

R. K.

Gizzarelli

Scott

(1994). The attitudes of incest offenders: Sexual entitlement and acceptance of sex with children. Criminal Justice and Behavior, 21, 187–202. https://doi.org/10.1177/0093854894021002001

20.

Hare

R. D.

(1991). Manual for the Revised Psychopathy Checklist. Multi-Health Systems.

21.

Hare

R. D.

(2003). The Hare Psychopathy Checklist-Revised (2nd ed.). Multi-Health Systems.

22.

Helmus

Hanson

R. K.

Babchishin

K. M.

Mann

R. E.

(2013). Attitudes supportive of sexual offending predict recidivism: A meta-analysis. Trauma, Violence, & Abuse, 14, 34–53. https://doi.org/10.1177/1524838012462244

23.

Helmus

Thornton

Hanson

R. K.

Babchishin

K. M.

(2012). Improving the predictive accuracy of Static-99 and Static-2002 with older sex offenders: Revised age weights. Sexual Abuse: A Journal of Research and Treatment, 24, 64–101. https://doi.org/10.1177/1079063211409951

24.

Hirschi

(1969). Causes of delinquency. University of California Press.

25.

McGrath

R. J.

Cumming

G. F.

Buchard

B. L.

Zeoli

Ellerby

(2010). Current practices and emerging trends in sexual abuser management: The Safer Society 2009 North American Survey. Safer Society Press.

26.

Mills

J. F.

Anderson

Kroner

D. G.

(2004). The antisocial attitudes and associates of sex offenders. Criminal Behaviour and Mental Health, 14, 134–135. https://doi.org/10.1002/cbm.578

27.

Mills

J. F.

Kroner

D. G.

(1997). The Criminal Sentiments Scale: Predictive Validity in a sample of violent and sex offenders. Journal of Clinical Psychology, 53, 399–404. https://doi.org/10.1002/(sici)1097-4679(199706)53:4<399::aid-jclp13>3.0.co;2-b

28.

Mills

J. F.

Kroner

D. G.

Forth

A. E.

(2002). Measures of Criminal Attitudes and Associates (MCAA): Development, factor structure, reliability, and validity. Assessment, 9, 240–253. https://doi.org/10.1177/1073191102009003003

29.

Mitchell

Tafrate

R. C.

(2012). Conceptualization and measurement of criminal thinking: Initial validation of the Criminogenic Thinking Profile. International Journal of Offender Therapy and Comparative Criminology, 56, 1080–1102. https://doi.org/10.1177/0306624X11416197

30.

Mitchell

Tafrate

R. C.

Hogan

Olver

M. E.

(2013). An exploration of the association between criminal thinking and community program attrition. Journal of Criminal Justice, 41, 81–89. https://doi.org/10.1016/j.jcrimjus.2012.09.003

31.

Murphy

W. D.

(1990). Assessment and modification of cognitive distortions in sex offenders. In Marshall

W. L.

Laws

D. R.

Barbaree

H. E.

(Eds.), Handbook of sexual assault: Issues, theories, and treatment of the offender (pp. 331–342). Plenum Press.

32.

Nuffield

(1982). Parole decision-making in Canada. Solicitor General Canada.

33.

Olver

M. E.

Eher

(2020). Predictive properties and factor structure of the VRS-SO: Results from an Austrian sample. European Journal of Psychological Assessment, 36(5), 748. https://doi.org/10.1027/1015-5759/a000551

34.

Olver

M. E.

Mundt

J. C.

Thornton

Beggs Christofferson

S. M.

Kingston

D. A.

Sowden

J. N.

. . . Wong

S. C. P.

(2018). Using the Violence Risk Scale-Sexual Offense Version in sexual violence risk assessments: Updated risk categories and recidivism estimates from a multisite sample of treated sexual offenders. Psychological Assessment, 30, 941–955. https://doi.org/10.1037/pas0000538

35.

Olver

M. E.

Nicholaichuk

T. P.

Wong

S. C. P.

(2014). The predictive and convergent validity of a psychometric battery used to assess sex offenders in a treatment program: An 18-year follow-up. Journal of Sexual Aggression, 20, 216–239. https://doi.org/10.1080/13552600.2013.816791

36.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2011). A meta-analysis of predictors of offender treatment attrition and its relationship to recidivism. Journal of Consulting and Clinical Psychology, 79, 6–21. https://doi.org/10.1037/a0022200

37.

Olver

M. E.

Stockdale

K. C.

Wormith

J. S.

(2014). Thirty years of research on the Level of Service Scales: A meta-analytic examination of predictive accuracy and sources of variability. Psychological Assessment, 26, 156–176. https://doi.org/10.1037/a0035080

38.

Olver

M. E.

Wong

S. C. P.

(2006). Psychopathy, sexual deviance, and recidivism among sex offenders. Sexual Abuse: A Journal of Research and Treatment, 18, 65–82. https://doi.org/10.1007/s11194-006-9006-3

39.

Olver

M. E.

Wong

S. C. P.

(2013). A description and research review of the Clearwater Sex Offender Treatment Program. Psychology, Crime & Law, 19, 477–492. https://doi.org/10.1080/1068316X.2013.758983

40.

Olver

M. E.

Wong

S. C. P.

Nicholaichuk

Gordon

(2007). The validity and reliability of the Violence Risk Scale-Sexual Offender version: Assessing sex offender risk and evaluating therapeutic change. Psychological Assessment, 19, 318–329. https://doi.org/10.1037/1040-3590.19.3.318

41.

Prochaska

J. O.

DiClemente

C. C.

Norcross

J. C.

(1992). In search of how people change: Applications to the addictive behaviors. American Psychologist, 47, 1102–1114. https://doi.org/10.1037/0003-066X.47.9.1102

42.

Raven

J. C.

(1938). Progressive matrices: A perceptual test of intelligence. H.K. Lewis.

43.

Rice

M. E.

Harris

G. T.

(2005). Comparing effect sizes in follow-up studies: ROC area, Cohen’s d, and r. Law and Human Behavior, 29, 615–620. https://doi.org/10.1007/s10979-005-6832-7

44.

Shields

I. W.

Simourd

D. J.

(1991). Predicting predatory behavior in a population of young offenders. Criminal Justice and Behavior, 18, 180–194. https://doi.org/10.1177/0093854891018002006

45.

Simourd

D. J.

(1996, June). Criminal attitudes: The silent partner in crime [Paper presentation]. Symposium on Violence and Aggression, University of Saskatchewan, Saskatoon, SK, Canada.

46.

Simourd

D. J.

(1997). The Criminal Sentiments Scale–Modified and Pride in Delinquency Scale: Psychometric properties and construct validity of two measures of criminal attitudes. Criminal Justice and Behavior, 24, 52–70. https://doi.org/10.1177/0093854897024001004

47.

Simourd

D. J.

Olver

M. E.

(2002). The future of criminal attitudes research and practice. Criminal Justice and Behavior, 29, 427–446. https://doi.org/10.1177/0093854802029004005

48.

Simourd

D. J.

Olver

M. E.

Brandenburg

(2016). Criminal attitudes among incarcerated offenders: Examination of a structured treatment program. International Journal of Offender Therapy and Comparative Criminology, 60, 1425–1445. https://doi.org/10.1177/0306624X15579257

49.

Simourd

D. J.

Van de Ven

(1999). Assessment of criminal attitudes: Criterion-related validity of the Criminal Sentiments Scale–Modified and Pride in Delinquency Scale. Criminal Justice and Behavior, 26, 90–106. https://doi.org/10.1177/0093854899026001005

50.

Simourd

Andrews

D. A.

(1994). Correlates of delinquency: A look at gender differences. Forum on Correctional Research, 6, 26–31.

51.

Smith

(1973). Symbol digit modalities test. Western Psychological Services.

52.

Sutherland

E. H.

(1947). Principles of criminology (6th ed.). Lippincott.

53.

Sykes

G. M.

Matza

(1957). Techniques of neutralization: A theory of delinquency. American Sociological Review, 22, 664–670. https://doi.org/10.2307/2089195

54.

Van de Ven

J. T. C.

(2004). Assessment of risk and need factors and service use in diverted youth [Unpublished Doctoral dissertation]. Carleton University.

55.

Walters

G. D.

(1995). The Psychological Inventory of Criminal Thinking Styles: I. Reliability and preliminary validity. Criminal Justice and Behavior, 22, 307–325. https://doi.org/10.1177/0093854895022003008

56.

Walters

G. D.

(2006). Risk appraisal versus self-report in the prediction of criminal justice outcomes: A meta-analysis. Criminal Justice and Behavior, 33, 279–304. https://doi.org/10.1177/0093854805284409

57.

Willis

Thornton

Kelley

de Vries Robbé

(2018). Structured Assessment of Protective Factor for violence risk—Sexual Offence version. Pilot Manual.

58.

Witte

T. D.

Di Placido

Wong

S. C. P.

(2006). An investigation of the validity and reliability of the Criminal Sentiments Scale in a sample of treated sex offenders. Sexual Abuse: A Journal of Research and Treatment, 18, 249–258. https://doi.org/10.1177/107906320601800303

59.

Woessner

Schwedler

(2014). Correctional treatment of sexual and violent offenders: Therapeutic change, prison climate, and recidivism. Criminal Justice and Behavior, 41, 862–879. https://doi.org/10.1177/0093854813520544

60.

Wong

Olver

M. E.

Nicholaichuk

T. P.

Gordon

(2003–2017). The Violence Risk Scale: Sexual Offense version (VRS-SO). Regional Psychiatric Centre; University of Saskatchewan.

61.

Wormith

J. S.

(1977). Converting prosocial attitude change to behavior change through self-management training [Unpublished Doctoral dissertation]. University of Ottawa.

62.

Wormith

J. S.

(1984). Attitude and behavior change of correctional clientele: A three year follow-up. Criminology, 22, 595–618. https://doi.org/10.1111/j.1745-9125.1984.tb00317.x

63.

Wormith

J. S.

Andrews

D. A.

(1984). Criminal sentiments and criminal behavior: A construct validation. Ministry of the Solicitor General of Canada.

64.

Wormith

J. S.

Olver

M. E.

(2002). Offender treatment attrition and its relationship with risk, responsivity, and recidivism. Criminal Justice and Behavior, 29, 447–471. https://doi.org/10.1177/0093854802029004006

65.

Wormith

J. S.

Olver

M. E.

Stevenson

H. E.

Girard

(2007). The long-term prediction of offender recidivism using diagnostic, personality, and risk/need approaches to offender assessment. Psychological Services, 4, 287–305. https://doi.org/10.1037/1541-1559.4.4.287

66.

Zagar

R. J.

Kovach

J. W.

Busch

K. G.

Zablocki

M. D.

Osnowitz

Neuthengen

. . . Zagar

A. K.

(2013). Quick test validity among randomly selected referrals. Psychological Reports, 113, 1–32. https://doi.org/10.2466/03.04.PR0.113x29z0