The Predictive and Incremental Validity of Two Self-Report Risk Assessment Measures With Adult Male Offenders Who Have Mental Health Problems

Abstract

We examined the predictive and incremental validity of two self-report risk assessment measures—the Self-Appraisal Questionnaire (SAQ) and the Measure of Criminal Attitudes and Associates (MCAA)—in a sample of 121 adult male offenders, with mental health problems in a correctional treatment setting. Both the SAQ and MCAA were significantly and positively correlated with a standard risk/need assessment currently used in corrections, the Level of Service Inventory–Ontario Revision (LSI-OR). All three risk measures significantly predicted general recidivism within 1 year of follow-up. The SAQ and LSI-OR also significantly predicted institutional incidents (threat, verbal aggression, or assault). In addition, the MCAA significantly added to the prediction of general recidivism provided by the LSI-OR, whereas the SAQ did not, likely reflecting the relatively high content overlap of the SAQ and LSI-OR. Neither self-report measure added to the ability of the LSI-OR to predict institutional incidents involving aggression.

Keywords

risk appraisal offenders self-report predictive validity mental health incremental validity

The deinstitutionalization movement of the 1950s and 1960s led to a dramatic increase in the number of mentally ill individuals seen in criminal justice settings. This, in part, was due to a lack of sufficient community mental health resources (Boschma, 2011; MacPhail & Verdun-Jones, 2013; Torrey, Kennard, Eslinger, Lamb, & Pavle, 2010). Consequently, there has been an internationally observed trend toward the criminalization of individuals with mental illness (Jansman-Hart, Seto, Crocker, Nicholls, & Cote, 2011).

Offenders with serious mental illness are twice as likely as non-disordered offenders to violate their probation or parole, are at an elevated risk of re-arrest and re-incarceration (Kesten et al., 2012; Prins & Draper, 2009), and are also more likely to experience unemployment and homelessness upon their release (Kesten et al., 2012; Wolff, Epperson, & Fay, 2010). Skeem, Winter, Kennealy, Louden, and Tatar (2013) discovered that mentally ill offenders had significantly more general risk factors for recidivism than their non-ill counterparts, including antisocial personality traits, and that this group had some unique risk factors such as their psychiatric symptoms. Similarly, the strongest predictors of a new violent offense among this subgroup of offenders include antisocial personality traits, juvenile delinquency, and criminal history, the same kinds of risk factors that are important predictors of violent recidivism among non-disordered offenders (Bonta, Blais, & Wilson, 2014; Harris, Rice, Quinsey, & Cormier, 2015). This suggests there are specific needs of mentally disordered offenders that have important implications for their rehabilitation and risk of reoffending. As such, this unique subgroup of offenders warrants special attention, especially in exploring the best ways to appraise their risk to reoffend and their criminogenic needs while they are in custody.

Currently, correctional services in Canada are delivered through an evidence-based approach to the assessment and treatment of offenders (Blanchette & Brown, 2006; Ward, Mesler, & Yates, 2007). The aim is to target changeable risk and protective factors to facilitate rehabilitation and decrease recidivism (Campbell, French, & Gendreau, 2007). As such, there are several formal methods of assessing the risk to reoffend based on this model. Given the high prevalence of offenders with mental health problems in the correctional system, a concern arises as to whether those methods of assessing risk are applicable to disordered offenders. The importance of appraising the likelihood of recidivism among all types of offenders is an integral part of delivering correctional services, and therefore it is imperative that this topic is explored.

Risk, Need, and Responsivity

There is ample evidence that the Risk–Need–Responsivity (RNR) model (Andrews, Bonta, & Hoge, 1990), which correctional services adhere to, can significantly reduce recidivism (see Andrews & Bonta, 2010). The RNR model is focused on the “Central Eight” criminogenic needs and based on three core principles. These central eight needs are a history of antisocial behavior, antisocial personality pattern, antisocial cognition, antisocial associates, family/marital circumstances, school/work functioning, leisure/recreational activities, and substance abuse. The three core principles are as follows: (a) The Risk Principle, which states that the level of intervention should be titrated to an individual’s level of risk to reoffend, meaning an individual with a higher level of risk should receive more intensive interventions, whereas someone with a low level of risk should receive minimal intervention. (b) The Need Principle refers to those criminogenic needs that are “problematic circumstances” troubling an individual (Andrews & Bonta, 2010). Finally, (c) the Responsivity Principle states that interventions are more effective when they are matched with the individual’s learning style. Services delivered to target criminogenic needs ought to be shaped to fit the individual instead of adopting a “one-size-fits-all” method.

Structured Risk Assessment

The Level of Service Inventory–Ontario Revision (LSI-OR; Andrews, Bonta, & Wormith, 1995) is a correctional assessment tool that addresses all three components of the RNR model. The LSI-OR is currently the standard risk measure used in Ontario’s provincial correctional system for offenders who are serving a sentence of less than 2 years (offenders serving sentences of 2 years or longer are held in federal correctional facilities in Canada). The LSI-OR significantly predicts both institutional misconducts and recidivism (Bonta & Motiuk, 1987; Campbell, French, & Gendreau, 2009; Girard & Wormith, 2004; Kroner & Mills, 2001).

The LSI-OR and similar structured risk measures are resource intensive, typically requiring a structured interview and file review by a trained professional. The LSI-OR takes about 30 to 45 min to complete, and it is intended to be administered at intake and re-administered every 6 months to monitor an offender’s progress. This means the LSI-OR is not always an option, for example, when screening a large number of offenders or when there is substantial information missing in the files.

Prediction of Institutional Misconducts

Not only is the ability to accurately assess the risk to recidivate vital but also the ability to predict institutional misconducts. By appraising the probability of an offender’s likelihood of committing an institutional infraction, decisions regarding offender placement and level of security can be better informed. Effective placement decisions can promote staff and offender safety and allow for the more efficient administration of correctional services (Kroner & Mills, 2001).

The LSI family of risk assessment tools is highly relevant to institutional placement decisions because it can predict institutional misconducts, with correlations ranging from .26 to .39 (Bonta & Motiuk, 1987). In a study conducted by Kroner and Mills (2001) comparing the accuracy of five risk appraisal instruments in predicting institutional misconduct as well as new convictions, it was found that the LSI had among the strongest correlations for institutional misconduct. In their review, Campbell et al. (2009) found that the LSI-OR was correlated (r = .24, k = 5, N = 650) with violent institutional misconduct.

Self-Reported Risk Assessment

Walters (2006) noted that there is skepticism about the validity or utility of self-reported risk relevant measures in criminal justice decision making, given individuals might lie and minimize their risk for self-serving reasons. Other concerns include questionable content validity of self-report measures and the cognitive capacity of respondents to answer the questions (Edens, Hart, Johnson, Johnson, & Olver, 2000). To test this idea, Walters (2006) conducted a meta-analysis of 22 studies reporting 27 effect sizes for examining self-report measures and risk measures completed by evaluators such as the Historical, Clinical, Risk Management-20 (HCR-20) and the Psychopathy Checklist–Revised (PCL-R). Results indicated no significant difference between the self-report and evaluator-completed measures in predicting institutional misconduct or recidivism. Of note, the self-report measures included two measures of criminal attitudes (the Criminal Sentiments Scale and the Psychological Inventory of Criminal Thinking Styles) and the Self-Appraisal Questionnaire (SAQ), described next. These measures fared better at predicting outcomes than more general measures of personality and psychopathology such as the Minnesota Multiphasic Personality Inventory. There was also some evidence to suggest that evaluator-completed and self-report measures could add to each other in the prediction of outcomes.

SAQ

The SAQ (Loza, 2005) is a self-report measure designed to predict violent and non-violent recidivism. More than half of the items of the SAQ assess dynamic factors that are helpful in measuring treatment gains (Loza & Loza-Fanous, 2000). The SAQ has been found to be useful in classifying offenders to different levels of security, assigning offenders to appropriate programs, and predicting offenders’ institutional adjustment (Loza & Loza-Fanous, 2003). This measure takes about 15 to 20 min to complete, and it can be administered to groups. The SAQ does not require an interview to be completed and it does not rely on the use of file information.

The Measure of Criminal Attitudes and Associates (MCAA)

The MCAA (Mills & Kroner, 1999) is also a self-report tool that assesses antisocial attitudes and quantifies antisocial associates. It requires about 20 min to complete and does not rely on file information as well. Antisocial attitudes and associates predict both institutional misconducts and future offending (Gendreau, Goggin, & Law, 1997; Gendreau, Little, & Goggin, 1996). Both antisocial attitudes and associates are criminogenic needs that can be targeted in treatment (Mills, Kroner, & Forth, 2002). The MCAA may also be sensitive enough to measure therapeutic changes in attitudes after participation in a treatment program (Bäckström & Björklund, 2008).

The Present Study

There is little empirical evidence regarding the predictive utility of the SAQ and MCAA with offenders detained in a correctional treatment center. Villeneuve, Oliver, and Loza (2003) cross-validated the SAQ in a sample of 107 offenders held in a maximum security psychiatric setting, but we are unaware of any study examining the MCAA. We predicted that the SAQ and MCAA would predict general recidivism in a correctional sample of offenders with serious mental health problems. Our study sample differed from that examined by Villeneuve et al. (2003) because it involved provincially sentenced offenders, that is, individuals who had been sentenced to less than 2 years of incarceration.

Another novel feature of this study is that we could also examine the ability of these measures to predict institutional misconducts. This information would be of great value when determining offender placement and treatment, as previously mentioned. We hypothesized that all three measures would significantly predict institutional misconduct, consistent with previous work (e.g., Campbell et al., 2009; Gendreau et al., 1997; Walters, 2006).

In addition, we wanted to examine whether self-report measures, such as the SAQ and MCAA, had incremental validity in the prediction of recidivism by an established risk measure (the LSI-OR), in line with results reported by Walters (2006). Examining the impact of combining multiple risk measures for sex offenders, Seto (2005) suggested that combining measures would not increase predictive validity unless the measures covered different content or used different methods to assess risk. Self-report measures might contain additional risk-related information, particularly regarding what the individual thinks and intends, which may not be available to evaluators and may be missing from the offender’s file. They also involve a different form of data collection, which can enhance the quality of risk information obtained. We hypothesized that the SAQ or MCAA would add to the prediction of recidivism offered by the LSI-OR, currently the standard risk assessment tool used in the correctional institution where we conducted this study.

Method

Participants

A sample of provincially sentenced offenders was recruited to take part in this study. All the offenders were adult males (n = 121), had been diagnosed with or were suspected of having a mental illness, and were in custody at the St. Lawrence Valley Correctional and Treatment Centre, also referred to as the Secure Treatment Unit (STU), in Brockville, Ontario. The mean age of the sample was 34.5 years (SD = 11.80, range = 18-67). The racial composition of the sample was Caucasian (83.5%, n = 101), Aboriginal (9.9%, n = 12), African Canadian (4.1%, n = 5), and Asian or other origins (2.5%, n = 3). The education levels were less than high school (62.8%, n = 76), high school completed (19.0%, n = 23), college/university completed (8.3%, n = 10), some college/university completed (7.4%, n = 9), some graduate studies (0.8%, n = 1), and graduate studies completed (1.7%, n = 2).

The primary psychiatric diagnosis was substance- and alcohol-related disorders (70.2%, n = 85), mood disorder (59.5%, n = 72), personality disorder (51.2%, n = 62), other disorders (50.4%, n = 61), anxiety disorder (47.9%, n = 58), paraphilias (16.5%, n = 20), developmental disorder (14.9%, n = 18), psychotic/schizophrenic spectrum disorder (12.4%, n = 15), and organic brain disorder (3.3%, n = 4). The “other disorders” category included disorders such as attention deficit hyperactivity disorder, sleep disorders, specific learning disorders, intermittent explosive disorder, and amnesic disorders. Ninety-four percent (n = 114) of participants had two or more diagnoses, 4% (n = 5) had one diagnosis, and 2% (n = 2) had no diagnosis at the time of the assessment but were admitted to the STU for further investigation of a suspected mental illness. The sample was representative of the institutional population.

The mean number of prior charges and hospitalizations for these participants was 31 (SD = 27.72) and 1.6 (SD = 3.48), respectively. In terms of index offenses, the most prevalent was a failure of conditional release (52.9%, n = 64), followed by property offenses (46.3%, n = 56), violent offenses (34.7%, n = 42), miscellaneous offenses (33.1%, n = 40), and drug offenses (10.7%, n = 13). Average time from sentencing to discharge date for participants was 280 days, and the average length of stay in the STU post-assessment was 83 days.

Measures

LSI-OR

The LSI-OR is divided into eight scales. Section A, which can also be referred to as the general risk/need section, contains the following subscales: Criminal History (8 items), Education/Employment (9 items), Family/Marital Relationships (4 items), Leisure/Recreation (2 items), Companions (4 items), Substance Abuse (8 items), Pro-Criminal Attitudes/Orientation (4 items), and Antisocial Pattern (4 items). These items require a rating of 0 to 3, where 0 is given when there is a very unsatisfactory situation with a very clear and strong need for improvement, and 3 is rated when the situation is satisfactory with no need for improvement. Section B is focused on specific risk/need items, and was added because they were thought to be criminogenic and useful if over-riding the actuarial estimate of risk was warranted (Wormith, 1997). This section contains two subscales: Personal Problems With Criminogenic Potential (14 items) and History of Perpetration (8 items). LSI-OR scores can be used to assign offenders to one of five risk categories, each associated with a probability of recidivism within 1 year. Higher scores reflect a greater risk of recidivism and a greater need for clinical intervention.

The LSI-OR has repeatedly demonstrated its ability to predict general and violent recidivism. Girard and Wormith (2004) discovered that inmates who scored higher on all scales were more likely to reoffend during a 31-month follow-up; the general risk/need score correlated highly with general recidivism and, to a lesser extent, with violent recidivism. Predictive validity was also found across different types of offenders, including sexual offenders, domestic violence offenders, and offenders with mental health problems. In a recent study by Canales, Campbell, Wei, and Totten (2014), the Level of Service (LS) family of risk tools was also valid for community-supervised mentally disordered offenders. Other studies have also demonstrated the reliability and validity of the LSI-OR (Andrews, Kiessling, Mickus, & Robinson, 1986; Bonta & Motiuk, 1987; Loza & Simourd, 1994; Stevenson & Wormith, 1987). The LSI-OR is an earlier version of the LS/Case Management Inventory (LS/CMI; Andrews, Bonta, & Wormith, 2004). Olver, Stockdale, and Wormith (2014) conducted a comprehensive meta-analysis of the family of LSI tools and concluded that they significantly predicted general and violent recidivism. In addition, Skeem et al. (2013) found that mental illness did not moderate the effect of the general risk factors; that is, the LSI worked with mentally ill offenders just as well as it did for the offenders without mental illness.

SAQ

The SAQ contains 72 true-or-false items. It has the eight following subscales: Criminal Tendencies (27 items), Antisocial Personality Problems (5 items), Conduct Problems (18 items), Criminal History (6 items), Alcohol/Drug Abuse (8 items), Antisocial Associates (3 items), Anger (5 items), and a Validity scale (8 items). The first six subscales (67 items) contribute to the measure’s ability to predict recidivism. Scores on these 67 items can be used to assign offenders to one of four risk categories: low (scores of 0-10), low-moderate (11-22), high-moderate (23-42), and high (43-67). The Anger scale is used to identify individuals for whom anger is a treatment need, and the Validity scale provides an assessment of the offender’s truthfulness in reporting.

The SAQ has demonstrated good psychometric properties and has been validated across cultures (Loza, Conley, & Warren, 2004; Summers & Loza, 2004) and for both male and female offenders (Loza, Neo, Shahinfar, & Loza-Fanous, 2005). Offenders with higher SAQ scores committed significantly more offenses than did those with lower SAQ scores, and offenders with a history of violence had higher SAQ total scores than offenders with no history of violence (Loza & Loza-Fanous, 2000). Loza and Loza-Fanous (2000) also demonstrated that the SAQ’s subscales were correlated with other valid measures of assessing risk. These correlations ranged from .28 to .65, illustrating concurrent validity. Concurrent validity was also assessed by examining the correlations between SAQ scores and other measures for assessing violent and general risk (i.e., General Statistical Information of Recidivism [GSIR], LSI-OR, PCL-R, and Violence Risk Appraisal Guide [VRAG]). All four measures shared strong correlations with the SAQ total and subscale scores. Criterion-related validity was also assessed using the total number of past offenses. Participants were divided into two equal groups. These groups included low (less than 9 offenses) and high (9 to 70 offenses). Groups were compared on the SAQ total and subscale scores. Correlations between the total number of prior offenses and the SAQ total and subscale scores were calculated. All comparisons and correlations were statistically significant.

The predictive validity of the SAQ has been shown through follow-up studies over 2-, 5- and 9-year intervals (Loza & Loza-Fanous, 2000, 2003; Loza, MacTavish, & Loza-Fanous, 2007). These studies have not been conducted by independent investigators, however. Other studies showed that the SAQ was as effective at predicting post-release outcomes as the VRAG or the PCL-R (Kroner & Loza, 2001; Loza & Green, 2003; Loza & Loza-Fanous, 2001).

In a sample of federally sentenced Canadian male offenders (serving a sentence 2 years or longer), the total scale test–retest reliability was .95, and ranged from .69 to .93 for the six subscales. Total scale to subscale correlations ranged from .52 to .87 and subscale to subscale correlations ranged from .25 to .58. Coefficient alphas for all subscales ranged from .42 to .87 (Loza, Dhaliwal, Kroner, & Loza-Fanous, 2000).

MCAA

The MCAA is comprised of two parts: Part A is a measure of criminal associates where participants can identify up to four friends they spend the most time with. They then indicate how much of their free time is spent with each associate (0%-25%, 25%-50%, 50%-75%, and 75%-100%). The participant also describes their associates’ involvement in crime by responding to four questions with “yes” or “no.” Part A is used to calculate two measures of criminal associations, number of criminal friends and criminal friend index. Part B is composed of 46 agree/disagree items that measure attitudes in four domains: Violence (12 items), Entitlement (12 items), Antisocial Intent (12 items), and Associates (10 items). Factor analysis supported the existence of these four attitude domains (Mills et al., 2002).

The MCAA has demonstrated predictive validity for both general and violent recidivism in adult male offenders (Mills, Kroner, & Hemmati, 2004). These studies have also not been conducted by independent investigators. With the exception of the Attitudes Toward Violence scale, which did not predict general recidivism, the MCAA scales were found to be significantly related to both violent and general recidivism; the Associates scale was most strongly correlated to general recidivism, while Antisocial Intent scale was most strongly related to violent reoffending (Mills, Anderson, & Kroner, 2004). The MCAA may have predictive validity among sex offenders (Mills, Anderson, et al., 2004), and in a Canadian sample of federally incarcerated offenders, the MCAA was strongly correlated with two other measures of antisocial attitudes, the Criminal Sentiments Scale and the Pride in Delinquency Scale (rs = .40 and .76). The MCAA is a valid measure of antisocial attitudes and associates (Bäckström & Björklund, 2008; Mills et al., 2002).

Regarding test–retest reliability of the MCAA, interclass correlations exceeded .72 for most scales: specifically, MCAA Total = .81, Violence = .73, Entitlement = .74, Antisocial Intent = .79, and Associates = .65 (Mills et al., 2002). Overall, the MCAA demonstrated acceptable reliability and validity in a sample of federally incarcerated men.

Outcomes

Institutional outcomes, such as incidents of physical aggression, verbal aggression, self-harm, rule-breaking (e.g., non-compliance with institutional rules, substance use), and use of seclusion and/or restraints, were recorded from an electronic incident reporting database up to 1 year after the risk assessment information was obtained. Only one research assistant was responsible for collecting this information from the database. Nursing and allied health staff are mandated to record any of the above-mentioned incidents into this electronic database. Upon orientation to the institution, all staff are trained on how to complete the incident form and detailed instructions on how to complete the form are also kept next to all computers in the nursing units. Recidivism outcomes, defined as any new charges within the first year following release into the community, were obtained from an electronic database—the Offender Tracking Information System—maintained by provincial correctional authorities. All outcome data were gathered after completion of the risk assessment measures.

Procedure

Residents at the STU were randomly approached by one of three research assistants who were trained in all aspects of the study protocol and asked whether they would be willing to participate in the study. If they agreed, the research assistant described the study in more detail and reviewed the informed consent process. Correctional policy prohibits compensation of individual residents; instead, $11 was donated to an inmate recreation fund for each participant. If informed consent was obtained in writing, the research assistant asked the individual to complete the SAQ and MCAA. Completion of the battery took approximately 1 hr, and longer if items had to be presented orally by the research assistant because the offender had reading difficulties. The research assistants then scored the measures according to each respective scoring guide. All LSI-OR scores were obtained from the Ministry of Community Safety and Correctional Services as they are responsible for completing this measure. All LSI-OR assessments were completed shortly following an offender’s sentencing, by classification officers prior to participants being admitted to the STU.

We also obtained consent to review institutional files to code sociodemographic variables (age, gender, ethnicity, civil status, residential status, education and employment status, and primary source of revenue), clinical variables (previous hospitalizations, age of first psychiatric hospitalization, admission psychiatric diagnoses, medication use, and treatment involvement), and criminal history variables (previous charges, age at first criminal charge, index offense leading to current sentence, and age at index offense). Outcome data were collected only after file coding was completed.

Results

Most types of institutional outcomes were rare. However, 41% of the entire sample had one or more aggressive incidents during their admission to the STU (lower-level aggression, for example, pushing, shoving, and threats were combined in the incidents database); this outcome was missing for one offender. A little over a third of the collective sample (35.5%, n = 43) were charged and deemed guilty of committing a new offense within 1 year of their release.

Table 1 illustrates the demographic characteristics for both non-recidivists and recidivists. An independent samples t test was conducted to examine whether any of the differences were statistically significant. There were statistically significant differences for age at time of assessment, t(113.85) = 3.86, p < .001; age of first psychiatric consultation, t(106) = 3.39, p = .001; age of first psychiatric hospitalization, t(108) = 3.41, p = .001; age at time of first offense, t(107.46) = 4.32, p < .001; and age at time of index offense, t(115.62) = 3.24, p = .002, with recidivists being younger. Table 2 displays the frequency of specific psychiatric diagnosis among recidivists and non-recidivists. The non-recidivist sample tended to have higher frequencies of mental illness compared with recidivists, but the two groups did not significantly differ in specific diagnoses.

Table 1:

Demographic Characteristics and t-Test Results for STU Non-Recidivist and Recidivist Sample

	Non-Recidivist (n = 78)	Recidivist (n = 43)	t test	df	p
Demographic	M (SD)	M (SD)	t test	df	p
Age at time of assessment	37.14 (12.69)	29.67 (8.55)	3.86	113.85	<.001
Age at first psychiatric consultation	25.91 (13.92)	16.82 (10.48)	3.39	106.00	.001
Age at first psychiatric hospitalization	33.35 (12.94)	25.06 (9.67)	3.41	108.00	.001
Total number of charges at time of assessment	23.79 (32.19)	31.00 (27.72)	−1.24	119.00	.219
Number of previous non-violent charges	21.35 (31.43)	27.40 (26.66)	−1.07	119.00	.288
Number of previous violent charges	2.81 (3.14)	3.60 (3.26)	−1.30	119.00	.191
Age at time of first offense	24.31 (11.33)	18.05 (4.31)	4.32	107.46	<.001
Age at time of index offense	35.24 (13.04)	28.88 (8.48)	3.24	115.62	.002

Note. STU = Secure Treatment Unit.

Table 2:

Disorders and Chi-Square Results for STU Recidivist and Non-Recidivist Sample

Disorder	Non-Recidivist (%)	Recidivist (%)	χ²	p
Psychotic/schizophrenia spectrum	10.3	16.3	0.454	.500
Mood	61.5	55.8	0.177	.679
Anxiety	51.3	41.9	0.645	.422
Organic	5.1	0.0	0.958	.328
Substance/alcohol related	66.7	76.7	0.908	.341
Personality	50.0	53.5	0.031	.859
Developmental	10.3	23.3	2.744	.098
Paraphilias	20.5	9.3	1.778	.182
Other	48.7	53.5	0.098	.755

Note. Recidivist sample (n = 43), non-recidivist sample (n = 78). STU = Secure Treatment Unit.

Associations of Risk Measures

The correlations between the SAQ, MCAA, and LSI-OR total scores demonstrated that all risk measures were significantly and positively correlated with each other. The SAQ was strongly correlated with both the MCAA total score, r = .79, p < .01, and LSI-OR general risk score, r = .73, p < .01, whereas the MCAA total score was less strongly correlated with the LSI-OR general risk score, r = .57, p < .01.

Tables 3 and 4 report correlations between SAQ and MCAA subscales and LSI-OR subscales. There was some evidence of domain specificity; for example, the MCAA subscale pertaining to associates had the strongest correlation with the LSI-OR subscale regarding companions, r = .69, p < .01. Similarly, the SAQ subscale regarding alcohol and drug use problems had its strongest correlation with the LSI-OR substance abuse subscale, r = .74, p < .01.

Table 3:

Correlations Between LSI-OR and SAQ Subscale and Total Scores

	SAQ Subscales and Total Score
LSI-OR Subscales	Criminal Tendencies	Antisocial Personality Problems	Conduct Problems	Criminal History	Alcohol and Drug Problems	Antisocial Associates	Anger	Total Score
Criminal History	.41**	.58**	.47**	.68**	.43**	.44**	.26**	.58**
Education/Employment	.39**	.42**	.42**	.42**	.32**	.42**	.34**	.48**
Family/Marital	.32**	.33**	.33**	.35**	.15	.41**	.22*	.37**
Leisure/Recreation	.35**	.33**	.32**	.39**	.30**	.33**	.18	.40**
Companions	.52**	.60**	.49**	.62**	.60**	.56**	.26**	.66**
Pro-Criminal Attitudes/Orientation	.31**	.32**	.26**	.40**	.11	.26**	.16	.33**
Substance Abuse	.49**	.53**	.44**	.49**	.74**	.38**	.20*	.62**
Antisocial Pattern	.39**	.46**	.43**	.47**	.29**	.42**	.23*	.49**
General risk score	.58**	.66**	.58**	.69**	.58**	.59**	.34**	.73**

Note. LSI-OR = Level of Service Inventory–Ontario Revision; SAQ = Self-Appraisal Questionnaire.

p < .05. **p < .01.

Table 4:

Correlations Between LSI-OR and MCAA Subscale and Total Scores

	MCAA Subscales and Total Score
LSI-OR Subscales	Attitudes Toward Violence	Attitudes Toward Entitlement	Antisocial Intent	Attitudes Toward Associates	Total Score
Criminal History	.25**	.20*	.42**	.47**	.42**
Education/Employment	.31**	.18	.44**	.43**	.43**
Family/Marital	.20**	.25**	.31**	.33**	.33**
Leisure/Recreation	.24*	.16	.34**	.36**	.34**
Companions	.33**	.22*	.53**	.69**	.55**
Pro-Criminal Attitudes/Orientation	.19	.17	.22*	.25**	.26**
Substance Abuse	.34**	.19	.38**	.40**	.41**
Antisocial Pattern	.30**	.18	.44**	.49**	.44**
General risk score	.40**	.27**	.56**	.60**	.57**

Note. LSI-OR = Level of Service Inventory–Ontario Revision; MCAA = Measure of Criminal Attitudes and Associates.

p < .05. **p < .01.

SAQ Subscales

There was also evidence of divergent validity for the SAQ, because SAQ subscales were not significantly correlated with subscales on the LSI-OR that assessed different domains. For example, the SAQ Alcohol and Drug Problems subscale was not significantly correlated with the LSI-OR Family/Marital (r = .15, p = .12) or Pro-Criminal Attitudes subscales (r = .11, p = .16). The SAQ Anger subscale was weakly correlated with most of the LSI-OR subscales and did not show significant correlations with the LSI-OR Leisure/Recreation subscale (r = .18, p = .06) or the Pro-Criminal Attitudes subscale (r = .16, p = .09).

MCAA Subscales

There was evidence for divergent validity for MCAA subscales as well. The Attitudes Toward Entitlement subscale of the MCAA showed no significant correlation with five of the LSI-OR subscales, specifically the Education/Employment (r = .18, p = .06), Leisure/Recreation (r = .16, p = .10), Pro-Criminal Attitudes (r = .17, p = .08), Substance Abuse (r = .19, p = .05), and Antisocial Pattern subscales (r = .18, p = .06). Entitlement, according to Mills and Kroner (2001), revolves around the idea that an individual believes he or she has the right to take whatever they want. This concept is not explicitly included in the LSI-OR.

Surprisingly, the MCAA Attitudes Toward Violence subscale also did not show any correlation with the LSI-OR Pro-Criminal Attitudes subscale, r = .19, p = .05. This violence subscale is indicative of an endorsement of attitudes that are supportive of violence (Mills & Kroner, 2001), whereas the Pro-Criminal Attitudes subscale, captured within the LSI-OR, is defined more broadly as attitudes that support criminal activity and are unfavorable of convention. Individuals who score high on this LSI-OR subscale often think their sentence is unfair and have a poor attitude toward supervision (Andrews & Bonta, 1995).

Prediction of Outcomes

We conducted receiver operating characteristic (ROC) analyses to examine the predictive validity of the two self-report risk measures and the LSI-OR. As expected, the LSI-OR significantly predicted whether an offender had an incident involving aggression while in the institution (area under the curve [AUC] = .62, 95% CI = [.52, .73]), and significantly predicted general recidivism within 1 year of release (AUC = .72, 95% CI = [.62, .81]). The SAQ also significantly predicted both institutional aggression (AUC = .61, 95% CI = [.51, .72]) and general recidivism (AUC = .74, 95% CI = [.65, .83]). The MCAA did not significantly predict institutional aggression (AUC = .59, 95% CI = [.48, .70]) but did predict general recidivism (AUC = .73, 95% CI = [.64, .82]).

Logistic Regression

Tables 5 and 6 report the maximum likelihood logistic regression results for predicting institutional aggression or predicting a new offense of any kind, respectively. There was no evidence of multicollinearity in either regression model based on variance inflation factor or condition index. In both regression models, we entered LSI-OR score as the first block because it is currently the standard risk measure used in provincial corrections in Ontario, and we were interested in seeing whether the two self-report risk measures could add to its predictive utility. We then entered the SAQ and MCAA total scores in a second block, using a conditional forward criterion for selection. Only the LSI-OR risk score entered as a significant predictor of institutional aggression, with a weak model overall (McFadden’s R² = .038, last row, Table 5).

Table 5:

Logistic Regression Analysis for Institutional Aggression (n = 110)

									95% CI
Predictor	β	SE β	Wald’s χ2	df	p	AIC	McFadden’s R²	Exp (β)	LL	UL
LSI-OR only	.047	.021	4.85	1	.028	149.12	.034	1.048	1.005	1.092
Constant	−1.486	.584	6.47	1	.011			0.226

LSI-OR and	.041	.025	2.61	1	.107	151.04	.036	1.042	0.991	1.094
MCAA	.010	.022	0.19	1	.661			1.010	0.967	1.055
Constant	−1.539	.598	6.61	1	.010			0.215

LSI-OR and	.028	.029	.94	1	.333	152.92	.034	1.029	0.971	1.090
SAQ	.015	.020	.54	1	.463			1.015	0.976	1.055
Constant	−1.443	.582	6.14	1	.013			0.236

LSI-OR and	.031	.030	1.105	1	.293	152.82	.038	1.032	0.973	1.093
MCAA and	−.003	.030	0.009	1	.925			0.997	0.939	1.058
SAQ	.017	.027	0.373	1	.542			1.017	0.964	1.072
Constant	−1.539	.600	6.573	1	.010			0.215

Note. AIC = Akaike Information Criterion; CI = confidence interval; LL = lower limit, UL = upper limit; LSI-OR = Level of Service Inventory–Ontario Revision; MCAA = Measure of Criminal Attitudes and Associates; SAQ = Self-Appraisal Questionnaire.

Table 6:

Logistic Regression Analysis for New Offense of Any Kind (n = 110)

							McFadden’s R²		95% CI
Predictor	β	SE β	Wald’s χ²	df	p	AIC	McFadden’s R²	Exp (β)	LL	UL
LSI-OR only	.092	.025	14.01	1	.001	137.29	.113	1.097	1.045	1.151
Constant	−2.90	.701	17.093	1	.001			0.055

LSI-OR and	.063	.029	4.85	1	.028	129.22	.159	1.065	1.007	1.127
MCAA	.059	.025	5.69	1	.017			1.060	1.011	1.113
Constant	−3.437	.792	18.84	1	.001			0.032

LSI-OR and	.054	.033	2.76	1	.097	131.39	.150	1.056	0.990	1.125
SAQ	.044	.022	4.13	1	.042			1.045	1.002	1.090
Constant	−3.326	.775	18.40	1	.001			0.036

LSI-OR and	.055	.033	2.72	1	.099	131.10	.161	1.056	0.990	1.127
MCAA and	.047	.033	2.03	1	.155			1.048	0.982	1.118
SAQ	.015	.029	0.28	1	.597			1.016	0.959	1.076
Constant	−3.447	.796	18.76	1	.001			0.032

Note. AIC = Akaike Information Criterion; CI = confidence interval; LL = lower limit, UL = upper limit; LSI-OR = Level of Service Inventory–Ontario Revision; MCAA = Measure of Criminal Attitudes and Associates; SAQ = Self-Appraisal Questionnaire.

As expected, the LSI-OR was also a significant predictor of any new offense; MCAA score, but not SAQ score, significantly added to the LSI-OR (Table 6). This model predicted 25% of the variance in new offending (McFadden’s R² = .161, last row, Table 6). As shown in Table 6, comparing the Akaike Information Criterion (AIC) values, adding the SAQ on its own did not perform as well as adding the MCAA to the LSI-OR, and adding both the SAQ and the MCAA had relatively little impact on variance in new recidivism that was explained by adding the MCAA alone. When either the SAQ or MCAA was entered first, the LSI-OR did not add to its predictive validity (specific results not reported).

Discussion

We again demonstrated that the LSI-OR is useful in a correctional setting, including for offenders with mental health problems (Girard & Wormith, 2004; Kroner & Mills, 2001). In this study, LSI-OR general risk scores were significant predictors of both institutional aggression (threat, verbal aggression, or assault) and of general recidivism within 1 year of follow-up. These results are consistent with those obtained from the study conducted by Canales et al. (2014). We also extended previous research with the SAQ and MCAA, showing both have predictive validity for recidivism (e.g., Loza & Loza-Fanous, 2000; Mills, Kroner, et al., 2004). As expected, both self-report measures were significantly and positively correlated with each other and with the LSI-OR. There was also evidence of subscale specificity, because the self-report subscales had differential patterns of correlations with LSI-OR subscales, reflecting different underlying factors. The SAQ had a stronger correspondence with the LSI-OR domains than the MCAA, which was also reflected in the higher correlation between total SAQ and LSI-OR scores.

A novel aspect of this study was comparing the incremental validity of two self-report risk measures in predicting institutional misconduct and recidivism. The SAQ was a significant predictor of both institutional misconduct and general recidivism, whereas the MCAA was only a significant predictor of general recidivism. The MCAA, but not the SAQ, added to the predictive utility of the LSI-OR in predicting general recidivism, but not institutional misconduct. The SAQ did not significantly add to the LSI-OR. This result may reflect the closer correspondence between SAQ item and domain content and LSI-OR item and domain content. Because of this overlap, the SAQ did not provide a unique contribution in the logistic regression, whereas the MCAA tapped other risk-related information about attitudes and associates.

At the same time, this high overlap suggests the SAQ could be a better self-report alternative to the LSI-OR than the MCAA for research purposes or when the LSI-OR is not feasible to administer (e.g., screening large groups, insufficient file information). We noted that the LSI-OR did not add to the predictive contribution of the SAQ when their order was reversed in a logistic regression analysis. As noted earlier in the introduction, self-report measures have several advantages, including taking less time to administer, the option of group administration, not requiring a trained professional to complete ratings, and not requiring extensive file review. Potential disadvantages include the possible manipulation of responses if self-reported risk is used to make important decisions and skepticism about offender self-report in forensic practice.

Study Limitations

There were several limitations to this study. First, the sample size was small, thus effect inflation could be possible. Effect inflation is more likely to happen in low-powered studies with smaller sample sizes which can only detect large effects (Button et al., 2013). Our study was sufficiently powered to detect moderate effects.

Second, the participants completed the self-report measures under the assurance that research data would be kept confidential and would not influence treatment or other decisions made about them. An important question before implementing self-reported risk measures in practice is to see whether self-reports under more typical risk assessment conditions are still predictive of recidivism. This limitation does not affect the utility of self-report risk measures for research purposes, where investigators may want to know the risk of recidivism in their sample or variations in risk factors and can assure confidentiality. In fact, Loza, Cumbleton, et al. (2004) conducted a cross-validation study utilizing the SAQ on samples from Australia, Canada, England, Singapore, and two samples from the United States. Their Australian and Canadian samples were aware that their scores on the SAQ would be used in the process of evaluating their requests for early release. Results suggested that the SAQ has sound psychometric properties, with acceptable reliability and concurrent validity on each of the aforementioned samples.

Another limitation is that our analysis of institutional incidents did not control for time at risk. Offenders differed at the time of the assessment in terms of how long they had been in the institution, and we would expect fewer incidents over time as offenders become adjusted to a new setting. We would note that the average length of stay is typically between 3 and 4 months, so variation in time at risk for institutional misconduct is not typically large. This limitation did not apply to our analysis of the recidivism outcome, where all offenders had 1 year of opportunity post-release.

Finally, completion of the SAQ and MCAA could have occurred at any time of their sentence; participants may have been initiating treatment, in the midst of treatment or completing treatment, which therefore could affect their scores, given the dynamic nature of the SAQ and at least the attitudes section of the MCAA while in custody.

Future Directions

A number of avenues can be pursued in further research. Future work could examine the impact that these risk measures have on the treatment planning of offenders. Because all measures are intended to assess risk to reoffend, and correctional interventions are most effective when titrated to level of risk and matched to needs, the results derived from scoring such instruments should be related to subsequent treatment plans. The stronger these associations, the greater the potential impact on recidivism (Andrews & Bonta, 2010). It would be interesting to administer the SAQ and MCAA pre- and post-treatment, to examine the relationship between any observed changes and recidivism. The given correctional programs target many of the domains described by these two self-report measures, we would expect treatment-related changes to be predictive of outcomes.

It would also be useful to look at the association and possible contribution of self-report risk measures, such as the SAQ and MCAA, in other risk assessment contexts, including young offenders, sex offenders, and female offenders, both with mental health issues or not. Many of the risk factors for recidivism—such as substance use, criminal history, and antisocial personality—are similar across these groups (Hanson & Morton-Bourgon, 2005; Penner, Viljoen, Douglas, & Roesch, 2014; Putkonen, Komulainen, Virkkunen, Eronen, & Lonnqvist, 2003). We could expect the SAQ and MCAA to demonstrate similar results regarding predictive validity for these samples as well (e.g., Loza & Loza-Fanous, 2000; Mills, Kroner, et al., 2004).

Footnotes

Acknowledgements

Acknowledgments go to Ms. Susan Curry for her help with data analysis. Wagdy Loza is the developer of the Self-Appraisal Questionnaire (SAQ) and has a financial interest in this risk assessment measure. He receives royalties from sales of the SAQ from its publisher, Multi-Health Systems.

This work was supported by the University Medical Research Fund (Grant G6302292) and the AAPL Institute for Education and Research.

This is a potential conflict of interest as favorable results from this study could financially benefit Wagdy Loza in the future through increased SAQ sales.

Nicole C. Rodrigues, MEd, CCC, is a research coordinator at the Brockville Mental Health Centre. She is also a Canadian Certified Counselor with a focus on forensic psychiatric populations.

Michael C. Seto, PhD, is a forensic research director with the Royal Ottawa Health Care Group. He is cross-appointed to the University of Toronto, Ryerson University, Carleton University, and the University of Ottawa.

Adekunle G. Ahmed is the associate chief for the Royal Ottawa Health Care Group (Forensic) and the director of the Anger Disorders Clinic. He is an associate professor at the University of Ottawa and an adjunct professor at Queen’s University.

Wagdy Loza is an adjunct assistant professor (psychiatry, Queen’s University) and ex. adjunct professor (psychology, Carleton University). He is former chief psychologist of Kingston Penitentiary and founder of the extremism/terrorism section of the Canadian Psychological Association (CPA).

References

Andrews

D. A.

Bonta

(1995). The Level of Service Inventory–Revised. Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Bonta

(2010). The psychology of criminal conduct (5th ed.). New Providence, NJ: LexisNexis/Matthew Bender.

Andrews

D. A.

Bonta

Hoge

R. D.

(1990). Classification for effective rehabilitation: Rediscovering psychology. Criminal Justice and Behavior, 17, 19-52. doi:10.1177/0093854890017001004

Andrews

D. A.

Bonta

Wormith

(1995). Level of Service Inventory–Ontario Revision (LSI-OR): Interview and scoring guide. North Bay, Ontario, Canada: Ministry of Correctional Services.

Andrews

D. A.

Bonta

Wormith

(2004). The Level of Service/Case Management Inventory (LS/CMI). Toronto, Ontario, Canada: Multi-Health Systems.

Andrews

D. A.

Kiessling

J. J.

Mickus

Robinson

(1986, June). Some convergent and divergent validities of the LSI. Paper presented to the annual meeting of the Canadian Psychological Association, Winnipeg, Manitoba, Canada.

Bäckström

Björklund

(2008). The Measures of Criminal Attitudes and Associates (MCAA): Further testing of structural and criterion-related validity. Criminal Justice and Behavior, 35, 1389-1410. doi:10.1177/0093854808322239

Blanchette

Brown

S. L.

(2006). The assessment and treatment of women offenders: An integrative perspective. Chichester, UK: John Wiley.

Bonta

Blais

Wilson

H. A.

(2014). A theoretically informed meta-analysis of the risk for general and violent recidivism for mentally disordered offenders. Aggression and Violent Behavior, 19, 278-287. doi:10.1016/j.avb.2014.04.014

10.

Bonta

Motiuk

L. L.

(1987). The diversion of incarcerated offenders to correctional halfway houses. Journal of Research in Crime & Delinquency, 24, 302-323. doi:10.1177/0022427887024004006

11.

Boschma

(2011). Deinstitutionalization reconsidered: Geographic and demographic changes in mental health care in British Columbia and Alberta. Histoire Sociale/Social History, 44(88), 223-256. doi:10.1353/his.2011.0020

12.

Button

K. S.

Ioannidis

J. P. A.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S. J.

Munafo

M. R.

(2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews, 14, 365-376. doi:10.1038/nrn3475

13.

Campbell

M. A.

French

Gendreau

(2007). Assessing the utility of risk assessment tools and personality measures in the prediction of violent recidivism for adult offenders (User Report 2007-04). Ottawa, Ontario: Public Safety Canada.

14.

Campbell

M. A.

French

Gendreau

(2009). The prediction of violence in adult offenders: A meta-analytic comparison of instruments and methods of assessment. Criminal Justice and Behavior, 36, 567-590. doi:10.1177/0093854809333610

15.

Canales

Campbell

M. A.

Wei

Totten

(2014). Prediction of general and violent recidivism among mentally disordered adult offenders: Test of the Level of Service/Risk-Need-Responsivity (LS/RNR) instrument. Criminal Justice and Behavior, 41, 971-992. doi:10.1177/0093854814523003

16.

Edens

J. F.

Hart

S. D.

Johnson

D. W.

Johnson

J. K.

Olver

M. E.

(2000). Use of the Personality Assessment Inventory to assess psychopathy in offender populations. Psychological Assessment, 12(2), 132-139. doi:10.1037/1040-3590.12.2.132

17.

Gendreau

Goggin

C. E.

Law

M. A.

(1997). Predicting prison misconducts. Criminal Justice and Behavior, 24, 414-431. doi:10.1177/0093854897024004002

18.

Gendreau

Little

Goggin

C. E.

(1996). A meta-analysis of the predictors of adult offender recidivism: What work! Criminology, 34, 575-607. doi:10.1111/j.1745-9125.1996.tb01220.x

19.

Girard

Wormith

(2004). The predictive validity of the Level of Service Inventory–Ontario Revision on general and violent recidivism among various groups. Criminal Justice and Behavior, 31, 150-181. doi:10.1177/0093854803261335

20.

Hanson

R. K.

Morton-Bourgon

K. E.

(2005). The characteristics of persistent sexual offenders: A meta-analysis of recidivism studies. Journal of Consulting and Clinical Psychology, 73, 1154-1163. doi:10.1037/0022-006X.73.6.1154

21.

Harris

G. T.

Rice

M. E.

Quinsey

V. L.

Cormier

C. A.

(2015). Violent offenders: Appraising and managing risk (3rd ed.). Washington, DC: American Psychological Association.

22.

Jansman-Hart

E. M.

Seto

M. C.

Crocker

A. G.

Nicholls

T. L.

Cote

(2011). International trends in demand for forensic mental health services. International Journal of Forensic Mental Health, 10, 326-336. doi:10.1080/14999013.2011.62559

23.

Kesten

K. L.

Leavitt-Smith

Rau

D. R.

Shelton

Zhang

Wagner

Trestman

R. L.

(2012). Recidivism rates among mentally ill inmates: Impact of the Connecticut offender reentry program. Journal of Correctional Health Care, 18, 20-28. doi:10.1177/1078345811421117

24.

Kroner

D. G.

Mills

J. F.

(2001). The accuracy of five risk appraisal instruments in predicting institutional misconduct and new convictions. Criminal Justice and Behavior, 28, 471-489. doi:10.1177/009385480102800405

25.

Kroner

D. G.

Loza

(2001). Evidence for the efficacy of self-report in predicting violent and non-violent criminal recidivism. Journal of Interpersonal Violence, 16, 168-177. doi:10.1177/088626001016002005

26.

Loza

(2005). Self Appraisal Questionnaire (SAQ): A tool for assessing violent and non-violent recidivism. Toronto, Ontario, Canada: Multi-Health Systems.

27.

Loza

Conley

Warren

(2004). Concurrent cross-validation of the Self-Appraisal Questionnaire: A tool for assessing violent and nonviolent recidivism and institutional adjustment on a sample of North Carolina offenders. International Journal of Offender Therapy and Comparative Criminology, 48, 85-95. doi:10.1177/0306624X03257714

28.

Loza

Cumbleton

Shahinfar

Neo

Evens

Conley

Summers

(2004). Cross-Validation of the Self-Appraisal Questionnaire (SAQ): An offender risk and need assessment measure on Australian, British, Canadian, Singaporean, and American offenders. Journal of Interpersonal Aggression, 19, 1172-1190. doi:10.1177/0886260504269180

29.

Loza

Dhaliwal

Kroner

D. G.

Loza-Fanous

(2000). Reliability, construct, and concurrent validities of the Self-Appraisal Questionnaire: A tool for assessing violent and non-violent recidivism. Criminal Justice and Behavior, 27, 356-374. doi:10.1177/0093854800027003005

30.

Loza

Green

(2003). The Self-Appraisal Questionnaire: A self-report measure for predicting recidivism versus clinician administered measures. A five-year follow-up study. Journal of Interpersonal Violence, 18, 781-797. doi:10.1177/0886260503253240

31.

Loza

Loza-Fanous

(2000). Predictive validity of the Self-Appraisal Questionnaire (SAQ): A tool for assessing violent and non-violent release failures. Journal of Interpersonal Violence, 15, 1183-1191. doi:10.1177/088626000015011005

32.

Loza

Loza-Fanous

(2001). The effectiveness of the Self-Appraisal Questionnaire (SAQ) in predicting offenders’ post-release outcome: A comparison study. Criminal Justice and Behavior, 28, 105-121. doi:10.1177/009385401028001005

33.

Loza

Loza-Fanous

(2003). More evidence for the validity of the Self-Appraisal Questionnaire (SAQ) for predicting violent and non-violent recidivism: A five year follow-up study. Criminal Justice and Behavior, 30, 709-721. doi:10.1177/0093854803256448

34.

Loza

MacTavish

Loza-Fanous

(2007). A nine-year follow-up study on the predictive validity of the Self-Appraisal Questionnaire for predicting violent and nonviolent recidivism. Journal of Interpersonal Violence, 22, 1144-1155. doi:10.1177/0886260507303730

35.

Loza

Neo

L. H.

Shahinfar

Loza-Fanous

(2005). Cross-validation of the Self- Appraisal Questionnaire: A tool for assessing violent and nonviolent recidivism with female offenders. International Journal of Offender Therapy and Comparative Criminology, 49, 547-560. doi:10.1177/0306624X04273433

36.

Loza

Simourd

D. J.

(1994). Psychometric evaluation of the Level of Service Inventory (LSI) among male Canadian federal offenders. Criminal Justice and Behavior, 21, 468-480. doi:10.1177/0093854894021004006

37.

MacPhail

Verdun-Jones

(2013, January). Mental illness and the criminal justice system. Paper presented at the Re-Inventing Criminal Justice: The Fifth National Symposium Montreal, Québec, Canada.

38.

Mills

J. F.

Anderson

Kroner

D. G.

(2004). The antisocial attitudes and associates of sex offenders. Criminal Behaviour and Mental Health, 14, 134-146. doi:10.1002/cbm.578

39.

Mills

J. F.

Kroner

D. G.

(1999). Measures of Criminal Attitudes and Associates. Unpublished User Guide.

40.

Mills

J. F.

Kroner

D. G.

(2001). Measures of Criminal Attitudes and Associates (MCAA). Unpublished instrument and user guide.

41.

Mills

J. F.

Kroner

D. G.

Forth

A. E.

(2002). Measures of Criminal Attitudes and Associates (MCAA): Development, factor structure, reliability and validity. Assessment, 9, 240-253. doi:10.1177/1073191102009003003

42.

Mills

J. F.

Kroner

D. G.

Hemmati

(2004). The Measures of Criminal Attitudes and Associates (MCAA): The prediction of general and violent recidivism. Criminal Justice and Behavior, 31, 717-733. doi:10.1177/0093854804268755

43.

Olver

Stockdale

Wormith

(2014). Thirty years of research on the level of service scales: A meta-analytic examination of predictive accuracy and sources of variability. Psychological Assessment, 26(1), 156-176. doi:10.1037/a0035080.

44.

Penner

E. K.

Viljoen

J. L.

Douglas

K. S.

Roesch

(2014). Procedural justice versus risk factors for offending: Predicting recidivism in youth. Law and Human Behavior, 38, 225-237. doi:10.1037/lhb0000055

45.

Prins

Draper

(2009). Improving outcomes for people with mental illness under community corrections supervision: A guide to research informed policy and practice. New York, NY: Council of State Governments Justice Center.

46.

Putkonen

Komulainen

E. J.

Virkkunen

Eronen

Lonnqvist

(2003). Risk of repeat offending among violent female offenders with psychotic and personality disorders. American Journal of Psychiatry, 160, 947-951. doi:10.1176/appi.ajp.160.5.947

47.

Seto

M. C.

(2005). Is more better? Combining actuarial risk scales to predict recidivism among adult sex offenders. Psychological Assessment, 17(2), 156-167. doi:10.1037/1040-3590.17.2.156

48.

Skeem

J. L.

Winter

Kennealy

Louden

J. E.

Tatar

J. R.

II . (2013). Offenders with mental illness have criminogenic needs, too: Toward recidivism reduction. Law and Human Behavior, 38(23), 212-224. doi:10.1037/lhb0000054

49.

Stevenson

H. E.

Wormith

S. J.

(1987). Psychopathy and the Level of Service Inventory (User Report No. 1987-25). Ottawa, Ontario: Solicitor General Canada.

50.

Summers

Loza

(2004). Cross validation of the Self-Appraisal Questionnaire (SAQ): A tool for assessing violent and non-violent recidivism on Australian offenders. Psychiatry, Psychology and Law, 11, 254-262. doi:10.1375/pplt.2004.11.2.254

51.

Torrey

E. F.

Kennard

A. D.

Eslinger

Lamb

Pavle

(2010, May). More mentally ill persons are in jails and prisons than hospitals: A survey of the states. Retrieved from http://www.treatmentadvocacycenter.org/storage/documents/final_jails_v_hospitals_study.pdf

52.

Villeneuve

D. B.

Oliver

Loza

(2003). Cross-validation of the Self-Appraisal Questionnaire with a maximum-security psychiatric population. Journal of Interpersonal Violence, 18, 1325-1334. doi:10.1177/0886260503256841

53.

Walters

G. D.

(2006). Risk-appraisal versus self-report in the prediction of criminal justice outcomes a meta-analysis. Criminal Justice and Behavior, 33, 279-304. doi:10.1177/0093854805284409

54.

Ward

Mesler

Yates

(2007). Reconstructing the risk-need-responsivity model: A theoretical elaboration and evaluation. Aggression and Violent Behavior, 12, 208-228. doi:10.1016/j.avb.2006.07.001

55.

Wolff

Epperson

Fay

(2010, October). Mental health probation officers: Stopping justice involvement before incarceration (Policy Brief Series). New Brunswick, NJ: Center for Behavioral Health Services & Criminal Justice Research, Rutgers University. Retrieved from http://www.cbhs-cjr.rutgers.edu/pdfs/Policy_Brief_Oct_2010.pdf

56.

Wormith

(1997). Research to practice: Applying risk/needs assessment to offender classification. Forum on Corrections Research, 9, 26-31.