Counselor Knows Best? Clinician Assessments of In-Prison Substance Abuse Treatment Participants

Abstract

Research has examined responses to substance abuse treatment among criminal justice populations primarily through client self-assessments, and comparatively fewer studies have focused on clinician ratings of client progress. Recently, Blasko and Hiller examined counselor ratings and their association with reincarceration but found no relationship between the two. The current study partially replicated their analyses and explored its own factor structure to compare the findings from a different sample using the same instrument, the Criminal Justice Counselor Rating of Client (CJ CRC). Analyses revealed that although none of the replicated counselor measures predicted reincarceration, higher levels of one measure produced from the factor analysis, Psychological Barriers to Recovery, indicated a significantly higher probability for reincarceration. In light of these conflicting findings, practical implications of this research suggested that clinical assessments of clients may be useful for criminal justice decision making.

Keywords

assessment corrections drug treatment therapeutic community

Introduction

Assessment is a way of planning and evaluating treatment progress. The National Institute on Drug Abuse (NIDA; 2006) has stated that “assessment is the first step in the treatment of drug involved offenders” (p. 2). Furthermore, assessment is best when it is an ongoing process, designed to identify drug or alcohol problems over the course of treatment, and to evaluate the dynamic need for intervention and the risk of relapse (Bonta & Andrews, 2007). For maximum utility, client assessments should be conducted before and during treatment to identify and address treatment needs (Hiller, Belenko, Welsh, Zajac, & Peters, 2011; Knight, Simpson, & Hiller, 2002; Roberts, Contois, Willis, Worthington, & Knight, 2007) and posttreatment planning (Farabee, Knight, Garner, & Calhoun, 2007). Client assessment is especially salient among prison populations, where rates of drug use have been shown to be relatively high. Although nearly 642,000 state prisoners were estimated to be drug dependent in the year prior to incarceration, only about 15% participated in professionally delivered treatment while in prison (Mumola & Karberg, 2006).

Most research on client treatment progress has focused on self-reported assessments from the participants themselves (e.g., Garner, Knight, Flynn, Morey, & Simpson, 2007; Joe, Broome, Rowan-Szal, & Simpson, 2002; Welsh, 2010). However, researchers note the inherent limitations of self-reported instruments which could affect the validity of the scales due to the possibility for human error (Garner et al., 2007). As an alternative to self-ratings, a limited number of studies have examined clinician assessments of treatment progress (Blasko & Hiller, 2014; Czuchry, Dansereau, Sia, & Simpson, 1998; Farabee et al., 2007). Analyses of the relationship between counselor ratings of treatment responses and the likelihood of recidivism (Farabee et al., 2007) or treatment retention (Hiller, Knight, Sandhya, Simpson, & Rao, 2000) are important, but the relative dearth of such research is unfortunate as the information may be of practical use to treatment providers and criminal justice officials.

Moreover, replicability is essential to the scientific endeavor to confirm, refine, or disconfirm existing findings (Bohannon, 2015; Winerman, 2013). For instance, a recent study of probationers who participated in a therapeutic community (TC) is one of the few that has explored this critical gap of counselor assessments in the literature (Blasko & Hiller, 2014). Despite finding no association between clinical measures of client progress and posttreatment recidivism in their sample, findings from one study can hardly be considered definitive. As such, using the same counselor rating instrument to predict reincarceration, the current study partially replicated Blasko and Hiller’s findings and explored the factor structure of the instrument using data from a prison-based substance abuse treatment sample.

Literature Review

Self-Assessment

Studies that have analyzed self-reported instruments, such as the Criminal Justice Client Evaluation of Self and Treatment (CJ CEST; Institute of Behavioral Research, 2005) and the Resident Evaluation of Self and Treatment (REST; Simpson, Knight, & Broome, 1997), two slightly modified versions of the same instrument, to assess drug treatment progress of offenders have shown improvements in psychosocial functioning among participants (Hiller, Knight, Rao, & Simpson, 2002; Roberts et al., 2007; Welsh, 2010). For example, self-esteem and confidence levels increased and depression decreased over the course of treatment (Hiller et al., 2002), and self-assessments have been found to adequately measure variations in client-level attributes of psychosocial functioning and treatment engagement over time (Garner et al., 2007; Roberts et al., 2007; Welsh & McGrain, 2008).

Hiller and colleagues (2002) assessed the psychosocial functioning and treatment motivation of felony probationers participating in the Dallas County Judicial Treatment Center, a 6-month modified TC using the REST (Simpson, Knight, & Broome, 1997). Over the course of treatment measured at 1, 3, and 6 months, probationers showed improvements in levels of self-esteem and decision-making confidence, and a reduction in depression. However, treatment motivation and readiness decreased over time. Likewise, Welsh (2010) examined similar measures of psychosocial functioning on a sample of prison-based TC participants. Although treatment participants generally showed improvements in psychological and social functioning over time, the patterns of results varied depending upon individual levels of risk (e.g., risk of relapse and recidivism) and motivation for treatment and the time periods examined (at 1, 6, and 12 months of treatment).

Furthermore, in a study on resident perceptions of a TC environment, Carr and Ball (2014) analyzed the impact of the residents’ perceptions on drug treatment retention among a sample of 9-month residential TC participants. Findings revealed that heightened perceptions of the orderliness of the therapeutic environment best predicted treatment completion. Importantly, the authors note that similar assessments could be completed by the staff as well as the residents to determine variations in environmental perception using multiple perspectives.

Although these studies have demonstrated improvements in client-level psychosocial functioning and treatment completion (e.g., Hiller et al., 2002; Roberts et al., 2007; Welsh, 2010), self-ratings may be less valid indicators of actual progress, especially in criminal justice settings (Czuchry et al., 1998), even if such ratings may provide information not readily available to counselors. As such, drug treatment assessments using self-reported measures only could perhaps benefit from a more comprehensive assessment by including the views of staff on the treatment environment (Carr & Ball, 2014) and the implementation of treatment programs (Welsh et al., 2015).

Counselor Assessment

Relatively little attention has been paid to the relationship between counselor assessments and responses to treatment and posttreatment outcomes of participants. Czuchry and colleagues (1998) found that counselor ratings were consistent with self-ratings for predicting treatment engagement and the likelihood of abstaining from drug use. Farabee et al. (2007) discovered that only treatment engagement as measured by counselors was significantly related to recidivism. Interestingly, however, low correlations were found between client and counselor ratings on similar measures, suggesting that counselor ratings should be investigated in addition to self-ratings when predicting treatment progress.

Although little research has examined counselor ratings of client responses to substance abuse treatment, we can look to other therapeutic areas for guidance. The psychiatric assessment literature, for instance, includes clinician ratings of treatment progress using the Brief Psychiatric Rating Scale (BPRS; Overall & Gorham, 1962). The BPRS is one of the most frequently used clinician-completed instruments for evaluating anxiety and depression in patients with schizophrenia (Leucht et al., 2005; Ventura, Nuechterlein, Subotnik, Gutkind, & Gilbert, 2000). Despite the common use of the BPRS, its ability to determine treatment responses remains unclear (Leucht et al., 2005). Often used as a complement to the BPRS is the Brief Symptom Inventory (BSI; Derogatis & Melisaratos, 1983), which is completed by the patient regarding his or her symptoms. The BSI has been used for clinical (Khalil, Hall, Moser, Lennie, & Frazier, 2011) and nonclinical populations (Urbán et al., 2014). One study found that the BSI accurately measured anxiety and depression in patients with heart failure (Khalil et al., 2011). The authors indicate that although self and clinician assessment is frequently used to identify psychosocial symptoms in clinical patients, drug treatment assessment in general is lacking, as little is still known about these types of assessments in either correctional or community settings.

Returning to the substance abuse literature, researchers have examined the factor structures of counselor assessment instruments to understand important themes in the drug treatment process (e.g., Blasko & Hiller, 2014; Hiller et al., 2002). Simpson, Knight, and Hiller (1997) analyzed 19 characteristics of counselor ratings of patient functioning and progress in treatment by performing a factor analysis where three factors emerged—Rapport, Self-Confidence, and Motivation. Hiller and colleagues (2000) examined counselor ratings of treatment clients as measured by the Criminal Justice Counselor Rating of Client (CJ CRC; Simpson, Knight, & Hiller, 1997). Their factor analysis uncovered four measures of client attributes: Therapeutic Engagement, Rapport With Others, Denial, and Psychological Problems, and Blasko and Hiller’s (2014) factor analysis of the CJ CRC client attributes among a sample of felony probationers likewise identified four factors they labeled Agreeableness and Confidence, Rapport, Resistance, and Psychological Discomfort. The authors found that Agreeableness and Confidence and Rapport were positively correlated with one another, as were Resistance and Psychological Discomfort. The authors also reported that the four subscales were not predictive of rearrest at 1, 2, or 3 years after release.

Despite the lack of an association found between counselor assessment and offender recidivism, the Blasko and Hiller (2014) study was one of the few—and the only one in recent years—to closely examine professional drug counselors’ opinions on their charges’ progress through treatment, including how those measures were related to future criminal behavior. Because of the critical role assessment plays in the success of treatment participants (Bonta & Andrews, 2007; NIDA, 2006) and in light of recent replicability issues (Bohannon, 2015), it would be unwise to conclude from a single study that counselor assessments are unworthy of additional empirical scrutiny and unrelated entirely to participants’ later outcomes.

As such, the current study sought to partially replicate the analyses performed by Blasko and Hiller (2014) and further explore the relationship between counselor assessment and recidivism using data from a different sample of drug-involved offenders from a different treatment setting. The data for the study came from offenders who participated in one of two prison-based substance abuse treatment programs within the same facility. It is important to note here the critical similarity between Blasko and Hiller’s study and the current one is that the same assessment instrument—CJ CRC—was used in both. We initially recreated the subscales based on Blasko and Hiller’s psychometric analyses and examined their intercorrelations and association with reincarceration. Furthermore, we conducted our own examination of the factor structure of the CJ CRC using our prison-based sample and subjected the resulting factors to the same analyses predicting reincarceration.

Method

Setting

The State Correctional Institution in Chester, Pennsylvania (SCI-Chester) is a 1,200-bed medium security prison for males in need of specialized drug treatment. At the time of data collection, Gaudenzia Inc., a well-known substance abuse treatment organization served as the third-party treatment provider of the programs at SCI-Chester. Programs were either an intensive, 12-month TC or a less intensive 12-month Outpatient (OP) treatment program. Graduates of both programs were required to complete a mandatory, 6-month aftercare program upon release to the community.

The modified TC program was designed for inmates with a moderate to severe need for drug treatment. Inmates in this program received approximately 24 to 37 hr of treatment a week for up to 12 months depending on an inmate’s individual needs. The program began with an orientation phase (1 month) followed by intensive treatment (6 months) where inmates are given more responsibility and program involvement as they progress in the program. The last 5 months were focused on community reentry and individual aftercare plans. Outpatient treatment was designed for inmates with a lower need for drug treatment where clients received approximately 6 to 8 hr of treatment a week over the 12 months. Upon release from custody, 6 months of outpatient-level community aftercare (CAC) was required for all inmates released from SCI-Chester.

Participants

Between January 2003 and March 2004, all inmates transferred to SCI-Chester were approached and asked to participate in the study, with an overall participation rate of 95% (Welsh, 2006). In total, 731 inmates who had a documented history of substance abuse met the requirements for need for treatment measured by the TCU Drug Screen II (TCUDS-II; Simpson, Knight, & Broome, 1997) and had 18 to 34 months left of their prison sentence gave written informed consent to be part of the study. The final sample for the present analyses included 616 inmates, representing those who had been released from prison at the time of data collection for the outcome measure and who had a valid CJ CRC score at the end of the treatment experience. A small number of inmates (n = 28) were still in custody at the time reincarceration data were collected in September 2008, and the remaining missing cases are due to inmates failing to complete their in-prison treatment programs (e.g., misconduct, transfer to another prison, or release from custody). As such, the current sample consists only of inmates who successfully graduated from treatment due to the missing data.

Participants were randomly assigned to treatment modality based on the first digit in their inmate identification number: Inmates with an even number were assigned to one of five TC units and those with an odd number were assigned to one of seven OP units. A less intensive treatment group was preferable to a no-treatment group, as no previous study has compared two in-prison treatment modalities. Random assignment of inmates was successful in that the TC and OP groups were statistically equal in age at admission, time remaining until release, criminal history, current offense severity, and TCUDS-II score to make valid comparisons of the treatment process and posttreatment outcomes (Welsh, 2006).

Measures

CJ CRC

The CJ CRC (Simpson, Knight, & Hiller, 1997) form provides a clinical assessment of treatment progress. A CJ CRC was completed for each client during his 12th month of treatment (i.e., the last month). The CJ CRC consists of 25 items on client attributes in treatment, such as engagement in treatment, rapport with counselors and other participants, and psychosocial issues (Hiller et al., 2000). Items were rated on a 7-point Likert-type scale ranging from strongly disagree (1) to strongly agree (7).

To address the first objective of the study, four subscales were created by averaging the item scores that corresponded to the factor analysis of 18 CJ CRC items reported by Blasko and Hiller (2014), and we conducted reliability analyses to evaluate the appropriateness of grouping the items together with the SCI-Chester data. It must be noted, however, that the Cronbach’s coefficient alpha is affected by the number of items included in a subscale. In other words, the more items per scale, the bigger the alphas are likely to be. Agreeableness and Confidence (α = .906) included the items regarding client self-confidence, freely expresses wishes, participates in group discussions, clearly expresses thoughts, and has good memory and recall. Rapport (α = .863) consisted of easy to talk to, warm and caring, cooperative, liked by other clients, and liked by staff. Resistance (α = .702) consisted of honest and sincere (reverse scored), manipulative, motivated to recovery (reverse scored), and in denial about problems. Finally, Psychological Discomfort (α = .822) included hostile or aggressive, depressed, impulsive, and nervous or anxious. These subscales were ultimately compared with SCI-Chester’s factor analysis (see Table 4).

Control Variables

Several inmate characteristics were collected upon admission to prison to control for their relationship to recidivism. Offense Gravity Scores (OGS) based on Pennsylvania sentencing guidelines provided information on an inmate’s criminal history (OGS-Prior) and the severity of the offense (OGS-Current) that resulted in the current sentence to SCI-Chester. Other inmate characteristics such as age were obtained through the Department of Corrections (DOC). Participants’ need for treatment was measured using the TCUDS-II (Knight et al., 2002). The total score can range from 0 to 9, in which a score of 3 or higher represents a drug dependence diagnosis according to the Diagnostic and Statistical Manual of Mental Disorders (5th ed.; DSM-5; American Psychiatric Association, 2013). Posttreatment time at risk in the community was measured by subtracting the date of release from prison from the date of when the outcome data were collected. Finally, program modality (TC vs. OP) was used to control for the type of treatment received.

Dependent Variable

The outcome measure in this study was reincarceration, defined as any return to prison either due to a technical parole violation or new conviction. Technical parole violations account for the majority of reasons for reincarceration, with new convictions accounting for only 23.5% of those who were reincarcerated. Reincarceration data were obtained through the DOC Inmate Records System.

Analytic Plan

Analyses were performed to replicate Blasko and Hiller’s (2014) study on counselor ratings in predicting reincarceration using computed subscales based on their factor solution, and measures from a principal components analysis (PCA) that best fit the current study’s data. To first explore the relationship between counselor assessment and reincarceration with Blasko and Hiller’s four CJ CRC subscales, bivariate correlations and intercorrelations between the variables were examined. Next, binary logistic regression was used, first with only the four CJ CRC subscales of Agreeableness and Confidence, Rapport, Treatment Resistance, and Psychological Discomfort to predict reincarceration, and then controlling for program modality, age, OGS, TCUDS-II, and time at risk. We conducted a PCA using the Chester sample and for a comparison, examined confirmatory factor analysis (CFA) goodness-of-fit indices for this solution and that of Blasko and Hiller. Again, bivariate correlations between the variables were examined, and binary logistic regression was used to examine the relationship between the factors produced and reincarceration, first using only the Chester CJ CRC factors and then adding the control variables. Receiver operating characteristics (ROC) with area under the curve (AUC) analyses were then conducted to determine the appropriateness of each model using the current data.

Results

Descriptive Statistics

Descriptive statistics are presented in Table 1. Overall, 37% of the sample was reincarcerated over the follow-up period, which averaged 30.69 months (SD = 10.33). The average participant age was 32.38 years (SD = 8.87), the average OGS-Current score was 6.79 (SD = 1.21) and 5.99 (SD = 1.91) for OGS-Prior, and the mean TCUDS-II score was 4.2 (SD = 2.78). Also included in Table 1 are the descriptive statistics for the four factors reported by Blasko and Hiller (2014) and the corresponding computed subscales from the current sample. In general, comparing the two samples indicates similar average ratings for the four subscales, with the replicated scores from the Chester sample being slightly higher for the Agreeableness and Confidence and Rapport measures, and slightly lower ratings for Treatment Resistance and Psychological Discomfort in the current sample.

Table 1:

Participant and Instrument Descriptive Statistics

Variables	n	Minimum	Maximum	M	SD
Reincarceration (0 = no, 1 = yes)	616	0.00	1.00	0.37	0.48
Demographic and control variables
Treatment Modality (0 = OP, 1 = TC)	616	0.00	1.00	0.48	0.50
Age (years)	616	18.00	73.00	32.14	8.78
OGS-Current	616	2.00	10.00	6.79	1.19
OGS-Prior	545	0.00	10.00	5.99	1.89
TCUDS-II	599	0.00	9.00	4.17	2.78
Time at Risk (months)	616	0.03	55.30	30.69	10.33
Replicated CJ CRC subscales
Agreeableness and Confidence (M = 4.80, SD = 0.98)	616	1.00	7.00	5.24	0.86
Rapport (M = 5.13, SD = 0.96)	616	1.00	7.00	5.35	0.82
Resistance (M = 4.53, SD = 0.58)	616	1.00	7.00	2.99	1.05
Psychological (M = 3.68, SD = 1.10)	616	1.00	6.00	2.78	1.24

Note. OP = outpatient; TC = therapeutic community; OGS = Offense Gravity Scores; TCUDS-II = TCU Drug Screen II; CJ CRC = Criminal Justice Counselor Rating of Client.

Partial Replication of Blasko and Hiller Subscales and Analyses

Bivariate correlations between reincarceration and the four subscales and the subscales with one another were examined to begin the replication of Blasko and Hiller’s (2014) counselor assessments (see Table 2). The four CJ CRC subscales are all highly correlated with one another, especially Agreeableness and Confidence and Rapport (r = .74, p < .001) and Treatment Resistance and Psychological Discomfort (r = .68, p < .001). Only Treatment Resistance (r = .14, p < .001) and Psychological Discomfort (r = .17, p < .001) were significantly associated with reincarceration.

Table 2:

Correlations Between Reincarceration and Blasko and Hiller’s CJ CRC Subscales

Outcome variable and subscales	Reincarceration	Agreeableness and Confidence	Rapport	Treatment Resistance	Psychological Discomfort
Reincarceration	—	−.05	−.08	.14*	.17*
Agreeableness and Confidence		—	.74*	−.57*	−.26*
Rapport			—	−.59*	−.26*
Treatment Resistance				—	.68*
Psychological Discomfort					—

Note. CJ CRC = Criminal Justice Counselor Rating of Client.

p < .001.

Table 3 presents the results of the two binary logistic regression models predicting reincarceration, with the first examining only the four CJ CRC computed subscales. In this model, Psychological Discomfort was a significant predictor for reincarceration (odds ratio [OR] = 1.24, p = .023), though none of the others reached statistical significance. In the second model, four CJ CRC subscales were used to predict reincarceration while controlling for program modality, age, the two OGS variables, TCUDS-II, and time at risk in the community. Once the control variables were added to the model, Psychological Discomfort was no longer a significant predictor of reincarceration. In addition, age (OR = 0.949, p < .001) and time at risk (OR = 1.002, p < .001) were significant predictors; as age increased, the probability of reincarceration decreased, and as time at risk increased, the probability of reincarceration increased. OGS-Prior (OR = 1.168, p = .003) was also a significant predictor of the outcome, with higher scores on this variable leading to an increased probability of reincarceration.

Table 3:

Logistic Regression Predicting Reincarceration Using Blasko and Hiller’s CJ CRC Subscales

	Model 1				Model 2
Subscales and control variables	B	SE	p	OR	B	SE	p	OR
Agreeableness and Confidence	.140	.151	.353	1.150	.211	.176	.231	1.235
Rapport	−.128	.162	.428	0.880	−.105	.143	.463	0.901
Treatment Resistance	.119	.137	.385	1.126	.115	.149	.441	1.121
Psychological Discomfort	.213	.094	.023	1.237	.133	.112	.235	1.142
Program Modality					−.125	.200	.523	0.882
Age					−.053	.012	<.001	0.949
OGS-Current					.057	.083	.496	1.058
OGS-Prior					.155	.053	.003	1.168
TCUDS-II					.041	.035	.244	1.041
Time at Risk					.002	.000	<.001	1.002
Constant	−2.262	1.429	.114	0.217	1.782	1.296	.169	0.168

Note. AUC = .70; Nagelkerke R² = .042; −2 log likelihood = 794.949. CJ CRC = Criminal Justice Counselor Rating of Client; OR = odds ratio; OGS = Offense Gravity Scores; TCUDS-II = TCU Drug Screen II; AUC = area under the curve.

SCI-Chester Factor Analysis and Reincarceration

Next, we conducted a PCA with Varimax rotation using the full complement of 25 CJ CRC items related to client attributes in treatment. Eigenvalues >1 were used to determine factors, and then reliability statistics were examined for the appropriateness of grouping the items together. A clear two-factor solution emerged representing two distinct themes in participant responses to treatment (see Table 4 for factor loadings).¹ The first factor, Treatment Engagement and Rapport (α = .96), indicating the participants’ treatment attitudes and engagement, is similar to Blasko and Hiller’s (2014) two factors of Agreeableness and Confidence and Rapport. Likewise, the second factor, Psychological Barriers to Recovery (α = .86), indicating a client’s struggle with recovery and mental health issues, is similar to Blasko and Hiller’s two subscales of Treatment Resistance and Psychological Discomfort. A one-item factor, self-confident, was ultimately removed from the analyses as it did not load onto either of the other two factors (nor was it in any way related to the outcome measure).

Table 4:

CJ CRC Factor Analysis

Counselor rating items	Treatment Engagement and Rapport (α = .959)	Psychological Barriers to Recovery (α = .861)
Pays attention	0.854	−0.141
Thinks clearly	0.819	−0.190
Consistent	0.814	−0.068
Liked by staff	0.807	−0.071
Cooperative	0.805	−0.234
Clearly expresses thoughts	0.800	−0.093
Reviews problems logically	0.791	−0.275
Participates	0.790	−0.038
Honest and sincere	0.790	−0.226
Easy to talk to	0.755	−0.193
Motivated to recover	0.753	−0.199
Warm and caring	0.749	−0.283
Follows through commitments	0.745	−0.087
Responsible	0.698	−0.246
Liked by others	0.664	0.069
Good memory and recall	0.655	−0.046
Freely expressed wishes	0.540	−0.006
Impulsive	−0.151	0.811
Nervous or anxious	−0.060	0.762
Depressed	0.000	0.744
Manipulative	−0.097	0.743
Hostile or aggressive	−0.118	0.706
Easily distracted	−0.173	0.696
In denial about problems	−0.237	0.615
Initial Eigenvalues	11.370	3.329

Note. Loadings from rotated factor matrix. CJ CRC = Criminal Justice Counselor Rating of Client. Shading is used to aesthetically accentuate the negative coefficients from the positive coefficients for the 24 items on the CRC scale related to Treatment Engagement and Rapport and Psychological Barriers to Recovery used in the factor analysis.

Both the two-factor solution from the current PCA and the items that loaded onto Blasko and Hiller’s (2014) four-factor solution were compared using CFA. The goodness-of-fit indices examined were very close for the two models: The comparative fit index (CFI) for the current factor structure was .883 versus .855 for the Blasko and Hiller solution, and the root mean square error of approximation values (RMSEA) where lower values indicate a better fit were .09 and .11, respectively. As demonstrated in the CFA, the two-factor solution is a marginally better solution, and the standardized scores from the PCA were used in the logistic regression models. The factors were orthogonal, and bivariate correlations showed a significant, positive correlation between reincarceration and Psychological Barriers to Recovery (r = .159, p < .001).

To compare predictions of reincarceration with that of Blasko and Hiller’s (2014) replicated subscales, two final binary logistic regression models were performed. As presented in Table 5, the first model included only the two CJ CRC variables produced from the factor analysis reported above. In this model, Psychological Barriers to Recovery was a significant predictor for reincarceration (OR = 1.390, p < .001). In the second model, the control variables were added to the two CJ CRC scales. Even when controlling for program type, age, TCUDS-II, current and prior OGS, and time at risk, Psychological Barriers to Recovery remained a significant predictor of reincarceration (OR = 1.296, p = .010) where participants with higher levels of Psychological Barriers to Recovery had a significantly increased probability of reincarceration. Finally, ROC with AUC analyses provide a more precise picture regarding the predictability of the CJ CRC using Blasko and Hiller’s four subscales and our two factors. Results showed that both of the final logistic regression models—using the two Chester factors and the four Blasko and Hiller subscales and control variables—demonstrated an equally good fit for the data (AUCs = .70).

Table 5:

Logistic Regression for Reincarceration Using Factor Scores

	Model 1				Model 2
Subscales and control variables	B	SE	Significance	OR	B	SE	Significance	OR
Treatment Engagement and Rapport	−.090	.083	.281	0.914	−.130	.095	.170	0.878
Psychological Barriers to Recovery	.329	.084	<.001	1.390	.259	.101	.010	1.296
Program Modality					−.138	.203	.496	0.871
Age					−.053	.012	<.001	0.949
OGS-Current					.062	.083	.455	1.064
OGS-Prior					.154	.053	.004	1.167
TCUDS-II					.039	.035	.257	1.040
Time at Risk					.002	.000	<.001	1.002
Constant	−0.530	0.085	<.001	0.588	−1.64	0.835	.049	0.194

Note. AUC = .70; Nagelkerke R² = .151; −2 log likelihood = 645.829. OR = odds ratio; OGS = Offense Gravity Scores; TCUDS-II = TCU Drug Screen II; AUC = area under the curve.

Discussion

Although there is a vast body of literature on drug treatment assessment, most research on responses to drug treatment has focused on the self-reported experiences from the clients themselves. Self-reports can be useful to identify and evaluate client functioning in the treatment process, but using only self-assessments could present issues of bias and faulty memory. In other therapeutic settings, counselor-completed assessments are more common, such as the BRPS used for patients diagnosed with schizophrenia (Leucht et al., 2005; Ventura et al., 2000). Few studies have examined counselor perspectives of substance abuse treatment clients in a correctional setting (Czuchry et al., 1998; Farabee et al., 2007), and one of the few, if only, recent examples of counselor rating analyses, Blasko and Hiller (2014) was central to the present article.

In social science research, the ability to replicate studies is desirable and should be performed to ensure the generalizability of published findings (Winerman, 2013). As such, it is important to test previous findings for their replicability, reliability, and validity, using similar instruments, but with different samples or populations. Although the present study is only a partial replication of Blasko and Hiller (2014), we analyzed posttreatment recidivism using the same counselor rating instrument, but with a sample of prison-based drug treatment participants.

The primary focus of this study was to examine counselor ratings from the CJ CRC (Simpson, Knight, & Hiller, 1997) using both Blasko and Hiller’s reported subscales, as well as analyzing the underlying factor structure of the SCI-Chester sample. We reproduced Blasko and Hiller’s (2014) four subscales—Agreeableness and Confidence, Rapport, Treatment Resistance, and Psychological Discomfort—and then produced two factors from a PCA—Treatment Engagement and Rapport and Psychological Barriers to Recovery. Also of interest was whether the CJ CRC measures, both the replication scales and the current study’s factors, were significant predictors of reincarceration. This was of particular interest because criminal justice practitioners can use clinician reports to make informed decisions regarding prisoner release and treatment implementation. Similar to Blasko and Hiller’s (2014) study on the CJ CRC, we did not find a significant relationship between any of the four replicated CJ CRC subscales and reincarceration. In an analysis using the current study’s two factors, however, we found that higher scores of Psychological Barriers to Recovery indicate an increased probability for reincarceration even after controlling for treatment modality, age, current and prior OGS, TCU-II, and time at risk.

The factor analysis conducted for this study differed greatly from that in Blasko and Hiller (2014) who reported a clear four-factor solution (Table 2, p. 262). It is important to note, however, that they dropped several items from the CJ CRC, including reviews problems logically, thinks clearly, pays attention, easily distracted, consistently follows through on commitments, and responsible. The first factor was Agreeableness and Confidence and the second Rapport. However, in the current study, many of these items loaded together onto one factor, labeled Treatment Engagement and Rapport. Likewise, the replicated subscales Treatment Resistance and Psychological Discomfort revealed factor loadings similar to the current study’s factor labeled Psychological Barriers to Recovery.

Our results in comparison with Blasko and Hiller (2014) present interesting findings with regard to recidivism. None of the replicated subscales in the logistic regression model for predicting reincarceration were significant, similar to Blasko and Hiller’s factors which also failed to reach significance. In contrast, however, one factor from the present factor analysis, Psychological Barriers to Recovery, was a significant predictor of reincarceration in the logistic regression model controlling for other variables. The latter results, to some degree, warrant further consideration of counselor ratings to measure client responses to prison-based treatment.

Implications

Staff perceptions of client responses to drug treatment are critically important as these professionals serve as important facilitators for change (Aarons, Hurlburt, & Horwitz, 2011). A practical implication of this research is that clinical assessments of client progress may be useful for criminal justice decision making. These findings highlight the importance of using multiple methods of assessment to evaluate offender progress for in-prison drug treatment programs. Criminal justice practitioners would benefit from considering clinician ratings of progress in treatment in addition to other forms of assessment when making informed decisions regarding offender release and program implementation.

Based on our significant finding of Psychological Barriers to Recovery, clients experiencing high levels of depression, anxiety, and other psychological difficulties have an increased probability of being reincarcerated. Clients identified with these conditions could perhaps benefit from particular attention by treatment staff for more intensive counseling that deals specifically with these problem areas. Moreover, correctional decision makers could give more direct attention and resources to offenders with difficulties in psychological functioning that could ultimately limit their ability to make progress in treatment or have difficulty after release from custody.

We recognize the ability to provide mental health services in conjunction with substance abuse treatment; it is often not feasible due to the scarcity of resources for such programming, and we acknowledge the uniqueness of SCI-Chester in this regard. Regardless, as NIDA (2006) asserted, the first crucial step in effective service delivery is assessment, and the CJ CRC could be beneficial in resource-depleted correctional settings. Notably, the CJ CRC is brief, is provided at no cost by the Texas Christian University Institute for Behavioral Research, and it does not require significant training or an advanced credential (e.g., LCSW, PhD, PsyD) to administer the instrument. With its established validity with a criminal justice population, actuarial assessments like the CJ CRC would be important to include in conjunction with the less systematic counselor impressions (e.g., first-generation assessment tools). The properties of the CJ CRC pertaining to psychological needs could be of practical use in at least identifying and perhaps tailoring the services some offenders receive while participating in drug treatment.

On the contrary, one could argue that having better-trained clinicians would be more accurate (e.g., clinical psychologists, psychiatrists) in assessing treatment progress and identifying client symptoms as opposed to correctional staff who have not received extensive training in clinical decision making. While many of the items on the CJ CRC are easily identifiable by those without extensive training, (e.g., easy to talk to, participates in group discussions), other items (e.g., nervous/anxious, depressed) might not be as recognizable to the untrained eye. Interestingly, according to Welsh (2006), both Gaudenzia and SCI-Chester correctional and administrative staff received additional, mandatory training on the TC method. Furthermore, as helpfully suggested by one of the reviewers of this article, it is worthwhile to consider the advantages of a briefer instrument if fewer items do not result in any degradation of its relationship to future outcomes. Completing 18 instead of 25 questions may result in a substantial reduction over time in the effort needed to assess client performance, but on balance, the results from the current study suggest that the 25 items may be preferable to the 18 used by Blasko and Hiller (2014). We acknowledge that more research from a greater number of samples is needed, but in the Chester sample, the data fit the two-factor solution better than the four subscales did. Furthermore, the parsimony of having only two factors is preferable to four, as one of them was found to be a significant predictor of reincarceration.

Limitations, Future Directions, and Conclusions

One limitation to the study is that it is not necessarily generalizable. Because of the uniqueness of the prison as a specialized drug treatment facility and that all prisoners required some form of drug treatment, these results may be different in prisons not specialized in drug treatment or on a sample of general population inmates not in treatment. Thus, this limitation affirms the importance of replication. More research on drug treatment assessment using the CJ CRC is needed using different samples, such as women, pretrial defendants, drug court participants, and drug users with mental health issues.

Another limitation is the final sample included only those who had been released and completed the program. Participants who failed to complete the treatment were removed. In addition to the 29 participants who had not been released to the community at the time of data collection, this brought the sample to 616. Including all participants might have affected the results; however, as this study was primarily focused on examining counselor ratings (which were not available for the excluded participants), they could not be included in the sample.

A formal risk of recidivism measure was not included in the final logistic regression model as a control variable (i.e., variables known to influence the dependent variable) as such data from the Level of Service Inventory – Revised (LSI-R) were only available for about half of the sample (see Kelly & Welsh, 2008). Although age, prior criminal history (OGS), need for treatment (TCUDS), and time in the community were used as control variables, as they were significant predictors of reincarceration, this limitation should still be noted.

Furthermore, there was a potential issue with Blasko and Hiller’s (2014) CJ CRC correlations in that Agreeableness and Confidence was highly correlated to Rapport. This is most likely because many of the items in both subscales are similar as they are positive attributes of clients in treatment. For example, consider the items warm and caring in Agreeableness and Confidence, and cooperative and liked by staff in Rapport. They both indicate a positive response to treatment. Therefore, it may not be feasible to use both in the same factor because they are simply too similar. However, because Blasko and Hiller (2014) used all four subscales, we attempted to replicate their results.

There is growing support in the literature for using multiple indicators to assess client responses to treatment. For example, Farabee and colleagues (2007) examined risk of relapse by comparing self and counselor ratings and found low correlations between self and counselor ratings on similar measures. Carr and Ball (2014) indicated that one significant limitation to their study was that questionnaires “were only completed by the study participants, thus offering one set of perspectives on the conditions of this therapeutic program” (Carr and Ball, 2014, p. 572). For a more comprehensive assessment, it would be beneficial to include the views of staff as well on similar measures. Counselor perspectives could potentially differ from client perspectives due to the professional training counselors endure that allows them to observe and assess these responses to treatment, compared to the bias in self-assessment when, for instance, a client believes he is doing better in treatment than he really is.

Using the same data as the current study, the next step in this line of research is to not only examine counselor ratings between treatment modalities but also compare further self and counselor ratings. This research could explore differences and similarities in responses to treatment between participants in outpatient/group counseling (OP) compared with those in a TC. Welsh, Zajac, and Bucklen (2014) found that TC treatment participants who self-reported higher Negative Affect (e.g., depression, anxiety) scores had significantly higher reincarceration rates compared with OP. This relationship between treatment modalities and self versus counselor ratings could be further explored using the same data as the present study as Negative Affect is a similar construct to Psychological Barriers to Recovery. An analysis on the comparison of self and counselor ratings on these similar measures could be used to explore possible discrepancies between self-reports and client assessments.

The current study examined measures from the CJ CRC and their association with reincarceration among a sample of in-prison treatment participants. Although the vast majority of drug treatment literature focuses on self-assessments obtained from clients, there has been little research on counselor ratings of drug treatment participants. Our findings, in contrast with Blasko and Hiller, suggest that counselor assessments should also be taken into consideration in assessing client responses to drug treatment and outcomes such as recidivism. While there is a need for further examination and replication of these findings, the improved assessment of client responses to treatment has critical implications for improving client outcomes.

Footnotes

Acknowledgements

The authors wish to thank the anonymous reviewers, Kim Logio, Susan Clampet-Lundquist, and Keith Brown, for their invaluable assistance on earlier drafts.

This research reported here was supported by Grant 2002-RTBX-1002 from the U.S. Department of Justice, National Institute of Justice. Opinions expressed here are those of the authors and not necessarily of the U.S. government. Any errors or omissions are the responsibility of the authors alone.

An earlier version of this paper was presented at the 2015 American Society of Criminology Annual Meetings in Washington, DC.

Notes

Jennifer N. Stanley, MS, is a recent graduate of Saint Joseph’s University’s program in criminal justice. In addition to her interest in corrections and substance abuse treatment research, she has contributed to studies exploring formal and informal processing of juveniles in the justice system and a field experiment regarding interview rooms with the Philadelphia Police Department.

Christopher E. Kelly, PhD, is an assistant professor in the Department of Sociology and Criminal Justice at Saint Joseph’s University where he studies interrogation, jury decision making, and correctional treatment. His work has been published in Law and Human Behavior, Justice Quarterly, and Annual Review of Law and Social Science.

Wayne N. Welsh, PhD, is a professor of criminal justice at Temple University. His research focuses on violence, organizational change, and substance abuse treatment in criminal justice settings. He is author of Counties in Court: Jail Overcrowding and Court-Ordered Reform (Temple University Press, 1995), Criminal Violence: Patterns, Causes, and Prevention, coauthored with Marc Riedel (Oxford, 2015, 4th ed.), and Criminal Justice Policy and Planning, with Philip Harris (Routledge, 2016, 5th ed.). Recent articles have appeared in Journal of Experimental Criminology, Criminal Justice and Behavior, and Drug and Alcohol Dependence.

References

Aarons

G. A.

Hurlburt

Horwitz

S. M.

(2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4-23.

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA: American Psychiatric Publishing.

Blasko

B. L.

Hiller

M. L.

(2014). Clinician ratings of client progress in a therapeutic community treatment setting: Do ratings predict outcomes? Journal of Offender Rehabilitation, 53, 253-272. doi:10.1080/10509674.2014.902007

Bohannon

(2015). Many psychology papers fail replication test. Science, 349, 910-911. doi:10.1126/science.349.6251.910

Bonta

Andrews

D. A.

(2007). Risk-need-responsivity model for offender assessment and rehabilitation. Ottawa, Ontario: Her Majesty the Queen in Right of Canada.

Carr

W. A.

Ball

S. A.

(2014). Predictors and treatment outcomes of perceived ward atmosphere among therapeutic community residents. Journal of Substance Abuse Treatment, 46, 567-573.

Czuchry

Dansereau

D. F.

Sia

T. L.

Simpson

D. D.

(1998). Using peer, self, and counselor ratings to evaluate treatment progress. Journal of Psychoactive Drugs, 30, 81-87.

Derogatis

Melisaratos

(1983). The Brief Symptom Inventory: An introductory report. Psychological Medicine, 13, 595-605.

Farabee

Knight

Garner

B. R.

Calhoun

(2007). The inmate prerelease assessment for reentry planning. Criminal Justice and Behavior, 34, 1188-1197. doi:10.1177/0093854807304429

10.

Garner

B. R.

Knight

Flynn

P. M.

Morey

J. T.

Simpson

D. D.

(2007). Measuring offender attributes and engagement in treatment using the client evaluation of self and treatment. Criminal Justice and Behavior, 34, 1113-1130.

11.

Hiller

M. L.

Belenko

Welsh

Zajac

Peters

R. H.

(2011). Screening and assessment: An evidence-based process for the management and care of adult drug-involved offenders. Issues in Children’s and Families’ Lives, 11, 45-62.

12.

Hiller

M. L.

Knight

Rao

S. R.

Simpson

D. D.

(2002). Assessing and evaluating mandated correctional substance-abuse treatment. In Leukefeld

C. G.

Tims

Farabee

(Eds.), Treatment of drug offenders: Policies and issues (pp. 41-56). New York, NY: Springer.

13.

Hiller

M. L.

Knight

Sandhya

Simpson

D. D.

Rao

S. R.

(2000). Process Assessment of Correctional Treatment (PACT) (Report No. 184507). Retrieved from ncjrs.gov

14.

Institute of Behavioral Research. (2005). TCU CJ Client Evaluation of Self and Treatment (TCU CJ CEST). Fort Worth, TX: Institute of Behavioral Research, Texas Christian University. Retrieved from Ibr.tcu.edu

15.

Joe

G. W.

Broome

K. M.

Rowan-Szal

G. A.

Simpson

D. D.

(2002). Measuring patient attributes and engagement in treatment. Journal of Substance Abuse Treatment, 22, 183-196.

16.

Kelly

C. E.

Welsh

W. N.

(2008). The predictive validity of the level of service inventory-revised for drug involved offenders. Criminal Justice and Behavior, 35, 819-831.

17.

Khalil

A. A.

Hall

L. A.

Moser

D. K.

Lennie

T. A.

Frazier

S. K.

(2011). The psychometric properties of the Brief Symptom Inventory depression and anxiety subscales in patients with heart failure and with or without renal dysfunction. Archives of Psychiatric Nursing, 25, 419-429.

18.

Knight

Simpson

D. D.

Hiller

M. L.

(2002). Screening and referral for substance-abuse treatment in the criminal justice system. In Leukefeld

C. G.

Tims

Farabee

(Eds.), Treatment of drug offenders: Policies and issues (pp. 259-272). New York, NY: Springer.

19.

Leucht

Kane

J. M.

Kissling

Hamann

Etschel

Engel

(2005). Clinical implications of Brief Psychiatric Rating Scale scores. British Journal of Psychiatry, 187, 366-371.

20.

Mumola

C. J.

Karberg

J. C.

(2006). Drug use and dependence, state and federal prisoners, 2004 (Bureau of Justice Statistics Special Report; NCJ 213530). Washington, DC: U.S. Department of Justice, Bureau of Justice Statistics.

21.

National Institute on Drug Abuse. (2006). Principles of drug abuse treatment for criminal justice populations. Rockville, MD: Author.

22.

Overall

J. E.

Gorham

D. R.

(1962). The Brief Psychiatric Rating Scale. Psychological Reports, 10, 790-812.

23.

Roberts

E. A.

Contois

M. W.

Willis

J. C.

Worthington

M. R.

Knight

(2007). Assessing offender needs and performance for planning and monitoring criminal justice drug treatment. Criminal Justice and Behavior, 34, 1179-1187. doi:10.1177/0093854807304350

24.

Simpson

D. D.

Knight

Broome

K. M.

(1997). TCU/CJ forms manual: TCU drug screen and initial assessment. Fort Worth: Institute of Behavioral Research, Texas Christian University.

25.

Simpson

D. D.

Knight

Hiller

M. L.

(1997). TCU/DCJTC forms manual: Intake and during-treatment assessments. Fort Worth: Institute of Behavioral Research, Texas Christian University.

26.

Urbán

Kun

Farkas

Paksi

Kökönyei

Unoka

. . . Demetrovics

(2014). Bifactor structural model of symptom checklists: SCL-90-R and Brief Symptom Inventory (BSI) in a non-clinical community sample. Psychiatry Research, 216, 146-154.

27.

Ventura

Nuechterlein

K. H.

Subotnik

K. L.

Gutkind

Gilbert

E. A.

(2000). Symptom dimensions in recent-onset schizophrenia and mania: A principal components analysis of the 24-item Brief Psychiatric Rating Scale. Psychiatry Research, 97, 129-135.

28.

Welsh

W. N.

(2006). Evaluation of drug treatment programs at the State Correctional Institution (SCI) at Chester: A partnership between the Pennsylvania Department of Corrections, Gaudenzia Inc., and Temple University (NIJ No. 2002-RT-BX-1002). Washington, DC: National Institute of Justice.

29.

Welsh

W. N.

(2010). Inmate responses to prison-based drug treatment: A repeated measures analysis. Drug and Alcohol Dependence, 109, 37-44. doi:10.1016/j.drugalcdep.2009.11.024

30.

Welsh

W. N.

Lin

H.-J.

Peters

R. H.

Stahler

G. J.

Lehman

W. E. K.

Stein

L. A. R.

. . . Frisman

L. K.

(2015). Effects of a strategy to improve offender assessment practices: Staff perceptions of implementation outcomes. Drug and Alcohol Dependence, 152, 230-238.

31.

Welsh

W. N.

McGrain

P. N.

(2008). Predictors of therapeutic engagement in prison-based drug treatment. Drug and Alcohol Dependence, 96, 271-280.

32.

Welsh

W. N.

Zajac

Bucklen

K. B.

(2014). For whom does prison-based drug treatment work? Results from a randomized experiment. Journal of Experimental Criminology, 10, 151-177. doi:10.1007/s11292-013-9194-z

33.

Winerman

(2013). Interesting results: Can they be replicated? Monitor on Psychology, 44, 38-41.