Structured decisions about Dutch probation service interventions

Introduction

Decisions of probation officers can have a great impact on the lives of offenders and on the safety of the public. One of the tasks of the Dutch probation service is to advise the Public Prosecution Service, the courts and the prison system about ways to minimize the chances of recidivism by offenders and to promote their reintegration in society. In order to be able to provide good advice the Dutch probation service has introduced instruments that can assist probation officers to make a decision. The Dutch probation service is not alone in this. The use of assessment tools is becoming a standard for many probation services in Europe, the United States and Canada (Bonta, 2002; Hanson and Morton-Bourgon, 2007). More and more organizations in the criminal justice system use forms of structured prediction to support their decision making. For example, forensic hospitals use instruments for risk assessment, the Dutch Public Prosecution Services use a system (‘BOS Polaris’) ¹ to assist in the formulation of sentencing demands and an instrument (‘BooG’; Kordelaar, 2002) has been developed at the Netherlands Institute of Forensic Psychiatry and Psychology (NIFP) to enable structured decisions to be made about whether or not to produce a report on the mental faculties of an offender.

What instruments of this kind have in common is that a conclusion can be reached by answering a standard set of questions, sometimes supported by a weighting of the answers and decision rules to generate advice. Instruments for offender risk assessment have evolved over time. Initially, actuarial instruments were developed, containing mainly static risk factors such as age or prior convictions. Although these instruments can validly predict recidivism, they lack information about dynamic criminogenic needs, such as drug abuse or unemployment, which are also relevant for intervention planning (Bonta, 2002). Therefore instruments have been developed that contain both static and dynamic risk factors. Some of these instruments produce an automatic conclusion on the risk of recidivism and are thus actuarial instruments, for example the Offender Assessment System (OASys; Home Office, 2002). Other instruments only provide a structure by prescribing which risk factors should be assessed, after which the professional draws the conclusion. These systems thus structure but do not replace professional decisions, for example the Historic Clinical Risk - 20 (HCR-20; Webster et al., 1997) for assessing the risk of violent recidivism.

This article examines the assumption that more structure leads to higher quality decisions, and discusses how this relates to the expertise of the probation officer. It describes that structuring the decision making process in probation so far has been focused on risk and needs assessment, but that intervention planning is still purely a matter of professional decisions. This leads to the conclusion that further structuring of the decision making process in the probation service is indeed desirable.

Structured decisions are better than unstructured decisions

There is convincing evidence that structuring decision making leads to better results. As long ago as 1954, Meehl published a study comparing structured (mechanical/statistical) predictions and clinical predictions (professional judgement without instruments). The predictions concerned diverse topics, such as probation outcome, academic success and the outcome of electroshock therapy in schizophrenia. After comparing 20 studies Meehl concluded that the statistical (formal) approach produced predictions that were as good as and often better than those produced by the clinical (informal) approach (Meehl, 1954). A more recent meta-analysis of 136 studies of predictions in the field of psychology and medicine was carried out by Grove and colleagues (Grove et al., 2000). They compared statistical (actuarial) and clinical predictions on a wide variety of topics such as criminal recidivism, treatment outcome, suicide attempt, personality characteristics or probation outcome. It was found that statistical predictions were better in 47 per cent of the studies and clinical predictions in 6 per cent of the studies. In the remainder of the studies the two sets of predictions were equally good. It may therefore be concluded that, in general, a structured (actuarial) prediction is to be preferred over an unstructured (clinical) prediction. Although the actuarial decisions were based on a fixed set of variables while clinicians often had extra information from a file or an interview, having extra information was not found to influence the correctness of the prediction (Dawes et al., 1989; Grove et al., 2000). Most relevant to our topic, there is also convincing evidence that structured predictions are better than unstructured professional judgements of the risk of recidivism (Andrews and Bonta, 2006; Bonta, 2002; Hanson and Morton-Bourgon, 2007). Both actuarial risk assessment and structured professional assessments show better predictive validity than clinical predictions of the risk of recidivism. The plausible conclusion is that structuring the decision making process can improve the outcome.

Why would an assessment carried out with the help of an instrument be better than an unaided professional assessment? It might reasonably be expected that professionals’ knowledge and experience would enable them to form a good judgement. That they often do not reach the best conclusion is explained by the fact that professionals, like anyone else, can make numerous errors of judgement. The quantity of information that people are capable of processing and analysing is limited. In order to be able to make decisions in complex situations such as the probation setting, people use rules of thumb (know as heuristics) that can lead to biases (Galanter and Patel, 2005; Garb, 1998; Van Schie, 2003). A number of these biases has been identified through experimental research (Arkes, 1991). For example, when a hypothesis is formed about a specific issue, people (including experts) tend to seek, use and remember information that confirms their hypothesis (the so-called ‘confirmation bias’). In judging the likelihood of a situation, people tend to look for examples of similar situations in their memory. The number of examples they can think of, and the ease with which they remember them, are considered as an indication of the likelihood (the so-called ‘availability heuristic’). The availability of examples, however, can be biased because extreme or vivid occurrences are better remembered. Using a structured decision aid helps avoid these biases, for example by asking professionals to consider counter-examples and by making base rates of occurrences available (Garb, 2005).

Assessors quite often differ from one another in their opinions. This can have various causes. For example, they may not (consistently) use the same information or reasoning, they do not assess the information in the same way, they apply different criteria as to what is an acceptable conclusion or they integrate information in different ways. Many probation officers believe that structuring the decision making process is worthwhile, albeit mainly for beginners. Once officers are experienced they should, it is argued, be able to make do without instruments. This is probably not the case. Indeed, a study by Tazelaar and Snijders (2004) actually shows that experience can result in poorer decisions. In some experiments they found that experts performed worse in purchasing decisions with growing experience, and that combining expert judgement with formal models improved the predictive accuracy of purchasing predictions. Garb (1998) has shown that assessments of personality and psychopathology of experienced psychologists are better than those of lay people but often not better than those of recently graduated psychology students. A possible explanation for this is the lack of feedback. As professionals generally get little feedback on the correctness of their decisions, they cannot learn from their mistakes (Dawes et al., 1989). As a result, they can be lulled into thinking that their own professional judgements are good. Studies in the field of medical decision making have shown that experts are better at managing information and filtering relevant and irrelevant information. They also rely more on their clinical experience than on basic scientific principles. Non-experts are less efficient in selecting relevant information, but more likely to rely on the available scientific evidence (Galanter and Patel, 2005). Both strategies have advantages and can lead to mistakes. Because decision strategies of experts are not perfect, structured decision making can help them improve their decisions, or at least help them make decisions with explicit and comprehensive justifications (Witteman and Kunst, 1999).

Structured approaches do not replace the professional

An often voiced objection to the greater use of structured decision making is that this would eliminate the professional. However, such beliefs are mistaken. Most instruments for structured risk assessment require a high degree of professional skill. It means, for example, that a standard set of items must be processed, and in each case the severity of the problem must be gauged by a professional using a set of rating instructions. Excellent communication skills, a thorough knowledge of the target group and their problems and strong analytical powers are essential for this purpose. Professional input will always remain requisite.

Even when an instrument contains decision making rules and automatically generates conclusions, this does not eliminate the professional. Instruments and decision making rules always have to be interpreted. In probation, there may be specific offender characteristics or circumstances which necessitate a qualified decision. A professional who uses the instrument must therefore critically assess the conclusions based on the decision making rules and, if necessary, make a reasoned different decision. A structured system of working helps professionals apply relevant scientific knowledge, not to overlook anything and to view their own opinions critically. But ultimately the decision is made not by the instrument but by the professional who is responsible and accountable for it (possibly in consultation with others). It follows that introducing a decision-support system does not mean that professionals can sit back and take things easy. Instead, they should be guided but not led by the instrument and use their professional expertise in a responsible manner, taking account of the latest scientific knowledge in their field.

Structured support of offender assessment decisions by the Dutch probation service

Since late 2004 the Dutch probation service has used an instrument for structured risk assessment called RISc (Adviesbureau Van Montfoort en Reclassering Nederland, 2004). RISc stands for ‘Recidive Inschattings Schalen’ (recidivism assessment scales). It is a form of structured support for decision making, based on the English and Welsh offender assessment system (OASys). Weighting a standard set of static and dynamic items results in a conclusion about the risk of reoffending and about the criminogenic needs (this part can be called actuarial). Additionally, the risk of serious harm to others and to oneself, and the responsivity to interventions are assessed by the professional. The psychometric qualities of RISc are fairly good. The inter-rater reliability (the extent to which different probation officers come to the same conclusion when assessing the same offender) was moderate to substantial for most items (Cohens K (for nominal items) and Tinsley and Weiss’s T (for ordinal items) varied between .30 and .87 with most items having values between .41 en .79). In general, a value of K and T in the range of .41 to .60 is considered moderate and in the range of .61 to .80 substantial (Landis and Koch, 1977). The inter-rater reliability of the total score is substantial (T = .68; Van der Knaap et al., 2010). Often the so-called Area Under the Curve (AUC) is used for measuring the predictive validity of tools for risk assessment. As a rule of thumb, AUC’s of .70 and higher denote satisfactory predictive validity, and AUC’s between .60 and .70 low to moderate predictive accuracy (Brennan et al., 2009). The predictive validity for general recidivism of RISc is moderate (AUC = 0,70; Van der Knaap and Alberda, 2009).

RISc is based on the premise of the ‘What Works’ theoretical framework, that is research that focuses on ‘What Works’ in reducing recidivism. The basic premise is that the greater the risk, the more intensive must be the interventions. This is why RISc focuses on assessing the risks of recidivism and harm. A second principle is that interventions should be aimed at the criminogenic needs that underlie these risks. Exerting a positive influence on these needs will reduce the risks. RISc therefore requires an assessment of the needs related to the offending behaviour of a specific offender. A third principle is that interventions should reflect the responsivity of an offender. In other words, the chance that an intervention will be effective can be increased by taking the possibilities and limitations of an offender into account. More in-depth diagnosis, such as an examination to identify psychiatric problems, can be used to supplement the standard set of questions on which the basic diagnosis is founded. Conclusions about risks of recidivism and harm, about criminogenic needs and responsivity provide the basis for the decision on the advisability of one or more behavioural interventions to resolve the identified problems and reduce the risks.

Before the introduction of RISc, the Dutch probation service did not systematically assess the risk of recidivism, and probation officers did not use an assessment tool. Non-structured decisions were the base for pre-sentence reports and supervision plans. Implementing RISc not only introduced structured decision making, but it also meant a change in the basic views on probation work by introducing the risk-needs-responsivity approach. To enable them to work with RISc, probation officers get four days of training in risk assessment. RISc is now used by the probation service to write pre-sentence reports, to advise the prison service on sentence plans for prisoners and on the conditional release of prisoners, and to make a supervision plan. Approximately two-thirds of the Dutch probation officers work with RISc.

On the basis of conclusions about risk, criminogenic factors and responsivity, a decision is made on what interventions are necessary to influence the behaviour and circumstances of an offender in such a way that the risks of recidivism and harm are reduced. A decision is also made on what interventions and/or criminal justice settings are necessary to mitigate risks in the short term, especially the risk of harm. Previously in RISc the decision about the behavioural interventions was a matter for professional judgement. The format in which the intervention decision was structured, was limited to text fields that had to be completed and tables from which a choice could be made. Although this functioned to some extent as a checklist of the aspects that must be dealt with, it provided few guidelines for the content of the decision. Internal research by the Dutch probation service has shown that the quality of intervention decisions leaves something to be desired. Probation officers have different ways of arriving at a decision. Furthermore, the planned interventions did not always follow logically from the preceding problem analysis. For some criminogenic needs no intervention was planned, or interventions were planned for non-criminogenic needs, without a proper explanation (Bosker and Ruijter, 2006). This leads to the question whether this part of the decision making process can be improved. In view of the convincing evidence that structured decisions produce better results, structuring the intervention planning might be the answer.

Research about structured intervention planning

Although a lot of research is available on clinical or structured prediction and assessment, relatively little research has been done into intervention decisions. As far as we know, no research at all has been done on this part of the decision process in the field of probation. Some research has been carried out on intervention decisions by psychologists and physicians. This research shows that different professionals often decide differently about what intervention would be best in a specific case (Garb, 1998, 2005). Professional agreement may be reasonably good in a given institution or clinic, but this is often not the case between different institutions. A way to establish general agreement in a specific profession is to develop practice guidelines, based on evidence-based practice. In the field of medicine and psychiatry working with practice guidelines is customary. Although practitioners report that these guidelines are helpful, little is known about their effect on the quality of intervention decisions (Galanter and Patel, 2005). There is some evidence that professionals frequently deviate from guidelines (Garb, 2005; Merkx et al., 2006). Some studies show that decision support systems can improve practitioners’ performance on intervention decisions, but further research is needed to support this conclusion (Galanter and Patel, 2005). Witteman and Kunst (1999,) for example, evaluated the use of a computer system to support psychotherapists in their decisions about the treatment of depression. The system helped practitioners order relevant decision elements and criticize their initial decisions by giving possible contra-indications. Practitioners evaluated the system as useful. In the field of medicine, experiments with computer based decision support systems show that this could enhance clinical performance for treatment. In a systematic review of 100 controlled trials assessing the effects of the use of decision support systems in health care, Garg and colleagues found that practitioners’ performance improved in 64 per cent of the studies (Garg et al., 2005). Practitioners’ performance was for example measured by medication use and identification of at-risk behaviors. The effect of structured decision support on patient outcomes has been studied insufficiently and, when studied, showed inconsistent results (Garg et al., 2005; Hunt et al., 1998).

The main conclusion about the question whether structured decision support can improve intervention decisions is that more research is needed, certainly in the field of probation. Nevertheless, some positive results can be reported on structuring intervention planning in other fields. Based on these results we expect that positive results can be achieved for probation also, but the way decision support systems are implemented in the professional organisation and introduced to the professional is an important aspect that needs attention to facilitate acceptance and actual use (Galanter & Patel, 2005; Shook & Sarri, 2007).

Conclusion: Structured intervention planning in probation is desirable

In recent years the work of the probation service has become increasingly structured and influenced by research. This structuring has mainly concerned the identification of the criminogenic needs and the assessment of the risk of recidivism. The body of evidence about the advantage of actuarial (structured, formal) predictions over clinical predictions justifies this development. Decisions on intervention planning following the assessment are still mainly a matter for individual professional judgement. Both experience in Dutch probation practice and research in the fields of psychology and health care show that improvement of this part of the decision process is necessary and possible. It is argued that it is therefore now time for the next step: structuring the intervention decision. This is, after all, the vital decision. The probation service does not identify risks and needs simply for the sake of it. The crucial question is what must be done in order to reduce the identified risks and needs. The answer must be evidence-based and may not differ too much between different probation officers. As with structured risk assessment, structured intervention planning would not replace the professional. Final decisions must be made by probation officers, taking into account all specific information about the case. A decision support system could however improve the quality of the decision process and thereby presumably of the decisions.

Recently, a decision support system that compels probation officers to decide on all relevant steps and issues of intervention planning has been integrated in RISc. Where possible, interventions are suggested that meet the assessed criminogenic needs. For some parts of the decision process, guidelines are developed and added to the system. But in all decisions, probation officers have the last word and can overrule the suggestions if they have good reasons to do so. Further research will show whether this system will produce the desired improvement in the quality of intervention decisions.