Abstract
The interrater reliability of an internationally renowned crime linkage system—the Violent Crime Linkage Analysis System (ViCLAS)—was tested. Police officers (N = 10) were presented with a case file and asked to complete a ViCLAS booklet. The level of occurrence agreement between each officer was calculated. Results showed a 30.77% level of agreement across the 106 variables examined. Agreement ranged from 2.36% for weapon variables to 62.87% for administration variables. Only 11 (10.38%) of the variables reached an acceptable level of agreement. Concerns pertaining to the validity of inferences produced using ViCLAS data are discussed, along with potential explanations for the findings, limitations of the study, and future research directions.
Keywords
The task of determining whether the same offender has committed multiple crimes is important to both the effectiveness and efficiency of police investigations (Grubin, Kelly, & Brunsdon, 2001). The ability to perform this linking task accurately often depends on the availability of trace or biological evidence (e.g., fingerprints, DNA). In the absence of such evidence, other evidence such as behavioral information must be relied on (e.g., verbal exchanges between offender and victim; Woodhams, Hollin, & Bull, 2007). To obtain the behavioral information required to link crimes, law enforcement agencies in North America and much of Europe use computerized linkage systems that are populated with information extracted from case files (Bennell, Snook, MacDonald, House, & Taylor, 2012).
Currently, the most widely used linkage system is the Violent Crime Linkage Analysis System (ViCLAS). A system developed by the Royal Canadian Mounted Police (RCMP; Collins, Johnson, Choy, Davidson, & MacKay, 1998), ViCLAS is used within Canada and is reportedly being used in Australia, Austria, Belgium, England, France, Germany, the Netherlands, New Zealand, Switzerland, and two U.S. states (i.e., Indiana and Tennessee; RCMP, n.d.). Although ViCLAS has existed for nearly 20 years, there is no published account of its effectiveness being evaluated systematically. In the current article, we test one of the key assumptions underlying ViCLAS, namely, that data included in the system may be coded reliably (see Bennell et al., 2012). Before examining the issue of reliability, we outline the origins of ViCLAS and describe some of the ways that the system is (and potentially may be) used.
Origins and Use of Viclas
The origins of computerized crime linkage systems, including ViCLAS, can be traced back to the FBI’s Violent Criminal Apprehension Program (Collins et al., 1998), a system developed to avoid “linkage blindness,” which is the lack of communication between law enforcement agencies across jurisdictional boundaries that can inhibit the apprehension of serial offenders (Clark, 2002; Egger, 1984). The RCMP subsequently developed their own crime linkage system known as the Major Crimes File (RCMP, n.d.), which was later replaced by ViCLAS (Clark, 2002). It has been argued that the use of ViCLAS has become widespread since its beginnings in the 1990s and that it has gained a reputation as the best crime linkage system available (Bijleveld & Smit, 2006; Collins et al., 1998).
Although there are variations in how ViCLAS is used within and between countries, the procedure in North America generally involves five steps (RCMP, n.d.). First, an officer enters data collected from his or her investigation of a ViCLAS-appropriate crime (e.g., abduction, sexual assault, homicide) into a 38-page booklet or e-booklet. It is important to note that training is not required before officers are allowed to complete ViCLAS booklets. Second, completed booklets are forwarded to a quality control center to be reviewed (Friesen, 2004). Third, data are entered into the database of previously solved and unsolved cases. Fourth, the data are compared against the ViCLAS database by analysts who are trained to search for potential links. Fifth, once the search for linked crimes is complete, relevant investigators are informed about any potential links that have been identified. These investigators then attempt to confirm or eliminate the potential link through further investigation.
In addition to assisting with investigations, ViCLAS has the potential to be used in at least two other ways. One additional use of ViCLAS relates to data storage and management, often for the purpose of conducting research (e.g., on serial homicide behavior—Bijleveld & Smit, 2006; rape typologies—McCabe & Wauchope, 2005; criminal profiling—Kocsis, Cooksey, & Irwin, 2002; and child care providers who commit sexual offenses—Moulden, Firestone, & Wexler, 2007). A second use stems from ViCLAS’s potential to be used as the basis for similar fact evidence in court cases where questions arise about whether or not a defendant is responsible for multiple crimes (Ormerod & Sturman, 2005). On several occasions, law enforcement officers have provided expert testimony about the likelihood of a series of crimes having been committed by the same offender (e.g., Labuschagne, 2006; State v. Code, 1994; State v. Pennell, 1989; State v. Prince, 1992). Crime linkage systems, such as ViCLAS, may be used to support such testimony (State v. Fortin, 2004).
A Preliminary Test of the Reliability of Viclas
For the data contained in ViCLAS to be of value for the aforementioned purposes, the assumptions on which it is based must be valid. Perhaps one of the most fundamental assumptions underlying ViCLAS (and all other all linkage systems) is that the database contains reliable data (for a review of the central assumptions underlying ViCLAS, see Bennell et al., 2012). The primary type of reliability of concern is interrater reliability. A test of interrater reliability involves determining the extent to which two (or more) different investigators enter the same information about a case into a ViCLAS coding booklet (e.g., both investigators agree that the victim was a nurse). In scientific research, a minimum level of 80% agreement is typically deemed acceptable before the inferences and conclusions drawn from coded data are trusted (e.g., Hartmann, 1977). It seems logical that a similarly high level of agreement should be demanded from crime linkage systems because of the consequential nature of the inferences drawn from the data held in these systems.
To date, the only test of the reliability of ViCLAS is Martineau and Corey’s (2008) study. 1 They provided 237 police officers with a vignette of either a sexual assault or homicide case and asked them to complete a ViCLAS booklet (see Martineau and Corey’s Appendices A and B for the information provided to participants). The participants were also given the ViCLAS field investigator’s guide—a resource that explains each of the questions in the booklet—to assist them with the task. Once the test was completed, Martineau and Corey calculated three measures of interrater reliability, namely, overall percentage agreement, percentage occurrence agreement, and percentage nonoccurrence agreement. In terms of overall percentage agreement, they reported a rate of 88% for the sexual assault case and 79% for the homicide case. Based largely on these results, they concluded that the data contained within ViCLAS were likely to be reliable.
However, in this case, overall percentage agreement inflates the actual agreement between individuals because it is influenced largely by the high levels of nonoccurrence agreement between investigators (i.e., instances where both investigators did not record the occurrence of a variable). Consider the example where investigators have five options available to them in the ViCLAS booklet when coding blunt force trauma. If we assume for a moment that the investigators disagree on the type of trauma that was present in a case, they would agree on three of the options (the nonoccurrences) and disagree on two of the options (the occurrences). Thus, they would exhibit a 60% agreement rate despite not agreeing on the behavior being coded (i.e., actual type of trauma). Although it is useful that investigators agreed on what did not occur in a particular case (e.g., that the blunt force was not minimal), it is more important from an operational standpoint that investigators agree on what actually happened (e.g., that the blunt force was extreme). In coding situations where multiple options cannot logically be selected simultaneously, such that there is inevitable agreement on what did not occur, the most appropriate measure of agreement is percentage occurrence agreement. Percentage occurrence agreement captures the number of instances that two coders indicated that a particular piece of information was present in a case file (Hartmann, 1977).
When percentage occurrence agreement values were calculated, Martineau and Corey’s (2008) reliability scores were less impressive. Specifically, they reported an overall percentage occurrence agreement of 38% for the homicide case and 25% for the sexual assault case. For the homicide case, occurrence agreement was approximately 4% for crime scene information, 9% for offense information, 13% for offender information, 23% for information pertaining to administration questions, 27% for deceased victim information (a subsection of the offense section), and 32% for information about the victim. Two of the sections (vehicle and weapon) exceeded 80% agreement, which Hartmann (1977) recommended as a desired benchmark. Similarly, for the sexual assault case, occurrence agreement was approximately 5% for information about the biological sample, 10% for the scene information, 13% for both offense and offender information, 18% for victim information, and 25% for the administration questions. The weapon section exceeded the 80% level of agreement.
The low percentage occurrence agreement values reported by Martineau and Corey (2008) demonstrate that officers almost always disagreed with each other about what occurred in certain aspects of the case. Although they are from only a single study, their findings suggest the opposite of what they concluded. Rather than supporting the reliability of ViCLAS data, the results actually indicate that the many sections of ViCLAS data are likely to be unreliable and raise concerns about the validity of the inferences derived from using these data. That being said, there are several limitations associated with this study that make it inappropriate to draw any strong conclusions about the operational value of ViCLAS.
One of the most obvious limitations is that Martineau and Corey (2008) did not use actual case files in their study, preferring instead to rely on short vignettes. Thus, their results were based on less detailed information than would normally be available in more naturalistic settings where officers have access to larger amounts of investigative material. A second limitation is that their participants consisted of a relatively heterogeneous group of police officers who may have differed from each other in important ways (e.g., whether they serve in a role that requires the completion of ViCLAS booklets).
The Current Study
The goal of the current study is to conduct an independent assessment of the interrater reliability of ViCLAS data in a manner similar to Martineau and Corey (2008) while also addressing some of the methodological limitations inherent in that previous study. We examine the level of percentage occurrence agreement associated with a wide range of ViCLAS variables using actual case file materials that are longer and more detailed than the previously used vignettes. We also sample a more homogeneous group of police officers by testing officers who investigate ViCLAS-appropriate crimes and are in a position to complete ViCLAS booklets. These changes to Martineau and Corey’s design will allow us to determine the degree of interrater reliability associated with ViCLAS data in a more ecologically valid fashion.
Method
Participants
Participants were 10 police officers (7 men and 3 women) working as crime investigators in a Canadian police organization. They completed the study voluntarily and did not receive any compensation. Of the participants, 1 reported being in the age range of 26 to 30 years, 1 in the 31 to 35 range, 3 in the 36 to 40 range, and 4 in the 41 to 45 range, and 1 reported being older than 45 years of age. Also, 4 participants indicated that they had between 5 and 10 years of policing experience, and 6 indicated that they had more than 10 years of experience.
In response to the open-ended question about their experiences with ViCLAS, nine participants reported having completed ViCLAS booklets. Three participants reported having spent less than 1 year of service completing booklets as part of their job, and the remaining seven indicated spending between 2 and 8 years of service completing booklets as part of their job. Seven participants indicated that they did not complete any booklets in the past year, two reported completing between one to two booklets per week in the past year, and one reported completing four booklets in the past year. In addition, two participants reported spending less than 1 hr per week completing booklets in the last year, one reported dedicating between 2 and 3 hr per week on the task, and seven stated that they did not dedicate any hours to the task in the past year. Six participants reported not having any previous training on how to complete ViCLAS booklets, and the other four indicated receiving an introduction to the task during their cadet training.
Participants were also asked six questions about the field investigator’s guide. Only one participant reported awareness that the guide existed, one participant reported using the guide previously, none of the participants indicated using the guide routinely, and only one participant reported receiving training on how to use the guide. When asked to rate their familiarity with the field investigator’s guide on a 10-point scale, where 1 = not at all familiar and 10 = extremely familiar, the average familiarity score was 1.11 (SD = 0.33). None of the participants indicated being aware of any successful links based on ViCLAS booklets they completed.
Materials
A genuine case file from a solved criminal investigation was used as the stimulus for the current study. The file contained an anonymous 29-page victim interview transcript and a 3-page case report from a case involving forcible confinement and sexual assault. The case report contained details of the accused (i.e., name, gender, date of birth, height, weight, previous offenses) and a summary of the criminal events. The information included in the transcript and case report provided details necessary for completing Version 4.0 of the ViCLAS booklet (e.g., descriptors and behaviors of the victim and offender, aspects of the crime scene). On a 5-point realism scale, where 1 = not realistic and 5 = very realistic, participants’ average rating of the realism of the experimental material was 4.00 (SD = 1.25).
The 38-page ViCLAS booklet contains 156 variables. The variables are subdivided into the following eight categories: administration variables (n = 11) pertaining to information regarding the police organization, victim variables (n = 23) such as the victim’s height and build, offender variables (n = 40) such as the offender’s height and build, vehicle variables (n = 16) pertaining to aspects of any vehicles involved in the crime, crime scene variables (n = 9) pertaining to details of the crime scene such as location, offense variables (n = 48) such as the nature of the offense (e.g., sexual), weapon variables (n = 5), and summary variables (n = 4) such as an open-ended question requiring them to write out a summary of the incident and space to provide any additional information pertinent to the offense.
Procedure
Each participant was tested individually at the participating police organization’s headquarters. All participants were presented with both an informed consent form and a short verbal presentation on the purpose of the study and their role as a participant. The participants were informed that the goal of the study was to measure the data entry accuracy of police officers. Participants were then given the case file to read at their desired pace and were asked to complete a ViCLAS booklet as they would normally do following one of their investigations. The participants were also told to be as accurate as possible when completing the booklet. After completing the booklet, each participant was asked to complete an eight-item demographic questionnaire. On average, the participants took approximately 45 min to read the case file and 70 min to complete the ViCLAS booklet. To prevent the integrity of the study from being compromised by having debriefed participants communicate about the study with future participants, a debriefing form outlining the true purpose of the study (i.e., reliability in coding) was emailed to the participants after all testing was completed.
Interrater Reliability Calculations
Percentage occurrence agreement was calculated using the method outlined by Hartmann (1977). For each variable, all possible pairwise comparisons between participants were first established. The total number of instances where two participants in a pair agreed that a specific variable was present in the case file was then tallied. This value was then divided by the total number of pairs where at least one participant in the pair indicated that the variable was present. This proportion was then multiplied by 100 to arrive at a percentage occurrence agreement. This was the same procedure used by Martineau and Corey (2008).
For clarity, consider an example where four police officers provide the following yes or no decisions about the use of a knife in an offense: Officer 1 = yes, Officer 2 = no, Officer 3 = yes, Officer 4 = no. From this example, six pairwise comparisons would emerge: yes–no (Officer 1 vs. Officer 2), yes–yes (Officer 1 vs. Officer 3), yes–no (Officer 1 vs. Officer 4), no–yes (Officer 2 vs. Officer 3), no–no (Officer 2 vs. Officer 4), yes–no (Officer 3 vs. Officer 4). The no–no pairwise comparison would be removed as this would represent a nonoccurrence agreement (i.e., neither of the police officers indicated that a knife was used in the offense). Of the five remaining pairwise comparisons, there is one agreement and four disagreements, thus resulting in a percentage occurrence agreement of 20% (i.e., 1/5).
Percentage occurrence agreement was calculated for 106 of the 156 variables composing the ViCLAS booklet. One of the variables contained in the summary section (i.e., a summary of the case material) was not included in the analysis because the information provided by this variable is redundant (i.e., provided elsewhere throughout the booklet) and is a replication of the materials provided in the experiment. In addition, as this study used percentage occurrence agreement, any time a participant answered “unknown” or left a variable blank, his or her answer was not included in the calculations. There were 49 variables throughout the booklet that were left blank by all 10 participants and were omitted from the analysis.
All of the 106 variables requiring an open-ended response were coded to identify the total number of discrete responses. Each unique response was then treated as a subvariable, subjected to interrater reliability analysis, and collapsed to form the percentage occurrence agreement for that main variable. For example, a response to a question (i.e., main variable) regarding what the offender said to a victim (verbal exchange) that included “I am going to kill you,” “You better listen to me,” and “Keep quiet” would have been treated as three subvariables. How often officers agreed on each of these phrases would be calculated and then combined for a total score for that main variable.
Similarly, every variable that had the option to “check all that apply” was also subjected to multilevel interrater reliability analysis and collapsed to produce a single interreliability score for that main variable. For example, a question pertaining to whether the offender was at risk of being detected at the scene and had three options (e.g., area deserted, potential to see offender, potential to hear or interrupt offender) would have been treated as three subvariables. How often officers agreed on each of these options would be calculated and then combined for a total score for that main variable.
We assessed the reliability of our data entry by having an independent researcher code each of the booklets that was completed by the participants. The independent coder was provided with a 1-hr training session that covered the practical aspects of coding the booklets, the structure and content of the coding guide, and the content dictionary. The reliability of coding was measured using Cohen’s kappa (Cohen, 1960). The kappa value for the administration section was .86, .88 for the victim section, .98 for the offender section, 1.00 for the vehicle section, .98 for the scene section, .87 for the offense section, and 1.00 for both the weapon and summary sections. The average kappa across all eight sections was .95, thus suggesting excellent agreement between the coders (Fleiss, 1981; Landis & Koch, 1977).
All analyses reported below are derived from 106 reliability scores. Across the 106 variables (including all subvariables), there were a total of 8,863 pairwise comparisons. In the current study, an 80% level of agreement was used as the minimum acceptable level of occurrence agreement (Hartmann, 1977).
Results
Across the 106 variables, the average overall percentage occurrence agreement was 30.77% (SD = 30.23, 95% CI = 25.02, 36.53). The levels of percentage occurrence agreement for each of the eight sections of the ViCLAS booklet are shown in Table 1. As can be seen, agreement ranged from a low of 2.36% for weapon variables to a high of 62.87% for administration variables. In addition, the width of the confidence intervals exceeded 20% for the administration, victim, offender, and scene sections and exceeded 10% for the vehicle and offense sections, thereby suggesting some imprecision in our estimates of the true level of reliability for this coding task. There was also variability within seven of the categories, with a range of 5.90% for weapon variables and a range of 100% for victim, offender, and offense variables.
The Total Number of Variables Tested, Mean Percentage Occurrence Agreement, 95% Confidence Interval, Range of Percentage Occurrence Agreement, and Number of Variables Meeting the Acceptable 80% Agreement Criterion
Note. The average of the mean percentage occurrence agreements for the eight sections is 28.71% (95% CI = 14.75, 42.68).
The number of variables in each section that met or exceeded an 80% level of agreement is also shown in Table 1. In total, 11 (10.38%) of the variables met the acceptable level of agreement. For the administration section, only incident type reached the acceptable benchmark. In the victim section, 3 of the 15 variables met the acceptable level of agreement. These 3 variables were the victim’s status (i.e., he or she survived an attack), the victim’s gender, and the victim’s occupation. For the offender section, 4 of the 30 variables met the acceptable level of agreement. These 4 variables were the offender’s gender, hair shade, hair color, and language spoken. Of the 9 variables composing the scene section, 2 had an acceptable level of agreement. These variables were the offender’s risk of being detected at the scene and a description of the scene (i.e., indoors or outdoors). For the offense section, only the variable pertaining to how the offense ended (i.e., release, escape, rescue, or death) reached an acceptable level of agreement. None of the variables for the vehicle, weapon, and summary sections exceeded the level of acceptable agreement.
Discussion
The purpose of this study was to conduct an independent test of interrater reliability associated with ViCLAS—a popular crime linkage analysis system that is used internationally for both investigative and research purposes. As predicted on the basis of previous research (Martineau & Corey, 2008), the level of occurrence agreement found in the current study was, for the vast majority of ViCLAS variables, unacceptably low. Even if one focuses on the upper limits of the confidence intervals, the most optimistic estimates of the levels of reliability, the levels of agreement are still very low (with the exception of the administration section). Given that our study dealt with some of the limitations of Martineau and Corey’s (2008) study and still resulted in low levels of reliability, our findings raise serious concerns about the validity of inferences drawn using ViCLAS data and, potentially, the validity of inferences drawn from other linkage systems in use around the world. Similar concerns exist for researchers who are using ViCLAS data as the primary source of information for their research endeavors. Relying on ViCLAS requires empirical evidence demonstrating the reliability of data contained in the system, evidence that is not yet available.
What about the 11 variables that exceeded acceptable levels of agreement? Although it is a positive sign that certain ViCLAS variables could be coded reliably, close inspection of these variables shows that they may be of little use for making linkage decisions. For example, some of the variables that participants agreed on may be too common (e.g., the offender’s gender) or too subjective or subject to change (e.g., the offender’s hair color). Moreover, many of the variables that one would intuitively think of as useful for establishing crime linkages (e.g., the sexual acts that occurred during the offense, whether or not the victim was targeted by the offender, the type of con or deception used) had very low levels of agreement (all <25%). Understanding why these variables are associated with low levels of interrater reliability may be important for improving the reliability of ViCLAS coding and ultimately the effectiveness of this renowned linkage system.
Potential Explanations for the Low Levels of Interrater Reliability
There are several potential explanations for the low levels of agreement observed here. First, it is possible that participants may have viewed the task as boring and/or unimportant and, thus, put little effort into their coding of the case material. Social psychological research has provided evidence for both motivational and performance decreases when individuals view tasks as mundane and inconsequential (Bartis, Szymanski, & Harkins, 1988; Healy, Kole, Buck-Gengler, & Bourne, 2004; Sheppard, 1993). If boredom with the coding task, or its perceived lack of importance, is causing motivation and/or performance decreases, then it will be important to ensure that this is not the case for coders using ViCLAS booklets in naturalistic settings. The ViCLAS booklet contains a large number of variables and takes time to complete, and officers are often not made aware of linkage successes (as our poststudy questionnaire results suggest). These conditions may erode the extent to which investigators perceive the coding task as important, and as a consequence of less motivation, interrater reliability is likely to suffer. Examining how boredom, or perceived lack of importance, influences the reliability of ViCLAS coding is an important issue to be examined in future research.
Second, the lack of intimacy with the case file information might also explain the low level of agreement among our participants. Of course, participants would be likely to have more knowledge of cases that they have investigated. It is therefore possible that the increased level of intimacy with those cases could result in higher interrater reliability scores. That being said, the participants in this study had the opportunity to become as familiar with the case material as they wished. They were also given the opportunity to make notes, refer back to the file, and work at their own pace. Moreover, the entering of data from case files in some jurisdictions is conducted by individuals with no experience of the case; for example, in the Netherlands the data entry process is centralized, and the investigator does not complete a ViCLAS booklet (Abraham & O’Dwyer, 2011). Even within North America, there are instances where police officers not associated with the investigation are asked to complete the ViCLAS booklets because of resource demands. Nevertheless, the effect of case familiarity on coding reliability is an empirical question that requires testing.
Third, inexperience with ViCLAS booklets may be another explanation for the low levels of interrater reliability. Some of the participants reported having little experience filling out ViCLAS booklets and minimal (or no) training on how to complete them. Potentially compounding this issue is the fact that the participants were not provided with a field investigator’s guide. Taken together, these two issues might have caused the participants to struggle with completing the various sections of the booklet. On the other hand, it should be noted that our findings are likely to have a high degree of external validity because our participants investigate ViCLAS-appropriate crimes and are in a position to complete ViCLAS booklets. The participants also rated the case file as being very similar to the materials typically available to them when completing ViCLAS booklets. Finally, the participants reported that they are unfamiliar with the field investigator’s guide and do not tend to use it when completing ViCLAS booklets, thus validating our decision not to provide it to them during the coding task. Furthermore, the level of reliability exhibited by the participants in Martineau and Corey’s (2008) study (who were given the investigator’s guide) was similar to the reliability exhibited by our participants (who were not given the investigator’s guide). Examining how ViCLAS training, or experience in completing ViCLAS booklets, affects the reliability of ViCLAS coding should be a priority in future research.
Fourth, the nature of the ViCLAS booklet itself may have caused the low levels of interrater reliability. For example, the sheer number of variables (and subsections) contained in the booklet may make it difficult for participants to obtain high levels of reliability—which is what officers contend with in reality. It may be the case that officers also struggle with the ambiguity and interpretation of some of the variables. For example, variables such as “Area was essentially deserted” and “How would you rate the extent of offender/victim negotiation?” are highly subjective variables that are likely to lead to disagreements among officers. In addition, there are a number of seemingly complex and confusing questions that may cause problems for officers. For example, a few of the questions in the ViCLAS booklet require multiple forced-choice answers that provide an increased probability of obtaining disagreements. It is possible that the size and nature of the ViCLAS booklet may be cause for concern, and research should continually be undertaken to determine if and how the ViCLAS booklet can be revised so as to increase its reliability.
Fifth, and perhaps related, is the role that the case material played in the reliability calculations. Regardless of participant knowledge or experience with ViCLAS, providing people with a great deal of crime-related information and then asking them to dissect and transfer that information into a detailed booklet (as done after actual investigations) would likely result in low levels of interrater agreement. Comprehension of the material is a possible concern, as is retention of the information if investigators rely on their memory of the case material (vs. direct referencing) when completing the ViCLAS booklet. 2 It may also be the case that the information contained in our experimental material was particularly vague, convoluted, or complex. The reality, however, is that the case file used in the current study is a genuine case that officers investigated and had to enter into a ViCLAS booklet. Careful attention should nevertheless be paid to these issues when preparing experimental material for future studies.
Limitations of the Research
There are some potential limitations with this study. For example, it could be argued that our small sample size limits our ability to generalize the findings. In response to this, we can simply say that our findings based on the use of 10 participants are consistent with Martineau and Corey’s (2008) findings that were based on more than 200 participants, and our sample reflects well the type of investigators that will complete ViCLAS booklets in naturalistic settings (i.e., experienced investigators working as criminal investigators). Moreover, in reality, finding disagreements between just two officers should be enough to raise concerns about the reliability of ViCLAS data.
Another potential limitation pertains to the reliability of our results, given that we used only one case file (and one crime type). Our results could be the result of something unique about our experimental material. Ideally, researchers would have participants in future studies complete multiple ViCLAS booklets using a range of case files. However, getting officers to complete this task will pose logistical challenges (e.g., with respect to the amount of time required). It is likely this issue is best resolved through a convergence of evidence, where interrater reliability is tested in a series of studies that use a single case file, but these case files would vary systematically (e.g., crime type, amount of information in the case file).
Some may also argue that our findings are not generalizable because of the artificial nature of our testing conditions. We disagree with this argument. If participants are unable to obtain high levels of interrater reliability under highly controlled experimental conditions, where they are not distracted or overburdened, then it is likely that coding ViCLAS booklets in actual police settings would result in even lower levels of reliability. For example, research shows that distractions have a negative impact on performance and accuracy in a range of complex tasks (Banbury & Berry, 1998; Kemker, Stierwalkt, LaPointe, & Heald, 2009; Sanders & Baron, 1975). The crowded offices, background noise, telephone calls, and interruptions from colleagues that are a part of every investigator’s work environment are likely to have a detrimental rather than facilitatory effect on the reliability of genuine ViCLAS coding.
Furthermore, the current study tested the interrater reliability of ViCLAS as it is implemented in one jurisdiction in North America. Future research could examine the interrater reliability of ViCLAS as it is implemented in other jurisdictions and countries. For example, examining whether or not interrater reliability is improved by centralizing that process—as is done in the Netherlands—is an interesting avenue of future research.
Conclusion
Regardless of whether crime linkage systems are used for operational or academic purposes, the ability to use these systems to make valid inferences is dependent on the reliability of the data that are stored in them. Our results, and those of others (Martineau & Corey, 2008), suggest that the data contained in ViCLAS may be unreliable. Continued use of a system that may contain unreliable data is difficult to justify, especially given the many serious consequences attached to linkage decisions. For example, pursuing potential linkages derived from unreliable data can be a waste of valuable police resources and taxpayer dollars and may even result in individuals being improperly considered or even falsely accused of crimes they did not commit. Time spent on ViCLAS-related activities also takes investigators away from other important tasks that may be more pertinent.
Police agencies will have to decide for themselves how much weight to put on our findings and the subsequent conclusions to draw regarding the potential value of linkage systems. As we have argued elsewhere, these systems ought to be evaluated extensively as a matter of urgency because there exists a real risk that current linking efforts are not achieving optimal results. We anticipate that such an argument will be viewed negatively by some and will be met with counterarguments that police organizations do not have the time to wait for research to be conducted as crimes continue to be committed. In response to such anticipated arguments, we simply state that without such research, there is a real risk that linkage systems will never reach their full potential and may cause more harm than good. Instead of viewing our findings in a negative light, we hope our findings act as an impetus (especially for researchers and practitioners who advocate such systems) to find ways to improve the reliability of data contained in linkage systems.
Footnotes
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
