Abstract
Allen and Pardo’s explanation of Relative Plausibility as a theory of evidence and proof in litigation is ambiguous and underspecified. Their account suggests at least three different interpretations of what they mean. They might be advocating “anti-halfism,” which tracks the “conventional account” but merely rejects >0.5 as the proper standard of proof. Or they might be advocating “probabilistic holism,” in which trial decision-makers apply probability to whole claims but not elements – in which case it remains to be explained how such an approach is internally coherent. Or they might be endorsing “total anti-probabilism,” in which “plausibility” obeys rules and axioms different from those of probability – rules and axioms that Allen and Pardo have yet to identify. To date, Allen and Pardo have side-stepped criticisms by shifting from one interpretation to another, strategically. Aside from presenting a theory too formless to determine how well it fits actual jury behavior, Allen and Pardo have not presented any robust empirical observations about how juries actually decide cases (despite their claims to do so). Before we can really assess whether Relative Plausibility is a new paradigm for understanding the structure of evidence and proof in litigation, Allen and Pardo must tell us much more about what it actually is.
We are grateful to Ron Allen and Mike Pardo, not only for including us in this symposium of responses to their defence of their Relative Plausibility theory, but for engaging so closely with our article, ‘The conjunction problem and the logic of jury findings’ (Schwartz and Sober, 2017: 619). They are highly critical of it, but in academia as elsewhere, the only thing worse than being talked about is not being talked about. In our article, we discussed Relative Plausibility in a limited way, to point out that it does not avoid the conjunction problem. By treating us as full-fledged critics of Relative Plausibility, Allen and Pardo have exceeded the scope of our argument, but we welcome the invitation to engage in a more wide-ranging critique of their theory.
Allen and Pardo claim that Relative Plausibility is a new paradigm for understanding the structure of evidence and proof in litigation, one that displaces what they call the ‘conventional account’. The conventional account, as they see it, is an approach that employs probability theory and adopts a standard of proof in civil cases expressed as ‘more probable than not’ or >0.5 on the standard 0 to 1 probability scale. Allen and Pardo’s account of Relative Plausibility has three elements: a purported refutation of probability theory as inapplicable to trial evidence, a description of Relative Plausibility, and an ambitious empirical claim that Relative Plausibility better fits the observable evidence about jury behaviours. We don’t have the space here to mount an extended defence of probability theory, but we note that even if Allen and Pardo could demonstrate that probability theory is incoherent as applied to trial evidence questions, it does not follow that Relative Plausibility is itself a coherent alternative.
Allen and Pardo’s explanation of Relative Plausibility is ambiguous and underspecified, suggesting at least three different interpretations of what they might mean. The ambiguity of their account allows them to side-step certain criticisms by shifting from one interpretation to another. In this essay, we will try to pin them down. As for their empirical argument, that too is unpersuasive. Their theory is at present too formless to allow us to determine how well it fits actual jury behaviour, and in any case Allen and Pardo have not presented the robust empirical observations they sometimes claim.
A. The conjunction problem
Allen and Pardo’s attack on the coherence of probability theory as applied to trial fact-finding relies heavily on the conjunction problem. In our article, we argued that the conjunction problem does not seriously undermine the standard probabilistic account of proof at trial and therefore doesn’t provide a good reason for rejecting that account in favour of an alternative, such as Relative Plausibility. The conjunction problem is not an inherent problem within the probabilistic account of proof, but is created by a particular interpretation of jury instructions that says that a plaintiff wins whenever she proves each element of her claim is more probable than not, interpreted as Pr(X)>0.5, where X is the plaintiff’s claim and Pr(X) + Pr(not-X) = 1. The conjunction problem occurs when the plaintiff meets the ‘each element >0.5’ condition but fails to meet the intuitive and legally mandated ‘whole claim >0.5’ condition due to the multiplication rule for conjunctions. (Cases where the plaintiff meets neither condition or both conditions are unproblematic.) But, we argued, no practical problem arises unless there are many cases falling into this ‘probability gap’, as we called it (Schwartz and Sober, 2017: 626).
While evidence theorists have long assumed that there must be numerous cases in the probability gap, we argued that there are two reasons to believe that the number of such cases is too small to present a serious concern to evidence theory or practice. First, because most elements of most claims are probabilistically dependent, the correct multiplication rule requires use of conditional probabilities. This would reduce the number of cases in the probability gap, since it is likely that most elements have high conditional probabilities. Second, jury instructions are linguistically consistent with a procedure in which juries are asked to determine whether the probability of the whole claim >0.5; and only then do they check to make sure that each element exceeds the >0.5 threshold, as an imperfect but useful double-check (an ‘entailment check’ in our article) that the whole claim indeed exceeds 0.5. That approach harmonises the >0.5 conditions for whole claims and individual elements, since the probability of the whole claim cannot exceed the probability of any conjunct element (Schwartz and Sober, 2017: 670–673).
Allen and Pardo’s criticism of our account sheds no new light on the matter, and a detailed response would be a mere rehash. Two points about Allen and Pardo’s criticism are curious, however. First, their own ‘holistic’ account of jury decision-making posits that juries decide whole claims and then check to make sure that each element of the claim is ‘included’ or ‘instantiated’ (their words) (Allen and Pardo, 2019: 16, 31 fn. 216, 35). Though attacking our account vehemently, they fail to demonstrate how their holistic version differs. They remain vague about how juries can determine ‘inclusion’ without viewing the whole claim as a conjunction subject to the multiplication rule.
Second, Allen and Pardo’s current article implies that the conjunction problem is our own hobby horse, attributing to us ‘the apparent belief that the “conjunction problem” is “the most serious of the [proof] paradoxes, posing the greatest challenge to probability theory”’ (Allen and Pardo, 2019: 31–32). But that’s not our belief—we argued the opposite. It is Allen and Pardo who think this, once again listing ‘avoids the conjunction problem’ as one of the top reasons to prefer Relative Plausibility over conventional probability theory (Allen and Pardo, 2019: 18; see also Schwartz and Sober, 2017: 624, n 6).
B. The theoretical account
Allen and Pardo’s account of Relative Plausibility says that juries compare degrees of things like ‘coherence’, ‘consistency’ and ‘consilience’ in the plaintiffs’ and defendants’ ‘stories’ and choose the more plausible of the two, but little more (Allen and Pardo, 2019: 7, fn 7; Allen and Stein, 2013: 568; see also Pardo and Allen, 2008: 223, 229–30). They leave key points undeveloped or ambiguous, such that we can’t even be sure whether their account requires a rejection of probability theory. Allen and Pardo’s arguments are amenable to at least three possible interpretations, which we list in terms of their increasing hostility to a probability model of legal fact-finding.
1. Anti-halfism
At times, Allen and Pardo seem to accept probability theory, but merely reject >0.5 as the proper standard of proof. Here, as before, they demonstrate Relative Plausibility with the ‘stylized example’ in which the plaintiff’s case has a probability of 0.4 and the defendant’s 0.2, saying that the plaintiff deserves to win because ‘the plaintiff’s version is twice as likely to be true as the alternative’ (Allen and Pardo, 2019: 14, 18; Pardo, 2013: 592–593). ‘Twice as likely’ is clearly a probability statement, suggesting that their concept of ‘plausibility’ just means probability, but with no requirement that the more plausible case has a probability >0.5.
If this is the core of Relative Plausibility, its distinction from the conventional account depends upon the existence of a non-trivial number of cases like their ‘0.4 to 0.2’ example—cases, that is, in which the jury self-consciously chooses between X and Y while being aware that there are possibilities beyond X and Y that collectively have a non-trivial probability. That’s because the ‘more plausible’ story will always have a probability >0.5 in cases where the parties’ competing stories fill the entire probability space. Moreover, unless the probability of unpresented stories leaves a non-trivial residue of unknowns, such that Pr(X) + Pr(Y) << 1, Relative Plausibility fails to distinguish itself from the >0.5 standard.
The viability of Relative Plausibility as anti-halfism thus depends on an empirical claim about the existence of cases with large unknown probabilities. To date, Allen and Pardo have offered only plausible speculation on this point, relying instead on intuitions and aphorisms such as ‘The probability space of conventional probability is filled with events the probability of which collectively sum to one. Knowledge of that sort simply is not part of the human condition’ (Allen and Pardo, 2019: 35). That sweeping statement is demonstrably false. Wholly apart from games of chance, which are structured to have probability spaces summing to one and are nevertheless subjects of human knowledge, there are many practical problems that can be cast in terms of probabilities summing to one. Statements like ‘It will it rain tomorrow’ are easily accommodated in a structure where Pr(X) + Pr(not-X) = 1.
More importantly, numerous litigated questions are naturally presented as requiring findings of X or not-X, where a defendant does not have to specify a particular story of not-X, and the probability space consequently sums to 1. As we pointed out in our article, defeating the causation element in various types of cases does not always require specifying ‘more plausible’ alternative causes (Schwartz and Sober, 2017: 651–653). Allen and Pardo cavalierly dismiss this counterexample as ‘idiosyncratic’, along with whole swathes of litigation like ‘toxic tort cases’ (Allen and Pardo, 2019: 34). But even in cases where we would expect a specified counter-narrative from a defendant, where there are theoretically many different possible narratives of how things could have come about, the case might still be constructed so as to occupy the entire probability space. The parties may implicitly agree or expressly stipulate that the plaintiff’s and defendant’s narratives are the only possibilities. The court may instruct the jury not to speculate about unpresented hypotheses of liability or non-liability. Or the jury may make the probability space sum to 1, either by disregarding the existence of unpresented versions of events, or by speculating to fill probability gaps on their own. It is possible, certainly, that those things don’t happen. But the anti-halfism version of Relative Plausibility depends on jurors deciding plausibilities while consciously refusing to view the probability space as fully occupied by the parties’ presented cases. As discussed below, Allen and Pardo have not shown that that is what actually happens.
2. Probabilistic holism
Allen and Pardo’s account at times seems to endorse a kind of ‘probabilistic holism’. Their caginess about how juries determine the ‘inclusion’ or ‘instantiation’ of elements while deciding claims holistically creates an ambiguity. They might mean that jurors do not, or are not expected to, posit specific numerical probabilities for each element so long as they can say that each element has a probability greater than the probability of the whole. If so, then Allen and Pardo are probabilists who pragmatically recognise the cognitive limits of jurors. This would make them no different from most other probabilists in the context of trial decision-making.
But there is a crucial difference between saying that jurors reason probabilistically about conjuncts only to a limited extent and saying that they do so not at all. If that is Allen and Pardo’s claim—if they mean to argue that jurors can decide the probabilities of a whole but can’t say anything at all about the probabilities of elements—then we fail to see the logic. Claims and elements do not have different natures from the perspective of probability theory: even elements can be cast as conjunctions. Probability theory isn’t an appliance that can be switched on and off to suit one’s convenience.
3. Total anti-probabilism
Finally, Allen and Pardo might be rejecting probability theory entirely. In places, they suggest that ‘plausibility’ cannot be described as a probability because it obeys different rules. This version of Relative Plausibility is suggested when they say, for example, that ‘Schwartz & Sober seem to assume that one can assign numbers representing the “probability” of events occurring, and that, without more, such statements are meaningful. They are not.’ (Allen and Pardo, 2019: 33) Let’s remove the ‘without more’ straw-man from this statement: no serious theorist claims that probability numbers can be pulled out of thin air, without evidence or a reasoning process, or reference to any degrees of certainty or belief, and ‘assigned’ to facts. Let’s also hold aside their 0.4/0.2 example and their sometime assertion that the multiplication rule is a ‘an aspect of the real world’ of litigation, both of which tend to undermine their sometime anti-probabilism (Allen and Pardo, 2019: 36 n 198).
Allen and Pardo need to provide a more thorough explanation of what they mean by ‘plausibility’ if their concept is intended somehow to resist probabilistic description. Telling us that more ‘plausible’ cases are superior in terms of ‘coherence’, ‘consistency’ and ‘consilience’ fails to explain how ‘plausibility’ differs from probability, since those qualities could well make some cases more probable than others. How, for example, do we compare the strength of the plaintiff’s and defendant’s cases without saying that one is more probably true than the other? What are the rules that ‘plausibility reasoning’ needs to follow? Can the conjunction X and Y be more plausible than X and more plausible than Y? Without knowing that, we can’t even tell whether Relative Plausibility avoids the conjunction problem.
Allen and Pardo’s attack on the validity or utility of probability theory is no substitute for answering these questions, and the attack itself is not without problems. For example, their contention that numerical probability estimates are meaningless fails on two counts. First, numbers or point values are not meaningless, but are useful expressions to estimate and compare things that exist by degrees. Consider the volume control on a big screen TV that goes from 0 to 70: the numbers may be somewhat arbitrarily assigned but are not ‘meaningless’. We know that 60 is louder, indeed ‘much’ louder than 20, and the numbers are no more subjective or less meaningful than saying that the plaintiff’s case is ‘very consilient’. Second, probability reasoning does not always require that point values be assigned to hypotheses in any event.
Allen and Pardo also assert that probability reasoning is doomed from the start because it must involve either a frequency interpretation of probability or a subjective-degrees-of-belief interpretation. You can’t interpret legal evidence in terms of frequencies, they say, and subjective degrees of belief are too subjective. Allen and Pardo neglect to consider other interpretations of probability, of which there are many. They also don’t consider the possibility that probability doesn’t need to be interpreted in terms of something else. The axioms of probability define probability, and when probability is used in science, it often stands on its own as a theoretical concept. Geneticists and physicists make intelligible use of probability in their theories without having a reductive account of what probability statements mean.
We are not trying to rule out the possibility of a viable eclectic theory. But Allen and Pardo’s arguments seem to have been chosen more for their ability in isolation to rebut some aspect of ‘the conventional account’, than for their consistency with one another as part of a system. This places their critics in the awkward position of playing a game of intellectual whack-a-mole, rebutting an argument here only to have one pop up randomly there, on a different logical basis.
C. The empirical evidence
Allen and Pardo rely heavily on their contention that Relative Plausibility closely fits actual jury behaviour. As seen above, they offer an underdeveloped model, telling us that juries compare the two cases and pick the more plausible, without making clear which of the three different versions of Relative Plausibility they think describes juror behaviour: anti-halfism, probabilistic holism or total anti-probabilism. It would also help to know more about how Allen and Pardo think jurors use such concepts as ‘consilience’ etc. These shortcomings aren’t fixed by the very limited additional empiricism Allen and Pardo offer: piecing together arguable inferences from story model research, trial lawyer behaviour and jury instructions.
Story model research offers little if any empirical support for the claimed superiority of Relative Plausibility as a behavioural description. The conclusion that juries prefer or require coherent narratives as the basis for their decisions is consistent with the notion that jurors consider the probability of a story, that jurors consider probabilities of elements of the story, and that they apply a >0.5 standard in doing so. Therefore, the story model proves neither that juries reject probability nor that they embrace pure holism. On the other hand, story model research suggests that jurors are uncomfortable with large information gaps, and will tend to fill them in by speculation in order to tell a complete story (see Allen and Stein, 2013: 567–569). Rather than supporting, this seems to undermine Relative Plausibility’s necessary assumption that juries decide between known probabilities summing to much less than one. Story model research seems at best agnostic with respect to Allen and Pardo’s dispute with the ‘conventional account’.
Allen and Pardo lean heavily on the intuition that juries do not assign numerical probabilities to elements of a claim and then multiply them to generate a probability for the whole claim. We can agree with this intuition while disagreeing with Allen and Pardo’s inferential leap from it, that probability is simply not consistent with jury reasoning. Even though jurors may not walk around with Kolmogorov’s axioms in their heads, this doesn’t mean they don’t reason probabilistically at all. Some aspects of basic probability are intuitive to many people. Most people understand the ‘odds’ involved in a coin toss, and understand that a weather forecast putting the chance of rain at 80% means that they would be well-advised to carry an umbrella or raincoat even though rain is not a certainty. Most probably also understand that an 80% chance that it will rain means a 20% chance that it will not. It strikes us as overconfident for Allen and Pardo to simply infer that jurors can’t or won’t apply a basic probability framework when instructed.
Allen and Pardo believe they have an empirical ace-in-the-hole in their sweeping empirical assertion that trial lawyers always try to cast their cases as persuasive stories. That assertion is plausible, but the question is not what trial lawyers do, but what juries do. Allen and Pardo treat these two as the same: because each party offers a single* ‘explanation’, juries must naturally be basing their decision on single* explanations. (We use the asterisk* to reflect Allen and Pardo’s sometimes question-begging insistence that the single explanation may be disjunctive.) From here, they simply assume – again, based on mere intuition – that juries will choose the more plausible story even if it is not probable. The plaintiff who proves her case to a 0.4 probability wins the verdict over the defendant whose case is proven to a 0.2 probability. Bam—relative plausibility! But as we have seen, juries can recast such a case into one that occupies the entire probability space, by disregarding unpresented hypotheses (either on their own or under instruction from the court) or by speculating to fill evidentiary gaps. In Allen and Pardo’s example, the jury might simply speculate about that additional 0.4 residual unpresented probability to put the plaintiff over the top at >0.5.
Allen and Pardo also point to jury instructions as support for their empirical claim that jury decision making tracks Relative Plausibility. Their argument on this point is particularly hard to follow given the lack of clarity about which of the three versions of Relative Plausibility they are advancing. While a small number of jurisdictions expressly ask juries to compare the relative strength of plaintiffs’ and defendants’ cases, most speak in terms of plaintiffs proving their claims to be ‘more probable than not’, a formulation entirely consistent with the standard probabilistic approach (Schwartz and Sober, 2017: 673–685). Allen and Pardo respond that most jury instructions (or special verdict forms) replicate the conjunction problem and claim that this causes the whole probability edifice to collapse, thereby preventing juries from deciding cases probabilistically. But even if Allen and Pardo are right about jury instructions—and our article finds that they are not—there is no strong reason to assume that jurors work out the purportedly conjunction-paradoxical aspect of their instructions by defaulting to a relative-plausibility mode of decision. Indeed, Allen and Pardo insist both that jurors decide claims holistically and that jury instructions place primary emphasis on element-by-element decision making. It is curious that Allen and Pardo seem to argue that jurors both obey and disobey the same aspect of their instructions at the same time.
Allen and Pardo’s empirical claims about jury decision making remain speculative. They offer little information about how jurors actually reason, and they provide no persuasive grounds to think that jurors decide cases without reference to probability. They offer no empirical evidence suggesting that jurors take account of unknowns and thereby consciously decide between cases whose probabilities sum to substantially less than 1.0. The conventional account assumes that jurors make probability statements such as ‘the plaintiff’s case is probably true’. We don’t know what kind of ‘plausibility’ statements, if any, Allen and Pardo think jurors make that rule out probabilism. And because we don’t know from Allen and Pardo exactly how plausibility differs from probability, it’s hard to know what kinds of juror statements we would look for to confirm or deny that they are ‘relative plausibilists’.
D. The normative claim
In the short space allotted to this response, we have focused on Allen and Pardo’s descriptive claims. While Allen and Pardo seem somewhat ambivalent on this point (Allen and Pardo, 2019: 8, 11, 25, 27), their argument plainly has normative implications. However, the normative aspects of Relative Plausibility are, like the descriptive aspects, underdeveloped. To say that the plaintiff should win in the 0.4 to 0.2 scenario, Allen and Pardo assert the normative superiority of their comparative standard to one in which the plaintiff must attain a ‘probably true’ threshold of 0.5. Why is that normatively preferable? Moreover, as argued above, to distinguish itself from the >0.5 standard, Relative Plausibility entails the belief that juries decide plausibilities while remaining conscious of unpresented explanations with substantial probabilities (the 0.4 residuum in their 0.4 to 0.2 example). Why should a jury be asked to do this?
These questions are probably answerable, but what Allen and Pardo have said so far is mostly that it is unfair to require plaintiffs to disprove unpresented explanations (Allen and Pardo, 2019: 18–19). The implication is that the current system unduly favours defendants. We are not unsympathetic to that intuition, but much more needs to be said about it. For starters, unpresented explanations might favour plaintiffs too. And not all plaintiffs are alike in terms of the pre-litigation distribution of evidence and their ability to gather it in litigation to meet a burden of proof. Without a deeper investigation into these matters, the normative aspects of Relative Plausibility, which would alter the conventional understanding of the burden of proof, remain open questions.
Conclusion
The real business of assessing and critiquing Relative Plausibility awaits a more robust and unambiguous description of what it actually is. Echoing Mark Twain, we think that reports of the death of probability theory and its paradigm-shifting replacement by Relative Plausibility are greatly exaggerated, at least in part because it remains uncertain exactly what we would be shifting to.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.