Burning the straw men: Intellectual integrity in the early intervention debate

Analytic competence and intellectual integrity

Although they do not explicitly identify factual errors versus unbalanced conclusions, Mihalopoulos et al.’s (2012) analysis demonstrates that what they believe to be error arises only from their poor understanding of the research. They suggest that Amos (2012a) is in error to note a change in the parameter of transition to psychosis used to construct the model of Valmaggia et al. (2009). They say that an estimate of transition to psychosis by different authors in different countries does not invalidate the estimate used in the model. This demonstrates that they are unaware that the original estimate is based upon work by Yung et al. (2003, 2004), and that these same authors later changed their estimate (Yung et al., 2008). They have concluded that Amos (2012a) is in error reflecting perhaps their poor understanding of the published literature.

Similarly, Mihalopoulos et al. (2012) contradict Goldberg et al. (2006), who conclude that ‘time series analysis of our data did not confirm that the reduction in costs was clearly the result of the introduction of the [EI] service’ (p. 902). Mihalopoulos et al. (2012) suggest that the latter authors, despite not finding that EI freed up beds, were arguing that ‘beds freed by [EI] patients are available to be alternatively used’ (p. 809). Nowhere does Amos (2012a) claim that these authors did not include behavioural measures, and refers directly to such measures in Table 1.

Some of Mihalopoulos et al.’s (2012) points may be rooted in personal sensitivity to impersonal criticism (p. 809): ‘Amos also describes the retrieval of patient files in the Mihalopoulos et al. study as ‘ad hoc’. This is an unfairly disparaging description and one designed to seed doubt in readers’ minds about the rigour of this study.’

An ad hoc measure is one where the method was ‘made for purpose’ after the original design, such as file retrieval 5 to 8 years after the initial intervention. Ad hoc measures are a potential source of bias. Again, Mihalopoulos et al. (2012) appear to experience the identification of possible bias as an unfair attack, even when accurate.

Factual inaccuracies

Mihalopoulos et al. (2012) make the serious accusation of factual inaccuracies (p. 808):

Amos begins his review by claiming that the expansion of EIP services in Australia is responsible for the contraction of other services, specifically general practitioner managed psychological interventions. No evidence for this assertion, which has been made previously by others, is provided.

The authors state I have not provided evidence for an assertion I have not made. Moreover, they do not identify which others have previously made this assertion. The relevant passage in Amos (2012a: 720) reads:

Recent events in Australia demonstrate the competition for resources within mental health. Despite resistance from the medical profession, funding of GP managed psychological interventions has been significantly reduced at the same time as large financial resources have been steered towards EI.

This juxtaposes two events occurring at the same time with different outcomes for different groups with no suggestion of a causal relationship. The statement is accurate, unbiased, and relevant, as it identifies a set of resource-limited systems in which winners and losers are determined by economic considerations.

Again introducing a bias that does not exist in the original, Mihalopoulos et al. (2012) propose (p. 808):

…because some critics in response to the reforms underway in Australia have questioned the evidence showing that better functional outcomes are achieved by specialised EIP services, Amos asserts that early psychosis researchers have tacitly accepted these critiques and sought to buttress the case with cost-effectiveness data.

Compare this with my statement (Amos, 2012a: 719): Faced with ongoing scepticism about clinical outcomes, proponents have argued that EI programmes should be implemented to save mental health resources to be used in other areas (McGorry, 2011).

I have not argued that EI researchers have accepted the criticisms of others. I have also provided a reference to the arguments of a strong EI advocate to illustrate my meaning. Mihalopoulos et al. (2012) appear to commit the error they accuse me of.

Basic accounting principles

Mihalopoulos et al. (2012) do not acknowledge the accounting points in the review. In a public mental health system where clinician resources are constrained by caseloads and paid for at the level of individual workers, cost attribution based upon individual instances of care introduces substantial distortion.

Imagine two services, A and B, each managing 100 outpatients with caseloads of 10:1 and 25:1. On average, service A patients consume 10% of their clinician’s contact time, service B patients 4%. Service A requires 10 clinicians, B only 4. Assuming a base wage of $100,000, the annual cost of clinician care per patient in service A and B is $10,000 and $4000, respectively. The yearly wage bill of service A is $1,000,000; of service B it is $400,000. These costs do not change with variation in number of contacts. All the reviewed literature calculate unit costs, as described by Mihalopoulos et al. (2012); it is because they calculate costs in this way that their estimates are distorted. If average service A and B patients each have 10 1-hour appointments per year, the unit-cost approach calculates the cost to the services to be identical. However, costs remain $6,000 higher for the average patient in service A regardless of the number of contacts.

The authors fail to acknowledge a related potential distortion in McCrone et al. (2010). Amos (2012a) notes that the provision of an out-of-hours service is likely to increase costs. While Mihalopoulos et al. (2012) insist that the paper included all costs, the failure to adequately describe how they did this does not allow the review to come to the same conclusion. A unit-cost approach to an out-of-hours service is particularly prone to distortion because it is likely that much of the time is spent without patient contact. If a weekend clinician spends an 8-hour day without contact, none of the cost of providing the service is measured by the unit-cost approach.

It is curious that Mihalopoulos et al. (2012) identify a lack of attention to the concept of opportunity cost as a weakness of Amos (2012a). Of course, the reason that it is not discussed in the review is because the concept is universally absent from the reviewed literature. A review of a different set of articles in which one or more papers considered opportunity cost or in any way tried to measure or analyse these concepts might have usefully included them. Nevertheless, much criticism of EI refers to opportunity cost in non-technical language. Castle (2012), for example, questions why resources are sequestered for particular populations when the same techniques can be shown to be effective for all patients.

Mihalopoulos et al. (2012) do not consider cited evidence that attempts to reduce total costs by reducing hospitalisation have failed. Although they acknowledge that short-term cost savings are unlikely from EI, they also state (p. 809):

… a bed not used for an EIP patient will be available to be used for another patient or purpose, with benefits associated with such utilisation. If the bed is not used, the variable costs associated with that bed are released as financial cost savings. In the long-term all resources are variable and there can be planned contraction of service use.

This is fine as theory. However, Amos (2012a) refers to evidence that past attempts to achieve cost savings by reducing hospitalisation rates, motivated by such theories, have led to increased total costs. Again, Mihalopoulos et al. (2012) do not provide evidence for their assertions.

Minor points

Mihalopoulos et al. (2012) find it unreasonable to detect the possibility of bias in Mihalopoulos et al. (2009). This is a historical case-control study, with large attrition differences, where most of the observed effect is due to a subgroup, and the outcomes are grossly different from a comparison study with more rigorous design (Bertelsen et al., 2008) as well as the rest of the literature. The study is an ad-hoc addition to a pre-existing experiment (eg it was not part of the original design). It is therefore reasonable to conclude that the study involves multiple potential sources of bias.

Discussing Valmaggia et al. (2009), Mihalopoulos et al. (2012) misinterpret the statement that subsequent research by the team that justified an initial parameter had ‘invalidated the assumptions’ of the model as ‘invalidating the model’ (p. 810). They also do not acknowledge that I had in fact praised the model for its sensitivity analysis, which allowed it to survive invalidated assumptions, by comparison with Serretti et al. (2009), where the absence of a sensitivity analysis meant that the model was invalidated along with its assumptions.

When I propose that the evidence for the long-term efficacy of EI is not compelling, they accuse me of asserting that ‘EIP services do not work over the longer term’ (p. 810). The latter is in reference to a reviewed article which itself speculates that the long-term impact of EI ‘may not be substantial’ (McCrone et al., 2010: 382). Where I note that the use of patient report measures ‘reduc[ed] the accuracy of their cost estimates’, they report this as ‘Amos argues that McCrone et al. did not accurately assess the costs associated with EIP services’ (p. 809). This clearly alters the meaning, and to proffer it as evidence of bias appears to contradict McCrone et al. (2010) themselves, who list patient report as a limitation of their study (p. 381).

It should be clear that Mihalopoulos et al. (2012) have attempted to create the impression of a lack of balance in my review by removing the qualified, measured language I use and substituting dogmatic, absolute statements whose accuracy varies from misleading to mistaken.

Concessions

Mihalopoulos et al. (2012) do identify one error: Phillips et al. (2009) should be identified as CMA rather than CEA in Table 3 (Amos, 2012a). It is also true that I was unable to recruit a second reviewer. I approached many people, including the Clinical Director of the EI service in my district. The director indicated that, interesting as the paper would be, given the ongoing tender process for funding for EI projects, it might be discreet to leave the name of the EI service off the paper. The current circumstances might support that director’s wisdom.

Conclusions

In sum, then, Mihalopoulos et al. (2012) suggest that Amos (2012a) is unbalanced because it effectively identifies multiple sources of bias in the articles it reviews, suggests sources of error in the calculation of outpatient costs, and expresses doubt that savings can be realised in practice, based on cited evidence. However, the authors fail to cite or analyse any evidence for the proposition that EI costs less than treatment as usual, and argue that identifying possible biases in their research is akin to accusing them of professional misconduct. Consequently, Hegelstad et al. (2012) is presented as an example of the results of this approach to research.