The relationship between NHS performance indicators and suicide rates

Introduction

Suicides are of considerable social and medical significance, are emotive, cause distress to relatives and professionals and may lead to litigation. ¹ The risk of suicide is at its highest during in-patient psychiatric care and the immediate post-discharge period.^2,3 Mental health services provided for those at risk of suicide may be important in suicide prevention and are considered a priority for health services in England. ⁴ The suicide prevention strategy requires improvement in the health service as a whole, ⁵ and many such measures leading to the prevention of suicide have been described in the national suicide prevention strategy for England, the national strategy, and the cross-party consultations on the national strategy.^6–8 Additionally, a national data from the National Confidential Enquiry into suicide and Homicide ⁹ and data on adverse events from ‘An Organisation with Memory' ¹⁰ could allow improvement in suicide prevention. Suicide rates have been considered an outcome measure of mental health strategy, mental health service provision and mental health quality assurance. ¹¹

The government’s performance ratings for all NHS trusts in England and Wales were introduced in 2003. ¹² The performance ratings incorporate key targets that cover governmental priority areas and a broader set of improvement indicators. Mental health trusts were judged on 30 targets encompassing the domains of risk management, clinical audit, research and education, patient involvement, information management, staff involvement and education, training and development.

Specific areas of relevance to suicide addressed for mental health trusts included: (1) access to crisis; (2) Care Programme Approach; (3) Child Adolescent Mental Health Services (CAMHS); (4) delay transfer of care; (5) patient experience and (6) drug misuses identify in effect. This system was modified in 2005, but the key determinants were similar. The performance ratings are categorised into two broad categories (the quality of service and the use of resources) and each is measured on a four-point scale (excellent, good, fair and weak).

As mental health services have an important role in prevention of suicides, the relationship between performance ratings of mental health trusts and the local suicide rates were examined with the null hypothesis that there would be no relationship between them.

Method

A total of 60 mental health trusts were identified from 20 strategic health authorities (SHAs) and a list of primary care trusts (PCTs) in England for the year 2009. SHAs and PCTs were contacted by phone and email to enable confirmation of the geographical area covered by each mental health trust. Data on the performance ratings for the quality of service and use of resources for each mental health trust for the year 2009 were ascertained from the Care Quality Commission (CQC) website ¹³ (www.chi.nhs.uk & www.doh.gov.uk/clinicalgovernance/index.htm).

Data on suicide rates for men and women aged 16–65 for the district covered by each mental health trust for the year 2009 were ascertained from the database of the Office of National Statistics (http://www.nchod.nhs.uk). Suicide rates for local areas were closely matched with the local mental health trusts. When one mental health trust covered more then one geographical district, data from the district with the lowest suicide rate was used for analysis in order to allow for conservative estimate of any relationship.

The relationship between performance ratings targets and the local suicide rates was examined using (1) the Kruskal-Wallis one-way analysis of variance and (2) the Mann-Whitney U test because the data were not normally distributed. The Kruskal-Wallis test was used to examine differences within the group. The Mann-Whitney U test was used to compare suicide rates between individual pairs of performance ratings. Analyses were conducted for both performance indicators: the quality of service and the use of resources.

Results

Complete data sets were available for 60 mental health trusts. Initially, the Kruskal-Wallis test was used to compare the suicide rates between trusts with different quality ratings. The figures reported are the median and inter-quartile suicide rate in each quality group. Also presented are p values indicating the significance of the difference between the four quality groups. The results are summarised in Table 1.

OPEN IN VIEWER

Table 1.

Suicide rate in each quality group.

Gender	Weak median (IQR)	Fair median (IQR)	Good median (IQR)	Excellent median (IQR)	p Value
Male	8.1 (8.1, 10.0)	7.1 (6.3, 7.6)	8.7 (7.5, 10.1)	7.4 (5.5, 8.2)	0.04
Female	2.7 (2.6, 3.1)	1.9 (1.7, 2.7)	2.4 (2.1, 2.7)	2.6 (2.0, 3.0)	0.16
Total	5.5 (5.3, 6.5)	4.4 (4.0, 4.9)	5.8 (4.9, 6.5)	4.8 (4.0, 6.5)	0.07

The results indicated that there was some evidence of a significant difference in suicide rates for males among the four groups. However, for both figures there was no clear trend in the data, with the weak and good groups having the highest rates and the fair and excellent groups having the lowest rates.

The Mann-Whitney test with a Bonferroni adjustment was used to compare between pairs of groups for the two outcomes where there was a suggestion of a difference between groups. The p values from these analyses are summarised in Table 2. There were no significant differences in suicide rates between different pairs of groups.

OPEN IN VIEWER

Table 2.

Comparison between pairs of groups.

Comparison	Males Adjusted p value	Total Adjusted p value
Weak vs. Fair	1.00	0.68
Weak vs. Good	1.00	1.00
Weak vs. Excellent	1.00	1.00
Fair vs. Good	0.11	0.08
Fair vs. Excellent	1.00	1.00
Good vs. Excellent	0.13	0.76

The final set of analyses used the Kruskal-Wallis test to compare the outcomes between trusts with different finance ratings. A summary of the analysis results is given in Table 3. There were no significant differences between the four groups.

OPEN IN VIEWER

Table 3.

Comparison between trusts with different financial ratings.

Gender	Fair median (IQR)	Good median (IQR)	Excellent median (IQR)	p Value
Male	9.3 (7.7, 11.2)	7.7 (6.45, 9.5)	8.2 (6.3, 9.3)	0.41
Female	2.5 (1.5, 2.6)	2.4 (1.7, 2.8)	2.5 (2.1, 2.8)	0.38
Total	6.0 (4.7, 6.9)	4.8 (4.2, 5.8)	5.1 (4.1, 6.6)	0.41

Suicide rate by quality of service according to 4 main indicators for males and females are represented in Figure 1. OPEN IN VIEWER

Suicide rate by use of resources for males and females regarding 3 main indicators are represented in Figure 2. OPEN IN VIEWER

Discussion

There were no significant differences in suicide rates in men, women and both combined across the four performance ratings for the quality of service and the use of resources. Similarly, there were no significant differences in suicide rates in men, women and both combined between pairs of ratings for the quality of service and the use of resources.

Before discussing the results, some of the methodological difficulties require careful consideration. A four-point scale may not be sensitive to changes in the performance of mental health trusts. Data on the validity of performance ratings by the CQC is not available. There is also the possibility that errors occurred during the process of matching geographical areas for suicide rates with the areas covered by the mental health trusts. At the time of the study, there were important changes in the structure of the NHS and PCTs, although all SHAs and PCTs were contacted on the phone and by email in order to provide accurate and updated local data. Some mental health trusts covered several geographical districts with data on suicide rates. Any bias due to this was minimised by using the lowest suicide rate of the several districts covered by a single mental health trust. Additionally, only pure suicide rates were considered; this may be an underestimate as accidental deaths include concealed suicides. However, this is the only study that has examined the relationship between CQC performance indicators and suicide rates at a geographical level. Hence, there is no comparative data.

The negative findings may be explained by the above methodological issues. However, the possibility that the results are genuine requires consideration. First, the performance ratings may not be a valid measure for the quality of service and the use of resources. Data on the validity of the performance indicators are absent. Second, performance ratings are based on data provided by the mental health trusts to CQC, usually without inspection, and raise concerns about their accuracy. Third, the study null hypothesis assumed the possibility of a causal link between performance indicators and suicides, but the specific characteristics measured by the performance ratings may not be of direct relevance to suicides. The performance indicators may not be accurately measuring the variables that have been measured. Additionally, a number of suicide prevention strategies described in the national suicide prevention strategy are not measured by these performance indicators, for example the use of collapsible curtain rails on inpatient psychiatric wards. Fourth, other local characteristics such as socio-demographic factors, social deprivation and the prevalence of psychiatric morbidity may be more important than performance ratings in the genesis of suicides.^14,15

This study was confined to England and Wales and thus the findings cannot be generalised to the rest of the United Kingdom or to other countries. Nevertheless, the negative findings suggest that the performance ratings of mental health trusts do not influence local suicide rates. However, if a measure could be developed to quantify the local implementation of the revised national suicide prevention strategy, then the impact of that measure on suicide rates could be evaluated.