Preventable Hospital Mortality: Learning from Retrospective Case Record Review

Abstract

Summary

Objective

To determine the proportion of hospital deaths associated with preventable problems in care and how they can be reduced.

Design

A two phase before and after evaluation of a hospital mortality reduction programme.

Setting

A district general hospital in Warwickshire, England.

Participants

In Phase 1, 400 patients who died in 2009 at South Warwickshire NHS Foundation Trust had their case notes reviewed. In Phase 2, Trust wide measures were introduced across the whole Trust population to bring about quality improvements.

Main outcome measures

To reduce the crude mortality and in effect the risk adjusted mortality index (RAMI) by 45 in the three years following the start of the programme, from 145 in 2009 to 100 or less in 2012.

Results

In total, 34 (8.5%) patients experienced a problem in their care that contributed to death. The principal problems were lack of senior medical input (24%), poor clinical monitoring or management (24%), diagnostic errors (15%) and infections (15%). In total, 41% (14) of these were judged to have been preventable (3.5% of all deaths). Following the quality improvement programme, crude mortality fell from 1.95% (2009) to 1.56% (2012) while RAMI dropped from 145 (2009) to 87 (2012).

Conclusion

A quality improvement strategy based on good local evidence is effective in improving the quality of care sufficiently to reduce mortality.

Declaration

Competing interest

None declared

Funding

None

Ethical approval

Clinical Governance and R & D approval obtained

Guarantor

Olanrewaju O Sorinola

Contributorship

OS was responsible for the original study idea. All authors contributed to the design of the study and the review forms. OS and RB were responsible for recruiting and training reviewers. CW and RB were responsible for data collection and analysis. All authors contributed to data interpretation. OS drafted the manuscript and all authors contributed to its revision.

Reviewer

Olwyn Westwood

Introduction

Patient safety is a major concern in all healthcare systems. Based on studies across the world, it is estimated that as many as one in 10 patients admitted to hospital in a developed country will be the victim of an unintentional error, and up to 50% of these are preventable.^1–6 The number of preventable deaths due to problems in care in NHS hospitals in England is uncertain,⁷ and wide variations exist. Some of this can be explained by variables such as patients case-mix; however, much remains unexplained and may reflect variation in quality of care.⁸ Variation also reflects different methods of assessment and differences in a key underlying assumption: some studies assume that if a problem in care was preventable, the death was preventable, whereas others assume that a problem in care will not always be a causal or contributory factor, e.g. if a patient is terminally ill.⁹ In a recent study in the Netherlands, based on rigorous case records review, the incidence of preventable deaths was reported to be 4.1% of all hospital deaths¹⁰; a figure that would be consistent with approximately 9000 annual deaths in England.

Our aim was to obtain a more accurate estimate of the number of preventable deaths resulting from problems in care, describe the problems and determine their preventability. It has been recognized that a greater understanding of the nature of adverse events and the processes surrounding them may lead to improved quality of care and patient safety.^2,11–13

We used the crude mortality rate and risk adjusted mortality index (RAMI) as outcome measures, as studies have reported falls in mortality linked to quality improvement programmes that used ratios as outcomes.¹⁴ Measurement itself may encourage improvements in processes of care that ultimately lead to improvements in patient safety.¹⁵ The RAMI provided by CHKS Ltd has been validated and shown to perform well.^16–18 The index excludes obstetrics and neonatal death as well as patients admitted exclusively for palliative care.

Setting

South Warwickshire Foundation NHS Trust (SWFT) is an acute healthcare trust in the Midlands. It has 396 inpatients and day case beds, and treats 170,000 outpatients, 50,000 inpatients and 51,500 emergency attendees every year. In 2009 there were 902 hospital deaths.

Methods

The study was conducted in two phases.

Phase 1: Review of hospital deaths

The study methods were adapted from four previous retrospective case record reviews with comparable endpoints.^6,9,10,14 Vincent et al.⁶ examined the incidence of adverse events in 1014 admissions in two London hospitals in 1999. Hayward et al.⁹ in a US study in 2001 and Zegers et al. 2009¹⁰ study of 3983 deaths in 25 Dutch hospitals focused on preventable deaths, while Wright et al.¹⁴ reported on a British hospital mortality reduction programme. A retrospective case record review of 400 consecutive hospital deaths in 2009 was undertaken to identify gaps in the quality of care. This convenience sample was deemed sufficient to account for the two-stage strategy and to maintain a 95% confidence interval of 2.5% to both sides. As in other studies and to aid comparison, obstetric, psychiatric and paediatric patients as well as patients admitted exclusively for palliative care were excluded.¹⁹

Definitions

For each case, reviewers were initially asked to determine whether there had been any problem in care that had contributed to the patient's death. Problems in care were defined as patient harm resulting from acts of omission (inactions), such as failure to diagnose and treat, or from acts of commission (affirmative actions), such as incorrect treatment or management, or harm as a result of unintended complications of healthcare. This definition was preferred to others such as adverse event, error or patient safety incident^6,9,14,20 because it focuses beyond single discrete incidents to take a wider view of the overall quality of care provided and its contribution to a patient's death. The definition was also more likely to ensure that deaths related to failure to act or omissions were recognized. For each case where a problem in care had been identified, reviewers were then asked to make a judgement as to the preventability of death on a six-point Likert scale. Deaths were categorized as non-preventable if rated 1–3 and preventable if rated 4–6 (Box 1).²¹

Box 1

Scale used to determine preventability of death ²¹

Definitely not preventable

Slight evidence for preventability

Possibly preventable but not very likely, less than 50–50 but close call

Possibly preventable, more than 50–50 but close call

Strong evidence for preventability

Definitely preventable

Preventable and non-preventable deaths

Some problems in care can result from exemplary clinical practice and would not be regarded as preventable (e.g. where there is a known risk of a complication that could lead to death, such as a patient experiencing an intracerebral bleed after appropriate administration of a thrombolytic drug following myocardial infarction). In other cases, patients may have experienced a problem in care but their concurrent illness was so complex or grave such that even if the problem had contributed to their death, the death itself was not judged preventable during that admission.

The review process

The in-depth review focused on the admissions during which death occurred, although reviewers were also asked to identify problems that occurred prior to that admission if these appeared to have contributed to a patient's death. All reviewers underwent one day of training in the review technique to pilot and adapt the review form. Case reviews were undertaken by a first set of reviewers consisting of two recently retired doctors (with extensive experience as generalist and intensivist), a retired nursing sister, a pharmacist and a palliative care nurse. All cases considered to be preventable deaths were blindly reviewed by a second group of reviewers comprising a consultant (specialist), a senior nurse and a clinical audit personnel (all currently practising). In addition, this group double-reviewed a randomly selected 25% of the records to test inter-rater reliability. All necessary reports including coroners, pathology and postmortems were made available and specialist medical advice was available from subspecialists.

Medical Review Form

We adapted a standardized Medical Review Form (MRF) previously used in other reviews.^11,12 This structured MRF contained no identifiable data as each patient was allocated a unique number. Demographic and clinical information was collected including age, sex, admitting specialty (medical, surgical), type of admission (elective, emergency) and co-morbidity. In all cases where a problem in care was adjudged to have contributed to death, reviewers reported on the type of problem, its timing and any contributory factors before making a judgement as to whether the death was preventable. Finally, reviewers recorded information about the completeness and nature of any missing information for each record and rated the overall quality of care.

Analyses

Data were entered onto Microsoft Access database and analysed using STATA (version 11.2) SciencePlus.com Personal software. Summary statistics included proportions, means and medians. For all comparisons of rates, descriptive statistics were used and tests for comparison of proportions in two independent groups corrected for binomial distribution.

Results

Of the 400 cases, 197 were men and 203 were women. Age range was 40–106 with a median of 84 years. There were 1218 co-morbidities: a median of 3 (range 0–8). In total, 384 (96%) patients were admitted as emergencies, 16 (4%) as elective and the majority (83%) were in medical specialties. Thirty-four patients (8.5%) experienced problems in care, of which 76% were medical specialties admission. There were no statistically significant differences (at P < 0.05) in the characteristics of patients who experienced a problem in care and those who did not (n = 366) as regards age, sex, admitting specialty or co-morbidity. Reviewers rated the overall quality of care received by patients to be excellent or good for 297 (81%; 95% CI 78.9–83.2) patients. The proportion was lower for the 34 patients deemed to have experienced a problem in care, 14 (41.2%; 95% CI 36.1–45.8; P < 0·001). Interrater reliability of the reviews key variable, ‘determination of a problem in care’, was good as determined by the Cohen κ coefficient (κ = 0.75).

A wide range of problems were identified in patients who experienced problems in their care, as listed in Table 1. In 14 (41%) patients more than one problem in care was identified. The most frequent related to medical input/supervision 13 (24%), clinical monitoring or management 13 (24%), diagnosis 8 (15%), infections 8 (15%), technical problems 5 (9%) and drugs or fluid management 4 (7%). Descriptions of the different types of problems with examples of each category are provided in Table 2, and contributory factors noted in the cases are listed in Table 3.

Table 1

Types of problems that contributed to patients’ death. More than one option may apply for each patient^*

Type of problem	N	%
Senior medical input/supervision	13	24
Clinical monitoring/management	13	24
Diagnostic problems	8	15
Infections	8	15
Technical problems	5	9
Drugs or fluid related	4	7
Resuscitation	1	2
Others, e.g. discharge planning	2	4

14 (41%) of the patients had more than one problem area

Table 2

Descriptions of problems in care that contributed to patients’ death with examples

Descriptors of problems in care	Examples
Senior medical input/supervision
Failure or delay to get senior/expert opinion	Elderly woman admitted with decreased mobility, fall and possible chest infection. Initially being treated as pneumonia with antibiotics. Subsequently no medical review (senior or junior) for nine days including weekend. Gradual deterioration with increasing Early Warning Score (EWS). Documented request for medical review but still no medical input. Some gaps in medical review related to ward moves and consultant changes with lack of continuity. Last medical entry was three days before patient died of pneumonia
Lack of adequate supervision
Lack of continuity of care
Clinical monitoring/management
Failure to recognize/manage problems with fluids/electrolytes	Male patient admitted with stroke. Clerked on the stroke pathway but no SALT assessment was done. Two days later patient died of aspiration pneumonia
Failure to recognize side-effects of medication	A patient with known asthma was admitted to A & E with fall due to shortness of breath. PaCO₂ was noted to be high (8.4) but was still given high flow oxygen. Serum potassium noted to be 2.5 and intravenous potassium was given. An hour later potassium level was 6.3 and insulin dextrose infusion was started (possible contamination of previous sample, i.e. intravenous fluid site). EWS was 7 and rising. Outreach nurses not contacted nor transferred to ITU. The patient arrested and died
Failure to recognize changes in patient's general condition
Failure to draw up a comprehensive management plan
Failure to take note of observations or check if charts completed properly including EWS
Diagnosis
Failure to take an adequate history	Woman admitted with shortness of breath, cough, sputum and pleuritic chest pain. Had non-union of a fractured ulna at the elbow of about eight-week duration. Focus was on fracture and respiratory tract infection. CTPA was not done until 12 days later, which confirmed pulmonary embolus. Patient collapsed and died the same day of diagnosis
Failure to examine carefully
Failure to consider the full range of possible diagnoses
Failure/delay to act upon results of tests or clinical findings
Infection related
Hospital acquired diarrhoea	Woman admitted with obstructed stoma. Urinary symptoms present on admission and urine dipstick was positive. No action taken regarding the urinary symptoms for six days. She developed urosepis/septicaemia and died eight days post admission
MRSA septicaemia	A patient who had recently had a left arthroplasty for fractured neck of femur was re-admitted within two days of discharge with an infected operation site. The prosthesis was removed and the pus drained cultured MRSA. Postoperatively the patient deteriorated and died
Hospital acquired pneumonia
Urinary tract infections
Technical problem
Sitting of prosthesis, tubes	Elderly woman, admitted after fall at home was found to have anaemia and swallowing problem. Seen by the ENT team and barium swallow advised, which suggested oesophageal malignancy. Upper GI endoscope showed a benign stricture which was dilated, but during the process the tip of the endoscope fell off. This was retrieved but unfortunately the oesophagus was ruptured. This was managed conservatively, however, swallowing problems persisted and patient was fed by percutaneous endoscopic gastrostomy (PEG). Her condition deteriorated over the next few days and died
Post op septicaemia
Post op bleeding
Inadvertent organ damage
Drug or fluid related
Drug/fluid side-effect	Elderly patient with history of arrhythmia admitted with confusion, dehydration and possible dementia. Plan was for hydration, urea/electrolytes bloods sent but excessive dose of digoxin was prescribed and administered despite blood results showing high urea/creatinine levels. She was also given morphine and midazolam in her confused/agitated state. She died of cardiac arrest
Failure to give indicated drug or wrong drug prescribed	Elderly man had made a good recovery from pneumonia and was ready for discharge when his blood urea was found to have risen from 12.0 on admission to 14.9. Intravenous fluids in the form of normal saline were commenced and the patient's regular frusemide discontinued. Two days later the patient was in intractable heart failure and died
Right drug but wrong dose or length of treatment (anticoagulant, intravenous fluids)
Inadequate monitoring

GI, gastrointestine; CTPA, computed tomography pulmonary angiography; ITU, intensive care unit

Table 3

Factors contributing to the problems in care. More than one option may apply

Contributory factors	Number of cases with factors identified as contributing to patient deaths
Patient characteristics
Patient not able to understand/communicate with clinical staff, e.g. dementia or acute confusion, poor English language	4
Co-morbidity	10
Task factors
New, untested or difficult procedure	1
Test results unavailable (not obtained) or difficult to interpret	4
Monitoring of INR	2
Lack of holistic view of patient's problems	1
Individual staff factors
Lack of knowledge of individuals	8
Lack of skill	4
Team factors
Poor teamwork	8
Poor written communication	3
Inadequate handover	6
Work environment
Infection control	3
Staffing issues	1
Failure to deal with falls hazards on the ward	1
Hospital/Trust factors
Poor coordination of overall services	5
Discharge planning	2

INR, international normalized ratio

Preventable deaths

Fourteen deaths (41%; 3.5% of all deaths) were judged to be preventable of the 34 cases found to have a problem in care contributing to death. Only one (7%) of these patients was judged to have received good quality care. In 9 (64%) patients suffering a preventable death, the overall quality of care was rated poor compared with four (8.2%; P < 0·001) in those with no problem in care. While problems occurred in all phases of care, 23 (67.6%) of the problems that contributed to a preventable death occurred during ward care. The rest occurred during or after a procedure, or while awaiting placement.

Phase 2: Interventions/strategies

The top six areas identified from Phase 1 as contributing to patients’ deaths (Table 1) were prioritized to direct the change strategies. All strategies were implemented in 2009/2010 and are discussed next.

Mortality group/surveillance

A mortality group was established by the chief executive to coordinate a strategic approach to reducing hospital mortality using a surveillance system to increase awareness. A mortality scorecard was developed in conjunction with CHKS Limited to monitor Trust performance on a number of mortality indicators. Summaries of data collected from hospital data, death certificates and departmental mortality reports were examined bimonthly by the mortality group which reported to the Trust Board and management committees. Information from the data was used to trigger a more in-depth investigation through case-note review and meetings.

Senior medical input/supervision

Daily structured medical ward (or board) rounds to review patient information, diagnoses and response or deterioration on current management was instituted. Senior supervision of junior staff diagnostic/management decisions, especially out of hours and on weekends, was implemented. Documented care plans from beginning of admission, providing clear directions and timelines, were reviewed daily by the medical team using a holistic problem listing approach. Verbal and written communication of management plans to the ‘on call’ clinical team helped to improve communication about patients’ condition among the team.

Clinical monitoring and management

Minimal delays in transfer of patients who needed more intensive care, reinforcement of protocols for prevention of thrombosis, falls and pressure ulcers was ensured by the ward sisters. All junior medical staffs were tasked with daily recording of patient results in the notes and prompt action on abnormal results, including contacting specialists for advice. The Trust already uses the Early Warning Score (EWS) as part of clinical observation record, but the study revealed miscalculation, inaction and sporadic measurements. A series of training sessions for nursing staff were undertaken to strengthen the use of the EWS and identify when intervention is required.

Diagnostic problems

Senior supervision of diagnostic decisions made by trainees and review of diagnosis based on patients’ condition was reinforced. The internal referral system was changed from a paper-based system to realtime electronic referral thereby providing timely access to specialist opinions. All wards were required to maintain case records in the same filing sequence so that vital medical information or tests results can be easily located and all records transferred with the patient along the care pathway. Telephone protocols for communicating important results from the laboratory to the clinical team were established.

Infection control

The infections identified from the study were mostly, methicillin resistant Staphylococcus (MRSA), Clostridium difficile and hospital acquired pneumonia. An infection prevention team was created comprising a microbiologist, a matron and three infection prevention nurses. The changes introduced by the team included isolation of positive cases, hand hygiene campaigns, improved ward cleaning routines, compulsory training for all staff, a review of the hospital antibiotic guidelines, increased surveillance and feedback of infection rates to the Trust board.

Technical problems

Contributory technical problems identified included a misplaced endotracheal tube, three failed attempts (by a junior doctor) at a nasogastric tube insertion leading to cardiac arrest and an endoscope tip falling off in the oesophagus leading to further surgery. The Trust policy was revised and the following changes were made:

•

Adequate risk assessment before undertaking hazardous procedures with appropriately trained staff for the complexity of the procedure;

•

Adequate senior supervision of junior staff undertaking procedures. Senior staff to undertake reoperations, anticipate complicated procedures or other procedures where juniors have failed;

•

Availability of appropriate equipment (as needed) for emergency procedures or for patients with special needs, such as obese patients, short neck, etc. and the use of guided procedures (if necessary) to avoid complications.

Drugs and fluids

Drug errors reported in the study such as overdose of digoxin, amiodarone and morphine led to prompt changes led by the chief pharmacist. All potentially hazardous intravenous fluids or drugs, e.g. intravenous albumin, intravenous diuretics, heparin, potassium, methotrexate, had to be administered and monitored according to protocol. Daily review of drug charts by the clinical pharmacist and the clinical team as well as pharmacy reconciliation of drugs on admission and discharge for all patients with alert to clinical teams of potential drug interactions. Staff training in the management of fluid balance, particularly in the elderly, and a robust monitoring system for fluid balance was led by an elderly care physician.

Phase 2 outcomes

The crude mortality rate was high in 2008 and 2009 (2.1% and 1.95% respectively) as was the RAMI (167 in 2008 and 145 in 2009). Following the implementation of the strategies, both indices fell over the subsequent three years (2010–2012), as shown in Figure 1. The crude mortality rate fell to 1.66% (2010), 1.58% (2011) and 1.56% (2012). Similarly, the RAMI fell to 117 (2010) and 87 (2011 and 2012). This translates as fewer hospital deaths than expected during the period 2010-2012.

Figure 1

The five-year trend in the Crude Mortality and Risk Adjusted Mortality Index (RAMI)

Discussion

Main findings

A significant number of hospital patients experience problems in care that contributed to their death. The first part of this study showed that 8.5% of adults dying at SWFT experienced a problem in their care contributing to their death. Not surprisingly, their quality of care was more likely to be judged to be poor (30% versus 3%). In 41% of those experiencing a problem in care, death was considered preventable (3.5% of all reviewed deaths). Problems were most likely during ward care, and usually involve inadequate medical input/supervision, poor clinical monitoring or management, diagnostic errors, infections and inadequate drug/fluid management.

Following these findings, themes of quality interventions were introduced which led to a successful reduction in mortality in the subsequent three years (2010–2012). While attribution is sometimes difficult in studies using before and after comparison, we believe the interventions significantly contributed to the reduction in death. Though our results might not be generalizable, we believe there are three key factors in our strategic plan that made it successful and applicable to other healthcare organizations. They are:

(a)

Leadership/Trust wide approach: The desire to reduce preventable deaths was a commitment by the Trust board, executives, senior medical staff and strong professional support from all staff. This high level backing was needed to align hospital systems and clinical directorates to a common goal and encourage a learning culture from deaths;

(b)

Contextual analysis: The mortality reduction approach was based on good evidence obtained from a rigorous hospital-wide audit of deaths which identified key gaps in the quality of care;

(c)

Communication and feedback: Mortality data were presented in simple formats using the mortality scorecard and summaries of patient deaths to encourage a reflective approach. The mortality scorecard has been a valuable tool as it is easy to use and simple to understand for non-statisticians. More importantly, it is robust and promote an objective assessment of variation across specialties (not misinterpreted or used for judgement and blame).²² Its introduction has led to a greater understanding of the processes of care to reduce preventable deaths.

Strengths and limitations of the study

Our study has a number of strengths: the contextual approach using local mortality data; our use of ‘problem in care’ rather than ‘adverse event’ to minimize the risk of overlooking errors of omission; and the use of standardized data collection with a high-quality review process.

However, several limitations need to be considered. First, is that retrospective case record reviews are restricted to the information available in the records. However, our reviewers reported only 4% of records with some information missing; hence, we believe our results are robust enough. Second, retrospective methods run the risk of hindsight bias. Knowing the outcome may influence the judgement of causation and preventability,²³ and third, reviews based on implicit criteria have been found to have only fair to moderate reliability.²⁴ We used a number of approaches to improve reliability including reviewer training, use of a structured review form, and two sets of reviewers with the second group blinded to the outcome of the first group on type of problem in care/preventability. Furthermore, our inter-rater reliability (κ = 0.75) was good, and comparable with several other studies.^3,4,12

Finally, because the Trust mortality index was above average to begin with, some will argue that this is just a case of a hospital with a high rate regressing to the mean. However, we believe that the changes made were significant enough to reduce preventable deaths as detected by our robust surveillance system.

Comparison with existing evidence

We acknowledge that our finding is from only one hospital and our more inclusive definition (i.e. errors of omission in addition to acts of commission) might lead to higher estimates; however, our preventable death findings are similar to estimates from Canada³ and the Netherlands.¹⁰ This may be because many of these deaths occur in elderly, frail patients with multiple co-morbidities, and maybe where life-expectancy is short, reviewers might be less inclined to view death as preventable. Similarly, the types of problems observed in care were similar to those reported by Wright et al., ¹⁴ Sari et al.¹² and in the Dutch study,¹⁰ where the common problems were in clinical management, diagnosis and drug/fluid management.

Implications for practice, policy and research

We believe it is a moral and social duty for all healthcare organizations to know what they are doing and how well they are doing it, by using a range of indicators that are appropriate for different clinical contexts and concentrate on how these can support internal learning. Mortality review is important, but it would be unwise to limit safety monitoring to this small group of patients, as the majority of problems in care may result in morbidity and disability rather than death.

Footnotes

Acknowledgements

None

References

Vincent

. Risk, safety and the dark side of quality. Br Med J 1999; 314: 1775–6.

Neale

, Woloshynowych

. Retrospective case record review: a blunt instrument that needs sharpening. Qual Saf Health Care 2003; 12: 2–3.

Baker

, Norton

, Flintoft

. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. Canad Med Assoc J 2004; 170: 1678–86.

Thomas

, Studdert

, Burstin

. Incidence and types of adverse events and negligent care in Utah and Colorado. Med Care 2000; 38: 261–71.

Wilson

, Runciman

, Gibberd

. The Quality in Australian Health Care Study. Med J Austr 1995; 163: 458–71.

Vincent

, Neale

, Woloshynowych

. Adverse events in British hospitals: preliminary retrospective record review. Br Med J 2001; 322: 517–9.

Thomson

, Black

, Neale

, Scobie

, Vincent

, Hogan

. Do we need better estimates of the impact of patient safety incidents on mortality/survival. Improving patient safety. In: Hignett

, Norris

, Catchpole

., eds. Improving Patient Safety 2008 ‘From Safe Design to Safe Practice’. Robinson College, Cambridge: The Ergonomics Society, 2008

Jarman

, Gault

, Alves

. Explaining differences in English death rates using routinely collected data. Br Med J 1999; 318: 1515–20.

Hayward

, Hofer

. Estimating hospital deaths due to medical error - preventability is in the eye of the reviewer. J Am Med Assoc 2001; 286: 415–20.

10.

Zegers

, de Bruijne

, Wagner

. Adverse events and potentially preventable deaths in Dutch hospitals: results of a retrospective patient record review study. Qual Saf Health Care 2009; 18: 297–302.

11.

Woloshynowych

, Neale

, Vincent

. Case record review of adverse events: a new approach. Qual Saf Health Care 2003; 12: 411–5.

12.

Sari

, Sheldon

, Cracknell

. Extent, nature and consequences of adverse events/ results of a retrospective casenote review in a large NHS hospital. Qual Saf Health Care 2007; 16: 434–9.

13.

Garcia-Martin

, Lardelli-Claret

, Bueno-Cavanillas

, Luna-del-Castillo

, Espigares-Garcia

, Galvez-Vargas

. Proportion of hospital deaths associated with adverse events. J Clin Epidemiol 1997; 50: 1319–26.

14.

Wright

, Dugdale

, Hammond

. Learning from death: a hospital mortality reduction programme. J R Soc Med 2006; 99: 303–8.

15.

Wright

, Shojanla

. Measuring the quality of hospital care: should focus on effective learning and improvement rather than judgment. Br Med J 2009; 338: 783–4.

16.

DesHarnais

, McMahon

Jr , Wroblewski

. Measuring outcomes of hospital care using multiple risk-adjusted indexes. Health Serv Res 1991; 26: 425–45.

17.

DesHarnais

, McMahon

Jr , Wroblewski

, Hogan

. Measuring hospital performance: the development and validation of risk-adjusted indexes of mortality, readmissions, and complications. Med Care 1990; 28: 1127–41.

18.

Iezzoni

, Ash

, Shwartz

, Daly

, Hughs

, Mackiernan

. Predicting who dies depends on how severity is measured: implications for evaluating patient outcomes. Ann Intern Med 1995; 123: 763–70.

19.

Rosenbaum

, Lenoch

, Ferguson

. Increasing departmental and college-wide faculty development opportunities through a teaching scholars program. Acad Med 2006; 81: 965–8.

20.

Zegers

, de Bruijne

, Wagner

, Groenewegen

, Waaijman

, van der Wal

. Design of a retrospective patient record study on the occurrence of adverse events among patients in Dutch hospitals. BMC Health Serv Res 2007; 7: 27

21.

Hogan

, Healey

, Neale

, Thomson

, Vincent

, Black

. Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf 2012; 21: 737–745.

22.

Lilford

, Mohammed

, Spiegelhalter

, Thomson

. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet 2004; 363: 1147–54.

23.

Robb

, Seddon

. A multi-faceted approach to the physiologically unstable patient. Qual Saf Health Care 2010; 19: e47.

24.

Lilford

, Edwards

, Girling

. Inter-ratter reliability of case-note audit: a systematic review. J Health Serv Res Policy 2007; 12: 173–80.