How to improve the quality of evidence when new treatments are funded conditional on collecting evidence of effectiveness and safety

Abstract

Coverage with evidence development (CED) has been promoted as a means of providing early access to innovative technologies. The policy provides for funding of a treatment or technology conditional on gathering data through a clinical trial or registry designed to determine its effectiveness and to identify rare adverse events. CED has been used with varying degrees of success in the US, Switzerland, Canada and the UK,¹ but the rigour of data collected has been a particular concern.

Treatment registers have been criticized for their poor quality of data, particularly inadequate coverage and incompleteness.^2,3 Inadequate coverage may lead to selection bias when patients who fit inclusion criteria for the register are missed. Incomplete data entry risks biased outcomes reporting. Both affect the reliability of the conclusions drawn. Randomized controlled trials (RCT) therefore continue to be considered the ‘gold standard’ in evidence-based medicine and are found at the summit of hierarchies of evidence, with observational studies in the foothills.⁴ However, RCTs have important limitations including uncertain generalizability and failure to identify important, rare safety concerns. Bradford Hill, the architect of the RCT stated: ‘Any belief that the controlled trial is the only way would mean not that the pendulum had swung too far but that it had come right off the hook’.⁵ Observational data give insight into real-world clinical practice and can be used by decision makers in health technology assessment (HTA).^6–9 They could be a powerful tool for CED because they have the potential to reduce uncertainty around treatment costs, prevalence, safety, natural history and provider experience.^10,11

Routine use of active surveillance would enhance the standing and usefulness of treatment register data and so increase their value in CED decision-making. Active surveillance is related to the approach taken by the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) group which has provided reporting standards that aim to foster and improve quality in the use and production of observational studies.¹² This is important work but is limited to describing problems which have occurred in the past and the ex post statistical methods to adjust for these failings (i.e. handling missing data and loss to follow-up). By contrast, active surveillance seeks to drive quality prospectively, rather than in retrospect.

The active surveillance approach involves proactive monitoring of the quality and coverage of data submitted to a register, and regular communication with those submitting data, with a view to improving their quality. There are five important aspects of active surveillance:

Establishing a management structure. The register should be managed by a steering group responsible for undertaking tasks listed below. This group is tasked with raising data quality. A primary concern should be transparency, with detailed minutes and information on group members that is easily accessible. The independence of members should be demonstrated through declaration of all possible conflicts of interest.

Agreeing a mandatory data subset. Time for data entry is severely limited for clinical staff and therefore the full dataset associated with a register may contain too many fields to allow it to be completed with good coverage. For this reason, a subset of the full data set should be agreed which includes only the most important items. This subset should be the focus of subsequent close monitoring and action where gaps persist.

Preventing and monitoring incoherent entries. The database should be user-friendly and designed so that implausible entries are discouraged by inbuilt rules within the software. Regular checks of the data should be carried out by people with clinical understanding to monitor the face validity of data submitted. Where entries are clinically incoherent, enquiries should be made.

Motivating those submitting data. High-quality data submission should be encouraged by provision of regular feedback on data coverage and completeness. Benchmarking against other users may also be used to encourage data submission.

Triangulation and data linkage of register data to external data sources. Triangulation and linkage of register data to other databases collecting information on the target population may be used to identify cases missed by the register and to supplement outcomes. A variety of databases may be used for this purpose including routine sources, e.g. Hospital Episode Statistics in England, the Manufacturer and User Facility Device Experience (MAUDE) database in the US and bespoke sources such as company sales data. Triangulation of data, though complex, may be essential for the viability of a project.

The components of active surveillance underpinned a successful national audit in the UK of the management of the open abdomen.¹³ The abdomen is sometimes left ‘open’ as part of the surgical treatment of complex intra-abdominal problems that make closure difficult, such as severe sepsis. In 2009, the National Institute for Health Care Excellence (NICE) published guidance on negative wound pressure in the management of the open abdomen. It considered that the evidence on safety was inadequate, recommending that if hospitals use the procedure they should make special arrangements for clinical governance and data collection. To collect and analyse the data, a national audit of management and outcomes of open abdomen was organized.

The project posed a particular challenge in terms of coverage because the clinical need for management of an open abdomen is a relatively rare event, and patients present acutely in both specialist units and general hospitals. The audit was led by a steering group with a smaller data subgroup of academics, clinicians and representatives of NICE to oversee active surveillance.

A minimum dataset was agreed by the groups to focus efforts. As the event is rare and acute, it was decided that hospitals needed to be alerted to the presence of patients on their wards for whom data submission was necessary. To achieve this, submissions from hospitals were triangulated against those in a database maintained by the Intensive Care National Audit and Research Centre (ICNARC). When patients were identified on the ICNARC database but were not on the audit database, their clinicians were contacted and encouraged to submit data to the National Audit. To maintain quality, data validation was undertaken by academics and specialist clinicians who identified clinically questionable entries, and communicated with hospital staff to verify the information.

The study yielded the largest independent survey of management of the open abdomen ever reported and was published in 2013.¹³

Use of active surveillance would improve the quality of observational data and thereby increase their value as an adjunct to clinical trial data when making decisions about the use of health-care interventions. Active surveillance involves simple practical steps that should be incorporated into the planning of any new register.

References

Brugger

Horisberger

Ruckstuhl

. Health technology assessment in Switzerland: a descriptive analysis of “Coverage with Evidence Development” decisions from 1996 to 2013. BMJ Open 2015; 5: e007021–e007021.

Smeeth

. We still need observational studies of drugs – they just need to be better. Int J Epidemiol 2006; 35: 1310–1311.

Westaby

. Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality. BMJ 2007; 335: 759–759.

Rawlins M. Harveian oration: DE TESTIMONIO on the evidence for decisions about the use of therapeutic interventions. Speech presented at The Royal College of Physicians, London, 16 October 2008.

Hill B. Heberden Oration: reflections on the controlled trial. Speech presented at The Royal College of Physicians, London, 1965.

Patrick

Gallaugher

Czoski-Murray

. Usefulness of a short-term register for health technology assessment where the evidence base is poor. Int J Technol Assess Health Care 2010; 26: 95–101.

Lyratzopoulos

Patrick

Campbell

. Registers needed for new interventional procedures. Lancet 2008; 371: 1734–1736.

Dreyer

Garner

. Registers for robust evidence. JAMA 2009; 302: 790–791.

Grilli

Taroni

. Managing the introduction of expensive medical procedures: use of a registry. J Health Serv Res Policy 2006; 11: 89–93.

10.

Black

. Why we need observational studies to evaluate the effectiveness of health care. BMJ 1996; 312: 1215–1215.

11.

Barkun

. Evaluation and stages of surgical innovations. Lancet 2009; 374: 1089–1096.

12.

von Elm

Altman

Egger

. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61: 344–349.

13.

Carlson

Patrick

Amin

. Management of the open abdomen: a national study of clinical outcome and safety of negative pressure wound therapy. Ann Surg 2013; 257: 1154–1159.