Abstract
The remit of research funding bodies is to prioritise funding for research that is of relevance and of high quality. This in turn will aim to raise the quality of healthcare and benefit to patients. Researchers are faced with increasing demands and expectations from the public purse and patients. The emphasis is to improve the quality of their research, with the ultimate aim of improving population health. While guidelines on feasibility study methods concentrate heavily on trials, there appears less guidance on application of health economics within feasibility studies, yet these are a less costly way to determine first of all if a full randomised controlled trial (RCT) is feasible. A feasibility study assesses if the study can be done in a small RCT type study. Since by definition, a feasibility study does not evaluate the outcome, researchers often omit the health economics aspects but do however include statistical analysis. This leaves a gap in interpretation for policy makers and potential funders. It also means that any resulting publication does not include relevant information and therefore comparison across studies in terms of difficulty in collecting cost data is not possible. The main aim of this commentary therefore, is to demonstrate a suggested health economics analysis within a feasibility study and to recommend to researchers to include these aspects from the conception of their intervention. This paper proposes a number of points, with rationale for each point, to indicate the health economics data and the potential benefits required for coherent interpretation of the feasibility of future economic evaluations in a full trial. Economic evaluation is necessary if implementation into standard care is anticipated. Therefore, collection and summary analysis of relevant data is good practice at each point of the intervention development. Current guidelines for economic evaluation, for example, The Medical Research Guidelines in the UK, conclude by stating that ‘to address these sufficiently, it is best to involve health economists early in the planning of design of the evaluation, so that the economic evaluation is fully integrated’. I argue in this commentary that similar early involvement is critical in feasibility studies to ensure that all eventualities are considered before the main phase III/IV trial begins.
Introduction
The remit of research funding bodies is to prioritise funding for research that is of relevance and of high quality. High-quality research addressing relevant questions would ultimately lead to patient benefits. The research and health services funding agencies face constant pressure to re-evaluate their budgets and improve the efficiency of their funded research. This in turn means that potentially funded project applications and projects come under welcome scrutiny and this would potentially help their findings, policy impact and patient benefit. The argument that weaknesses in the design, conduct, and analysis of research reduces value and increases waste has been highlighted recently (1, 2). However, health economics has been overlooked in this series (1, 2) so this paper is now a good opportunity to emphasise the importance of health-economic evaluation from the beginning of project design.
To that end, a range of tools and references for feasibility studies and general checklists are available to researchers. In the UK, guidelines on how to evaluate complex interventions have been introduced (3). These guidelines do not refer to Clinical Trial of an Investigational Medicinal Product (CTIMP) nor to other medical devices. Researchers should reflect on these to ensure they firstly align their research with standard practice and secondly, improve the quality of their trials. Indeed, Cochrane systematic reviews, which many clinical guidelines are heavily informed by, are consistent in their findings that the overall quality of the body of evidence in trials (using Grades of Recommendation, Assessment, Development and Evaluation Working Group (GRADE) is low or very low. This includes relevant points on how to perform a cost-effective analysis and what data to collect within the trial timeframe. The key point here is ‘trials’, i.e., randomised controlled trials (RCT). While these guidelines concentrate on both randomised and non-randomised designs, there is less specific guidance on feasibility studies, yet these are a less costly way to determine first of all if a full trial, RCT, is feasible and to be considered. It is common for large-scale RCTs to include economic evaluations. In order to design the RCT well to answer the cost-effectiveness questions, we need to consider cost-effectiveness prior to the RCT. This can be done during the feasibility/pilot phase.
Trials are very expensive and it would be preferable to ascertain indicative average costs of a pragmatic trial in the early stage of health-technology assessment, for example, using data from feasibility studies. The section in Craig et al (3), “Assessing feasibility and piloting methods” does not clearly set out what should and should not be included, from a health economics perspective. Furthermore, some current guidelines, e.g., the UK National Institute of Health Research (NIHR) guidelines on Feasibility and Pilot Studies do not explicitly mention health economics (4). The current Medical Research Council (MRC) guidance (3) provides a discussion of initial development and piloting of studies and an example of early economic evaluation during the pilot phase to inform decisions about a further trial was included in this guidance. In addition to well-documented examples of the value of early considerations of economic evaluation (5, 6), this commentary now suggests further detail that may benefit the researcher.
A feasibility study assesses if the study can be done in a small randomised control type study. Since by definition a feasibility study does not evaluate the outcome (7) researchers often omit the health economics aspects, even though they often provide statistical analysis of the clinical efficacy. This leaves a gap in interpretation for policy makers and potential funders of the larger scale trials. It also means that any resulting publication does not include relevant information on cost-effectiveness data. Therefore, direct comparison of cost and quality-of-life data collection across feasibility studies is not possible.
Indeed, this issue even arises in full trials as there is no standardisation for collection of resource use and cost data in trials (8). However, it was recommended that piloting of patient and carer completed forms should be done to test for clarity and ease of use and to determine potential completion rates (8). Piloting is also useful in determining the main cost-driving events related to the health technology under assessment. This current lack of standardisation and exclusion of economic aspects in the feasibility study guidelines creates confusion amongst clinicians when applying for research grants and conducting feasibility studies. Many do not include any health economics, only to get asked to re-submit and include more details – the crux is that they do not know what to include and sometimes even the range of different types of experienced health economists available do not know either. Furthermore, without access to an established Clinical Trials Unit, with relevant health economics expertise in situ, the decision on what and what not to include is difficult. This is similar to the lack of statistical expertise sought in earlier trials, before detailed guidelines were necessitated. Both statisticians and health economists are necessary members of a multidisciplinary team from the conception of the study.
The main aim of this commentary, therefore, is to demonstrate a suggested health economics analysis within a feasibility study and to recommend to researchers to include these aspects from the conception of their intervention to the operation of the study. This approach is based on my experience in working on various trials and as a reviewer for the UK NIHR and health economist on the NIHR Research for Patient Benefit (RfPB) Funding Agency Advisory Committee. The views herewith are entirely my own and are open to critique.
Health economics issues
While some reporting checklists are available, they are much less used than they should be, and funders should be as demanding of the health-economic aspects as they are for RCTs, via the CONSORT 2010 Statement (9), an evidence-based minimum set of recommendations for reporting randomized trials, and for reviews, via PRISMA (10), (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses.
More specific guidance could help funding bodies and readers to understand what an appropriate feasibility analysis would involve, from a health economics perspective. Three main issues are evident: (i) are the costs of a full trial justified; (ii) how should costs and utility be measured as outcome measures and (iii) how should the data be analysed and potentially modelled. The aim of such guidance would be to provide a list of parameters that could be used to indicate the health economics data required and the potential benefits of the intervention. This could serve as a framework for future data collection in the full trial and adherence to the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) (11) reporting guidelines that are being increasingly advocated by a range of high-impact journals, e.g., Implementation Science, The BMJ and the Biomed Group.
I now propose the following core points that could be addressed, and provide rationale for each.
Aim
The aim is to determine indicative costs and benefits to the healthcare provider (e.g., National Health Service [NHS]) and society that will be used in the full RCT.
Rationale
When a full trial is conducted, the new intervention or treatment will only be recommended for population health, if cost effective. Indicative costs are therefore an important part of the pre-trial process.
Objective
An objective of a feasibility study is to define and refine methods for data collection. This includes health economics data.
Rationale
Data collection should also include collection of health economics data – if patients refuse to provide such information, strategies to improve collection of data must be put in place before the full trial begins.
Healthcare use
Include healthcare utilisation in the data collection at all time-points. Self-reported resource use measures, can be determined by the health economist for collection of relevant service use data (12) e.g., a variant of Client Service Receipt Inventory (CSRI).
Rationale
If the clinical evaluation takes place at baseline and follow-up time points, the economic evaluation must be considered also at all those points. These are used to allow patients to provide information on healthcare resource use that is not evident from routine databases.
Reference costs
Propose reference costs that will be used, e.g., Personal Social Services Research Unit (PSSRU) or NHS unit costs.
Rationale
Combined with healthcare use, this will provide useful cost-per-patient information.
QALYs
Include a measure of quality of life (QOL), e.g., EQ5D that will allow for calculation of quality-adjusted life years (QALY). For robustness, also include an alternative, such as SF12, SF36 or a disease-specific QOL measure that can be mapped onto utility values.
Rationale
Using utility values, economists can then calculate the QALYs – differences in scores can be used to show potential differences that may emerge in the RCT. In many instances, we wouldn't expect to see a difference in EQ-5D scores during the course of a trial. For example, although generic instruments generate utilities that can enable comparisons across different disease areas, they can sometimes be insensitive to some aspects of certain conditions, e.g., asthma or inflammatory bowel disease.
Inclusion of EQ-5D in the feasibility stage could potentially illustrate this point, but there are likely many cases where we beforehand could say that collection of these data would be futile and hence use alternatives.
Summarise
Once all the relevant cost and benefit data are collected, provide summary statistics (e.g., mean values) for each arm of the trial and indicate potential outliers.
Rationale
This will highlight any discrepancies in data, missing data and need for advanced modelling of outliers in the full trial.
Modelling
Some exploratory modelling work could be done beforehand to ensure that it is possible to demonstrate potential cost effectiveness.
Rationale
In order to conduct an economic evaluation, it is necessary to follow patients until there are no meaningful differences between the arms. This is of course impossible within the scope of most trials, which is one reason why we use models. A key aspect of a trial would therefore be that it provides the relevant data to be used as model inputs. It is therefore important to ensure that it is feasible to collect this information in the first place.
Prioritise
The feasibility trial can inform the study that would determine clinical efficacy and often is the main goal of the clinical researchers in the first instance. However, the health economics analysis also deserves prioritisation.
Rationale
This will ensure a transparent analysis of cost data and a smooth transition into the economic evaluation in the main trial.
Conclusions
By including the above information into study designs and grant applications for feasibility studies, researchers may be less likely to be asked to re-submit, and have a higher chance of a successful outcome. Similarly, systematic reviews of feasibility trials will be easier to conduct if researchers have a standardised data collection process, such as the one that I suggest. Furthermore, the potential to pool data across studies will be greater if there is consistency in health economics information and outcomes. Economic evaluation is necessary if implementation into standard care is anticipated. Therefore, collection and summary analysis of relevant data is good practice at each point of the intervention development. Some guidelines, e.g., the MRC guidelines (4) conclude by stating that “to address these sufficiently, it is best to involve health economists early in the planning of design of the evaluation, so that the economic evaluation is fully integrated”. I argue in this commentary that similar early involvement is critical in feasibility studies to ensure that all eventualities are considered before the main phase III/IV trial begins.
Footnotes
Acknowledgements
I would like to thank Martin Tickle, Tanya Walsh and Paul Wilson at The University of Manchester for very helpful comments on an earlier version of this paper.
Financial support: No grants or funding have been received related to this study.
Conflict of interest: The author has no financial interest related to this study to disclose.