Abstract
Background
In reaction to weaknesses in feasibility studies reporting, the Consolidated Standards of Reporting Trials (CONSORT) statement published an extension for feasibility studies in 2016.
Aim
The aim of this study was to systematically review and appraise the reporting of feasibility studies in the nursing intervention research literature based on the CONSORT statement extension for feasibility studies.
Method
Papers published prior to January 2018 that described feasibility studies of nursing interventions were retrieved. Components of feasibility studies were coded, and code frequencies were analysed.
Results
The review included 186 papers. Although most papers (n = 142, 76.3%) included the label ‘pilot’ or ‘feasibility’ in their title, reporting for other components generally did not adhere to one or several CONSORT recommendations. Most papers reported objectives (n = 116, 62.4%), designs (n = 95, 51%), or rationales for sample size (n = 165, 88.7%) that were incongruent with the purpose of feasibility studies.
Discussion
This review results in two main implications for nursing research. First, we noted that the reporting of feasibility studies is weak. While all papers described feasibility studies, almost half focused exclusively on testing the effectiveness of an intervention. Second, we identified rationales for sample size along with key references that could offer guidance in reporting feasibility studies while being coherent with the CONSORT recommendations.
Background
Distinction of the subtypes of feasibility studies.
An ongoing critique of feasibility studies is that they report objectives similar to those of full-scale trials without sufficient statistical power to achieve those aims (Arain et al., 2010; Kistin and Silverstein, 2015). This is problematic, as the results from underpowered studies risk being misinterpreted (over- or underestimation or effect sizes) and lead to biased recommendations regarding the value of an intervention (Arain et al., 2010; Kistin and Silverstein, 2015).
The inconsistent use of the term ‘feasibility’ is deemed to reflect a lack of agreement and guidance on the conduct and reporting of such studies (Eldridge et al., 2016b). In recent years, the scientific community has come together in hopes of addressing this issue and created guidelines to support the design and reporting of feasibility studies. In 2015, nursing researchers Feeley and Cossette (2015a, 2015b) published guidance on the purpose and conduct of feasibility studies and highlighted their role in assessing not only the feasibility, but also the acceptability of interventions. One year later, the Consolidated Standards of Reporting Trials (CONSORT) statement – the reference for reporting randomised trials – added an extension for feasibility studies in reaction to severe weaknesses in reporting (Eldridge et al., 2016a). In comparison with the original CONSORT statement, the extension added the following recommendations for reporting key components that are characteristic of feasibility studies:
In their title, feasibility studies must be clearly identified as such. The background of a feasibility study must include the rationale for the larger trial and the reasons for conducting a feasibility study first, which should be coherent with objectives of such studies. Objectives that are appropriate for feasibility studies include investigating any component of a research protocol or intervention that is uncertain and could hamper the success of a larger trial (Eldridge et al., 2016a, 2016b; Feeley and Cossette, 2015a, 2015b). The study design must be labelled with ‘pilot’ or ‘feasibility’ (e.g. pilot randomised controlled trial). For sample sizes, authors should provide a rationale for the number of participants recruited but are not expected to report the calculations by which the numbers were determined (e.g. power calculations).
With respect to the latter recommendation, determining the sample size for a feasibility study is a topic of debate. It is widely accepted that power calculations for testing the effectiveness of an intervention (formal hypothesis testing) are not appropriate to determine the sample size for a feasibility study as they are not congruent with the purpose of such study (Eldridge et al., 2016a). To be congruent with the purpose of feasibility studies, a more acceptable approach is to determine key objectives to be achieved (e.g. recruitment and retention rates), and to adjust numbers to ensure a desired degree of precision around these estimates (Eldridge et al., 2016a). Another approach is to use a proportion (e.g. 10–15%) of the expected sample size of the full-scale trial (Cocks and Torgerson, 2013; Eldridge et al., 2016a; Whitehead et al., 2016).
The extent to which these recommendations have influenced the use and reporting of feasibility studies in nursing intervention research remains unclear. Thus, our objective was threefold: (a) to systematically review the literature on feasibility studies in nursing intervention research; (b) to assess the reporting of characteristic components of feasibility studies in nursing intervention research, based on the CONSORT statement extension for feasibility studies; (c) to identify the rationales and key references used by authors to support the sample sizes in their feasibility studies that are coherent with the CONSORT recommendations.
Materials and methods
This descriptive review (Paré et al., 2015) was conducted to examine literature and assess the reporting of feasibility studies in nursing intervention research. Through structured search methods, descriptive reviews aim to identify interpretable patterns and gaps in the literature with respect to pre-existing propositions, theories, methodologies or findings (Paré et al., 2015). In the descriptive review reported here, following the formulation of our aims we performed five steps:
developed the search strategy based on key feasibility study literature (for example: Eldridge et al., 2016a; 2016b; Feeley and Cossette 2015a, 2015b); conducted a systematic search of multiple databases to identify feasibility studies in nursing intervention research; selected studies using pre-established eligibility criteria; extracted the data focused on key features of feasibility studies; synthesised and analysed data.
Steps 1 and 2: search strategy and systematic search
A search strategy was defined by a librarian in collaboration with review authors. The search strategy used a combination of keywords and medical subheadings related to feasibility studies and nursing interventions (e.g. ‘Pilot Projects’, ‘Feasibility Studies’, ‘Nursing’, ‘Intervention’).
We searched seven electronic bibliographical databases in January 2018 for eligible primary research articles: Cumulative Index to Nursing and Allied Health Literature (CINAHL), via EBSCOhost; Embase, via Ovid SP; Google Scholar; PsycINFO, via APA PsycNet; PubMed (including MEDLINE), via NCBI; Web of Science—Science Citation Index (SCI) Expanded and Social Sciences Citation Index (SSCI), via Clarivate Analytics.
Step 3: eligibility criteria and selection of papers
To be included papers had to describe a feasibility study of a nursing intervention delivered to patients. Nursing intervention was defined as ‘treatments, therapies, procedures, or actions implemented by healthcare professionals to and with clients, in a particular situation, to move the client’s condition toward desired health outcomes that are beneficial to the clients’ (Sidani and Braden, 2011, p. 17). In terms of study design, no restrictions were used aside from our search strategy keywords which already included ‘Pilot Projects’ and ‘Feasibility Studies’. Papers written in English and French from 2015 to 2017 were retained to portray the reporting practices around and after the publication of recommendations cited in this paper’s introduction (Eldridge et al., 2016a, 2016b; Feeley and Cossette 2015a, 2015b).
Exclusion criteria were: (a) secondary analysis, literature review, or meta-analysis; (b) conference abstract; (c) full-scale randomised controlled trial; (d) intervention delivered to healthcare professionals; and (e) intervention not led or delivered by a nurse.
All citations were imported in EndNote X8.1 and duplicates were removed. Based on titles and abstracts, two independent researchers (TM, MHG) screened the first 100 citations. Given the high inter-rater agreement – Kappa reached 0.86 (95% CI, 0.84–0.88) – the remaining citations (n = 909) were split into two equal sets and each was screened by one researcher (TM or MHG). When a researcher doubted whether to include a citation or not, another researcher (PL) examined the citation and consensus was reached by discussion. Following this, full texts were retrieved for the papers selected, and additional papers were excluded per criteria described above.
Step 4: data extraction
Five researchers (TM, MHG, PL, MAMC, GF) extracted the following data for included studies: year of publication, journal, country, specialty, title, objectives, study design and rationale for sample size. Double data extraction was performed for 12% of the sample as suggested for medical record review (Worster and Haines, 2004). As the extractor agreement was high, the rest of the data was extracted by only one extractor for the remainder of the sample.
Step 5: data synthesis and analysis
Codification scheme for data analysis.
As the guidance for the appropriate rationales to support the sample size in feasibility studies is still vague, we coded the rationales for sample sizes from the papers using an inductive coding procedure, that is creating and adjusting codes depending on what was described in the papers. The analysis was mainly descriptive and based on the frequencies of codes for each category of interest.
Results
The flowchart of the literature is presented in Figure 1. Titles and abstracts were screened for 909 papers. Following this, full texts were retrieved for the papers selected (n = 206), and additional papers were excluded per criteria described above (n = 20). Leaving 186 papers to be included in the analyses.
Flow diagram of the article selection process.
Papers included in the review were published in 2015 (n = 83, 44.6%), 2016 (n = 67, 36.0%), and 2017 (n = 36, 19.4%); see Supplemental Digital Content 1 for the list of papers reviewed. Papers were published in 116 different journals (Supplemental Digital Content 1). Papers originated from North America (n = 105, 56.5%), Asia (n = 36, 19.4%), Europe (n = 26, 14.0%), Oceania (n = 13, 7.0%) and South America (n = 4, 2.2%). The most frequent specialties were geriatrics (n = 32, 17.2%), paediatrics (n = 24, 12.9%), oncology (n = 28, 15.1%) and chronic care (n = 25, 13.4%).
The majority of papers included the label ‘pilot’ or ‘feasibility’ in their title (n = 142, 76.3%); the remainder included ‘pilot’ or ‘feasibility’ in their abstract (n = 32, 17.2%) or main text (n = 12, 6.5%). In addition to the label ‘pilot’ or ‘feasibility,’ the title of 26 papers (14%) included labels suggesting formal hypothesis testing (e.g. ‘efficacy’, ‘effectiveness’, ‘effect’ or ‘impact’).
Descriptive statistics for characteristics of feasibility studies.
In 91 papers (48.9%), the study design was characterised with the label ‘pilot’ or ‘feasibility’, 88 papers (47.3%) reported a study design without using the word ‘pilot’ or ‘feasibility’. Although the included papers had either ‘Pilot Projects’ and ‘Feasibility Studies’ in the title as per our search keywords, seven papers (3.8%) did not specify any study design within their main text. Among papers that described a design, descriptors included: ‘randomised’ (n = 76, 40.9%), ‘quasi-experimental’ (n = 68, 36.6%), ‘mixed’ (n = 19, 10.2%), ‘observational’ (n = 10, 5.4%) and ‘qualitative’ (n = 6, 3.2%).
The majority of papers (n = 125, 67.2%) did not report any rationale to support their sample size. The other papers (n = 61, 32.8%) reported one or several rationales, including power calculations (n = 40, 21.5%), sample size of previous feasibility studies (n = 17, 9.1%), expected effect size of an intervention (n = 12, 6.5%), various methodological references from the feasibility study literature (n = 11, 5.9%), or qualitative data saturation (n = 3, 1.6%). Among the methodological references identified in this review (Cocks and Torgerson, 2013; Feeley et al., 2009; Hertzog, 2008; Kraemer et al., 2006; Lancaster et al., 2004; Leon et al., 2011; Thabane et al., 2010), the confidence interval approach (degree of precision) was the most frequent (n = 4 (Eldridge et al., 2016a; Hertzog, 2008; Thabane et al., 2010)). Of note, sample sizes were increased in anticipation of attrition rates in 12 papers (6.5%).
Congruency with reporting standards.
aBased on the definitions presented in Table 2.
Discussion
This systematic descriptive review appraised the reporting of 186 feasibility studies in the nursing intervention research literature and found that the majority did not adhere to one or several recommendations of the CONSORT statement extension for feasibility studies. The observations from this descriptive review result in two important contributions. First, our results highlight that this design remains misused as a large proportion of feasibility studies in the nursing intervention literature still focuses on hypothesis testing rather than on acceptability and feasibility of research protocols or interventions. Indeed, we observed that while most papers included the label ‘pilot’ or ‘feasibility’ in their title, the majority reported objectives, designs, or rationales for sample size that were not consistent with the purpose of feasibility studies. The purpose of feasibility studies is to investigate features of a research protocol or intervention that could hamper the success of a full-scale trial (Eldridge et al., 2016a, 2016b; Feeley and Cossette, 2015a, 2015b). Second, our results highlight the rationales used in nursing intervention feasibility studies to support the sample size. As guidance on the subject remains vague, highlighting the rationales that are coherent with the purpose of feasibility studies and the references to support these rationales will inform the nursing scientific community and may contribute to better practice in reporting feasibility studies.
Perhaps the most interesting but concerning finding from this review was the large number of papers misleadingly claiming to report on a feasibility study, when in fact they were presenting results regarding the effectiveness of an intervention. While all papers described feasibility studies, almost half of the papers focused exclusively on testing the effectiveness of an intervention. This goes against the purpose of the feasibility study design, which should not aim to test hypotheses regarding the effectiveness of an intervention (Arain et al., 2010). Another concerning finding was the lack of rationale to support sample sizes in more than 60% of the papers included in the review, and the presentation of power calculations to support sample sizes in 20% of the papers. This means that the vast majority of the papers reviewed did not follow the CONSORT recommendations regarding sample sizes, an issue that probably reflects misunderstandings regarding the purpose of the feasibility study design.
These results are consistent with previous reviews in which poor reporting of feasibility studies in various fields was highlighted (Arain et al., 2010; Eldridge et al., 2016a, 2016b; Lancaster et al., 2004). However, this is the first review to focus on the field of nursing intervention research. Considering the importance of intervention development and testing in nursing scholarship, it appears as a field conducive to the improvement of reporting practices for feasibility studies. However, the results of this review show that the recommendations for reporting feasibility studies have yet to have an impact in the field of nursing intervention research.
Nevertheless, some papers exemplified best practices for reporting feasibility studies. For example, Walker et al. (2015) reported on a protocol for a pilot study to evaluate the feasibility of study administration, resource and data management, intervention fidelity and effect size of a prophylactic dressing intervention to minimise sacral pressure. Verloo et al. (2016) investigated the feasibility and acceptability of delirium assessment methods in the context of home care for a randomised controlled trial. Cossette et al. (2017) assessed the feasibility, acceptability and preliminary efficacy of a nursing intervention to enhance patient acceptance of implantable cardioverter defibrillators. In all three cases, titles and designs clearly included the label ‘pilot’ or ‘feasibility’, objectives were congruent with the purpose of feasibility studies, and rationales for sample sizes are provided without involving power calculations for hypothesis testing.
Limitations
It could be argued that the timespan that this review covered was relatively close to the publication date of the recommendations; the review could be repeated in a few years to get a better sense of the impact that the recommendations had on the reporting of feasibility studies in nursing intervention research.
Conclusion and recommendations
In this descriptive review, we have systematically reviewed the literature on feasibility studies in nursing intervention research and appraised the reporting of characteristic components of feasibility studies in nursing intervention research, based on the CONSORT statement extension for feasibility studies. Our results highlight that the reporting of feasibility studies is still poor. This study design remains misused, as evidenced by the fact that a large proportion of feasibility studies in the nursing intervention research literature still focus on hypothesis testing.
Another objective was to identify the rationales and key references to support sample sizes in feasibility studies in nursing intervention research. We found that the confidence interval approach (degree of precision) was the most frequent. Additionally, key methodological references used by authors of feasibility studies in nursing intervention research, that are coherent with CONSORT recommendations, were highlighted in this review (Cocks and Torgerson, 2013; Feeley et al., 2009; Hertzog, 2008; Kraemer et al., 2006; Lancaster et al., 2004; Leon et al., 2011; Thabane et al., 2010).
Based on the results of this descriptive review, we would argue that there is a need for more sensitisation and education regarding the purpose, conduct and reporting of feasibility studies among the nursing scientific community, and that there is a need for more scrutiny of any manuscript that claims to report on a feasibility study.
Key points for policy, practice and/or research
The results of this descriptive review highlight that the reporting of feasibility studies is still weak. The findings highlight that this study design remains misused, as evidenced by the fact that a large proportion of feasibility studies in the nursing intervention research literature still focus on hypothesis testing. In terms of rationale to support sample sizes in feasibility studies, this descriptive review found that the confidence interval approach (degree of precision) was the most frequent. Results of this descriptive review suggest that there is a need for more sensitisation and education regarding the purpose, conduct, and reporting of feasibility studies among the nursing scientific community.
Supplemental Material
JRN883404 Supplemetal Material - Supplemental material for Methodological reporting in feasibility studies: a descriptive review of the nursing intervention research literature
Supplemental material, JRN883404 Supplemetal Material for Methodological reporting in feasibility studies: a descriptive review of the nursing intervention research literature by Tanya Mailhot, Marie-Hélène Goulet, Marc-André Maheu-Cadotte, Guillaume Fontaine, Pierre Lequin and Patrick Lavoie in Journal of Research in Nursing
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.
Ethics
Ethical permissions were not required for this work as it is a literature review and does not involve any participants.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by a postdoctoral fellowship from the Quebec Research Funds (Mailhot).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
