What does meaningful look like? A qualitative study of patient engagement at the Pan-Canadian Oncology Drug Review: perspectives of reviewers and payers

Abstract

Objectives

While there is wide support for patient engagement in health technology assessment, determining what constitutes meaningful (as opposed to tokenistic) engagement is complex. This paper explores reviewer and payer perceptions of what constitutes meaningful patient engagement in the Pan-Canadian Oncology Drug Review process.

Methods

Qualitative interview study comprising 24 semi-structured telephone interviews. A qualitative descriptive approach, employing the technique of constant comparison, was used to produce a thematic analysis.

Results

Submissions from patient advocacy groups were seen as meaningful when they provided information unavailable from other sources. This included information not collected in clinical trials, information relevant to clinical trade-offs and information about aspects of lived experience such as geographic differences and patient and carer priorities. In contrast, patient submissions that relied on emotional appeals or lacked transparency about their own methods were seen as detracting from the meaningfulness of patient engagement by conflating health technology assessment with other functions of patient advocacy groups such as fundraising or public awareness campaigns, and by failing to provide credible information relevant to deliberations.

Conclusions

This study suggests that misalignment of stakeholder expectations remains an issue even for a well-regarded health technology assessment process that has promoted patient engagement since its inception. Support for the technical capacity of patient groups to participate in health technology assessment is necessary but not sufficient to address this issue fully. There is a fundamental tension between the evidence-based nature of health technology assessment and the experientially oriented culture of patient advocacy. Divergent notions of what constitutes evidence and how it should be used must also be addressed.

Keywords

health technology assessment oncology drug funding Pan-Canadian Oncology Drug Review patient engagement

Introduction

Patient engagement in health technology assessment

Health technology assessment (HTA) is a multidisciplinary, evidence-based process ‘that seeks to examine the consequences of using a health care technology by considering the medical, social, economic and ethical issues’¹ involved. HTA processes are sometimes used in public health care systems to support policy decision-making about whether a particular drug, procedure or medical device should be covered under the public system. In addition to considering clinical efficacy and cost, some HTA agencies also solicit input from patient advocacy groups with the aim of ensuring that patient perspectives inform decision-making about reimbursement in publicly funded health care systems. However, while the perceived need for social accountability and alignment with the ethos of patient-centred care have thus led HTA agencies to seek such input,^2–6 neither the processes for soliciting and incorporating patient input nor the capacity of patient advocacy groups to participate in HTA are currently optimal.^7–9 Many patient groups are under-resourced, have limited capacity to engage in HTA and have expectations that diverge from those of other HTA stakeholders. Organizational efforts to support patient engagement in HTA have emerged over the past few years^10–12 but only relatively recently and the long-term impacts remain to be seen.

While the principle of patient engagement in HTA is widely supported, determining what meaningful patient engagement looks like is a difficult task. The challenge is especially great in oncology where public perception of cancer as a ‘special case’ disease fuels concerns about access to all available treatments.¹³ The prevalence of cancer and the rapid proliferation and high cost of new oncology drugs further highlight the importance of determining what meaningful patient engagement looks like in HTA.

The Pan-Canadian Oncology Drug Review programme process

In Canada, cancer drugs are assessed by the Pan-Canadian Oncology Drug Review (pCODR), a programme of the Canadian Agency for Drugs and Technologies in Health (CADTH). pCODR uses a structured deliberative framework to consider the clinical benefit, cost-effectiveness, patient values and feasibility of adoption for each drug under review, and then uses this information to make non-binding recommendations to the federal drug programmes, provincial/territorial (P/T) ministries of health and provincial cancer agencies to help guide their drug-funding decisions (except in Quebec, which has its own process).

At the heart of pCODR is the 16-member pCODR Expert Review Committee (pERC), comprising practitioners (primarily oncologists and pharmacists), health economists, an ethicist and patient representatives. This is the committee that considers the evidence and makes recommendations about whether a new drug, or new indication for an existing drug, should be funded. pERC bases its deliberations on the evidence provided by pCODR’s clinical and economic guidance panels (CGP and EGP), a submission on the feasibility of adoption from the Provincial Advisory Group (PAG), a submission from registered clinicians on local issues from a practice perspective, and a submission from a relevant patient advocacy organization. The CADTH pCODR Advisory Committee (PAC) is a steering group which provides strategic advice for the organization’s development and management and ensures the pCODR program meets the needs of provincial and territorial payers and cancer agencies. There is some overlap in membership between PAG and PAC and because both groups have a policy focus, for the purpose of this study, they were treated as a single entity.

The patient representatives on pERC are not connected to and do not advocate on behalf of any patient group. They are recruited through a public advertisement and interview process and perform the same deliberative work as their peers on the committee. They are therefore distinct from the patient advocacy groups that are the focus of this study.

Written submissions from patient advocacy groups on drugs under review are invited by the pCODR programme. From its first review in July 2011 to September 2016, 84 out of 87 drug reviews have included a patient submission. In order to participate in the process, groups must be registered with pCODR and must declare conflicts of interest. A conflict of interest declaration is requested for reasons of transparency and patient submissions may be used even if a conflict exists, provided it is declared and can be appropriately managed. To date, no patient submission has been rejected for reasons of conflict of interest. There are currently 34 groups registered with pCODR, spanning large national and small local organizations, groups focussed on specific disease sites and the more general category of rare disorders and patient-focused organizations concerned with cancer care more generally.¹⁴

Patient engagement at pCODR

The patient submission is intended to provide patient and caregiver perspectives on the day-to-day experience of living with a given disease and on currently available treatments, including issues related to drug administration, side-effects, perceived clinical benefit, care giver burden and out-of-pocket costs. The views of patients with experience of the drug under review (gained, for example, in clinical trials or as residents of a jurisdiction where it is already available) are sought in order to enable comparison of the new drug to existing treatments. Patient expectations generally for the new drug are also invited from those both with and without experience of the drug under review. A guide and template^15,16 (currently under revision), as well as periodic training events, are provided to support patient advocacy groups in the work of gathering and presenting relevant evidence. At present, the size, constitution and capacity of patient advocacy groups to participate in the pCODR process are highly variable and constitute the focus of a separate, dedicated study.

The strengths of the current patient engagement process at pCODR have been recognized and are evident in a number of ways.^17–19 The deliberative framework governing pERC’s work specifies that of the four quadrants of clinical benefit, cost-effectiveness, patient values and feasibility of adoption, ‘no single element over-rides another’.²⁰ Consideration of patient values is thus a core element of the process. In addition, two patient representatives (and one alternate) sit as full voting members of the committee. pCODR supports capacity building among patient advocacy groups through the provision of online resources, training opportunities, a patient navigator and direct outreach from the patient representatives on pERC, which are facilitated through pCODR to create a firewall while a review is ongoing. These efforts have led to tangible improvements in the quality of patient submissions over time. pCODR is also committed to transparency and its recommendations are publicly available. Patient advocacy groups have the opportunity to respond to these recommendations.

Despite these strengths, however, stakeholder expectations of patient engagement at pCODR vary and little has been reported about how participants in drug reviews consider patient input in relation to other evidence. The capacity of patient advocacy groups to participate in the process also varies and the expectations of such groups are poorly understood.

With a view to addressing some of these issues, this study set out to explore the experience of patient engagement at pCODR from the perspective of participants in the drug review process. Specifically, we wanted to know what meaningful patient engagement looks like to the members of pERC, CGP, EGP, PAG and PAC. A companion study exploring the perspectives of patient advocacy groups involved in the pCODR process is under way. It should also be noted that this study is focused on the direct engagement of patient advocacy groups in the pCODR drug review process. Broader questions of public engagement in health care priority setting and elicitation of societal values, while related, are beyond our scope.

Methods

This qualitative study is based on 24 semi-structured telephone interviews with participants in pCODR drug reviews. Interviews were conducted by the qualitative researcher (LR) between January and March 2016 and explored participants’ views on a range of issues relevant to the patient engagement process at pCODR. These included perceptions of the purpose and value added of patient engagement at pCODR and in HTA more generally; the strengths and weaknesses of current submissions from patient advocacy groups; the impact of patient submissions on the deliberative process and on pERC recommendations; perceptions of patient advocacy groups’ understanding of HTA and the drug funding process in Canada and views on what an optimized patient engagement process might look like.

A list of all current and recent past participants in the pCODR drug review process (N = 111) was compiled by the pCODR programme, which also distributed study information and invitations to participate via e-mail. Everyone on the list was invited to participate, and interested individuals replied directly to the qualitative researcher (LR) to protect their anonymity from pCODR. The qualitative researcher followed up with non-responders by e-mail after three weeks and the identities of participating and non-participating individuals were not disclosed to pCODR. All responders were interviewed. As this is a qualitative descriptive study, we were not seeking saturation²¹ and focused rather on transparency around data collection and the internal coherence of our analysis as markers of quality.

Participants provided recorded verbal consent to participate. The study was approved by the Research Ethics Board at the University of Toronto.

Interviews averaged 40 minutes in length. All interviews were digitally audio recorded and transcribed verbatim by a professional transcriptionist. Transcripts were checked against the sound files for accuracy and corrected where necessary. A coding framework was developed in discussion with the study team incorporating both a priori categories embedded in the interview guide and additional concepts emerging from interviews. Initial organization of the data using open coding was undertaken by the qualitative researcher (LR). Discussion with the study team then informed development of axial codes in order to map out the relationship between categories. Selective coding was then used to articulate a thematically organized narrative account of the data. The constant comparative method was used to test the integrity of the coding framework and included searches for disconfirming evidence.²² A qualitative descriptive approach²³ informed the analysis. This was considered most appropriate given the applied health services research context and the aim of producing a detailed account of an organizational process as perceived by participants in that process. HyperResearch²⁴ software was used to facilitate data coding and management and the qualitative researcher (LR) undertook all coding and analysis, supported by the study team.

Findings

A total of 24 individuals participated in interviews. A breakdown of participants by role and length of involvement with pCODR is presented in Table 1.

Table 1.

Participants by role and length of time in role.

Role	No. of participants
pERC (pCODR expert review committee)	9
PAG/PAC (Provincial advisory group/ CADTH pCODR advisory committee)	6
CGP (Clinical guidance panel)	5
EGP (Economic guidance panel)	4
Length of time in role
<1 year	4
1–2 years	2
2–3 years	5
>3 years	13

Meaningful patient engagement provides information unavailable from other sources

One of the most widely shared perceptions expressed by participants was that meaningful patient submissions were those that provided information unavailable from other sources. Put another way, a meaningful submission was seen as one that changed the committee’s deliberations and associated recommendation in some way by supplying insights uniquely articulated through patient values. As one pERC member explained:

There are things that are significant to patients’ well-being that don’t get well recorded or recognized easily in the clinical context. Factors that don’t lead you to go back for treatment or can’t be treated easily and that you just live with, are things that have, on occasion, actually significantly affected our discussion at the pERC table. (P05 pERC)

There were a number of situations in which the patient submission was identified as adding value in this way.

Several examples concerned the limitations of clinical trials. For example, a number of participants pointed out that clinical trial data often did not include patient-reported outcomes or quality of life data and that the patient submission could be key in these situations:

Often clinical trials are designed in such a way that we only have the primary endpoint and some safety data and not much about the patient’s experience or patient’s benefit on it. When there’s no quality of life data, the patient submission, when properly done, can be quite rich and helpful. (P23 pERC)

In a similar vein, some participants observed that structured instruments used to measure patient quality of life in clinical trials were too generic to provide anything other than high-level accounts of patient experience. Narrative accounts of symptoms, side-effects and other quality of life issues could, therefore, bring important considerations to light:

The patient input can help us value some of the more subtle things that you can’t really measure on the EORTC QLQ-C30 [instrument to assess the quality of life of cancer patients], such as the value of not having a portacath or the trade-off between some different toxicities. Sometimes the clinical benefit is borderline and, if it’s clear that patients would prefer to be receiving this treatment versus the standard treatment, then that can speak a lot. (P09 pERC)

Another frequently mentioned contribution of the patient submission was the real-world perspective it provided. This was considered important because of the artificiality of the clinical trial setting and the way in which this might over-represent the benefit of a drug or fail to account for negative impacts on a particular subpopulation of patients:

Clinical trial patients are often a very select group of people. They are pampered and looked after very well in the trials process. But then, when it comes into the real world, I think we need to step back and say, ‘What’s happening here?’ And, I think, the advocacy groups can provide that perspective. (P16 CGP)

Often, they are patients who are, in general, healthier than the corresponding real-world population. They’re the patients who were approached and said, ‘Yes, I want to be part of a clinical trial’. So, that quality of life assessment is coming down to us through a number of filters. What happened in the clinical trial doesn’t always reflect what happens in the real world. (P10 CGP)

Beyond the context of drug trials, patient perspectives were also seen as meaningful when they provided insight into particular clinical issues that might affect a deliberation. For example, one pERC member explained how information contained in the patient submission could inform recommendations when decisions hinged on specific clinical trade-offs:

Sometimes it comes down to, ‘How much do you value not having to have a portacath and come to the cancer centre every week?’ versus maybe different toxicities or side effects, or out-of- pocket costs. Often, it’s not until we’re deliberating that it comes down to, ‘How do you view nausea versus febrile neutropenia?’ (P09 pERC)

Finally, patient submissions were seen as meaningful when they provided insight into important aspects of patients’ lived experience. Geography, which is an ever-present issue in a country as large as Canada, was one such aspect:

Not everybody lives within 30 minutes of a major cancer treatment centre. [We need to consider] the dynamics of getting cancer treatment when you live in a remote place and have to travel a long way for treatment, possibly leave family and friends. (P02 pERC)

Participants were also emphatic about the importance of putting a human face on the evidence-based process of HTA, pointing out that this relied heavily on the patient submission. They maintained that simply reading about symptom burden or side effects did not automatically lead to an understanding of the realities of illness and treatment for patients and their families. Several suggested that it was neither adequate nor ethical to make assessments of drugs on the basis of clinical efficacy and cost alone without taking patient experience into account:

The practical appeal, the practical burden, all of that can sometimes be overlooked where you simply, dispassionately look at efficacy and cost. That’s simply not enough in medicine anymore. (P16 CGP)

They are the people that will benefit or be harmed by our decisions. So, I think it’s very important that they are part of this discussion. (P24 EGP)

It is, at the end of the day, their needs, their interests and their values that clinicians and the system are there to support. They deserve a place at the table because they are the subject of the treatment and the recipients of whatever benefits flow from it. (P05 pERC)

Attributes perceived as detracting from meaningful patient engagement

In the scenarios described above, patient engagement was experienced as meaningful when it made a positive contribution to the committee’s work of deliberating and formulating recommendations about cancer drugs. Participants also described scenarios where the opposite was true. These generally involved submissions that relied heavily on emotional appeals and, in a number of ways, were perceived as lacking credibility.

Submissions that relied on emotional appeals, were seen as diminishing the meaningfulness of the patient engagement process on several counts. First, they alienated other stakeholders by conflating participation in HTA with other functions of patient advocacy groups such as fundraising or public awareness campaigns. Many participants expressed concern about the possible misperception that a favourable recommendation from an HTA committee was, in some sense, a reward for suffering:

I’d give anything to live longer. I’d accept as many toxicities as possible … I’m at the end of the line of currently available treatments and now there’ll be nothing for me. (P21 EGP)

There was a comment from one of the patient advocate groups a while ago about the challenge of finding the most emotionally evocative vignette to include. Well, that’s not what we want. (P02 pERC)

Moreover, as one participant pointed out, responding to such appeals was liable to increase inequities in the health care system which was clearly not the purpose of HTA:

By favouring a horrible story, you might be putting other people that have equally horrible or worse stories to share into a disadvantage … It’s not helpful for decision-making. (P24 EGP)

Most importantly, however, participants consistently pointed out that emotional appeals were simply not relevant to the task of deliberation. Such information hindered rather than supported decision-making processes that were already inherently difficult:

There’s a lot that come in and have nothing to do with the things we need to make decisions about ‘…It’s bad to have cancer. I don’t like having an ostomy. There’s a lot of uncertainty about my future’. How does that help when you’re considering Treatment A versus Treatment B? (P09 pERC)

Another tendency that participants viewed as diminishing the meaningfulness of patient engagement was a perceived misunderstanding, on the part of patient groups, around what kind of information should be included in patient submissions. For example, one participant was puzzled by submissions which expressed expectations for the new drug but which did not align with the clinical trial data:

They [indicate] what they wish for this drug to provide but it very sporadically aligns with what the drug has been shown to do … They’ll say, ‘We want this drug to extend survival. We want it to improve quality of life’. And we have clinical trial data that clearly shows that this drug does not do that. So, I don’t know why they provide an answer that shows that they haven’t reviewed the clinical information. (P11 pERC)

Many others commented on the difficulty of incorporating the patient submission into the deliberation when insufficient information was provided about how the submission was prepared. The lack of specificity was experienced as incongruous with the evidence-based nature of HTA:

It’s been signed by one person. I don’t know who else was involved. I have no idea whether they interviewed five people or twenty people … I have no idea of the composition or breadth of the review. So, therefore, it lacks a certain level of credibility. (P16 CGP)

It will say, ‘Patients were derived from a variety of methods including online surveys, in-person interviews and attendance at groups. Patients ranged in age from 18-89’. And you don’t know how many were in the focus groups and how many were online and were they all from Canada, and how many had experience with the drug. (P02 pERC)

Discussion

Amidst growing interest in patient engagement in HTA, this study provides insights into what participants in a national drug review process perceived to be meaningful forms of engagement from patient advocacy groups. Participants in the pCODR process valued patient submissions that supplied evidence unavailable from other sources and that influenced deliberations. Limitations associated with clinical trials including the absence of quality of life data or patient-reported outcomes, reliance on generic quality of life indicators and the artificiality of the clinical trial setting were identified as areas where substantial value could be added to the HTA process by the patient submission. Patient submissions were also credited with providing important insights into particular clinical trade-offs which could be central to deliberations. Aspects of lived experience such as geographic considerations and patient and carer priorities were additional areas where the patient submission was seen as providing pivotal perspectives. Conversely, patient submissions that relied on emotional appeals or lacked transparency about their own methods were seen as detracting from the meaningfulness of patient engagement by conflating HTA with fundraising or public awareness campaigns and by failing to provide credible information relevant to deliberations.

There are several issues arising from these findings.

First, this study suggests that misalignment of stakeholder expectations remains an issue, even for a well-regarded HTA process that has promoted patient engagement since its inception. A similar observation has been made in the Australian context where ‘a significant gulf between patients’ and consumers’ expectations of the HTA process, and what decision makers want to know from these groups’ has been observed.²⁵ Support for the technical capacity building of patient advocacy groups to participate in HTA is clearly one element needed to bridge this gulf and it is hoped that the findings of this study will contribute to such work. However, while necessary, such support may not be sufficient to address this issue fully since too exclusive a focus on patient advocacy group capacity may prove problematic if it implies that patients are the only stakeholders who need to change in order to achieve the desired improvements in HTA. Increasing the technical capacity of patient advocacy groups to make submissions may result in these groups doing a better job at serving the needs of HTA but, in the absence of other changes, it would do little to improve HTA’s responsiveness to those directly affected by its decisions.

The second issue this study brings to light is thus the need to address not just issues of patient advocacy group capacity but of differences in stakeholder cultures and beliefs about what constitutes evidence and how it should be valued.²⁶ This aligns with the observations of Lopes et al.²⁷ who have shown how the divergent perspectives of stakeholders in HTA processes reflect competing notions of what constitutes relevant evidence in relation to patient engagement. The findings of this study are also consonant with their description of the fundamental tension that exists between evidence-based medicine, which HTA is intended to support, and the experientially oriented domain of patient-centredness, which provides the impetus for patient engagement practices in HTA.²⁸ The distinction between personal experiential knowledge and more professionalized notions of evidence has similarly been considered by Moreira,²⁹ whose work shows how patient advocacy organizations operate as part of wider knowledge networks. These approaches provide an important foundation for future work and seem more likely to be able to contribute to meaningful patient engagement practices in the future than do approaches which confine the patient contribution to the domain of private insight as distinct from evidence.³⁰

This study has a number of limitations. First, it is focused on one particular HTA process and it is likely that processes, dynamics and stakeholder expectations would differ in other national and institutional contexts. Second, while this study sheds light on what meaningful patient engagement looks like from the perspective of one group of stakeholders, building on this work will require similar exploration of the perspectives of patient advocacy groups. Patient submissions to pCODR come from a wide variety of groups ranging from large, well-resourced national charities to very small organizations representing rare disorders. Capturing this heterogeneity and the complex spectrum of perspectives emerging from it represents an important opportunity to extend our sense of what meaningful patient engagement in HTA could one day look like.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Health Equality Europe. Understanding health technology assessment (HTA), http://img.eurordis.org/newsletter/pdf/nov-2010/58-1%20HEE%20Guide%20To%20HTA%20for%20Patients%20English.pdf (2008, accessed 1 December 2016).

Kreis

and Schmidt

Public engagement in health technology assessment and coverage decisions: a study of experiences in France, Germany and the United Kingdom. J Health Polit Policy Law 2013; 38: 89–122.

Barham

Public and patient involvement at the UK National Institute for Health and Clinical Excellence. Patient 2011; 4: 1–20.

Bridges

JFP

and Jones

Patient-based health technology assessment: a vision of the future. Int J Technol Assess Health Care 2007; 23: 30–35.

Coulter

Perspectives on health technology assessment: response from the patient’s perspective. Int J Technol Assess Health Care 2004; 20: 92–96.

Scott

and Wale

JL.

Patient advocate perspectives on involvement in HTA: an international snapshot. Res Involv Engagem 2017; 3: 2.

Facey

Boivin

Gracia

et al . Patients’ perspectives in health technology assessment: a route to robust evidence and fair deliberation. Int J Technol Assess Health Care 2010; 26: 334–340.

Messina

and Grainger

DL.

A pilot study to identify areas for further improvements in patient and public involvement in health technology assessments for medicines. Patient 2012; 5: 199–211.

Gagnon

Desmartis

Lepage-Savary

et al . Introducing patients’ and the public’s perspectives to health technology assessment: A systematic review of international experiences. Int J Technol Assess Health Care 2011; 27: 31–42.

10.

European Patients’ Academy. European Patients’ Academy Expert Training Course for patients and patient representatives on the medicines research & development process, www.eupati.eu/eupati-training-course/ (2014, accessed 11 January 2017).

11.

Health Technology Assessment International. Values and quality standards for patient involvement in HTA, www.htai.org/interest-groups/patient-and-citizen-involvement/pcisg-home/values-andstandards.html (2014, accessed 4 December 2016).

12.

Health Information and Quality Authority. Guidelines for stakeholder engagement in health technology assessment, www.hiqa.ie/publications/guidelines-stakeholder-engagement-health-technology-assessment-ireland (2014, accessed 11 January 2017).

13.

Neumann

Bliss

and Chambers

JD.

Therapies for advanced cancers pose a special challenge for health technology assessment organizations in many countries. Health Aff 2012; 31: 700–708.

14.

Canadian Agency for Drugs and Technologies in Health. Patient groups registered with pCODR, www.cadth.ca/sites/default/files/pcodr/Submit%20%26%20Contribute/pcodr-registered-patientadgrps.pdf (2015, accessed 11 January 2017).

15.

Canadian Agency for Drugs and Technologies in Health. Guidelines, procedures, and templates, https://cadth.ca/pcodr/guidelines-procedures-and-templates (2017, accessed 12 December 2017).

16.

Canadian Agency for Drugs and Technologies in Health. Patient engagement guide, https://cadth.ca/sites/default/files/pcodr/pCODR%27s%20Drug%20Review%20Procss/pcodr-patient-engagement-guide.pdf (2017, accessed 12 December 2017).

17.

Rochi

Chabot

and Glennie

Evolution of health technology assessment: best practices of the pan-Canadian Oncology Drug Review. Clinocoecon Outcomes Res 2015; 7: 287–298.

18.

Hoch

and Sabharwal

Informing Canada’s cancer drug funding decisions with scientific evidence and patient perspectives: the Pan-Canadian Oncology Drug Review. Curr Oncol 2013; 20: 121–124.

19.

Hoch

Brown

McMahon

et al . Meaningful patient representation informing Canada’s drug funding decisions: views of patient representatives on the Pan-Canadian Oncology Drug Review. Curr Oncol 2014; 21: 263–266.

20.

Canadian Agency for Drugs and Technologies in Health. pCODR Expert Review Committee Deliverative Framework, www.cadth.ca/sites/default/files/pcodr/The%20pCODR%20Expert%20Review%20Committee%20(pERC)/pcodr_perc_deliberative_frame.pdf (2016, accessed 11 January 2017).

21.

O’reilly

and Parker

Unsatisfactory Saturation: a critical exploration of the notion of saturated sample sizes in qualitative research. Qual Res 2012; 13: 190–197.

22.

Strauss

and Corbin

Basics of qualitative research. Thousand Oaks, CA: SAGE, 1998.

23.

Sandelowski

Focus on research methods-whatever happened to qualitative description?

Res Nurs Health 2000; 23: 334–340.

24.

HyperRESEARCH version 3.7.3. Researchware, Randolph, MA.

25.

Wortley

Wale

et al . Moving beyond the rhetoric of patient input in health technology assessment deliberations. Aust Health Rev 2017; 41: 170–172.

26.

Tjornhol-Thomsen

and Hansen

HP.

Knowledge in health technology assessment: who, what, how?

Int J Technol Assess Health Care 2011; 27: 324–329.

27.

Lopes

Street

et al . Involving patients in health technology funding decisions: stakeholder perspectives on processes used in Australia. Health Expect 2016; 19: 331–344.

28.

Lopes

Carter

and Street

Power relations and contrasting conceptions of evidence in patient-involvement processes used to inform health funding decisions in Australia. Soc Sci Med 2015; 135: 84–91.

29.

Moreira

Understanding the role of patient organizations in health technology assessment. Health Expect 2015; 18: 3349–3357.

30.

Staley

and Doherty

It’s not evidence, it's insight: bringing patients’ perspectives into health technology appraisal at NICE. Res Involv Engagem 2016; 2: 4.