A protocol for the evaluation of the process and impact of embedding formal and experiential Public and Patient Involvement training in a structured PhD programme

Abstract

Background:

Incorporating Public and Patient Involvement (PPI) into doctoral research is valued by PhD scholars. The importance of providing early career researchers with appropriate education and training to develop skills to conduct meaningful involvement has been articulated. The Collaborative Doctoral Award in MultiMorbidity (CDA-MM) PhD programme embedded formal PPI training as a postgraduate education component. Four PhD scholars taking part in the CDA-MM established a PPI panel comprising people, and carers of people, living with multimorbidity (≥2 chronic conditions), presenting an opportuning for experiential PPI training. The proposed study aims to evaluate the process and impact of formal and experiential PPI training during a PhD programme.

Design:

Embedding PPI training in a PhD programme is a novel approach. This evaluation will include a process evaluation to provide an understanding of the workings of the PPI panel and explore the experiences of key stakeholders involved, and an impact evaluation to assess the impact of embedding PPI training in a PhD programme. This study is a longitudinal mixed-methods evaluation, conducted over 24 months. Participants include PhD scholars, PPI contributors and PhD supervisors. An independent researcher not aligned with the CDA-MM will lead the evaluation. Data collection methods include focus groups, individual interviews, an impact log and group reflections. Qualitative data will be analysed using thematic and content analysis and quantitative data will be analysed using descriptive statistics.

Discussion:

This evaluation will report the learnings from embedding formal and experiential PPI training and education across a PhD programme.

Plain English Summary

Why are we doing this research?

Some PhD students want to involve patients and members of the public in their research but there is not much formal training for how to do this successfully as a PhD student. Four PhD students established an advisory panel of people living with more than one ongoing health condition to advise their research projects. The students are learning about how to involve patients and members of the public in research through this experience, and through formal training built into their 4-year PhD programme.

What is this research study trying to find out?

We want to find out how the students involved patients and the public in their research and what the students, their supervisors, and the patient and public contributors learned during this process.

How are we doing this research?

The panel of people advising the research projects will be invited to take part in one group discussion in the middle of the PhD programme and another at the end. These discussions will aim to find out what their experience of being involved in PhD research was like. The PhD supervisors will also be invited to take part in one group discussion in the middle and one at the end of the PhD programme. These discussions will aim to find out if they think the training made a difference to the way PhD research is carried out. The PhD students will keep a record of the time they spent involving patients and the public in their research, and will record the impact of this involvement on their projects. The students will also keep a journal to reflect on their experiences and will be invited to take part in a discussion at the end of the 4-year PhD programme. All of this data will be analysed to understand how patients and the public were involved in the PhD research, how that impacted the research, and what everyone learned during the process.

What will happen with the results?

We will invite the advisory panel to help us tell other people living with more than one ongoing condition about our research findings. We will also invite the panel to tell members of the public about what it was like to work with PhD students and what everyone learned during the process. We hope that this might encourage other members of the public, including people living with health conditions, to get involved in research being carried out by PhD students. The results will also be published in an academic journal. We hope our results will provide encouragement and guidance for future PhD students to involve patients and members of the public in their research.

Introduction

Public and Patient Involvement (PPI) is defined as research undertaken with or by patients and the public, as opposed to research undertaken on, for, or about them.¹ Drawing on the lived experience of PPI contributors, good quality PPI can enhance the quality and relevance of studies,² whilst also contributing to the broader democratisation of research.³ Funders emphasise the importance of involving patients and the public, increasingly requiring that applications include PPI in proposed programmes of research. However without knowledge and skills, researchers may be apprehensive about using PPI⁴ and examples of tokenistic involvement are often reported, with patients and the public invited to ‘tinker at the edges’.⁵ The importance of providing early career researchers with appropriate education and training to enhance both understanding of and skills to conduct meaningful involvement has been articulated,⁶ and calls have been made to embed PPI as a component of postgraduate education.^6,7 In many countries worldwide, postdoctoral students can avail of PPI training through courses and workshops designed for researchers with varying levels of PPI knowledge; however formal incorporation of PPI training in postgraduate programmes is in its relative infancy.

In 2017, two national research funders in Ireland, the Health Research Board (HRB) and the Irish Research Council (IRC), funded the PPI Ignite programme, supporting five universities to embed PPI across their research culture.⁸ The PPI Ignite programme – focused on building PPI capacity amongst both researchers and the public and patients – has been a key driver in a ground-swell of interest in PPI amongst researchers in Ireland in recent years. Evidence of this growing interest includes an upturn in PPI training courses and conferences, development of general and disease-specific PPI groups, and growing numbers of partnerships between patient organisations and academic research groups. The increased awareness of PPI amongst doctoral scholars and their supervisors is apparent; however PPI education and training is not yet embedded in doctoral programmes in Ireland.

A Health Research Board-funded Collaborative Doctoral Award in MultiMorbidity programme (CDA-MM) commenced in Autumn 2018, involving a cohort of four PhD scholars in primary care. The single cohort of four PhD scholars in the CDA-MM are from different disciplinary backgrounds (general practice, health economics, health psychology and pharmacy). While each scholar conducts their own distinct project, they also collaborate on specific aspects of each other’s projects. The programme is underpinned by the research theme of multimorbidity, which has previously been defined as the presence of two or more chronic conditions in one individual.⁹ It has long been suggested that involving patients as partners in multimorbidity research may help to answer complex clinical questions for this population¹⁰ and may be key to determining what outcomes matter most to these patients.¹¹ Some evidence of the positive impact of PPI in multimorbidity research has already been reported.¹² It has recently been suggested that involving people living with multimorbidity in research is feasible, and recognised challenges can be overcome.¹³ Consequently, PPI has been a core element of the CDA-MM since its inception. During grant writing, an existing PPI group working on other primary care studies with members of the applicant team contributed to shaping the research questions of the individual PhD studies and the overall aims of the CDA-MM; the doctoral curriculum was designed to include PPI components; and the consortium of experienced inter-disciplinary multimorbidity researchers leading the CDA-MM is committed to embedding PPI across the doctoral programme. The CDA-MM is the first in Ireland to embed PPI education and training within its PhD programme. The PhD scholars completed a series of general training modules through the Structured Population Health and Health Services Research and Education Programme (SPHeRE),¹⁴ supplemented by formal CDA-MM multimorbidity training and formal PPI training (see Box 1).

Box 1

CDA-MM formal training components

SPHeRE programme modules	Multimorbidity formal training component	PPI formal training component
Perspectives on Population Health & Health Services Research – Linking Contexts and Methods Practical Approaches To Population Health & Health Services Research Methods & Study Design Health Systems, Policy & Management Systematic Reviewing & Protocol Development Working with Health Information – Informatics and Statistical Analysis Health Economics	Definition and measurement in multimorbidity Mental and physical health multimorbidity Multimorbidity: researching prognosis and impact (cross sectional and cohort studies) Qualitative research methods to support clinical research in multimorbidity Multimorbidity: Interventions focusing on key healthcare areas such as utilisation and medicines management Multimorbidity outcome data, primary care clinical data collection and management using practice EHRs Application of statistics to multimorbidity research (e.g., condition cluster analysis) Economic evaluation in clinical trials and multimorbidity	PPI in primary care research Establishing a PPI panel for the CDA-MM Online PPI facilitation skills Conducting a PPI evaluation study Communicating research findings to the public

In their first year, the PhD scholars worked together to establish a PPI panel to support their PhD research studies, comprising people living with multimorbidity and carers of people living with multimorbidity. The PPI panel meet approximately four times per year and advise on the PhD projects across the research trajectory, including intervention development, outcome prioritisation, and communication of research findings. Working with the PPI panel provides the PhD scholars with opportunities for ongoing experiential learning in PPI, building on the formal PPI training provided in the CDA-MM. Incorporating PPI into doctoral research has been reported by PhD scholars as valuable¹⁵, with the potential to positively contribute to the development of both the doctoral research projects and the PhD scholars, while also being a rewarding experience for PPI contributors.⁷ However it is not without its challenges, and the importance of training PhD scholars so they are equipped to overcome these challenges has been highlighted.¹⁶ Evidence is lacking on the impact of embedding PPI training across a doctoral programme. Therefore, we plan to evaluate how PPI education and training was embedded in a structured doctoral programme and the impact of this training, including the experiential component, on individual projects and the overall programme.

Aim of the evaluation study

The aim of the proposed study is to evaluate the inclusion of both formal and experiential PPI training within the CDA-MM. Specifically, the objectives of the study are to:

Explore the feasibility of embedding formal and experiential PPI training within a doctoral programme.

Explore the experiences and perspectives of PPI contributors participating in a doctoral programme PPI panel.

Explore the process and perceived impact of embedding formal and experiential PPI training in the CDA-MM from the perspectives of PhD scholars and PhD supervisors.

Assess the impact of embedding PPI in the CDA-MM on the design and conduct of the four individual research projects being undertaken by the PhD scholars, as well as on the overall CDA-multimorbidity programme, including training and dissemination activities.

Methods

This study will evaluate the novel approach of embedding formal and experiential PPI training in the CDA-MM from the perspective of the PPI contributors, PhD scholars and PhD supervisors. We will assess the impact of PPI on the PhD scholars’ research, learning and development, and on the overall collaborative doctoral programme. We consulted the PPI contributors to understand how they would like to contribute to the study; the methods outlined here incorporate their preferences.

Design

This study is a longitudinal mixed-methods evaluation conducted over a 24-month period, with the first round of data collection taking place 24 months into the 48-month CDA-MM and the second round of data collection taking place 36 months into the CDA-MM. Table 1 outlines how the evaluation study addresses each of the study objectives.

Table 1.

The evaluation process.

Objective	Data sources	Sample	Analysis
Explore the feasibility of embedding formal and experiential PPI training within a doctoral programme	Focus groups at 24 months and 36 months	PPI panel members PhD supervisors	Thematic analysis
	Semi-structured interviews at 36 months	PhD scholars	Thematic analysis
Explore the experiences and perspectives of PPI contributors participating in a doctoral programme PPI panel	Focus groups at 24 months and 36 months	PPI panel members	Thematic analysis
Explore the process and perceived impact of embedding formal and experiential PPI training in the CDA-MM from the perspectives of PhD scholars and PhD supervisors	Focus groups at 24 months and 36 months	PhD supervisors	Thematic analysis
	Semi-structured interviews at 36 months	PhD scholars	Thematic analysis
	Sample activity logs at 24 months and 36 months	PhD scholars	Quantitative summary statistics
	Notes from group reflections	PhD scholars	Content analysis
Assess the impact of embedding PPI in the CDA-MM on the design and conduct of the four individual research projects being undertaken by the PhD scholars, as well as on the overall CDA-multimorbidity programme, including training and dissemination activities	Semi-structured interviews at 36 months	PhD scholars	Thematic analysis
	Notes from group self-reflections	PhD scholars	Content analysis
	Impact log from each PPI meeting	PhD scholars	Quantitative summary statistics
	Focus groups at 24 months and 36 months	PhD supervisors	Thematic analysis

The evaluation will include two key elements:

(a) Process evaluation

Qualitative methods will be used to understand how the PPI panel was established and how it operates, and to explore the experiences and views of the PPI contributors, PhD scholars and PhD supervisors on embedding PPI in a structured PhD programme. The experiences and views of these key stakeholders will be explored using a combination of focus group and individual interviews conducted at specified time points after the establishment of the PPI panel.

Two focus groups will be conducted with PPI contributors (24 months and 36 months).

Two focus groups will be conducted with PhD supervisors (24 months and 36 months).

Individual interviews will be conducted with PhD scholars (36 months).

Focus groups and individual interviews will be semi-structured, guided by topic schedules informed by the existing literature and with input from the PPI panel. The qualitative approach will offer insight into the experiences and perspectives of key stakeholders within a PhD programme.¹⁷ The semi-structured approach to interviews and focus groups will allow the researcher to remain flexible and adapt questions in response to participants.¹⁸ Focus groups were identified as the preferred method of data collection by the PPI panel.

(b) Impact evaluation

A mix of qualitative and quantitative methods will be used to assess the impact of PPI on the PhD projects and scholars’ learning and development, and to assess PhD scholars’ time contributed to PPI activities. Data will be gathered from:

Impact log completed by PhD scholars

An impact log will be completed by PhD scholars after each PPI panel meeting. A common template developed for this purpose will be used (see Table 2), detailing what was discussed, what changes were made to the projects based on input from the panel, and the perceived extent of these changes (small, moderate or large), as rated by the PhD scholars; for example, changes to language used in written information given to research participants, decisions on where and how to share research findings with patients and the public, and changes to the logistics of delivering interventions will be recorded in the impact log.

Self-facilitated group reflections by PhD scholars after PPI panel meetings

For the duration of the doctoral programme, the PhD scholars will be seeking to learn from their experiences of working with the PPI panel, using self-facilitated group reflection. A group reflection will take place after each PPI panel meeting. Reflections will be guided by the Gibb’s Reflective Cycle,¹⁹ developed to give structure to learning from experiences, and to reflect on learning and development. The scholars will also reflect on time committed to organising, planning and conducting PPI panel meetings, as well as the process of embedding PPI throughout the doctoral programme. The PhD scholars will retain written records of the group reflections.

Two sample activity logs

To quantitatively assess the time involved in organising, planning and conducting PPI panel meetings during a PhD, the PhD scholars will complete a detailed activity log for two PPI panel meetings, one during the first 12 months of the study, and another during the final 12 months of the study. The sample activity logs will include details such as time spent contacting PPI contributors, preparing content and materials for PPI meetings, and facilitating PPI meetings.

The PPI panel will be involved throughout the evaluation, such as advising on focus group questions, sense-checking findings, reviewing drafts, preparing a summary of the evaluation report for a public audience, and communicating findings of the evaluation.

Table 2.

Impact log template.

Impact log items
Date
PhD scholar’s name
What did we do?
What was discussed?
What was changed?
What was the impact? (1 = small change; 2 = moderate change; 3 = large change)

Participants and recruitment

The sampling frame is finite and includes nine PPI panel members, four PhD scholars and nine PhD supervisors involved in the CDA-MM. If all those invited agree to take part, the study will involve 22 participants. The authors BK, LF, JL, and AC (PhD scholars) and SMS and BC (PhD supervisors) will be participants in the study. Therefore, for reasons related to objectivity, credibility and ethical considerations, an independent researcher (MP) who is otherwise not associated with the CDA-MM and not involved in CDA-MM governance will lead the evaluation. Potential participants will be invited to take part through a gatekeeper (EM for PhD students and PhD supervisors; BK, LF, JL, AC for PPI contributors), who will outline the details of the study, and with consent will share the contact details of participants with the independent researcher (MP). From this point onwards, the independent researcher will obtain informed consent from all participants and organise and complete data collection.

Data collection and analysis

In light of Covid-19 restrictions, the first round of focus group interviews will be conducted remotely by the independent researcher (MP) using video-conferencing facilities (Zoom) and organised to best suit the needs and preferences of participants. Where virtual communication is used, all participants will be supported in accessing remote communication facilities to ensure equality of participation using these means. It is anticipated that subsequent focus groups and the semi-structured interviews with the PhD scholars will be conducted (by MP) face-to-face in a location convenient to the participants, likely a private room in a university building. All interviews will be recorded and notes will be taken by the researcher. Interviews and focus groups will not be fully transcribed. There is ongoing debate about the role of field notes, transcriptions and audio-recordings in qualitative data analysis.^20,21 In this study, a decision was taken not to fully transcribe audio-recordings. This is partly for resource reasons, but also to avoid the loss of an overview of interviews. Salient issues will be captured in notes taken during interviews and field notes written up after the interview. The audio-recordings will be returned to and listened to, following which initial field notes will be expanded upon and, where necessary, amended/revised. This will ensure that participants’ perspectives are fully represented. Part of the audio recording may be transcribed verbatim as needed.

Data from the focus groups and individual interviews will be analysed inductively and organised into themes and sub-themes, following the principles of thematic analysis outlined by Braun and Clarke.²² Content analysis will be conducted using data from the group self-reflections to describe the impact of PPI on the individual PhD scholars and the overall PhD programme. Qualitative data from the impact logs will be scrutinised to summarise the perceived impact of PPI on the individual research projects. Quantitative data from the impact logs will be summarised using descriptive statistics. Data from the activity logs will be used to summarise the activities involved and to calculate the time commitment required for PPI activities as part of a PhD programme. Participants will be given an opportunity to sense-check the findings before the analysis is finalised. Following the first round of data collection, the researcher (MP) will identify and communicate findings that offer a learning opportunity for the participants. This feedback loop will guide the PhD scholars, PhD supervisors and PPI contributors to refine the PPI process and address any issues identified by stakeholders.

Data management and confidentiality

To facilitate pseudonymisation, participants will be assigned a unique identification number within a separate password-protected file that will not be linked to the study data. No information relating to participants’ identity will be included in the notes taken during interviews or focus groups. Written and oral reports of study findings will not contain any information that could potentially identify participants. Given the small number of participants from each group (PhD scholars, supervisors, PPI contributors), care will be taken when reporting results to ensure anonymity as far as possible, e.g. gender/age of participants will not be attached to quotes; potentially identifying comments will not be used as quotes. Because the CDA-MM is a distinct and unique programme, because of the small and finite number of study participants, and because participants are known to each other, there are limits on the extent to which the anonymity of participants can be ensured. Participants will be informed of this during the consenting process and at the start of interviews. Data associated with the study will be stored in a named folder on a secure drive at the National University of Ireland Galway, compliant with National University of Ireland Galway data protection policies. Once the study is completed, this data will be destroyed.

Ethical issues and ethical approval

The nine PPI panel members, four PhD scholars and nine PhD supervisors will potentially be sources of research data during the evaluation of the process and impact of embedding PPI training in the CDA-MM. Therefore, we obtained approval for the evaluation study from the National University of Ireland Galway Research Ethics Committee (8 August 2020, ref: 2020.08.006).

In designing this evaluation protocol, we considered potential ethical issues. The PPI panel will contribute to planning the logistics of the focus group, ensuring that the known burden associated with living with multimorbidity will be considered. We do not anticipate that there will be any disadvantages or risks to PPI contributors or PhD supervisors from participating in this study. The involvement of an independent researcher who is responsible for managing and analysing the data produced from scholar self-reflections will mitigate any potential risks arising from this source of data, and participants will have an opportunity to sense-check findings before the analysis is finalised.

A particular risk, however, that arises in this study relates to the nature of the CDA-MM and the small pool of people from which to recruit study participants. This means that participants (PhD scholars, PhD supervisors, and PPI contributors) are known to one another. In addition, the participants have pre-existing relationships with each other, e.g. PhD supervisor/PhD scholars, PhD scholar/PPI contributor, which may be regarded as ‘dependent or unequal relationships,’ and accordingly power differentials may exist amongst the participants. The decision to involve an independent researcher will facilitate open and honest discussion of the experience of using PPI and contribute to the alleviation any power differentials.

Dissemination of findings

We will publish an evaluation report. Additionally, we will publish the findings in a peer review journal using the Guidance for Reporting Involvement of Patients and the Public (GRIPP) tool.²³ The PPI contributors will be invited to co-deliver the findings of the evaluation study to people living with multimorbidity, other PPI contributors, researchers and members of the public.

Discussion

Over the past 10 years, the international literature evaluating PPI has more than tripled.²⁴ Demands for empirical evidence of the impact of PPI abound and a recent systematic review identified at least 65 frameworks developed to assess the nature and impact of PPI in health research.²⁵ While guided by the principles of INVOLVE,¹ our approach to embedding formal and experiential PPI training in a structured PhD programme is largely informed by experience, as opposed to an existing theory or framework.²⁶ Therefore, we elected not to adhere strictly to a single existing framework to explore process and impact in our evaluation study.

We agree with those who urge that public involvement is considered ‘not as an instrumental intervention, but a social practice of dialogue and learning between researchers and the public’²⁷ and accordingly engaged in such inter-stakeholder dialogue while planning this evaluation study. We contemplated incorporating inter-stakeholder dialogue during data collection; however, considering the finite sample and dual roles of all participants, we chose to maintain a clear distinction between the research participant role and stakeholder role.¹ In doing so, we aim to distinguish this research piece from our usual PPI activities, where we regularly engage in dialogue and learning through reflection amongst stakeholders, including during the development of this evaluation study, and following our regular PPI meetings.

We anticipate that the proposed evaluation will demonstrate how embedding formal and experiential PPI training in a doctoral programme can change the course of the research projects and the doctoral programme itself, as well as the impact on relevant stakeholders and their relationships and power balances.

Footnotes

Acknowledgements

We would like to thank members of the PPI panel and acknowledge their contribution to planning this evaluation study protocol and reviewing the Plain English Summary.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work is supported by a Health Research Board Ireland Collaborative Doctoral Award, grant number CDA-2018-003.

ORCID iDs

Bridget Kiely

Aisling Croke

Susan M Smith

References

INVOLVE. NIHR. 2021 [cited 2021 March 3]; https://www.invo.org.uk/. (2012; accessed March 2021).

Brett

Staniszewska

Mockford

, et al. Mapping the impact of patient and public involvement on health and social care research: a systematic review. Health Expect 2014; 17(5): 637–650.

Gradinger

Britten

Wyatt

, et al. Values associated with public involvement in health and social care research: a narrative review. Health Expect 2015; 18(5): 661–675.

Brett

Staniszewska

Mockford

, et al. A systematic review of the impact of patient and public involvement on service users, researchers and communities. Patient 2014; 7(4): 387–395.

Boaz

Biri

McKevitt

. Rethinking the relationship between science and society: has there been a shift in attitudes to patient and public involvement and public engagement in science in the United Kingdom? Health Expect 2016; 19(3): 592–601.

Biggane

Olsen

Williamson

. PPI in research: a reflection from early stage researchers. Res Involv Engagem 2019; 5(1): 35.

Coupe

Mathieson

. Patient and public involvement in doctoral research: Impact, resources and recommendations. Health Expect 2020; 23(1): 125–136.

Health Research Board Ireland. Public and patient involvement in research [Internet]. [cited 2021 March 3]. https://www.hrb.ie/funding/funding-schemes/public-and-patient-involvement-in-research/ (2018, accessed 3 March 2021).

Fortin

Bravo

Hudon

, et al. Prevalence of multimorbidity among adults seen in family practice. Ann Fam Med 2005; 3(3): 223–228.

10.

Fortin

Soubhi

Hudon

, et al. Multimorbidity’s many challenges. BMJ 2007; 334: 1016.

11.

Mair

Gallacher

. Multimorbidity: what next? Br J Gen Pract 2017; 67(659): 248–249.

12.

Mann

Chilcott

Plumb

, et al. Reporting and appraising the context, process and impact of PPI on contributors, researchers and the trial during a randomised controlled trial – the 3D study. Res Involv Engagem 2018; 4(1): 15.

13.

Markle-Reid

Ganann

Ploeg

, et al. Engagement of older adults with multimorbidity as patient research partners: lessons from a patient-oriented research program. J Multimor Comorb 2021; 11: 2633556521999508.

14.

The SPHeRE Programme | Structured Population and Health-services Research Education [Internet]. [cited 2021 March 26]. https://www.sphereprogramme.ie/ (2020; accessed 26 March 2021).

15.

Dawson

Ruddock

Parmar

, et al. Patient and public involvement in doctoral research: reflections and experiences of the PPI contributors and researcher. Res Involv Engagem 2020; 6: 23.

16.

Tomlinson

Medlinskiene

Cheong

V-L

, et al. Patient and public involvement in designing and conducting doctoral research: the whys and the hows. Res Involv Engagem 2019; 5(1): 23.

17.

Braun

Clarke

. What can “thematic analysis” offer health and wellbeing researchers? Int J Qual Stud Health Well-Being 2014; 9: 26152.

18.

Guest

Namey

Mitchell

. Collecting qualitative data: a field manual for applied research. Thousand Oaks, CA: SAGE, 2013.

19.

Gibbs

Unit

GBFE

. Learning by Doing: A Guide to Teaching and Learning Methods. Oxford: Oxford Polytechnic: FEU, 1988.

20.

Tessier

From field notes, to transcripts, to tape recordings: evolution or combination?

Int J Qualitat Methods 2012; 11(4): 446–460.

21.

Rutakumwa

Mugisha

Bernays

, et al. Conducting in-depth interviews with and without voice recorders: a comparative analysis. Qualitat Res 2020;20(5):565–581.

22.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3(2): 77–101.

23.

Staniszewska

Brett

Simera

, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 2017; 358: j3453.

24.

Boivin

Richards

Forsythe

, et al. Evaluating patient and public involvement in research. BMJ 2018; 363: k5147.

25.

Greenhalgh

Hinton

Finlay

, et al. Frameworks for supporting patient and public involvement in research: Systematic review and co-design pilot. Health Expect 2019; 22(4): 785–801.

26.

Moore

Audrey

Barker

, et al. Process evaluation of complex interventions: medical research council guidance. BMJ 2015; 350: h1258.

27.

Russell

Fudge

Greenhalgh

. The impact of public involvement in health research: what are we measuring? Why are we measuring it? Should we stop measuring it? Res Involv Engagem 2020; 6(1): 63.