Patient and public involvement in the design,administration and evaluation of patient feedback tools,an example in psychiatry: a systematic review and critical interpretative synthesis

Abstract

Background

Patient feedback is considered integral to healthcare design, delivery and reform. However, while there is a strong policy commitment to evidencing patient and public involvement (PPI) in the design of patient feedback tools, it remains unclear whether this happens in practice.

Methods

A systematic review using thematic analysis and critical interpretative synthesis of peer-reviewed and grey literature published between 2007 and 2017 exploring the presence of PPI in the design, administration and evaluation of patient feedback tools for practising psychiatrists. The research process was carried out in collaboration with a volunteer mental health patient research partner.

Results

Fourteen articles (10 peer-reviewed, four grey literature) discussing the development of nine patient feedback tools were included. Six of the nine tools reviewed were designed from a professional perspective only. Tool content and its categorization primarily remained at the professional’s discretion. Patient participation rates, presence of missing data and psychometric validation were used to determine validity and patient acceptability. In most instances, patients remained passive recipients with limited opportunity to actively influence change at any stage. No article reviewed reported PPI in all aspects of tool design, administration or evaluation.

Conclusions

The majority of patient feedback tools are designed, administered and evaluated from the professional perspective only. Existing tools appear to assume that: professional and patient agendas are synonymous; psychometric validation is indicative of patient acceptability; and psychiatric patients do not have the capacity or desire to be involved. Future patient feedback tools should be co-produced from the outset to ensure they are valued by all those involved. A reconsideration of the purpose of patient feedback, and what constitutes valid patient feedback, is also required.

Keywords

health policy mental health patient and public involvement patient experience psychiatry systematic review

Introduction

Patient feedback is considered integral to professional development, healthcare service design, delivery and reform.^1–4 However, despite extensive policy requirements, recent research challenges the perceived efficacy, validity and acceptability of existing patient feedback tools.^5,6 Furthermore, while policy-driven discourses promote patient and public involvement (PPI) in the design, administration and evaluation of such tools,⁷ critical exploration of this in practice is severely limited, particularly in the context of psychiatry.^8–10

Currently, patient feedback is typically collected through validated questionnaires or surveys.^11,12 However, some doctors and patients remain sceptical about their reliability due to: the adoption of generic questionnaires that intentionally, or unintentionally, exclude vulnerable patient groups; the inability of existing tools to ensure sufficient patient representation; biased patient selection, or choosing the ‘right’ patient to participate; and biased patient responses, such as patients not providing honest feedback due to a fear of reprisals.⁶ Building on these concerns, Sir Keith Pearson, the lead for a recent independent review of medical regulation in the United Kingdom expressed some doubt that existing questionnaires ‘provide sufficient quality and breadth of information to enable a doctor to reflect properly on their interaction with patients.’⁵ The practicality and acceptability of existing patient feedback tools has therefore been contested from both a patient and professional perspective, inhibiting their adoption, value and subsequent benefit to both patients and the profession.^5,6

Furthermore, in addition to the generic issues described above, psychiatrists report specific concerns in regards to patient feedback. Psychiatrists frequently express concerns about a patient’s ability to ‘know their own mind’,^13,14 reflecting issues of capacity. In some instances, patients who ‘lack capacity’ are excluded from patient feedback opportunities. While substantial improvements have been made with regard to shared decision-making processes,¹³ evidence suggests mental health patients are frequently excluded, or at best under-represented in experiencing patient feedback opportunities.¹⁵

Such findings are concerning given the valuable learning opportunities attributed to this activity.^16,17 Pertinent to the context of this review, some authors argue that exploring patient feedback in the context of mental health is of paramount importance as patients are more likely to maintain contact with medical services if they are satisfied with their care.¹⁸ This in turn has implications for reducing clinical relapse incidents and hospital admission rates, affecting patient wellbeing and resource expenditure.¹⁸ Understanding and acknowledging the patient perspective in the design, administration and evaluation of patient feedback tools is therefore imperative as the continued use of ineffective methods has been shown to be detrimental to the overall quality of care received.¹⁹ Despite this, critical exploration of PPI in the design of patient feedback tools is limited, particularly within the field of psychiatry.^8–10

This review therefore aims to critically examine how, if at all, patients and the public are involved in the design, administration and evaluation of patient feedback tools for practising psychiatrists. By identifying any potential flaws, assumptions and contradictions (the central aim of critical interpretative synthesis) that underpin existing feedback tools, practical and policy-related implications can be identified. Although psychiatry is used as an example for the purposes of this review, its findings and conclusions are likely to have valuable lessons for the use of PPI in the design of patient feedback tools in healthcare more generally.

For the purposes of this review, the term ‘patient’ includes service-users, consumers, clients, carers and family members although the important distinctions between these terms are acknowledged.

Methods

This systematic review followed the preferred reporting items for systematic reviews and meta-analysis (PRISMA) flow diagram²⁰ and guidance set out by the Centre for Reviews and Dissemination.²¹ The review’s protocol is registered on the PROSPERO database, registration number CRD42016050533.

All research was conducted in collaboration with a volunteer mental health patient research partner who has extensive experience of receiving psychiatric care. Published principles of PPI were followed to support this involvement.²²

Search strategy

Peer-reviewed literature

Search terms listed in Table 1 were designed and reviewed using the Peer Review of Electronic Search Strategies (PRESS) guidance.²³ A scoping exercise revealed the need to produce an extensive list of patient synonyms to remain sensitive to different contexts (Table 1). As advised by an information specialist, agreed search terms were used to systematically search: MEDLINE, PubMed, PsycINFO, Embase, CINAHL and Cochrane Library databases for articles published in the English language. Database searches were supplemented by reference list searches of included studies. Two independent reviewers conducted the research process.

Table 1.

Search term strategy.

Setting: Psychiatric OR psychiatrist OR psychiatry OR inpatient OR “mental health” NOT child

AND

Perspective: Patient OR user OR service user OR service-user OR client OR consumer OR survivor OR representative(s) OR citizen OR family OR relative OR carer

AND

Intervention: “multisource feedback” OR “multi-source feedback” OR “360 degree feedback” OR “360 degree evaluation” OR MSF OR “performance feedback” OR “patient feedback” OR “patient experience” OR “patient survey” OR “patient questionnaire” OR “online feedback”

AND

Evaluation: “professional development” OR regulation OR behaviour OR attitude OR change OR improve OR quality of care OR learn OR reflect OR impact OR outcome OR “patient safety”

Table 2.

Inclusion criteria form.

Is the study published between 2007 and 2017?Yes (proceed) No (reject)

Is the study available in English?Yes (proceed) No (reject)

Does the study talk about the experience and/or use of patient feedback for a practising psychiatrist?Yes (proceed) No (reject)

Does the study solely focus on psychiatric care associated with children/dementia/Alzheimer’s or learning difficulties?Yes (reject) No (retain)

Grey literature

Grey literature, defined as ‘that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers,’²⁴ was also searched using Google to ensure sufficient coverage. Screening was limited to the first 10 pages in order to maintain a manageable sample size. The peer-reviewed search strategy was also used to search Google Scholar.

Inclusion criteria

Articles were reviewed independently by two authors using a two-stage process. Firstly, all identified articles were screened by their title and abstracts using a pre-defined inclusion criteria form (Table 2). Rayyan, a web application for systematic reviews was used to facilitate this process.²⁵ Where a decision could not be made, the full article was retrieved. Full texts of included articles were then reviewed for inclusion. Discrepancies would have been resolved with reference to a third reviewer. This process was not required.

To develop a manageable focus, certain studies were excluded. Articles not in the English language were excluded as the authors could not ensure an accurate translation. The date parameters of 2007–2017 were selected to ensure only contemporary information was included. Studies with a specialty focus of child, dementia/Alzheimer’s or learning difficulties were excluded as these areas were likely to require tailored patient feedback activities beyond the remit of this review. Any study design except opinion, commentary or letter articles was included. For clarity, articles that described the evaluation or assessment of mental health services, but did not differentiate individual psychiatrist involvement, were excluded. Figure 1 documents the inclusion decisions.

Figure 1.

PRISMA peer review inclusion process.PRISMA: preferred reporting items for systematic reviews and meta-analysis.

Data analysis and synthesis

Data were thematically analysed using an inductive approach and synthesized using critical interpretative synthesis (CIS).^26,27 Two researchers were involved in the coding and analysis of results. Relevant text under the headings ‘results’, ‘discussion’ and/or ‘conclusions’ were extracted electronically with thematic analysis supported through the use of Nvivo 11 software.²⁸ Two researchers initially reviewed two papers line by line to develop a comprehensive coding framework. The framework was then used to individually analyse all included studies and to iteratively compare emerging themes across studies. The themes were then synthesized using CIS.

CIS is an adaptation of meta-ethnography and borrows techniques from grounded theory.²⁹ CIS was chosen as the synthesis methodology for this review due to its ability to integrate both qualitative and quantitative evidence, enabling theory generation.³⁰ It incorporates conventional systematic review methodology with traditional techniques of qualitative enquiry, enabling the aggregation and examination of existing literature as a corpus of information, and generation of new perspectives through the critical interrogation of existing contradictions, flaws and assumptions between, not within individual studies. Essentially, CIS seeks to problematize the literature. One of its defining features is its ability to generate synthetic constructs, a ‘third order construct’, the result of transforming underlying evidence into new conceptual forms, and synthesizing argument(s).²⁷ As Dixon-Woods explains: ‘this argument integrates evidence … into a coherent theoretical framework … to provide more insightful, formalized, and generalizable ways of understanding a phenomenon’.²⁷ The CIS stages set out by Flemming³⁰ were followed. The flaws, assumptions and contradictions identified through this review are used to structure its discussion.

Quality assessment

The criteria described by Buckley et al.³¹ were used to quality-appraise included peer-reviewed studies. In line with previous research,³² studies scoring seven or above were regarded as high quality. Finally, sensitivity analyses, a form of analysis that tests for the effect on conclusions drawn when lower quality studies are excluded, were also conducted. Such analyses are considered an important focus of any qualitative synthesis method warranting their inclusion.³³ Consistent with the synthesis method chosen, conceptual relevance took preference over methodological rigour.²⁷ Quality appraisals of included studies were conducted independently by two reviewers until consensus was achieved.

Results

Study characteristics

Peer-reviewed literature

Database and reference list searches of included articles led to the inclusion of 10 peer-reviewed studies discussing nine unique patient feedback tools (Figure 1). Studies were published primarily in the United Kingdom (n = 5), with others published in Canada (n = 1), Sweden (n = 1), the United States (n = 1), Italy (n = 1) and France (n = 1).

Table 3.

Included study characteristics.

Author	Study location	Study population	Intervention	Setting	Design perspective	Assessment areas	No. of items	Scale used	Quality appraisal^a
Peer-reviewed literature
Schröder et al.⁴⁴	Sweden	116 patients	Quality in psychiatric care (QPC)	In-patient psychiatry	Patient with limited professional involvement – patient interviews	Patient dignity and respect, security, participation, recovery and environment	69	4-item Likert scale (1 = totally disagree, 4 = totally agree). Not applicable option.	8
Violato et al.³⁷	Canada	101 psychiatrists, 2456 patients	CPSA-PAR	Psychiatry	Professional – working group inclusive of psychiatrists and other physician specialists. No patient involvement discussed	Medical knowledge and skills, attitudes and behaviour, professional responsibilities, practice improvement activities, administrative skills and personal health	40	5-point rating scale (1 = strongly disagree, 5 = strongly agree). Unable to assess option.	9
Campbell et al.⁴⁰	UK	13,754 patients, 380 participant doctors	GMC patient and colleague questionnaire	Doctor performance including psychiatry	Professional – authoritative guidelines	Good medical practice	18	5-point Likert scale (1 = poor, 5 = very good or 1 = strongly disagree, 5 = strongly agree) and two binary responses (yes/no)	6
Lelliott et al.⁴²	UK	347 consultant psychiatrists, 6657 patients	ACP 360	Psychiatry	Professional – interviews with 24 specialist mental healthcare workers only	Communication, availability, emotional intelligence, decision making, relationship with patients, relationships with patients’ relatives, partners and carers	17	6-point scale (1 = very low, 6 = excellent)	7
Mason et al.³⁸	UK	554 consultants with over 16,000 replies	360 degree appraisal	Doctor performance including psychiatry	Professional and existing literature	Respect and consideration, involvement, clarity of communication, carer/family involvement, information provision	5	4-point scale (1 = poor, 4 = very good). Unable to comment option	5
Stewart et al.³⁹	US	149 pre-implementation and 137 post implementation surveys	PSQ-18	Psychiatric out-patients	Professional and existing literature	“General satisfaction”, “technical quality”, “interpersonal manner”, “communication”, “time spent with doctor”, “anxiety”, “computer use” and “confidentiality”	23	5-point Likert scale (1 = strongly agree, 5 = strongly disagree)	8
Campbell et al.⁴¹	UK	1065 doctors, 30,333 patients	GMC patient questionnaire	Doctor performance including psychiatrists	Professional – authoritative guidelines	Good medical practice	9	5-point Likert scale (1 = poor, 5 = strongly agree). Not applicable or do not know option available	7
Laugharne and Pant⁴³	UK	7500+ patient surveys	Care Quality Commission in-patient survey	In-patient satisfaction with psychiatrists	Not discussed	Listening abilities, time, confidence and trust, respect and dignity	4 items of interest	Not discussed	6
Barbato et al.⁸	Italy	204 people with severe mental disorders	Quality assessment of mental health care by people with severe mental disorders	Public agency providing mental health care in Tuscany	Literature review and patient-review of evaluation tools then discussed and developed by consumer focus groups (n = 204)	Relations with the professional (behaviour, accessibility, competence), interventions received, environment and facilities, shared decision making, organization aspects, waiting time on the phone, home visits and help in an emergency	45	5 point Likert scale associated with smiles (1= very positive, 6 = very negative) and yes/no responses	5
Zendjidjian et al.⁴⁵	France	270 responders	SATISPSY-22	Psychiatry	Patient with limited professional involvement – 80 interviews with 80 hospitalized psychiatric patients. Professional steering committee	Staff, quality of care, personal experience, information, activity and food	22	5-point Likert scale (1 = extremely less than expected, 5 = better than expected)	8
Grey Literature
Royal College of Psychiatrists, 2011	UK	NA	ACP 360	Psychiatry	Professional – interviews with 24 specialist mental healthcare workers only	Communication, availability, emotional intelligence, decision making, relationship with patients, relationships with patients’ relatives, partners and carers	17	6-point scale (1 = very low, 6 = excellent)	-
Royal College of Psychiatrists, 2014	UK	NA	ACP 360/GMC	Psychiatry	Professional	Good medical practice	Not discussed	Not discussed	-
Academy of Medical Royal Colleges	UK	NA	Patient feedback	Doctor performance including psychiatry	Professional – authoritative guidelines, good medical practice	Good medical practice	Not discussed	Not discussed	-
Royal College of Psychiatrists, 2018	UK	NA	ACP 360	Psychiatry	Professional	Good medical practice	15	6-point scale (1 = very low, 6 = very high)	-

^aQuality appraisal score out of 11: 7 or above indicates high quality.

Grey literature

107 unique documents were identified through grey literature searching leading to the inclusion of four relevant documents (Figure 2).^7,34–36 These are presented alongside the peer-review findings.

Figure 2.

PRISMA grey literature inclusion process.PRISMA: preferred reporting items for systematic reviews and meta-analysis.

Fourteen articles (10 peer-reviewed, four grey literature) discussing the development of nine patient feedback tools were therefore included. Table 3 lists the characteristics of included studies.

Study quality

Six peer-reviewed articles were appraised as high quality (Table 3). Sensitivity analyses indicated that the exclusion of studies assessed as low quality (an appraisal score of below seven), did not affect the conclusions drawn. All studies were therefore retained in order to be inclusive of the available literature.

Results are presented in order of some of the generic steps involved in tool development:

Step 1: Tool design, e.g. whose perspective will the tool be designed from? What information will the content of the tool be designed from? Which domains and questions will the tool cover? How will the questions be phrased? And how will the respondents answer the questions planned, what response methods will they use?

Step 2: Tool administration, e.g. who will administer the tool? How?

Step 3: Tool evaluation, e.g. piloting and validation.

PPI in patient feedback tool design

Design perspective

Six of the nine tools reviewed were designed from a professional perspective only.^7,34–42 For example, the ACP 360, a tool used by the Royal College of Psychiatrists in the UK, is described as, ‘the only tool designed and validated for psychiatrists, by psychiatrists’.³⁶ One article reviewed did not discuss the design perspective and was assumed to be from that of the professional.⁴³ Professional perspectives were often obtained through professional steering or working groups, and pre-defined authoritative guidelines, such as Good Medical Practice.⁴⁰ One tool was designed from a combination of the patient and professional perspective.⁸ Two tools reviewed were also designed from the patient perspective with minimal professional input (Table 3).^{44
^,
45}

Tool content

Where discussed, the generation of tool criteria was dominated by professional input (n = 3/5).^37,40,42 Authors of one article reported that the only criteria included were those that, ‘the regulatory authority and the physicians themselves believed to be important’ (p.529).³⁷ In most instances, professional opinion and authoritative guidelines were taken as a proxy measure for the patient voice.

However, in two cases tool content was not driven by the profession alone.^44,45 Schröder et al.⁴⁴ interviewed 20 patients, of whom 17 had in-patient psychiatric care experience, and developed five descriptive categories of care quality relating to: patient dignity and respect; a patient’s sense of security, for example, trust; patient participation in care; patient recovery, for example, provision of supportive guidance and opportunities for post-care follow-up; and quality of the patient’s care environment, for example, personal space and aesthetics. Another study used face-to-face, semi-structured interviews with 80 in-patients with various diagnoses, including schizophrenia, bipolar disorder and mental and behavioural disorders due to psychoactive substance use.⁴⁵ Patient interview data were then used to determine the question design and response scales used.

However, while often described as patient-generated, the categorization or analysis of suggested content was ultimately at the professionals’ discretion.^42,45 No article reviewed discussed the analysis of data in collaboration with patients or members of the public. A tool’s content was a further area in which professional opinion typically superseded patient contributions.

Finally, the number of domains covered in patient feedback tools varied from four⁴³ to eight.⁴⁵ The number of questions used to assess these domains also varied (n = 5–69). No papers reported patient involvement in discussions around the number of domains included or number of questions asked. No clear rationale behind the variability of included questions or domains was identified.

Question design

PPI in the formatting of proposed questions was mixed. Where drawn upon, patient involvement was favourably described.⁸ For example, Barbato et al.⁸ acknowledged how participating patients prompted the importance of a more direct and friendly style of questioning, for example, ‘do you get on well with your psychiatrist?’ When patients were not involved, authors reported a high ‘unable to rate’ response⁴² or patient response confusion.⁴⁰ For example, despite being tested for ‘face validity and feasibility by eight consultants and their colleagues only’, one acknowledged limitation of the tool developed by Lelliot et al.⁴² was the ‘substantial number of patients unable to rate some items.’ Campbell et al.⁴⁰ also identified one item that caused some patient confusion. When responding to the statement ‘I have no reservation about seeing this doctor again’, 87 respondents altered their initial binary response, i.e. ‘yes’ or ‘no’, following a misunderstanding or misreading of the question.⁴⁰ This statement (related to reservation) also had a substantially higher proportion of adverse ratings in comparison to other questions asked.⁴⁰

Response scales

Response scales were defined by patients in two cases.^8,45 Following patient suggestions, Barbato et al.⁸ used smiley faces in conjunction with a five point Likert scale (very positive–very negative). Zendjidjian et al.⁴⁵ adopted the language used by patients in preceding interviews as the response scale modalities for a five point Likert scale, from ‘extremely less than expected’ to ‘better than expected’ to facilitate patient understanding. No other articles discussed the decision process of response scale agreement in collaboration with patients or members of the public.

PPI in patient feedback tool administration

One article described directly involving patients in the tool’s delivery.⁸ Barbato et al. employed six patients to deliver the questionnaire, offer assistance if required, and to collect completed tools. Professional involvement in this process was deliberately kept to a minimum to limit possible conformity and overrating of positive answers, i.e. social desirability bias.⁸ The low refusal rate of 12% and wide patient representation achieved were directly attributed to PPI by the article’s authors.⁸

PPI in patient feedback tool evaluation

Piloting

Four articles reported the involvement of patients in the piloting of proposed tools.^38,39,44,45 In one instance, six patients recruited through a local patients’ association with experience of psychiatric care were asked to complete the proposed questionnaire at home, and then evaluate it using a piloted checklist.⁴⁴ Participating patients were required to assess each tool item in terms of: importance in care quality (1 = very important, 5 = of little importance); clarity, for example, clear and easy to understand, acceptable, or unclear and hard to understand; general structure; relevance; and usefulness.⁴⁴ Participant evaluations were then discussed with researchers either face-to-face or by phone, leading to the exclusion of 58 items due to: perceived lack of importance (n = 10); emotionally charged or overlapping content (n = 20); and low levels of perceived importance (n = 28). Two other articles reported the reduction of question items, and re-wording of questions to facilitate understanding, following patient involvement in the piloting stage.^38,39

However, in most cases, the piloting of tools was assessed by the profession with ‘their patients’ (p.157)⁴² as passive recipients. In one instance, tool appropriateness was assessed by ensuring that ‘every physician to be assessed … was provided an opportunity to review the questionnaires and provide feedback that was incorporated into the final [version]’ (p.256).³⁷ Two reviewed articles also reported the use of informal feedback by participating doctors only.^36,40

Validation

With the exception of one tool not yet validated,³⁹ the psychometric properties of a patient feedback tool were often used to determine all aspects of validity and acceptability. For example, as stated by Campbell et al.,⁴⁰ ‘analysis of psychometric properties showed that both surveys were acceptable to patients and colleagues’ (p.192). Tool acceptability was also assessed through the examination of patient participation and levels of missing data (n = 5/8).^37,40,42,45 One study reported, ‘The high rate of return from patients suggests that raters did not find the questionnaire over-burdensome’ (p.159).⁴² Patient participation or response omission was therefore used as a default measure of tool acceptability, regardless of which questions were ignored or responses provided.

Finally, although large numbers of patients participated in the validation of proposed tools,⁴⁰ validation was often passive with limited opportunities to influence change. In some instances, physicians selected their own patients to be involved in the tool’s evaluation, introducing possible bias.⁴²

Discussion

This review addressed limitations of existing research by exploring the presence of PPI in the design, administration and evaluation of patient feedback tools for practising psychiatrists. The discussion below presents potential flaws, assumptions and contradictions in the arguments or reasoning underpinning existing tool design, administration and evaluation. A synthesizing argument is then generated that contributes to policy debates in healthcare more broadly and highlights a number of solutions for consideration.

Identified flaws

Despite a strong policy commitment to PPI,⁷ existing patient feedback tools primarily rely on professional opinion. Professional discretion often supersedes patient input, or remains restricted, with the ability to influence change severely limited. While this may reflect a need for ‘scientific or statistical’ assurances as later discussed, it may also reflect a power imbalance between patients and the profession, and the traditional approach of keeping these two communities distinct.⁴⁶ A lack of consistent PPI across the tool development stages may also reflect acknowledged difficulties in conducting effective PPI.²² However, difficulties encountered should not be used to deter PPI efforts.

Existing assumptions and contradictions

In addition to limited PPI, existing patient feedback tools are also potentially undermined by three key assumptions. Firstly, tool developers assume that professional and patient agendas are aligned. Three included studies demonstrate that this is not the case.^8,44,47 A lack of concordance between patients and psychiatrists has been reported, as has a lack of commonality between carer, patient and psychiatrist experiences.⁸ Lelliot et al.⁴² acknowledged that colleagues and patients are two independent groups whose perceptions and experiences of consultants are derived from different perspectives. It is therefore imperative that those responsible for designing existing patient feedback tools acknowledge the disparity between these populations and do not take one to be representative of the other, i.e. do not assume colleague interests to be indicative of patient desires, or vice versa. Secondly, those responsible for evaluating patient feedback tools assume psychometric validation, patient participation, or high response rates are indicative of patient acceptability. We argue that to determine patient acceptability from these measures alone is reductionist and jeopardizes tool validity.⁴⁴ For example, if patients perceive the content or process of the tool to be irrelevant, they are unlikely to be motivated to complete it or provide accurate results, undermining the tools’ content and response process validity. While important, the current reliance on psychometric validation alone is therefore unfavourable.

Finally, those looking to receive patient feedback sometimes assume that psychiatric patients do not have the capacity or desire to be involved. This assumption is directly challenged by a number of the articles reviewed.^8,44,45,47 An alternative interpretation of the reportedly low levels of patient engagement is that it is the tool – and the traditionally exclusive approach to its design, administration and evaluation – that lacks the capacity to facilitate effective patient involvement, not the participating individual. For example, while patients may not have the required capacity at a given point in time, this is unlikely to be true for the entirety of their patient journey. The opportunity to provide patient feedback should therefore be patient-dependent not physician-dependent.

Potential solutions

We propose three solutions to address the identified flaws, assumptions and contradictions discussed. Firstly, co-producing tools from the outset is imperative. By its nature, co-producing tools may help to address acknowledged power imbalances and the underpinning assumptions that currently exist. Secondly, clarification of the intended purpose of patient feedback is required. The purpose should be distinguished as either summative, that is, a point in time evaluation of performance against existing standards or benchmark, or formative, that is, an ongoing monitoring of performance. Likewise, the intended focus of feedback should be specified.⁴⁰ Failure to do so is likely to create unnecessary confusion and development of poorly planned tools. Although often used interchangeably, patient experience, patient satisfaction and patient expectations, each term represents discrete concepts that require different questions to be asked. As described by Gill et al.,⁴⁹ patient experience explores the specific experience of individuals, while patient satisfaction evaluates those individual experiences. In contrast, patient expectation is a measure of how well a patient’s experience met their individual, and at times, highly variable expectations. If it is patient experience that regulators, employers or healthcare organizations are looking to explore, then the overriding emphasis placed on psychometric validation must be reviewed. While it is important to test the psychometric properties of tools if they are to be used as a summative assessment, questions should be asked as to what constitutes valid patient feedback from both a professional and patient perspective. If the acceptability and feasibility of patient feedback tools continue to be judged on their internal structure alone, we are at risk of developing tools that ignore other influential variables such as narrative comments⁵⁰ and subjective experiences capable of facilitating learner engagement and reflection.⁵¹ Is ‘valid’ patient feedback a psychometrically sound tool that can be ‘gamed’ by both patients and physicians,⁵² or is a broader understanding of validity that relates to its ability to encourage actionable and beneficial change regardless of its source, format or design now required? Experience is valid for the person involved. As stated by Downing et al.,⁴⁸ validity is comprised of five domains including: internal structure, (psychometric validation); response process, (how participants involved respond to the assessment); and consequence, (the intended and unintended consequences of the intervention/assessment). Typically, those involved in the design, administration and evaluation of patient feedback tools have focused on the internal structure only. Emerging evidence suggests some patients alter their response scores due to anticipated fear of reprisals.⁵³ Once one validity domain is undermined, so is that tool’s ability to be used as an equitable summative assessment tool. This reiterates the need to clarify the intended purpose of patient feedback. Is it formative or summative. Why are we collecting patient feedback? What do we intend to do with it? If we are trying to achieve enhanced reflection, quality improvement and professional development through the exploration of patient experience, then traditional notions of validity (that is, psychometric validation only), arguably become redundant. As a result, in contrast to statements such as ‘high quality feedback will only be achieved if the instruments have good psychometric properties,’⁵⁴ the authors of this review conclude it is the ability of patient feedback tools to encourage reflective practice and actionable change that determines its level of quality, with feedback tools that have demonstrable PPI throughout their development now requiring more attention, support and development.

Finally, the existing culture of organizations and professionals arguably restricts patient involvement and subsequent acceptance. Cultures across the education continuum favour objective and behavioural criteria over subjective experience. Exclusion, or rejection, of feedback can lead to patient experience being medicalized. This situation may be further exacerbated by application of a professional framework that is perhaps currently misaligned with patient perspectives and desires.⁵⁵

Implications

Although primarily focused on psychiatry, this review has clear practical and policy related implications for other healthcare disciplines and services more generally. Firstly, patient feedback tools should be co-produced from the outset to ensure patient and professional desires are acknowledged. Secondly, the intended purpose and scope, i.e. patient experience, satisfaction or expectation, must be clearly articulated prior to the design and evaluation of any patient feedback tool. Thirdly, patient participation, psychometric validation, or absence of missing data should not be used to determine patient acceptability alone. Finally, those working in healthcare services should reconsider what constitutes valid patient feedback for both patients and physicians. Ultimately, what is it we are trying to achieve when asking for patient feedback? If it is to enhance patient experience, patient safety and quality of care then we must ask what matters most to patients to ensure services and individuals deliver this.

Strengths and limitations

Strengths of this research include its rigorous application of accepted guidelines,^20,21 collaboration with a patient research partner and inclusion of grey literature. However, its limitations must also be acknowledged. Most of the evidence reviewed relied primarily on volunteer samples.⁴⁰ Results may not therefore be representative of the wider population. In some cases, doctors also chose their own patients, introducing possible patient selection and response bias. This review also only included articles published in the English language. Potential publication bias is therefore acknowledged. Finally, included articles rarely described the level or detail of PPI initiatives. The review is therefore reliant on available information at the time of publication.

Conclusion

While it is generally agreed that patient feedback tools should incorporate PPI throughout their development, current methods rely primarily on the professional perspective only. Existing tools are therefore undermined by a number of identified flaws, assumptions and contradictions, including the beliefs that: professional and patient agendas are synonymous; psychometric validation is indicative of patient acceptability; and psychiatric patients do not have the capacity or desire to be involved. A reconsideration of what constitutes valid patient feedback is required. Future patient feedback tools should also be co-produced from the outset to ensure they are valued by all those involved.

Footnotes

Acknowledgements

The authors would like to thank John Donovan for his continued enthusiasm, commitment and captivating personality, and Dr Ellen Wilkinson for her invaluable feedback on draft iterations of this paper.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Sheard

Marsh

O'Hara

et al . The patient feedback response framework – understanding why UK hospital staff find it difficult to make improvements based on patient feedback: a qualitative study. Soc Sci Med 2017; 178: 19–27.

Reinders

Ryan

Blankenstein

et al . The effect of patient feedback on physicians' consultation skills: a systematic review. Acad Med 2011; 86: 1426–1436.

Bodur

Filiz

A survey on patient safety culture in primary healthcare services in Turkey, Int J Qual Health Care 2009: 21: 348–355.

Altmiller

The role of constructive feedback in patient safety and continuous quality improvement.

Nurs Clin N Am 2012; 47: 365–374.

Pearson

Taking revalidation forward: improving the process of relicensing for doctors, 2017, p.80.

Archer

Cameron

Laugharn

et al . Interim report, 2016, http://www.gmc-uk.org/UMbRELLA_interim_report_FINAL.pdf_65723741.pdf

Royal College of Psychiatrists. Supporting information for appraisal and revalidation: guidance for psychiatrists, 2014, http://www.rcpsych.ac.uk/usefulresources/publications/collegereports/cr/cr194.aspx

Barbato

Bajoni

Rapisarda

et al . Quality assessment of mental health care by people with severe mental disorders: a participatory research project. Community Ment Health J 2014; 50: 402–408.

Delaney

Johnson

Fogg

Development and testing of the combined assessment of psychiatric environments: a patient-centered quality measure for inpatient psychiatric treatment. J Am Psychiatr Nurses Assoc 2015; 21: 134–147.

10.

Bjertnaes

Iversen

Kjollesdal

PIPEQ-OS – an instrument for on-site measurements of the experiences of inpatients at psychiatric institutions. BMC Psychiatry 2015; 15: 234.

11.

Campbell

Narayanan

Burford

et al . Validation of a multi-source feedback tool for use in general practice. Educ Prim Care 2010; 21: 165–179.

12.

Campbell

Wright

GMC multi-source feedback questionnaires. Interpreting and handling multisource feedback results: guidance for appraisers, 2012.

13.

Tait

Lester

Encouraging user involvement in mental health services. Adv Psychiatr Treat 2005; 11: 168–175.

14.

Goodwin

A qualitative analysis of the views of in-patient mental health service users. J Ment Health 1999; 8: 43–54.

15.

Gayet-Ageron

Agoritsas

Schiesari

et al . Barriers to participation in a patient satisfaction survey: who are we missing? PLoS ONE 2011; 6: e26852.

16.

Crawford

Rutter

Manley

et al . Systematic review of involving patients in the planning and development of health care. BMJ 2002; 325: 1263.

17.

Armstrong

Herbert

Aveling

et al . Optimizing patient involvement in quality improvement. Health Expect 2013; 16: e36–e47.

18.

O'Regan

Ryan

Patient satisfaction with an emergency department psychiatric service. Int J Health Care Qual Assur 2009; 22: 525–534.

19.

Thornicroft

Brohan

Kassam

et al . Reducing stigma and discrimination: candidate interventions. Int J Ment Health Syst 2008; 2: 3.

20.

Moher

Liberati

Tetzlaff

et al . Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 2009; 151: 264–269.

21.

Khan

Ter Riet

Glanville

et al. Undertaking systematic reviews of research on effectiveness: CRD's guidance for carrying out or commissioning reviews. NHS Centre for Reviews and Dissemination, 2001.

22.

Baines

Regan de Bere . Optimizing patient and public involvement (PPI): identifying its “essential” and “desirable” principles using a systematic review and modified Delphi methodology. Health Expect 2018; 21: 327–335.

23.

Sampson

McGowan

Cogo

et al . An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol 2009; 62: 944–952.

24.

In: Fourth international conference on grey literature: new frontiers in grey literature, Washington, DC, 4–5 October 1999. Grey Literature Network Service.

25.

Mourad

Hossam

Zbys

et al . Rayyan – a web and mobile app for systematic reviews, https://rayyan.qcri.org/ (2016, accessed 30 November 2016).

26.

Gysels

Evans

Higginson

IJ.

Patient, caregiver, health professional and researcher views and experiences of participating in research at the end of life: a critical interpretive synthesis of the literature. BMC Med Res Methodol 2012; 12: 1–17.

27.

Dixon-Woods

Cavers

Agarwal

et al . Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol 2006; 6: 1–13.

28.

NVivo 11. NVivo qualitative data analysis software. NVivo Qualitative data analysis Software; QSR International Pty Ltd. Version 11, 2012. http://www.qsrinternational.com/nvivo/support-overview/faqs/how-do-i-cite-nvivo-for-mac,-nvivo-11-for-windows

29.

Glaser

Discovery of grounded theory: strategies for qualitative research. New York: Routledge, 2017.

30.

Flemming

Synthesis of quantitative and qualitative research: an example using critical interpretive synthesis. J Adv Nurs 2010; 66: 201–217.

31.

Buckley

Coleman

Davison

et al . The educational effects of portfolios on undergraduate student learning: a best evidence medical education (BEME) systematic review. BEME Guide No. 11. Med Teach 2009; 31: 282–298.

32.

Tai

Molloy

Haines

et al . Same-level peer-assisted learning in medical clinical placements: a narrative systematic review. Med Educ 2016; 50: 469–484.

33.

Thomas

Harden

Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol 2008; 8: 1–10.

34.

Royal College of Psychiatrists. Revalidation guidance for psychiatrists. London, UK: Royal College of Psychiatrists, 2011, pp.1–56.

35.

Academy of Medical Royal Colleges. Multisource feedback, patient surveys and revalidation: report and recommendations from the MSF Work Group of the Academy of Medical Royal Colleges, pp.1–14.

36.

Royal College of Psychiatrists. Why choose ACP 360?, http://www.rcpsych.ac.uk/workinpsychiatry/qualityimprovement/acp360/whychooseacp360.aspx (2017, accessed 7 July 2016).

37.

Violato

Lockyer

Fidler

Assessment of psychiatrists in practice through multisource feedback. Can J Psychiatry 2008; 53: 525–533.

38.

Mason

Power

Parker-Swift

et al . 360-degree appraisal: a simple pragmatic solution. Clin Governance 2009; 14: 295–300.

39.

Stewart

Kroth

Schuyler

et al . Do electronic health records affect the patient-psychiatrist relationship? A before & after study of psychiatric outpatients. BMC Psychiatry 2010; 10: 3.

40.

Campbell

Richards

Dickens

et al . Assessing the professional performance of UK doctors: an evaluation of the utility of the General Medical Council patient and colleague questionnaires. Qual Saf Health Care 2008; 17: 187–193.

41.

Campbell

Roberts

Wright

et al . Factors associated with variability in the assessment of UK doctors' professionalism: analysis of survey results. BMJ 2011; 343: d6212–d6d12.

42.

Lelliott

Williams

Mears

et al . Questionnaires for 360-degree assessment of consultant psychiatrists: development and psychometric properties. Br J Psychiatry 2008; 193: 156–160.

43.

Laugharne

Pant

Sector and functional models of consultant care: in-patient satisfaction with psychiatrists. Psychiatrist 2012; 36: 254–256.

44.

Schröder

Wilde Larsson

Ahlström

Quality in psychiatric care: an instrument evaluating patients' expectations and experiences. Int J Health Care Qual Assur 2007; 20: 141–160.

45.

Zendjidjian

Auquier

Lancon

et al . The SATISPSY-22: development and validation of a French hospitalized patients' satisfaction questionnaire in psychiatry. Eur Psychiatry 2015; 30: 172–178.

46.

Laugharne

Priebe

Trust, choice and power in mental health. Soc Psychiatry Psychiatr Epidemiol 2006; 41: 843–852.

47.

Boyer

Lançon

Baumstarck

et al . Evaluating the impact of a quality of life assessment with feedback to clinicians in patients with schizophrenia: randomised controlled trial. Br J Psychiatry 2013; 202: 447–453.

48.

Downing

SM.

Validity: on the meaningful interpretation of assessment data. Med Educ 2003; 37: 830–837.

49.

Gill

Dolley

Dunning

et al . Evaluating health services with point of service feedback: perspectives and experiences of patients, staff and community volunteers in an inpatient rehabilitation facility. Disabil Rehabil 2015; 37: 1997–2005.

50.

Baines R, Regan de Bere S, Stevens S, et al. The impact of patient feedback on the medical performance of qualified doctors: a systematic review. BMC Medical Education 2018; 18:173.

51.

Overeem

Lombarts

MJMH

Arah

et al . Three methods of multi-source feedback compared: a plea for narrative comments and coworkers ‘perspectives’. Informa Healthc 2010; 32: 141–147.

52.

Rozenblum R and Bates DW. Patient-centred healthcare, social media and the internet: the perfect storm? BMJ Qual Saf 2013: 22: 183–186.

53.

Edwards

Evans

White

et al . Experiencing patient-experience surveys: a qualitative study of the accounts of GPs. Br J Gen Pract 2011; 61: e157–e166.

54.

Royal College of Psychiatrists. What is ACP 360?, http://www.rcpsych.ac.uk/workinpsychiatry/qualityimprovement/acp360/whatisACP360.aspx (2017, accessed 7 July 2017).

55.

Jacob

KS.

Patient experience and psychiatric discourse. Psychiatrist 2012; 36: 414–417.