Article Commentary: Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs)

Abstract

In recent years, there has been an increased focus on placing patients at the center of health care research and evaluating clinical care in order to improve their experience and ensure that research is both robust and of maximum value for the use of medicinal products, therapy, or health services. This paper provides an overview of patients’ involvement in clinical research and service evaluation along with its benefits and limitations. We describe and discuss patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs), including the trends in current research. Both the patient-reported experiences measures (PREMs) and patient and public involvement (PPI) initiative for including patients in the research processes are also outlined. PROs provide reports from patients about their own health, quality of life, or functional status associated with the health care or treatment they have received. PROMs are tools and/or instruments used to report PROs. Patient report experiences through the use of PREMs, such as satisfaction scales, providing insight into the patients’ experience with their care or a health service. There is increasing international attention regarding the use of PREMS as a quality indicator of patient care and safety. This reflects the ongoing health service commitment of involving patients and the public within the wider context of the development and evaluation of health care service delivery and quality improvement.

Keywords

patient reported outcomes patient reported measures clinical research patient experience

Introduction

High-quality clinical care requires patients to provide information regarding how they are feeling, their symptoms, and any effects of prescribed treatment. The medical outcomes study was at the forefront of this concept as patient outcomes were examined and differences in care, clinicians, and communication styles were reported for both patient and clinical outcomes.¹ A further extension of this concept into drug development research has only been evident in the past two decades.^2,3 The pharmaceutical industry recognized the importance of considering patient-reported outcomes (PROs) along side biomarkers of health improvement.⁴ The separation between health outcomes and treatment outcomes became clearer when research into health services began to focus on improving the patients’ health-related quality of life, particularly when patients were undergoing optimal medical therapy.⁴ This, in turn, created the need for identifiable, valid, and reliable patient-reported measures (PROMs).

The increasing attention being given to patients involved in research studies has led to other initiatives such as Patient and Public Involvement in the United Kingdom.⁵ This initiative builds upon commissioned research by the Health Technology Assessment program that sought to identify the benefits and barriers to patient involvement in research and the associated research governance activities, such as human research ethics committees.⁶ Therefore, this paper provides an overview of patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs) in research and clinical practice. We also describe patient-reported experiences measures (PREMs) and initiatives for patient involvement in research such as the Patient and Public Involvement program (PPI).

Patient-Reported Outcomes (PROs)

A PRO is directly reported by the patient without interpretation of the patient's response by a clinician or anyone else and pertains to the patient's health, quality of life, or functional status associated with health care or treatment.⁷ These outcomes may be measured in absolute terms, such as a patient's rating of the severity of pain. PROs can also be used to report changes from a previous measure such a new onset of nausea following administration of a new drug. Traditional survival, disease, and physiological outcomes may demonstrate the physiological benefits of treatment; however, the patient perspective provides a more holistic interpretation and a comprehensive assessment of the benefits of the treatment under investigation.^8,9 For example, a new drug may demonstrate good clinical outcomes in terms of improving the length of survival for a particular patient group, while PROs may identify that patients are non-compliant with the drug regime due to reported adverse or side effects, complexity of the drug regime, and or a poor quality of life. The effectiveness of any therapeutic intervention, therefore, has many dimensions, including clinical effectiveness of the intervention and the benefit felt by patients as a direct result of having the intervention. The assessment by the benefit to the patient using a variety of instruments or tools may include functional status, service satisfaction, and quality of life.

The patient experience has played a part in clinical research for some time and continues to increase with the recognition that a whole system, such as a patient-centered approach, is necessary for comprehensive assessment of the impact of treatment and care. In clinical pharmacology trials, PROs are used as either primary outcome measures or to complement primary outcome measures.⁷ When clinical trials involve conditions in which there is no objective outcome measurement, such as the degree of morbidity or biomarkers for symptoms, and in which outcomes can only be observed subjectively to the patient in terms of impact, PROs can be used as primary outcome measures. PROs are also used to complement primary outcomes such as survival rates and biomarkers as they reflect components important to the patient and may include patient reports of symptoms and other indices such as quality of life.⁷ Therefore, PROs can be used as either a primary outcome or as a secondary outcome of a study, while a PROM is the measurement of the PRO, such as quality of life.

Patient-Reported Measures (PROMs)

PROMs are the tools or instruments used to measure PROs. These tools may measure the patient's health status such as health-related quality of life. These tools are often (patient) self-completed questionnaires.¹⁰ PROMs may include instruments or tools that measure functional status, health related quality of life, symptom and symptom burden, personal experience of care, and health-related behaviors such as anxiety and depression.^8,11 They can be either general in nature or disease-specific.⁸ Broader PROMs examine aspects that fit a variety of different conditions and allow comparison across these various medical conditions to assist in the evaluation and the implementation of new methods of providing care and equity of service delivery.^8,12 Broad-based PROMs, such as the EuroQol EQ-5D, also enable cost-effectiveness analysis as part of a cost-utility analysis to examine the cost of a health-related intervention and the benefit it produces in terms of the number of years lived in full health.^8,12

In contrast, disease-specific PROMs are designed to identify specific symptoms and their impact on the function of those specific conditions. Disease-specific PROMs have greater face validity and credibility than generic PROMs, but these comparisons cannot always be made across a variety of conditions.⁸ Often, clinical studies use a combination of generic and disease-specific PROMs. For example, a study involving asthma patients may include a PROM of ‘asthma control’ (disease-specific) along with a generic PROM such as the EuroQol EQ-5D as a measure of quality of life.¹¹

Development of PROMs in research

PROMs were initially developed for use in pharmacological and health service research⁸ and were largely restricted to England, Sweden, and parts of the US as a way to improve the clinical care of patients.¹¹ In 1975, the medical profession in Sweden established the nation-wide use of PROMs using disease-specific clinical databases known as quality registers.⁸ By 2000, PROMs were introduced into some parts of the US with the aim of extending PROMs as a reimbursement mechanism for accountability within care organizations.⁸

The use of PROMs continues to expand beyond clinical research in recognition of its potential to transform health care, as well as improve quality and safety by placing the patients at the center of decision-making. This increasing usage of PROMs has culminated in PROs attaining greater credibility amongst regulatory bodies who aim to standardize their use and interpretation in clinical trials.¹³ For instance, both the US Food & Drug Administration and the European Medicines Agency have both released guidelines that mandate the use of PROMs to support labeling claims.^2,13

Since 2009, it has become mandatory in the UK to use PROMs to report outcomes for certain elective surgical patients as a method of collecting information on the effectiveness of patient care within the NHS from the patient perspective. This mandatory adoption of PROMs has been driven by the UK government to enable comparison of health service⁸ and to identify strengths and weaknesses of health care delivery, drive quality improvement, inform commissioning, and promote choice.^12,14 The Department of Health¹² also released guidelines on the national standards (in the NHS) for the mandatory routine collection of PROMs involving some elective surgical procedures.¹² Building on this work, The Patient-Reported Outcomes group at Oxford University has produced evidence-based reports on PROMs for chronic conditions such as chronic obstructive pulmonary disorder (COPD)¹⁵ along with recommendations for selection of the best measurement instruments according to literature reviews. The formalization of, and broader emphasis on, PROs and PROMs has led to need for guidance on the development, use, measurement, and analysis PROs and the establishment of a conceptual foundation for PRO assessment to overcome issues with PROMs.

Overcoming issues in the development and selection of PROMs

Development and selection of appropriate PROMs, whether for clinical trial research or other uses such as quality improvement initiatives, requires consideration of several methodological issues including validity, sensitivity, reliability generalizability, and feasibility. One issue involves the identification and selection of valid, sensitive PROM instruments. A multitude of validated PROMs instruments have stemmed from earlier clinical research involving the patient perspective as a health care outcome.¹⁵ This work focuses on developing valid and responsive measurement instruments to produce empirical evidence regarding health from the patient perspective.¹⁶ The validity of PROM instruments is based on whether these tools represent or measure what they are intended to (construct validity) from the patient perspective.⁷ For example, a study investigating the impact of pulmonary rehabilitation for COPD patients on breathlessness would require the use a PROM of breathlessness.

The content validity of PROMs is equally important in selecting a PROM and is established through analysis of the instrument's content (or items) and the concept that the test is designed to measure.¹⁷ Therefore, a precise and reliable instrument must be valid and responsive and/or sensitive to change when evaluating treatment differences in order to measure differences between groups, if they exist.⁷ Content validity is important in selecting or modifying existing PRO instruments, but there is a lack of consensus regarding the best practices for establishing and documenting validity within the research community.¹⁷

Within clinical research, a variety of instruments may be chosen to measure a specific PRO, such as dyspnea, making comparisons between PROMs challenging. Issues regarding comparisons of PROM instruments arise when there are different definitions of a particular PRO (eg, quality of life) and there is a choice to use a number of instruments such as generic (EQ-5D) and a disease-specific [St. George's Respiratory Questionnaire (SGRQ)] instrument of a patient with COPD.⁷ Traditionally, the choices of PROMs are often based on professional judgment versus strong conceptual models creating issues with grouping and scoring items into domains. An unclear conceptual match between the PRO instrument and the intended claim threatens its content validity.¹⁷ This may result in problems with analysis and interpretation of study data as links and comparability between certain tools may come into question.

To overcome this issue, Rothman et al suggested beginning with a conceptual model and framework to guide the selection, analysis, and interpretation of PROs.¹⁸ Conceptual models should form the rationale and specification for selecting the appropriate PRO outcome(s) for a clinical trial.¹⁸ The PRO concept guides the development of a framework for measurement. For example, if the health behavior of a patient is central to the study, then a health belief model may be a relevant framework to assist with the selection of appropriate outcomes and outcome measures. Hence, all variables and their relationships to each other are given an operational meaning that guide the selection and approaches to analysis.¹⁹ This allows the focus to be maintained on interrelationships among PRO domains being measured, the content validity of each PRO instrument, and construct validity, reliability, and responsiveness of each PRO instrument when applied within a specified population being studied.^17,18 The complexity of the PRO concept should be reflected in the decision to use either one-dimensional instruments (eg, rating of the perception of dyspnea) or complex tools such as health-related quality of life to measure a patient-reported multidimensional outcome.

Identification, evaluation, and recommendations of validated PROMs through national guidelines from organizations such as the Food & Drug Administration and the Oxford Patient-Reported Outcomes Group assist ininitiating and standardizing clinical research methods. In the US, the National Institutes of Health joined with several outcome scientists from across the country to collaboratively develop the Patient-Reported Outcomes Measurement Information System (PROMIS). This group of research scientists aim to create the next generation of PROMs and standardize and promote a common measurement system for PROs across clinical research.^20,21

A second issue relates to the reliability, size, and content of existing PROMs. There is growing recognition that combining good measurement properties with shorter, more reliable instruments could result in better response rates by reducing the burden to responders.²² In order to produce these faster and more reliable instruments, work is being carried out not only to determine how well the PROM instruments work as a whole but also how well individual items on the PROM work based on how people respond to the individual questions. The PROMIS team uses item response theory (IRT) and computerized adaptive testing tools^20,21 to assist in the development of measurement systems comprised of carefully calibrated questions that define and quantify a common concept and provide operational definitions of the trait. IRT is a probabilistic, mathematically based model used to describe the relationship between an individual's response to questions about his or her health and an underlying variable measured by the instrument (eg, strength of attitude, intelligence). This analysis informs the tools development and helps to strengthen the content validity of the PRO instrument by identifying the most relevant and important item content in the instrument.¹⁷ The result is the production of shorter, more reliable measures which allow comparisons to be made between measurement instruments.

The aim of developing shorter and more reliable tools is to increase the rates of patient participation from target groups, particularly among underrepresented patient populations.⁸ While the production of shorter, more reliable measures may help improve the response rate by decreasing the time and effort required to complete them, a secondary aim would be to demonstrate the benefit and relevance of the item being measured, such as quality of life, to a previously difficult to attract target group. Rothman et al¹⁷ suggests that this will add to the content validity of the measure and can be achieved by involving patients directly with selecting the item content of the PROMs rather than presuming to know what is significant to them.

Higher response rates could also be obtained by widening the population base through the translational and cultural adaption of PROMs. Inclusion of under-represented patient groups would decrease the risk of bias and increase the generalizability of results. However, caution is required when utilizing PROMs in different populations (eg, pediatrics versus adults, South African versus European), as validation and normalization to each cultural group must be carried out prior to their administration as part of a research study. It has been suggested that translational and cultural adaptation of PROs should be directed by guidelines and standards.²³

Lastly, the widespread use and feasibility of PROMs has been limited by the time-consuming and costly process of collection, analysis, and presentation of PROMs data in the conventional paper format. The internet opens up many opportunities to help improve the feasibility and cost-effectiveness of collecting and aggregating both PROMs and patient experience data. Greaves et al²⁴ uses the term ‘cloud of patient experience’ to describe the collection of this type of information via various internet sources (eg, social networking sites, twitter, hospital review sites, etc). The potential to collect data in real-time to uncover poor clinical care and potential areas of excellence is an additional advantage over the infrequently administered paper-based tools (eg, hospital surveys).²⁴However, mailed surveys have the advantage of higher response rates.²⁵ This may reflect the fact that internet users are generally younger, of higher economic status, and have less contact with health care,^24,25 and therefore may not necessarily represent the target group. Offering both internet and paper-based PROMs options could minimize selection bias while improving the feasibility and cost-effectiveness of PROMs.

Patient-Reported Experience Measures (PREMs)

Patient-reported experience measures (PREMs) are tools and instruments that report patient satisfaction scores with a health service and are generic tools that are often used to capture the overall patient experience of health care. PREMs are often used in the wider population and in non-specific settings such an outpatient department. Patient experience tools, for example, may be used to monitor patient feedback and focus on the general experience such as customer service rather than an experience related to a specific disease.^26,27 These instruments or tools have revealed positive associations between patient satisfaction and safety^28,29 and are a reliable measure of how well a hospital is able to provide good quality service from a patient's perspective.³⁰ Internationally, PREMs are used to evaluate health care in terms of clinical effectiveness and economic efficiencies.²⁷

Limitations and challenges of patient experience also need to be considered when developing policies based on patient experience and PREMs. In the US, the Centres for Medicare and Medicaid services and private insurers are basing reimbursement methods on patient satisfaction scores.²⁸ Lyu et al²⁸ argue that the reliability and adequacy of such policies that rely on patient satisfaction scores as metrics without consideration of the context may be questionable in some circumstances such as surgery. Patient satisfaction was found to be independent of hospital compliance with surgical process-of-care measures.²⁸ The lack of correlation between patient satisfaction and how well surgical teams comply with best practice may be in part due to the fact that the processes are unknown to and unseen by the patient. For example, an anesthetized patient will not be able to comment on the quality of either the surgical or safety procedures within the operating theatre. However, they would be able to relate their health care processes outside of the theatre and familiar to them (eg, administrative processes, ward cleanliness, discharge practices) which would be beneficial to improving the overall quality of care.

In the UK, there is emphasis on measuring patient experience with both indicators and metrics. Metrics are precise measures of a known attribute²⁶ such the number of falls occurring on a ward. Indicators are a type of metric that identifies issues requiring further investigation (eg, increase in number of falls) (NHS Institute for Innovation and Improvement/Public Health Observatories, 2007) and reflects how effectively an organization is performing on a set of metrics. Patient experience indicators are being embedded within NHS accountability frameworks, such as the NHS Outcomes 2012/13 framework.³¹ These measures increasingly used at Ministerial, policy, commissioning, and management levels as part of policy and legislation as the emphasis of the patient experience as a marker of patient care quality becomes more accepted and embedded in NHS contracts.^26,32 The NHS National Quality Board produced a working definition of patient experience with eight indicators to guide the measurement.³³ These indicators include: respect for patient-centered values, preferences, and expressed needs; coordination and integration of care; information, communication, and education; physical comfort; emotional support; welcoming and involvement of family and friends; transition and continuity; and access to care.³³

Patient and Public Involvement (PPI)

The value of patient and public involvement (PPI) in the delivery and design of health care services has become increasingly recognized internationally. In the context of research within the UK, PPI actively involves patients and the public in the actual research process from conception through to analysis and dissemination of research findings. Thus, the role of patients and the public in PPI differs from research involving PROs and PROMs. PROs and PROMs involve research “about” or “to” patients in contrast to PPI, in which involvement in research is “with” or “by” the public.³⁴ Actively involving consumers in the research process by obtaining their unique perspectives is thought to result in greater quality and clinical relevance of health research.^6,35–37A body of research is now evolving on the benefits and impact of PPI at different stages of the research process (eg, identification of research questions and research prioritization, research design, and protocol development, peer review of research proposals and funding decisions, medical device/intervention development, research advisory groups and other committees, data collection, data analysis and interpretation data), and on how to involve different members of the public (eg, older people, young people and parents, people with learning difficulties, mental health problems, or cancer).³⁴ In 1999, the Department of Health introduced a policy directive to involve patients and the public in the National Health Service's research and development process.⁵ This has been supported by INVOLVE, partially funded by the National Institute for Health Research (NIHR), which was set up in 1996 to promote, monitor, and evaluate PPI in research in the NHS, public health, and social care. Funding bodies such as the NIHR now require evidence of PPI in their research applications submitted for funding.

Conclusion

There is increasing recognition of the importance of involving patients and the public in in clinical research and within the wider context of development and evaluation of health care service delivery and quality improvement. PROS are reports provided by patients about their own health, quality of life, or functional status associated with the health care or treatment they have received.

PROMs are the tools and/or instruments that have been developed to ensure both a valid and reliable measurement of these patients-reported outcomes, such as quality of life measures. The inclusion of PROs through the use of PROMs with clinical outcomes in research and clinical practice provides a more complete understanding of the impact of an intervention, therapy, and/or service on the patient. PPI in the research process ensures that the research is relevant to patients and that the appropriate measures are selected.

Patient reported experiences through the use of PREMs, such as satisfaction scales, provide insight into the patients’ experience with their care or a health service. There is increasing international attention regarding the use of PREMs as a quality indicator of patient care and safety.

Author Contributions

Analyzed the data: TW, SMSS. Wrote the first draft of the manuscript: TW, SMSS. Contributed to the writing of the manuscript: TW, SMSS. Agree with manuscript results and conclusions: TW, SMSS. Jointly developed the structure and arguments for the paper: TW, SMSS. Made critical revisions and approved final version: TW, SMSS. All authors reviewed and approved of the final manuscript.

Funding

Author(s) disclose no funding sources.

Competing Interests

Author(s) disclose no potential conflicts of interest.

Footnotes

As a requirement of publication the authors have provided signed confirmation of their compliance with ethical and legal obligations including but not limited to compliance with ICMJE authorship and competing interests guidelines, that the article is neither under consideration for publication nor published elsewhere, of their compliance with legal and ethical guidelines concerning human and animal research participants (if applicable), and that permission has been obtained for reproduction of any copyrighted material. This article was subject to blind, independent, expert peer review. The reviewers reported no competing interests. Provenance: the authors were invited to submit this paper.

References

Tarlov

A.R.

, Ware

J.E.

Jr , Greenfield

, Nelson

E.C.

, Perrin

, Zubkoff

The Medical Outcomes Study. an application of methods for monitoring the results of medical care. JAMA. 1989; 262(7): 925–930.

Food and Drug Administration. Guidance for industry patient-related outcome measures: use in medical product development to support labeling claims. In: Services DoHaH, ed. Washington: U.S. Department of Health and Human Services Food and Drug Administration; 2009.

Patrick

D.L.

, Burke

, Powers

J.H.

Patient-reported outcomes to support medical product labeling claims: FDA perspective. Value Health. 2007; 10 Suppl 2: S125–S137.

Wilke

R.J.

, Burke

L.B.

, Erickson

Measuring treatment impact: a review of patient-reported outcomes and other efficacy endpoints in approved product labels. Control Clin Trials. 2004; 25(6): 535–552.

NHS Executive. Patient and public involvement in the new NHS. Leeds: Department of Health; 1999.

Oliver

S.R.

How can health services users contribute to the NHS research and development programme?

BMJ. 1995; 310(6990): 1318–1320.

Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. Higgins

J.P.T.

, Greene

, eds. The Cochrane Collaboration, 2011. http://handbook.cochrane.org/. Accessed date.

Black

Patient reported outcome measures could help transform health care. BMJ. 2013; 346: f167.

Rathert

, Huddleston

, Pak

Acute care patients discuss the patient role in patient safety. Health Care Manage Rev. 2011; 36(2): 134–144.

10.

Using the commissioning for quality and innovation (CQUIN) payment framework. UK Department of Health. https://www.gov.uk/government/publications/using-the-commissioning-for-quality-and-innovation-cquin-payment-framework-guidance-on-new-national-goals-for-2012–13. Updated April 30, 2012. Accessed date.

11.

Black

, Jenkinson

Measuring patients’ experiences and outcomes. BMJ. 2009; 239: b2495.

12.

Guidance on the routine collection of Patient Reported Outcome Measures (PROMS). UK Department of Health. https://www.gov.uk/government/publications/patient-reported-outcome-measures-proms-in-england-a-methodology-for-identifying-potential-outliers–2. Updated July 18, 2011. Accessed date.

13.

Bottomley

, Jones

, Claassaens

Patient-reported outcomes: assessment and current persepctives of the guidelines of the Food and Drug Administration and the reflection paper of the European Medicines Agency. Eur J Cancer. 2009; 45(3): 347–353.

14.

Darzi

High quality care for all. NHS next stage review final report. Norfolk, UK: Stationary Office Books; 2008.

15.

University of Oxford. Patient Reported Outcomes Measurement Group. http://phi.uhce.ox.ac.uk/about.php. Accessed date.

16.

Revicki

, Hays

R.D.

, Cella

, Sloan

Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008; 61(2): 102–109.

17.

Rothman

, Burke

, Erickson

, Leidy

N.K.

, Patrick

D.L.

, Petrie

C.D.

Use of existing patient-reported outcome (PRO) instruments and their modification: The ISPOR good research practices for evaluating and documenting content validity for the use of existing instruments and their modification PRO taskforce report. Value Health. 2009; 12(8): 1075–1083.

18.

Rothman

M.L.

, Beltran

, Cappelleri

J.C.

, Lipscomb

, Teschendorf

; Mayo/FDA Patient-Reported Outcomes Consensus Meeting Group. Patientreported outcomes: conceptual issues. Value Health. 2007; 10(2): S66–S77.

19.

Devlin

J.N.

, Appleby

, Buxton

Getting the most out of PROMS. Putting health outcomes at the heart of the NHS decision making. In: Fund

T.K.

, ed. London, UK: The Kings Fund; 2010.

20.

Carle

A.C.

, Cella

, Cai

Advancing PROMIS's methodology: results of the Third Patient Reported Outcomes Measurement Information System (PROMIS^®) Psychometric Summit. Expert Rev Pharmacoecon Outcomes Res. 2011; 11(6): 677–684.

21.

Cella

, Yount

, Rothrock

The Patient-Reported Outcomes Measurement Information System (PROMIS): progress of an NIH Roadmap cooperative group during its first two years. Med Care. 2007; 45(5 Supp 1): S3–S11.

22.

Edwards

, Roberts

, Clarke

Increasing response rates to postal questionnaires: systematic review. BMJ. 2002; 324: (7347)1183.

23.

Wild

, Grove

, Martin

; ISPOR Task Force for Translation and Cultural Adaptation. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR taskforce for translation and cultural adaptation. Value Health. 2005; 8(2): 94–104.

24.

Greaves

, Ramirez-Cano

, Millett

, Darzi

, Donadson

Harnessing the cloud of patient experience: using social media to detect poort quality healthcare. BMJ Qual Saf. 2013; 22(3): 251–255.

25.

Lagha

, Noble

, Smith

, Denvir

M.A.

, Leslie

S.J.

Patient reported experience measures (PREMS) in chronic heart failure. J R Coll Physicians Edinb. 2012; 42(4): 301–305.

26.

Lakhani

Indicators for measuring patient experience. http://patientexperienceportal.org/article/indicators-for-measuring-patient-experience. Accessed April 18, 2013.

27.

Staniszewska

, Bullock

, Avital

, O'Flynn

Developing and implementing NICE guidance on patient experience. http://patientexperienceportal.org/article/devleoping-and-implementing-nice-guidance-on-patient-experience. Accessed April 18, 2013.

28.

Lyu

, Wick

E.C.

, Housman

, Freischlag

J.A.

, Makary

M.A.

Patient satisfaction as a possible indicator of quality surgical care. JAMA Surg. 2013; 148(4): 362–367.

29.

Doyal

, Lennox

, Bell

A systematic review of evidence on the links between patient experience and clinical safety and effectiveness. BMJ Open. 2013; 3(1): e001570.

30.

Glickman

S.W.

, Boulding

, Manary

Patient satisfaction and its relationship with clinical quality and inpatient mortality in acute myocardial infarction. Circ Cardiovasc Qual Outcomes. 2010; 3(2): 188–195.

31.

National Clinical Guideline Centre (UK). Patient experience in adult NHS services: improving the experience of care for people using adult NHS services: patient experience in generic terms. London, UK: Royal College of Physicians; 2012.

32.

Gibbons

, Fitzpatrick

Patient reported outcomes measures: their roles in measuring and improving patient experience. http://patientexperienceportal.org/article/patient-reported-outcome-measures-their-role-in-measuring-and-improving-patient-experience. Accessed April 18, 2013.

33.

National Health Service. NHS Patient Experience Framework. 2011; https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/146831/dh_132788.pdf.pdf. Accessed June 15, 2013.

34.

Boote

Patient and public involvement in Health and Social Care Research: A bibliography: National Institute of Health Research (NIHR). http://www.rdsyh.nihr.ac.uk/_file.ashx?thematic+PPI+bibliography+latest+version+5.doc. Accessed date.

35.

Boote

, Telford

, Cooper

Consumer involvement in health research: a review and research agenda. Health Policy. 2002; 61(2): 213–236.

36.

Entwhistle

V.A.

, Renfrew

M.J.

, Yearley

, Forrester

Lay perspectives: advantages for health research. BMJ. 1998; 316(7129): 463–466.

37.

Chalmers

What do I want from health research and researchers when I am a patient?

BMJ. 1995; 310(6990): 1315–1318.