Online patient feedback: a scoping review and stakeholder consultation to guide health policy

Abstract

Objective

To provide a synthesis of the current evidence base of online patient feedback using a scoping review and a consultation of stakeholders in England, UK.

Methods

We searched MEDLINE, EMBASE, PsycINFO, CINAHL and the Social Science Citation Index and conducted hand searches up to January 2018. We included primary studies of internet-based reviews and other online feedback (e.g. social media and blogs) from patients, carers or the public about health care providers (individuals, services or organizations). Key findings were extracted and tabulated for further synthesis guided by the themes arising from a stakeholder consultation.

Results

The review found that awareness and usage of online feedback is increasing. Most feedback is about physicians, and is typically positive. Online reviews and ratings are used by some service users to inform choice of provider or treatment while providers tend to be concerned about the validity and representativeness of feedback. Reviewed studies found that those who post feedback are generally not representative of the general population, tending to be younger and more educated, but online feedback does broadly correlate with some other measures of health care quality.

Conclusions

In an increasingly digital society, where citizens provide and use feedback for a range of goods and services, online patient feedback can offer a convenient, low cost and widely accessible mechanism to capture experiences of health care, while being mindful to avoid issues of digital exclusion. This review provides important insights to inform policy development seeking to harness the opportunities offered by online feedback.

Keywords

health policy patient experience patient satisfaction quality improvement

Background

Patient experience and satisfaction data are routinely collected by health services in many countries to ensure the delivery of patient-centred care; to monitor patient safety; to improve quality in service delivery and care and/or to aid patients in choosing services or providers. Incorporating patient experience data into the monitoring and delivery of health care is associated with a range of potential benefits including increased patient-centredness, satisfaction, quality improvement and cost-effectiveness.¹ Increasingly, patient feedback is being collected online, mirroring common practices in other industries. Previous reviews of the evidence in this emerging field showed that the use of online rating sites is increasing, that they have the potential to provide information about patients’ views on quality of care and that their value to policymakers should be explored.^2,3

Online patient feedback offers an opportunity for collecting experience, satisfaction and improvement data in addition to the more traditional approach that uses paper-based or digitized experience or satisfaction surveys. Providing multiple and inclusive avenues for feedback is important. The internet offers advantages to patients and providers in that it is convenient, low-cost, accessible to most, immediate and transparent. However, the health care sector in general has been slow to adopt online feedback routinely in the way other industries have, and guidance on the collection and implementation of online patient feedback is scant. This scoping study brings together existing research in this area, alongside views of key stakeholders. It aims to provide a state-of-the-art snapshot of this emerging field to inform policy and practice and to set the agenda for future research. Stakeholders consulted for this work were based in England so their views are pertinent to the English National Health Service (NHS). However, the review itself covers the international evidence.

Method

We used a scoping review as the topic was broad and we aimed to include studies of multiple designs, following Arksey and O’Malley.⁴ We defined online patient feedback as qualitative (e.g. narrative) or quantitative (e.g. star/numeric ratings) information in any format which described the experience of using a health service by a patient or carer in an online setting, whether solicited or unsolicited.

Search strategy

MEDLINE, EMBASE, PsycINFO, CINAHL and the Social Science Citation Index were searched in January 2017 using a list of terms likely to retrieve articles about online patient feedback, compiled in consultation with an information specialist. The search was updated in 2018 (see Online Supplement 1 for the detailed search strategy).

Inclusion criteria

Original primary research on online feedback (provider reviews, ratings and other online feedback, e.g. on social media/blogs) from patients, carers or the public about health care providers (individuals/organizations) and published 2000–2018 from any discipline or design.

Screening and data extraction

Titles and abstracts, and then full texts were screened independently by two reviewers. Disagreements were resolved with a third reviewer. Selected full-text articles were randomly assigned to either of two reviewers, and data on authors, publication date, study aims, sample, methods and findings were extracted and tabulated.

Analysis

Included studies were thematically grouped by aim for further analysis. Findings were synthesized using a narrative approach, including individual analysis and discussion within the research team, guided by the themes that arose from the stakeholder consultation (see below). Formal critical appraisal was not conducted due to the broad and inclusive nature of this review and the variation in study designs.

Stakeholder consultation

We approached organizations with experience of (collecting or using) online feedback, including feedback companies in the health and retail sectors, national health policymakers, regulators of health services, senior clinicians working in the area of patient experience and patient experience managers from NHS organizations, as well as service users with experience of reading and providing feedback who were known to the research team through Patient and Public Involvement work (service-user as used here includes patients, family, carers and those who use services but are not ill, e.g. pregnant women). Potential participants were invited by email to take part in the consultation; the email also explained the practicalities of participating and what topics would be covered. All invited stakeholders accepted the invitation; some participants were already known to the project team and had previously expressed interest. The consultation sought to inform the questions being asked in the literature review and therefore did not require formal ethics approval. Each stakeholder was consulted by the lead author (AMB) about their views, key concerns and future priorities, either in-person or by telephone using a broad interview schedule. Conversations typically lasted 60–90 min. Topics included how online feedback was defined, how feedback is collected and used in commercial and health settings, what feedback is for and how and why is it collected, used and analysed in the NHS, the usefulness of numeric and narrative data, and inclusion and diversity. Notes were taken and analysed inductively by the lead author in consultation with the research team.

Results

The literature searches yielded a total of 28,613 records; the total number of included studies was 78, of which 12 were identified through hand-searching (Figure 1). The majority of included studies were conducted in the USA, followed by the UK and Germany, with a smaller number of studies from China, the Netherlands, Canada and Austria. An overview of key characteristics of included studies is provided in Online Supplement 2.

Figure 1.

PRISMA diagram demonstrating the search and screening process.

The analysis of the stakeholder consultation identified six overarching evidence gaps that would need to be addressed to help policymakers navigate the online patient feedback landscape (see also Box 1 for more detail on the stakeholders’ views): Who provides and who uses feedback? How do organizations currently use feedback? What is the content of online feedback? Why is online feedback given? What are staff and services user attitudes towards online feedback? How reliable is online feedback? We address each of these evidence gaps raised by the stakeholders in turn using the literature found in the scoping review. Box 1.

Summary of findings and evidence gaps identified by stakeholders

Defining online feedback

Online feedback was broadly defined as any insight that can be found online either on structured platforms that collect it (e.g. NHS Choices, CareOpinion, iWantGreatCare) or on other unstructured websites, particularly social media (Twitter, Facebook, blogs) and that includes digitized paper-based feedback. It could be solicited or unsolicited. Online feedback was described as useful and important and something that should be promoted and welcomed.

Online feedback is produced by patients or carers, but little is known about the characteristics of those who post. This information is needed to assess the usefulness of such feedback. Equally, who uses this feedback and to what end is also unclear. Feedback may help service users to make choices about health care, and it is believed that feedback is also used to provide reassurance about others service users’ experiences and service quality.

Identified evidence gaps

Who provides and who uses online feedback? Why is online feedback given and how is it used?

Stakeholders identified these two important questions on who provides online feedback, who uses it and how and the motivations behind it. Online feedback was perceived to be largely positive and probably useful in identifying what works well in the NHS. Negative feedback was also seen as a force for good as it could help organizations to understand what improvements they should make and what lessons to learn. A central role of feedback is to provide insight to staff (frontline staff, bodies with responsibility for oversight, policymakers) to inform quality improvement. It can be seen as a means to initiate and expedite change or improvements in the NHS, perhaps because of concerns over reputation management. It could shift the power differential between health professionals and service-users. However, who provides online feedback and how it is used by service users is unclear.

How do organizations currently use feedback?

There is a widely held assumption that service users comment on issues of convenience and access (car parking, catering). This may be because it is what (they feel) is expected of them and, as such, may inflate the importance of these issues. The NHS should make more effort to encourage discussion of whatever is important, including inviting patients to comment on every aspect of their care, from their views on their treatments to their relationships with staff.

What is the content of online feedback?

The volume of online feedback was perceived to be low, and the stakeholders were concerned that this might make it unusable skewing how issues of user experience are perceived.

How reliable is online patient feedback?

Addressing feedback is likely to add a burden to already over-stretched services, and negative feedback could reduce staff morale, particularly in small organizations in which staff are more identifiable. It could undermine professionals’ reputations, consequently demotivating hardworking NHS staff. Inviting feedback from service users could add an additional burden to those who are ill or caring full-time.

What are health professionals’ and service user attitudes towards online feedback?

Stakeholders were interested to understand the evidence on attitudes to online feedback among professionals and patients. As a group, they held mixed attitudes towards it. They raised issues for professionals, such as fears around reputational damage and wondered if patients trusted online feedback, particularly as they felt it was becoming more commonplace for the public to review products and services online.

Who provides and who uses online feedback?

The review found that awareness of rating sites varies, that providing an online review is a minority activity and that frequency of use has increased over time. A public survey in Germany noted that 32% of respondents were aware of rating and review sites.⁵ Awareness of such sites tends to be lower than for those for consumer goods, with a representative survey of the US population finding that respondents considered such sites less important than other sources of information (e.g. recommendations of friends and family).⁶ The frequency of posting a rating varied across included studies. One survey of an online panel in Germany found that 11% of respondents had posted feedback online,⁷ while a survey of a representative sample of the population in the UK reported that 8% of respondents had done so,⁸ with studies from the USA and Austria reporting figures of around 6%.^9,10 Women were found to be more likely to post online feedback.^11,12

The survey of the UK population mentioned above found that 42% of respondents reported reading online health reviews; it also noted that the only significant association with providing feedback was being a more frequent internet user.⁸ One nationally representative survey in the USA found parents to be frequent users; they would use review sites to choose doctors for their children.⁹ Other users are likely to be younger people, with higher levels of education,¹² and those with a long-term condition.^12,13 One small survey of adults in London, UK, found that being of the same gender as their general practitioner (GP), feeling listened to and perceiving the relationship with their GP as friendly seem to be potential predictors of use of rating sites, whereas willingness to use such sites was predicted by patients’ autonomy in health care decisions.¹⁴ Having clear explanations from a GP meant that patients were less likely to use such sites.

How do organizations currently use feedback?

The scoping review did not identify studies that considered the role or function of online patient feedback or uncovered the practices and processes governing its use in health care organizations. One study in Germany found that physicians have begun to respond to online reviews,¹⁵ and a survey of providers by the same authors reported that ophthalmologists and gynaecologists were the most likely to implement change based on online reviews.¹⁶ One Dutch study highlighted the potential value of online feedback for regulatory supervision although the health care inspectors interviewed expressed reservations about this type of feedback.¹⁷ An analysis of different forms of online feedback in the Netherlands further noted that structured patient feedback websites contained more relevant additional information than social media sites (e.g. Facebook, Twitter), and the authors argued for the use of the former by health care inspectors because dedicated patient feedback websites were seen as providing ‘on the ground’ or ‘bottom-up’ quality monitoring by patients.¹⁸

What is the content of online feedback?

Characteristics of feedback

The literature consistently showed that the majority of feedback was positive, and ratings for health care providers were generally high;^11,19–21 service user reviews often recommended the health service to others.^20,22 An analysis of ratings of dentists on a German physician rating website found that female or older patients and those who had private health insurance were more likely to give positive feedback, but the reason for this was unclear.¹⁹ Patients who had a long-standing relationship with their doctor were also more likely to provide positive feedback.²³

Online feedback is usually relatively succinct. A sentiment analysis (an automated process of identifying, quantifying and studying information in text that is often used to analyse consumer reviews) of 33,654 reviews of 12,898 medical practitioners in the New York State area noted that they contained on average 4.17 sentences per review and 15.5% contained only one line.²⁴ An analysis of narrative comments posted on a German physician rating website reported that longer narratives were more likely to be negative.¹¹

Patient feedback was usually positive but reviews also highlighted negative aspects of the care experience.²⁵ The feedback in the included studies largely pertained to services or providers, clinical and administrative staff and the physical environment. The content of feedback often related specifically to physicians and focused on perceived knowledge and competency,^26,27 patient-centred communication,^26,27 personal character traits,^23,27,28 professional conduct,²⁷ dignified care²⁰ and coordination of care.²³ Waiting times and length of appointments were also discussed,^20,28,29 with other themes including cleanliness,^20,30 scheduling appointments,³⁰ insurance,²⁹ access,²³ administrative staff^23,29 and parking.³¹ Studies did not report patients commenting on medical or clinical aspects of their care. However, one study noted that including reviews from family members increased the amount of patient safety information contained therein.³²

Who is reviewed?

Analyses of physician rating sites in Germany found that male physicians were more likely to be reviewed than females, with female physicians more likely to receive positive feedback than their male colleagues.^7,19 Online feedback typically concerns physicians; feedback on nurses and allied health professionals appears to be neglected in the current literature.

Some specialities (e.g. otolaryngology) appear to be under-represented on rating sites, and some sub-specialities received a higher number of reviews than others.²¹ Surgeons in the US appear to be among the most frequently rated health professionals with one study showing that in California 97% had been rated on at least one patient feedback website.³³

Why is online feedback given and how is it used?

Included studies reported that people use online feedback sites to post reviews or ratings or consult them to aid choosing a doctor or other health professionals, e.g. dentist.^5,6,9,10,13 In a national representative US survey, 28% of respondents had used such websites to search for a physician,⁹ and in an online panel survey in Germany, 25% of respondents had done so.⁵ The latter study also found that 65% of its 1505 respondents had chosen a doctor based on their website rating while 52% did not consult a specific doctor because of their online ratings. A cross-sectional survey of a representative sample of the UK population showed that most people provide feedback to help other patients and to express positive experiences (praise) and that they read it to find out about tests or treatments and to help choose a treatment or provider.⁸

There has been little qualitative work exploring the motivations to post (and read) online reviews. In a qualitative study of patients’ views on giving online feedback and ratings to GPs in England, study participants did not consider feedback necessary and questioned whether GPs would use it.³⁴ They noted that they would only post feedback if they had had an extremely positive or extremely negative experience. The study also reported that some participants would provide online feedback because this could be done remotely, that feedback can be shared with others and because they believed it would be taken seriously by GPs. Others, however, raised issues of accessibility, were concerned about privacy and security and thought that online feedback might be ignored. Similarly, a cross-sectional survey of people who had used physician rating websites in Germany found that having had a very positive or negative experience would motivate them to post feedback whereas concerns about possible negative consequences, technical issues or about site operators would stop them from doing so.³⁵

A nationally representative survey of the US population reported that of those who had used rating sites in the preceding year, 35% said that good ratings had helped them select a physician while roughly the same proportion (37%) said that poor ratings led them to not consult the physician in question.⁶ An experimental study of 500 adults recruited through a crowdsourcing website who were presented with a hypothetical review of a physician (without any other information about quality or performance) showed that predominantly negative reviews reduced the willingness of participants to consult the physician; willingness was also lower when negative reviews were presented before positive reviews.³⁶

The perceived trustworthiness, credibility and expertise of the reviewer and the content of the review were also found to be important to patients reading feedback. For example, a randomized experiment with 168 adults found that respondents preferred fact-oriented content over emotional feedback (i.e. containing slang or humour).¹⁰ An online survey of a random sample of patients in Germany noted that prior use of a rating site increased the likelihood of rating a health care experience in the future,¹² while a similar survey in the USA reported that patients are likely to spend more time on sites that contain patient comments but that this reduced their attention to other standardized measures of quality, increasing the potential for making ill-informed choices.³⁷

What are health professionals and services user attitudes towards online feedback?

Health professionals described a range of reservations about online feedback, while patients’ attitudes were more mixed. For example, a qualitative interview study with GPs in England highlighted several concerns, including issues of validity and representativeness.³⁸ A survey by the American College of Physicians Executive reported that only 12% of the 730 physicians who responded thought online rating websites were useful and 39% agreed with the ratings they had received.³⁹ In a survey of hand surgeons in the USA, 65% of respondents reported holding negative views of online physician rating websites, and 82% felt that the contents were not useful for their practice.⁴⁰

How reliable is online feedback?

Studies in England, Germany and the USA reported that online ratings and free-text reviews were correlated with established measures of patient feedback derived from more traditional patient satisfaction or experience surveys.^{22,24,41–43} For example, one analysis of aggregate website visitor ratings (1–5 stars) on Yelp.com and patient experience as assessed through the Hospital Consumer Assessment of Healthcare Providers and Systems survey in the USA found these to be positively correlated (r = 0.49, p < 0.001).⁴¹ Likewise, an analysis of patient web-based ratings of all acute hospitals in England found a statistically significant positive correlation between the number of patients willing to ‘recommend the hospital to a friend’ with a hospital’s overall rating of patient experience (Spearman ρ = 0.41, p < 0.001).⁴⁴ A related study also found significant correlations for positive recommendations online with lower hospital standardized mortality ratios (Spearman ρ = −0.20, p = 0.01) and lower readmission rates (ρ = −0.31, p < 0.001), and better ratings of hospital cleanliness were associated with lower rates of health care-associated infections.²² Other domains such as being treated with respect and dignity, involvement in decisions about care and staff working well together were also found to be significantly associated.⁴³ Evidence from the USA highlights that online feedback could usefully complement existing measures of patient experience using traditional survey methodology,^41,45 although additional content tends to pertain more to some services than others.²⁴

Discussion

This scoping study explored the uses and usefulness of online patient feedback, highlighting the potential of such feedback to influence service inspection, quality improvement, and to inform patient choice of provider and health policy more broadly. Our review found that awareness and usage are increasing; that most feedback is about physicians, and typically positive; that reviews and ratings are used by some service users to inform choice of provider or treatment; and that providers have reservations about the validity and representativeness of feedback. Included studies found that those who post feedback are generally not representative of the general population, tending to be younger and more educated, but online feedback does broadly correlate with some other measures of health care quality. Our review complements and expands two previous reviews that also reported a relationship between health care quality and online feedback and called for more research to be done in this emergent area.^2,3 It is important to note that the included literature comes from a range of settings where approaches to service user feedback and the options for exercising choice of provider vary.

Online patient feedback can be seen as having functions of ‘choice’ and ‘voice’. Adams⁴⁶ argued that the ‘voice’ function of online feedback provides an opportunity for ‘sousveillance’ or bottom-up monitoring and can be an important tool for gathering insight to guide or inform inspections. The internet may well be perceived to provide a safe space where patients can speak safely and honestly about their care. However, this is contradicted by reviewed studies that suggest that patients may fear repercussions of providing online feedback. Like with other forms of feedback, patients may not be motivated to help others’ choose, but to help health care providers identify the positive and negative aspects of their care provision and drive improvement. Several studies looked at the usefulness of online feedback websites in uncovering patient safety issues.⁴⁷ Conceptualizing and promoting them as sources of intelligence for monitoring might provide service users with a further reason to contribute. However, little is known about the motivations of those who post feedback, and available research on public and provider attitudes to online feedback is limited.^38,40

Only a small number of studies included in this review considered the potential harms of online patient feedback; these were small-scale qualitative interview studies with patients and GPs that voiced their concerns rather than exploring actual harms (direct or indirect).^34,38 We speculate that the harms for providers might be associated with reputation, demoralization and risks to confidentiality by responding to feedback online. For service users, reading online reviews might induce or increase anxiety or distort reality where feedback is extreme (highly positive or highly negative). Narratives offer the opportunity to organize and make sense of their experiences, but they can also wield undue influence and may cause harm if, for instance, a small number of ‘unrepresentative’ stories are seen as hard and fast evidence.¹ At the same time, ‘unrepresentative stories’, that is, unusual single stories, outliers or discrepant feedback, might provide valuable lessons for service delivery and improvement as signals that point towards problems amidst the general noise of feedback. A challenge for decision makers is to ensure that these single stories are not overlooked or dismissed, but instead used to inform improvement.

We identified several gaps in the literature. Studies have tended to focus on feedback on physicians or surgeons but there was little evidence of feedback on other health care workers, such as nurses. We also did not identify studies that looked at the practices and processes of using online feedback to improve health care. Equally, while there is some evidence pointing to a reasonably strong link between established measures of patient satisfaction and online feedback, the evidence of a relationship with indicators of clinical quality remains less well established and warrants further exploration.⁴² Finally, none of the reviewed studies presented research on the potential for patient feedback to be manipulated.

This study has several limitations. The stakeholder consultation was carried out within the English context and as such reflects a limited perspective. At the same time, the consultation provided an additional layer of insight that helped shape and strengthen the review. In line with the scoping review method, we did not appraise the quality of any of the included studies meaning that this scoping review can only describe what has been reported. Finally, it is possible that some studies are omitted from this review; despite our attempts to be as comprehensive as possible, we may have overlooked important keywords in the search or missed relevant articles when hand-searching.

Conclusion

Efforts to ensure patient-centred care that is responsive to the needs of service users must include patients’ experiences in the design, evaluation and monitoring services and determining quality. In an increasingly digital society, where citizens provide and use feedback for a range of goods and services, online patient feedback can offer a convenient, low cost and widely accessible mechanism to capture experiences of health care, while being mindful to avoid issues of digital exclusion. This review provides important insights to inform policy development seeking to harness the opportunities offered by online feedback.

Supplemental Material

Supplemental material for Online patient feedback: a scoping review and stakeholder consultation to guide health policy

Supplemental Material for Online patient feedback: a scoping review and stakeholder consultation to guide health policy by Anne-Marie Boylan, Veronika Williams and John Powell in Journal of Health Services Research & Policy

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was conducted as part of the National Institute for Health Research Health Services and Delivery Research Programme funded project ‘Improving NHS Quality Using Internet Ratings and Experiences’. AMB and JP were also supported by the NIHR Collaboration for Leadership in Applied Health Research and Care at Oxford Health NHS Foundation Trust (NIHR CLAHRC Oxford). The views expressed in this research are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

References

Ziebland

Wyke

Health and illness in a connected world: how might sharing experiences on the internet affect people’s health?

Milbank Q 2012; 90: 219–249.

Verhoef

Van de Belt

Engelen

, et al. Social media rating sites as tools to understanding quality of care: a scoping review. J Med Internet Res 2014; 16: 252–261.

Emmert

Sander

Pisch

Eight questions about physician-rating websites: a systematic review. J Med Internet Res 2013; 15: e24.

Arksey

O'Malley

Scoping studies: towards a methodological framework. Int J Social Res Methodol 2005; 8: 19–32.

Emmert

Meier

Pisch

, et al. Physician choice making and characteristics associated with using physician-rating websites: cross-sectional study. J Med Internet Res 2013; 15: e187.

Hanauer

Zheng

Singer

, et al. Public awareness, perception, and use of online physician rating sites. JAMA 2014; 311: 734–735.

Emmert

Meier

An analysis of online evaluations on a physician rating website: evidence from a German public reporting instrument. J Med Internet Res 2013; 15: e157.

van Velthoven

Atherton

Powell

A cross sectional survey of the UK public to understand use of online ratings and reviews of health services. Patient Educ Couns 2018; 101: 1690–1696.

Hanauer

Zheng

Singer

, et al. Parental awareness and use of online physician rating sites. Pediatrics 2014; 134: e966–e975.

10.

Grabner-Krauter

Waiguny

MK.

Insights into the impact of online physician reviews on patients' decision making: randomized experiment. J Med Internet Res 2015; 17: e93.

11.

Emmert

Meier

Heider

A-K

, et al. What do patients say about their physicians? An analysis of 3000 narrative comments posted on a German physician rating website. Health Policy 2014; 118: 66–73.

12.

Terlutter

Bidmon

Rottl

Who uses physician-rating websites? Differences in sociodemographic variables, psychographic variables, and health status of users and nonusers of physician-rating websites. J Med Internet Res 2014; 16: e97.

13.

Thackeray

Crookston

West

JH.

Correlates of health-related social media use among adults. J Med Internet Res 2013; 15: e21.

14.

Galizzi

Miraldo

Stavropoulou

, et al. Who is more likely to use doctor-rating websites, and why? A cross-sectional study in London. BMJ Open 2012; 2: pii: e001493.

15.

Emmert

Sauter

Jablonski

, et al. Do physicians respond to web-based patient ratings? An analysis of physicians’ responses to more than one million web-based ratings over a six-year period. J Med Internet Res 2017; 19: e275.

16.

Emmert

Meszmer

Sander

Do health care providers use online patient ratings to improve the quality of care? Results from an online-based cross-sectional study. J Med Internet Res 2016; 18: e254.

17.

Kleefstra

Zandbelt

Borghans

, et al. Investigating the potential contribution of patient rating sites to hospital supervision: exploratory results from an interview study in the Netherlands. J Med Internet Res 2016; 18: e201.

18.

van de Belt

Engelen

Verhoef

, et al. Using patient experiences on Dutch social media to supervise health care services: exploratory study. J Med Internet Res 2015; 17: e7.

19.

Emmert

Halling

Meier

Evaluations of dentists on a German physician rating Website: an analysis of the ratings. J Med Internet Res 2015; 17: e15.

20.

Lagu

Goff

Hannon

, et al. A mixed-methods analysis of patient reviews of hospital care in England: implications for public reporting of health care quality data in the United States. Jt Comm J Qual Patient Saf 2013; 39: 7–15.

21.

Sobin

Goyal

Trends of online ratings of otolaryngologists: what do your patients really think of you?

JAMA Otolaryngol Head Neck Surg 2014; 140: 635–638.

22.

Greaves

Pape

King

, et al. Associations between web-based patient ratings and objective measures of hospital quality. Arch Intern Med 2012; 172: 435–436.

23.

Detz

Lopez

Sarkar

Long-term doctor-patient relationships: patient perspective from online reviews. J Med Internet Res 2013; 15: e131.

24.

Brody

Elhadad

An unsupervised aspect-sentiment model for online reviews. Human language technologies: the 2010 annual conference of the North American chapter of the association for computational linguistics. Los Angeles, CA: Association for Computational Linguistics, 2010, pp.804–812.

25.

Brookes

Baker

What does patient feedback reveal about the NHS? A mixed methods study of comments posted to the NHS Choices online service. BMJ Open 2017; 7: e013821.

26.

Macdonald

Beaudin

Pineda

What do patients think about dental services in Quebec? Analysis of a dentist rating website. J Can Dent Assoc 2015; 81: f3.

27.

Adams

SA.

Sourcing the crowd for health services improvement: the reflexive patient and “share-your-experience” websites. Soc Sci Med 2011; 72: 1069–1076.

28.

Lopez

Detz

Ratanawongsa

, et al. What patients say about their doctors online: a qualitative content analysis. J Gen Intern Med 2012; 27: 685–692.

29.

Kinast

Barker

Day

, et al. Factors related to online patient satisfaction with ophthalmologists. Ophthalmology 2014; 121: 1843–1845.

30.

Smith

Lipoff

JB.

Evaluation of dermatology practice online reviews: lessons from qualitative analysis. JAMA Dermatol 2016; 152: 153–157.

31.

Emmert

Sander

Esslinger

, et al. Public reporting in Germany: the content of physician rating websites. Methods Inf Med Suppl 2012; 51: 112–120.

32.

Bardach

Hibbard

Greaves

, et al. Sources of traffic and visitors’ preferences regarding online public reports of quality: web analytics and online survey results. J Med Internet Res 2015; 17: e102.

33.

Lewis

Kobayashi

Gupta

An online review of plastic surgeons in southern California. Ann Plast Surg 2015; 74: S66–S70.

34.

Patel

Cain

Neailey

, et al. Exploring patients’ views toward giving web-based feedback and ratings to general practitioners in England: a qualitative descriptive study. J Med Internet Res 2016; 18: e217.

35.

McLennan

Strech

Kahrass

Why are so few patients rating their physicians on German physician rating websites? A qualitative study. BMC Health Serv Res 2018; 18: 670.

36.

Feng

Chen

, et al. Physician review websites: effects of the proportion and position of negative reviews on readers’ willingness to choose the doctor. J Health Comm 2015; 20: 453–461.

37.

Kanouse

Schlesinger

Shaller

, et al. How patient comments affect consumers’ use of physician performance measures. Med Care 2016; 54: 24–31.

38.

Patel

Cain

Neailey

, et al. General practitioners’ concerns about online patient feedback: findings from a descriptive exploratory qualitative study in England. J Med Internet Res 2015; 17: e276.

39.

Johnson

Survey finds physicians very wary of doctor ratings. Physician Exec 2013; 39: 6–8.

40.

Samora

Lifchez

Blazar

, et al. Physician-rating web sites: ethical implications. J Hand Surg Am 2016; 41: 104–110.

41.

Bardach

Asteria-Penaloza

Boscardin

, et al. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Qual Saf 2013; 22: 194–202.

42.

Emmert

Adelhardt

Sander

, et al. A cross-sectional study assessing the association between online ratings and structural and quality of care measures: results from two German physician rating websites. BMC Health Serv Res 2015; 15: 414.

43.

Greaves

Ramirez-Cano

Millett

, et al. Use of sentiment analysis for capturing patient experience from free-text comments posted online. J Med Internet Res 2013; 15: e239.

44.

Greaves

Pape

King

, et al. Associations between Internet-based patient ratings and conventional surveys of patient experience in the English NHS: an observational study. BMJ Qual Saf 2012; 21: 600–605.

45.

Ranard

Werner

Antanavicius

, et al. Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Affairs 2016; 35: 697–705.

46.

Adams

Post-panoptic surveillance through healthcare rating sites: who’s watching whom?

Inf Comm Soc 2013; 16: 215–235.

47.

Greaves

Laverty

Ramirez-Cano

, et al. Analysis of patients’ comments about hospitals in the English NHS via Twitter, and comparison with patient surveys. BMJ Qual Saf 2014; 23: 348.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.79 MB