Poor Online Patient Ratings of Otolaryngologists in the United States: What are Patients Saying?

Abstract

Objectives

Online patient forums have become a platform for patient education and advocacy in many areas of medicine. The anonymity provided by such forums may encourage honest, candid responses. Using patient online reviews, this study sought to explore themes that arose from negatively perceived care interactions with American otolaryngologists using the Accreditation Council for Graduate Medical Education (ACGME) competency framework.

Study Design

Qualitative thematic analysis.

Methods

Through an iterative multistep process, a qualitative thematic analysis was conducted on negative reviews (defined as ratings of two or less out of five) of all American otolaryngologists found on a popular online physician-rating website (RateMDs.com).

Results

A systematic search through the RateMDs website revealed 2950 separate comments of negative reviews. Of these negative reviews, 350 were randomly selected for thematic analysis. The predominant themes that emerged aligned closely with the Accreditation Council for Graduate Medical Education (ACGME) competencies, in particularly with professionalism and interprofessional skills and communication.

Conclusions

The negative reviews of American otolaryngologists revealed a number of areas where improvements could be made to quality of care. Patients value evidence-based medicine delivered by compassionate and respectful physicians. Isolating and aligning predominant themes within the ACGME framework proved a productive method to collect and organize pertinent patient feedback and integrate teaching into the post-graduate training and continuing professional development in order to avoid such negatively perceived interactions in the future.

Keywords

qualitative research healthcare quality assessments core competency/competencies continuing education medical education

Introduction

Physician-rating websites have gained popularity and are frequently used by patients.^1,2 Despite the increased popularity of these sites, many physicians view them unfavorably.^3,4 Available online platforms provide the public visibility for unfiltered feedback and retroaction of healthcare encounters, often free of charge and protected by anonymity.^5-7 Opponents fear defamation, false accusations, and inability to respond in preserving at all cost confidentiality.^8-11 Negative feelings aside, 69% of physicians report having checked their online profile at least once.³ In addition to using online resources to research health topics, patients are using these resources to access information about their physicians.^12-16 Such information is easily available as more than 90% of physicians have professional information available online.¹⁷

A recent American study specific to online ratings of otolaryngologists (OTLs) revealed largely positive reviews.¹⁸ Amongst negative narratives, Calixto et al. (2018) observed that content frequently pertained to unsatisfactory bedside manner, wait time, accessibility, clinical staff, and environment.¹⁹ It has long been reported that patient satisfaction is strongly correlated with maintaining the patient-physician relationship, patient adherence, and reported outcomes. A further exploration of negative reviews is essential as it may provide cues for improvements in quality and delivery of care.

Using patient online reviews, this study sought to explore themes that arose from negatively perceived care interactions with American otolaryngologists using the Accreditation Council for Graduate Medical Education (ACGME) competency framework.

Materials and Methods

Due to the public availability of the online comments, ethics approval was waived for this study.

Study Design

This qualitative thematic analysis used a deductive, constant comparative analysis method applied to a forum. The content used for assessment was retrieved from RateMDs, an online patient rating platform, in January 2016.

Search Strategy

Reviews were obtained by selecting “Specialty: Ears Nose and Throat (ENT) doctor” on RateMDs.com for all American states.²⁰ This well-known English-language online forum was selected given its popularity and accessibility. Patient reviews that were given a negative global score (i.e., two or less out of a five-point Likert scale, with three considered neutral) were included in our study. If a physician received greater than five negative comments, only the most recent five were included.

Data Analysis

Online patient responses fitting the inclusion criteria were extracted from RateMDs.com into Microsoft Excel Workbook (Version 16.22). Each patient’s responses were divided into units of phrases and each phrase was entered into a new row. Using the ACGME competencies as a conceptual framework, a deductive theoretical analysis was completed. Comments were analyzed to develop a coding framework for dominant themes. In this way, emerging themes from the text were translated and categorized into the ACGME core competencies.

Phrases were coded through an iterative multistep process to identify major themes and sub-themes. A random sequence generator was used to establish a list of random list of RateMD comments by Microsoft Excel (Version 16.22) entry number. Three independent reviewers were first assigned the same set of 50 entries to code. These codes were used for training purposes to achieve congruence among reviewers. During training and discussion, the framework underwent numerous iterations. Ambiguities and potential missing themes were identified and discussed between the reviewers to clarify the coding framework until consensus was reached. After this time, a second set of 50 entries was assigned. At this point, the reviewers achieved congruence (kappa range 0.8–1.0 of assigned codes) to indicate high inter-rater reliability. The same investigators then coded entries independently. After 350 entries were coded, saturation was reached. This was determined by the saturation ratio, whereby the number of new and unique themes in a run of five comments was below the threshold (<5%). This signified that saturation was reached and further coding would have been redundant. In addition to the major and sub-themes, thematic analysis was used to assess the indirect emotion presented in the data. Coding for the indirect emotion was developed from common themes including punctuation, acronyms, tone, and sarcasm. The final coding framework can be found in Tables 2 and 3.

Results

Patient reviews were anonymized and recorded in a Microsoft (Microsoft Corporation, Redmond, WA) Excel 2011 database (Version 16.22) in order of global physician rating. A total of 2950 entries were extracted. Comments corresponded to 1455 OTLs, with an average star rating of 1.5.

A breakdown of OTLs included by state is presented in Table 1. As per the 2010–2019 American population census, states were classified in decreasing order of resident population.²¹ The three states with the highest number of OTLs with negative reviews were California, Florida, and New York. Conversely, the three states with the least number of OTLs with negative reviews were Alaska, North Dakota, and Wyoming. There was relative overlap with the geographic distribution and population, such that the states with the greatest number of negative reviews are also most populated and vice versa.

Table 1.

Percent of Otolaryngologists Included by State.

State	Percent (%)	State (Cont’d)	Percent (%)
California	9.35	Kentucky	1.10
Texas	7.90	Oregon	1.10
Florida	8.93	Oklahoma	1.44
New York	10.03	Connecticut	2.06
Illinois	2.75	Utah	0.96
Pennsylvania	3.92	Iowa	0.34
Ohio	1.99	Nevada	1.51
Georgia	3.02	Arkansas	0.41
North Carolina	2.54	Mississippi	0.34
Michigan	3.64	Kansas	0.55
New Jersey	6.60	New Mexico	0.55
Virginia	3.09	Nebraska	0.41
Washington	1.92	West Virginia	0.41
Arizona	3.37	Idaho	0.76
Massachusetts	2.20	Hawaii	0.27
Tennessee	2.13	New Hampshire	0.76
Indiana	1.31	Maine	0.21
Missouri	0.89	Montana	0.21
Maryland	2.34	Rhode Island	0.55
Wisconsin	1.17	Delaware	0.34
Colorado	1.24	South Dakota	0.34
Minnesota	0.89	North Dakota	0.14
South Carolina	0.89	Alaska	0.07
Alabama	1.92	Vermont	0.21
Louisiana	0.76	Wyoming	0.14

While the development of the coding framework was taking place, several salient comments emerged. These comments, as well as the referenced codes can be found in Tables 2 and 3. Patient care, professionalism, and system-based practice were the most prominent ACGME themes. 8.1% of comments rated patient care experience as unhelpful (“this doctor actually said to me ‘I have another patient to see’, pushed two sample bottles of Nasonex at me and said I needed to see a psychiatrist and a neurologist.”), and 5.3% found the encounter was rushed (“Our entire conversation lasted 5 MINUTES!”).

Table 2.

Summary of Themes and Sub-Themes (ACGME).

Theme/Sub-theme	Code	Relative frequency (%)	Representative quote
Patient Care
Unhelpful	PU	8.1	“this doctor actually said to me "I have another patient to see", pushed two sample bottles of Nasonex at me and said I needed to see a psychiatrist and a neurologist.” “If you have a lot of time on your hands, don't care if you actually see the doctor and generally have a low opinion of yourself this may be the doctor for you. Serious waste of time!”
Time spent/rushed	PT	5.3	“Our entire conversation lasted 5 MINUTES!” “I have to say, I was so dissapointed on this Dr. He took only 5 min to see my son and he was very irresponsible, he only checked his throat and he didn't even take a look at his nose and ears and he sent me right away to schedule the surgery, […]”
Perceived poor decision-making/reasoning/pressure for surgery	PP	3.6	“Doctor doesn't even give you time to talk he will say tubes need to be installed and thats the only way.” “This doctor was the worst. He tried to convince me to have my children get a procedure that they did not need. […] He was visibly upset when we opted not to have the expensive procedure he suggested.”
Poor follow-up	PF	3.5	“the worst service and follow through of any doctor i have been patient to...”
Refusal to see	PR	1.3	“I couldnt even get an appt the receptionist told me to go back to my primary care”
Patient safety	PS	1.0	“I found this dangerous and unnecessary”
Lack of advocacy	PL	0.2	“not a doctor with dignity who gave an oath to Hippocrates”
Medical Knowledge
Perceived clinical skill	MC	5.8	“Rather than checking out all of my symptoms, he was firm in the fact he could do nothing and that I needed to live with the condition.” “he misdiagnosed me after using his torture device up my nose. told me i had allergies when i had a sever sinuis infection.”
Perceived failure/complication of treatment	MF	3.5	“performed a "routine" sinus and deviated septum surgery on me and damaged the structure of my nose, causing the bridge to lose shape.”
Perceived surgical skill	MS	1.2	“She completely botched my sinuses and claims to be a cosmetic surgeon.”
Perceived knowledge	MK	0.8	“he seemed as he had no idea what he was doing.”
Patient Based Learning and Improvement
Lack of evidence-based practice	LE	1.2	“He didn't even culture me!! He actually told me that if he couldn't see fungus, I didn't have it! “
Lack of recognition of one’s roles and limitations	LR	0.7	“He nearly killed my son in the hospital by misdiagnosis, refusal to listen and final refusal to step aside when he couldn't help.”
Systems-Based Practice
Staff, medical learners	SS	8.7	“The front desk is horrible. Those ladies clearly do not want to work there.” “I don't have a complaint with Dr. NAME but I do with his staff, specifically whoever answers the phones and makes the appointments. […] With this incompetence you might have to catch Dr. NAME in the parking lot to get an appointment.”
Time management (wait for apt and in office)	ST	7.6	“never less than an hour in the waiting room, and then another 20-30 minutes in the room waiting for the doctor.” “I have used Dr. NAME for years. I find his practice now is nearly impossible to get into. […]”
Cost of healthcare	SC	5.6	“To my disbelief I was charged over $2,000” “I was charged for services that was never rerendered. There was no follow up .”
Poor service	SV	0.8	“I felt that I was courteous enough to call and let them know that I might be late. They or he could have extended the same courtesy to me.”
Practice environment	SP	0.8	“and plus his office is huge and so noisy and out of control.”
Medico-legal	SL	0.2	“He has been sued for medical malpractice numerous times, as of this writing he has one open/active medical malpractice case pending against him in the state”
Professionalism
Behavior_(arrogant/condescending/unapproachable/dismissive/rude/lack of empathy)	RB	12.6	“This is the rudest man I have ever met in my life. He insulted me, made derogatory and very personal remarks, used profanity and mocked my condition right at the front desk in front of his staff” “Agree with unsympathetic, rude, arrogant and add a few more similar remarks...worst doctor visit ever.”
Conflict of interest (personal/ financial)	RC	3.8	“He examined me for something I didn't come to him for (tried to bill me for this) and tried to cover up those results and he tried to protect my doctor, his friend, by trying to fool me” “Not very thorough. Poor interaction. Its all about money not care. Thats my honest opinion.”
Untrustworthy	RU	1.5	“Claimed "surgery," when there was none; and indicated that the visit was 45 minutes when she spent less than 18 minutes.”
Privacy and confidentiality	RP	0.2	“They insisted even though I told them i did not feel comfortable in them taking pictures, they had to be taken for my file”
Interpersonal Skills and Communication
Inadequate communication (explanation/listening)	IC	5.0	“After about a 2 minute consult he told me there was nothing wrong. After re-explaining my symptoms to him, he told me to go see a psychiatrist because "it's all in your head.” “If he read the audiogram, he would have known that I couldn't hear what he was saying. If he paid any attention to my asking him "What?"/"Excuse me?"/"Pardon me?", he would have spoke up. You really can't trust someone who has such a lack of interest in what they do. He provided no feedback on my hearing loss. Gave no impression or take on how to approach the issue. The visit was really a total waste of time and energy.”
Poor accuracy/detail in records	IP	1.3	“Even though my visit was a follow-up, in their system they had my diagnosis wrong, as well as a misunderstanding about what medication they had prescribed and why”
Lack of respect for healthcare team members	IL	0.7	“He threw things around the room, shoved the Charge Nurse out of his way, ripped my wife's headphones right out of her ears, yelled at ALL of the nurses and performed some procedure on my wife while she was in her bed and wouldn't give the opportunity for even local anesthesia.”
Informed consent	II	0.2	“Staff moves you along too quickly and if you are having surgery you are given papers to read and pushed out the door.”

Table 3.

Summary of Themes and Sub-Themes (non-ACGME).

Theme/Sub-theme	Code	Relative frequency (%)	Representative quote
Indirect Emotion
Exclamation marks, numerous question marks	IEQ	6.3	“I WOULDN'T LET THIS MAN DO SURGERY ON A CORPSE!!!” “[…]DONT GO HERE!!”
Capitalization of words or sentences	IEC	5.6	“TO SAY THAT NAME WAS EXTREMELY ARROGANT AND PATRONIZING WOULD BE A GROSS UNDERSTATEMENT. “ “STAY AWAY from this charlatan.”
Sarcasm or disrespect	IED	1.0	“Not a doctor but a businessman!” “Avoid this "genius" at all cost if you want to be healthy!”
Hyperbole	IEH	1.0	“The absolute worst doctor ever!” “The absolute worst experience I have ever had with a so-called medical professional. […]NEVER, EVER again. WORTHLESS doctor, and vile staff.”

Moreover, 8.7% had complaints about the staff and medical learners (“The front desk is horrible. Those ladies clearly do not want to work there.”), 7.6% for time management (“never less than an hour in the waiting room, and then another 20–30 min in the room waiting for the doctor.”), and 5.6% for cost of healthcare (“To my disbelief I was charged over $2,000”).

“Importantly, 12.6% were not satisfied with their physicians behavior:

“This is the rudest man I have ever met in my life. He insulted me, made derogatory and very personal remarks, used profanity and mocked my condition right at the front desk in front of his staff”.

Finally, the emotion coding segment of this study revealed that excessive punctuation and capitalization were used quite frequently in the negative comments to exacerbate comments. These codes accounted for 6.3 and 5.6% of all codes assigned, respectively:

“I WOULDN’T LET THIS MAN DO SURGERY ON A CORPSE!!!”

“TO SAY THAT NAME WAS EXTREMELY ARROGANT AND PATRONIZING WOULD BE A GROSS UNDERSTATEMENT”

Sarcasm and disrespect (“Not a doctor but a businessman!”) as well as hyperbole (“The absolute worst doctor ever!”) were both encountered in 1.0% of comments.

Discussion

To our knowledge, this was the first study to specifically examine the content of negative online reviews of American otolaryngologists in relation to ACGME competencies. Negative comments translated well onto the ACGME competencies. The most common themes of negative comments, which were related to perceived unhelpfulness, staff/medical learners, and overall behavior. These sub-themes corresponded to the ACGME competencies of patient care, systems-based practice, and professionalism. Excessive punctuation and capitalization were some of the means used to convey dissatisfaction. Trends in the geographic distribution of online patient negative reviews were also noticed.

In translating the themes from the negative comments onto the pre-existing ACGME competencies framework, learners, educators, and practicing physicians may get a new perspective on patient concerns. The alignment with the core competencies may allow for targeted post-graduate and continuing medical education tailored to these specific areas. This analysis of negative reviews revealed a number of areas where improvement could be made to quality of care. The three most frequently identified codes were “unhelpful,” “staff/medical learners,” and “behavior (arrogant/dismissive/lack of empathy).” None of these three related to medical knowledge, but instead stemmed from the patient care, systems-based practice, and professionalism ACGME competencies. This information confirms that patients value evidence-based medicine delivered by compassionate and respectful physicians.

Physician-rating websites have an increasing presence on the Internet and an unspoken presence in the physician’s office. Regardless of the reliability of the data, patients are increasingly visiting these sites for guidance in choosing healthcare professionals and providing feedback on their experiences. Previous research identified differences between ratings and reviews among subspecialties. In general, facial plastics had lower ratings than other otolaryngology and head and neck surgery subspecialties. In addition, associate professors had lower scores than assistant professors and professors.³ Though the present study did not isolate negative ratings by subspecialty and academic appointment, some of these trends may apply.

Previous research has shown that a majority of physicians check their online ratings.² If OTLs are reviewing these comments, there may be a unique opportunity to reflect and learn. It may serve as an opportunity to see how we are viewed in the eyes of patients, the people to whom we are accountable as physicians. This is why, as the medical community strives to endorse the dual role of professional and healer, it is imperative that the aforementioned areas of unmet patient needs be tackled. Some system-based factors such as insurance claim denials, coverage, and overall policies might not be readily and solely solved in an otolaryngology office. However, other areas such as patient care and professionalism can be improved at an individual level, through education and training. Importantly, the assumption is that the feedback provided online is accurate and valuable. As an open access, unregulated forum, this may prove to be a challenge. Interestingly, other systems such as the Patient Advocacy and Co-Worker Observation Reporting Systems were designed to capture and act upon unsolicited complaints and pitfalls encountered during the delivery of care.^22,23 Altogether, these online platforms can provide the necessary feedback to promote patient advocacy, safety, and satisfaction.

Overall, this provocative overview of negative patient-clinician interactions may serve as a useful tool to identify initiatives in the otolaryngology head and neck surgery field. However, there are a few limitations to note. Through the anonymity of this process, patients naturally feel comfortable in being overtly open in their responses. Patrons are free to express themselves how they see fit. This may cause patients to use stronger language to describe the interaction, which in turn would inflate the results of this study. Patients with experiences at either end of the spectrum may have been more prone to publishing a comment, adding to the inherent selection bias. Additionally, patient reviews were not actively monitored for modifications of the original publication and rerating of the same otolaryngologist in a separated posting. Due to the anonymity of the postings, it was not possible to account for the possibility of multiple posting from the same individual or related to one incident. Furthermore, it is foreseeable that negative statements might have been observed even within good overall ratings, which would not have been captured in this present study. Future work to explore the common themes across all ratings is warranted. Lastly, based on the volume of entries on this site, we did not cross-check the names of the otolaryngologists on the RateMDs site with the national registry. As such, the list of otolaryngologists may not be entirely complete and other online platforms such as Press Ganey may provide valuable insight and complementary catchment for future research.

Conclusions

Analysis of negative reviews of American otolaryngologists by patients reveals a number of areas where improvements could be made to quality of care. By aligning predominant themes within the ACGME framework, we may be able to integrate teaching into the post-graduate training and continuing professional development areas. Furthermore, this may serve as a template for a self-assessment tool for practicing OTLs. Such targeted teaching may mitigate some of these negatively perceived interactions and enhance the delivery of care to future patients.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethics Approval and Consent to Participate

Our institution does not require ethical approval for reporting publicly available data.

Data Availability Statements

All data was accessed online and continues to be available online via .

ORCID iD

Lily HP Nguyen

References

Hanauer

Zheng

Singer

Gebremariam

Davis

. Parental awareness and use of online physician rating sites. Pediatrics. 2014;134(4):e966-e975.

Trehan

Daluiski

. Online patient ratings: why they matter and what they mean. J Hand Surg. 2016;41(2):316-319.

Johnson

Survey finds physicians very wary of doctor ratings. Physician Executive. 2013;39(1):6.

Gao

McCullough

Agarwal

Jha

. A changing landscape of physician quality reporting: analysis of patients’ online ratings of their physicians over a 5-year period. J Med Internet Res. 2012;14(1):e38.

Emmert

Meier

Pisch

Sander

. Physician choice making and characteristics associated with using physician-rating websites: cross-sectional study. J Med Internet Res. 2013;15(8):e187.

Emmert

Sander

Pisch

. Eight questions about physician-rating websites: a systematic review. J Med Internet Res. 2013;15(2):e24.

Hanauer

Zheng

Singer

Gebremariam

Davis

. Public awareness, perception, and use of online physician rating sites. Jama. 2014;311(7):734-735.

Woodward

“Anti-defamation” group seeks to tame the rambunctious world of online doctor reviews.

López

Detz

Ratanawongsa

Sarkar

. What patients say about their doctors online: a qualitative content analysis. J General Intern Med. 2012;27(6):685-692.

10.

Strech

. Ethical principles for physician rating sites. J Med Internet Res. 2011;13(4):e113.

11.

Emmert

Meier

Heider

Dürr

Sander

. What do patients say about their physicians? An analysis of 3000 narrative comments posted on a German physician rating website. Health Policy. 2014;118(1):66-73.

12.

Aungst

. Patients say the darnedest things. You can't stop online ratings, but you can stop fretting about them. Med Econ. 2008;85(23):27.

13.

Bacon

Will doctor rating sites improve standards of care?

Yes. Bmj 2009;338:b1030.

14.

Greaves

Ramirez-Cano

Millett

Darzi

Donaldson

. Harnessing the cloud of patient experience: using social media to detect poor quality healthcare. BMJ Qual Saf. 2013;22(3):251-255.

15.

McCartney

Will doctor rating sites improve the quality of care?

No. Bmj. 2009;338.

16.

Pasternak

Scherger

. Online reviews of physicians: what are your patients posting about you? Family Practice Manag. 2009;16(3):9.

17.

Mostaghimi

Crotty

Landon

. The availability and nature of physician information on the internet. Journal of General Intern Med. 2010;25(11):1152-1156.

18.

Sobin

Goyal

. Trends of online ratings of otolaryngologists: What do your patients really think of you? JAMA Otolaryngol Head Neck Surg. 2014;140(7):635-638.

19.

Calixto

Chiao

Durr

Jiang

. Factors impacting online ratings for otolaryngologists. Ann Otol Rhinol Laryngol. 2018;127(8):521-526.

20.

RateMDs. www.RateMDs.com

21.

United States Census Bureau . Annual estimates of the resident population for the Unites States, Regions, States and Puerto Rico: April 1, 2010 to July 1, 2019. Microsoft Excel file. https://www.census.gov/data/tables/time-series/demo/popest/2010s-state-total.html

22.

Vanderbilt University Medical Center . Vanderbilt center for patient and professional advocacy: the co-worker observation reporting system (CORS) program. https://www.vumc.org/patient-professional-advocacy/cors-program [online] (consulted on November 28th, 2022).

23.

Vanderbilt University Medical Center . Vanderbilt center for patient and professional advocacy: the patient advocacy reporting system (PARS) program [online] https://www.vumc.org/patient-professional-advocacy/pars-program (consulted on November 28th, 2022).