Data Mining of the Reviews from Online Private Doctors

Abstract

Background:

User-generated content shared in the online health communities (OHCs) is becoming a valuable resource for researchers to understand patients' decision-making behaviors in the management of their health. Many studies have focused on how to obtain useful information from online reviews in OHCs.

Introduction:

This study focuses on a telemedicine service called Online Private Doctor (OPD), which is offered by a leading Chinese physician review website (PRW). OPD reviews have not received much attention. By data mining the reviews, our goal is to determine what patients are talking about when they use the OPD service and whether they are satisfied with the service or not.

Materials and Methods:

We used a Python web crawler to collect 41,029 reviews and 84,510 short reviews (labels) of all 5,645 physicians who offered the OPD service on a PRW (haodf.com) in China. Mixed methods (i.e., a literature review, topic discovery, annotation, and a sentiment analysis) were used to determine the information that the OPD reviews are meant to express.

Results:

We discovered that the OPD reviews can be categorized into four subjects: competence (35.1%), communication (29.4%), treatment (26.0%), and convenience (9.5%). In terms of previously discovered topics, we found that competence, communication, and treatment have been discussed before, but convenience is an emerging subject. The sentiment analysis indicated that 93.67% of the reviews indicated positive emotions, and the area under the receiver operating characteristic (ROC) curve is 0.64. Furthermore, the labels indicated that only 0.72% (603/84,570) of reviews were negative toward the OPD service. The subjects of the labels were distributed according to competence (34.7%), communication (23.8%), treatment (33.5%), and convenience (8.0%).

Discussion:

The findings of our study suggest that patients who ever used OPD have been quite satisfied with the service. From their reviews, we discovered that OPD has its special characteristics and is convenient. However, it still has some shortcomings, for example, the quality of the phone connection. In terms of both the platform and the doctors, more efforts should be made to make the OPD better and more regulated.

Conclusion:

OPD is an emerging telemedicine service that still needs more time and space to evolve. For patients, it helps reduce problems such as scheduling and queuing. Therefore, it brings more convenience to people's daily lives. In the future, more attention should be paid to this service, as it is helpful in reducing the uneven distribution of medical resources.

Introduction

In recent years, the internet has developed so much that an increasing number of people turn to web-based searches for information before they go to the hospital to visit health care providers,^1,2 and they would prefer to browse reviews before visiting a doctor.³ Web-based physician reviews help patients express their opinions and share their experiences so they can have a significant impact on the patients' choice of physicians.^4
–6 Therefore, online physician reviews have recently gained increasing attention. Physician review websites (PRWs) grant patients to give a score for both physicians and hospitals.⁷ Despite some subjective reviews, most reviews directly reflect patients' real opinions on physicians or hospitals.⁸ On these online health platforms, there are many reviews from the patients' real experiences after they visit doctors. For instance, in the United States, a typical example of a PRW is RateMDs, whereas in China, the representative PRW is Good Doctor Online.⁹ Good Doctor Online is not only a PRW but also includes many online health services, such as online outpatient registration and information on searching for doctors.

A very large population base, large urban/rural differences, and the wide disparity between the rich and poor have led to an imperfect medical system in China.^10,11 Since the 1970s and 1980s, China's overall medical service level has improved significantly, but problems such as weak primary health care service capacity, unreasonable medical treatment for residents, and rapid growth in medical expenses have gradually become prominent.¹² Currently, the greatest difficulty in promoting family doctors in China is the lack of enough qualified doctors. For instance, in some areas, although there are some primary care physicians, residents still do not trust these physicians and prefer to go to large hospitals for treatment. Due to the rapid development of information technologies and the high demand for medical services, it is becoming increasingly popular to deliver health services through the internet. Many internet-based platforms are built to deliver online health services, from which patients can obtain both informational and emotional support.¹³

A special service called Online Private Doctor (OPD), offered by Good Doctor Online, is quite good to help solve the problem of the lack of qualified doctors, as it allows patients to select a doctor and sign and pay a weekly, monthly, or yearly contract with the doctor; in other words, it is similar to employing an OPD. Then, patients can send online messages to the doctors or make a phone call to consult on health issues. This phenomenon is a new trend in visiting doctors, and in some ways, it is also a pattern of telemedicine. Telemedicine addresses two problems facing the health care system: inadequate access and uneven resource distribution.¹⁴ The instructions for an OPD contract are shown in Figure 1.

Fig. 1.

Graphical instructions for an OPD contract. OPD, Online Private Doctor. Color images are available online.

To the best of our knowledge, although many studies have focused on Good Doctor Online as a PRW, few studies are related to the OPD reviews. Therefore, we try to determine whether there are some new insights in the OPD reviews, and we compare these reviews with other online reviews from traditional doctor consultations^{1,2,8,9,15

–18} to determine whether there are some interesting phenomena in the OPD reviews.

The rest of the article is organized as follows. First, the study reviews related works. Then, it introduces the methodological design. Subsequently, it illustrates and discusses the results. Finally, the article concludes with the major findings and their implications.

Related Works

Many studies have used different research methods to discover what patients are most concerned about when they participate in the online health communities (OHCs), such as surveys,^16,19,20 statistical content analyses,²¹ and data mining.²² In Table 1, we list some previous studies on the topics that have already been found. But sometimes, the scores were likely to be biased because of different specialties or evaluation systems. Thus, conveying consumer ratings data in a clear, fair way, really affects health care consumers' decisions.²³

Table 1.

Previous Related Topic-Modeling Works

AUTHOR	TOPICS DISCOVERED IN THE PHYSICIAN REVIEWS
Liu et al.¹	Demeanor, clinical outcomes, office or staff, scheduling, billing
Li et al.²	Helpful, knowledge, punctual, staff
Wallace et al.⁸	Medical ethics, competence, advice and prescriptions, communication skills, financing, operation process, symptoms
Hao et al.⁹	Medical treatment, recommendations, bedside manner
Brody and Elhadad¹⁵	Recommendations, bedside manner, anecdotal, attention, scheduling
Lu et al.¹⁶	Symptoms, complications, examinations, procedures, drugs
Detz et al¹⁷	Personality traits, technique competence, communication, access to the physician, office staff/environment
Hao and Zhang¹⁸	Treatment effects, technique skills, registration, bedside manner, finding doctors

PRWs are a popular trend worldwide. RateMDs was reported to be among the top ten most visited PRWs. PRWs have become more common in the German online environment.²⁰ The awareness of using PRWs to provide feedback about a general practitioner (GP) is also high in Great Britain.²⁴ These PRWs, which are also online communities, can serve as a low-cost alternative or complement to existing health programs.

Prior research highly related to our study is summarized in Table 1. The most relevant study is from Hao and Zhang,¹⁸ as they also used latent Dirichlet allocation (LDA) to derive hidden topics from online physician reviews, and performed a noteworthy comparison of topic discoveries between the United States and China. However, there are some significant differences between our study and theirs. For Hao and Zhang, they just used the data-driven method to extract the topics. However, in this study, we took related topic discoveries in the literature as a theoretical basis to support our data-driven approach. In addition, they tried to compare the differences in the web-based physician reviews between China and the United States, while we limited our study to China and tried to make comparisons between the ordinary online physician reviews and the newly emerging OPD reviews.

Materials and Methods

Data

In this study, the PRWs that we examined are from a special service called OPD, offered by haodf.com, which is one of the leading OHCs in China. We crawled the homepages of 5,645 physicians and obtained a total of 41,029 reviews. After cleaning the incomplete or invalid data, 22,530 reviews of 2,588 physicians were collected, and all of these reviews are of the physicians who do offer the OPD service.

In addition, some patients are not willing to write reviews, as it is time consuming, so these patients are more likely to click simple labels than to write reviews. The labels on the website are all created by previous patients, and every patient is authorized to add new ones when they write reviews. Then, the new labels are saved in the system and are later selected by other patients. We crawled all of the labels of the OPD reviews and obtained a total of 84,570 labels from 26 categories.

Preprocessing the raw data is very important for the quality of the data-mining process. To analyze the reviews with a text-mining technology, the first step is word segmentation, which is the basis for natural language processing (NLP). In our research, all the reviews were Chinese texts, so a useful Python Kit called Jieba was adopted. This module helped segment the Chinese sentences into separate words. Then, the stop words had to be filtered out. In NLP, stop word removal refers to the terms that are widespread and frequent in any kind of text.²⁵ After removing the stop words, the result of the NLP becomes more accurate.

Figure 2 illustrates the framework of our study.

Fig. 2.

The structure of the study. Color images are available online.

Topic Discovery

Topic modeling is an NLP technique for discovering the topics that occur in collections of documents.²⁶ The topics which are derived from the reviews are important as they can offer a better perspective to observe what the patients actually cared about. In this study, we choose the LDA model as a topic-modeling approach because it is often used as a statistical model to discover unobserved information that depended on observed data.²⁷ The LDA model considers each document in a corpus as a random mixture of latent topics, meanwhile, each latent topic is characterized by a distribution of words.^28,29 Thus, a three-layer structure model could be considered; the three layers refer to the document, topic, and word. It builds a topic-per-document model and a words per-topic-model, which are modeled as Dirichlet distributions. To sum up, the process of how the LDA model works can be regarded as two phases. First, choose a distribution including K topics. Then, pick up a topic randomly from the topic distribution, and draw a word from that topic according to the topic's word probability distribution.²⁹

Gensim, an open-source library for unsupervised topic modeling and NLP, is our topic-modeling tool.

Sentiment Analysis

Reviews express subjective information, such as the author's thoughts and opinions about certain topics. A sentiment analysis is used to identify positive and negative opinions, emotions, and evaluations. It helps to classify the given text into one specific polarity: positive or negative (or neutral).³⁰

A sentiment analysis is able to obtain the emotional tendency of the text through text mining and other means. Through a sentiment analysis, we can understand whether the patient's attitude toward the doctor is positive or not, and we can understand the acceptance and satisfaction with the OPD service.

We employed SnowNLP to perform the sentiment analysis, which is a Python Kit that specializes in sentiment analyses of Chinese texts. The algorithm of SnowNLP is actually a Naive Bayes algorithm: a simple probabilistic model often used for binary classification.³¹ The return value for each sentence is the probability of a positive emotion. Thus, the closer the value is to 1, the more likely it is to be a positive emotion; the closer the value is to 0, the more likely it is to be a negative emotion. For example, one review said that “Doctor Wang is very kind to his patients. He tried his best to explain my condition to me, which was comforting.” The emotional value analyzed by SnowNLP is 0.998, which means the probability that the text is positive is 99.8%.

Results

Topic Discovery

With many repeated experiments, we found that the LDA model performed best and identified the most easily explained and clearest topics when the cluster topic number was set to 10. The results are shown in Table 2. While each topic has its specific key words, we have provided an explanation for each topic.

Table 2.

Topic Discovery of Online Private Doctor's Reviews

NO.	KEY WORDS	EXPLANATION
Topic 1	Skill, state of illness, treatment, well, price, comment, words, professional, attitude, consult	Doctors are equipped with great skills.
Topic 2	Careful, SMS, thanks a lot, reply, solution, doctor, skill, price, appreciation, professor	Doctors carefully reply through SMS.
Topic 3	Listen, doctor, illness, good, give, heart, express, careful, solution, professor	Doctors carefully listen to the patients.
Topic 4	Suggestions, details, medicine, consult, know, surgery, worried, hospital, service	Doctors give suggestions about the medicine.
Topic 5	Assured, treatment, directions, service, illness, director, appreciate, patience, easy, recent	Doctors give directions on the treatment.
Topic 6	Enthusiastic, communication, diagnosis, patients, be able to, specify, experience, in place, remember, someone	Doctors are enthusiastic and have experience in diagnosing patients.
Topic 7	Meet, lucky, help, professional, relationship, receive, service, long time, rules, new	Patients feel lucky when meeting the doctor and when building a good relationship.
Topic 8	Eliminate, confusion, doctors, patience, grateful, professional, condition, clear, director, explanation	Doctors help the patients to understand the condition.
Topic 9	Useful, attitude, plan, thanks, listen, doctors, hurry, patient, message, talk	Doctors offer a useful plan for patients.
Topic 10	Phone, doctors, good, convenient, schedule, really, considerate, symptoms, attitude, next time	Talking with doctors through the phone is convenient.

SMS, short message service.

The 10 topics above are more or less similar. To better explain this finding, we categorized these 10 small topics into 4 subjects based on previous research: competence, communication, treatment, and convenience. Table 3 shows the details.

Table 3.

Four Subjects of the Online Private Doctor Reviews

CATEGORY	TOPIC NOS.	COUNT (%)	EXPLANATION
Competence	1, 6, and 9	35.1	The quality and skill of the doctor.
Communication	3 and 7	29.4	The communication skills of the doctor.
Treatment	4, 5, and 8	26.0	Diagnosis, prescriptions, and medicines.
Convenience	2 and 10	9.5	This consultation mode is convenient.

Competence, communication, and treatment are the common topics discovered by prior studies, as shown in Table 1. The competence subject mainly refers to the quality that the physicians are supposed to have, such as being professional, experienced, skilled, patient, and kind. It is the most important subject that patients are affected by when giving OPD reviews (35.1%). Medicine is a complicated field, and for doctors, establishing trust from patients mostly depends on their own competence, so being qualified is always the priority. The second most important subject (29.4%) is communication. For many patients, communication is quite important when meeting a doctor. Some of these patients not only hope to receive medical treatment but also emotional comfort. If the communication is good enough, they will feel relaxed, and a good relationship between both sides is built. Next is the subject of treatment (26.0%), which includes the diagnosis, prescriptions, and medicines. When quickly browsing the reviews, we found that patients are not that likely to share thoughts and opinions related to their personal illness, such as symptoms and drugs, as they are quite private. They are more willing to express simply that they received good treatment.

The three discovered topics in our study (i.e., competence, communication, and treatment) are quite consistent with previous research. In the OPD reviews, convenience is a special topic that is highly related to the typicality of the OPD service. Some patients wrote comments such as “I feel it's so convenient to call doctors at home. Doctor Lin listened to me carefully and was patient enough” and “I saved much time because I contacted a doctor online.” These reviews are the opposite of the ordinary online physician reviews, for which scheduling is one the most important topics for both patients and doctors. OPD offers a convenient way to solve scheduling problems for both patients and doctors. Patients can easily connect with doctors, and they do not need to worry about scheduling, as they can choose a time that is the most convenient for them. In addition, with fewer patients rushing into the hospital, the heavy workloads of doctors are alleviated.

Figure 3 presents the word clouds for the 10 topics shown in the Table 4. The larger the font size of the word is, the more representative of the special topic. With these images, we can observe the topics graphically and more intuitively. The highest frequency words are “skill,” “careful,” “phone,” “suggestions,” etc.

Fig. 3.

Ten topic word clouds for the OPD reviews. Color images are available online.

Table 4.

The Results of the Sentiment Analysis of the Online Private Doctor Reviews

	COUNT	PERCENTAGE
Negative sentiment	1,363	6.33
Positive sentiment	20,167	93.67

Sentiment Analysis

We employed SnowNLP to analyze sentiment. First, we need to train our data to fit the model. We invited two researchers to choose 500 positive and negative reviews each, and we saved the chosen texts. Then, we used these specific 1,000 reviews to train the model, and the rest were used to perform a sentiment analysis by using the trained model. When the probability is above 0.5, the emotion is regarded as positive; otherwise, it is regarded as negative.

The final results in Table 4 shows that patients are quite satisfied with the OPD service, as the positive sentiment proportion is ∼93.67%.

To evaluate the accuracy of our model performance, we randomly selected 100 reviews; two researchers helped us annotate whether each was positive or not, and we then used these labeled data as the test set. Then, we used the sentiment scores from SnowNLP as the prediction set. After obtaining the test set and the prediction set, we used a receiver operating characteristic (ROC) curve to obtain the true positive rate and the false positive rate. The true positive rate means that all of the positive comments are correctly identified as positive by the algorithm.³² On the other hand, the false positive rate means that all of the negative comments are mistakenly identified as positive. The area under the ROC curve (AUC) also represents the accuracy of the classifier. If the value of the AUC is between 0.5 and 1, the accuracy of this classifier is better than that of a random guess. In our case, the AUC is 0.6359, which indicates that the result of the sentiment score is quite satisfactory. Figure 4 shows the ROC curve of our study.

Fig. 4.

ROC curve for evaluating the sentiment analysis. ROC, receiver operating characteristic. Color images are available online.

Labels of the OPD Reviews

In total, 26 types of labels were classified as positive or negative by our study group: 12 types are positive, and 14 are negative. Additionally, each label is annotated with the topic we discovered. The results are shown in Table 5.

Table 5.

Subject of Positive and Negative Review Labels

LABEL NAME	PERCENTAGE (COUNT)	SUBJECT
Positive
Given clear advice	17.08 (14,447)	Treatment
Very patient	16.11 (13,627)	Communication
Fully explained	15.30 (12,939)	Competence
Eliminated my confusion	12.32 (10,422)	Competence
Clarified the treatment directions	12.12 (10,247)	Treatment
More convenient than going to a hospital	8.03 (6,794)	Convenience
Knew my condition very well	7.10 (6,007)	Competence
Considerate and thoughtful	3.87 (3,270)	Communication
Asked for details about the state of my illness	3.77 (3,189)	Communication
Emphasized many precautions	3.38 (2,858)	Treatment
Offered some opinions	1.58 (134)	Treatment
Useful drug guidance	0.039 (33)	Treatment
Total	99.28 (83,967)
Negative
Did not answer questions seriously	0.11 (98)	Communication
Did not meet expectations	0.088 (75)	Competence
Offered little explanation	0.078 (66)	Treatment
Answered questions too simply	0.078 (66)	Competence
Not patient	0.076 (64)	Communication
No advice offered	0.057 (48)	Treatment
No good treatment advice offered	0.051 (44)	Treatment
Answered my questions vaguely	0.049 (42)	Communication
Just asked a few questions	0.048 (41)	Communication
Did not know my condition at all	0.034 (29)	Competence
Too noisy	0.011 (10)	Convenience
Spent a long time waiting for the line to connect	0.01 (9)	Convenience
Spoke only a few words	0.007 (6)	Convenience
Hung up the phone after saying a few words	0.006 (5)	Convenience
Total	0.72 (603)

From Table 5, we can see that among all the review labels, the majority ones are positive (99.28%), whereas only 0.72% are negative (0.72%). As observed from the labels, patients are sufficiently satisfied with the doctors involved with the OPD service. Thus, the sentiment analysis of the reviews is quite consistent with the labels: the positive reviews accounts for 93.72% of all reviews. In conclusion, most patients are satisfied with the OPD service.

The top three selected labels are “given clear advice,” “very patient,” and “fully explained,,” which are all positive and refer to treatment, communication, competence, respectively. Additionally, some patients (8.03%) believe that this kind of service is “much more convenient than going to a hospital,” which is a compliment for OPD's convenience. However, there are some complaints about the phone consultation, such as being “too noisy,” “long time waiting to connect the line,” “hung up the phone after saying a few words.” These labels describe some drawbacks when consulting doctors online, such as the poor quality of the voice call. In other words, the labels related to convenience are special and different from ordinary online reviews.

To count the labels in the subject, the annotation result shows that a large number of labels are about competence (34.7%), followed by treatment (33.5%), communication (23.8%), and convenience (8.0%). Figure 5 shows the proportion that each subject takes both in the reviews and labels. Competence is always the most significant point both in reviews and labels, while for treatment and communication, there is little difference. Both in the reviews and labels, some patients would usually say something about OPD's convenience, as it is quite convenient but still with some shortcomings that need to be improved.

Fig. 5.

The proportion of discovered subjects in the reviews. Color images are available online.

Discussion

This article collected nearly 20,000 reviews from OPD and conducted a topic discovery analysis and sentiment analysis on these OPD reviews. During the topic discovery process, we found that these reviews could be categorized into four types, including competence (35.1%), communication (29.4%), treatment (26.0%), and convenience (9.5%). Convenience is an interesting topic that never appears in the previous research related to online physician reviews, as OPD is such a new service and these reviews have not been focused on before. In addition, this article conducted a sentiment analysis to investigate the patients' opinions of the OPD service; 93.67% of the review texts show positive emotions, and the accuracy of the sentiment classification algorithm is 0.64, which is a good accuracy level for a sentiment analysis. Moreover, we collected 83,000 labels as short reviews, totaling 26 categories. With our annotation, 99.72% of them were positive, which is quite consistent with the result of the sentiment analysis. The distribution of the subject's labels (i.e., competence [34.7%], treatment [33.5%], communication [23.8%], and convenience [8.03%]) is also consistent with that of the reviews.

With the advent of Web2.0, user-generated content has become increasingly popular. There are already many studies of OHC reviews worldwide, including various websites and diversified diseases. The PRW our study focused on was Good Doctor Online. There are many studies based on this website, as it is very well known in China. However, few studies are connected to the OPD service, which is a special service for medical conditions in China. Therefore, we try to understand the OPD service by data mining the reviews. Then, we found that OPD has received a great reputation from patients, and we discovered people like to talk about competence, treatment, communication, and convenience when they comment on doctors. Convenience is an interesting insight, as it has never appeared in the prior research. Our research has theoretically enriched the studies of PRWs.

Practically, this study discovered what patients care about when they use the OPD service, and we now know their attitudes toward the service. From the negative labels, we know that the quality of the phone connection is quite a problem for OPD. Therefore, the platform should find some solution to this problem, such as suggesting that patients consult more through the internet. This suggestion can help the platform to rethink if there is any problem with its policies and how to make any improvement to better serve the patients and bring them more convenience.

In summary, the performance of Chinese health care nowadays is not satisfactory enough, so there is much room to make efforts, such as a quicker response to patients, better efficiency, and reduced cost. The OPD service is not only a major reform of the traditional medical industry but also a major innovation in the internet era. China is such a large country with a very large population, so medical resources are limited and distributed inequitably. The OPD service is significantly helpful for those patients who cannot go to hospital or find great doctors. In short, it is no doubt a great approach to enrich the bilateral communication and build a great relationship between both sides.

Our study provides interesting insights but has a number of limitations. First, only one disease was examined in the research data. Reviews of different types of diseases may differ (e.g., chronic vs. acute, serious vs. mild, and male vs. female), and the hidden topics and sentiment expressions of these reviews also differ. Therefore, in the future, we should try to obtain a variety of data on different diseases. Second, the data were from only one Chinese PRW, and the result may be slightly biased. There are some people who doubt that the Good Doctor Online has its own auditing system and that the negative reviews are not supposed to be published on the website. Then, the reviews with strong negative opinions are possibly filtered. Therefore, more time and energy should be put to expand the data source, and obtaining more high-quality reviews should be considered. Last but not least, the OPD service is essentially a special family doctor service, which is quite developed in some western countries. Therefore, we should look further to foreign studies on family doctors, or so-called GPs to be inspired to improve the service itself.

Footnotes

Disclosure Statement

No competing financial interests exist.

Funding Information

This research was supported by the Natural Science Foundation of Shanghai under grant number 19ZR1419400.

References

Liu

, Uffenheimer

, Nasseri

, Cohen

, Ellenhorn

. “But his yelp reviews are awful!”: Analysis of general surgeons' Yelp reviews. J Med Internet Res, 2019; 21:e11646.

, Liu

, Li

, Liu

. Developing embedded taxonomy and mining patients' interests from web-based physician reviews: Mixed-methods approach. J Med Internet Res, 2018; 20:e254.

Hong

, Liang

, Radcliff

, Wigfall

, Street

. What do patients say about doctors online? A systematic review of studies on patient online reviews. J Med Internet Res. Chicago, IL, 2019;21: e1, 2521.

Alodadi

, Zhou

. Predicting the helpfulness of online physician reviews. IEEE International Conference on Healthcare Informatics (ICHI). 2016; 1–6.

Harst

, Lantzsch

, Scheibe

. Theories predicting end-user acceptance of telemedicine use: Systematic review. J Med Internet Res, 2019; 21:e13117.

Yang

, Zhang

. Investigating the effect of paid and free feedback about physicians' telemedicine services on patients' and physicians' behaviors: Panel data analysis. J Med Internet Res, 2019; 21:e12156.

Hao

. The development of online doctor reviews in China: An analysis of the largest online doctor review website in China. J Med Internet Res, 2015; 17:e134.

Wallace

, Paul

, Sarkar

, Trikalinos

, Dredze

. A large-scale quantitative analysis of latent factors and sentiment in online doctor reviews. J Am Med Inform Assoc, 2014; 21:1098–1103.

Hao

, Zhang

, Wang

, Gao

. A tale of two countries: International comparison of online doctor reviews between China and the United States. Int J Med Inf, 2017; 99:37–44.

10.

Eggleston

, Li

, Meng

, Lindelow

, Wagstaff

. Health service delivery in China: A literature review. Health Econ, 2008; 17:149–165.

11.

Blumenthal

, Hsiao

. Privatization and its discontents—The evolving Chinese health care system. N Engl J Med, 2005; 353:1165–1170.

12.

Yin

, Zhang

, Wang

, et al. Delivery status of contractual services from family doctors in China and advances in the related studies [in Chinese]. Chinese Gen Pract, 2018; 21:753–760.

13.

Zhang

, Guo

, Lai

, Yin

, Meng

. From offline healthcare to online health service: The role of offline healthcare satisfaction and habits. J Electron Commer Res, 2017; 18:138–154.

14.

Miller

. Telemedicine and doctor–patient communication: A theoretical framework for evaluation. J Telemed Telecare, 2002; 8:311–318.

15.

Brody

, Elhadad

. Detecting salient aspects in online reviews of health providers. AMIA Annu Symp Proc, 2010; 2010:202–206.

16.

, Zhang

, Liu

, Li

, Deng

. Health-related hot topic detection in online communities using text clustering. PLoS One, 2018; 8:e56221.

17.

Detz

, López

, Sarkar

. Long-term doctor-patient relationships: Patient perspective from online reviews. J Med Internet Res, 2013; 15:e131.

18.

Hao

, Zhang

. The voice of Chinese health consumers: A text mining approach to web-based physician reviews. J Med Internet Res, 2016; 18:e108.

19.

Daskivich

, Luu

, Noah

, Fuller

, Anger

, Spiegel

. Differences in online consumer ratings of health care providers across medical, surgical, and allied health specialties: Observational study of 212,933 providers. J Med Internet Res, 2018; 20:e176.

20.

Emmert

, Meier

, Pisch

, Sander

. Physician choice making and characteristics associated with using physician-rating websites: Cross-sectional study. J Med Internet Res, 2013; 15:e187.

21.

Carbonell

, Brand

. Choosing a physician on social media: Comments and ratings of users are more important than the qualification of a physician. Int J Hum-Comput Int, 2018; 34:117–128.

22.

Deng

, Hong

, Zhang

, Evans

, Chen

. The effect of online effort and reputation of physicians on patients' choice: 3-Wave data analysis of China's Good Doctor Website. J Med Internet Res, 2019; 21:e10170.

23.

Zhang

, Bantum

, Owen

, Bakken

, Elhadad

. Online cancer communities as informatics intervention for social support: Conceptualization, characterization, and impact. J Am Med Inform Assoc, 2017; 24:451–459.

24.

Patel

, Cain

, Neailey

, Hooberman

. Public awareness, usage, and predictors for the use of doctor rating websites: Cross-sectional study in England. J Med Internet Res, 2018; 20:e243.

25.

Ferilli

, Esposito

, Griecoa

. Automatic learning of linguistic resources for stopword removal and stemming from Text. Proc Comput Sci, 2014; 38:116–123.

26.

Menner

, Höpken

, Fuchs

, Lexhagen

. Topic detection: Identifying relevant topics in tourism reviews. Sping Comp Sci, 2016; 411–423.

27.

Blei

, Carin

, Dunson

. Probabilistic topic models: A focus on graphical model design and applications to document and image analysis. IEEE Signal Process Mag, 2010; 27:55–65.

28.

Lin

, He

. Joint sentiment/topic model for sentiment analysis. Proceedings of the 18th ACM Conference on Information and Knowledge Management. Hong Kong, China, 2009; 375–384.

29.

Yau

, Porter

, Newman

, Suominen

. Clustering scientific documents with topic modeling. Scientometrics, 2014; 100:767–786.

30.

Wang

, Lu

, Tan

, Impact of product attributes on customer satisfaction: An analysis of online reviews for washing machines. Electron Commer Res App, 2018; 29:1–11.

31.

Chang

, Xu

, Zhou

, Cao

. Research on detection methods based on doc2vec abnormal comments. Future Gener Comp Sy, 2018; 86:656–662.

32.

Hajian-Tilaki

. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med, 2013; 4:627–635.