Net Promoter Score as a Reflection of Patients’ Opinions About Telemedical Visits: A Mixed Methods Analysis

Abstract

Introduction:

In order to assess patient experiences of telemedicine, researchers and administrators use the net promoter score (NPS), based on a likelihood to recommend (LTR) question. However, there is reason to doubt validity of this metric for this purpose. We assessed the degree to which the LTR question reflects actual patient preferences about telemedicine.

Methods:

Using data from a patient experience survey collected in Spring 2020, we compared LTR responses to open comments. Through content analysis, we transformed comments into categorical variables and used those variables in a multiple logistic regression model to predict LTR responses. We also thematically analyzed comments to further elucidate our results.

Results:

Only about half the comments mentioned telemedicine at all. Around 6% of comments were wholly incongruent with LTR responses. In many comments, ideas about telemedicine were semantically entangled with ideas about providers. Our logistic regression found strong associations between sentiments expressed in comments and LTR responses. However, comments about telemedicine were relatively poor predictors for LTR compared to comments about the provider.

Discussion:

NPS, which is included on many patient experience surveys used by health systems across the United States, has limitations for use as a measure of the acceptability of telemedicine for patients. Patients have more than telemedicine in mind when responding to the LTR question, and ratings conflate attitudes about providers, office policies, and staff with the telemedicine modality. More direct measures are necessary for meaningful research on the acceptability and usability of telemedicine for patients.

Introduction

After the COVID-19 pandemic accelerated telemedicine use across health systems,^1,2 health systems sought to evaluate how its continued use impacts patient experience.³ The net promoter score (NPS) is a leading performance metric used by health care executives for strategy development and quality improvement initiatives. Researchers have used NPS to evaluate the acceptability of a variety of health care interventions, including e-Health^4

–8 and telemedicine.^{9

–18} One review found that NPS was the most widely used measure of satisfaction for m-health.¹⁹ However, there is cause to question its validity.

Prior to use in health care settings, NPS was developed in business as an alternative to measuring customer/patient satisfaction.^20,21 Customers respond to a likelihood to recommend (LTR) question on a scale from 0 to 10. People responding with a 9 or 10 are defined as “promoters,” 7 or 8 are “passives,” and 0–6 are “detractors.” NPS is calculated across groups by subtracting percentage of detractors from percentage of promoters.

Use of NPS is debated among business scholars,^22
–24 but it may be even less relevant in health care where lack of choices and urgent need constrain customers.^25
–27 Validity studies find that NPS is less informative and reliable than multi-item satisfaction measures.²⁸ The measure is overly sensitive to patient panel compositions and modes of administration.^25,29,30 The interpretability of responses is uncertain, particularly across cultures and settings,³¹ and the NPS has been shown to lack specificity.^22,32,33

Before using NPS to evaluate telemedicine acceptability, we wanted to assess the extent to which responses to an LTR question on a patient experience survey following a telemedicine encounter reflect attitudes about telemedicine. We inferred that comments would reflect a patient’s state of mind when answering the LTR, so these comments could serve as a secondary measure of patient opinions that could be compared with ratings.

Methods

We used a mixed methods design to understand what LTR is measuring. The approach involved triangulating LTR ratings and qualitative comments. We used content analysis to transform content from comments into categorical data, which could be used in regression models to predict LTR ratings. To better understand regression results, we used inductive analysis to identify themes across comments, including references to telemedicine, understandings of the survey, and entanglements between provider and telemedicine comments.

The Maine Medical Center Institutional Review Board determined this study did not constitute human subjects research, and survey respondents did not provide informed consent.

DATA COLLECTION

The LTR question was included on a survey (see Appendix 1) conducted for our health system by a third party (NRC Health) to get feedback relevant to telemedicine. Patients completed the survey by phone (automated), email, or text message following a telemedicine encounter. They answered nine questions about their experience and then rated the following LTR question on a 0–10 scale: “How likely are you to recommend this service to your family and friends?” A 10 is “extremely likely,” whereas a 0 is “not at all likely.” Finally, patients are given the opportunity to leave a comment.

Table A1.

NRC Patient Satisfaction Survey for Telehealth Encounters

QUESTION TEXT	QUESTION SCALE
Were you able to talk to a care provider in a timely manner?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Was this method of connecting with a care provider easy to use?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Was the quality of the video or call good enough?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Did the care provider give you enough information?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Did the care provider listen carefully to you?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Did you trust the care provider?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Did the care provider seem to know your medical history?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Did you know what to do if you had more questions afterwards?	(1) No, (2) Yes somewhat, (3) Yes mostly, (4) Yes definitely
Where would you have gone, if this service was not available?	Emergency Dept, Walk in UC, Primary Care Provider, Opt for no care, other
For this question, please listen very carefully as you will be using a scale from 0 to 10 and it is very important that we get an accurate response from you.
How likely would you be to recommend this service to your family and friends?	Likely 0–10
What else would you like to say about your experience?	Open Question

SAMPLE

Our dataset included 5,148 encounters, from April until June 2020, representing 5,022 unique patients. For analyses, we excluded 2,874 encounters without comments. For patients with multiple encounters, we included only the patient’s most recent encounter, excluding 43 encounters. The sample for these analyses (n = 2,231) is compared with the larger sample in Table 1. The sample with comments is slightly older, with a greater proportion of women, compared with the larger sample. We do not believe these differences affect our goal to understand the relationship between responses to the LTR question and comments.

Table 1.

Characteristics of Sample

CHARACTERISTIC	COMMENTER, N = 2,231	OVERALL, N = 5,022
Age-group
12 and under	11 (0.5%)^a	149 (3.0%)
18–44	271 (12%)	826 (16%)
45–64	712 (32%)	1,588 (32%)
65–79	1,048 (47%)	2,063 (41%)
80 and over	189 (8.5%)	396 (7.9%)
Sex
Female	1,327 (59%)	2,914 (58%)
Male	904 (41%)	2,106 (42%)
Unknown	0	2
Ethnicity
Non-Hispanic	2,187 (99%)	4,911 (99%)
Hispanic or Other	23 (1.0%)	47 (0.9%)
Unknown	21	64
Race
White or Caucasian	2,141 (97%)	4,795 (97%)
Non-White	60 (2.7%)	159 (3.2%)
Unknown	30	68
Medicaid
Yes	102 (4.6%)	318 (6.3%)
No	2,129 (95%)	4,704 (94%)
Rurality
Metro	830 (39%)	1,841 (39%)
Large rural	623 (30%)	1,482 (31%)
Small or isolated rural	656 (31%)	1,442 (30%)
Unknown	122	257
Existing patient
Yes	2,108 (94%)	4,713 (94%)
No	123 (5.5%)	309 (6.2%)
Physician specialty
Primary	1,138 (54%)	2,424 (51%)
Medical	828 (39%)	1,992 (42%)
Surgical	142 (6.7%)	310 (6.6%)
Unknown	123	296

n(%)

CONTENT ANALYSIS Table 2

We used content analysis, with MAXQDA™ (Version 22.8.0), to transform comments into categorical data.³⁴ We developed a codebook following review of 100 comments (see Table 2 for codebook). One code was used to identify when telemedicine was mentioned in a comment, even implicitly. Other codes captured instances in which participants either directed comments at the survey itself or left comments that included no interpretable information. Finally, we used object-valence codes to capture positive or negative (e.g., reporting problems) valence and the object of the valence, where applicable. Throughout the coding process, M.K. used memoing to note broader patterns (themes) not captured through other codes.

To avoid issues with unitization,³⁵ we applied codes to entire responses, which varied in length. Responses often included mixed commentary on more than one topic, so responses were given as many codes as were applicable. In order to ensure codes were applied consistently and nonarbitrarily,^36,37 we employed a codebook and checked intercoder reliability. A second, novice coder (R.A.) was trained to use the codebook and then blind-coded a subset of 249 comments. Coding decisions were compared using Cohen’s Kappa, with a preset threshold of kappa = 0.7. For all codes, except the Provider Negative code, kappa exceeded the threshold, the minimum being kappa = 0.78. Given the infrequency of Provider Negative code application (used 34 times across 2231 comments), we used the negotiated coding procedure to review this specific code.^35,38

Table 2.

Codebook Used for Content Analysis

CODE	DEFINITION
Telehealth mentioned	Any allusion to the telemedicine format, including implicitly.
No information/Uninterpretable	The comment cannot be given any other code.
Survey	The comment references the survey itself, including to report an error or confusion.
Positive	Any comment that is positive, but which is not clearly attributed to the provider, the format of service provision, or something else specific. Comments about the “visit” or the “appointment” would also go here unless a more specific code applies.
Provider positive	The comment refers to the provider or a quality of the provider in a positive way. If an unidentified person is mentioned in reference to the positive statement, then it should be assumed to apply to the provider unless otherwise stated (e.g., a staff member).
Telehealth positive	The comment refers to the format or nature of the visit as being telemedicine, and assesses it in a positive way. The pronouns “it” or “that” can be assumed in certain contexts to refer to the format, as can “experience.”
Staff positive	The comment refers to a staff member or the office generally in a positive way.
Provider negative	Any statement that refers to or reports a problem or issue that was caused by the provider or suggested to be the responsibility of the provider, based on the respondent’s perspective.
TH negative	Any statement that refers to or reports a problem or issue related to the telemedicine format of the visit. Do not include statements about how telemedicine is new, different, strange or otherwise novel, but which do not report a negative impression of telemedicine (e.g., “I wasn’t sure about meeting my doctor on Zoom...”).
Other negative	Any statement that refers to or reports a problem or issue, which is not about the provider, the format of the appointment, or the office/staff. This includes health issues, issues with medication, or anything else which does not seem to place responsibility with the provider, the staff/office, or the format.
Staff negative	Any issue/problem that refers to the staff or office/clinic specifically. Include issues with transferring records, being able to setup appointments, getting follow-up calls from office, appointment starting late (unless provider is blamed), etc.

MULTIPLE LOGISTIC REGRESSION

We assigned binomial predictor variables based on applications of each content analysis code. The variables were Telehealth Positive, Provider Positive, Staff/Office Positive, Telehealth Negative, Provider Negative, and Staff/Office Negative.

To test whether positive comments about telemedicine would predict Promoter Ratings better than other comments, we used a logistic regression to produce odds ratios. All six individual variables were significant with bivariate modeling, so all six variables were included in the final multiple logistic model.

We used LTR responses to create a binomial outcome variable. Because the LTR ratings were heavily skewed toward recommending (1,813 patients gave ratings of 9 or 10, 266 gave ratings of 7 or 8, and 152 gave ratings of 6 or lower), we combined “passives” and “detractors,” converting LTR ratings into a binary outcome: “promoter” (rating 9–10) or “nonpromoters” (rating 0–8).

The quantitative analysis was conducted using R Version 3.5.1.

Results

Of 2,231 comments, 64 comprised uninterpretable or no information. Fourteen were solely about the survey, including to report errors or confusion. The remaining 2,153 comments included information related to patient experience.

PATIENT COMMENTARY DOES NOT FOCUS ON TELEMEDICINE

Though the survey was conceived to elicit patient feedback on telemedicine, only 50.0% of comments mentioned telemedicine (explicitly or implicitly), and only 46.3% included a positive or negative comment about telemedicine. In contrast, 53.1% of comments included either a positive or a negative comment about providers. In some cases, comments suggested that patients believed the survey was about the provider:

This was my first experience with [Doctor] and she was exceptional. I find it disturbing that you do not have enough faith in your own medical staff and you think I do not have the tools to find a way to communicate my feelings if I found I had a bad experience.

Others indicated confusion about what opinions the survey was trying to elicit:

What kind of question is this? “How likely would you be to recommend this service to your family and friends?” you had questions regarding [Doctor], is this about using telemed?

These comments, and the relative infrequency of comments about telemedicine, suggest that many participants are thinking about other topics than telemedicine when they responded to the LTR question.

INCONGRUENCE BETWEEN THE LTR AND COMMENTS

LTR ratings were sometimes incongruent with telemedicine-related comments. About 7.1% of promoters expressed a problem with or distaste for telemedicine without saying anything positive about it. The following comment was left after responding with “10”:

I understand the reason for this Zoom call and I’m fine with this appointment. I would prefer to actually be in the doctor’s office and I hope Zoom does not become the norm for doctor’s appointments.

Meanwhile, 5.3% of “Detractors” were positive about telemedicine and said nothing negative about it. For example, a respondent who gave a “5” on LTR said:

Grateful for the opportunity to visit a provider without taking any risks at this time. Also the convenience of not dealing with the 45 minute drive helps to relieve additional stress.

Altogether, 6.1% of respondents gave opinions of telemedicine incongruent with their LTR ratings—either promoters expressing problems with telemedicine (and nothing positive) or detractors only expressing positivity about telemedicine. For comparison, LTR ratings were only incongruent with 1.3% of commentary about providers.

ENTANGLED IDEAS ABOUT PROVIDERS AND TELEMEDICINE

Patient telemedicine comments often overlapped with comments about providers. Many patients qualified their assessment of telemedicine based on the provider:

[Doctor]… does an excellent job balancing the use of technology with connecting with the patient.

Familiarity with a provider prior to telemedicine made the technology more acceptable to some patients:

I have been with [Doctor] for many years so I am comfortable with him on line. A new doctor maybe not so much.

Some providers were able to overcome reluctance through attention to patients’ discomfort:

It was a little weird but [Doctor] made it very comfortable.

Other patients referenced provider skills and proficiency using telemedicine technology:

I realize that there might be circumstances in which a virtual visit with a medical professional might not work, but in my case, it worked very well. [Doctor] made very good use of the Zoom platform.

Patients with positive experiences mentioned the provider competence and ability to effectively navigate telemedicine. Conversely, negative experiences with telemedicine resulted from providers being unfamiliar with the technology:

My suggestion is to get more organized before you do this again to an unknowing person. THE WHOLE meeting was very frustrating.

These comments suggest that patients’ perceptions of telemedicine are difficult to disentangle from perceptions of the providers and offices using them.

DO COMMENTS PREDICT PROMOTER RESPONSES? RESULTS OF MULTIPLE LOGISTIC REGRESSION

If the LTR question is a reliable measure of patient opinions about telemedicine, we would expect that the strongest predictor of LTR rating would be positive or negative comments about telemedicine and that comments about other topics would be less predictive.

Our analysis found that there were significant associations between content in comments and the patient LTR rating. Positive comments, whether about the provider, staff, or telemedicine, correlated with being a promoter, while negative comments correlated with lower likelihoods of being a promoter, in both the bivariate and multivariate logistic models. Fig. 1 plots odds ratios on a logarithmic scale to allow readers to visually compare strength of associations, both positive and negative.

Fig. 1.

Odds ratios for code applications to predict NPS score of 9 or 10, shown on log scale. Significance (α = 0.001) is denoted with ***.

As can be seen overall, negative comments were more strongly associated with promoter status than positive comments. The strongest predictor for being a promoter was whether a respondent made negative comments about the provider. Our bivariate logistic regression yielded an odds ratio of 0.036, meaning that if the patient gave a negative comment about a provider, they were nearly 28 times as likely to be a nonpromoter (p < 0.001). A multivariate analysis, which controlled for all six comment variables, yielded an identical odds ratio of 0.036.

The weakest association was between positive comments about telemedicine and being a promoter. The bivariate analysis odds ratio of 1.738 indicated that if the patient made a positive comment about telemedicine in general, they were about 1.7 times more likely to be a promoter (p < 0.001). The multivariate analysis produced an odds ratio of 1.975.

Discussion

Because the open comment question immediately follows the LTR question, patient comments should be consistent with LTR ratings, and yet we identified some incongruence. We found that NPS reflects experiences during medical encounters, but responses to the LTR question are nonspecific, biased toward providers and away from the telemedicine modality. Based on the qualitative comments, some patients understood the questions to be focused on telemedicine, while others may have understood it to be about the provider or medical office.

Thus, patients may be miscategorized as promoters while harboring negative attitudes about telemedicine, or detractors while feeling positive about it. Finally, LTR ratings were better predicted by comments about providers or staff than comments about telemedicine.

Previous research has also identified similar issues in terms of specificity and interpretability of NPS.^25,28,29,33 Providers appear to play as strong a role for LTR ratings as with other satisfaction measures.³⁹ We agree with previous assessments that found NPS to be helpful only in combination with deeper qualitative data.⁴⁰

These results suggest that NPS scores should be interpreted cautiously, understanding that LTR responses are based on multiple considerations that vary across patients, clinical settings, and points in time. Respondents will interpret the LTR prompt differently and may not consider telemedicine in their ratings. We echo previous research in observing that telemedicine as a modality is difficult to separate from the providers and staff who employ the technology.⁶ Patient comments about telemedicine were nearly always framed in relation to its skillful or inept deployment. Thus, poor patient experience with telemedicine may be improved with strategic and effective telemedicine usage coupled with robust training and support for health care professionals in optimizing telemedicine encounters.

LIMITATIONS

Many patients referenced the COVID-19 pandemic in their comments, noting that telemedicine was preferable for avoiding infection. While recognizing the impact of the pandemic context on the acceptability of telemedicine, we see no reason to believe it impacted the relationship between NPS responses and comments.

Elements of the survey design—for example, switching from a 4-point to a 10-point scale—made respondent errors on LTR rating more likely. Comments confirmed that some “detractors” were misidentified based on such errors (e.g., accidentally entering 1 rather than 10). Notably, the LTR prompt ambiguously uses the term “this service” to refer to the telemedicine modality, which may have contributed to respondent confusion.

As elsewhere observed,¹⁰ the vast majority of respondents on the survey were promoters. This pattern suggests a positivity bias,⁴¹ and creates a highly skewed distribution, violating the assumptions of many statistical models. This distribution limited possible analyses and should be considered a more general limitation of LTR response data.

RECOMMENDATIONS

These results imply several recommendations to improve the utility of patient surveys with regard to telemedicine. First, improvements to the design of the survey noted above would likely have produced more reliable results. Second, a separate measure of likelihood of recommending the provider could accompany this question to allow for contrast between opinions about the service modality and the provider. Finally, qualitative responses, while more time-consuming to analyze, offer more direct expression of patient experience than NPS and should be utilized when possible.

Conclusions

NPS, which is included on many patient experience surveys used by health systems across the world, has limitations for use as a measure of acceptability of telemedicine for patients. Patients have more than telemedicine in mind when responding to the LTR question, and ratings conflate attitudes about providers, office policies, and staff with the telemedicine modality. More direct measures are necessary for meaningful research on the acceptability and usability of telemedicine for patients.

Footnotes

Acknowledgments

We would like to thank Adam Ouellette for assistance with data collection and Lee Lucas for feedback throughout the analysis.

Authors’ Contributions

M.K. conducted the qualitative analysis, conceptualized the article, wrote significant portions of the methods and results, and edited the article. T.J. wrote most of the first draft of the article. J.D. conducted the logistic regression, wrote significant portions of the methods and results, created tables and figures, and edited the article. R.A. assisted with qualitative analysis and reviewed the article. E.A. and J.B. reviewed the article.

Data Availability

We cannot share our full dataset to protect the privacy and confidentiality of patients.

Disclosure Statement

The authors declare that there is no conflict of interest.

Funding Information

This work was completed without external funding.

Appendix 1

References

Patel

, Mehrotra

, Huskamp

, et al. Trends in outpatient care delivery and telemedicine during the COVID-19 pandemic in the US. JAMA Intern Med, 2021; 181(3):388–391; doi: 10.1001/jamainternmed.2020.5928

Johnson

, Taff

, Lee

, et al. The rapid increase in telemedicine visits during COVID-19. Patient Exp J, 2020; 7(2):72–79; doi: 10.35680/2372-0247.1475

Chuo

, Macy

, Lorch

. Strategies for evaluating telehealth. Pediatrics, 2020; 146(5):e20201781; doi: 10.1542/peds.2020-1781

J de

, Massip

, Vargiu

, et al. CONNECARE-Lleida Group. Implementing mobile health–enabled integrated care for complex chronic patients: Patients and professionals’ acceptability study. JMIR Mhealth Uhealth, 2020; 8(11):e22136; doi: 10.2196/22136

Jonker

, Plas

, de Bock

, et al. Remote home monitoring of older surgical cancer patients: Perspective on study implementation and feasibility. Ann Surg Oncol, 2021; 28(1):67–78; doi: 10.1245/s10434-020-08705-1

Ratanjee-Vanmali

, Swanepoel

, Laplante-Lévesque

. Patient uptake, experience, and satisfaction using web-based and face-to-face hearing health services: Process evaluation study. J Med Internet Res, 2020; 22(3):e15875; doi: 10.2196/15875

Daly

, Cracchiolo

, Holland

, et al. Digitally enabled transitional care management in oncology. JCO Oncol Pract, 2024; 20(5):657–665; doi: 10.1200/OP.23.00565

Singh

, Diamantidis

, Ramani

, et al. Patients’ and nephrologists’ evaluation of patient-facing smartphone apps for CKD. Clin J Am Soc Nephrol, 2019; 14(4):523–529; doi: 10.2215/CJN.10370818

Powell

, Stone

, Hollander

. Patient and health system experience with implementation of an enterprise-wide telehealth scheduled video visit program: Mixed-methods study. JMIR Med Inform, 2018; 6(1):e8479; doi: 10.2196/medinform.8479

10.

Garcia-Huidobro

, Rivera

, Chang

, et al. System-wide accelerated implementation of telemedicine in response to COVID-19: Mixed methods evaluation. J Med Internet Res, 2020; 22(10):e22146; doi: 10.2196/22146

11.

Martin

, Hinkley

, Stockman

, et al. Capitated telehealth coaching hospital readmission service in Australia: Pragmatic controlled evaluation. J Med Internet Res, 2020; 22(12):e18046; doi: 10.2196/18046

12.

Febrero

, Balaguer

, López-Picazo

, et al. [Evaluation of the telematic consultation in endocrine surgery during confinement due to COVID-19]. J Healthc Qual Res, 2022; 37(5):299–302; doi: 10.1016/j.jhqr.2022.02.009

13.

Madhusudhan

, Watts

, Lord

, et al. Employer-sponsored health centers provide access to integrated care via a hybrid of virtual and in-person visits. Telemed Rep, 2021; 2(1):247–257; doi: 10.1089/tmr.2021.0027

14.

Carneiro

, de Pinho

, Belo

, et al. Outcomes of telemedicine care during the COVID-19 pandemic: Experience from an intervention program designed for vulnerable population in Brazil. J Telemed Telecare, 2024; 30(4):715–721; doi: 10.1177/1357633X221089151

15.

Labinsky

, May

, Boy

, et al. Evaluation of a hybrid telehealth care pathway for patients with axial spondyloarthritis including self-sampling at home: Results of a longitudinal proof-of-concept mixed-methods study (TeleSpactive). Rheumatol Int, 2024; 44(6):1133–1142; doi: 10.1007/s00296-024-05581-w

16.

Gordon

, Wright

, Savona

, et al. Abstract TMP106: Use of net promoter score to quantify the participant experience with a telehealth lifestyle management and cardiovascular risk reduction coaching program. Stroke, 2024; 55(Suppl 1):ATMP106–ATMP106; doi: 10.1161/str.55.suppl_1.TMP106

17.

Summers

, Kuo

, Zaback

, et al. Pediatric ophthalmology provider and staff attitudes and patient satisfaction in telehealth implementation during COVID-19. Telemed J E Health, 2022; 28(5):675–681; doi: 10.1089/tmj.2021.0189

18.

Zachrison

, Yan

, White

, et al. Factors associated with the digital patient experience of virtual care across specialties. Telemed Rep, 2023; 4(1):227–235; doi: 10.1089/tmr.2023.0032

19.

Hajesmaeel-Gohari

, Khordastan

, Fatehi

, et al. The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health. BMC Med Inform Decis Mak, 2022; 22(1):22; doi: 10.1186/s12911-022-01764-2

20.

Grisaffe

. Questions about the ultimate question: Conceptual considerations in evaluating Reichheld’s net promoter score (NPS). J Consum Satisf Dissatisfaction Complain Behav, 2007; 20:36–53.

21.

Reichheld

. The one number you need to grow. Harv Bus Rev, 2003; 81(12):46–54, 124.

22.

Kristensen

, Eskildsen

. Is the NPS a trustworthy performance measure? TQM J, 2014; 26(2):202–214; doi: 10.1108/TQM-03-2011-0021

23.

Baehre

, O’Dwyer

, O’Malley

, et al. Customer mindset metrics: A systematic evaluation of the net promoter score (NPS) vs. alternative calculation methods. J Bus Res, 2022; 149:353–362; doi: 10.1016/j.jbusres.2022.04.048

24.

Baehre

, O’Dwyer

, O’Malley

, et al. The use of Net Promoter Score (NPS) to predict sales growth: Insights from an empirical investigation. J of the Acad Mark Sci, 2022; 50(1):67–84; doi: 10.1007/s11747-021-00790-2

25.

Adams

, Walpola

, Schembri

, et al. The ultimate question? Evaluating the use of Net Promoter Score in healthcare: A systematic review. Health Expect, 2022; 25(5):2328–2339; doi: 10.1111/hex.13577

26.

Garman

, Garcia

, Hargreaves

. Patient satisfaction as a predictor of return-to-provider behavior: Analysis and assessment of financial implications. Qual Manag Health Care, 2004; 13(1):75–80.

27.

Manacorda

, Erens

, Black

, et al. The friends and family test in general practice in England: A qualitative study of the views of staff and patients. Br J Gen Pract, 2017; 67(658):e370–e376; doi: 10.3399/bjgp17X690617

28.

Wilberforce

, Poll

, Langham

, et al. Measuring the patient experience in community mental health services for older people: A study of the net promoter score using the friends and family test in England. Int J Geriatr Psychiatry, 2019; 34(1):31–37; doi: 10.1002/gps.4978

29.

Hamilton

, Lane

, Gaston

, et al. Assessing treatment outcomes using a single question: The Net Promoter Score. Bone Joint J, 2014; 96-B(5):622–628; doi: 10.1302/0301-620x.96b5.32434

30.

Sizmur

, Graham

, Walsh

. Influence of patients’ age and sex and the mode of administration on results from the NHS Friends and Family Test of patient experience. J Health Serv Res Policy, 2015; 20(1):5–10; doi: 10.1177/1355819614536887

31.

Krol

, de Boer

, Delnoij

, et al. The net promoter score – An asset to patient experience surveys? Health Expect, 2015; 18(6):3099–3109; doi: 10.1111/hex.12297

32.

Bitencourt

, Crestani

, Peuckert

, et al. Net Promoter Score (NPS) as a tool to assess parental satisfaction in pediatric intensive care units. J Pediatr (Rio J), 2023; 99(3):296–301; doi: 10.1016/j.jped.2022.11.013

33.

Lewis

, Mehmet

. Does the NPS® reflect consumer sentiment? A qualitative examination of the NPS using a sentiment analysis approach. Int J Mark Res, 2020; 62(1):9–17; doi: 10.1177/1470785319863623

34.

Elo

, Kyngäs

. The qualitative content analysis process. J Adv Nurs, 2008; 62(1):107–115; doi: 10.1111/j.1365-2648.2007.04569.x

35.

Campbell

, Quincy

, Osserman

, et al. Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement. Sociol Methods Res, 2013; 42(3):294–320; doi: 10.1177/0049124113500475

36.

Krippendorff

. Reliability in content analysis. Human Comm Res, 2004; 30(3):411–433; doi: 10.1111/j.1468-2958.2004.tb00738.x

37.

Burla

, Knierim

, Barth

, et al. From text to codings: Intercoder reliability assessment in qualitative content analysis. Nurs Res, 2008; 57(2):113–117; doi: 10.1097/01.NNR.0000313482.33917.7d

38.

Garrison

, Cleveland-Innes

, Koole

, et al. Revisiting methodological issues in transcript analysis: Negotiated coding and reliability. Internet High Educ, 2006; 9(1):1–8; doi: 10.1016/j.iheduc.2005.11.001

39.

Martinez

, Nakonezny

, Batty

, et al. The dimension of the press ganey survey most important in evaluating patient satisfaction in the academic outpatient orthopedic surgery setting. Orthopedics, 2019; 42(4):198–204; doi: 10.3928/01477447-20190625-03

40.

Koladycz

, Fernandez

, Gray

, et al. The Net Promoter Score (NPS) for insight into client experiences in sexual and reproductive health clinics. Glob Health Sci Pract, 2018; 6(3):413–424; doi: 10.9745/GHSP-D-18-00068

41.

Posavac

, Brakus

, Jain

, et al. Selective assessment and positivity bias in environmental valuation. J Exp Psychol Appl, 2006; 12(1):43–49; doi: 10.1037/1076-898X.12.1.43