Abstract
We investigated how social diffusion increased client participation in an online rating system and, in turn, how this herding effect may affect the metrics of client feedback over the course of years. In a field study, we set up a transparent feedback system for university services: During the process of making service requests, clients were presented with short-term trends of client satisfaction with relevant service outcomes. Deploying this feedback system initially increased satisfaction moderately. Thereafter, mean satisfaction levels remained stable between 50% and 60%. Interestingly, at the individual client level, satisfaction increased significantly with experience despite the lack of any global trend across all users. These conflicting results can be explained at the social network level: If satisfied clients attracted new clients with more negative attitudes (a herding effect), then the net increase in service clients may dampen changes in global trends at the individual level. Three observations support this hypothesis: first, the number of service clients providing feedback increased monotonically over time. Second, spatial analysis of service requests showed a pattern of expansion from floor to floor. Finally, satisfaction increased over iterations only in clients who scored below average.
Introduction
Feedback has an important role in a wide range of social phenomena, from the evolution of cooperation (Nowak, 2006) to sustainable economic development (Platteau, 2000) and, conversely, poverty traps (Adato, Carter, & May, 2006). In social organizations, the issue of engendering a feedback loop is a central conundrum: Many institutions attempt to get the public more involved in improving their services by developing systems for eliciting feedback and evaluations from their clients (Miller, Resnick, & Zeckhauser, 2002; Pavett, 1983; Vigoda, 2000). However, creating a sustained feedback loop between users and public service providers is difficult (Jaeger & Thompson, 2003). The public sector adds an additional obstacle: The monopoly nature of service provision. In e-commerce sites, the provider has strong market incentives to address the concerns of online reviews lest business go elsewhere. In contrast, below we describe a system where both the provider and the consumer of the service are beholden to each other through monopolistic (and monopsonstic) dynamics (Holmes, Levine, & Schmitz, 2012). Little is known about how individual satisfaction changes over time and how social networks affect participation, and thus free ridership (Marwell & Ames, 1979)—especially in monopolistic service provision settings such as found in the public sector (Vedung, 1997).
Traditionally, feedback systems were deployed to facilitate learning at the institutional level, where the role of the clients is merely to provide information (Gerson, 1993). In recent years, however, several businesses (e.g., Google, Yelp, SeeClickFix) and nonprofits (e.g., FixMyStreet.org) have been experimenting with two-way communications systems (Lee, 2014), where citizens can follow up on their requests and post comments about service outcome. Consequently, public opinion about service quality is both mirrored and affected by highly distributed rating systems, whose statistical and dynamic features are poorly understood. For example, recent data suggest that the increased usage of government websites may negatively influences citizens’ satisfaction with public service provision despite (or perhaps because) the increase in accessibility and transparency (Porumbescu, 2015).
Indicators of service provision, such as mean satisfaction levels, might be affected by sampling biases at two levels (Kaufmann, Kraay, & Mastruzzi, 2004; Stipak, 1979): client decision to request a public service (participation level) and client decision to rate the service outcome (feedback level). Participation level may bias service provision indicators in a complex manner. For example, people who are less likely to participate might have a more negative attitude toward public services. A major factor in participation level is the herding effect (Hardin, 1968; Helbing, Farkas, & Vicsek, 2000), where social diffusion may gradually change public opinion and either facilitate or decrease participation levels, potentially leading to sampling biases over time. That is, early on, people with more positive attitudes toward the service might be more likely to try the service and to send positive feedback, while the herding effect may attract more reluctant customers who might send more critical evaluations.
A second source of potential bias is feedback level, as client tendency to send feedback may change with experience. In the classic “tragedy of the commons” scenario (Hardin, 1968), a common grazing area is overused and eventually destroyed by the farmers (free riders) who only care about their individual gains. Empirically, the principal factors that affect the level of free ridership are group size, the distribution of interest within the group, and the distribution of resources (Marwell & Ames, 1979). We do not know to what extent these factors might also apply in the case of monopolistic public service provision, potentially affecting client likelihood of submitting feedback (Hu, Zhang, & Pavlou, 2009; Moe & Trusov, 2011). Feedback level might also depend on the expectation of reward (the likelihood of a satisfactory service outcome), which may change with policies and/or with time.
Taken together, complexities in the temporal dynamics of client expectations versus experience, combined with the effect social diffusion, may impose long-term biases in indicators of service provision. Slow accumulation of biases can make it difficult to detect causality of observed changes, or alternatively, it might mask real changes in public service quality. This study is a preliminary attempt to disentangle some of those factors, by analyzing changes in participation level, feedback level, and satisfaction level over years.
The data used for this study were obtained from an administrative initiative at The City College of New York (CCNY) Science Division, which aimed at resolving long-lasting problems with maintenance services provided to research labs. The Science Division deployed a web application, which was developed to facilitate service request and to allow faculty and staff of research labs to request services directly. Current clients received feedback statistics about service outcome from recent users via a dashboard attached to service request forms (Figure 1A). In an attempt to reduce provider and client triage across service types (i.e., discouragement), the system presented client satisfaction as short-term trends rather than cumulative scores. This way, even a small change in mean client satisfaction could become immediately apparent in the dashboard display as a positive or a negative trend in client satisfaction (Figure 1, bar graphs). Our administration hoped that this would improve services without centralized sanctioning (Baldassarri & Grossman, 2011), by synchronizing social learning across workers and clients of monopolistic university services.
Transparent reporting system for services requests: (
We analyzed data collected via this system over 4 years to evaluate how slow changes in participation and feedback level may affect measures of client satisfaction level over time. We then performed spatial analysis to examine if changes in participation level may display any spatial structure, which could indicate a herding effect (Benkler, 2006; Helbing et al., 2000). Finally, at the individual client level, we analyzed changes with satisfaction level over iterations (experience), to assess if social learning (across clients and providers of services through the feedback system) might have influenced satisfaction with service outcome. Our results suggest that herding effect might have imposed sampling biases, resulting in opposing effects on individual and population-level performance indicators of service provision. We propose simple measures for reducing such biases, and potentially improve the reliability of similar information systems.
In sum, the problem that our field study attempted to address is how to improve public services by enabling social learning, online, across and among clients and providers of services. The approach was to set a voluntary service rating system and present client feedback statistics as short-term trends (as opposed to cumulative star rating). We tested for slow progressive changes in client satisfaction with service outcome, focusing on interactions between changes in participation levels (herding effect) and our estimates of changes in client satisfaction over years.
Materials and Methods
Analysis of data obtained by this administrative initiative has been approved by the CCNY Institutional Review Board (IRB). All institutional data were deidentified and then analyzed according to the CCNY IRB regulations.
System Design
A transparent reporting system was developed between 2005-2006 by faculty volunteers to address persistent problems in resolving maintenance issues in about 100 research laboratories and several core facilities in the CCNY Science Division. The Web Application includes forms designed for requesting services for problems of various categories, including electrical, plumbing, HVAC (heating and cooling), carpentry, building integrity, pest control, custodial, restrooms, and so on. The system replaced a facilitator, whose job was to receive reports from building occupants and submit work orders when necessary. Instead, it enabled direct communication with service providers via online forms accessible via any web browser without restrictions to all the occupants of the division building. Each form submission triggered an automated email to the appropriate service provider, but we had no access to their internal operations, hence it is an “external,” third-party system.
Client Access Control
The site was available online without restrictions (no login) from any device connected (wired or wirelessly) via the college division network, which was available via WIFI to all occupants of the Science building.
Requesting User Evaluations (Feedback)
The system relied entirely on requesting user evaluations, and implemented externally. Feedback was requested blindly from each user (Figure 1B) via email after a week delay. This approach allowed, at the cost of uncertainty, to arbitrarily attach our reporting system to any relevant university service. The feedback request email included a link to a simple form where user scored the outcome as either satisfied, partially satisfied, or unsatisfied/nothing happened. The feedback form did not present clients with any information (no dashboard).
Dashboard Presentation of Satisfaction With Service Outcome
Trends in mean client satisfaction were presented in the dashboard for each service type in monthly bins (Figure 1A). %satisfaction was calculated as the proportion of reports that received a “satisfied” score, presenting six monthly bins, with the first bin representing accumulating data from the current month.
Workflow
During the baseline period (first 9 months of 2006), the Web application for university services was deployed without presenting the dashboard. Service requests were automatically emailed to the appropriate service providers. Client feedback about service outcome was requested only sparsely (n = 33) to obtain baseline information about client satisfaction with service outcome. After 9 months, the feedback system was activated without otherwise changing the user interface. The system requested feedback from each client 1 week after each service request. A dashboard was attached to the service request forms with bar graphs presenting mean values and trends of monthly satisfaction with service outcome (Figure 1). Those graphs were visible to all users as well as to service management and workers. Service events that received “unsatisfied” score were automatically emailed to the service providers without any further follow-up. This mechanism remained intact without significant modifications for 4 years.
Data Analysis and Statistics
Users scored the service outcome as either satisfied, partially satisfied, or unsatisfied. We pooled the “partially satisfied” and “unsatisfied” scores into a single category to obtain a binary measure for each period, namely, the percentage of fully satisfied clients. Similarly, in the figures we present %satisfaction as the proportion of reports (or clients) that scored service outcome as fully satisfied. Data were analyzed using Matlab 8. The Matlab Statistics package was used to calculate Pearson correlation coefficients and p values. Spatial analysis was performed by computing a matrix with the number of unique users per quarter per floor. We then normalized each column to represent proportions, and smoothed the matrix using a 2 × 2 Hun filter. For analysis of the effect of repeated requests at the individual level, we considered the first 14 reports (submitted per individual), for which we had sufficient sample size of 53 users. We calculated Pearson correlation coefficients and p values over those first 14 requests per subject.
Programming
The web application was programmed using HTML, PHP, and MySQL. Feedback was managed via custom C++ application using Embarcadero RAD Studio. This way, dashboard images of graphs were automatically updated when the web page was open or refreshed.
Results
Presenting a dashboard with short-term trends of satisfaction with service outcome (Figure 1A) resulted in a moderate increase in the quarterly satisfaction rate with service outcomes. Satisfaction rate increased from 33% during the 6 months baseline period (n = 33 subjects) to 56.5 ± 1.7% (Figure 2A, means and SEM across quarters hereafter, Pearson’s χ2 = 4.1, p = .04). After the initial increase, client satisfaction levels remained stable with no apparent trend over time (Figure 2A, R2 = .02, ns). However, analysis at the individual client level (pooled across individuals) showed that satisfaction increased linearly with the chronological order of service requests (Figure 2B, Pearson correlation: R2 = .63, p = .0007). Limiting the analysis to clients who submitted at least 14 reports (our upper bound for analysis, see methods), to account for attrition (n = 53 subjects) we still identify a significant positive trend (Pearson correlation: R2 = .48, p = .006), indicating increased satisfaction over iterations. Client feedback levels remain stable at 43 ± 1.3% throughout the study, with no apparent trend (Figure 2C), excluding a scenario of changes in mean satisfaction through attrition.
Trends in client satisfaction with service outcome: (
Although the number of Science Division members was approximately the same during experiential period, we observed a persistent increase in the number of unique service clients per quarter, at a rate of about 13% per year (Figure 3A, Pearson correlation: R2 = .43, p = .006). Examining the role of social diffusion in facilitating participation through a spatial analysis, we observed a slow spatial diffusion, from floor to floor of the Science building over the course of the study (Figure 3B), suggesting that offline local communication might have played a role in the increased participation (Rogers, 1962).
Analysis of participation and satisfaction levels:
Finally, we analyzed satisfaction over time at the individual client level to test for heterogeneity across client pools with different attitude toward the university services. Dividing clients into those whose mean score was above the pooled average (positive attitude) versus those below the pooled average (negative attitude), we found a strong heterogeneity of responses: the positive trend was driven by clients who scored below average (Figure 3C) with a remarkable increase from about 20% to 60% satisfaction within that group. In contrast, no significant trends were observed in clients who scored above average (Figure 3D).
Discussion
Our results suggest that the effects of launching transparent feedback systems on monopolistic service provisions might be complex, and simple cumulative performance estimates have limited bearing on capturing it. In our case, even though we did not observe any global trends in client satisfaction over 4 years (Figure 2A), a detailed analysis detected an increased satisfaction with experience at the individual client level (Figure 2B). We suspect that this effect was masked by a secondary herding effect (Benkler, 2006; Helbing et al., 2000), where satisfied clients attracted more reluctant ones, who at least initially submitted lower satisfaction scores, dampening the global trend. Note that even if the expansion in client pool were a random effect, the positive correlation between iterations (experience) and satisfaction alone would dampen the global trend. For example, as service client pool increased at a rate of about 13% per year (Figure 3A), and as satisfaction rate were about 15% lower in new clients (Figure 2B), if participation was unbiased, then the flow of new customers should have reduced the mean annual satisfaction rate by about 2%. However, our data suggest that participation was biased: Spatial analysis of service requests showed a pattern of expansion from floor to floor (Figure 3B), suggesting that clients with positive attitude herded clients with a more negative attitude toward university services. In clients with negative attitude (Figure 3C-D), satisfaction increased much more strongly with experience, at about 30% to 40%. Therefore, herding effect could have reduced satisfaction by 4% to 5% given the flow rate. Note that the overall increase in individual clients satisfaction rate over 4 years was about 20% (about 5% per year), comparable to the putative dampening effect we propose. Overall, our results suggest that slow but persistent improvement in satisfaction for existing users and social contagion in utilization of the information system might have brought clients with less positive attitude toward the services into the information ecology, hence masking the increase in client satisfaction.
We found it interesting that client feedback levels remained high (between 40% and 50%) over the entire study (Figure 2C). Feedback was requested only once, a week after each report, and no additional measures were taken to encourage clients to send feedback. It is difficult to compare those rates to existing data because institutions rarely report the percentage of clients who responded to automated feedback requests. However, typically, feedback levels are lower than 20% and nonresponse bias is significant (Groves, 2006; Lambert & Harrington, 1990). Optimization of survey methods can increase client response rate to some extent (Kaplowitz, Hadlock, & Levine, 2004). Beyond that, in a repeated game (Xiao, Zhang, Shi, & Gao, 2012) client motivation to share information may depend on their expectation of reward, for example, in their perception of feedback usefulness (Racherla & Friske, 2012), or by deviations from prior expectation of reward (Moe & Schweidel, 2011). Perhaps presenting information about client satisfaction at the time of service request, had affected client expectation of reward and contributed to stability in feedback rate. Testing this hypothesis would require randomized control studies, which were not feasible in this administrative study.
Our analysis has many limitations. The most serious one is that without randomized controls, we could not test if the presentation of short-term trends on the dashboard was beneficial or not. However, we can certainly conclude that the trends presented to the clients in real time did not fully capture the long-term improvement in client satisfaction, which we observed only a posteriori at the individual client level. In retrospect, presenting trends of satisfaction as a function of client experience with the service (pooled across individuals as in Figure 2B) rather than monthly trends could have reduce biases, and could have provided a better chance to synchronized learning across clients and providers of services.
This study was conducted at a particularly interesting time in the history of communication technologies, just when the usage of online communication and mobile devices became widespread. It is therefore possible that the findings we present here are historically contingent. That is, there could be a novelty effect that resulted from the transition from a phone/paper-based request system to an online interface that would have faded had we continued data collection for another 5 or more years. However, as we did collect data for a 5-year time period that far exceeds that of most experimental timelines, we do not believe that our observed results are due to novelty or the particulars of the online interface but rather to the improved steady state relationship between service quality and user information. It is also possible that if our approach were so widely adopted such that users were saturated with such coupled, socially informed interfaces, our feedback dashboard would fail to elicit the level (and quality) of sustained user (or service provider) response. However, if presenting the short-term trends in client satisfaction is indeed the cause of the improved validity of information for both users and service providers, then there is little reason to expect the surrounding ecology of information to strongly bias the underlying relationships. That is, better quality (and more) information leading to better responsiveness in one service could be sustained without affecting usage or information feedback dynamics across services, especially in the case of monopolistic public services. This assumption rests on the notion that the ease of use and perceived usefulness of the interface remains constant even in a changing information technology landscape (Davis, Bagozzi, & Warshaw, 1989).
The ecological validity of this field study allows us to cautiously propose some general implications. Other than presenting trends, the manner in which we published voluntary client scores on service request forms is fairly similar to distributed platforms for client rating such as Yelp and Amazon, who impose external rating on many arbitrary services, as we did. It is well established that the dismal proportion of service clients who choose to post feedback in such platforms is unlikely to be representative (Gao, Greenwood, Agarwal, & Jeffrey, 2015; Zervas, Proserpio, & Byers, 2015). Our study adds to this by suggesting that herding effect may impose dynamic biases on indicators of service provision. Such effects should be taken into account when attempting to increase client base or feedback levels, which might slowly bias the sampling of clients with different attitude toward the service. Long-term dynamical biases in sampling should particularly concern public services, where incremental improvements over years can be guided by client feedback.
Despite the lack of persistent positive trends, deploying a transparent feedback system for university service resulted in positive outcomes: an improvement in client satisfaction with experience, an increase in participation rate, and persistently high feedback levels. Therefore, under certain conditions, designing a communication system that presents trends of client satisfaction with service outcomes may result in improved client satisfaction with monopolistic public service provision through learning. Future work studying how the design of communication ecosystems can affect the evolution of client engagement and satisfaction could become useful if the two effects we describe here can be separated. Achieving high public participation in reviewing service outcome can potentially allow distributed management of public services. Such systems can be tested on large scales. We hope that this preliminary study would facilitate efforts for embedding governance within Internet-based social dynamics. Such systems could make it possible to create communication ecologies that enable real-time (and bottom-up) social learning.
Footnotes
Acknowledgements
We thank CCNY Science Division deans M. Gunner and R. Stark, CCNY Vice Presidents R. Santos, CCNY Physical Plant Services managers R. Slowski and G. Miller, and K. Woods for their continuous support and cooperation. We thank L. Parra and D. Baldassarri for commenting on the manuscript.
Authors’ Note
The data reported in the paper are archived at the CCNY Science Division server.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: Financial support for this study was provided by the John D. and Catherine T. MacArthur Foundation as part of its support of the Research Network on Connected Learning to D. Conley.