Social media data used in the measurement of public services effectiveness: Empirical evidence from Twitter in higher education institutions

Abstract

Measuring public service effectiveness has become a central issue for public authorities worldwide, often driven by governmental pressures to ensure value for money. In this context, social media data represent a potential powerful tool in the hands of public authorities to support the evaluation of public service performance. By relying on an action research project in the higher education field, this study explores how social media data can contribute to measure service effectiveness by focusing specifically on Twitter in the higher education field. The final aim of the paper is to develop a set of measures, derived from Twitter data, to quantify the effectiveness of higher education services. This investigation supports a broader discussion about the extent to which social media data can contribute to performance measurement in the public sector.

Keywords

Accountability information communication technology (ICT)public performance management service delivery social media

Introduction

Measuring performance in the public sector was a central issue of the new public management (NPM) reforms dating back to 1990s (Hood, 1991, 1995) and is still a challenge for governments and academics (e.g. Matthews, 2016; Stejskal and Hájek, 2015). Among the various performance measures that can be analysed, effectiveness plays a central role, since it indicates whether a given public service meets user expectations (Boland and Fowler, 2000; Boyne, 2003). From the content of public sector reforms and number of academic papers debating these matters, it is clear that performance measurement and service effectiveness are of topical interest (e.g. Agasisti and Bonomi, 2014; Arena et al., 2010; Cutler, 2011; Johnsen, 2005; Matthews, 2016; Pettersen, 2015; Talbot, 1999). Although measuring performance is not new in the literature covering the public sector, its importance has revived with the potential offered by big data (Gandomi and Haider, 2015; George et al., 2014; Lavertu, 2016). The key features of big data are high volumes of data, the high speed with which data are generated and the great variety of data formats, including, alongside traditional excel spreadsheets, data from sensors, mobile phones and social media (Gandomi and Haider, 2015; McAfee and Brynjolfsson, 2012).

Social media data are a particular category of big data and are defined as a set of online tools centred on social interactions that generate a huge amount of user data (Kaplan and Haenlein, 2010). Among the main characteristics of social media are a more democratic access to communication channels, the generation of data in real time and the high level of interactivity, creating the conditions for establishing communities of users (Arnaboldi and Coget, 2016; Gao et al., 2011; Kaplan and Haenlein, 2010). Public authorities worldwide are endorsing these social tools, largely because they are free. Although the literature on how public authorities are implementing and using social media is expanding (e.g. Agostino and Arnaboldi, 2016; Kim et al., 2014; Lev-On and Steinfeld, 2015; Waters and Williams, 2011), there are still few contributions on the potential of social media data in terms of measuring how effective a given service is. The reason for this is because of the overriding interest in technical issues about how data are collected and analysed (e.g. Agostino and Sidorova, 2016; Archak et al., 2011; Bhardwaj et al., 2014; Netzer et al., 2012; Nguyen et al., 2014; Thelwall et al., 2010), rather than on how this data can be used for management purposes to measure performance. The importance of extracting value from these new types of data has been widely acknowledged: ‘categorizing big data, assessing its quality, and identifying its impact is radically new in social sciences’ (George et al., 2014: 324)

Starting from this premise, the aim of this paper is to gain an understanding of how social media data can contribute towards measuring the effectiveness of public sector services, focusing specifically on Twitter in the field of higher education. The ultimate objective of the paper is to develop a set of measures, based on data obtained from Twitter, to quantify the effectiveness of services provided within the domain of higher education. This investigation can support a broader discussion concerning the extent to which social media data can contribute towards measuring performance in the public sector.

This factor is significant at two different levels. First, while there is widespread recognition about the importance of exploiting social media data and finding value in the conversations taking place on social media (Criado et al., 2013; Gandomi and Haider, 2015; George et al., 2014), most extant literature is concerned with how well public authorities use these tools, rather than on how social media data can help to measure performance in the public sector. Second, studies on social media in the field of higher education are mainly interested in looking at the benefits of these tools, and how widely used they are for the purposes of advancing teaching and learning, and for improving library services, recruitment processes, marketing and student enrolment (e.g. Bélanger et al., 2013; Constantinides and Stagno, 2011; Manca and Ranieri, 2016; Palmer, 2013). The possibility of using social media to gain information about the effectiveness of the services on offer, bypassing the need for student and staff surveys, is in general much neglected.

From a methodological point of view, we set up an action research project, with the involvement of 21 Italian universities. A literature review was first carried out, together with interviews and other forms of interaction with university administrative staff and governance officials, which provide the basis for establishing a set of Twitter-based measures for quantifying service effectiveness. The framework was then applied to the universities taking part in the project, a process that involved downloading their Twitter dataset, computing the proposed measures and actively discussing the potential and limitations with the participants.

From the results, we can propose a framework composed of four different measures (i.e. discriminant ratio, non-official account ratio, temporal ratio and polarity ratio), whereby Twitter data are used to quantify the effectiveness of university services. The benefits and limitations of this framework are also discussed in connection with its application to Italian universities.

The rest of the paper is articulated as follows. The next section contains a review of extant literature concerned with the role of social media in supporting performance measurement. The action research methodology is then described, followed by the results of the project. The main contributions of this paper are discussed in the concluding section of the article.

Social media and performance measurement in the public sector: Linking two streams of analysis

This section contains a review of extant studies on social media within the public sector, with particular attention to contributions relating to the interconnection between measuring performance in the public sector and measuring social media. This section is divided into two parts, the former focusing on the measurement of performance in the public sector, and the latter on the type of measures that can be derived from social media.

Performance measurement in the public sector

Measuring performance has become of primary concern to public sector scholars in several European, American and Eastern countries (Blackman et al., 2006; Matthews, 2016; Moynihan and Pandey, 2010; Van Helden et al., 2008) because of the requirement to ensure value for money. NPM theory and the more recent new public governance (Hood, 1991; Osborne, 2006) have prompted public administrations at all governmental levels to use their resources more effectively and provide services of value to the community. This, in turn, has led to performance measurement and performance management systems becoming adopted widely, and they have become recurrent topics of investigation in public sector literature (e.g. Arena et al., 2010; Cutler, 2011; Johnsen, 2005; Matthews, 2016; Pettersen, 2015; Talbot, 1999). The input–output model, with its focus on measures of efficiency, effectiveness and economy, has become a common reference framework for classifying and analysing performance measures for public services (Jackson and Palmer, 1992; Pollanen, 2005). Efficiency measures compare the output of the service provided with the resources used to produce the service. Effectiveness measures evaluate the output of the service provided, and economy measures are related to the inputs used. Among these measures, effectiveness has attracted most academic attention, giving rise to a lively debate on how public service effectiveness can be measured and managed (e.g. Arena et al., 2010; Boyne, 2003; Dužević and Čeh Časni, 2015; Evans, 2013; Forbes and Lynn, 2005; Hodgson et al., 2007; Lægreid et al., 2006; Mansour et al., 2015; Osborne et al., 2015). One of the more recent contributions by Osborne et al. (2015), in particular, highlights that, for public bodies, it is of primary concern to rely on measures of effectiveness (rather than on measures of efficiency only), since they can be used to create value for users and ensure long-term sustainability. Despite the importance of measuring service effectiveness, there is still a level of criticism with regards to the complexity of quantifying this measure: ‘this is because a public service is without material substance and is therefore difficult to convert into units of performance’ (Stejskal and Hájek, 2015: 145). Given this limitation, most studies that try to measure service effectiveness do so through surveys sent to service users, and so evaluate the users’ perception about the public service being delivered (e.g. Arena et al., 2010; Brochado, 2009; Sarrico and Rosa, 2014).

Several contributions focus specifically on measuring performance in the field of higher education; these recognise the importance of measuring service effectiveness with reference both to the primary activity of teaching and research (e.g. Agasisti and Bonomi, 2014; Evans, 2013; Pettersen, 2015) and to support services (e.g. Arena et al., 2010; Craig, 2014; Dužević and Čeh Časni, 2015). In current studies, ‘traditional’ organisational data is used as the main source of information for performance measurement purposes, neglecting by and large the potential of social media data. On the contrary, social media have become an emergent field of investigation in terms of how this new technology can support teaching and learning, library services, recruitment, marketing and the students’ enrolment process (e.g. Bélanger et al., 2013; Constantinides and Stagno, 2011; Manca and Ranieri, 2016; Palmer, 2013).

In this paper, we tackle social media from a different perspective, by considering the contribution of social media data towards the measurement of performance, with particular reference to the effectiveness of higher education services.

Social media measurement in public administration

Social media data are a particular category of big data and are defined as a set of online tools centred on social interaction that generate a huge amount of user data (Kaplan and Haenlein, 2010). They comprise a variety of platforms, which range from social networks such as Facebook and Twitter, to video and photo sharing platforms such as YouTube or Pinterest, and wikis like Wikipedia or blogs. Despite the wide variety of tools known under the umbrella term of social media, they all share the common feature of users actively generating content in real time (Chun and Luna-Reyes, 2012).

The diffusion of social media is a worldwide phenomenon and has prompted several investigations into how these tools can be exploited and how value can be created from these new types of data (Criado et al., 2013; Gandomi and Haider, 2015; George et al., 2014). When specifically looking at social media in the public sector, more and more literature is emerging. These works are mainly explorative and based on case studies, providing preliminary evidence on how local authorities and other public sector institutions are exploiting the new social tools (e.g. Bertot et al., 2012; Bonsón et al., 2012; Grimmelikhuijsen and Meijer, 2015; Mergel and Bretschneider, 2013; Waters and Williams, 2011). Their findings are basically twofold. First, public authorities tend to use social media as traditional communication tools, rather than exploiting their interactive features (Bonsón et al., 2012; Campbell et al., 2014), and, conversely, the general public exploits social media with intentions connected to political mobilisation and/or legitimation, calling public authorities to account for their actions (Bekkers et al., 2011; Grimmelikhuijsen and Meijer, 2015). Second, it is possible to identify several phases and steps within the process of adopting and implementing social media (Mergel and Bretschneider, 2013), as well as policies concerning the use of social media at local administration level (Bertot et al., 2012).

Alongside this increasing use of social media by the public sector, there is an emerging stream of literature concerned with measuring social media. In this respect, several studies propose a set of measures that can evaluate the public sector’s capacity in using social media (e.g. Agostino and Arnaboldi, 2016; Bonsón et al., 2012; Campbell et al., 2014; Grimmelikhuijsen and Meijer, 2015; Lev-On and Steinfeld, 2015). For example, in the study by Bonsón et al. (2012), the authors developed a sophistication index to evaluate how extensively public authorities use social media. In a similar vein, Agostino and Arnaboldi (2016) developed a popularity and commitment indicator for evaluating the ability of municipalities to engage with the general public through social media. These studies are concerned with examining how well public bodies use social media, providing several specific measures for the purpose. While, however, these measures show how effectively social media are used, they do not evaluate the performance of public services.

The aim of our paper is to address this gap by connecting the two streams of literature relating to social media and to performance measurement in the public sector. More specifically, we aim to understand how social media data can contribute towards measuring the effectiveness of public services, by analysing Twitter posts within the field of higher education.

We decided to focus on service effectiveness because of the inherent feature of social media, that of being a tool in the hands of users. It is, therefore, highly probable that the users’ perceptions about a given service can be gleaned from social media data, potentially helping to measure that service’s effectiveness. Three distinctive features of social media come into play when measuring service effectiveness: democratic access to communication channels, real-time generation of data and interactivity. First, social media ensure that users have democratic access to communication channels, thereby rebalancing the power between those who broadcast and those who receive information (Arnaboldi and Coget, 2016). This means that users have a voice and can comment and offer their opinion, and even, in some exacerbated outbursts, encourage mass mobilisation on political topics (Bekkers et al., 2011).

Second, social media data are generated in real time, so users can give their comments and opinions 24/7. This has increased ‘the speed with which information about events is created, circulated and commented upon’ (Arnaboldi and Coget, 2016: 2). Due to this increasing rapidity, social media managers must monitor social platforms continuously, not only during office hours. On the other hand, social media have shown themselves to be priceless in the management of disasters, such as earthquakes and hurricanes (Freberg et al., 2013; Gao et al., 2011). Third, social media, being based on Web 2.0 technology, ensure interactivity, allowing public bodies to dialogue with and involve a participating public (Bonsón et al., 2012). Because of these three features, social media are particularly suited to collecting users’ opinions, a fact favouring the measurement of public service effectiveness.

Finally, our specific focus on the field of higher education is driven by the spreading of social media among young people, above all, and the general recognition that digital technologies are an ‘expected part of the routines of academic study and wider campus life’ (Henderson et al., 2015: 308). For this reason, we considered the higher education field as a representative public sector field to be investigated in connection with exploiting social media data for the purposes of performance measurement.

Methodology

In exploring how social media data can contribute towards measuring service effectiveness, we based our work on action research. The distinctive feature of this methodological approach is that action research ‘seeks to bring together action and reflection, theory and practice, in participation with others, in the pursuit of practical solutions to issues of pressing concern to people, and more generally the flourishing of individual persons and their communities’ (Reason and Bradbury, 2001: 1). It follows that action research can be used to address a practitioner problem that is also of interest to academics. The practitioner problem in our specific setting is the need felt by universities about whether data obtained from Twitter have the potential to evaluate the performance of their services. This problem is linked to the academic urgency of ‘categorizing big data, assessing its quality, and identifying its impact’ (George et al., 2014: 324).

The client system is the social domain where there is the need to solve a certain problem through action research (Baskerville and Wood-Harper, 1996). Our client system is represented by a group of 21 Italian universities belonging to a voluntary network set-up to benchmark the performance of their services and identify best practice, and ultimately improve their services (Table 1). The universities in the group differ in terms of size and year in which they started to use Twitter. Three universities (UNIV 12, UNIV 15 and UNIV 20) joined the project without having a Twitter account, because they were interested in having a benchmark of the current situation in the perspective of introducing social media in the future. Two of the three joined Twitter before the end of the project.

Table 1.

List of participant universities

	No. students	Twitter in use	Year when started to use Twitter
UNIV 1	19,460	Yes	2010
UNIV 2	4327	Yes	2009
UNIV 3	10,984	Yes	2014
UNIV 4	40,453	Yes	2011
UNIV 5	31,189	Yes	2011
UNIV 6	546	Yes	2012
UNIV 7	657	Yes	2012
UNIV 8	10,630	Yes	2013
UNIV 9	80,924	Yes	2010
UNIV 10	14,619	Yes	2011
UNIV 11	16,247	No	N/A
UNIV 12	60,826	Yes	2010
UNIV 13	32,340	Yes	2010
UNIV 14	8917	No	N/A
UNIV 15	25,738	Yes	2013
UNIV 16	22,200	Yes	2011
UNIV 17	17,867	Yes	2010
UNIV 18	13,740	Yes	2012
UNIV 19	15,478	No	2013
UNIV 20	15,672	Yes	2009
UNIV 21	22,497	Yes	2009

We, as a research team, were contacted with reference to a specific part of a larger project, with the purpose of examining how the group of universities uses social media data, benchmarking current practice and identifying the potential of the new tools. The project lasted one year, from October 2014 to October 2015 and involved four main phases (Table 2).

Table 2.

Phases of analysis

Phase	Description	Period	Data sources	Volume
First phase	Definition of social media measures and metrics	October 2014–February 2015	Literature review, active discussions with participants, interviews with the key people involved in social media, document analysis	Thirty-two papers reviewed, one plenary meeting, Skype and phone calls, emails, a total of 52 interviews with general directors, and heads of communication and social media units, 21 documents analysed
Second phase	Definition of the methodology of analysis	February 2015–September 2015	Active discussions with participants	One plenary meeting, Skype and phone calls, emails
Third phase	Collection and analysis of social media data	March 2015–June 2015	Twitter database	75,653 tweets
Fourth phase	Discussion of results	June 2015–October 2015	Active discussions with participants	One plenary meeting, face-to-face discussions with participants, phone discussions with participants

In the first phase, the objective was to develop a set of measures, derived from social media, to quantify how social media data contribute towards measuring services. The process started with a literature review, where we specifically looked for papers discussing social media measures or the type of data that can be obtained from social media. This research was carried out in Scopus, looking at the areas of business, management, public sector, public policy and social science, and generated 32 relevant papers. In addition, 52 interviews were conducted to gain information about the start of the art in social media within the participating universities and the informants’ information needs concerning social media. In each university, we interviewed the general director, the person heading the office managing social media (usually the communication office) and staff with the specific responsibility of managing social media. Lastly, we analysed the documents concerned with how each university measures social media, and the findings were shared through a plenary meeting, skype calls and emails. This first phase led to a preliminary model of the social media measures that can help in evaluating the effectiveness of higher education services.

The analysis methodology was clarified in the second phase, with the measures derived being framed within the context of the 21 universities. The boundaries of our analysis were set at the end of this phase, and were to concentrate on Twitter, to collect Twitter data over a period of five months (from February to June 2015) and to gather tweets posted by both the university and by third parties. With respect to this last point, each university provided us with its Twitter dataset, and we downloaded Twitter data from third parties by running the universities’ names as keywords through the public Twitter API. We chose Twitter because of two main reasons. First, Twitter is the only social media, among the many available, that allows users to set queries and download tweets for free; in Facebook, for example, due to privacy issues, free text downloads are not allowed unless you are a ‘friend’ of the relative account holder. Second, in this phase of the project, we investigated the distribution of social media within all universities in Italy and Twitter was the clear winner. Moreover, some other studies underlined that Twitter is among the three most commonly used social networks in Italy (Cosenza, 2014).

The third operative phase involved collecting Twitter data and cleaning and analysing the data. In this process, the Twitter dataset was extracted using pre-defined keywords and cleaned, starting with a manual tagging. The downloaded tweets (a total of 75,653) were grouped into discussion topics and, for each of the topics, a sentiment analysis was carried out to assign a polarity to each tweet. Each operation was followed by the data being cleaned manually and the initial data were further revised to ensure the validity of the analysis. In this phase, we used different statistic software. We used R software to download and manage Twitter data, CLUTO to build clusters around the discussion topics, and TreeTagger for the sentiment analysis part.

The final phase involved discussing the results with the 21 universities in the client system to share the results from the operative phase and validate the set of Twitter-derived measures. The benefits and potential problems associated with the proposed set of measures were highlighted during this phase.

Results

The results from the literature review and the action research project are presented in this section, making the distinction between the set of social media measures developed for the purpose of measuring public service performance, and how these measures are applied to the context of Italian universities to identify the relative benefits and limitations.

Framing a set of social media measures for public service performance measurement

The extant studies on social media data, together with active discussions with the participant universities, provided some insights into the types of data that can be derived from social media and used to measure the effectiveness of the public services provided. Current literature suggests four main areas in which social media data can contribute towards measuring public service performance: what service is discussed, who is talking about the service, when the service is evaluated and how the service is perceived (Table 3).

Table 3.

Proposed performance measures

Indicator	Definition	Metric
Discriminant ratio	Measures the importance of a topic compared to the overall discussions on Twitter	Total number of tweets where a topic is discussed/total number of tweets
Non-official account ratio	Measures the relative incidence of discussions made by accounts other than the university	Tweets posted by accounts other than the university/total number of tweets
Temporal distribution	Measures the distribution of tweets over a certain period	Number of daily tweets
Polarity ratio	Measures how users perceive the service	Number of the total scores of each word in a tweet/total number of words contained in a tweet

The first area relates to what service is being discussed. Studies on the usage of social media data have shown that it is possible to analyse word frequency in social media conversations and find topics of discussion on social media channels (Herring, 2010). On this point, a set of content analysis techniques have been proposed for downloading and identifying the discussion topics of users on social media (e.g. Agostino and Sidorova, 2016; Bhardwaj et al., 2014; Netzer et al., 2012). The distinctive feature of these techniques is that, rather than being a top-down driven process, the topics of discussion are derived bottom up from the most recurring words. It also means that the topics of discussion are not defined a priori. This picture led us to propose a discriminant ratio and so identify the type of service object of the social media conversation; therefore, answering the question about what services are users talking about. This measure quantifies in percentage terms the importance, on Twitter, of a given topic relating to the university, compared to the overall set of discussions, meaning that it is possible to identify which university services are being talked about on Twitter and their relative importance.

This measure is elaborated by dividing the total number of tweets where a certain topic is mentioned by the total number of tweets relating to the university. The discussion topics are identified in an intermediate step, involving statistical techniques based on word count: the words that recur most are ranked into descriptive item sets and then grouped together into clusters on the basis of a similarity index, where similar words are joined and those loosely linked separated.

The second area concerns who is talking about the service, which is connected with the feature of dialogue being intrinsic to social media. Users are at the centre of social media (Waters et al., 2009), and the fact that they are given a voice is widely acknowledged in the public sector, with several authors underlying that public authorities use social media to involve the public (i.e. social media users) in decision-making processes (Lev-On and Steinfeld, 2015). At the same time, the manager of the social media page, here the public body itself, also provides material on social media. Since dialogue is a major feature of social media (Chun and Luna-Reyes, 2012; Waters et al., 2009), this implies that the conversations on social media are the output of both the public authority that is posting and providing content, and also of the users who comment on the public body’s posts and offer their own comments. Since it is, therefore, also necessary to distinguish, among those talking about the public service, between the public authority itself and the service users, we have developed a non-official account ratio for this purpose.

This non-official account ratio can be calculated when the available social media dataset consists of tweets published by the public sector’s official account as well as those published by members of the general public talking about the public body. By distinguishing between these two categories of tweets, this non-official account ratio provides an insight into who is most interested in a specific topic; for example, the public sector mainly publishes posts on certain subjects, while it was found that users comment on other services. This indicator is elaborated by taking the number of tweets in which the public body is mentioned published by accounts other than the public body, and dividing it by the total number of downloaded tweets mentioning the public body. Results provide, in percentage terms, the incidence of comments made by non-official accounts.

This distinction between official and non-official tweets is relevant given that, for effectiveness to be elaborated, the users’ perception on a certain topic emerges in the non-official tweets.

The third area concerns when the service is evaluated. A distinctive feature of social media is that, by working in real time, users can leave their comments and material at all times (Kaplan and Haenlein, 2010). It follows that knowing when a conversation takes place is part of the information that can be extracted from social media data and, potentially, this can be at any moment of the day or week. Since social media can be accessed at all times, we decided to include an indicator of temporal distribution, to learn when the user is discussing the service. This time-linked distribution ratio is computed by summing the number of daily tweets, distinguishing between official and non-official accounts. This ratio, therefore, provides information about when users are talking about the public body, which means that it is possible to find out whether the users’ perception of the services is connected to any contingent situation.

Lastly, the last area of analysis focuses on how users talk about the service. On this point, a set of sentiment and opinion mining techniques are proposed in several social media contributions, and these can be used to explore the opinion of social media users (e.g. Archak et al., 2011; Nguyen et al., 2014; Thelwall et al., 2010). Given this context, we have included a polarity ratio within our framework to measure user opinion about the service. This polarity ratio provides information about how users perceive the public sector services previously identified through the discriminant ratio. The polarity ratio gives a score to each tweet, ranging from −6 (negative) to +6 (positive), and is computed by means of a weighted lexicon that contains a list of words and their associated scores in a [−6; +6] scale. A weight is given to every word in the tweet and the average of the scores of the words gives the polarity ratio of the tweet. The metric for the polarity ratio is the following

= \frac{\sum_{i = 1}^{n} S_{i}}{n}, where

s_i is the score of the word: s_i ∈ [−6; +6]

n is the number of words in a tweet

The model applied to Italian universities

Starting from the set of measures developed from literature, the research team, assisted by the university staff in charge of managing social media platforms, calculated the above Twitter-derived performance measures and discussed the potential and limitations of these measures with the universities involved. The validation of these measures implies first downloading the tweets and then calculating the proposed measures.

The discriminant ratio was the first indicator calculated and was obtained by using word count techniques associated with a similarity index to build clusters of similar words and separate those that are unconnected. This analysis generated an initial group of 30 clusters, which was further refined by aggregating similar clusters, still based upon the values of the similarity ratio. For example, in the initial larger dataset, news and events were two distinct clusters that were combined into the university life cluster during the final refinement. At the end of this phase, we reached a final number of four clusters, which are the main topics of discussion for our group of universities (Table 4): university life, teaching, research and administrative services.

Table 4.

Discriminant ratio

University name	University life (%)	Teaching (%)	Research (%)	Administration (%)	Total (%)
UNIV 1	51.95	14.50	10.33	23.22	100.00
UNIV 2	52.48	13.74	6.41	27.37	100.00
UNIV 3	40.81	19.29	14.69	25.20	100.00
UNIV 4	49.38	15.49	9.74	25.39	100.00
UNIV 5	50.55	20.44	8.17	20.83	100.00
UNIV 6	64.92	18.54	4.74	11.79	100.00
UNIV 7	46.62	21.90	13.30	18.19	100.00
UNIV 8	54.24	15.88	7.49	22.40	100.00
UNIV 9	55.27	18.65	10.43	15.65	100.00
UNIV 10	44.59	21.95	5.46	28.00	100.00
UNIV 11	51.45	17.85	6.23	24.47	100.00
UNIV 12	52.27	20.07	6.32	21.34	100.00
UNIV 13	48.27	18.54	9.42	23.78	100.00
UNIV 14	42.35	16.72	3.06	37.87	100.00
UNIV 15	47.79	21.42	7.77	23.02	100.00
UNIV 16	50.32	18.65	9.64	21.39	100.00
UNIV 17	50.83	16.19	9.14	23.83	100.00
UNIV 18	50.91	21.12	11.63	16.34	100.00
UNIV 19	40.21	25.29	7.85	26.65	100.00
UNIV 20	54.31	17.13	8.13	20.43	100.00
UNIV 21	52.89	17.23	10.14	19.74	100.00
AVG	49.69	18.43	8.77	23.11

The university life cluster refers to university operations other than teaching and research, including taking part in university events, university news and episodes in the students’ and staff’s everyday life (for example, students going to the library to study, events organised by student associations and student meetings in the university campus). The recurring words in this cluster include ‘events’, ‘life’, ‘pictures’, ‘news’ and ‘students’. Examples of tweets in this cluster are:

A meeting at the #UNIV20 to learn how to write a powerful cv.

Suggest a place where we can celebrate the end of exams, #UNIV14.

I feel sorry for those UNIV2 students walking back and forth on bridges with gigantic scale models.

The teaching cluster contains all the tweets that are concerned with academic courses, professors, the rector and preparing for exams. Tweets in this cluster mainly talk about how more or less satisfied students are about their classes. Examples of tweets in this cluster are:

An entire night lost in organising pdf documents uploaded by the professor after class #UNIV10 #study #nervous.

I’ve just finished my English class with a very good professor. I still have trust in human nature! #UNIV4.

The research cluster includes tweets about research projects, grants available and the results of the projects, as in the following tweet:

New research at #UNIV6: we may be wrong when interpreting other people’s feelings, but our brain corrects our interpretation.

According to the discriminant ratio, this category scored lowest, probably signalling that Twitter is not the tool used to discuss research projects and work.

Lastly, the administration cluster refers to comments about the support services offered by the university, such as job vacancies, recruitment matters, general services and enrolment procedures. An example of tweet in this cluster is the following:

Someone stole my university badge and I have to pay 30€ for a new one. UNIV12 is this fair?

Looking at the values of this discriminant ratio, we can make several reflections that can refer to the whole university system or to individual universities. With regards to the whole system, university life is the topic discussed most, in nearly half the Twitter conversations. Teaching and administrative life are about the same, while research is discussed least. These values provide an initial perception about the relative importance of university services, as seen through the eyes of university stakeholders. While teaching and research services are traditionally evaluated by universities through traditional customer satisfaction surveys, Twitter can provide a complementary view with conversations on university life, something not usually explored formally by universities through traditional monitoring tools.

The discriminant ratio was found to vary university by university, with some universities scoring more than 60% in the topic of university life and, indeed, seems to be the predominant topic in most universities, although in other cases, tweets are more balanced among the four categories. It is also important to underline that these four clusters are not expected to be general reference categories for universities analysis about Twitter. On the contrary, clusters in the discriminant ratio are expected to be data driven: it is the value of the similarity ratio that suggests topics of discussion and their connections. The identification of the final set of clusters is a managerial choice. For example, in our analysis we initially had 30 clusters, but participant universities found them being too many and not useful. Through discussions, cross-referenced with the value of the similarity ratio, we ended up with four categories.

The second proposed indicator is the non-official accounts ratio, which counts the number of tweets posted by the university official accounts and by third parties, compared to the overall number of tweets. Table 5 provides the detail of non-official account ratios for each topic identified previously and for each university.

Table 5.

Non-official account ratio

	University life	Teaching	Research	Administration	AVG
UNIV 1	28.99	28.48	25.46	34.04	29.24
UNIV 2	32.33	31.11	28.16	45.47	34.27
UNIV 3	70.48	63.29	78.61	77.49	72.47
UNIV 4	70.60	71.64	75.16	59.42	69.20
UNIV 5	63.87	64.31	53.21	60.17	60.39
UNIV 6	32.98	51.19	35.11	40.41	39.92
UNIV 7	14.74	13.10	8.69	6.98	10.88
UNIV 8	12.60	17.77	13.29	18.13	15.45
UNIV 9	49.21	61.57	49.27	59.36	54.85
UNIV 10	53.20	48.69	46.99	66.54	53.85
UNIV 11	100.00	100.00	100.00	100.00	100.00
UNIV 12	78.54	66.65	66.67	71.14	70.75
UNIV 13	49.56	48.08	48.03	50.16	48.96
UNIV 14	100.00	100.00	100.00	100.00	100.00
UNIV 15	56.89	59.59	67.70	72.42	64.15
UNIV 16	45.65	39.77	50.87	44.33	45.15
UNIV 17	45.16	36.31	50.70	60.90	48.27
UNIV 18	42.32	48.48	43.55	56.38	47.68
UNIV 19	100.00	100.00	100.00	100.00	100.00
UNIV 20	34.12	38.48	39.21	35.15	36.74
UNIV 21	29.34	28.55	14.65	34.08	26.65
AVG	50.75	50.80	49.79	55.12

Looking at the table, the first observation is that, taking the average for all the universities, the number of tweets published by official accounts is almost the same as that of those published by third parties, since the average value of the non-official account ratio is very close to 50%. When looking at the individual universities, the situation is different: in some universities, there are many more tweets posted by official accounts (e.g. UNIV 1 and UNIV 2) because the non-official account ratio is very small. This is the most common situation. In other universities, for certain topics, more posts are published by third parties than by the official accounts (i.e. UNIV 3, UNIV 4 and UNIV 5). Lastly, the universities with a non-official account ratio of 100% have no Twitter account. Although they are not on Twitter, users still talk about the university services and, therefore, it is possible to gain insights into the effectiveness of the universities’ services.

The third measure proposed is the temporal distribution ratio, which assesses the distribution of tweets over time.

We applied this indicator to our sample of universities (Figure 1), finding that the tweets have a cyclical distribution. First, for both official and non-official accounts, the tweets follow the same pattern, although the number of tweets is different. Second, we can identify ‘peak times’, corresponding to week days. Weekends and holidays are ‘non-peak times’. This is an interesting result, since Twitter is known to be a 24/7 relationship tool and is advertised as such by several university communication offices. However, despite the instrument potentially allowing users to communicate at every moment of every day and week, in practice, users talk mainly during week days.

Figure 1.

Temporal distribution ratio.

Finally, the fourth indicator refers to the polarity ratio, which measures the users’ opinion about the topics identified previously. In order to learn what users feel about the effectiveness of the services, we have computed the average polarity ratio for each topic (average of the polarity ratios of the tweets in the same cluster), obtaining the results shown in Table 6.

Table 6.

Polarity ratio

	University life	Teaching	Research	Administration	AVG
UNIV 1	0.74	0.61	0.97	0.28	0.65
UNIV 2	0.77	0.83	1.11	0.24	0.73
UNIV 3	0.51	0.66	1.10	0.40	0.67
UNIV 4	0.70	0.79	1.15	0.60	0.81
UNIV 5	0.56	0.52	0.92	0.15	0.54
UNIV 6	0.30	0.21	0.49	−0.41	0.15
UNIV 7	0.57	1.63	1.24	0.63	1.02
UNIV 8	0.98	1.25	0.67	0.20	0.77
UNIV 9	0.55	0.46	0.83	0.24	0.52
UNIV 10	0.64	0.42	0.89	0.60	0.64
UNIV 11	0.75	0.72	0.75	0.39	0.65
UNIV 12	0.52	0.62	0.42	0.38	0.49
UNIV 13	0.64	0.64	0.68	0.38	0.58
UNIV 14	0.85	0.46	0.32	0.25	0.47
UNIV 15	0.64	0.87	0.97	0.40	0.72
UNIV 16	0.82	0.60	1.01	0.05	0.62
UNIV 17	0.56	−0.40	0.89	−0.05	0.25
UNIV 18	0.61	0.54	0.41	0.71	0.57
UNIV 19	0.53	0.31	1.05	0.46	0.59
UNIV 20	0.73	0.60	0.68	0.60	0.65
UNIV 21	0.65	0.68	0.32	0.30	0.49
AVG	0.65	0.62	0.80	0.32

Negative scores correspond to negative perceptions about the service, while positive scores relate to positive opinions about the service. Values close to zero correspond to a neutral perception (i.e. neither negative nor positive). Looking at the values, on average the users’ perception about each service is close to zero (i.e. neutral). This is mainly because, when there are both positive and negative comments about a certain topic, when computing the average per cluster they cancel each other out.

There are a few exceptions, where the score is less than zero for some universities, mainly with reference to the administrative clusters. We specifically searched for tweets offering a negative perception about this cluster and found several examples:

I’m ill because the university heating system is not working @UNIV11. How is this possible?

Study room in the medicine faculty. Notice: unsafe furniture. @UNIV8 can you do something?

UNIV5 what about all the problems you don’t want to know about? What do you do all day? Play cards? Lazy office.

When discussing these Twitter-derived indicators with the university staff, they found the relative information extremely valuable for supplementing their yearly customer satisfaction surveys sent out to students, professors and administrative staff. Starting from the preliminary insights provided by our analysis, some universities asked us to carry out further vertical investigations in several areas they felt to be critical.

Discussion of the model with universities

The final phase of the action research project involved discussing the calculated Twitter-based measures with the participant universities. Both benefits and limitations of the four measures were highlighted.

With regards to the difficulties, university staff in charge of managing social media highlighted the importance of being able to analyse the data and of having the appropriate information technology and some degree of statistical competence to compute these measures.

I like these measures and found them very useful, but right now I cannot implement them without further support. I’m managing social media within the university. I have a background in communications and I would find it impossible to implement those measures without IT and maths support. We have to download Twitter data and this is a first problem. I can’t push a button and get the results automatically. I need to calculate them and I cannot do it by myself. (Social Media Manager, UNIV 7)

This quote highlights the first limitation related to the proposed measures that concerns the lack of adequate skills, an aspect shared by all the other participants at the round table discussion. It was the main reason why most of the universities involved were not applying any measurement process to Twitter data at the time when the investigation was being carried out but all relied on the same dashboard of measures provided by Twitter, as highlighted in this statement:

We check the Twitter dashboard regularly, usually on a weekly basis, to see if we are able to manage the tool. We are particularly interested in the hours of the day where interaction is highest and in the trend of number of followers. We have no structured system in place, we just check data on the Twitter dashboard. (Head of Communication Office, UNIV 1)

Twitter measures include the following: number of profile views, number of followers, number of posted tweets by the account and number of visualisations, number of retweets and number of replies for each tweet. These measures are provided automatically on the Twitter platform and are, therefore, available to all the universities. However, they quantify the level of Twitter activity, rather than giving information for measuring service effectiveness.

A second limitation, in part connected to the previous one, relates to the lack of resources to manage social media. The administrative staff of all the universities underlined that social media were introduced in the university without investing in additional personnel, relying instead on existing resources, mainly in the communication offices. Several participants underlined that some financial resources have been used to send communications staff on social media courses and improve their skills in this area. While this aspect is generally related to the difficult financial situation many Italian universities find themselves in, with their reduced spending power, in several universities this lack of resources was found to be connected to the negative perception of social media held by the administrative and political officials:

We introduced social media by ourselves. We saw what was happening in the outside world and in other universities: we could not miss out on these key tools to interact with our students. But we have to learn how to deal with social media on our own; when we present our work here at our general administrative meetings, we are told that it is a waste of time. (Head of Communications, UNIV 19)

It is important to point out that this negative perception of social media was found only in a few universities, and, in others, social media was planned and introduced top down or defined jointly between communication staff and the most senior university people. In both cases, however, only limited resources have been provided to manage social media, and this can pose problems when trying to implement further analyses concerning these social channels.

With regards to the benefits of the model, discussion with the universities on the four indicators has shown that Twitter has the potential to help in measuring service effectiveness, providing insight into university services without the university having to ask the service users for their views directly. This is an aspect of particular importance, as highlighted by the controller of UNIV 1:

Students are continuously receiving requests to fill in questionnaires: we have the teaching evaluation survey, the internal support service questionnaire, the final exam survey, the international survey. Students are complaining about these continuous demands; the less we ask, the better it is, otherwise we run the risk of getting a low response rate.

A further benefit emerged from discussions with participants was connected to the possibility of visualising the Twitter-derived measures to identify strong and weak service areas. Starting from this need, we developed a service effectiveness matrix to help university staff visualise the services rated highest and lowest by users.

The polarity ratio is positioned on the horizontal axis of the matrix, while the non-official account ratio is on the vertical axis. These axes are centred on the average value of the non-official account ratio and on the zero value for the polarity ratio. Each dot represents a topic of discussion (derived from the discriminant ratio). Four quadrants can be identified following this approach, and these correspond to four priorities for intervention: high performing area, risk area, indifferent area and unsafe area (Figure 2).

Figure 2.

Service effectiveness matrix.

The High performing area contains services with a high polarity ratio and a high non-official account ratio, corresponding to topics that are perceived positively by users (polarity ratio higher than zero) and where a number of users are talking about the service. These are the strongest areas for the university.

The Risk area includes services with a low polarity ratio but a high non-official account ratio. These have been defined as areas of risk since user perception is negative (polarity ratio lower than zero) and lots of people talk about the service. These areas require immediate intervention to avoid the negative trend increasing.

The Indifferent area has a high polarity ratio and a low-non official account ratio. It identifies services rated positively by users, but where there are only a few user comments, shown by the low value of the non-official account ratio.

Lastly, the Unsafe area corresponds to services perceived negatively, with a polarity ratio less than zero, and of scarce interest to users, as indicated by the low value of the non-official ratio. These services need to be monitored, since they are not perceived positively by users and the number of people talking about them could increase rapidly.

Figure 2 shows an example of this matrix for UNIV 18. Although the services are close to the axis, teaching services are positioned in a risk area, while university life is perceived positively by users. Administration services are neutral, since the polarity ratio is very close to zero, while research services are in the indifferent area because the related tweets are mainly posted by the university’s official account.

To sum up, this service effectiveness matrix can help staff visualise the effectiveness of a service from a Twitter perspective. Also limitations of this matrix need to be highlighted; the main limitation is related to positioning the axis of the polarity ratio in the average value of observations. This choice is affected by the universities included in the sample and does not provide a reference target value that is always valid. The same analysis performed in another context would have provided a different matrix. However, this opens up further research on how to define a reference value for the non-official account ratio, which is not dependent on the sample of observations.

Conclusion

The aim of this paper is to investigate how Twitter data can help measure the effectiveness of services offered by public bodies, with particular reference to the field of higher education. We carried out an action research project with the participation of a group of 21 Italian universities, and developed a set of Twitter-derived measures that have been applied to and discussed with our sample of universities. These measures comprise a discriminant ratio, a non-official account ratio, a temporal distribution ratio and a polarity ratio, which are used, respectively, to decide what higher education service is being talking about, who is talking about it, when it is being talked about and how it is being talked about.

Applying our model to the participant universities has highlighted two main benefits. These are the possibility of gaining insights into how higher education services are perceived without having to rely on surveys completed by students and staff, which often suffer from a low response rate; and that of identifying the priorities for action, by plotting the measures in a service effectiveness matrix. The limitations have also been highlighted; these mainly refer to the fact that statistical and mathematical skills are needed to elaborate the proposed measures, and that administrative and political officials can have a negative view of social media, associated in turn to the scarce resources available for social media measurement purposes. The key point is that Twitter-derived measures can complement these traditional customer satisfaction surveys filled in by students, professors and academic staff. While traditional surveys usually focus on teaching and support services, Twitter-derived measures mainly provide indications about university life. Further studies could investigate the correlation between how a service is perceived according to traditional surveys and according to social media data.

These results can serve both the academic and practitioner domains.

From an academic perspective, this study provides three main contributions. First, it contributes to extant literature on performance measurement in the public sector. While measuring performance in public administration is a long-standing issue (e.g. Hood, 1991, 1995), the available contributions to date propose performance measures that are based on traditional organisational data. We, instead, underlined the possibility of measuring public service performance using the newer social media data, providing evidence about a set of measures that can be derived from Twitter to quantify the effectiveness of public services, together with a discussion about their benefits and limitations.

Second, this study expands the literature on use of social media in the field of higher education. Current studies on social media in higher education focus on how to use social media to enhance the areas of learning, teaching, enrolment, marketing and library services (e.g. Bélanger et al., 2013; Constantinides and Stagno, 2011; Manca and Ranieri, 2016; Palmer, 2013), emphasising the fact that social media data are interactive and based on dialogue among users. The present study underlines the potential of social media data in supporting the back office administration work of planning and controlling performance in a university. This, therefore, stresses the importance of social media, not simply because it allows the university to interact with the ‘outside’ (i.e. students) but also in connection with internal management processes.

Third, this study contributes to the more general literature on social media and big data, by emphasising the added value of social media data from a manager’s perspective (e.g. Criado et al., 2013; George et al., 2014). While several investigations concentrate on developing a set of analytics techniques to download and analyse social media data (e.g. Agostino and Sidorova, 2016; Archak et al., 2011; Bhardwaj et al., 2014; Netzer et al., 2012; Nguyen et al., 2014; Thelwall et al., 2010), the management implications of the available set of measures is mainly underexplored. By developing the proposed set of measures, we underline the possibility of mining social media data as a means of supporting the work of managers to measure performance, with particular reference to service effectiveness, thereby answering the recent call for ‘categorizing big data, assessing its quality, and identifying its impact’ (George et al., 2014: 324). Moreover, this study further underlines the indirect power of social media, which is connected with literature on social media. We found that, even if a university does not have a Twitter account, Twitter users will talk about the university anyway. In our analysis, this was visible in universities with a non-official account ratio of 100%. This is a relevant issue given that, even when a university does not want (yet) to take the social media path, its stakeholders will inevitably talk about it, its initiatives, events and organisations.

From a practitioner perspective, and in line with the purposes of action research, we also addressed some practical concerns. The first relates to a benchmark among Italian universities about the uses of social media and the users’ perceptions about university services. While this contribution finds relevance at the Italian level only, at a more general level, this study provides a set of applicable measures that can be exploited by practitioners in the higher education field, as well as in other public sector fields, to compute the effectiveness of their services. In addition, a service effectiveness matrix was provided, which can help serve practitioners, in particular, to identify the risky and safe areas for intervention, on the basis of the measures previously computed.

Finally, we pointed to the issue of generalising the results, which is strictly connected to the action research methodology we adopted. While action research facilitates access to information and ensures continuous interaction with the observed reality, this study was conducted on a sample of Italian universities, leading to the question of whether it is, indeed, possible to generalise these results and apply them to similar institutions. In this respect, it is important to distinguish between what is general in scope and what is case specific. The type of measures that can be derived from Twitter endeavour to be general in scope. While the application and detailed values for each measure relate to our sample of Italian universities, the proposed measures can be applied to other institutions of higher education and public bodies with a Twitter account that want to exploit Twitter data to evaluate the effectiveness of their services. In the same vein, the benefits and limitations of the proposed set of measures also aim to be general in scope since they address several issues, from lack of commitment to lack of resources or competencies, that are found in similar organisations facing the same need to exploit social media data.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Agasisti

Bonomi

(2014) Benchmarking universities’ efficiency indicators in the presence of internal heterogeneity. Studies in Higher Education 39(7): 1237–1255.

Agostino

Arnaboldi

(2016) A measurement framework for assessing the contribution of social media to public engagement: An empirical analysis on Facebook. Public Management Review 18(9): 1289–1307.

Agostino

Sidorova

(2016) A performance measurement system to quantify the contribution of social media: New requirements for metrics and methods. Measuring Business Excellence 20(2): 38–51.

Archak

Ghose

Ipeirotis

(2011) Deriving the pricing power of product features by mining consumer reviews. Management Science 57(8): 1485–1509.

Arena

Arnaboldi

Azzone

(2010) Student perceptions and central administrative services: The case of higher education in Italy. Studies in Higher Education 35(8): 941–959.

Arnaboldi M and Coget J (2016) Social media and business: We’ve been asking the wrong question! Organizational Dynamics 45(1): 47–54.

Baskerville

Wood-Harper

(1996) A critical perspective on action research as a method for information system research. Journal of Information Technology 11: 235–246.

Bekkers

Edwards

Moody

et al. (2011) Caught by surprise? Micromobilization, new media and the management of strategic surprises. Public Management Review 13(7): 1003–1021.

Bélanger

Bali

Longden

(2013) How Canadian universities use social media to brand themselves. Tertiary Education and Management 20(1): 14–29.

10.

Bertot

Jaeger

Hansen

(2012) The impact of polices on government social media usage: Issues, challenges, and recommendations. Government Information Quarterly 29(1): 30–40.

11.

Bhardwaj N, Shukla A and Swarnakar P (2014) Users’ sentiment analysis in social media context using natural language processing. In: Proceedings of the International Conference on Digital Information, Networking, and Wireless Communications (DINWC2014), 24–26 June 2014, pp.103–111. Hong Kong: Society of Digital Information and Wireless Communication. Available at: http://search.proquest.com/docview/1703412001?accountid=28385.

12.

Blackman

Greene

Hunter

et al. (2006) Performance assessment and wicked problems: The case of health inequalities. Public Policy and Administration 21(2): 66–80.

13.

Boland

Fowler

(2000) A systems perspective of performance management in public sector organisations. International Journal of Public Sector Management 13(5): 417–446.

14.

Bonsón

Torres

Royo

et al. (2012) Local e-Government 2.0: Social media and corporate transparency in municipalities. Government Information Quarterly 29(2): 123–132.

15.

Boyne

(2003) What is public service improvement? Public Administration 81(2): 211–227.

16.

Brochado

(2009) Comparing alternative instruments to measure service quality in higher education. Quality Assurance in Education 17(2): 74–190.

17.

Campbell

Lambright

Wells

(2014) Looking for friends, fans, and followers? Social media use in public and Nonproﬁt human services. Public Administration Research 74(5): 655–663.

18.

Chun

Luna-Reyes

(2012) Social media in government. Government Information Quarterly 24(4): 441–445.

19.

Constantinides

Stagno

MCZ

(2011) Potential of the social media as instruments of higher education marketing: A segmentation study. Journal of Marketing for Higher Education 21(1): 7–24.

20.

Cosenza V (2014) World map of social networks. Available at: http://vincos.it/world-map-of-social-networks/ (accessed 10 July 2016).

21.

Craig

(2014) Would you like service with that? Journal of Higher Education Policy and Management 36(3): 293–304.

22.

Criado

Sandoval-Almazan

Gil-Garcia

(2013) Government innovation through social media. Government Information Quarterly 30(4): 319–326.

23.

Cutler

(2011) Performance management in public services ‘before’ new public management: The case of NHS Acute Hospitals 1948–1962. Public Policy and Administration 26(1): 129–147.

24.

Dužević

Čeh Časni

(2015) Student and faculty perceptions of service quality: The moderating role of the institutional aspects. Higher Education 70(3): 567–589.

25.

Evans

(2013) Making sense of assessment feedback in higher education. Review of Educational Research 83(1): 70–120.

26.

Forbes

Lynn

(2005) How does public management affect government performance? Findings from international research. Journal of Public Administration Research and Theory 15(4): 559–584.

27.

Freberg

Saling

Vidologg

et al. (2013) Using value modeling to evaluate social media messages: The case of Hurricane Irene. Public Relations Review 39(3): 185–192.

28.

Gandomi

Haider

(2015) Beyond the hype: Big Data concepts, methods and analytics. International Journal of Information Management 35: 137–144.

29.

Gao

Barbier

Goolsby

(2011) Harnessing the crowdsourcing power of social media for disaster relief. IEEE Intelligent Systems 26(3): 10–14.

30.

George

Haas

Pentland

(2014) Big Data and management. Academy of Management Journal 57(2): 321–326.

31.

Grimmelikhuijsen

Meijer

(2015) Does twitter increase perceived police legitimacy? Public Administration Review 75(4): 598–607.

32.

Henderson

Selwyn

Finger

et al. (2015) Students’ everyday engagement with digital technology in university: Exploring patterns of use and ‘usefulness’. Journal of Higher Education Policy and Management 37(3): 308–319.

33.

Herring

(2010) Web content analysis: Expanding the paradigm. In: Hunsinger

Allen

Klastrup

(eds) International Handbook of Internet Research, Dordrecht: Springer, pp. 233–249.

34.

Hodgson

Farrell

Connolly

(2007) Improving UK public services: A review of the evidence. Public Administration 85(2): 355–382.

35.

Hood

(1991) A public management for all seasons? Public Administration 69(1): 3–19.

36.

Hood

(1995) The ‘new public management’ in the 1980s: Variations on a theme. Accounting, Organizations and Society 20(2/3): 93–109.

37.

Jackson PM and Palmer B (1992) Developing Performance Monitoring in Public Sector Organizations. Leicester: The Management Centre, University of Leicester.

38.

Johnsen

(2005) What does 25 years of experience tell us about the state of performance measurement in public policy and management? Public Money and Management 25(1): 9–17.

39.

Kaplan

Haenlein

(2010) Users of the world, unite! The challenges and opportunities of social media. Business Horizons 53(1): 59–68.

40.

Kim

Trimi

Chung

(2014) Big-Data applications in the government sector. Communications of the ACM 57(3): 78–85.

41.

Lægreid

Roness

Rubecksen

(2006) Performance management in practice: The Norwegian way. Financial Accountability and Management 22(3): 251–270.

42.

Lavertu S (2016) We all need help: “Big Data” and the mismeasure of public administration. Public Administration Research 76(6): 864–872.

43.

Lev-On

Steinfeld

(2015) Local engagement online: Municipal Facebook pages as hubs of interaction. Government Information Quarterly 32(3): 299–307.

44.

McAfee A and Brynjolfsson E (2012) Big Data: The management revolution. Harvard Business Review, October: 1–9.

45.

Manca

Ranieri

(2016) “Yes for sharing, no for teaching!”: Social Media in academic practices. Internet and Higher Education 29: 63–74.

46.

Mansour

Heath

Brannan

(2015) Exploring the role of HR practitioners in pursuit of organizational effectiveness in higher education institutions. Journal of Change Management 15(3): 210–230.

47.

Matthews FM (2016) Letting go and holding on: The politics of performance management in the United Kingdom. Public Policy and Administration 31(4): 303–323.

48.

Mergel

Bretschneider

(2013) A three-stage adoption process for social media use in government. Public Administration Review 73(3): 390–400.

49.

Moynihan

Pandey

(2010) The big question for performance management: Why do managers use performance information? Journal of Public Administration Research and Theory 20(4): 849–866.

50.

Netzer

Feldman

Goldenberg

et al. (2012) Mine your own business: Market-structure surveillance through text mining. Marketing Science 31(3): 521–543.

51.

Nguyen

Quan

Phan

(2014) Sentiment search: An emerging trend on social media monitoring systems. Aslib Journal of Information Management 66(5): 553–580.

52.

Osborne

(2006) Editorial: The new public governance? Public Management Review 8(3): 377–387.

53.

Osborne

Radnor

Kinder

et al. (2015) The SERVICE framework: A public-service-dominant approach to sustainable public services. British Journal of Management 26(3): 424–438.

54.

Palmer

(2013) Characterisation of the use of Twitter by Australian Universities. Journal of Higher Education Policy and Management 35(4): 333–344.

55.

Pettersen

(2015) From metrics to knowledge? Quality assessment in higher education. Financial Accountability and Management 31(1): 23–40.

56.

Pollanen

(2005) Performance measurement in municipalities empirical evidence in Canadian context. International Journal of Public Sector Management 18(1): 4–24.

57.

Reason

Bradbury

(2001) Handbook of Action Research: Participative Inquiry and Practice, London: Sage Publications.

58.

Sarrico

Rosa

(2014) Student satisfaction with Portuguese higher education institutions: The view of different types of students. Tertiary Education and Management 20(2): 165–178.

59.

Stejskal

Hájek

(2015) Evaluating the economic value of a public Service – The case of the Municipal Library of Prague. Public Money and Management 35(2): 145–152.

60.

Talbot

(1999) Public Performance-towards a new model? Public Policy and Administration 14(3): 15–34.

61.

Thelwall

Wilkinson

Uppal

(2010) Data mining emotion in social network communication: Gender differences in MySpace. Journal of the American Society for Information Science and Technology 61(1): 190–199.

62.

van Helden

Johnsen

Vakkuri

(2008) Distinctive research patterns on public sector performance measurement of public administration and accounting disciplines. Public Management Review 10(5): 641–651.

63.

Waters

Burnett

Lamm

et al. (2009) Engaging stakeholders through social networking: How nonprofit organizations are using Facebook. Public Relations Review 35(2): 102–106.

64.

Waters

Williams

(2011) Squawking, tweeting, cooing, and hooting: Analyzing the communication patters of government agencies on Twitter. Journal of Public Affairs 11: 353–363.