The impact of the Balanced Scorecard in libraries: From performance measurement to strategic management

Introduction

Developing, implementing and using adequate performance measurement systems and strategic management systems have been essential issues in the managerial context during the last two decades. The Balanced Scorecard (BSC), created and developed by Robert Kaplan and David Norton in the early 1990s, is probably one of the most successful tools, used extensively in all types of organizations worldwide. As remarked by Meng and Minogue (2011), unlike traditional approaches primarily reliant on financial measures, the BSC evaluates whether an organization is moving towards its strategic goals from four different perspectives: financial, customer, internal business process, and learning and growth. It aims to balance long- and short-term objectives, financial and non-financial measures, leading (performance drivers) and lagging (outcome measures) indicators, and internal and external performance perspectives (Hepworth, 1998). First designed as a simple performance measurement framework, it has evolved into a comprehensive management system that enables organizations to clarify their vision and strategy and translate them into action.

Originally developed for the private for-profit sector, in order to bring about financial success, it was soon adopted by, and adapted to, public sector organizations as a system to achieve and communicate their effectiveness in serving the public. In 1999, Kaplan himself stated that the BSC was equally or even more appropriate for not-for-profit organizations: ‘Although the Balanced Scorecard has initially been applied to the for-profit (private) sector, its potential to improve the management of public-sector organizations is even greater’ (Kaplan, 1999: 1).

However, when the BSC is applied to public organizations, the framework should be changed to capture their mission-driven nature, placing more emphasis on accountability and results in meeting user expectations for public services and products (Rohm, 2004). In that regard, the literature review developed by Greatbanks and Tapp (2007) on public sector performance measurement and the use of the BSC confirmed that its application in public and not-for-profit organizations is not readily transferable from the private sector, and some modifications and adjustments are necessary. Furthermore, they determined that the implementation process was more complex and difficult than in the private sector, owing to the multiplicity of customers and stakeholders and the disparate nature of public sector organizations. Finally, they concluded that those difficulties appear to be responsible for the reported lack of empirical evidence regarding BSC implementation within public and not-for-profit organizations.

Thus, the present study about the use of the BSC in libraries can be located in this context of an expanding body of literature on BSC implementations in public and not-for-profit sector institutions and the need for empirical studies. Libraries became interested in the BSC at an early stage. Towards the end of the 1990s, some academic libraries decided to adopt it to face the challenges of a growing institutional demand for proving the effectiveness and efficiency of their services, and their impact and benefits in regard to the community of users. The background of libraries’ performance at that time, as described by Poll (2001b: 2), evidenced the need for adopting a strategic management system:

There is an immense pool of management data available today in libraries: statistics of resources, services and use, cost data and combined data like performance indicators for the quality of library services. The quantity, diversity and complexity of the data stress the need for an integrated system to make their management useful for evaluation, strategy and action.

Taking that diagnosis as the starting point of our research and aware that the BSC is a relatively new concept in libraries which has not been thoroughly investigated, this paper aims, firstly, to analyse and trace the development of the BSC in libraries worldwide through the testimonies and experiences that have appeared in the literature during the last 15 years; and secondly, to explore the use and impact of BSC implementation in libraries, surveying and providing evidence of the main characteristics and outcomes of BSC practice in these not-for-profit organizations.

The development of the BSC in libraries: Literature review

Libraries took an early interest in the BSC with the first documented cases at the end of the 1990s. Only one year after Kaplan and Norton published their book about the BSC (1996), the Academic Information Service of the University of Pretoria (South Africa) decided to adopt this tool as one of the mechanisms for its transformation to a virtual information service (Pienaar and Penzhorn, 2000). Another example of early adoption can be found in the Australian Deakin University Library, which, in a context of growing focus on its strategic plans, adopted this system in 1998 to drive the process of translating customer values into achievable objectives (Livingston, 2000; Wilson and Pitman, 2000). In 1999, the first initiative to develop the BSC in a collaborative context commenced: the German Research Council promoted the project ‘Integrative controlling for academic libraries’ aimed at implementing the BSC in three of the largest libraries in Germany. ¹ Unlike the previous cases, the original scorecard model was modified by placing the ‘customer’ perspective at the head of the system, instead of the ‘financial’ one, just as Kaplan and Norton recommended for not-for-profit organizations (Ceynowa, 2000; Poll, 2001a, 2001b). Many ideas and indicators resulting from that initiative were incorporated into the German academic library benchmarking project BIX, which started in the same year, with indicators grouped in the perspectives of the BSC (Poll and te Boekhorst, 2007). Poll (2006: 3) stated that ‘the use of the Balanced Scorecard is unique in BIX. It has been used by individual libraries, but not in benchmarking projects’. The German experience also influenced the adoption of the BSC structure in the second edition of the International Standard ISO 11620 Information and Documentation – Library Performance Indicators (ISO, 2008).

At the beginning of the 21st century, some international library organizations began to take an interest in the potential of the BSC as a comprehensive performance management system (Association of Research Libraries (ARL), 2011). Within this context, taking the initiative in the USA, the University of Virginia Library implemented this system as part of an effort to develop a culture of assessment (Self, 2003, 2004). After more than 10 years, it is currently one of the libraries with the longest, most successful and best-known experience in the use of the BSC.

During this period, we can also find the first examples of National Libraries using the BSC. Forced to review their historical modus operandi, owing to the explosion of information access afforded by the Internet, and the growing importance of value for money (Penny, 2005), National Libraries saw the BSC as a good tool to manage the implementation of the new strategies they had to set. This seems to have been the case for both the National Library of Australia (Cameron, 2008) and the Royal Library of Denmark (Krarup, 2004), which by the year 2000 had started to implement the BSC as a means to achieve their strategic goals and to perform better and more efficiently.

Australian libraries paid special attention to the BSC during these first years of the new century. Following the path opened by the National Library of Australia, at least three other large libraries applied the BSC for their strategic planning: in 2001 the Central Queensland University Library (CQU Library, 2001); in 2003, the Bond University Library (linked to the University’s institutional BSC in a context of massive growth and financial constraints) (Cribb and Hogan, 2003); and the State Library of Queensland (2004). This interest in the BSC went beyond national borders, giving rise to a very interesting project at the end of 2002: on 5 December, the National Library of Australia and the National Library of Singapore signed a Memorandum of Understanding that would allow them to benefit from each other’s expertise in different areas (strategic directions and performance, information-sharing and access, resource sharing) with the BSC a key tool in this collaboration (Kon, 2005). Another example in Asia is the National University of Singapore (NUS), which also in 2002 began the process for the implementation of the BSC to improve its evaluation process, within the broader context of the Quality Assurance Framework for Universities developed by the Ministry of Education. As part of this project, the NUS library presented its first BSC to the Deputy President in December 2002 (Leong, 2005).

By this time, there were also some academic libraries in Europe which had adopted the BSC model as part of a wider institutional process developed by their universities, which promoted the use of the BSC as a way of measuring the performance of the university as a whole: this is the case of the University of Hull Library Services (Stanley and Killick, 2009) and Leeds University Library (Franklin, 2010), both in the UK. There were also some independent initiatives, such as that developed by the University of Glasgow Library, which, in 2007, began a pilot project with the BSC to overcome the limitations of the traditional library metrics (Dowd, 2009).

There was also an interest in the BSC in European national libraries, not only as an evaluation tool, but again as a strategic framework for facing the important changes taking place in their operational environment. This seems to have been the case at the Finnish National Library, which, at the end of 2005, compiled its strategy for the following 10 years by using the BSC method (Finnish National Library, 2005). Another example of that strategic approach to the BSC can be found in the National Library of Scotland, which in 2005–2006 implemented a formal set of Key Performance Indicators (KPIs) as an outcome of its new strategy ‘Breaking through the Walls’ (Hunter, 2009). This experience was reported to the IFLA National Libraries Section Standing Committee, in the second meeting of its 71st General Conference held in Oslo on 19 August 2005 (IFLA, 2005) – clear proof of the interest aroused by this type of initiative. Following the same path, a few years later, in 2008, one of the three German National Libraries, the German National Library of Science and Technology, implemented the BSC as its strategic management system, to fulfil the new public management approach developed by the local government (Düren, 2010). All these experiences seem to support Penny (2005) who, based on his experience of the development of the BSC at the Australian National Library and elsewhere, had stated that the BSC was ‘an ideal format for the management of (national libraries) strategy implementation’ (Penny, 2005: 155 ).

Public libraries were also interested in the BSC approach, although the first experiences were developed some years later than in academic and national libraries. North America was the location for those first initiatives. In 2004, the Institute of Museum and Library Services (ILMS, ² Washington DC) funded a project ³ that explored the feasibility of adapting the BSC to the public library environment. The four libraries that took part in the first phase were the Chula Vista Public Library, the Carlsbad Public Library, the Cerritos Public Library and the Newport Beach Public Library, all located in southern California (Matthews, 2006). As result of these libraries’ experiences, a workbook was compiled and, in a second phase of the project, about 50 public libraries nationwide, including the Hartford Public Library and the Cleveland Heights-University Heights Public Library, agreed to put it through a trial over the period 2005–2006 and prepared their own BSC (Matthews, 2008). One of the most important outcomes of that project was a re-arrangement of the traditional perspectives of the BSC in order to create a possible Library Balanced Scorecard, which would place the processes perspective at the base of the structure and the customer perspective at the top, going through the learning and growth, financial and information resources perspectives (Matthews, 2011). There were also larger public library systems which started to develop a BSC model in those years, such as the Denver Public Library (Denver Public Library Commission, 2008), the Pierce County Library System (Pierce County Library System, 2007), the North Central Regional Library (Howard and Marney, 2007) and the Orange County Library System in Orlando, Florida (Orange County Library System, 2007).

Another initiative, noteworthy because of the context and the scope of the project, is the comprehensive planning effort carried out by a state library system, the Delaware Division of Libraries, in 2004–2005, aimed at improving its performance levels and becoming the ‘first in the nation in providing public library service’ (Himmel & Wilson; PROVIDENCE, 2005: 2). They decided to adopt the BSC as the method to develop their state-wide master plan, because it provided ‘a mechanism that lends itself to working together to achieve common objectives’, so that the state-wide plan, the Delaware Division of Libraries’ own plan, and individual library plans would be linked together (Himmel & Wilson; PROVIDENCE, 2005: 22). A special feature of that initiative, which differentiates its approach from some of the other examples, was the link between the adoption of the Malcolm Baldrige Criteria and the implementation of the BSC (Wilson et al., 2008).

Other examples can be found beyond the US, among Canadian public libraries, including the London Public Library (Ontario), which, in 2006, adopted the BSC as ‘an analysis framework’ that would allow the library to structure a set of measurements and to accomplish its strategic plan (London Public Library, 2006; Radcliffe and Nichol, 2007). In 2008, a Prospective Balanced Scorecard Model (at strategic and operational level) was proposed as a ‘progressive evaluation method’ of a 10-year Consolidation Plan (2005–2014) for the Montreal Public Libraries system (Meunier et al., 2009: 155).

In 2009, the Association of Research Libraries launched the Library Scorecard Pilot to test the use of the BSC in research libraries. This initiative began with four participating libraries: Johns Hopkins University, McMaster University (Canada), the University of Virginia (the only one with previous experience in the BSC) and the University of Washington (Kyrillidou, 2010; Lewis et al., 2013). This was one of the first collaborative projects in the adoption of the BSC for academic libraries launched after the early German initiative in 1999, and ‘aimed to test the viability of creating a community of practice to facilitate scorecard implementation and practice’ (Mengel and Lewis, 2012: 358). The experience suggested that the collaborative approach paid off: saving time, gaining high credibility and laying the ground for future benchmarking were the main benefits identified by the pilot libraries. In May 2011 a second cohort of 10 libraries ⁴ was formed, augmenting the initial design of the first cohort (ARL, 2011). In March 2013, ARL issued a call for participation in the third cohort, with the goal of recruiting five libraries that would like to use the BSC in a coordinated and collaborative approach during 2013-2014 (Kyrillidou, 2013).

Another collaborative initiative was developed by the International Coalition of Library Consortia (ICOLC) as result of its meeting held in April 2011. Eight important US Library consortia ⁵ began working together on the ICOLC Balanced Scorecard pilot project. The first phase was completed in the summer of 2012 and the results demonstrated that ‘the participants have benefited greatly from synergy and shared learning’ (Perry and Self, 2012: 44). Other interesting experiences had taken place previously in the use of the BSC in the European library consortia context, such as the study by Kettunnen (2007) which, with a narrower scope, aimed to test the applicability of the BSC approach in the evaluation of the strategic plans of a consortium of the digital libraries of 29 Finnish universities, although not for the creation of the libraries’ strategic plans.

Other noteworthy initiatives which feature the combined use of the BSC with other evaluation methods have been developed in Portuguese public and academic libraries. The first was a research project funded by INCITE (the Portuguese Association for Information Management) in 2004, which proposed a new model for public library and information services evaluation based on the integrated use of several evaluation methods: the Common Assessment Framework (CAF, based on the European Foundation Quality Management Excellence Model) for self-assessment; the BSC for strategic evaluation; and the Integrated System for Performance Appraisal in Public Administration (SIADAP) adopted by the Portuguese Government, covering organizational and individual results (Gaspar et al., 2006). Two years later, a Portuguese action model was proposed to measure the contribution of the academic library to the institution and to the society to which it belongs, based on a mixed method combining the CAF, the BSC and the Analytic Hierarchy Process (CAF-BSC-AHP) (Baptista Melo and Cardosa Sampaio, 2006). A similar mixed model combining the CAF and the BSC was used in the project ‘Recognizing Best Practices in Higher Portuguese Education Libraries’ resulting from a partnership of six Portuguese academic libraries and aimed at assessing the performance of library services (Baptista Melo et al., 2008).

The experience and example of all these libraries with the use of the BSC has encouraged other libraries to adopt it, for example in China where the ARL’s experience with the BSC has provided a reference for libraries to develop relevant measures of success (Huang and Song, 2010).

This review of the development of the BSC in libraries worldwide is not intended to be exhaustive, but rather representative of the main events and milestones along that path. Below, we attempt to investigate more deeply the features of that development through the testimony of libraries that use the BSC in their daily management.

Surveying the use of the BSC in libraries: Methodological issues

Paraphrasing Schick (2003), it is evident that the BSC is an idea whose time has come for libraries, but to what extent has it been taken up, and what has been the outcome for libraries?

To answer that question, we surveyed the development, use and impact of the BSC in libraries to identify its main characteristics and outcomes. Within that overall purpose, since the BSC framework emphasizes the need for strategy measurement as the way to turn the strategy into reality, we paid special attention to the issues related to the building of the strategic control system and the selection of relevant performance indicators.

Collection of data: Design and distribution of the questionnaire

In order to test these hypotheses, we used an electronic survey to collect empirical data on the profiles of participating libraries, the characteristics of BSC implementation and use, selection of performance measures and main outcomes. The questionnaire was developed using previously published research on the use of the BSC in libraries. In order to refine the design and the content, the survey was piloted with two experts in the use of the BSC in libraries, ⁶ whose comments and suggestions were incorporated in the final version of the questionnaire (see Appendix 1).

One of the most challenging issues of the research was to define a sample of libraries with experience in the use of BSC to which the questionnaire could be sent. As Lewis and his colleagues stated in 2013, ‘although the total number of libraries adopting the Balanced Scorecard is unknown, it is likely still only a few handfuls worldwide’ (Lewis et al., 2013: 2). Moreover our aim was to survey the libraries which had implemented the BSC fully, not just as a performance measurement system. For that reason, we did not include libraries involved in joint performance measurement projects, such as BIX, which have their metrics structured following the BSC perspectives, but do not connect the indicators with their individual mission and goals.

Devising a list of libraries with experience in using BSC that was as exhaustive as possible was a very demanding task in time and effort; 49 libraries were identified from all five continents and 12 different countries, and a mailing list was prepared (see Appendix 2).

An email was sent to all these libraries on 6 December 2012 inviting them to participate in the survey with an initial closing date of 31 December, extended until 9 January 2013; 15 responses were received (30%).

Figure 1 shows the breakdown of invitations sent and responses received, by region and by library type. Although there are insufficient data formally to test the representativeness of the responses on these criteria, the graph suggests that academic libraries may be under-represented in the responses, and North American libraries over-represented. However, as we are not attempting to make formal estimates of the numbers or proportions of libraries in the wider population which exhibit particular traits in their adoption and use of the BSC, nor are we aiming to compare different types of library or different regions, this imbalance does not affect our analysis or conclusions.

OPEN IN VIEWER

Figure 1.

Libraries by type and region.

If we take into account the identity of the respondent libraries, the sample is qualitatively very valuable because it includes some of the academic libraries with the longest tradition in the use of the BSC, and some libraries taking part in collaborative projects, such as the one promoted by the ALA. One limitation is that all the responding libraries are in westernized, developed, countries, and experience in Asian and developing world countries may differ. The respondent libraries vary in size, serving from 2500 to several hundreds of thousands of users, with a predominance of larger libraries.

Analysis of the results

Selection of indicators (H6 and H7)

The most challenging issue for 10 of the 15 libraries in implementing the BSC was selecting the KPIs. Other most challenging issues were elaborating the strategic map; the holistic use and sequence of business tools including the BSC; getting employees to focus attention on achieving the goals set; implementing fully; and ‘time!’ Only two libraries used specialist software (Clearpoint) to support their use of the BSC.

The number of KPIs used varied from just four in a state library agency and six in an academic library, to 35 in one national library and ‘around 50’ in another. Only four libraries had referred to national standards for library performance indicators for their KPIs and just two considered being ‘included in a national/international standard or recommendation’ a valid criterion in choosing them. All had used the relationship to strategic goals as one criterion when making their final selection; other common criteria included previous use, staff consensus, easy data collection and, surprisingly, use by benchmarking partners, although in the previous question no library referred to ‘local standards (e.g. benchmarking group)’ for selecting its KPIs (Figure 2).

OPEN IN VIEWER

Figure 2.

Which criteria did your library use to select the final set of KPIs?

Of the 15 libraries 12 use other performance indicators or management statistics in addition to their KPIs, for monitoring and evaluating services. Reasons for using other statistics varied, including for taking lower-level decisions, use at departmental level, and requiring other indicators for public-facing services. One library noted that the BSC works best with a limited number of measures, while another commented that a large amount of data was gathered in the course of its operations but this was not relevant to the strategic priorities that were the focus of their BSC. One library which did not use other statistics noted that their BSC was very large, and encompassed almost all of what they do.

Benefits from using BSC (H8)

Eleven libraries felt that they had benefited from using a BSC. The remaining four had not been using the framework for long (less than one year in three cases, and between one and three years for the other) and felt it was still too early to say. Libraries were asked to indicate which of a range of benefits they felt were the most important to have been realized in their library, and the responses are summarized in Figure 3.

OPEN IN VIEWER

Figure 3.

Please indicate the most important benefit(s) realized.

Libraries were also asked what they thought was the most positive outcome for the library from using the BSC technique. Responses were divided across four broad areas – strategic outcomes, operational outcomes, reporting and advocacy, and staff culture.

The BSC was thought to be a ‘great way to align effort with the organization’s mission, measure progress and communicate results’, providing a better overview of strategic goals, and helping to translate those goals into an action plan. For one library, focusing a large number of performance indicators down to just 20 KPIs was seen as the most positive outcome, while another had used it as a tool to organize their data as part of a wider quality initiative.

Transparency was also a key outcome, with consistency of reporting across an extended period perceived as an advantage. The BSC was thought to provide an important tool to demonstrate effectively the library’s importance both internally, at a corporate strategic level, and externally. One library noted that the strategy had proven to be valuable in articulating key library objectives to external audiences.

The opportunity to engage staff in planning, promote a culture of assessment and make sure that activity aligns with the BSC, were aspects of staff-related outcomes which can be summed up by one library’s comment, ‘cultural shift to importance of metrics and analyses for quality/performance improvement’. Staff awareness and ownership was a factor noted by one library, while another said that staff were ‘more aligned with strategic goals and objectives’ as a result of using the BSC, and were able better to understand how their work helps achieve community outcomes. Another, in the early stages of development of its BSC, felt that it had been useful having the entire library staff focused on strategy and institutional expectations.

Discussion and final remarks

Although the first initiatives to develop the BSC in libraries took place very early, in the 1990s, our study found that most of the libraries implemented it more recently, suggesting that the adoption of the BSC has not been a general or massive movement in the library sector, but rather selective and irregular, with an ascending tendency in the most recent past, possibly promoted by the collective initiatives of important libraries’ associations and consortia.

Despite the importance that cooperative projects may have had in endorsing the use of the BSC, most of the libraries surveyed said they had developed their scorecards individually rather than in collaboration, even though some of them are participating in the ALA project or in other collaborative programmes. The key reason for this would appear to be that the BSC ‘must be adapted or changed to fit a specific organization’ (Pienaar and Penzhorn, 2000: 203). So, although several libraries can work together to deploy the BSC model, every library must adapt it to its own project, strategy and reality. In this context, it makes sense that for the respondent libraries the most important motivation for using BSC has been to improve their own management, more than other contributory factors such as the experiences of other libraries with the BSC; being part of a benchmarking project; proving value to the parent organization; or changing the organizational culture.

It should also be taken into account that the implementation of the BSC is not an easy task. Understanding the concepts, being aware of the methodology and meeting the technical requirements of deploying the BSC are complex processes and most libraries needed some type of support in order to tackle them. The testimonies collected and the experiences published show that in many cases libraries have had to resort to external consultancy as the best means to resolve their doubts and achieve their objectives. Consultancy services are usually expensive and not all libraries can afford them, which presents another barrier to be overcome, either by sharing resources (e.g. the ALA case), or finding some other source of help or advice, such as consultation with other libraries using the BSC, support from within the library, or guidance provided by academic staff. In this context, we should also mention the role for specialized software in providing information and support for adopting the BSC, especially for collecting and analysing the performance data. However, although the use of an automated tool is a common recommendation made by professionals with experience in BSC implementation, only two of the respondent libraries were using specialist software. One reason may be the scant expertise of the libraries in using the BSC because, according to the advice offered by Matthews (2008: 73), automating the process should not be considered ‘until it has produced four or more scorecards’.

Another important issue in adopting the BSC framework is the need to provide library staff with the necessary education and training to work with it and make the most of its potential. Almost all the surveyed libraries confirmed that their staff received special training, either in short workshops or in intensive courses, to use the BSC.

To summarize, the evidence suggests that the adoption of the BSC system is a costly matter that may be unaffordable for small or medium-sized libraries, which may have more limited resources. Such libraries may also have less need for complex administration systems, so they may not consider the BSC as an appropriate tool for their management. The testimony of one of the surveyed libraries is very revealing:

When the Library originally implemented the BSC there was a significant investment in both staff time and IT system set up. Over time the IT and system support costs started to impact. For small organizations such as the Library the $$ investment in terms of more sophisticated data warehouse type technology has not proven successful or affordable. We have been required to modify some elements e.g. we no longer undertake regular evaluation of the cost of services but do report on key revenue and expenditure lines. We have developed and modified our own reporting system using Excel. There is still a considerable investment in staff time required to analyse the results, despite automatic calculation of data. (National Library)

The impact on smaller libraries of the resources required to implement and operate the BSC has been noted in some other types of organization: for example, in research to test the usage of the BSC in the most important publicly traded firms in Germany, Austria and Switzerland, Speckbacher et al. (2003: 381) found ‘a significant association of size (measured as the number of employees) and BSC usage; larger companies are more likely to have implemented a BSC’.

In addition to the supplies and support required for deploying the BSC system, our survey confirms the hypothesis from the literature that the most challenging issue in its implementation is selecting the KPIs, as stated in the analysis of the results. It is generally assumed that developing meaningful performance measures (metrics) and the expected levels of performance (targets) is difficult. Although the methodological process of building a BSC is clearly established and the development of the performance measures should be derived from the previous steps (assessment of the organization’s foundations; development of overall strategy, decomposition of strategy into objectives and creation of the strategic map), in practice, ‘the BSC approach has no specific mechanism in order to select metrics and targets or peculiar procedures/tools to test and validate such choices’ (Barnabé, 2011: 453). This difficulty is accompanied by the responsibility of knowing that selecting the right measures is possibly the most important factor for success in using the BSC, since measuring the wrong aspects would mean wasting the library’s resources and dynamism in activities that do not contribute to its success.

In that context, it is very important to know the main criteria used by libraries to select their KPIs. The criterion on which all the libraries agree is that KPIs should be based on the priorities of the library’s strategic plan – as Ceynowa (2000: 158) stated: ‘The suitability of an indicator for inclusion in the set depends primarily on its importance to strategy’. That choice would confirm that the library has implemented the BSC, not just as a framework to provide a series of performance measures that describe their assets, but as a ‘tool that can be used to describe, implement and measure an organization’s strategy’ (Matthews, 2008: 57). Other criteria identified as important in our survey were familiarity and ease of collection, although these are tendencies that need to be resisted (Matthews, 2008). More significant is the finding that their use by benchmarking partners was one of the principal reasons for selecting KPIs, despite the fact that participating in benchmarking projects was neither an important reason for libraries to decide to use the BSC nor a key factor in developing it. This indicates the importance afforded by libraries involved in joint performance measurement projects of interpreting their results relative to those of other libraries, rather than independently. However, there is also a need for each library to develop its own customized BSC programme, and, more specifically, a requirement that the selected KPIs reflect the success, or otherwise, of a particular strategy. The warning is very clear: ‘many companies use metrics that are not applicable to their own situation. It is vitally important when using BSC to make the information being tracked applicable to your needs. Otherwise, the metrics will be meaningless’ (Kootanaee et al., 2013: 52).

Poll, a promoter of the use of the BSC in benchmarking programmes, stated for academic libraries that, since their mission was in many aspects identical, it was possible to use a standardized indicators system structured in the perspectives of the BSC as a reference model for benchmarking purposes, and that the individual variations in libraries could ‘be expressed by different target values and operational actions’ (Poll, 2001b: 715). That perception led to the adoption of the BSC to structure the performance indicators included in the third edition of the international standard ISO11620, published in 2008 (ISO, 2008). According to Poll (2005: 65), ‘using such standardized methods does not only allow benchmarking but will give the individual library more confidence in its measuring process and will add reliability to funding institutions’. In that regard, one would expect libraries to consider inclusion in a standard as an important reference for configuring their performance measurement system. However, none of the surveyed libraries referred to international standards and only four referred to national standards when choosing their KPIs. The sample is not sufficiently broad to draw wider conclusions, but this suggests that awareness and use of standards in libraries’ performance measurement practices is not as extensive as might have been expected.

Another interesting issue is the number of KPIs used by libraries. The survey results show there is a broad range, from just four to 50. Most authors are in favour of considering a small number of indicators for the BSC, since ‘one of the factors at the basis for BSC success relies on its simplicity, largely ensured by the consideration of a limited number of parameters’ (Barnabé, 2011: 453), while Poll (2009) advocates having not more than 20 indicators. With a limited number of KPIs, the risk is, as stated above, not selecting the right indicators because that could generate the development of an incoherent strategy and its consequent breakdown. On the other hand, there is risk in selecting too many indicators ‘since a wide set of performance metrics may not reflect the organization’s strategy and produce an amount of data which will not be properly analyzed’ (Barnabé 2011: 453). In that context, adopting a performance measurement system including other metrics to be used, at operational level, to supplement the KPIs may be a viable solution. In this regard, one of the surveyed libraries stated that ‘a BSC works best when there are a limited number of measures. The KPIs are limited to processes so we use other performance indicators for public services’.

Our survey allows us to confirm that libraries’ performance metrics are not fully covered by their KPIs, as most also collect and use other indicators and statistics for monitoring their own services or, as argued by one of the libraries, to ‘do comparisons and rankings with peer libraries’, allowing them to meet the needs of benchmarking practice, without conditioning the selection of their KPIs.

The final assessment made by most of the surveyed libraries of their experience with the BSC is very positive, asserting they have benefitted from using it. However, awareness of the BSC impact takes some time and the benefits are not immediately realized in the management of the library. Respondent libraries which had fewer than three years’ experience with BSC declared that it was too early to evaluate its effects. Results from those with more experience of the technique show four broad areas in which libraries place the most positive outcomes: strategic management, operational control, transparency and reporting, and culture of assessment.

The benefits arising from using a BSC for strategic management purposes are those which receive the broadest consensus in the surveyed libraries, including setting strategic priorities for library improvement, organizational alignment of work with the strategic plan, and encouraging communication within the library about strategic goals and expectations. At operational level, the most noteworthy outcomes identified by libraries are that the BSC helps to focus attention on areas of mediocre or low performance which are impeding library success overall, and that it provides a more effective and rational system of performance metrics. A significant proportion of the respondent libraries also agree that growing transparency of information and visibility at all levels of the organization is one of the most important benefits of the BSC, which moreover provides the library staff with an important reporting tool at a corporate strategic level. Finally, the participant libraries note that the adoption of the BSC promotes the development of a culture of assessment, where the importance of metrics and their analysis for performance and quality improvement is recognized and where there is a greater ability to effectively demonstrate a library’s achievement and significance internally and externally.

Although the surveyed libraries were not specifically asked about the disadvantages or pitfalls of using the BSC, some of the comments they made related to such issues. Firstly, it is worth noting that it is difficult to progress from the partial use of the BSC as a performance measurement system to its full development as a holistic strategic management system. In that sense, an observation made by one of the libraries is very revealing. After more than three years using the BSC, they stated that: ‘the library approach is partly presentational – representing indicators in a coherent strategic framework. Much remains to be done to improve data quality, to automate data collection and to link active decision-making to performance information’. Another issue is that the BSC may not be adequate to manage the whole process of library performance and quality improvement and therefore, the use of some other management systems or tools could be necessary. This is the case in one of the surveyed libraries, which advises ‘do not promote the BSC as a stand-alone tool. The library profession needs to build the business model and implement all of the business tools together, they dovetail with each other’. Finally, the difficulty in balancing the multiple measures used in the BSC, noted by another library, highlights one of the potential pitfalls identified by Norreklit et al. (2008: 66) who state in this respect, that ‘the BSC is silent on the balance of significance across the measurements it incorporates, so users may become confused and the system may lead to frustrations’.