Performance indicators framework for assessment of national libraries using the analytic hierarchy process

Abstract

The National Library and Archives of Iran is the most important library in the country, taking on tasks such as providing and protecting the written and unwritten national heritage and organizing the services of other libraries in Iran. An assessment of the National Library and Archives of Iran is critical to understanding how the organization contributes to organizational and strategic goals. This article aims to show the rankings and priorities of performance indicators based on the four perspectives of the balanced scorecard framework (resources, access and infrastructure; use; efficiency; and potentials and development) and with use of the analytic hierarchy process. The perspectives of resources, use, efficiency, and potentials and development were rated. The top performance indicators included: ‘Level of user satisfaction with searching the National Library and Archives of Iran’s website’, ‘Percentage of acquired books in any medium and multimedia resources entered by legal deposit’ and ‘Qualitative evaluation using case studies of use and value of the collection and services’. Pervious literature shows various applications of a joint balanced scorecard and analytic hierarchy process. This study applies this integrated framework throughout an organization in a new service field.

Keywords

National Library and Archives of Iran performance indicators balanced scorecard analytic hierarchy process national library library assessment strategy

Introduction

Today, it is clear that success is not spontaneous and requires planning. The optimal execution of a programme, in turn, requires proper monitoring and control. One of the main obstacles to the success of an organization in the implementation of its plans and strategies is the lack of sufficient, accurate and timely information on the results of the implementation of plans. To do this, organizations need to measure their performance. Performance measurement is one of the first steps in the service-delivery improvement process. It involves the selection, definition and application of performance indicators, which quantify the efficiency and effectiveness of service-delivery methods (Fine and Snyder, 1999).

One purpose of performance indicators is to influence performance. If performance indicators are used seriously and real consequences follow from the judgments based on them, all concerned should be aware that the chosen performance indicators will come to be taken by staff to be representative of the objectives of an organization. Performance indicators must therefore be based on criteria that adequately reflect all of the objectives of an organization (Winkworth, 1990).

Every organization needs to determine and rank performance indicators to evaluate the desirability of its activities, especially in complex and dynamic environments (Adeli, 2005, cited in Tavallaei, 2007: 10). Libraries are at the heart of the social system; they exist to serve the needs of people, to help them live, learn and develop, and to act as part of the social glue that holds communities together (Brophy, 2006). In libraries, as in other organizations, we have seen in recent decades the development and application of performance indicators – in other words, performance measurement – which are often published in the form of handbooks and standards. These include ISO 11620 (Library performance indicators; International Organization for Standardization, 2014a), ISO 28118 (Performance indicators for national libraries; International Organization for Standardization, 2009), ISO 2789 (International library statistics; International Organization for Standardization, 2003), ISO 16439 (Methods and procedures for assessing the impact of libraries; International Organization for Standardization, 2014b) and ISO 21248 (Quality assessment for national libraries; International Organization for Standardization, 2019).

Increasingly, libraries must also be able to demonstrate the value of what they are doing and provide evidence of the impact they are making (Payne and Conyers, 2005). A problem at the comparative and judgement levels is that different institutions may legitimately have set different objectives and policies, or may have given their libraries wider or narrower responsibilities. The ‘family’ of performance indicators must therefore also be capable of dealing with the varying balances of emphasis in the activities of different libraries (Winkworth, 1990).

Performance indicators and measurements pertaining to libraries in Iran have not received adequate attention from researchers or practitioners. Little research has been done in this field and only one study on the balanced scorecard (BSC) framework was found in the National Library of Iran (Nabizadeh-Kivi et al., 2014). Various methods have been suggested over the years for evaluating the performance of libraries.

The BSC, a performance measurement framework that provides an integrated look at the business performance of a company by a set of both financial and non-financial measures, seems to be a good solution (Lee et al., 2008). The main contribution of the BSC is that it includes strategic objectives and performance measures that are not solely financial. The BSC considers four perspectives, where strategic objectives and performance measures are defined as (a) financial, (b) clients, (c) internal processes, and (d) learning and growth. There is a causal relationship among these perspectives: if the ‘learning and growth’ perspective is improved, then the ‘internal processes’ perspective will also be improved. Moreover, there is a positive effect on the ‘clients’ perspective, which will ultimately have an impact on the ‘financial’ perspective (Quezada and López-Ospina, 2014). According some studies, more than 80% of the top 1000 corporations in the world have adopted the BSC, and BSC adoption has expanded in an increasing number of countries over the last decade more than 80% of the top 1000 corporations in the world have adopted the BSC, and BSC adoption has expanded in an increasing number of countries during 2000 to 2013 (Lin et al., 2014). There are several articles that have discussed the application of the BSC in relation to performance indicators in libraries (De la Mano and Creaser, 2016; Hammes, 2011; Huang and Song, 2010; Krarup, 2004; Self, 2004).

Developed by Robert S Kaplan and David P Norton (1992), the BSC is a holistic performance measurement system that will reinforce a library’s strategy and complement its resources. The BSC is a performance measurement framework that links the short-term and long-term activities of an organization with the vision, mission and strategy of the organization through the establishment of measurable, consensus-driven goals. The BSC is particularly applicable to libraries as it:

Clearly links the strategy to the core activities of a library in a structured framework. Rather than focusing on just budget size or informal customer service feedback, the BSC allows library managers to focus on several different indicators in an organized manner.

Has a proven record of being effective and adaptable to all types of organizations, including thousands of non-governmental organizations, government agencies, libraries and business units.

Is easily communicable to staff, partners and the public.

Is consensus-driven, relying on the input of stakeholders that are both internal and external to the organization (Lloyd, n.d.).

Performance indicators at the National Library and Archives of Iran

The National Library and Archives of Iran (NLAI) was officially inaugurated in 1937 but its collection dates back 150 years. It contains large numbers of manuscripts, old printed books, old itineraries from European tourists, documents, rare periodicals and materials in non-print forms. It has grown gradually through donations, exchanges, acquisitions and legal deposits. The library is an educational, research and service institution, aiming to acquire, organize and disseminate information published or produced in Iran or in the fields of Iranian studies and Islamic studies in other countries; to promote research and planning in library and information science; and to provide consultation and devise appropriate methods for all Iranian libraries (Amirkhani, 2002).

The NLAI has always dealt with the issue of performance appraisal, but in order to improve the preparation of reports, it was felt that there was a need for standard performance indicators (International Organization for Standardization, 2015: 7). In 2015, Deputy of research of NLAI considered ‘development of key performance indicators, and improvement and promotion strategies’ among the important and urgent research projects list. (National Library and Archives of Iran, 2015). In the same year, the NLAI, based on IFLA guidelines on the performance evaluation indicators of national libraries according to ISO 11620 (Poll and Jönsson-Adrial, 2006), which was itself a prelude to the publication of ISO 28118 in 2009, introduced 23 performance indicators to evaluate the performance of the Library. However, it should be stated that these indicators have not met the needs of the NLAI, and several meetings have been held at the top and middle management levels of the organization, and the Library has been feeling the need for performance indicators more than ever. But so far, the performance indicators have not been defined based on the goals and tasks of the NLAI.

Because performance indicators are shaped by an organization’s goals and mission, existing ISOs and standards, and even the performance indicators of other national libraries, cannot be prescribed in the same way for every library. In other words, it is necessary to apply ISOs, standards and performance evaluation indicators that are appropriate to the needs of a library and its goals and mission. National libraries, in consultation with their funding institutions and relevant authorities, such as the national government, as well as their users and other stakeholders, need to decide which performance indicators are most appropriate for a particular situation (International Organization for Standardization, 2009: 11).

The NLAI, as a national and parent organization in Iran in the area of libraries, should continuously review its current status through predetermined indicators and standards. By describing the status of the changes it makes, it can examine, analyse and make necessary predictions through strategic and operational planning to address deficiencies and shortcomings in the dimensions of education, research, services, publishing and professional growth. Without determining performance indicators, the existing gaps cannot be identified and removed.

Since the NLAI lacks performance indicators (Azadeh et al., 2013), the present study intends to develop performance indicators for the NLAI based on standard indicators – including ISO 28118 (Performance indicators for national libraries; International Organization for Standardization, 2009), ISO 11620 (Library performance indicators; International Organization for Standardization, 2014a), ISO 2789 (International library statistics; International Organization for Standardization, 2003) and ISO 16439 (Methods and procedures for assessing the impact of libraries; International Organization for Standardization, 2014b) – as well as the performance indicators of national libraries around the world, taking into account the goals and strategies of the NLAI. The performance indicators obtained have been arranged based on the four dimensions of the BSC and then on the analytic hierarchy process (AHP) to rank and determine the weight of the performance indicators, so that the NLAI can evaluate them and obtain a good picture of its performance and, in the not too distant future, as a service organization enter the arena of the Iran Award.¹ The purposes of this research are therefore as follows: first, to justify the rankings and weightings of the performance indicators of the NLAI based on the AHP and, second, to determine the operational pattern of the NLAI’s performance indicators. Accordingly, the following research questions have been raised:

What are the rankings and weightings of the performance indicators of the NLAI based on the AHP?

What is the operational pattern of the functional evaluation indicators of the NLAI?

Literature review

The literature review showed that there has been little research conducted in the field of the performance indicators of national libraries. Of course, it should be noted that some prominent national libraries have performance evaluation indicators and carry out their annual performance evaluation according to these indicators. It was considered that examining the indicators of other national libraries could develop the vision of compiling performance indicators for the NLAI and be insightful. Therefore, the indicators related to eight important national libraries – Library of Congress, National Library of Spain, National Diet Library, Swiss National Library, National Library of France, National Library of Australia, Library and Archives Canada, and British Library – were received through email requests to the relevant expert of each of these libraries in May 2017. This group of libraries includes a rich mix of countries and continents, from Japan in Asia to Spain, France, Switzerland and Great Britain in Europe, and Canada and the USA in North America. With different geographical areas, high economic levels and a diversity of cultural backgrounds, it can be said that the study of these areas covers the entire world. After collecting the performance indicators of these national libraries, it was found that the Library of Congress does not currently have codified indicators and it is in the process of compiling such indicators. Table 1 presents a list of the performance indicators of the national libraries, along with their goals.

Table 1.

Performance indicators of national libraries worldwide.²

Library	Number of goals	Number of performance indicators	Period	Explanation of goals
National Library of Australia	3	14	2017–2020	Build the nation’s memory Make access happen Lead, partner, connect and excel (National Library of Australia, n.d.)
Library and Archives Canada	5	8	2017–2018	Development of disposition authorizations Collaboration in the management of government records Acquisition and processing of documentary heritage Preservation of documentary heritage Access to documentary heritage(Library and Archives Canada, 2018)
National Diet Library	3	101	2017–2020	Fundamental role: assisting Diet activities Basic role: gathering and preserving materials information Basic role: providing use of information resources (National Diet Library, 2018)
British Library	6	21	2015–2018	Custodianship Research Business Culture Learning International (British Library, 2018)
Swiss National Library	8	29	2015–2017	Uploading collections online Using digital collections (primary platforms) Using digital collections (secondary platforms) Reproductions Loan and users Advice Exhibitions, events Use of catalogues and the BN (Bibliothèque nationale suisse) website (Swiss National Library, 2018)
National Library of France	4	40	2016–2021	Renew relationship with the public Guarantee the continuity of physical and digital collections and facilitate their accessibility Produce and share content and services Adopt exemplary and responsible management, looking to the future (National Library of France, n.d.)
National Library of Spain	4	162	2016–2017	Evaluation of the collections Evaluation of library services: use and response timesCultural diffusion Potential and development (National Library of Spain, 2018)

According to the literature, some research on performance indicators in national libraries (e.g. Baba and Shukor, 2003) has discussed the need for national libraries to evaluate their performance and measure their effectiveness, and states that although performance indicators for academic and public libraries are well developed and used, little has been done in national libraries. Ambrožič et al. (2003) examined performance evaluation in European national libraries. They suggest that the performance indicators that are most important for national libraries include the following: correct answer fill rate, user satisfaction with library services in general or separately, speed of interlibrary loans, automated systems availability, title catalogue search success rate and subject catalogue search success rate. Poll (2008), in an article entitled ‘Quality indicators for national libraries: The new standard’, introduces ISO 28118. ISO 28118 (Performance indicators for national libraries; (2009)) describes 30 performance indicators that are appropriate for the tasks of national libraries. These indicators have either been adapted from ISO 11620 (2014a) or chosen from quality measures that have been used or tested by national libraries and cover the topics that are not considered in ISO 11620.

Also, other research studies the use of the BSC in national libraries. For example, Krarup (2004), in ‘Balanced scorecard at the Royal Library, Copenhagen’, states that, in 2000, the Royal Library started using the BSC as an instrument to create a better connection between its day-to-day activities and strategic goals. The visions of the Royal Library became 10 strategic goals. In 2002, these strategic goals were expressed in the four perspectives of the BSC (internal dimension, user dimension, potential dimension, economic dimension) and performance indicators were defined for each perspective (42 in total). In 2003, the Library defined 25 performance indicators to achieve its strategic goals.

Poll (2007), for the purpose of comparison, classified the indicators used in six projects – BIX – The Library Index (treated as two separate projects), the Council of Australian State Libraries, the Swedish Quality Handbook, The UK Higher Education Library Management Statistics (HELMS), and Benchmarking of the University Libraries Netherlands (UKB) – according to the four perspectives of the BSC, with further subdivisions for resources and services. In addition, they were compared to ISO 11620 to see whether standardized measures had been used. Poll (2007) states that there are 54 indicators. However, 24 of the 54 indicators are more or less identical to those of ISO 11620. The perspective of ‘resources/infrastructure’ has 16 indicators; ‘usage’, 16 indicators; ‘efficiency’, 14 indicators; and ‘development/potentials’, 8 indicators. No indicators for the impact or outcome of libraries on users or society have as yet been used in the projects. Such indicators are still in the testing phase.

Hunter (2009) describes how the National Library of Scotland has developed and improved the management of performance information and the cultural, management and implementation issues involved. At the same time, the Library has sought to align key performance indicators with the political agenda of the Scottish government, and Hunter’s article looks to future developments, particularly in developing outcome indicators that meet both the government’s and the National Library of Scotland’s needs. In this article, 16 key performance indicators were identified in the above four perspectives Developing the Collections, Developing the Organization, Building Relationships, and Widening Access to Knowledge.

Kim and Cha (2021) developed performance indicators as a rational tool to improve the management of the National Library of Korea for its continuing growth and development. The performance indicators are based on the BSC, which is useful for linking the organization’s strategies in response to changes in the library environment with performance measurement. As a result, four BSC-based perspectives (resources, access and infrastructure; usage; efficiency; and potential and growth), 19 key success factors and 71 performance indicators were developed for the National Library of Korea.

Reviewing the articles related to the present study, it can be said that most of the activities performed outside Iran have been related to determining and identifying performance indicators. Consequently, national, academic and school libraries have designed indicators to achieve their goals and continuously evaluated their libraries based on them. However, according to the research that has been done in this field, many countries, including Oman (Zahran Al Hijji and Cox, 2012), Vietnam (Kim Thi Ninh et al., 2010), Estonia (Nuut, 2002), China (Huang and Song, 2010) and Kenya (Odera-Kwach and Ngulube, 2011), have announced that they are at the beginning of the path of evaluating the performance of libraries, and more research is needed in this area. In Iran, research on this subject has been published in relation to academic, public and special libraries (Hariri and Sarafpour, 2015; Ketabi et al., 2011; Moazzen et al., 2017; Zaboli et al., 2015).

The NLAI has set 23 performance indicators in 10 main areas: building the national collection; building the foreign collection; making services accessible: cataloguing; making services accessible: digitization; offering reference services; preserving the collection; possibility of development; international cooperation; management; and Cataloguing in Publication (Office of Planning and Development, n.d.). In the present study, the performance indicators proposed by the NLAI have been used (which have been referred to in this article).

The literature review in Iran also showed that most of the research has been done using researcher-made questionnaires to evaluate the performance of libraries. In the field of performance indicators, in addition to translations of some library-related ISOs by the Iran National Standards Organization, NLAI and researchers, steps have been taken to develop and determine performance indicators for libraries. However, perhaps it can be said that this issue in Iran, like the other countries mentioned above, is in its early stages in libraries, and further research is needed in this field.

Methods

This study is applied in terms of its objectives because it focuses on a methodology that is used to solve a problem in the real world and the results can be applied in real decision-making situations. The statistical population includes heads of the National Library deputy and Research, and digital resources deputy along with high and middle managers as well as deputy directors.

The criteria for selecting the organizational experts and specialists were having at least two years of experience in the NLAI; being familiar with the goals and duties of the NLAI; and having at least a Master’s degree (Table 2).

Table 2.

Demographic characteristics of the statistical population.

Demographic characteristics		%
Sex	Female	53.8
	Male	46.2
Age	30–40	15.4
	40–50	61.5
	50+	23.1
Education	Master’s	53.8
	PhD	46.2
Work experience (years)	< 10	15.4
	10–20	38.5
	20+	46.2
Organizational post	Deputy head	15.4
	General manager	46.2
	Deputy general manager	30.8
	Boss	7.7

A total of 13 organizational experts participated in this research, with all of them having extensive work experience and a higher education degree, working in high-level management positions in the NLAI, and having valuable experience in the field of this research.

In this research, the four perspectives of the BSC used in ISO 21248 (Quality assessment for national libraries; International Organization for Standardization, 2019) – resources, access and infrastructure; use; efficiency; and potentials and development – were taken as the framework. The performance indicators obtained from the research by Hassanzadeh et al. (2022) were arranged in the form of the BSC perspectives. Next, a pairwise comparison questionnaire was designed based on the hierarchical structure of the BSC. Finally, the AHP was applied to obtain the weights of the performance indicators and rank the performance indicators of the NLAI.

In Hassanzadeh et al.’s (2022) study, 32 performance indicators were identified from eight prestigious national libraries (Library of Congress, National Library of Spain, National Diet Library, Swiss National Library, National Library of France, National Library of Australia, Library and Archives Canada, and British Library) and checked against ISO 21248 (International Organization for Standardization, 2019) and ISO 28118 (International Organization for Standardization, 2009) (see Table 3).

Table 3.

Performance indicators of the NLAI.³

Number	Performance indicators	Supporting references
1.	Percentage of national publications acquired by the NLAI	ISO 28118 (2009)ISO 21248 (2019)Office of Planning and Development (n.d.)
2.	Percentage of acquired books in any medium and multimedia resources entered by legal deposit	National Library of Spain (2018)
3.	Percentage of acquired serials entered by legal deposit	National Library of Spain (2018)
4.	Coverage of the national imprint in the new entries of the national bibliography	ISO 28118 (2009)ISO 21248 (2019) National Library of Spain (2018)Office of Planning and Development (n.d.)
5.	Percentage of national publications with Cataloguing in Publication	Office of Planning and Development (n.d.)
6.	Median time to catalogue books entered by legal deposit (in weeks)	National Library of France (n.d.)
7.	Median Time of Document Processing	ISO 21248 (2019)
8.	Number of searches for ‘national bibliography offering service’ of NLAI-Online Public Access Catalogue (OPAC)	National Diet Library (2018)
9.	Qualitative evaluation using case studies of use and value of the collection and services	National Library of Australia (n.d.)
10.	Satisfaction with ease of finding information on the NLAI’s website	British Library (2018)
11.	Satisfaction level of NLAI-OPAC user questionnaire	National Diet Library (2018)
12.	Overall satisfaction level of user service (user questionnaire)	National Diet Library (2018)
13.	Level of user satisfaction with searching the NLAI’s website	National Diet Library (2018)
14.	Number of active users of the NLAI	National Library of Spain (2018) Swiss National Library (2018)
15.	Number of titles on NLAI Press Online	Swiss National Library (2018)
16.	Increase percentage of digitized titles and pages per 1000 documents in the collection	ISO 28118 (2009)ISO 21248 (2019) National Library of Spain (2018)Office of Planning and Development (n.d.)
17.	Cumulative percentage of at-risk audiovisual material migrated from an obsolete to a current digital file format	Library and Archives Canada (2018)
18.	Number of views of digitized books and magazines	National Library of France (n.d.)
19.	Number of views of digitized press	National Library of France (n.d.)
20.	Rate of providing document reference services	National Diet Library (2018)
21.	Percentage of information requests submitted electronically	Office of Planning and Development (n.d.)
22.	Percentage of correct reference questions answered electronically	Researcher
23.	Number of users to whom direct bibliographic information has been provided in reading rooms with a maximum waiting time of 10 minutes and a percentage of the total	National Library of Spain (2018)
24.	Number of users to whom general information has been provided with a maximum waiting time of 10 minutes and a percentage of the total	National Library of Spain (2018)
25.	Percentage of library staff involved in resource processing	Office of Planning and Development (n.d.)
26.	Percentage of processing staff involved in Cataloguing in Publication	Office of Planning and Development (n.d.)
27.	Percentage of staff in national and international cooperation and projects	British Library (2018)ISO 28118 (2009)ISO 21248 (2019) National Library of Spain (2018)Office of Planning and Development (n.d.)
28.	Total square metres of surface in deposits of bibliographic funds with control of air conditioning and percentage	National Library of Spain (2018)
29.	Percentage of storage space which has an appropriate environment	ISO 28118 (2009)ISO 21248 (2019)Office of Planning and Development (n.d.)
30.	Employee productivity in document processing	ISO 28118 (2009)ISO 21248 (2019)Office of Planning and Development (n.d.)
31.	Employee productivity in Cataloguing in Publication	Office of Planning and Development (n.d.)
32.	Employee productivity in lending and delivery services	ISO 28118 (2009)ISO 21248 (2019)Office of Planning and Development (n.d.)

Source: Hassanzadeh et al. (2022).

The 32 indicators were categorized into the four dimensions of the BSC used in ISO 21248 (International Organization for Standardization, 2019: 133–134):

Resources, access and infrastructure: indicators that measure the comprehensiveness, integrity and availability of library resources and services (e.g. collections and catalogues).

Use: indicators that measure the usage of library resources and services (e.g. loans, access and facilities).

Efficiency: indicators that measure resource and service efficiency (e.g. costs per loan or title catalogued, time required to lend or process documents, and correct answer fill rate).

Potentials and development: indicators that measure the library’s input in training and emerging service areas and its ability to gain sufficient funding for development (e.g. attendance at training sessions by staff or staff in cooperation and projects).

Several studies have proven that the BSC framework (see introduction) is effective in helping organizations to evaluate performance. However, the BSC, a tool that describes the perspectives of performance, does not determine perspective and indicator weights (Noori, 2015). As a valuable tool for prioritizing and consolidating performance metrics based on multiple criteria, the AHP is a promising mechanism to help overcome the limitations of the BSC (Álvarez Pérez et al., 2017). It has been adopted in several cases as a method for calculating weights in an evaluation performance system (Lee et al., 2008; Quezada and López-Ospina, 2014; Wang and Xia, 2009; Yaghooi and Haddadi, 2016).

The AHP was first introduced by Saaty in 1971 to solve the problem of the scarce resources allocation and planning needs of the military (Saaty, 1980, cited in Wang and Xia, 2009: 2244). Since its introduction, the AHP has been studied extensively and used in almost all the applications related with Multi Attribute Decision Making (MADM) over the last 30 years (Yaghoobi and Haddadi, 2016). It has also been used to solve unstructured problems in different areas of human needs and interests, such as political, economic, social and management sciences (Wang and Xia, 2009). The AHP is a useful approach for solving complex problems. In this approach, selected stakeholders prioritize the relative importance of criteria by making pairwise comparisons of the factors using a 9-point scale. The most creative part of the decision-making is modelling the problem. The identification of the decision hierarchy is the key to success in using the AHP. The process of building a hierarchy structure is to identify more accurately all of the elements of the decision and recognize the interrelationships between them (Álvarez Pérez et al., 2017). In this method, criteria and indicators are compared with each other, and the degree of importance for each criterion or indicator with respect to each other is specified (Haddadi and Yaghoobi, 2014). Saaty (1990) proposed a fundamental scale of use in making the comparisons. It consists of verbal judgements ranging from equal to extreme (equal, moderately more, strongly more, very strongly more, extremely more); corresponding to the verbal judgements are numerical judgements (1, 3, 5, 7, 9) and compromises between these values (see Table 4).

Table 4.

Scale for making expert judgements.

Intensity of importance on an absolute scale	Definition	Explanation
1	Equally important	Two activities contribute equally to the objective
3	Moderate importance of one over another	Experience and judgement strongly favour one activity over another
5	Essential or strong importance	Experience and judgement strongly favour one activity over another
7	Very strong importance	An activity is strongly favoured and its dominance demonstrated in practice
9	Extreme importance	The evidence favouring one activity over another is of the highest possible order of affirmation
2, 4, 6, 8	Intermediate value between the two adjacent judgements	When compromise is needed
Reciprocals	If activity i has one of the above numbers assigned to it when compared with activity j, then activity i has the reciprocal value when compared with activity j

Source: Saaty (1990).

Saaty (1980) argued that the 9-point scale (Table 4) represents mangers’ ability to make qualitative distinctions (cited in Yaghoobi and Haddadi, 2016: 964). If an alternative A_i is preferable to an alternative A_j , then the value of the comparison scale P_c (A_i , A_j ) = $a_{i j}$ indicates the intensity of relative importance of A_i over A_j , assigned by the decision maker. The scale allows, in a pairwise comparison, the investigator to establish which alternative is better. Higher values of $a_{i j}$ indicate a stronger preference for alternative A_i over A_j . Finally, the matrix A is the result of all the comparisons and represents the relative importance $a_{i j}$ of each element. Given that an element is as important as itself, and taking into account the theorem of reciprocity, then $a_{i j}$ = 1 if i = j and $a_{i j}$ = 1/ $a_{i j}$ if i ≠ j. The calculation of weights relies on an iterative process in which matrix A is successively multiplied by itself, resulting in normalized weights w_i , and these weights represent the importance of alternative A_i relative to all other alternatives (Álvarez Pérez et al., 2017).

The application of the AHP to ranking the performance indicators of the NLAI involved two steps. The first was to structure the decision problem into a hierarchy. According to Saaty’s (1990). model, in the first step, a hierarchical framework is drawn according to the indicators at three levels: objectives, criteria and subcriteria. Accordingly, the goal – the performance indicators in the NLAI based on the BSC – was given the highest priority. In the second category were the criteria of resources, access and infrastructure with 11 subcriteria; use with 8 subcriteria; efficiency with 10 subcriteria; and potentials and development with 3 subcriteria (see Figure 1).

Figure 1.

Hierarchical framework of BSC performance indicators for NLAI.

The second step involved making pairwise comparisons and obtaining a judgemental matrix. For prioritizing, a questionnaire (Appendix 1) was designed to ascertain the judgments of the experts, with a conventional AHP questionnaire format, 9-point scale and pairwise comparison. The questionnaire included four parts: demographic information about the experts from the NLAI; the definition and meaning of each criterion; a comparison of the four BSC perspectives relating to the NLAI; and a comparison of the performance indicators relating to each perspective. On 5 February 2020, the pairwise comparison questionnaire was distributed among the 13 NLAI experts and, on 29 February 2020, 12 completed questionnaires were collected (one was discarded due to it being incomplete).

This study focused on geometric mean of organizational expert’s views, using Excel software. It was calculated using the following method:

μ_{g} = {(\prod_{i = 1}^{n} a_{j})}^{1 / n} = \sqrt[n]{a_{1}, a_{2} \dots a_{n}}

Here, a is the opinions of the experts and n is the number of experts.

The geometric mean is used to combine individual comparisons with group comparisons and form paired comparison matrixes. Since paired comparisons create ratio data, the geometric mean is the best for them. Moreover, the inverse matrix of comparison justifies the use of this mean more than anything else (Azar and Memariani, 1995). In order to rank the performance indicators, in this step, the local weight obtained from the previous step was multiplied by the local weight of the corresponding dimension to obtain the final weight.

The credibility of the information was determined based on the inconsistency rate for the paired comparisons. If the ratio (called the consistency ratio CR) of consistency index (CI) to that from the matrices is significantly small (carefully specified to be about 10% or less), then the degree of consistency is satisfactory (Saaty, 1990).

Results and discussion

In order to answer the research questions, it is necessary to re-examine them here.

What are the rankings and weightings of the performance indicators of the NLAI based on the AHP?

Thirty-two performance indicators according to four criteria were identified and weighted by the organization experts and prioritized through the AHP. The weight of each criterion determines the importance of the criterion in relation to the other criteria. Table 5 shows the raw and normalized weights of the 4 × 4 matrix. The results of the four BSC dimensions for the NLAI show that the use perspective, with a priority weight of 0.881, is the most important perspective. This is followed by resources, access and infrastructure with a priority weight of 0.871. The priority weight of the efficiency perspective is 0.704 and potentials and development has the lowest priority weight of 0.653.

Table 5.

Ranking of the four BSC perspectives for the NLAI.

Perspective	Resources, Access, and Infrastructure	Use	Efficiency	Potentials and Development	Weight
Resources, Access, and Infrastructure	1	1.122	1.078	1.475	0.871
Use	0.892	1	1.059	2.023	0.881
Efficiency	0.927	0.945	1	0.7	0.704
Potentials and Development	0.678	0.494	1.428	1	0.653

In Figure 2, the dimensions of the BSC are specified in order of importance. It shows the order of the main criteria according to their weight (also shown in Table 5).

Figure 2.

Prioritization of BSC perspectives for NLAI by the analytical hierarchy process (AHP).

Following the necessary mathematical calculations to obtain the eigenvector of each matrix, for the resources, access and infrastructure perspective, Performance Indicator 2 (Percentage of acquired books in any medium and multimedia resources entered by legal deposit), Performance Indicator 1 (Percentage of national publications acquired by the NLAI) and Performance Indicator 3 (Percentage of acquired serials entered by legal deposit) occupied the top positions, with local weights of 0.845, 0.811 and 0.647, respectively (see Table 6).

Table 6.

The pairwise comparison matrix of checklists in Resources perspective.

	PIs 1	PIs 2	PIs 3	PIs 4	PIs 5	PIs 8	PIs 15	PIs 16	PIs 17	PIs 28	PIs 29	LW
PIs 1	1	0.831	1.411	1.881	1.332	3.216	2.565	1.327	0.931	1.193	1.235	0.811
PIs 2	1.203	1	1.791	1.348	1.568	1.704	2.373	1.796	1.557	1.213	1.261	0.845
PIs 3	0.709	0.558	1	1.098	1.248	1.165	1.489	1.260	2.895	1.156	1.030	0.647
PIs 4	0.532	0.742	0.911	1	1.575	1.428	0.921	0.869	1.339	1.751	1.372	0.632
PIs 5	0.751	0.638	0.801	0.635	1	1.396	1.403	1.307	1.333	1.057	0.816	0.549
PIs 8	0.311	0.587	0.859	0.700	0.717	1	0.854	0.686	0.811	0.673	0.651	0.392
PIs 15	0.390	0.421	0.672	1.086	0.713	1.171	1	0.748	0.721	1.003	1.305	0.462
PIs 16	0.753	0.557	0.794	1.150	0.765	1.458	1.337	1	1.084	1.793	1.215	0.592
PIs 17	1.074	0.642	0.345	0.747	0.750	1.233	1.387	0.922	1	1.104	0.869	0.512
PIs 28	0.838	0.825	0.865	0.571	0.946	1.485	0.997	0.558	0.906	1	1.242	0.531
PIs 29	0.810	0.793	0.971	0.729	1.226	1.536	0.767	0.823	1.152	0.805	1	0.544

For the use perspective, Performance Indicator 13 (Level of user satisfaction with searching the NLAI’s website), Performance Indicator 9 (Qualitative evaluation using case studies of use and value of the collection and services) and Performance Indicator 10 (Satisfaction with ease of finding information on the NLAI’s website) were the top priority, with local weights of 0.969, 0.826 and 0.639, respectively (see Table 7).

Table 7.

The pairwise comparison matrix of checklists in Use perspective.

	PIs 9	PIs 10	PIs 11	PIs 12	PIs 13	PIs 14	PIs 18	PIs 19	LW
PIs 9	1	1.153	1.477	1.980	0.747	2.764	1.688	1.897	0.826
PIs 10	0.867	1	0.970	1.075	0.573	2.187	1.358	1.889	0.639
PIs 11	0.677	1.031	1	0.701	0.722	1.984	1.153	1.426	0.577
PIs 12	0.505	0.930	1.427	1	0.619	1.762	1.153	1.350	0.583
PIs 13	1.339	1.744	1.384	1.616	1	3.102	2.291	2.291	0.969
PIs 14	0.362	0.457	0.504	0.568	0.322	1	1.042	0.997	0.337
PIs 18	0.592	0.736	0.867	0.867	0.436	0.959	1	1.454	0.463
PIs 19	0.527	0.529	0.701	0.741	0.436	1.003	0.688	1	0.380

For the efficiency perspective, Performance Indicator 30 (Employee productivity in document processing), Performance Indicator 32 (Employee productivity in lending and delivery services) and Performance Indicator 20 (Rate of providing document reference services) occupied the top positions, with local weights of 0.864, 0.788 and 0.785, respectively (see Table 8).

Table 8.

The pairwise comparison matrix of checklists in Efficiency perspective.

	PIs 6	PIs 7	PIs 20	PIs 21	PIs 22	PIs 23	PIs 24	PIs 30	PIs 31	PIs 32	LW
PIs 6	1	0.659	0.578	1.501	1.610	0.597	0.820	0.523	0.884	0.716	0.479
PIs 7	1.517	1	0.915	1.804	1.912	0.913	0.849	0.690	1.131	0.776	0.632
PIs 20	1.730	1.092	1	0.961	1.461	1.376	1.289	1.377	1.731	1.378	0.785
PIs 21	0.666	0.554	1.040	1	0.529	0.467	0.630	0.350	0.292	0.286	0.346
PIs 22	0.621	0.523	0.684	1.892	1	0.528	0.713	0.555	0.539	0.452	0.401
PIs 23	1.674	1.096	0.727	2.143	1.894	1	0.612	0.550	0.751	0.762	0.6
PIs 24	1.220	1.178	0.776	1.588	1.402	1.633	1	0.756	0.693	0.545	0.601
PIs 30	1.912	1.450	0.726	2.854	1.802	1.819	1.323	1	2.078	1.190	0.864
PIs 31	1.132	0.884	0.578	3.420	1.855	1.331	1.442	0.481	1	0.958	0.662
PIs 32	1.398	1.288	0.726	3.496	2.213	1.312	1.833	0.840	1.044	1	0.788

For the potentials and development perspective, Performance Indicator 25 (Percentage of library staff involved in resource processing), with a local weight of 1, occupied the top spot (see Table 9).

Table 9.

The pairwise comparison matrix of checklists in Potentials and Development perspective.

	PIs 25	PIs 26	PIs 27	LW
PIs 25	1	1.242	3.25	1
PIs 26	0.805	1	1.709	0.712
PIs 27	0.308	0.585	1	0.362

In general, Table 10 shows the order of priority of the performance indicators selected by the organizational experts for all the dimensions and their rank in the relevant dimension, as well as their overall ranking. Based on this, it can be said that, from the point of view of the organizational experts, for the NLAI among the selected performance indicators, The ten most important performance indicators were ‘Level of user satisfaction with searching the NLAI’s website’; ‘Percentage of acquired books in any medium and multimedia resources entered by legal deposit’; ‘Qualitative evaluation using case studies of use and value of the collection and services’; ‘Percentage of national publications acquired by the NLAI’; ‘Percentage of library staff involved in resource processing’; ‘Employee productivity in document processing’; ‘Percentage of acquired serials entered by legal deposit’; ‘Satisfaction with ease of finding information on the NLAI’s website’; ‘Employee productivity in lending and delivery services’; and ‘Rate of providing document reference services’.

The consistency property of the matrix was then checked to ensure the consistency of the judgements in the pairwise comparison. In general, if the consistency ratio is less than 0.1, the comparisons are acceptable. Otherwise, the original values in the matrix must be revised by the evaluator or it could affect the overall results negatively. In this research, the consistency ratio of the comparison between the perspectives of the BSC (0.04) was calculated, and the consistency ratios for the perspectives of resources, access and infrastructure (0.022), use (0.008), efficiency (0.026), and potentials and development (0.018), which are less than 0.1, indicating the high accuracy of the pairwise comparison (see Table 10).

Table 10.

Local and overall weights and ranks of the perspectives and ranks of the personal indicators of the NLAI.

Perspective	PerformanceIndicator	Local weight	Rank of performanceindicator in perspective	Weight ofperspective	Rank ofperspective	Overallweight	Consistency ratio of perspective	Consistencyratio	Overall rank ofperformance indicator
Resources, Access, and Infrastructure	PIs 1	0.811	2	0.871	2	0.706	0.022	0.04	4
	PIs 2	0.845	1			0.736			2
	PIs 3	0.647	3			0.564			7
	PIs 4	0.632	4			0.55			11
	PIs 5	0.549	6			0.478			15
	PIs 8	0.392	11			0.341			26
	PIs 15	0.462	10			0.402			25
	PIs 16	0.592	5			0.516			12
	PIs 17	0.512	9			0.446			20
	PIs 28	0.531	8			0.463			19
	PIs 29	0.544	7			0.474			16
Use	PIs 9	0.826	2	0.881	1	0.728	0.008		3
	PIs 10	0.639	3			0.563			8
	PIs 11	0.577	5			0.508			14
	PIs 12	0.583	4			0.513			13
	PIs 13	0.969	1			0.854			1
	PIs 14	0.337	8			0.297			29
	PIs 18	0.463	6			0.408			24
	PIs 19	0.38	7			0.335			28
Efficiency	PIs 6	0.479	8	0.704	3	0.337	0.026		27
	PIs 7	0.632	5			0.445			21
	PIs 20	0.785	3			0.553			10
	PIs 21	0.346	10			0.243			31
	PIs 22	0.401	9			0.282			30
	PIs 23	0.6	7			0.422			23
	PIs 24	0.601	6			0.423			22
	PIs 30	0.864	1			0.608			6
	PIs 31	0.662	4			0.466			17
	PIs 32	0.788	2			0.555			9
Potentials and Development	PIs 25	1	1	0.653	4	0.653	0.018		5
	PIs 26	0.712	2			0.465			18
	PIs 27	0.362	3			0.237			32

What is the operational pattern of the functional evaluation indicators of the NLAI?

Figure 3 shows the operational pattern of the performance indicators of the NLAI resulting from the calculations. In this model, the weight of the dimensions and criteria for ranking the performance indicators of the NLAI are specified. The model is designed as a decision-making model for the NLAI.

Figure 3.

Operational model of the selected performance indicators of the NLAI based on the dimensions of the BSC (in order of preference).

The national library of any country is a practical model for other libraries in that country, and every library and librarian tries to present the activities in their library by imitating the work done in the national library. Thus, evaluating the performance of national libraries is essential to ensure that they move to improve the delivery and promotion of their services. An effective tool to evaluate the performance of an organization is the BSC. In this study, the performance indicators have been arranged according to the four perspectives of the BSC. The National Library of Scotland (Hunter, 2009), the Royal Library in Copenhagen (Krarup, 2004), ISO 21248 (International Organization for Standardization, 2019) and ISO 11620 (International Organization for Standardization, 2014a) have also ranked their performance indicators According to the BSC.

According to the research findings (Table 5), among all the dimensions of the BSC, use is at the top. This result is similar to the research conducted by Melo and Pires (2008), where, among the results criteria, the client perspective had the highest score. Libraries, like other educational, service and non-profit institutions, tend to define quality in terms of the richness of the resources in their collection, the number of volumes available, the number of staff and other such scales. However, according to Kovel-Jarboe (1996), in a conscious environment, quality is defined based on customer and user satisfaction (cited in Latifi and Mirhosseini, 2017: 180). Thus, users are an important pillar and when we accept that all the efforts of information centres are aimed at providing information services to information seekers, the importance of these services becomes clearer (Horri, 2001). In other words, the existential philosophy of the library is to serve the community, utilizing as many human resources as possible for the excellence of all human beings. The most important perspective in libraries is not the financial perspective but the user perspective, as libraries do not strive for maximum gains but to provide the best services (Poll, 2009). Thus, each national library should be a model for other libraries in the country and therefore try to provide user-friendly services as far as possible.

Also, the findings in Table 5 show that the resources, access and infrastructure perspective, with a slight difference, is in second place. Considering that, in this dimension, the indicators related to the comprehensiveness, integrity and availability of library resources and services are evaluated, the results of this study show that the importance of the national collection and the national bibliography in the NLAI is quite clear. However, it can be said that all the processes of collecting, organizing and maintaining in the NLAI end up with the user.

According to Table 6, the indicators in the resources, access and infrastructure perspective that are ranked first to third are ‘Percentage of acquired books in any medium and multimedia resources entered by legal deposit’, ‘Percentage of national publications acquired by the NLAI’ and ‘Percentage of acquired serials entered by legal deposit’. According to the results, it can be said that apart from indicators related to the issues of cataloguing the national bibliography, collection digitization, preservation and maintenance, the top three indicators in this perspective are related to the provision of the NLAI collections. What should be considered in these results is that, for the NLAI, the provision of books is more important than other information media.

For the use perspective, the indicators ‘Level of user satisfaction with searching the NLAI’s website’, ‘Qualitative evaluation using case studies of use and value of the collection and services’ and ‘Satisfaction with ease of finding information on the NLAI’s website’ are ranked first to third. The results for this dimension show that two of the top indicators are related to the NLAI’s website. Today, websites are one of the main tools for accessing information in libraries, especially national libraries. Given the important position of this technology in the field of information, one can expect a lot of information to be available on websites; Danesh and Soheili (2009) consider them to be the richest information treasuries in the world. The NLAI should have a more serious look at its website so that it can be present in the national and international arenas with more and better facilities, regardless of time and place restrictions. Undoubtedly, the NLAI, with its rich website, can meet many of the needs of its users at home and abroad.

For the efficiency perspective, the indicators ‘Employee productivity in document processing’, ‘Employee productivity in lending and delivery services’ and ‘Rate of providing document reference services’ are ranked first to third. All of the indicators mentioned are among the important basic tasks of any library, and it is clear that evaluating their performance provides a more comprehensive picture of evaluating the effectiveness of the services of the NLAI. The issue that needs to be addressed here is that the selected indicators in this perspective do not examine the lending and delivery services from the perspective of users, while in most of the libraries surveyed, including the national libraries of Spain, Switzerland and Japan, this issue is assessed from the perspective of users. Also, at the Royal Library (Krarup, 2004), the ‘Loans/e-usage’ indicator has been monitored in the user dimension. In Poll’s research (2007), the indicators of “Loans per capita”, ”Loans per user”, “Loans in the past year/acquisitions over the past five years” and ”Proportion of ILL loans (Interlibrary Loan) to total loans” were focused on. Melo and Pires’ (2008) study addressed “Loans per capita”. Odera-Kwach and Ngulube research (2011) also dealt with “Loans per capita”. And last but not least Hariri and Sarafpour’s research (2015) focused on “Per capita deposits per member” and “Percentage of deposits to foreign users”. This all shows that access to a national library’s collection is important for all users (individuals or other libraries), and therefore one important goal of a national library is to loan so that to user can have a fast and effective access to its collection. This issue has been overlooked by the organizational experts. Accordingly, it is suggested that the NLAI pursue the issue of lending from a user perspective, as well as the issue of interlibrary loans.

For the potentials and development perspective, among the three indicators selected by the organizational experts, ‘Percentage of library staff involved in resource processing’ is ranked first. The NLAI, as a model library, must show that it organizes efficient processing. Therefore, the percentage of staff allocated for this purpose is important.

Conclusion

This study proposes a valuable approach based on the AHP and BSC for evaluating the performance of the NLAI. The analytic hierarchy is structured by the four main perspectives of the BSC – resources, access and infrastructure; use; efficiency; potentials and development – followed by performance indicators.

In general, among the 32 indicators selected by the organizational experts from the NLAI, ‘Level of user satisfaction with searching the NLAI’s website’, ‘Percentage of acquired books in any medium and multimedia resources entered by legal deposit’, ‘Qualitative evaluation using case studies of use and value of the collection and services’, ‘Percentage of national publications acquired by the NLAI’ and ‘Percentage of library staff involved in resource processing’ were the top five, respectively. The first and third indicators in this ranking refer to services provided to users. The importance of user satisfaction and the quality of services provided to users in improving the performance of the library is clear because the ultimate goal of a library in any form is the user. This shows that user satisfaction is very important for the NLAI. However, ‘Level of user satisfaction with searching the NLAI’s website’ is ranked first, which indicates the importance of the NLAI’s website. Haji Zeinolabedini et al. (2006) point to the importance of the websites of national libraries as the gateway to the written culture of a country and state that, in addition to investing in design and management, national libraries must make the necessary investments to promote and make their website and its contents available. This ranking also shows that, after user satisfaction, the provision and subsequent processing of resources is more important than the other tasks of the NLAI.

Finally, national libraries are unique institutions in their countries and their data is not easily comparable to that of other libraries. As Line (2001) states, ‘no type of library varies so much in nature, size, types of media covered, range of acquisitions, functions and services’ (quoted in Poll, 2008: 2). But this is what matters, national libraries are a very special sector, with defined national roles which differ from country to country, and which are subject to political agendas which change, often more rapidly than for public or academic libraries (Cullen, 2003). Accordingly, the need to develop performance indicators for national libraries based on their goals and tasks is a serious issue. Therefore, it is suggested that the NLAI use the performance indicators obtained from this research, compare the results and methods used with other national libraries, and then, with frequent reviews and monitoring, reach a consensus on valid indicators. Since indicators are defined as the top levels of an organization, managers can use the findings of this study effectively, applying indicators to allocate resources, make plans and policies on the NLAI. This helps them to avoid any extra costs. Based on the current research, it can be stated that integrating the AHP and BSC is an appropriate means to prioritize and determine the performance indicators for national or other types of libraries.

Research limitations

The main limitation of the study was the NLAI managers’ difficulty in understanding the AHP concepts and how to complete the questionnaire. Approaches such as preparing a guide for completing the questionnaire and the presence of the researcher when filling out the questionnaire were implemented to overcome this issue.

Further research

National libraries, like other organizations, try to identify their strengths and weaknesses and modify their programmes by evaluating their activities and comparing them with those of the past. There is no doubt as to the importance of these evaluations, but expanding their scope through comparisons with other similar organizations would certainly add to their efficiency. Despite the unique cultural, historical and economic characteristics of each country, as national libraries are unique organizations, they have no choice but to compare themselves with the national libraries of other countries. Therefore, it is suggested that the NLAI evaluate its performance based on the indicators obtained from this research and benchmark itself with other national libraries with other national libraries around the world. It is also recommended that, in future studies, other methods be used for weighting and prioritizing the indicators.

Footnotes

Declaration of conflicting interests

The author declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

Funding

The author received no financial support for the research, authorship and/or publication of this article.

Notes

References

Álvarez Pérez

Rodríguez Montequín

Ortega Fernández

, et al. (2017) Integrating analytic hierarchy process (AHP) and balanced scorecard (BSC) framework for sustainable business in a software factory in the financial sector. Sustainability 9(4): Article 486.

Ambrožič

Jakac-Bizjak

Mlekuš

(2003) Performance evaluation in European national libraries: State-of-the-art. In: World Library and Information Congress: 69th IFLA general conference and council, Berlin, Germany, 1–9 August 2003. Available at: https://archive.ifla.org/IV/ifla69/papers/024e-Ambrozic_Jakac-Bizjak_Mlekus.pdf

Amirkhani

(2002) The National Library of Iran, past and present. Alexandria 14(2): 117–121.

Azadeh

Hassanzadeh Dizaji

Neshat

(2013) Evidence of excellence based on the EFQM model: A case study of the National Library and Archives of Iran. Libri 63(3): 240–249.

Azar

Memariani

(1995) The new technique AHP for group decision-making. Journal of Management Knowledge (27–28): 22–32. (In Persian)

Baba

Shukor

(2003) Performance indicators for national libraries in Asia/Oceania: Preliminary proposals based on a survey of Asia/Oceania libraries. In: World Library and Information Congress: 69th IFLA general conference and council, Berlin, Germany, 1–9 August 2003. Available at: https://archive.ifla.org/IV/ifla69/papers/025e-Baba_Shukor.pdf

British Library (2018) British Library annual report and accounts 2017/18. Available at: https://www.bl.uk/britishlibrary/∼/media/bl/global/about%20us/annual%20reports/45th%20annual%20report%202017-18.pdf

Brophy

(2006) Measuring Library Performance: Principles and Techniques. London: Facet.

Cullen

(2003) Benchmarking: Overview and context. In: World Library and Information Congress, 69th IFLA General Conference and Council, Berlin, 2003, pp. 1–11. DOI: http://www.ifla.org/IV/ifla69/papers/015e-Cullen.pdf

10.

Danesh

Soheili

(2009) Investigating the level of visibility and the effective factor of the websites of the Iranian government ministries. Library and Information Sciences 12(1): 203–222. (In Persian)

11.

De la Mano

Creaser

(2016) The impact of the balanced scorecard in libraries: From performance measurement to strategic management. Journal of Librarianship and Information Science 48(2): 191–208.

12.

Fine

Snyder

(1999) What is the difference between performance measurement and benchmarking? Public Management 81(1): 24–25.

13.

Haddadi

Yaghoobi

(2014) Key indicators for organizational performance measurement. Management Science Letters 4(9): 2021–2030.

14.

Haji Zeinolabedini

Maktabifard

Osareh

(2006) Analysis links the National Library Web world. Studies in Education and Psychology 7(1): 173–194. (In Persian)

15.

Hammes

(2011) Third generation balanced scorecards for libraries: From measuring instrument to core strategic management tool. In: Proceedings of the 8th Northumbria international conference on performance measurement in libraries and information services (ed. Graham

Thornton

), Florence, Italy,17–20 August 2009, pp. 43–53. Newcastle upon Tyne: Northumbria University Press.

16.

Hariri

Sarafpour

(2015) Introducing measurement indicators of Iranian special libraries within balanced scorecard approach. Journal of Epistemology (Library and Information Science and Information Technology) 7(27): 31–42. Available at: https://qje.ntb.iau.ir/article_516848.html?lang=en (InPersian)

17.

Hassanzadeh

Osareh

Khosravi

(2022) Performance indicators for the National Library and Archives of IR of Iran. Journal of Studies in Library and Information Science 14(3). DOI: 10.22055/slis.2020.35005.1788. (In Persian)

18.

Horri

(2001) Information: Attitudes and Research. Iran, Tehran: Nashre Ketabdar. (In Persian)

19.

Huang

Song

(2010) Application of balanced scorecard in library management and evaluation. In: MASS 2010: 2010 international conference on management and service science, Wuhan, China, 24–26 August 2010. Available at: https://ieeexplore.ieee.org/document/5577071

20.

Hunter

(2009) Development of strategic performance information at the National Library of Scotland. Performance Measurement and Metrics 10(2): 142–150.

21.

International Organization for Standardization (2003) Information and documentation – International library statistics. ISO 2789:2003. 3rd ed. Geneva: International Organization for Standardization.

22.

International Organization for Standardization (2009) Information and documentation – Performance indicators for national libraries. ISO/TR 28118:2009. Geneva: International Organization for Standardization.

23.

International Organization for Standardization (2014a) Information and documentation – Library performance indicators. ISO 11620:2014. 3rd ed. Geneva: International Organization for Standardization.

24.

International Organization for Standardization (2014b) Information and documentation – Methods and procedures for assessing the impact of libraries. ISO 16439:2014. Geneva: International Organization for Standardization.

25.

International Organization for Standardization (2015) Performance Indicators for National Libraries (ISO 28118). Trans. Kianmehr

. Tehran: National Library and Archives of the Islamic Republic of Iran. (In Persian)

26.

International Organization for Standardization (2019) Information and documentation – Quality assessment for national libraries . ISO 21248:2019. Geneva: International Organization for Standardization.

27.

Kaplan

Norton

(1992) The balanced scorecard - Measures that drive performance. Harvard Business Review 70(1): 71–79.

28.

Ketabi

Mirahmadi

SMR

Karimpoor Azar

(2011) Evaluating the performance of public libraries in provinces by data envelopment analysis. Research on Information Science and Public Libraries 17(1): 9–28. (In Persian)

29.

Kim

Cha

M-K

(2021) A study on the development of performance indicators for the National Library of Korea using balanced scorecard (BSC). Journal of the Korean Society for Library and Information Science 55(1): 251–283.

30.

Kim Thi Ninh

Tanner

Johanson

, et al. (2010) Systematic performance measurement for university libraries in Vietnam. Library Management 31(8–9): 702–716.

31.

Krarup

(2004) Balanced scorecard at the Royal Library, Copenhagen. LIBER Quarterly 14(1): 37–57.

32.

Latifi

Mirhosseini

(2017) Study of users’ utilization and satisfaction from Internet services of National Library of Islamic Republic of Iran. National Studies on Librarianship and Information Organization 27(4): 170–182. (In Persian)

33.

Lee

AHI

Chen

W-C

Chang

C-J

(2008) A fuzzy AHP and BSC approach for evaluating performance of IT department in the manufacturing industry in Taiwan. Expert Systems with Applications 34(1): 96–107.

34.

Library and Archives Canada (2018) Library and Archives Canada, 2017–18: Departmental results report. Available at: https://www.bac-lac.gc.ca/eng/about-us/departmental-results-reports/2017-18-DRR/Documents/drr-2017-2018-eng.pdf

35.

Lin

Zhang

(2014) Performance outcomes of balanced scorecard application in hospital administration in China. China Economic Review 30: 1–15.

36.

Lloyd

(n.d.) Building library success using the balanced scorecard. Available at: https://eis1.my.salesforce.com/sfc/p/#1H000000P2eP/a/1H000000UiCb/7kUU5cFolf2eysPcN4WtG3VaYmR0R9O3eiCXzqPpaIY (accessed 1 January 2022).

37.

Melo

Pires

(2008) Performance evaluation of academic libraries: Implementation model. In: 17th Hellenic conference of academic libraries: Academic library evaluation as a measure of institutional quality assessment, Ioannina, Greece, 24–26 September 2008. Available at: http://eprints.rclis.org/12374/

38.

Moazzen

Matlabi

Vazirpour Keshmiri

(2017) Performance assessment of Semnan academic libraries based on Standard 11620. Quarterly Journal of Knowledge Studies 10(36): 87–101. (In Persian)

39.

Nabizadeh-Kivi

Momeni

Khanipour

, et al. (2014) Performance appraisal of the Information Services Department of the National Library of Iran using balanced score card methodology. National Studies on Librarianship and Information Organization 24(4): 6–23. (In Persian)

40.

National Library and Archives of Iran (2015) Research priorities of the NLAI. Tehan, Iran. Available at: https://www.nlai.ir/documents/10184/0/%D8%A7%D9%88%D9%84%D9%88%DB%8C%D8%AA+%D9%87%D8%A7%DB%8C+%D9%BE%DA%98%D9%88%D9%87%D8%B4%DB%8C+%D8%B3%D8%A7%D8%B2%D9%85%D8%A7%D9%86+-+%D8%B3%D8%A7%D9%84+1394/295864fd-bf15-4012-9793-fcf245ac7

41.

National Diet Library (2018) Activity performance evaluation in FY 2017. Available at: https://warp.da.ndl.go.jp/info:ndljp/pid/11677496/www.ndl.go.jp/jp/aboutus/evaluation/pdf/evaluation_h29_all.pdf

42.

National Library of Australia (n.d.) National Library of Australia, 2017–18 corporate plan: Covering reporting periods 2017–18 to 2020–21. Available at: https://www.nla.gov.au/sites/default/files/nla_corporate_plan_2017-18_final.pdf

43.

National Library of France (n.d.) Contrat d’objectifs et de performance 2017–2021. Available at: https://www.bnf.fr/fr/mediatheque/contrat-dobjectifs-et-de-performance-2017-2021

44.

National Library of Spain (2018) Informe de indicadores: Biblioteca Nacional de España 2017. Available at: http://www.bne.es/export/sites/BNWEB1/webdocs/LaBNE/Transparencia/Institucional/Informe_indicadores_2017.pdf

45.

Noori

(2015) Prioritizing strategic business units in the face of innovation performance: Combining fuzzy AHP and BSC. International Journal of Business and Management 3(1): 36–56.

46.

Nuut

(2002) Library Statistics and Standardization: Performance Measurement and Possibilities for Applying New Methods on Performance Measurement and Benchmarking in Estonia, Statistics in practice Measuring and managing: proceedings of IFLA satellite conference, Loughborough, August 2002, (179-186). Loughborough (United Kingdom): Library and Information Statistics Unit, Loughborough University. Available at: https://www.researchgate.net/publication/237290844_Library_Statistics_and_Standardisation_Performance_Measurement_and_Possibilities_for_Applying_New_Methods_on_Performance_Measurement_and_Benchmarking_in_Estonia

47.

Odera-Kwach

Ngulube

(2011) Quality management framework for evaluating academic libraries in Kenya. South African Journal of Libraries and Information Science 77(2): 116–124.

48.

Office of Planning and Development NLAI (n.d.) Performance indicators of NLAI, Tehran. (In Persian)

49.

Payne

Conyers

(2005) Measuring the impact of higher education libraries: The LIRG/SCONUL Impact Implementation Initiative. Library and Information Research 29(91). DOI: 10.29173/lirg180.

50.

Poll

(2007) Benchmarking with quality indicators: National projects. Performance Measurement and Metrics 8(1): 41–53.

51.

Poll

(2008) Quality indicators for national libraries: The new standard. In: World Library and Information Congress: 74th IFLA general conference and council, Québec, Canada, 10–14 August 2008. Available at: https://archive.ifla.org/IV/ifla74/papers/160-Poll-en.pdf

52.

Poll

(2009) Benchmarking in the form of performance indicators and balanced scorecard. In: Heaney

(ed.) Library Statistics for the Twenty-First Century World. Berlin: Saur, 61–70.

53.

Poll

Jönsson-Adrial

(2006) Performance indicators for national libraries: A list of possible indicators, taken from the new draft of the standard ISO 11620 and from practical examples tested by national or regional libraries. Available at: https://archive.ifla.org/VII/s1/pub/s1-PerformanceIndicators2006.pdf

54.

Quezada

López-Ospina

(2014) A method for designing a strategy map using AHP and linear programming. International Journal of Production Economics 158: 244–255.

55.

Saaty

(1990) How to make a decision: The analytic hierarchy process. European Journal of Operational Research 48(1): 9–26.

56.

Self

(2004) Metrics and management: Applying the results of the balanced scorecard. Performance Measurement and Metrics 5(3): 101–105.

57.

Swiss National Library (2018) Statistiques d’utilisation 2017. Switzerland.

58.

Tavallaei

(2007) New approaches to evaluating the performance of organizations. Bi-Quarterly Journal of Police Human Development 4(12): 9–30. (In Persian)

59.

Wang

Xia

(2009) A fuzzy AHP and BSC approach for evaluating performance of a software company based on knowledge management. In: ICISE 2009: 1st international conference on information science and engineering, Nanjing, China, 26–28 December 2009. Available at: https://ieeexplore.ieee.org/document/5455616/metrics#metrics

60.

Winkworth

(1990) Performance indicators for polytechnic libraries. Library Review 39(5): 23–41.

61.

Yaghoobi

Haddadi

(2016) Organizational performance measurement by a framework integrating BSC and AHP. International Journal of Productivity and Performance Management 65(7): 959–976.

62.

Zaboli

Solimaninejad

Zaboli

(2015) Performance assessment of faculty libraries affiliated with Kerman University of Medical Sciences using balanced scorecard model. Journal of Health-Based Research 1(1): 47–59. Available at: http://hbrj.kmu.ac.ir/article-1-36-en.html(InPersian)

63.

Zahran Al Hijji

Cox

(2012) Performance measurement methods at academic libraries in Oman. Performance Measurement and Metrics 13(3): 183–196.