Evaluation of h and h-type Indices for Research Performance at Researcher's Level

Abstract

Various new performance evaluation indices for impact have been proposed and studied in various contexts. This study carried out a quantitative evaluation of the case of Malaysian engineering researchers at the micro level using a scientometric approach. In order to understand the behavior of new developments, a comparative performance evaluation is carried out of h-index, a set of h-type indices along with publication and citation metrics. Findings are compared with earlier major studies. We looked for institutional h-index and researchers’ h-index scores and did not find any relationship. Exploratory Factor Analysis is employed to examine the valid categorization and to study the underlying dimensions of the studied metrics and indices. The inter-correlation among h-index, its variants, and traditional metrics is probed in detail. The h, q and g-indices along with publication and citation hold the position on ‘quantity of the productive core’, while the R index showed equal loading on both cores. For the case of Malaysian engineering researchers, two conspicuous findings are observed about the total citation and g-index. These have association with the first component named as ‘quantity of the productive core.’ Our findings strengthen the point that citation count has a strong association with the ‘quantity of the productive core’ and cannot be used as sole impact evaluation measure.

Keywords

research performance evaluation engineering research h-index h-type indices bibliometric indicators Malaysia

Introduction

Research output dimensions such as the ‘quantity of the productive core’ and the ‘impact of the productive core’ are being explored at various aggregate levels in developed nations, or the ‘core countries’. The purpose is to get a better understanding of the usefulness of new indices for a fairer research performance decision-making process (Bornmann and Daniel 2005; Moed 2005; Cronin and Meho 2006; Saad 2006; Oppenheim 2007; Imperial and Rodriguez-Navarro 2007, Bouabid and Martin 2009, Norris and Oppenheim 2010a). An interesting research move in this area was to conduct case studies for the comparison of the h-index with other h-type indices. Costas and Bordons (2007) analysed the relationship of the h-index with activity and impact indicators at the micro level for natural resource scientists of the Spanish National Research Council (CSIC) in order to identify some of its advantages and limitations. Since the h-index is size-dependent (Van Raan 2006), they assumed that the total papers and citations had a profound effect on h-index. This is because of its failure to categorize authors who prefer to publish in noteworthy journals in their niche areas, putting them at a disadvantage when compared with those of intermediate level productivity but with high impact. They tested the hypothesis that the achievement of highly visible but intermediate-productive authors might be underestimated when compared with other scientists by means of the h-index. Using a class diagram to four groups, Costas and Bordons (2007) found that undervalued scientists (those with lower positions by h-index than by impact) presented higher numbers of citations per paper (CPP) and highly cited papers than overvalued scientists (those with higher positions by h-index than by impact). In addition, the former had fewer publications with zero citations as compared to the latter and a neutrally-valued group. The best association was found between the h-index and numbers of papers and citations. Similar to Van Raan (2006), they viewed h-index as size-dependent and pointed out that such indicators might lead the trend towards more productivity and lower quality publications. This reservation was also raised by Butler (2003) and Weingart (2005) for other bibliometric indicators. This contradicts the real strength of h-index; publishing and getting citations are both crucial to improving the h-index of a scientist. They also emphasized its use with other indicators as a complementary tool. They argued that “if the h-index were widely adopted as the only measure of scientific performance, these selective scientists could be unfairly treated” (p.202).

It is a fact that h-index generally plays in favor of more senior researchers – it indicates lifetime achievement instead of current or most recent performance. Fiala (2013), who asserted the need for an “age normalization” factor to be able to fairly compare researchers of different ages, introduced the Current Index which is an h-index based on a 3-year publication/citation window combined with a citation count for that time period. The Current Index is able to change over time (increase as well as decrease) even if the scientist under study is not active, because the new indicator considers a 3-year time window for both publications and citations, and therefore reflects current performance rather than lifetime achievement. Using publication and citation data of 20 winners of the ACM SIGMOD Edgar F. Codd Innovations Award2 from Scopus, Fiala (2013) demonstrated that if the rating is updated regularly (possibly on a yearly basis), it may present a dynamic rating framework in which researchers’ ratings (and ranks) can grow as well as decline in time according to their most recent performance.

In order to compare the ability of ‘h’ and another notable evaluative measure, ‘g’-index, Costas and Bordons (2008) applied the Cole and Cole (1973) criteria to the natural resource scientists at CSIC based on data from the Web of Science from 1994 to 2004. These two indices loaded on the same factor during factor analysis, along with Publication (P) and Citations (C), and could distinguish the extreme cases (top and low producers). However, these indices failed to discriminate the middle order groups (big and selective producers). These findings strengthened Jin et al.’s (2007) point of view that this index is only useful for comparing the better scientists in the field, and does not discriminate among average scientists. These findings were in line with the conclusions made by Costas and Bordons (2007) in their first study. A strong positive relationship was observed between these indices, where the ‘g-index’ showed a better relation with Citations Per Publication (CPP) and Highly Cited Publications (HCP).

In another study, Bornmann et al. (2008) studied the relationship between h-index and several of its variants, i.e., m quotient, g index, h² index, the A- index, R- index, AR -index, and hw- index, using data from biomedicine research. These variants were uploaded on two components during Exploratory Factor Analysis (EFA) with h, m quotient, h² and g on the same factor while others were uploaded on the second factor. They assumed that there were two types of indices that stood for different quantity and impact dimensions and complemented each other. In a subsequent study, they included the Standard Bibliometrics Measures (SBMs) with these nine variants (Bornmann et al. 2009a) using molecular life sciences data. They hypothesized whether these new developments made any incremental contributions. Based on both studies they concluded that one type of index describes the most productive core of the output of a scientist and shows the number of papers in the core, while the second depicts the impact of the papers in the core (2009b). The first factor was named ‘Quantity of the productive core’ and second was named ‘Impact of the productive core’. They found high inter-correlation among one set of indices and argued that these made hardly any incremental contribution. They proposed to use either one of the pairs of indicators. One pair is associated with the total research articles in the researcher’s productive core, and the second is related to the impact of the said documents.

A series of case studies was conducted by Schreiber during 2007 to 2013 based on 26 non-prominent physicists’ data from WoS. Based on this dataset, he studied g, h, A and R indices (Schreiber 2008). Egghe (2006a) proposed that the g-index is successful in dealing with citations having a highly skewed frequency distribution. Based on the different ranking of data by these indices, Schreiber (2008) observed a large Pearson correlation coefficient between g and R and argued that the g-index discriminates better between different citation patterns.

In an extension of his study, Schreiber (2010a) differed from the conclusion by Bornmann et al. (2009b) on the use of two separate indices, h and A, to gauge the quantitative and impact dimensions. In his two papers, Schreiber (2010a; 2010b) argued that g and R indices were similar (based on a significant correlation), and both comprise the A-index qualities;, whereas g-index is elegant and comprises information from the productive and impact core. Therefore, he voted for it and proposed to study its application in further studies. In another extension of his study, Schreiber et al. (2011) found the nearly equal loading of the factor for g in the Exploratory Factor Analysis (EFA), which verified his previous assumption. Schreiber (2013) argued in favour of g-index (n = 26); however, De Visscher (2011) claimed, based on a small data set (n = 8), that it was nearly equal to the square root of the total number of citations.

Besides the advantages and disadvantages of the h-index and h-type indices, its application for research performance in many cases is found meaningful. Variants of the h-index are correlated more strongly with peer judgment than the original h-index (Li et al. 2010). Comparison of different indices indicates that the studied indicators are loaded on two different dimensions of the research components (Bornmann et al. 2008; 2009a; Schreiber 2008; 2010a; 2010b; Schreiber et al. 2011). Several studies found that h-type indices fall clearly on one dimension. Derrick et al. (2011) evaluated the relationship of four citation-based metrics, h-index, m-index, m-quotient and q²-index, with peer judgment for six fields. They emphasized the need to explore field data to find out the appropriate metric. These studies reveal that among the most noted h-type indices are g-index, followed by several other indices such as A, R, AR, q², m-indices, m quotient, etc.

Based on various disciplines’ data, Van Leeuwen (2008) reported that both productivity and h-index scores of these disciplines had a noticeable difference. Another noteworthy finding was that age that did not seem to be correlated with h-index scores. Tyson (2009) studied the effect of gender, country of residence, institutional affiliation, and scientific age on 35 library and information science (LIS) researchers’ h-index of New Zealand and Australia. Only a positive relationship was found between scientific age and h-index scores.

In this paper, the relationship of h-index and institutional affiliation is explored, to position the traditional metrics and newly developed research performance indices for research performance evaluation (RPE) in higher education. To do so, an experimental work of researchers’ publication data at the micro level is scientometrically designed, and quantitative evaluation is carried out to get an answer about the feasibility of the h-index and h-Type Indices (h-TI) for research evaluation.

Objectives and method

The main objective of this study is to compare the performance, underlying dimensions and positioning of traditional metrics and newly developed research evaluation indices to evaluate university research performance. The EFA is applied to explore the underlying dimensions of h-TI indices for RPE. Malaysian engineering research is our unit of analysis for the reason that engineering is a multidisciplinary and interdisciplinary applied field and has broader applications. A scientometric analysis is carried out with the application of a set of three indicators at the micro level (a) Activity Indictor (AI); (b) Observed Impact Indicator (OII), and (c) h-index; with a set of the most studied h-Type Indices (h-TI) consisting of Publication (P), Citation (C) and Citations Per Publication (CPP), and g, A, R, m, q², H’. The selected set of indices is based on core citation distribution issues.

The case of the first 100 most productive Malaysian engineering researchers from WoS over a 10-year period (2001-2010) is analyzed. Appendix 1 presents the detailed results for the top 10 researchers; details of the results for all 100 researchers may be obtained from the corresponding author. The results of the analysis reported below apply to the full sample of 100 researchers.

The research output is refined in terms of time span, document type, engineering research categories and selection of database. The search term used was ‘Malaysia’ and it was limited to only the 9 engineering categories from WoS that have the word ‘engineering’ in common:

engineering – electrical and electronic

engineering – manufacturing

engineering – biomedical

engineering – industrial

engineering – civil

engineering – chemical

engineering – mechanical

engineering – environmental

engineering – multidisciplinary.

The term ‘Malaysian engineering researchers’ is used for researchers who are affiliated with 11 Malaysian universities under nine WoS engineering categories. As a rule, these universities have published >50 publications for two document types: articles and reviews. They are:

Universiti Sains Malaysia (USM)

Universiti Putra Malaysia (UPM)

University of Malaya (UM)

Universiti Teknologi Malaysia (UTM)

Universiti Kebangsaan Malaysia (UKM)

Universiti Teknologi Mara (UiTM)

International Islamic Universiti Malaysia (IIUM)

University of Multimedia (MMU)

University of Nottingham Malaysia Campus (UNMC)

Universiti Teknologi PETRONAS (UTP)

Monash University Sunway Campus (MONASH).

The metrics and indices used at micro level in the study are defined as follows:

1. Activity Indicator (AI)

Publications (P): Total publications over the period of 10 years (2001-2010) of researcher including the research articles and reviews only from WoS via Web of Knowledge.

2. Observed Impact Indicator (OII)

Citations (C): Total citations during the period 2001–2010 of publications records, including self citations.

Citation Per Publication (CPP): Citation per publication for the 10 years.

3. h and h-Type Indices

h-index: A scientist has index h if h of his /her Np papers has, at least, h citations each and the other (Np − h) papers have no more than h citations each (Hirsch 2005 p.16569).

h-Type Indices (h-TI)

g-index: The g-index is the highest number g of articles that together received g² or more citations (Egghe 2006a p.8).

A-index: A-index is the average number of citations received by the articles in the h-core (Jin 2006).

R-index: R-index is the square root of the total number of citations received by the articles in the h-core. (Jin et al. 2007)

m-index: m-index is the median number of citations received by papers in the Hirsch core (Bornmann et al. 2008).

q²-index: A composite index computed by the product of the h-index and median of the h-core citations (Cabrerizo et al. 2010).

H’: It deals with the citation distribution function with head and tail core ratio and formalizes as h′ = e.h/t (Zhang 2013). Where ‘e’ is the excess citation above the h-core and t is the tail core. h and g-indices are counted manually while, other are computed by applying their respective formula.

To verify the data, publications and citations record were checked one by one manually. For citation count, we used time cited features of WoS and checked all citations manually for the period 2001–2010. Websites for both the universities and the researchers were checked to detect variations in affiliation and author’s name. Author’s last affiliation was considered as final. All the data at this level were also cleaned manually for affiliation, homonym, and homograph problems. The most common problems noted in our dataset were: using different names with a slight change in spelling, changing universities and having same initials and last names. Further data were verified by senior librarians from the respective universities.

The top 100 prolific authors belonged to 11 universities. These researchers contributed a significant share of total research output and visibility of their institutions over the entire period of study. Table 1 presents the universities’ total share of publication and citation percentage. The five research universities have a total of 80 authors who share more than 80% in terms of productivity and impact, while six other non-research universities share about 20% of the sample data.

Table 1.

Productivity and impact of Malaysian engineering researchers from 11 universities.

No.	University name	Total authors	Total publication	Total citation	University type	Total share	Approx % share
1	USM	24	668	4918	Research Universities	1898 (P) 11519 (C)	82% 81%
2	UPM	23	413	2150
3	UM	13	323	1753
4	UTM	9	202	1453
5	UKM	12	292	1245
6	UiTM	1	15	69	Non-research universities	438 (P) 2628 (C)	19% 19%
7	IIUM	1	12	23
8	MMU	9	156	999
9	UNM	6	147	1385
10	UTP	2	36	65
11	MONASH	1	15	78
	Total	100	2264	14138

The values of different indicators for the top 10 researchers in the sample are computed and presented in Appendix 1. Based on the data, we checked the ranking, descriptive evaluation and performance of these indices. There is a significant fluctuation in ranking order of these researchers with respect to the application of all indicators.

Results

Descriptive information of a set of indicators for the performance evaluation purpose is presented in Table 2. The h-TI set consists of six h index variants (g, A, R, m, q² and H’). These variants are based on citation distribution function to overcome the disadvantages of the original h-index. As can be seen in Table 2, P and C exhibit a noticeable variation in the mean and median values with a substantial standard deviation compared to other traditional metrics in the same set. Quite a new development, H’ by Zhang (2013) shows a large variation between mean and median values with a high standard deviation.

Table 2.

Descriptive statistics of researchers’ indices data.

	Mean and SD	Median	Mode	Range(min-max)
Activity Indicator (AI)
P	22.9, 16.049	17	15	11-100
Observed Impact Indicator (OII)
C	141.4, 183.22	86.50	90	16-1138
CPP	5.739, 3.86	4.56	6	0.8421-21.3
h and h-type Indices (hTI)
h-index	6.05, 3.109	5	4	1-21
g-index	9.56, 4.916	8	7	3-31
H’	22.1, 45.3	10	7	1.22-37.84
A	14.71, 8.16	11.7	5.3	5.04-48.75
R	9.22, 4.408	8.18	4	3.16-28.67
M	11.7, 5.86	10	10	4.5-32.5
q2	8.27, 3.803	7.1	7.07	3.16-25.5

The h-index is sensitive to disciplinary perspective (Iglesias and Pecharroman 2006; Van Leeuwen 2008). An overview of h-index of various disciplines based on different strength of data, number of researchers and regional perspective, stated that the average h-index for information sciences was 11 in the case of scholars from the USA and ranges from 20-5 (Cronin and Meho 2006); in the case of British LIS scholars, the average h-index was 7 and in the range of 31-5 (Oppenheim 2007). Saad (2006) found the h-index of consumer researchers was between 17-3, whereas computer scientists had the h-index in the range of 24 to 14 (Sidiropoulos et al. 2007). An important aspect is that the above mentioned findings are considerably lower than world prolific authors in physics and the life sciences, that range between 110-62 and 191-120 respectively (Hirsch 2005). In our case, the average h-index of engineers in our dataset is 6 and in the range of 21-1.

Box Plots illustration

The box plots of AI, OII and h and h-TI as seen in Figure 1(a-i) indicate the comparative performance of these indicators. This log-transformed data helps to understand the degree of dispersion, outliers and skewness of the data. These plots tell us the median, the upper and lower quartiles of central tendency, and the highest and lowest values of the data set. C and CPP illustrate a better median and extreme positions among AI and OII case; whereas, among h and h-type indices, h-index and R-index exhibit better position for lower and upper levels.

Figure 1. (a-i).

Box plots illustrations of AI and OII and h-index and h-TI.

Institutional affiliation and researcher’s h-index score

We employed Kruskal-Wallis, a popular non-parametric test for comparing K-independent samples inference (Siegel and Castellan 1988). We applied this test to examine the influence of institutional affiliation on a Malaysian researcher’s h-index scores (Table 3). The result indicates that there is no significant relationship (asymptotic. Sig. >.05) between the institutional affiliation and researcher’s h-index score. Therefore, our hypothesis that there is a relationship between institutional affiliation and researcher’s h-index score is rejected.

Table 3.

Kruskal-Wallis test for institutional affiliation and researcher’s h-index score.

Test Statistics ^ab
	h-index
Chi-Square	9.618
Df	10
Asymptotic. Sig.	.475

^aSignificant at .05 level.

^bgrouping variable: affiliation.

Typically, researchers belonging to prestigious institutions have an influence on the productivity and quality of research produced (Meaddows 1998; Allison and Long 1990). Table 1 reveals a significant difference in contributions from researchers of research and non-research university status. Notwithstanding, we have also observed that the highest h-index score is concentrated in a few universities. This endorses the results of a study by Oppenheim (2007). Kelly and Jennions (2006) for the first time raised the issue that there might be some relation between institutional affiliation and researcher’s h-index score. However, empirically this is found false. Tyson (2009) also made such an observation for 35 LIS academics of Australian and New Zealand universities.

Functional relationship and predictive values

We conducted a statistical analysis under the prescribed research objectives. The objective was to examine the functional relationship, predictive values and to explore the underlying dimensions of this relationship and a potential set of these indices for an objective solution.

A linear regression model fit was employed to observe the association of h-index with activity and impact metrics. Figure 2a shows a strong functional relationship with citation (R² = 0.86). This relation is found to be weaker (Figure 2b) for publications (R² = 0.635), while CPP (Figure 2c) displays a weak positive (R² = 0.369) relation with h-index. The regression power trends (Figures 2 a-c) illustrate that all traditional metrics exhibit a predictive value, which is nearly equal to the square of the multiple of h = 0.934p^0.594; 0.562C^0.499 and 2.398CPP^0.521. At the author level, the total citation is found to be the best predictor of h-index for the given dataset followed by P and CPP.

Figure 2 (a-c).

Functional relationship among h-index and C, P and CPP.

To “explore the factors of research performance dimension and to answer which index is feasible” is a crucial issue raised by Burell (2007 p.168). To further examine the relationship between different indices, Exploratory Factor Analysis (EFA) was applied. Factor analysis was used to study the structure of inter-correlation of the indices that purported to measure something similar (Harnad 2007). Factor analysis is a statistical method “to reduce the dimensionality of the data space to discover, visualize and interpret dependencies among sets of variables” (Timm 2002: 445). EFA is employed to examine the valid categorization and study the underlying properties by probing the inter-correlation among the h-index, its variants, AI and OII at the researchers’ level.

Exploratory Factor Analysis (EFA)

The descriptive analysis of the dataset indicates that few indices exhibit a significant variation between mean and median with a large standard deviation (Table 2). Scientometric data are usually not symmetrically distributed and are skewed (Egghe 2006; Bornmann et al. 2008; Moed, 2005). We therefore applied Egghe’s assumption (2005; 2006) that the relationship between any two indices is nonlinear and can be described as a power function. To get a more likely approximation for a normal distribution, we applied square root and logarithmic transformation to the raw dataset. For the case, we found that the logarithmic transformation was best. Table 4 shows the descriptive statistics with skewness and kurtosis results of transformed data. The values of skewness and kurtosis in between ±2 are acceptable (George and Mallery 2003). The log-transformed data result indicates that the data is now adequately normal for Exploratory Factor Analysis (EFA).

Table 4.

Descriptive statistics, skewness and kurtosis results of all indicators.

Indices	N	Minimum	Maximum	Mean	Std. Deviation	Skewness		Kurtosis
Indices	Statistic	Statistic	Statistic	Statistic	Statistic	Statistic	Std. Error	Statistic	Std. Error
P	100	2.48	4.62	3.027	.4863	1.288	.241	1.166	.478
C	100	2.83	7.04	4.564	.8025	.810	.241	.976	.478
CPP	100	.61	3.11	1.79	.4754	.366	.241	.058	.478
h	100	.69	3.09	1.878	.3727	.541	.241	1.528	.478
H’	100	.69	5.88	2.494	.9818	.904	.241	1.410	.478
g	100	1.39	3.47	2.274	.3927	.621	.241	.786	.478
A	100	1.79	3.91	2.646	.4510	.523	.241	-.030	.478
R	100	1.43	3.39	2.251	.3708	.609	.241	.610	.478
m	100	1.70	3.51	2.459	.4065	.517	.241	-.087	.478
q2	100	1.43	3.28	2.160	.3519	.682	.241	.673	.478
Valid N (list wise)	100

Note Data is Log normalized.

We applied EFA to our transformed dataset. The Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy of researchers’ data should be greater than 0.5 to decide to apply EFA. In our case the KMO value is adequate (.>65) to apply EFA (Table 5) with high commonalities (>.85) except for m-index (>0.75). The scree plot at Figure 3 also shows a clear bend for two components. The percentages of total variance accounted for by each factor (P and C) are 76% and 16% respectively, and these two indices together accounted for 92% of the total variance (Table 6).

Table 5.

Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy of researchers’ data.

Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO)	0.677
Sig	0.000

Figure 3.

Scree plot for all components.

Table 6.

Total variance explained among indicators.

Total Variance Explained
Component	Initial Eigen values			Extraction Sums of Squared Loadings			Rotation Sums of Squared Loadings
Component	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	7.581	75.815	75.815	7.581	75.815	75.815	4.921	49.211	49.211
2	1.598	15.979	91.793	1.598	15.979	91.793	4.258	42.582	91.793
3	.400	4.000	95.793	–	–	–	–	–	–
4	.274	2.744	98.538	–	–	–	–	–	–
5	.123	1.231	99.769	–	–	–	–	–	–
6	.016	.164	99.933	–	–	–	–	–	–
7	.005	.054	99.987	–	–	–	–	–	–
8	.001	.011	99.998	–	–	–	–	–	–
9	.000	.002	100.000	–	–	–	–	–	–
10	7.129E-6	7.129E-5	100.000	–	–	–	–	–	–

Component matrix executes the results for a two-factor solution (Table 7). Bornmann et al. (2008) used a greater than 0.6 cut-off threshold for categorization of the indices between the factors. For this case, we have also used >0.6 cut-off thresholds to make clear categorization. Rotated component matrix (Table 7) is loaded h-index along with P, C, q² and g on the first component, while CPP, H’, A, and m indices are uploaded on the second component. We made a comparison of conspicuous findings from the other studies conducted in different contexts and disciplines at the micro level to seek the relation between indices, and prescribed two factors. Based on the categorization by Bornmann et al. (2008; 2009a) and Schreiber et al. (2012), we named these two components as ‘quantity of the productive core’ and ‘impact of the productive core’. Our findings demonstrate that the h-index, q and g-index along with P, C hold the position on the ‘quantity of productive core’ while, CPP, H’ A, and m indices are positioned on ‘impact of the productive core’. The m index is observed having less correlation with h-index in a study by Bornmann et al. (2011). In our case, the m-index has comparatively less association with component 2. We cannot make a clear categorization for R-index on either factor because it is uploaded on both components based on the threshold value >0.6. Earlier Schreiber et al. (2012) has reported similar results. However, this index was placed on the second component in both studies by Bornmann et al. (2008; 2009b).

Table 7.

Rotated component matrix for all indicators.

Indices	Component
Indices	1	2
P	.940	−.178
C	.849	.514
CPP	.364	.883
h	.879	.319
H’	−.076	.950
g	.790	.597
A	.486	.817
R	.754	.649
m	.569	.659
q²	.807	.547

Extraction Method: Principal Component Analysis.

Rotation Method: Varimax with Kaiser Normalization

Values >.6 are in boldface

Costas and Bordons (2007), in a study of natural resources researchers at CSIC using WoS data (1994-2004) reported the same tendencies for h, P and C as we observed in this study. In a second study (Costas and Bordons 2008), they also included g-index, that holds a position on the first factor. Notably, other indices in their set are related to the relative impact and IF median, and they extracted four components. The same result for g-index was reported by Bornmann et al. (2008; 2009a) for postdoctoral researchers in biomedicine. However, two noticeable findings are about the citation metric (Bornmann et al., 2008) and g-index (Schreiber et al., 2012). They reported their position on the second component with R-index, while we differ for C and g position (Table 7) as observed by aforesaid studies. Except for these conspicuous findings, the current study is mostly in agreement with Costas and Bordons (2007) as well as Bornmann et al. (2008; 2009a).

Discussion and conclusions

To observe a general ranking behavior, we analyzed the scientific research productivity and citation input of the first 100 most productive Malaysian engineering researchers. Eighty percent of the most productive authors have an affiliation with five research universities. USM and UPM have almost equal numbers of prolific authors. Other prominent universities are UM, UKM, UTM, and MMU respectively. We applied four set of indicators (AI, OII, h, and h-TI), which demonstrate discriminating power for the ranking purpose. Application of indices shows a large variation in the positioning order of researchers. This is more evident in the case of m, R and H’ indices when citations are being normalized.

About half (47) of the authors in our dataset belong to two universities (USM and UPM). Despite this fact, no influential relation of Malaysian institutional affiliation and researcher’s h-index scores was found. Nevertheless, these indices play a key monitoring role in decision making for the recruitment process and may be a valid supporting tool for the peer review process in higher education institutions. In addition, h-index has gained popularity and immediately become part of global citation databases. It has been considered for evaluation purposes at the individual level. It seems that to link prolific researchers’-h-index with their respective institutional affiliation is still not mature for incentive or selection purposes. Nonetheless, we have observed the place of this index and its discriminatory power in the present and certain past studies. One of the strong reasons for such a finding might be the time frame of the study. Another strong reason is the disciplinary perspective (Iglesias and Pecharroman 2006). Furthermore, the culture of research, the policies of the Ministry of Education and uneven international visibility can be influential factors.

To examine the appropriateness of this index for the researchers, we have explored power regression trends, which illustrate that total citation is the best predictor and has strong model fit with h-index at this level. Predictive power is a square of the multiple of traditional metrics P, C, and CPP. To find the best index in the RPE process, EFA was applied. A two-factor solution was extracted, named ‘quantity of the productive core’ and ‘impact of the productive core.’ Our findings strengthen the point that citation count has a strong association with the ‘quantity of the productive core’ and cannot be used as sole impact evaluation measure (Garfield, 1983; Coastas and Bordons, 2008; Leydesdorff, 2009; Schreiber et al., 2011). Therefore, more indices are required to address the impact core. Interestingly, h-index also shows its place on the quantity of the productive core in the present and various other past studies. The potential impact indices at the micro level for the present case are CPP, H’, and A-index. In our case, two conspicuous findings are observed about C and g- index that showed their association on the first component. Moreover, it is worthwhile to notice that only R-index shows its association on both cores. However, only one study supported the argument for this finding. This might be due to some variation in the studied indices or a disciplinary perspective of the study.

Our findings for the studied indices at this level are in general agreement with the previous studies. To examine the feasibility of the h-index for the researcher, we have explored regression power trends, which illustrate that total citation is the best predictor and has strong model fit at this level. Predictive power is the square of the multiple of traditional metrics P, C, and CPP. The high correlation, predictive value and loading of h-index on the same component (in EFA), depicts that this index has the potential to work as traditional P and C metrics for broader impact performance evaluation purposes.

Footnotes

Appendix 1. Application of AI,OBI h and H-type Indices to most Prolific Malaysian Engineers

Note: Appendix 1 presents the detailed results for the top 10 researchers; details of the results for all 100 researchers may be obtained from the corresponding author: Abrizah Abdullah, Department of Library and Information Science, University of Malaya, Kuala Lumpur 50603, Malaysia. Email: abrizah@um.edu.my.

The results of the analysis reported in the text apply to the full sample of 100 researchers.

Acknowledgement

The work of Muzammil Tahira and A. Abrizah was supported by the Ministry of Higher Education Malaysia (HIR-MOHE) UM.C/HIR/MOHE/FCSIT/11.

References

Allison

Long

(1990) Departmental effects on scientific productivity. American Sociological Review 55, 469–478.

Bornmann

Daniel

(2005) Does the h-index for ranking of scientists really work? Scientometrics 65(3): 391–392.

Bornmann

Mutz

Daniel

(2008) Are there better indices for evaluation purposes than the h-index? A comparison of nine different variants of the h-index using data from biomedicine. Journal of the American Society for Information Science and Technology 59(5): 830–837.

Bornmann

Mutz

Daniel

Wallon

Ledin

(2009a) Are there really two types of h-index variants? A validation study by using molecular life sciences data. Research Evaluation 18(3): 185–190.

Bornmann

Mutz

Daniel

(2009b) Do we need the h-index and its variants in addition to standard bibliometric measures? Journal of the American Society for Information Science and Technology 60(6): 1286–1289.

Bouabid

Martin

(2009) Evaluation of Moroccan research using a bibliometric-based approach: investigation of the validity of the h-index. Scientometrics 78(2): 203–217.

Burrell

(2007a) On the h-index, the size of the Hirsch core and Jin’s A-index. Journal of Informetrics 1: 170–177.

Butler

(2003) Explaining Australia’s increased share of ISI publications; the effects of a funding formula based on publication counts. Research Policy 32 (1): 143–155.

Cabrerizo

. (2010) q²-index: quantitative and qualitative evaluation based on the number and impact of papers in the Hirsch core. Journal of Informetrics 4, 23–28.

10.

Cole

(1973) Social Stratification in Science. University of Chicago: Press, Chicago.

11.

Costas

Bordons

(2007) The h-index: advantages, limitations and its relation with other bibliometric indicators at the micro level. Journal of Informetrics 1, 193–203.

12.

Costas

Bordons

(2008) Is g-index better than h-Index? An exploratory study at the individual level. Scientometrics 77(2): 267–288.

13.

Cronin

Meho

(2006) Using the h-index to rank influential information scientists. Journal of the American Society for Information Science and Technology 57(9): 1275–1278.

14.

De Visscher

(2011) What does the g-index really measure? Journal of the American Society for Information Science and Technology 62 (11): 2290–2293.

15.

Derrick

Haynes

Chapman

Hall

(2011) The association between four citation metrics and peer rankings of research influence of Australian researchers in six fields of public health. PLoS ONE 6(4): e18521.

16.

Egghe

(2005) Power Laws in the Information Production Process: Lotkaian informetrics. Kidlington, UK: Elsevier Academic Press.

17.

Egghe

(2006a) An improvement of the h-index: the g-index. ISSI Newsletter 2(1): 8–9.

18.

Egghe

(2006b) The power of power laws and an interpretation of Lotkaian informetric systems as self-similar fractals. Journal of the American Society for Information Science and Technology 56(7): 669–675.

19.

Fiala

(2013) Current index: a proposal for a dynamic rating system for researchers. Journal of the Association for Information Science and Technology 65(4): 850–855. DOI: 10.1002/asi.23049.

20.

George

Mallery

(2003) Frequencies. In: George

Mallery

(eds.) SPSS for Windows Step by Step: A Simple Guide and Reference 11.0 Update. Pearson Education, New York, pp: 20–52.

21.

Harnad

(2007) Open access scientometrics and the UK Research Assessment Exercise. In Torres-Salinas

Moed

(eds.) Proceedings of the 11th Conference of the International Society for Scientometrics and Informetrics (Vol. 1, pp. 27–33). Madrid, Spain: Spanish Research Council (CSIC).

22.

Harnad

(2008) Validating research performance metrics against peer rankings. Ethics in Science and Environmental Politics 8(1): 103–107 (2008).

23.

Iglesias

Pecharromán

(2007) Scaling the h-index for different scientific ISI fields. Scientometrics 73(3): 303–320.

24.

Imperial

Rodriguez-Navarro

(2007) Usefulness of Hirsch’s h-index to evaluate scientific research in Spain. Scientometrics 71(2): 271–282.

25.

Jin

(2006) h-index: an evaluation indicator proposed by scientist. Science Focus 1(1): 8–9 (in Chinese).

26.

Jin

Liang

Rousseau

Egghe

(2007) The R- and AR indices: complementing the h-index. Chinese Science Bulletin 52(6): 855–863.

27.

Kelly

Jennions

(2006) The h index and career assessment by numbers. Trends in Ecology and Evolution 21 (4): 167–170.

28.

Sanderson

Willett

Norris

Oppenheim

(2010) Ranking of library and information science researchers: comparison of data sources for correlating citation data and expert judgments. Journal of Informetrics 4(4): 554–563.

29.

Meadows

(1998) Who does research and with what results? In: Meadows

, Communicating Research (pp. 79–114). San Diego, California: Academic Press.

30.

Moed

(2005) Citation Analysis in Research Evaluation. Dordrecht, Springer.

31.

Norris

Oppenheim

(2010a) Peer review and the h-index: two studies. Journal of Informetrics 4(3): 221–232.

32.

Oppenheim

(2007) Using the h-index to rank influential British researchers in information science and librarianship. Journal of the American Society for Information Science and Technology 58(2): 297–301.

33.

Saad

(2006) Exploring the h-index at the author and journal levels using bibliometric data of productive consumer scholars and business-related journals respectively. Scientometrics 69: 117–120.

34.

Schreiber

(2007) A case study of the Hirsch index for 26 non-prominent physicists. Annalen der Physik (Leipzig) 16(9): 640–652.

35.

Schreiber

(2008) An empirical investigation of the g-index for 26 physicists in comparison with the h-index, the A-index and the R-index. Journal of the American Society for Information Science and Technology 59(9): 1513–1522.

36.

Schreiber

(2010a) Revisiting the g-index: the average number of citations in the g-core. Journal of the American Society for Information Science and Technology 61(1): 169–174.

37.

Schreiber

(2010b) Twenty Hirsch index variants and other indicators giving more or less preference to highly cited papers. Annalen der Physik (Berlin) 522(8): 536–554.

38.

Schreiber

(2013) Do we need the g-index? Journal of the American Society for Information Science and Technology 64(11): 2396–2399.

39.

Schreiber

Malesios

Psarakis

(2011) Categorizing Hirsch index variants. Research Evaluation 20(5): 397–409.

40.

Schreiber

Malesios

Psarakis

(2012) Exploratory factor analysis for the Hirsch Index, 17 h-type variants and some traditional bibliometric indicators. Journal of Informetrics 6: 347–358.

41.

Sidiropoulos

Katsaros

Manolopoulos

(2007) Generalized Hirsch h-index for disclosing latent facts in citation networks. Scientometrics 72(2): 253–280.

42.

Siegel

Castellan

NJR

(1988) Nonparametric Statistics for the Behavioral Sciences, McGraw-Hill Book Company, New York.

43.

Tyson

(2009) Using the h-index to Measure Research Performance in Higher Education: A case study of Library and Information Science faculty in New Zealand and Australia. Degree of Master of Library and Information. Victoria University of Wellington.

44.

Timm

(2002) Applied Multivariate Analysis. New York, NY, USA: Springer.

45.

Van Leeuwen

(2008) Testing the validity of the Hirsch-index for research assessment purposes. Research Evaluation 17(2): 157–160.

46.

Van Raan

AFJ

(2005) Fatal attraction: conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics 62(1): 133–143.

47.

Weingart

(2005) Impact of bibliometrics upon the science system: inadvertent consequences? Scientometrics 62(1): 117–131.

48.

Zhang

(2013) The h′ Index, effectively improving the h-index based on the citation distribution. PLoS ONE 8(4)e59912.