Research Performance Evaluation in China: A Big Data Analysis

Abstract

China’s scientific achievement has received considerable international attention due to a large amount of research and development (R&D) spending. This article aims to study the performance of China’s R&D expenditures (in the form of research funding) by examining the research performance of individual researchers based on bibliometric measures. This study concludes that research practice is not merely determined by capital possessed. Besides, international collaboration primarily accounts for research performance of scholars, whereas research funding and publishing in Chinese-based journals do not impact research performance significantly.

Keywords

big data research performance China grant

In an editorial commentary in Science, Shi and Rao (2010) commented on China’s problematic funding system and entrenched research culture that impedes scientific progress and innovation. These comments were, however, refuted by government authorities (Xin, 2010). Seemingly to echo the Chinese authority’s stance, Nature (Morrison, 2014; Noorden, 2016; Witze, 2016) reported that China’s expenditure in research and development (R&D) ranked second only to the United States (see Figure 1) and produced the third-highest number of publications following the European Union (EU) and the United States. However, Nature (Morrison, 2014; Noorden, 2016) also mentioned that very few articles authored by scholars based in mainland China had been cited by international scholars. Moreover, M. J. Smith et al. (2014) discovered that China’s share of top-cited papers is below the worldwide average and also below the EU28’s average performance (see Noorden, 2016).

Figure 1.

Comparisons of R&D expenditures among the major economies and of national grants between China (shown as CN in the graph) and the United States.

China’s research expenditures have been booming in the wake of rapid economic growth, but it is controversial to evaluate the effectiveness of the massive R&D spending when different criteria were employed (Jauch & Glueck, 1975). To contribute to the discussion, we present an empirical analysis of China’s “science enterprise” from the Bourdieusian point of view. Specifically, the research purpose of the present study is to examine the outcome of China’s expenditures for research projects by examining the research performance of individual scholars as well as the factors affecting their performance.

Literature Review

Conceptualization and Operationalization of Research Performance

Performance is a multidimensional concept and has been evaluated by a variety of criteria (Ford & Schellenberg, 1982). Therefore, evaluating any type of performance is always controversial, and conceptualizing and assessing research performance is not an exception (Daniel & Fisch, 1990; Froghi et al., 2012; Jauch & Glueck, 1975; Moed, 2002).

Bazeley (2010) subdivides the construct of research performance into two components: “research activity” and ‘performance’—the outcome of research practice made visible and passed on to others. In a more straightforward form of the conceptualization (Bazeley, 2010), research performance is defined as the anticipated research outcomes of researchers in concrete products (e.g., publications), academic standing, personal understanding, and benefits to the community. This theorizing is illuminating in that it has encompassed all the essential elements of research performance, but its operationalization is far more contentious. Many (e.g., García & Sanz-Menéndez, 2005; Hicks et al., 2015; Potì & Reale, 2007; Roessner, 2002) suggest that measures of research performance may include bibliometric measures, research funding, awards, academy memberships, royalty income, activity measures, mid-term impact measures, long-term measures, and other indicators of increased competitiveness.

Bibliometric measures may include both quantitative and qualitative indicators that measure the performance of a journal, researcher, or research group (Durieux & Gevenois, 2010). Quantity may include the number of publications and number of publications in top-ranked journals, while quality may include the journal’s impact factor (IF), 5-year journal IF, immediacy index, cited half-life, Crown indicator, H-index, and other indicators (Durieux & Gevenois, 2010). In spite of the prevalence, many scholars (Kurmis, 2003; Milesi et al., 2014; Moed, 2002; Seglen, 1997) argue that bibliometric indicators do not reflect scientific quality and only provide useful supplementary tools for evaluating academic research (for an overview, see D. R. Smith, 2012). Nature (2002) contended that comparing IFs of journals is meaningless if the journals serve different disciplines (see Moed, 2006). A similar argument is shared by many other scholars (Amin & Mabe, 2000; Durieux & Gevenois, 2010; Hicks et al., 2015; Kurmis, 2003; Seglen, 1997). In addition, certain scholars (Bordons et al., 2002; Kurmis, 2003) note that IFs may be easily manipulated and may also be limited and biased. Consequently, many (Daniel & Fisch, 1990; García & Sanz-Menéndez, 2005; Hicks et al., 2015; “The Maze of Impact Metrics,” 2013; Roessner, 2002) strongly advocate for non-bibliometric measures. Nevertheless, bibliometric measures are still the most popular in practice, primarily because other measures could also backfire (see “The Maze of Impact Metrics,” 2013), and bibliometric measures are readily accessible (Bordons et al., 2002) and are defended by numerous scholars (e.g., Bornmann & Williams, 2017; Durieux & Gevenois, 2010; Garfield, 1999, 2001; Ketcham, 2007; Moed, 2002).

Theoretical Framework Explaining Research Performance

Credibility cycle model

Among many conceptual frameworks regarding research performance evaluation, the credibility cycle model (Latour & Woolgar, 1982) may be the most notable perspective. Latour and Woolgar (1982) explain how struggles for reputation influence the behavior of individual scientists whose primary motivation is the quest for credibility. Latour and Woolgar (1982) argue that the research process is a repetitive cycle involving conversions between funding, staff, data, arguments, articles, and recognition. The credibility cycle model has its origin in the Matthew Effect in Science (Merton, 1968a; Zuckerman, 1977), and has been widely applied in research evaluation (Hessels et al., 2011; Hessels & van Lente, 2011; Leisyte et al., 2008). However, scientists’ behavior and motivations are hardly homogeneous, especially considering different scientific systems. Noting the limitations of Latour and Woolgar (1982), Horta and Santos (2016) further added the indicator of Drive to Publish or Perish in the ambition factor, in addition to a few separate factors, for example, collaboration, mentor influence, and dispositional propensities.

Bourdieu’s capital theory

As a well-known sociologist of science, Bourdieu (1975) believes that every scientific choice is “a political investment strategy directed toward maximization of strictly scientific profit.” Consistent with his other famous works (Bourdieu, 1984, 1986), Bourdieu (1975) held that the extent to which they are motivated to fulfill their self-interests depends on the amount of scientific resources (capital) they possess, specifically, the economic, cultural and social network resources, and the symbolic capital¹ (see Bourdieu, 1986), and the power that the resources can bring. Researchers occupying the varying amounts of resources have discrepant research capabilities and varying levels of productivity (Bourdieu, 1975). All this ends up with the social distinction of researchers in a field to which they belong.

According to Bourdieu (1984, 1986), in addition to resources or capital, other factors are at play to cause scholars to have very distinctive practices, all of which are represented in the following formula:

[(h a b i t u s) \times (c a p i t a l)] + f i e l d = p r a c t i c e s .

In a nutshell, habitus, which is a set of preconscious dispositions including tastes, translates agents’ different social positions in social space specified by different forms of capital into observable practices or behavior in a particular field (Bourdieu, 1984; also see Feng et al., 2019). That is, the practices that habitus produces vary according to agents’ position in social space (Weininger, 2005). Social space is the very structural factor. Scientists’ activities occur in the context of a “research system,” that is, filed, consisting of universities, research institutes, funding agencies, governmental organizations, and firms, among others (Hessels et al., 2011; Rip & Van der Meulen, 1996). Capital is distributed in particular types, and habitus operates in a delimiting structure in this institutional environment (Bourdieu, 1984), which also provides research organizations and researchers with incentives and constraints to conduct research (Bourdieu, 1975; Hessels et al., 2011).

Consequently, scholars have drastically distinctive research practices and performance under the varied structure of the scientific system. Scholars are motivated to have power, and they strive for opportunities to acquire power (Bourdieu, 1975). Those who have resources, such as social networks, and the ability to engage in political struggle, will choose more prestigious high-impact journals. They will also be more likely to succeed in taking power, which in turn brings more resources to them, eventually facilitating the cycle of capital (Latour & Woolgar, 1982). In contrast, those who do not enjoy power for its own sake and who are less competitive in terms of academic ability due to a lack of resources will choose domestic and low-impact international journals. Scholars who work in diverse circles are rewarded with different kinds of power and capital (Latour & Woolgar, 1982).

The conceptual model

Based on the literature review and data availability, a theoretical model is subsequently proposed (see Figure 2), in which the effects of various types of resources or capital on scholars’ research performance (measured by the number of citations) are mediated by the research practice or strategy adopted (measured by choice of different tiers of journals). Specifically, the amount of a grant, the number of foreign coauthors, the number of China-based international, referring in particular to Sciences Citation Index (SCI)/Social Sciences Citation Index (SSCI)/Arts & Humanities Citation Index (A&HCI) listed journals, publications (for comparing research performance evaluation between international and China’s domestic indices, see Liang (2003)), and the number of collaborative national grants, indicate the economic, social, and cultural capital, respectively.

Figure 2.

Theoretical model specification.

Hypotheses

Science in China, like other sectors of the Chinese system, has been reigned in under a Soviet-style of top-down planned control, although decentralization of control exists to the extent that the government feels politically safe (see Yang et al., 2007). Both scientists and universities are subject to direct orders from higher entities on the one hand, whereas research institutions compete for resources and power among each other, on the other hand (Yang & Welch, 2012). Such a structural idiosyncrasy of China’s science system influences agency without doubt. Therefore, the accumulation of competitive advantage of scholars in China is not merely determined by academic ability or any single one capital but the totality of all forms of abilities to capitalize on available resources.

Hypothesis 1

Among all the forms of capital, social capital has been widely studied with respect to its antecedents as well as impact in a society (e.g., Lawson et al., 2008). Bourdieu (1986, p. 51) defined social capital as “the aggregate of the actual or potential resources which are linked to possession of a durable network of more or less institutionalized relationships of mutual acquaintance and recognition.” In the context of the research community, Luukkonen (1990) discovered that social capital indicated by social relations in the scientific community affects researchers’ citations received.

Social capital could take on various forms, but the number of collaborators could be one of the crucial indicators. Some scholars (Garfield, 1999; Leimu & Koricheva, 2005) have reported that the number of authors primarily affects citation rates rather than IFs. In the context of this study, the number of authors is indicated by the number of grants reported in the paper in addition to the principal investigator’s (PI’s) grant. As a result, the number of collaborative grants has a positive effect on the choice of tiers of journals (H1).

Hypothesis 2

In addition, international collaboration is another form of valuable social capital. Numerous scholars (Adams, 2013; Aksnes, 2003; Arunachalam et al., 1994; Glänzel & de Lange, 2002; Tang, 2013; Wagner et al., 2017) have discovered that highly cited articles are typically authored by a large number of researchers and often involve international collaboration. This social network capital is particularly true for Chinese scholars because their first language is not English, and most high-impact journals published overseas are produced in English (also see Li et al., 2015; Tang, 2013). Although the effect of the number of authors on IFs is debatable, the presence of foreign collaborators in authorship may be influential in affecting the publishing probability in high-impact journals. Therefore, collaboration with foreign scholars may have a positive effect on the choice of tiers of journals (H2).

Hypothesis 3

More than 100 international journals that are listed in the Science Citation Index (SCI) are based in China (Wagner & Wong, 2012). Publishing in China-based international journals is relatively easier for Chinese scholars because of professional connections and other non-academic cultural factors (Jiang, 2007). Nonetheless, networking with these journals needs not only social capital and sometimes economic capital, but also cultural capital. Chinese scholars are familiar with the habitus as well as favored research objects of these journals, and even endowed with institutionalized privileges (these are embodied, objectified, and institutionalized cultural capital, according to Bourdieu (1986)). However, such cultural capital is a double-edged sword. That is, scholars who put considerable effort into securing such capital will instead choose lower impact journals in general (H3).

Hypothesis 4

Economic capital is generally considered as the root of other forms of capital (Bourdieu, 1986). Research funding is one of the most essential and precious forms of economic capital. Although its effect on IFs has been well explored, findings are mixed. Gaughan and Bozeman (2002) and Campbell et al. (2010) maintained that research productivity might not have a direct relationship to the magnitude of funding (also see Larivière et al., 2010). Nevertheless, despite the fact that amounts of grants are often related to field- and discipline-specific dynamics, such as the expense of equipment, certain scholars (Defazio et al., 2009; Lee & Bozeman, 2005; Milesi et al., 2014) reported that a relationship exists between the amount of the grant and publication productivity. A consensus has not yet been reached; however, it is hypothesized that a positive relationship exists between the amount of funding and the choice of tiers of journals (H4).

Hypothesis 5

As suggested by the theories reviewed above, practices or strategies mediate the effects of other predictors on performance. Although citation rates mathematically determine IFs, the research practices are quite divergent. For instance, Seglen (1997) found that some authors select journals based on their IFs prior to submission of an article. Papers published in journals with larger IFs are generally cited more often (Tang, 2013), albeit this has been challenged by Finardi (2013). Consequently, the choice of tiers of journals should mediate the effects of resource (capital) factors, such as the total number of collaborative grants (Hypothesis 5a), number of foreign coauthors (Hypothesis 5b), amount of grant funds (Hypothesis 5c), and number of publications in China-based SCI/SSCI/A&HCI journals (Hypothesis 5d), on research performance, specifically the average citation rate of articles (ACA hereinafter) generated by a grant (Callaham et al., 2002).

In addition to the hypotheses proposed above, two research questions were raised according to the suggestions of the editor and an anonymous reviewer:

Do scholars with a higher return on investment (ROI) (citations per dollar), which is a performance measure used to evaluate the efficiency of research funding, continue to dominate research funding?

Are scholars’ publication records linked to succeeding funding?

Methods

Data

Publications funded by major Chinese grants

Information concerning bibliometric measures was obtained from the Web of Science (WoS). For analysis of the Web of Science database, we used the keywords “FO=National*” combined with “CU=China” to search for publications funded by China’s national grants. The search yielded over 830,000 results. After irrelevant records were removed, the attribute information of 808,247 journal articles funded by major grants in China was retrieved.

NSFC and NSSFC grant information

We retrieved grant information from two of China’s national funding initiatives; data from the National Natural Science Foundation of China (NSFC) were retrieved from ScienceNet (http://fund.sciencenet.cn), and data from the National Social Science Foundation of China (NSSFC) were retrieved from the database located at http://fz.people.com.cn/skygb/sk/index.php/Index/seach. The data analysis revealed 315,862 and 28,498 grants, ranging from 1999 through 2016.

To test the hypotheses of interest, we derived a new dataset by merging the databases of WoS, NSFC, and NSSFC with the grant number as the unit of analysis. The sample size is 339,302.

Measures

The predictors, the mediator, and the outcome variable are measured by mostly bibliometric measures.

Dependent variables

The dependent variables include research performance and research strategies, which are measured by the two primary bibliometric measures, that is, ACA and average impact factor (AIF), respectively.

AIF

Because yearly IFs of some journals have changed dramatically, the 5-year IF better captures the stable quality of journals (Amin & Mabe, 2000). Journal Citation Reports (JCR) accessed through the WoS database provides these data. Moreover, because the grant is the unit of analysis, IFs of all journal publications funded by a single grant are simply averaged to form the AIF.

ACA

The number of citations is recorded for each journal article in the database of Web of Science. The number of citations of all articles funded by a grant is simply averaged to derive the ACA, which reflects the average actual impact or performance of a grant or the PI.

Independent variables

Independent variables are three forms of capital mentioned above, that is, social capital, cultural capital, and economic capital, which are respectively measured by the number of foreign coauthors and number of collaborative national grants, number of publications in China-based SCI (Sciences Citation Index)/SSCI (Social Sciences Citation Index)/A&HCI (Arts & Humanities Citation Index) journals, and the amount of grant. All measures but the amount of grant (data were from the NSFC and NSSFC) used the data provided by the WoS database.

Results

Descriptive Analysis

Before testing hypotheses, a series of descriptive analyses were conducted.

The characteristics of Chinese scholars’ publications

China’s social scientists account for only 1.60% (the sum of the number of SSCI and A&HCI listed articles divided by the total number of SSCI, A&HCI, and SCI listed articles of Chinese scholars) of all of the publications authored by Chinese scholars. In total, 45.83% of Chinese scholars’ articles are published in journals whose IFs are in the first quantile. The AIF of these journals is 2.93 (SD = 2.76), and the maximum is 54.39. The ACA is 6.56 (SD = 18.10), with a maximum of 2,610, and 76.49% of the articles have been cited at least 10 times. Besides, 20.6% are collaborations with foreign authors, and 1.81% are published in China-based international journals, which have an AIF of 0.64.

How China’s major funding initiatives have distributed grants

The NSFC (representing science) and NSSFC (representing social sciences) have distributed over 30.24 billion U.S. dollars over the past 20 years, of which the NSFC accounts for 94.35%. Universities included in “Project 985” have received half of the grants. Provinces with more key universities (listed in either “Project 985” or “Project 211,” two of which aimed to strengthen the competitiveness of certain selected universities in China (see Huang, 2005; Yang & Welch, 2012), but have been replaced with the “Double First-Class” program in September 2017) and are more affluent (based on gross domestic product [GDP] per capita) have secured more grants, and nearly half of the total number of grants (46.90%) have been distributed in Beijing, Shanghai, Jiangsu, Guangdong, and Hubei (also see Chu & Li, 2000; Tang, 2013). $R^{2}$ s are above .70 for the correlations between the number of grants and the number of key universities, and $R^{2}$ is .524 for the correlation between the number of grants and GDP per capita. NSFC grants were primarily distributed to universities and research institutes, while approximately 10% of NSSFC grants² were distributed to non-research-related organizations such as governments, the subordinate organizations of the Communist Party of China, media, and even commercial firms.³

Hypothesis Testing

Multiple-group analysis of structural invariance

To examine whether AIF mediates the effects on ACA (average citation of articles) and whether the grant type (NSFC vs. NSSFC) moderates the structural relationships, a multi-group mediation analysis was performed. The multiple-group analysis of structural invariance is used to test whether differences observed in the structural parameters across groups are statistically significant. The analysis begins by fitting a model to the data for each group considered separately, with none of the parameters constrained to be equal across groups. This unconstrained model serves as a baseline model and is compared to constrained models, in which constraints are placed by specifying parameters of interest to be equal across groups (Koufteros & Marcoulides, 2006). In light of the principles stated above, two models (i.e., the un-constrained baseline model, and the constrained model) in which all of the path coefficients were constrained to be invariant between the two groups, were compared. Given the complexity of the data, the unconstrained and constrained structural models were compared by conducting chi-square tests for nested models based on Log-likelihood values and scaling correction factors obtained by the MLR estimator (see Asparouhov & Muthén, 2006). A significant $χ^{2}$ difference represents a rejection of the null hypothesis (that the parameters are equal), whereas a non-significant $χ^{2}$ indicates otherwise.

Following the above-mentioned procedures, the multi-group mediation analysis with a bias-corrected bootstrap procedure (5,000 resamples) was performed, and we determined that the constrained model was insignificant compared to the baseline model, $χ_{d i f f e r e n c e}^{2}$ (df = 13) = 10.925, p = .617. The model with equal path coefficients across the groups (NSFC vs. NSSFC) fits the data better, $χ^{2}$ (df =13, N = 143,026) = 21.849, p = .06; standardized root mean squared residual (SRMR) = .002; root mean square error of approximation (RMSEA) = .003; comparative fit index (CFI) = 1; Tucker-Lewis index (TLI) = 1, with the addition of a few predictions beyond the proposed theoretical model by means of modification indices.

Results of hypothesis testing

As reported in Table 1, both direct effects and indirect effects of all exogenous variables on ACA are significant. AIF, therefore, only partially mediates the effects of exogenous variables. R squared values of citation rates in the group NSFC and NSSFC are .525 and .306, respectively, which indicates that overall, the model has satisfactory explanatory power. In addition, although all of the effects of predictors on AIF were significant, the positive effects of the number of collaborative grants (Hypothesis 1, $β = - . 052, p < . 001)$ and the amount of grant (Hypothesis 4, $β = - . 003, p < . 001)$ were reversed and hence rejected. In view of the test results reported above, Hypothesis 2 (positive effect of foreign co-authorship, $β = . 147, p < . 001),$ Hypothesis 3 (negative effect of cultural capital, $β = - . 084, p < . 001),$ Hypothesis 5a (the indirect effect of the total number of grant collaborators on ACA, $β = - . 007, p < . 001),$ Hypothesis 5b (the indirect effect of the number of foreign collaborators on ACA, $β = . 018, p < . 001),$ Hypothesis 5c (the indirect effect of the amount of grant funds on ACA, $β = . 000, p < . 05),$ and Hypothesis 5d (the indirect effect of the number of publications in China-based SCI/SSCI/A&HCI journals on ACA, $β = - . 011, p < . 001)$ were supported (see Table 1 and Figure 3).

Table 1.

Results of Multi-Group Mediation Analysis.

DVs	Variables	Direct effect	Indirect effect	Total effect	Hypothesis
ACA	Amount of grant	.017***	.000*	.016***	H5c
	Foreign co-authorship	.169***	.018***	.188***	H5b
	China-based international publications	.012***	−.011***	.002	H5d
	Total collaborative grants	.596***	−.007***	.589***	H5a
	AIF	.126***
AIF	Amount of grant	−.003***			H4
	Foreign co-authorship	.147***			H2
	China-based international publications	−.084***			H3
	Total collaborative grants	−.052***			H1
Intercepts	ACA	.000***
Intercepts	AIF	.001***
Residual variances	ACA	.373***
Residual variances	AIF	.99***

Note. Although H1 and H4 were significant, the directions of the effects were opposite to what is hypothesized. DV = dependent variable; ACA = average citation rate of articles; AIF = average impact factor.

and *** indicate the p value is less than .05 and .001, respectively.

Figure 3.

Result of model estimation.

Results of research questions

A multiple regression analysis was performed to answer the two research questions, in which the amount of grant was the dependent variable, and ROI and the number of publications were the independent variables. The higher ROI is, the lower the amount of grant is $(β = - . 121, p < . 001) .$ The higher the number of publications is, the greater the amount of grant is $(β = . 207, p < . 001) .$ There was a negative interaction effect of ROI and the number of publications $(β = - . 096, p < . 001) .$ That is, when ROI is higher, the higher the number of publications, the lower the amount of grants. Corresponding to the questions raised above, scholars with higher ROI will not dominate research funding, but scholars’ publication records were linked to succeeding funding.

Discussion and Conclusion

Capital Discount

There are some quite counterintuitive findings concerning the effects of the number of collaborative grants (Hypothesis 1) and the amount of grant (Hypothesis 4) on the choice of tiers of journals. Although significant, their effects were opposite to what hypothesized. That is, both the effects of the number of collaborative grants and the amount of grant had negative effects. Furthermore, given the negative effect of cultural capital on the choice of tiers of journals (Hypothesis 3), such a result could be attributable to the phenomenon of so-called “capital discount,” a term inspired by the concept of cultural discount (see Hoskins & Mirus, 1988). That is, the presumed positive effect of capital on the choice of high-impact journals is discounted by the decreased motivation due to the excessive possession of resources. The reason why the motivation is decreased when possessing excessive resources may be that needed resources exceed the threshold level, and this causes the satiation effect (see Mojzisch & Schulz-Hardt, 2007; Reutskaja & Hogarth, 2009). In addition, such a finding may confirm the totality of the Bourdieusian view of research practice, which is determined by habitus in addition to capital, and field, in which habitus is formed and capital is distributed, that is,, the structural factor (Bourdieu, 1984).

Positive Effect of Social Capital

As expected, Hypothesis 2 (a positive effect of foreign co-authorship) was supported. In general, foreign collaboration has been effective in terms of producing high-impact publications (Glänzel & de Lange, 2002; Glänzel & Schubert, 2001). Chinese research institutions have attached greater importance to high-impact publications, but only the negligible portion of prestigious international journals are based in China. In view of the importance and effectiveness, these institutions, orchestrated by the Chinese government, who launched a series of initiatives (such as Thousand Talents Program, or Qianren Jihua) to lure foreign top scholars to collaborate with the Chinese counterparts, have been craving for international collaboration (Li et al., 2015; Lundh, 2011).

Indirect Effects of the Resource Predictors on Research Performance

According to the results of mediation analysis (hypothesis 5), the mediator, that is, AIF, partially mediates the effects of exogenous variables on ACA in that all the resource predictors have significant effects on research performance (measured by ACA). Moreover, AIF strengthens the positive effect of foreign co-authorship, whereas it weakens the effects of the amount of grant, the number of grants collaborated on, and publication in China-based SSCI/SCI/A&HCI journals. In addition, collaboration with foreign scholars has been beneficial in achieving both AIF and ACA.

Total Effects of the Resource Predictors on Research Performance

Due to mostly the competing role that the mediation effects play, some of the total effects of the resource variables on research performance have changed compared to the direct effects. As seen in Table 1 with respect to the total effects of the predictors on research performance, we conclude that all the forms of capital but cultural capital (measured by publishing in China-based SCI/SSCI/A&HCI journals) contribute to the research performance of scholars. This finding generally confirms the applicability of the proposed theoretical framework and is consistent with the classic capital theory of Bourdieu (1975, 1984, 1986). However, not all the effects of capital are equally important. The social capital, that is, the number of collaborative grants and foreign coauthors, play the most critical role, and yet the economic capital, that is, the amount of grant, and the cultural capital, that is, the number of publications in China-based SCI/SSCI/A&HCI journals, exert the minor or inconsequential role in affecting research performance.

Implications of the Findings

Theoretical implications

The author has tried to transform some grand theories from the humanistic paradigm (e.g., Bourdieu’s capital theory) into an empirical-based middle-range theory (Merton, 1968b) so that the study can propose a theoretical model that explains research performance. This might have made a theoretical contribution to the literature.

Practical implications

This study also has practical implications. In light of the findings on the unequal importance of capital factors, the university’s management should encourage international collaboration and help faculty expand their social networks. Moreover, submission to international journals instead of domestic English journals should be given greater priority, and yet research funding should not be stressed in tenure and promotion (T&P) review practices of assessing faculty’s performance (cf. García & Sanz-Menéndez, 2005).

Besides, considering the above-mentioned finding of “capital discount,” funding agencies should decide on the amount of funds of each grant proposal based on needs case by case, rather than evenly distribute funds among successful applicants, which has been a long-time practice in China, particularly in the area of the social sciences.

Methodological implications

As previously stated, China’s scientific achievement has received significant international attention, but few studies have thoroughly analyzed and reported on specific outcomes. This study scraped big data related to China’s grant information and publications funded by these grants. We have not only described the data but also presented them interactively in a web application (https://public.tableau.com/profile/publish/my_submission/Story1#!). Data were further analyzed using innovative modeling techniques. This work is the first study to adopt multi-group mediation analysis in the context of research evaluation, as far as the author is aware. Consequently, this study has made methodological contributions.

Limitations

This study has limitations. Some factors in the Bourdieusian view of practice, for example, habitus, were not tested due to the problem of data availability. Likewise, the concept of field (Bourdieu, 1984) is confined to the context of the Chinese science system. Consequently, this proposed theoretical framework is not meant to be applied to any science systems, but only applicable in non-autonomous or semi-autonomous science systems similar to China’s. In addition, the unit of analysis is the grant; data were retrieved from two major national funding initiatives, the NSFC (concerning science) and NSSFC (concerning social sciences). Although these two agencies are the primary funders of research, grants are also available from certain ministries and local governments. Hence, it is possible to compare the differences in certain bibliometric measures among studies funded by the aforementioned grants and non-funded studies by expanding the publication search in WoS and available grant data in China.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by a grant of National Social Science Fund of China (grant NO.: 18BXW082).

ORCID iD

Guangchao Charles Feng

Notes

Author Biography

Guangchao Charles Feng (PhD, Hong Kong Baptist University) is a distinguished professor in the College of Communication, Shenzhen University, Shenzhen, China. His main research areas are research methods, new media studies, computational advertising, message effects, and political communication. He is currently serving on the editorial board of many respectable journals, including the senior editor of Cogent Social Sciences (Taylor & Francis), and the board members of Mass Communication and Society, and Chinese Journal of Communication.

References

Adams

(2013). Collaborations: The fourth age of research. Nature, 497(7451), 557–560. https://doi.org/10.1038/497557a

Aksnes

D. W.

(2003). Characteristics of highly cited papers. Research Evaluation, 12(3), 159–170. https://doi.org/10.3152/147154403781776645

Amin

Mabe

(2000). Impact factors: Use and abuse. International Journal of Environmental Science and Technology, 1(1), 1–6. https://doi.org/10.1007/BF03325810

Arunachalam

Srinivasan

Raman

(1994). International collaboration in science: Participation by the Asian giants. Scientometrics, 30(1), 7–22. https://doi.org/10.1007/BF02017209

Asparouhov

Muthén

(2006). Robust chi square difference testing with mean and variance adjusted test statistics. Matrix, 1, 1–6.

Bazeley

(2010). Conceptualising research performance. Studies in Higher Education, 35(8), 889–903. https://doi.org/10.1080/03075070903348404

Bordons

Fernández

Gómez

(2002). Advantages and limitations in the use of impact factor measures for the assessment of research performance. Scientometrics, 53(2), 195–206. https://doi.org/10.1023/A:1014800407876

Bornmann

Williams

(2017). Can the journal impact factor be used as a criterion for the selection of junior researchers? A large-scale empirical study based on ResearcherID data. Journal of Informetrics, 11(3), 788–799.

Bourdieu

(1975). The specificity of the scientific field and the social conditions of the progress of reason. Information (International Social Science Council), 14(6), 19–47. https://doi.org/10.1177/053901847501400602

10.

Bourdieu

(1984). Distinction: A social critique of the judgement of taste. Harvard University Press.

11.

Bourdieu

(1986). The forms of capital. In Szeman

Kaposy

(Eds.), Cultural theory: An anthology (Vol. 1, pp. 81–93). Greenwood.

12.

Callaham

Wears

R. L.

Weber

(2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Journal of the American Medical Association, 287(21), 2847–2850. https://doi.org/10.1001/jama.287.21.2847

13.

Campbell

Picard-Aitken

Côté

Caruso

Valentim

Edmonds

. . . Bastien

(2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83.

14.

Chu Ng

S. K.

(2000). Measuring the research performance of Chinese higher education institutions: An application of data envelopment analysis. Education Economics, 8(2), 139–156.

15.

Daniel

H. D.

Fisch

(1990). Research performance evaluation in the German university sector. Scientometrics, 19(5), 349–361. https://doi.org/10.1007/BF02020698

16.

Defazio

Lockett

Wright

(2009). Funding incentives, collaborative dynamics and scientific productivity: Evidence from the EU framework program. Research Policy, 38(2), 293–305. https://doi.org/10.1016/j.respol.2008.11.008

17.

Durieux

Gevenois

P. A.

(2010). Bibliometric indicators: Quality measurements of scientific publication. Radiology, 255(2), 342–351. https://doi.org/10.1148/radiol.09090626

18.

Feng

G. C.

Zhang

Lin

(2019). A meta-analysis of the effects of socio-demographic factors on the social media adoption. International Journal of Communication, 13, 1996–2025. https://ijoc.org/index.php/ijoc/article/view/10088/2637

19.

Finardi

(2013). Correlation between Journal Impact Factor and Citation Performance: An experimental study. Journal of Informetrics, 7(2), 357–370. https://doi.org/10.1016/j.joi.2012.12.004

20.

Ford

J. D.

Schellenberg

D. A.

(1982). Conceptual issues of linkage in the assessment of organizational performance. Academy of Management Review, 7(1), 49–58. https://doi.org/10.5465/AMR.1982.4285450

21.

Froghi

Ahmed

Finch

Fitzpatrick

J. M.

Khan

M. S.

Dasgupta

(2012). Indicators for research performance evaluation: An overview. BJU International, 109(3), 321–324. https://doi.org/10.1111/j.1464-410X.2011.10856.x

22.

García

C. E.

Sanz-Menéndez

(2005). Competition for funding as an indicator of research competitiveness. Scientometrics, 64(3), 271–300.

23.

Garfield

(1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.

24.

Garfield

(2001). Impact factors, and why they won’t go away. Nature, 411(6837), 522–522. https://doi.org/10.1038/35079156

25.

Gaughan

Bozeman

(2002). Using curriculum vitae to compare some impacts of NSF research grants with research center funding. Research Evaluation, 11(1), 17–26. https://doi.org/10.3152/147154402781776952

26.

Glänzel

de Lange

(2002). A distributional approach to multinationality measures of international scientific collaboration. Scientometrics, 54(1), 75–89.

27.

Glänzel

Schubert

(2001). Double effort = double impact? A critical view at international co-authorship in chemistry. Scientometrics, 50(2), 199–214.

28.

Hessels

L. K.

Grin

Smits

R. E. H. M

. (2011). The effects of a changing institutional environment on academic research practices: Three cases from agricultural science. Science and Public Policy, 38(7), 555–568. https://doi.org/10.3152/030234211X12960315267976

29.

Hessels

L. K.

van Lente

(2011). Practical applications as a source of credibility: A comparison of three fields of Dutch academic chemistry. Minerva, 49(2), 215.

30.

Hicks

Wouters

Waltman

De Rijcke

Rafols

(2015). The Leiden Manifesto for research metrics. Nature, 520(7548), 429–431.

31.

Horta

Santos

J. M. J. S

. (2016). An instrument to measure individuals’ research agenda setting: The multi-dimensional research agendas inventory. Scientometrics, 108(3), 1243–1265. https://doi.org/10.1007/s11192-016-2012-4

32.

Hoskins

Mirus

(1988). Reasons for the US dominance of the international trade in television programmes. Media, Culture & Society, 10(4), 499–515. https://doi.org/10.1177/016344388010004006

33.

Huang

(2005). Qualitative enhancement and quantitative growth: Changes and trends of China’s higher education. Higher Education Policy, 18(2), 117–130. https://doi.org/10.1057/palgrave.hep.8300076

34.

Jauch

L. R.

Glueck

W. F.

(1975). Evaluation of university professors’ research performance. Management Science, 22(1), 66–75. https://doi.org/10.1287/mnsc.22.1.66

35.

Jiang

(2007). The transformation of core journals in China 2. http://shc2000.sjtu.edu.cn/0702/xueshuj.htm

36.

Ketcham

C. M.

(2007). Predicting impact factor one year in advance. Laboratory Investigation, 87(6), 520–526. https://doi.org/10.1038/labinvest.3700554

37.

Koufteros

Marcoulides

G. A.

(2006). Product development practices and performance: A structural equation modeling-based multi-group analysis. International Journal of Production Economics, 103(1), 286–307. https://doi.org/10.1016/j.ijpe.2005.08.004

38.

Kurmis

A. P.

(2003). Understanding the limitations of the journal impact factor. The Journal of Bone & Joint Surgery, 85(12), 2449–2454.

39.

Larivière

Macaluso

Archambault

É.

Gingras

(2010). Which scientific elites? On the concentration of research funds, publications and citations. Research Evaluation, 19(1), 45–53.

40.

Latour

Woolgar

(1982). The cycle of credibility. MIT Press.

41.

Lawler

(2011). Symbolic capital. In Encyclopedia of consumer culture (pp. 1418–1421). Sage. https://doi.org/10.4135/9781412994248.n532

42.

Lawson

Tyler

B. B.

Cousins

P. D.

(2008). Antecedents and consequences of social capital on buyer performance improvement. Journal of Operations Management, 26(3), 446–460. https://doi.org/10.1016/j.jom.2007.10.001

43.

Lee

Bozeman

(2005). The impact of research collaboration on scientific productivity. Social Studies of Science, 35(5), 673–702.

44.

Leimu

Koricheva

(2005). What determines the citation frequency of ecological papers? Trends in Ecology & Evolution, 20(1), 28–32. https://doi.org/10.1016/j.tree.2004.10.010

45.

Leisyte

Enders

De Boer

(2008). The freedom to set research agendas—Illusion and reality of the research units in the Dutch universities. Higher Education Policy, 21(3), 377–391.

46.

Miao

Yang

(2015). How do alumni faculty behave in research collaboration? An analysis of Chang Jiang Scholars in China. Research Policy, 44(2), 438–450. https://doi.org/10.1016/j.respol.2014.09.002

47.

Liang

(2003). Evaluating China’s research performance: How do SCI and Chinese indexes compare? Interdisciplinary Science Reviews, 28(1), 38–43.

48.

Lundh

(2011). Assessing the impact of China’s Thousand Talents Program on life sciences innovation. Nature Biotechnology, 29(6), 547–548.

49.

Luukkonen

(1990). Invited review article: Bibliometrics and evaluation of research performance. Annals of Medicine, 22(3), 145–150. https://doi.org/10.3109/07853899009147259

50.

The maze of impact metrics. (2013). Nature, 502(7471), 271. https://doi.org/10.1038/502271a

51.

Merton

R. K.

(1968a). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159(3810), 56–63.

52.

Merton

R. K.

(1968b). On sociological theories of the middle range. In Merton

R. K.

(Ed.), Social theory and social structure (p. 423). Free Press.

53.

Milesi

Brown

K. L.

Hawkley

Dropkin

Schneider

B. L.

(2014). Charting the impact of federal spending for education research: A bibliometric approach. Educational Researcher, 43(7), 361–370. https://doi.org/10.3102/0013189x14554002

54.

Moed

H. F.

(2002). The impact-factors debate: The ISI’s uses and limits. Nature, 415, 731–732. https://doi.org/10.1038/415731a

55.

Mojzisch

Schulz-Hardt

(2007). Being fed up: A social cognitive neuroscience approach to mental satiation. Annals of the New York Academy of Sciences, 1118, 186–205. https://doi.org/10.1196/annals.1412.006

56.

Morrison

(2014). China becomes world’s third-largest producer of research articles. Nature. https://doi.org/10.1038/nature.2014.14684

57.

Nature. (2002). Errors in citation statistics. Nature, 415(6868), 101–101.

58.

Noorden

R. V.

(2016). China by the numbers. Nature, 534, 452–453. https://doi.org/10.1038/534452a

59.

Potì

Reale

(2007). Changing allocation models for public research funding: An empirical exploration based on project funding data. Science and Public Policy, 34(6), 417–430. https://doi.org/10.3152/030234207X239401

60.

Reutskaja

Hogarth

R. M.

(2009). Satisfaction in choice as a function of the number of alternatives: When “goods satiate.” Psychology and Marketing, 26(3), 197–203. https://doi.org/10.1002/mar.20268

61.

Rip

Van der Meulen

B. J.

(1996). The post-modern research system. Science and Public Policy, 23(6), 343–352.

62.

Roessner

J. D.

(2002). Outcome measurement in the USA: State of the art. Research Evaluation, 11(2), 85–93. https://doi.org/10.3152/147154402781776880

63.

Seglen

P. O.

(1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314(7079), 498–502.

64.

Shi

Rao

(2010). China’s research culture. Science, 329(5996), 1128–1128. https://doi.org/10.1126/science.1196916

65.

Smith

D. R.

(2012). Impact factors, scientometrics and the history of citation-based research. Scientometrics, 92(2), 419–427. https://doi.org/10.1007/s11192-012-0685-x

66.

Smith

M. J.

Weinberger

Bruna

E. M.

Allesina

(2014). The scientific impact of nations: Journal placement and citation performance. PLOS ONE, 9(10), Article e109195. https://doi.org/10.1371/journal.pone.0109195

67.

Tang

(2013). Does “birds of a feather flock together” matter—Evidence from a longitudinal study on US–China scientific collaboration. Journal of Informetrics, 7(2), 330–344. https://doi.org/10.1016/j.joi.2012.11.010

68.

Wagner

C. S.

Whetsell

T. A.

Leydesdorff

(2017). Growth of international collaboration in science: Revisiting six specialties. Scientometrics, 110(3), 1633–1652.

69.

Wagner

C. S.

Wong

S. K.

(2012). Unseen science? Representation of BRICs in global science. Scientometrics, 90(3), 1001–1013.

70.

Weininger

E. B.

(2005). Foundations of Pierre Bourdieu’s class analysis. Approaches to Class Analysis, 4, 82–118.

71.

Witze

(2016). Research gets increasingly international. Nature. https://doi.org/10.1038/nature.2016.19198

72.

Xin

(2010, November 10). Defying critics, China’s science ministry defends research culture. Science. http://www.sciencemag.org/news/2010/11/defying-critics-china-s-science-ministry-defends-research-culture

73.

Yang

Vidovich

Currie

(2007). “Dancing in a cage”: Changing autonomy in Chinese higher education. Higher Education, 54(4), 575–592.

74.

Yang

Welch

(2012). A world-class university in China? The case of Tsinghua. Higher Education, 63(5), 645–666.

75.

Zuckerman

(1977). Scientific elite: Nobel laureates in the United States. Transaction Publishers.