Bibliometric Approach to Evaluating the Impact of a Building Interdisciplinary Research Careers in Women's Health K12 Research Career Development Program

Abstract

Background:

Mentored research career development programs are excellent training opportunities for junior faculty/early-stage investigators to transition into independent research careers. However, there is limited evidence that provides guidance on best practices for measuring the impact and reach of these programs, both for individual Scholars and the program as a whole. This article evaluates both the individual and overall impact of the Building Interdisciplinary Research Careers in Women's Health (BIRCWH) National Institutes of Health research career development award at the University of Minnesota.

Materials and Methods:

BIRCWH Scholars (n = 16) and a comparison group (n = 17) were evaluated on traditional metrics (e.g., publications, grant funding) in addition to bibliometrics (e.g., network growth, interdisciplinary collaborations, international reach, policy impact).

Results:

Traditional metric findings showed that BIRCWH Scholars had significantly more publications from pre- to post-BIRCWH experience than the comparison group and more grant funding. Bibliometric findings showed exponential network growth, interdisciplinary collaborations, international citations, and policy impact from pre- to post-BIRCWH Scholar experience.

Conclusion:

Findings from this evaluation have potential important implications. At the Scholar level, the results can be used to provide evidence of research impact in materials developed for merit review and promotion as well as in job and research grant applications. At the program level, the results can be used at the institutional level to gain broad administrative support and leverage additional funds for program activities and for evidence of program success for continuation funding from federal agencies.

Introduction

Mentored research career development programs supported by the National Institutes of Health (NIH) K12 Programs, the Patient-Centered Outcomes Research Institute (PCORI) Learning Health Systems Researchers K12 Program, and the NIH Clinical and Translational Science Institute (CTSI) Early Career Investigator KL2 program are excellent training opportunities for junior faculty/early-stage investigators (ESIs) to transition into independent research careers. These institutional K-award programs support and foster the interdisciplinary research careers of junior faculty, providing them with 75% (or 50% for surgeons) protected time to conduct their research and pairing them with senior investigators in a mentored, interdisciplinary scientific environment.

These research career development award programs are administered within academic institutions, they allow for direct training and mentoring of junior faculty (i.e., Scholars). In addition, institutional accountability ensures the necessary knowledge and skills are acquired for a scholar to lead their own research team(s) and conduct cutting-edge science.

However, there is limited evidence that provides guidance on best practices for measuring the impact and reach of these programs, both for individual Scholars and the program as a whole. Yet, progress reports and competitive renewal funding are dependent on showing Scholar and institution-level program impact. Prior research is also limited in evaluating the impact of NIH research career development training programs. The majority of previous studies have focused on evaluating postdoctoral trainees and fellows (e.g., T32s, F31s, F32s) and individual career development awards (e.g., K01, K08, K23), with few studies examining the impact of mentored career development programs.¹

One previous study that did evaluate NIH mentored research career development programs examined the impact of the overall Building Interdisciplinary Research Careers in Women's Health (BIRCWH) K12 program for ESIs.² They found that among 542 total BIRCWH Scholars ever trained on the K12 mechanism (across 39 institutions), most Scholars had applied for a competitive grant with approximately two-thirds successfully funded.² In addition, current BIRCWH Scholars had a 38% grant award success rate in comparison to the NIH overall ESI rate that averaged 29%.²

Another study focusing on research career development training programs advanced a process evaluation framework for systematic and continuous evaluation to improve junior faculty career development program quality.³ Utilizing logic models and process-based evaluation measures were proposed as tools to improve the quality of faculty research career development programs.³ In addition, some studies have evaluated the impact of independent K-awards given by NIH Institutes. For example, in 2012 the National Cancer Institute evaluated their career development mechanism and found that ESIs' funded on their K grant had higher grant funding and publications compared with a match comparison group (selected based on application priority scores of K applicants).⁴

While these prior studies have provided important new ideas and next steps for evaluating the impact of research career development programs, there is a need to show both individual Scholar and overall program impact over time, as well as broader reach beyond traditional grant awards and publication metrics. Evaluation of programmatic and Scholar impact based on policy influence, international reach, co-author/collaborator network growth, and interdisciplinary journal citations could provide important information demonstrating the impact of a research career development program beyond scholarly productivity. New innovative tools, including bibliometrics, may provide a promising tool for demonstrating impact, scope, and reach of both individual Scholars and research career development programs at the academic institution level.

Bibliometrics is a form of meta-research that refers to the quantitative analysis of research activities and outputs. This field represents an attempt to understand, quantify, and describe the impact of research through the development and use of surrogates or indicators. These indicators range from simple, easily understood measures to elaborate calculations. While traditional evaluation methods assess scholarly productivity, such as number of grants, publications, and presentations, bibliometric analysis can be used to describe the breadth of impact, both geographically and across disciplines, as well as growth and diversification of collaboration and citation networks.

Furthermore, bibliometrics have the capacity to show longer term impacts such as economic impact generated by patents and technology commercialization, clinical impact through improved diagnosis and patient satisfaction, and increased public engagement and policy change.^5,6 Programs and funding agencies employ bibliometrics as a data point in the evaluation of research investments.⁷ While there is no one gold standard for assessment, the use of a range of intentionally selected indicators in conjunction with qualitative assessment allows for an evidence-based approach to the evaluation of investments. In addition to bibliometrics being useful for showing impact and reach of research career development programs, these tools may also be useful for T32 pre- and postdoctoral programs, research networks, consortia, and individual promotion and tenure dossiers.

Using the University of Minnesota's (UMN) BIRCWH K12 program as an exemplar, the main aims of this article are to: (1) describe the process for utilizing bibliometrics to report program impact of junior faculty/ESIs research training programs; (2) report on the overall impact of the UMN's BIRCWH program; and (3) present individual BIRCWH Scholar case examples of different types of impact reports (i.e., network growth impact; international impact; interdisciplinary journal citations; and policy and media impact).

Materials and Methods

UMN's BIRCWH Program

The BIRCWH is an institutional mentored career development program that supports the development of research careers of junior faculty/ESIs (i.e., BIRCWH Scholars). One of the BIRCWH's primary aims is to connect junior faculty to an interdisciplinary team of senior faculty mentors with shared research interests in women's health and sex-difference research. Since the establishment of this K12 mechanism in 2000, 44 institutions supporting more than 700 junior faculty have been awarded by the Office of Research on Women's Health (ORWH) and BIRCWH program cosponsors from several institutes and centers.⁸

The UMN was awarded our first BIRCWH program grant in 2007, with two successful renewals in 2012 and 2017. As of 2020, there have been 16 UMN BIRCWH Scholars who have completed their award. The UMN BIRCWH program has a national reputation for interdisciplinary collaborations, including a variety of diverse disciplines from schools within and outside the health sciences, as well as diversity across Scholar, mentors, and faculty on the Interdisciplinary Advisory Committee and Leadership Team. Alumni from our program go on to be prolific Scholars with extramural and intramural funding in the realm of women's health and sex-difference research.

The UMN BIRCWH program has been diligent in assessing traditional key Scholar success metrics as determined by NIH, including publication rates and successful extramural funding. In addition, we use other metrics, including bibliometrics that portray more comprehensively interdisciplinary scholastic success (i.e., policy impact, network growth, etc.). Using both of these metric approaches, we advance our program mission of increasing the number of independently funded researchers who advance scientific knowledge in women's health and wellness across the lifespan and in sex/gender differences in health and disease. The Institutional Review Board (IRB) determined this study to be non human subjects.

Procedures

We studied BIRCWH Scholars (n = 16) and a comparison group (n = 17) of junior faculty from 2007 to February 2020. Our selection process for each group followed the method that the NIH uses to measure the impact of their individual career development K awards.^4,9,10 (i.e., comparing successful applicants to those applicants who were not funded). Our comparison group included all faculty who applied for the BIRCWH program and advanced to the final selection process but were ultimately not selected. We excluded five applicants who received a subsequent K12 award in another program because their experience in a mentored research career development program would be similar to the BIRCWH program. One BIRCWH Scholar who withdrew from the program soon after enrollment was also excluded.

We collected demographic and other background information (e.g., gender, race/ethnicity, terminal degree, discipline, postdoctoral research fellowship completion, surgeon specialty, time since BIRCWH application, current position [academic or nonacademic]) from both BIRCWH Scholars and the comparison group. We also obtained outcome data on publications, citations, and grants from both groups.

Program impact was further evaluated by assessing the reach of individual BIRCWH Scholars in four case examples. We examined bibliographic methods to illustrate individual Scholars' pre- and post-BIRCWH reach/growth, including: (1) network growth, (2) interdisciplinary journal citations, (3) international reach, and (4) health policy and media impact. To highlight the strength and capability of using bibliometrics to show growth/reach of individual BIRCWH Scholars, we intentionally chose specific Scholars who had excelled in these four areas.

Using data from the UMN BIRCWH program from 2007 to 2020, publication data were retrieved from Scopus for both BIRCWH Scholars and the comparison group of BIRCWH applicants. Scopus was selected as a large, interdisciplinary database containing ∼60 million publication records. Bibliographic information and citation information were extracted for all researchers in the two groups. Beyond times cited, characteristics of those citing publications, including author affiliations and journal subject area, were extracted for all included publications. Data were normalized using OpenRefine, an open-source desktop application designed for cleaning and normalizing data. Assessment of social and public impact was completed by the Policy and News Media Impact team at the UMN Libraries using 12 data sources summarized in Table 1.

Table 1.

Data Sources Used to Assess Social and Public Impact

	Policies and grey literature	News and media	Social media
Subscription-based resources
Access UN	X
Factiva		X
Gale student resources in context		X
ProQuest congressional Publications and executive branch documents	X	X	X
ProQuest news and newspapers		X
Publicly available resources
Altmetric explorer^a	X	X	X
Australian policy online	X
DuckDuckGo	X	X	X
Global voices		X
Google^b	X	X	X
OpenGrey	X
World news network		X

Users must request a personal account from Altmetric. These accounts are granted at the discretion of that company.

Searches were restricted using domain and format limits.

The 12 data sources were chosen based on the comprehensiveness of their coverage, the quality of available data, the transparency and accuracy of the data collection methods, and the flexibility and functionality of the tool. The tools represent a mixture of subscription-based and publicly available resources that provide broad coverage of news resources and policy materials across disciplines, time periods, and geographic regions. The methods used to generate these reports are fully described elsewhere.¹¹

Journal impact factor, subject categorization, and average journal impact factor percentile were retrieved from Journal Citation Reports (JCR). Grants data were retrieved from NIH RePORTER, which includes records of previously funded projects from all NIH Institutes and Centers (since 1993), the Administration for Children and Families (2013–2018), the Agency for Healthcare Research and Quality (since 1993), Centers for Disease Control and Prevention (since 1993), Health Resources and Services Administration (1993–1997), Food and Drug Administration (since 1993), and Veterans Affairs (since 2009). Searches were conducted for each Scholar and all retrieved records exported as spreadsheets. Collaborations associated with network growth were developed using VosViewer, while all other analyses and visualizations were completed using R version 3.6.0.

BIRCWH Scholar and comparison group demographic characteristics were summarized using descriptive statistics. We assessed for statistical differences between the two groups on demographic characteristics using chi-square tests. To determine statistical significance in scholarly productivity between groups (while controlling for years since applied to the BIRCWH program), we performed one-way analysis of covariance. One-way analysis of variance was utilized when controlling for years was not appropriate (i.e., preprogram metrics). A p-value of 0.05 was set for significance.

Results

Study sample

Characteristics of the 16 BIRCWH Scholars and the 17 BIRCWH applicant comparison group are summarized in Table 2. BIRCWH Scholars and applicants were similar in terms of gender and race/ethnicity but differed by terminal degree. A higher percentage of BIRCWH Scholars had PhDs, completed postdoctoral research fellowships, and were surgeons. The BIRCWH Scholars also came from a greater number of schools/colleges. A higher percentage of the comparison group had left academic employment at the time of this study.

Table 2.

Characteristics of Building Interdisciplinary Research Careers in Women's Health and Nonbuilding Interdisciplinary Research Careers in Women's Health Scholars

Characteristic	BIRCWH Scholars (n = 16)	BIRCWH applicant comparison Scholars (n = 17)	Chi-square, χ²(DF), p-value
Gender, n (%)			0.007 (1), 0.9346
Female	13 (81.3)	14 (82.4)
Male	3 (18.7)	3 (17.6)
Racial minority, n (%)	6 (37.5)	6 (35.3)	0.017 (1) 0.8953
Terminal degree, n (%)			2.991 (3), 0.3931
MD	4 (25.0)	7 (41.2)
PhD/ScD	11 (68.7)	9 (52.9)
MD and PhD	1 (6.3)	—
MD and Pharm D	—	1 (5.9)
Scholars with a postdoctoral research fellowship, n (%)	14 (87.5)	13 (76.5)	0.6740 (1), 0.4117
Research primary focus	13 (81.3)	10 (58.8)	1.963 (1), 0.1612
Clinical primary focus	1 (6.3)	3 (17.6)	1.005 (1), 0.3161
Surgeon specialty, n (%)			1.281 (1), 0.2577
Surgeon	3(18.8)	1 (5.9)
Nonsurgeon	13 (81.2)	16 (94.1)
School/division			7.167 (6), 0.3057
Kinesiology	1 (6.3)	—
Liberal arts	1 (6.3)	—
Medical school	8 (50.0)	13 (76.5)
Nursing	3 (18.6)	—
Pharmacy	1 (6.3)	1 (5.8)
Public health	2 (12.5)	2 (11.8)
Social work	—	1 (5.9)
Time since initial application, years			3.991 (3), 0.2624
3–5	4 (25.0)	2 (11.7)
6–8	3 (18.7)	4 (23.5)
9–10	5 (31.3)	2 (16.6)
>13	4 (25.0)	9 (52.9)
Left academic employment	1 (5.9)	4 (23.5)	1.914 (1), 0.1665

BIRCWH, Building Interdisciplinary Research Careers in Women's Health; DF, Degrees of Freedom.

Overall UMN BIRCWH program impact

Impact results comparing BIRCWH Scholars with the BIRCWH applicant comparison group are first reported by traditional metrics (i.e., publications, grants) and then bibliometrics (e.g., network growth impact, international reach, interdisciplinary journal citations)

Traditional metrics

Publications

The overall impact of the BIRCWH program using traditional metrics of success is summarized in Tables 3 and 4, which compares BIRCWH Scholars to the comparison group of BIRCWH applicants.

Table 3.

Productivity and Impact of Building Interdisciplinary Research Careers in Women's Health and Comparison Scholars

	BIRCWH Scholars (n = 16)	BIRCWH applicant comparison Scholars (n = 17)	ANCOVA
	BIRCWH Scholars (n = 16)	BIRCWH applicant comparison Scholars (n = 17)	Df	F value	p>F
Before BIRCWH application^a			1	3.02	0.0924
Total publications	264	178
Mean publications per author	16.5	10.5
Total peer-reviewed research articles	238	151
Mean peer-reviewed research articles per author	14.9	8.9
After BIRCWH application			1	7.21	0.0117
Total publications	736	475
Mean publications per author	46.0	28.0
Total peer-reviewed research articles	619	390
Mean peer-reviewed research articles per author	38.7	22.9
Total			1	8.87	0.0057
Total publications	1,000	653
Mean publications per author	62.5	38.4
Total peer-reviewed research articles	857	541
Mean peer-reviewed research articles per author	53.6	31.8

One-way ANCOVA determining statistical significance between groups by the variable of interest, controlling for years since applied for the UMN BIRCWH program. Bolded text indicates statistical significance and p < 0.05.

One-way ANOVA determining statistical significance between groups on the variable of interest.

ANCOVA, analysis of covariance; ANOVA, analysis of variance.

Table 4.

Grants Retrieved from National Institutes of Health Reporter With Scholars As Principal Investigators Or Co-Principal Investigators

	BIRCWH Scholars (n = 16)		BIRCWH applicant comparison Scholars (n = 17)		Difference, pre(post)
	Pre	Post	Pre	Post	DF	Value	p
Grants received (n)	4	20	3	10	1 (1)	0.18 (1.45)^a	0.6702 (0.2375)^a
PIs or Co-PIs (n)	4	10	2	6	1 (1)	0.94 (2.48)^a	0.3399 (0.1256)^a
Awarded funds (USD)	$548,005	$48,239,932	$1,073,017	$9,859,946	1 (1)	0.25 (1.56)^a	0.6222 (0.2218)^a
Funding mechanisms as PI or other investigator (n)					—	—	—
R-level	3	17	1	9	—	—	—
U-level	0	2	0	1	—	—	—
K-level	0	1	1	0	—	—	—
F-level	1	0	0	0	—	—	—
M-level	0	0	1	0	—	—	—

One-way ANCOVA determining statistical significance between groups by the variable of interest.

Controlling for years since applied for the UMN BIRCWH program.

PIs, principal investigators; UMN, University of Minnesota.

The BIRCWH Scholars produced 1,000 publications, 264 (Mean = 16.5/Scholar) of which were produced before their BIRCWH award and 736 (Mean = 46.0/Scholar) of which occurred after their award, resulting in an average growth percent change per Scholar as a 178.8% increase. The comparison group produced 653 publications, 178 (Mean = 10.5/comparison group member) of which were produced before applying for the BIRCWH award, and 475 (Mean = 28.0/comparison group member) of which occurred after their award, resulting in an average growth percent change per comparison group member as a 166.7% increase. There were no significant differences between BIRCWH Scholars and the comparison Scholars in preapplication publications, F(1) = 3.02, p = 0.0924.

After controlling for years since the BIRWCH application, BIRCWH Scholars published significantly more postapplication publications, [F(1) = 7.21, p = 0.012] and more total publications [F(1) = 8.87, p = 0.006] than the comparison Scholars. The majority of the pre- and post-BIRCWH application publications (85.7%; n = 857) were original research articles. The 857 research articles were published in 367 scholarly journals, including 325 journals with journal impact factors; 137 (42.2%) of these journals ranked in the top quartile of journals as ranked by average journal impact factor percentile, and these top-ranked journals published 347 (40.5%) of the 857 articles.

The comparison group had 653 total publications, including 539 research articles (82.5%). These 539 articles were published in 313 scholarly journals, 279 of which have journal impact factors, with 119 (42.7%) of these journals ranked in the top quartile by average journal impact factor percentile.

Grant funding

Before participation in the BIRCWH program, four BIRCWH Scholars had each received one extramural research grant, including two R36 awards and one R03 award. Awarded funds for these grants totaled $548,005. In the comparison group, three had each received an external research grant, including one R03 award, totaling $1,073,017 in awarded funds.

Following involvement in the BIRCWH program, 10 BIRCWH Scholars received 20 grants with a total post-BIRCWH application awarded value of $48,239,932. This included five R01s, seven R03s, and four R21s. Six comparison group members received 10 grants totaling $9,859,946 in funding. These grants included four R01s, one R03, and two R21 awards. Although the amount of funding that BIRCWH Scholars received (i.e., $48,239,932) was ∼$40,000 higher than the comparison group (i.e., $9,859,946), this difference was not statistically significant between groups, as data regarding individual Scholar funding dollars were skewed.

Individual Scholar bibliometric impact case examples

Network growth impact case example

Figure 1 presents an example of the impact of the BIRCWH program on one Scholar's collaboration network growth. This Scholar was in the inaugural BIRCWH cohort in 2007, is a PhD clinician and researcher, and is currently a tenured professor and Vice Chair for Research in a clinical department in the medical school. This Scholar's research expertise centers around examining risk and protective factors for childhood obesity disparities by gender and race/ethnicity. This Scholar's publications were separated into those produced before involvement in the BIRCWH program and those produced during and after the program.

FIG. 1.

Pre- and post-BIRCWH Network Growth Impact Example Depictions of one BIRCWH Scholar's collaboration network growth, including on the left (A) the Scholar's collaboration network before their appointment as a BIRCWH Scholar, and on the right (B) the Scholar's collaboration network after their appointment (e.g., February 2020). The circle size relates to the prevalence of collaborations (e.g., grants, publications) with the larger circle meaning a larger number of collaborations associated with that individual Scholar. The thickness of the network lines relates to the strength of the collaboration (e.g., number of coauthors). The colors represent different collaboration clusters. BIRCWH, Building Interdisciplinary Research Careers in Women's Health.

This BIRCWH Scholar produced 11 articles with 19 unique collaborators before BIRCWH involvement with an average of 3.3 co-authors per article, compared with 122 publications with 197 collaborators during and after BIRCWH involvement with an average of 4.6 co-authors per article. As can be seen, this Scholar began with a healthy set of collaborators on publications pre-BIRCWH award, but these collaborations exploded exponentially across the BIRCWH program and post-BIRCWH.

Interdisciplinary journal citations impact case example

To examine the impact of the program on citations in interdisciplinary research journals, the journal in which articles citing one BIRWCH Scholar's publications were examined. This Scholar was in the third BIRCWH cohort in 2017, is a PhD researcher, and is currently an associate professor in a School of Journalism. This Scholar's research examines exposure to health information in the media, with a focus on cancer prevention and screening in women.

Before involvement in the BIRCWH program, this Scholar's work had been cited 65 times in 43 journals. Considering only the period during and after BIRCWH involvement, the work was cited 590 times in 277 journals. Of the cumulative 320 journals, 261 are included in JCR, the resource that provides access to journal impact factors. JCR organizes journal impact factors into subject categories.

Subject classification of each journal was extracted from this data source. Citations before BIRCWH award were produced by journals in 13 different disciplines, most frequently Oncology and Public, Environmental, and Occupational Health. Citations during and after BIRCWH were found in journals associated with 45 different disciplines, most frequently in public health, environmental, and occupational health; communication; oncology; and general and internal medicine. The total 48 JCR Subject Categories were organized into 10 broader categories based on the work of Leydesdorff and Rafols,¹² which are presented in Table 5.

Table 5.

Example of Journal Citations for One Building Interdisciplinary Research Careers in Women's Health Scholar's Publications Organized by Discipline of Citing Journal

	Before BIRCWH			During and after BIRCWH			Total
	JCR categories	Journals	Citations	JCR categories	Journals	Citations	JCR categories	Journals	Citations
General medicine and health	4	15	23	7	89	237	7	104	260
Social sciences	3	5	11	8	32	116	10	37	127
Biomedical sciences	2	11	16	8	47	80	8	58	96
Neuroscience	2	2	2	7	26	39	8	28	41
Infectious disease	1	1	1	1	5	17	1	6	18
Clinical medicine	1	2	2	8	12	13	8	14	15
Computer science	0	0	0	2	7	10	2	7	10
Agriculture	0	0	0	2	5	7	2	5	7
Engineering	0	0	0	1	1	3	1	1	3
Environmental sciences	0	0	0	1	1	1	1	1	1
N/A	—	7	10	—	52	67	—	59	77
Total	13	43	65	45	277	590	48	320	655

JCR, journal citation reports.

International impact case example

An example of a BIRCWH Scholar's international impact is shown in Figure 2. This Scholar was in the inaugural BIRCWH cohort in 2007, is a PhD researcher, and is currently an associate professor and Vice President for Clinical Affairs in a Health Science Center. This Scholar's research expertise is in global women's health.

FIG. 2.

Pre- and post-BIRCWH International Impact Example Depictions of one BIRCWH Scholar's international impact, including on the top (A) the Scholar's cited publications on an international scale before their appointment as a BIRCWH Scholar, and on the bottom (B) the cited publications on an international scale after their appointment (e.g., February 2020). Each dot on the map represents citation(s).

Citations of this BIRCWH Scholar's work were assessed for both their pre- and post-BIRCWH research outputs. The institutional affiliations of authors of these citing works were extracted from Scopus and normalized using OpenRefine. This researcher's work was cited by 642 documents, produced by 1,757 authors. Work produced before the BIRCWH award has been cited by 263 researchers in 30 countries while work produced during and after the BIRCWH program has been cited by 1,494 individuals in 54 countries. As shown in Figure 2, this Scholar had some international citations pre-BIRCWH award, but the citations increased dramatically during and after the BIRCWH program.

Policy and media impact case example

Table 6 shows an example of the policy and media impact of a BIRCWH Scholar. This Scholar was in the second BIRCWH cohort in 2012, is a PhD researcher, and is a professor in a School of Public Health. This Scholar's research focuses on health care delivery and women's reproductive health. Since involvement in BIRCWH program, this Scholar's work has been referenced in 52 government and association documents produced by 41 different sources, including the Department of Health and Human Services, the U.S. Agency for International Development, the National Bureau of Economic Research, and the World Health Organization.

Table 6.

Summary of the Policy and Media Impact of One Building Interdisciplinary Research Careers in Women's Health Scholar

Category	Sources (n)	Documents (n)	Times republished
United States—Federal	4	10	N/A
United States—State and local	10	12	N/A
International—Federal	1	1	N/A
International—Regional and local	1	1	N/A
Organizations and associations	25	27	N/A
Newspapers and magazines	41	70	30
Websites	36	53	23
Television and radio	4	11	14
Press releases	8	11	42
Total	130	197	109

This BIRCWH Scholar's work was also referenced in 134 news stories from 81 popular media channels, including outlets such as the New York Times, the Washington Post, Forbes, CBS News, and Minnesota Public Radio. These news stories were subsequently republished in 67 additional venues, while press releases on this Scholar's research were published in 42 syndicated sources. These policy documents, guideline reports, and recommendation publications are powerful indicators of this Scholar's translational impact and reach. In addition, the media impact report shows this BIRCWH Scholar's research reaching communities and laypersons.

Discussion

Overall, bibliometric results showed that the BIRCWH program had both a strong impact on the research career development of individual Scholars, as well as a strong collective impact as an entire program compared with a comparison group.

At the collective program level, BIRCWH Scholars published more articles (1,000 vs. 653) and were awarded more NIH grants (20 vs. 10) compared with the comparison group. Results on the individual Scholar-level impact emphasize the high level of productivity of BIRCWH Scholars following their involvement in the research career development program. Specifically, on average, BIRCWH Scholars produced 36.5 peer-reviewed articles following their involvement, and cumulatively received almost $50 million in federal funding during that same time period. While these Scholars had published articles and some had received grant funding before the BIRCWH program, their involvement marked an inflection point, after which their productivity showed lofty increase.

At the individual BIRCWH Scholar level, bibliometric results showed strong impact of Scholar work using other important metrics that allow for demonstrating the scope and reach of individual Scholar impact. For example, exponential Scholar network growth, broadening of Scholar international publication impact, strong spread of interdisciplinary work, and policy impact and reach of individual Scholars pre- to post-BIRCWH were all demonstrated. Additionally, results included both standardized indicators, such as articles published and grants awarded, as well as more advanced bibliometric strategies to demonstrate the growth of authorship networks, international reach, and influence on policy and media.

These analyses highlight the diverse nature of the BIRCWH Scholars' impact. Their work is well cited in academic spaces, both in the United States and internationally, and is being used and referenced by researchers in a broad range of fields. Networks of collaborators see similar growth and diversification, as BIRCWH Scholars integrate with new groups of researchers. Beyond these academic impacts, the research of BIRCWH Scholars is communicated through popular media channels and is reflected in policy documents, reflecting public engagement and the effective mobilization of knowledge.

Results of this study also reflect the goals of the BIRCWH program overall: to foster collaboration, promote interdisciplinary research, and accelerate women's health and sex differences research productivity. By aligning analyses and program goals, it is possible to move beyond documenting the growth of individual researchers, to assessing the effectiveness of the program in meeting its objectives. Results also have implications for other federal research career development programs, in addition to pre- and postdoctoral programs. These training programs may want to consider using bibliometric analyses to compliment traditional metrics such as publication and grant funding to demonstrate program impact.

There were both strengths and limitations of this evaluation. One strength is that this is one of the first articles to demonstrate impact at both the program and individual level for federal research career development awards. In addition, the use of both traditional metrics and bibliometrics to demonstrate impact was a strength. Furthermore, the use of a comparison group to evaluate the impact of the program was an additional strength.

Limitations of this study include not being able to capture internal funding or other non-NIH funding for Scholars or the comparison group, which also demonstrates success. Another limitation of this study was that although our comparison group is comparable to our BIRCWH program across most characteristics (e.g., gender, race/ethnicity, terminal degree), the fact that these individuals were ultimately not selected for our program presents a limitation of nonequivalent groups, common to quasiexperimental designs.

Additionally, there were limitations with our comparison group that may have led to selection bias. For example, there were more doctors of medicine (MDs) in the comparison group than the BIRCWH Scholar group. MDs often have more clinical responsibilities that must be balanced with research and this may negatively impact productivity.¹³ Also, our Scholar sample size was relatively small and was from one BIRCWH program, thus limiting generalizability of findings.

Another limitation of our approach was that the bibliometric examples were meant to highlight novel approaches to examine individual BIRCWH Scholars and are not generalizable to all Scholars. Furthermore, citation, whether in scholarly materials, policy documents, or media resources, may not always indicate agreement or endorsement of ideas. While tools, such as SciTE, attempt to provide categorization of the sentiment of citations in academic articles, no similar tool exists for nonacademic sources. Moreover, even in academic spaces, the ability to categorize citations consistently across disciplines and at scale remains challenging.¹⁴ However, engagement, whether positive or negative, may still signal impact of the Scholar's work.

Conclusions

This article highlights how both traditional and bibliographic methods can be used to provide evidence of the visibility and overall impact of a research career development program. Traditional metrics of Scholar and program success provide promising opportunities to compare outcomes across local, regional, and national career development programs. Bibliometrics can provide additional metrics on Scholar productivity and impact in assessing program quality beyond the scope of traditional metrics.

Bibliometrics have value for the program and the individual Scholar. At the program level, the results can be used at the institutional level to gain broad administrative support and leverage additional funds for program activities. At the Scholar level, the results can be used to provide evidence of research impact in materials developed for merit review and promotion as well as in job and research grant applications.

Footnotes

Disclaimer

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Author Disclosure Statement

J.M.B., S.A., and B.T. are current UMN BIRCWH K12 program Principal Investigators. J.F.W. is a UMN BIRCWH K12 program Principal Investigator. K.M. is a current UMN BIRCWH K12 program Project Coordinator.

Funding Information

Research reported in this publication was supported by the Eunice Kennedy Shriver National Institute of Child Health and Human Development of the National Institutes of Health under Award No. K12HD055887.

References

Sherman

. Chapter 12: Evaluation of research training and career development programs at NIH: Current capabilities and continuing needs. In: National Research Council, enhancing philanthropy's support of biomedical scientists: Proceedings of a workshop on evaluation. The National Institutes of Health. Washington, DC: The National Academies Press,, 2006:111–124.

Guise

, Winter

, Fiore

, Regensteiner

, Nagel

. Organizational and training factors that promote team science: A qualitative analysis and application of theory to the National Institutes of Health's BIRCWH career development program. J Clin Transl Sci, 2017; 1:101–107.

Raymond

, Wyman

, Dighe

, Harwood

, Hang

. Process evaluation for improving K12 program effectiveness: Case study of a National Institutes of Health building interdisciplinary research careers in women's health research career development program. J Womens Health (Larchmont), 2018; 27:775–781.

Discovery

Logic

, a Thomson

Reuters business

. Outcome evaluation of the NCI career development (K) awards program. The National Cancer Institute, Center for Cancer Training, 2012. Available at: https://www.cancer.gov/grants-training/training/funding/kaward-report/full-kaward-report.pdf Accessed March 29, 2021 .

Greenhalgh

, Raftery

, Hanney

, Glover

. Research impact: A narrative review. BMC Med, 2016; 14:78.

Graham

, Chorzempa

, Valentine

, Magnan

. Evaluating health research impact: Development and implementation of the Alberta innovates-health solutions impact framework. Res Eval, 2012; 21:354–367.

Bakker

, McBurney

, Chew

, Aho

, Reed

. Implementing and assessing a service to demonstrate public impact of faculty research in news and policy journals. J Med Libr Assoc, 2019; 107:579–587.

Building Interdisciplinary Research Careers in Women's Health (BIRCWH). National Institute of Health (NIH) Office of Research on Women's Health (ORWH) website. Available at: https://orwh.od.nih.gov/career-development-education/building-interdisciplinary-research-careers-womens-health-bircwh Accessed March 29. 2021.

Bol

, Vaan

M de

, Rijt

A van de

. The Matthew effect in science funding. PNAS, 2018; 115:4887–4890.

10.

Haggerty

, Fenton

. Outcomes of early NIH-funded investigators: Experience of the National Institute of Allergy and Infectious Diseases. PLoS One, 2018; 13:e0199648.

11.

Rivera

, Kyte

, Aiyegbusi

, Calvert

. Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Med, 2017; 14:e1002370.

12.

Leydesdorff

, Rafols

. A global map of science based on the ISI subject categories. JASIST, 2009; 60:348–362.

13.

Bell

, Valsangkar

, Joshi

, et al. The role of PhD faculty in advancing research in Departments of Surgery. Ann Surg, 2017; 265:111–115.

14.

Nicholson

, Mordaunt

, Lopez

, et al. Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. bioRxiv. Published online January 1, 2021:2021.03.15.435418.