Scope,Influence,and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017

Abstract

The Clinical and Translational Science Awards (CTSA) program sponsors an array of innovative, collaborative research. This study uses complementary bibliometric approaches to assess the scope, influence, and interdisciplinary collaboration of publications supported by single CTSA hubs and those supported by multiple hubs. Authors identified articles acknowledging CTSA support and assessed the disciplinary scope of research areas represented in that publication portfolio, their citation influence, interdisciplinary overlap among research categories, and characteristics of publications supported by multihub collaborations. Since 2006, CTSA hubs supported 69,436 articles published in 4,927 journals and 189 research areas. The portfolio is well distributed across diverse research areas with above-average citation influence. Most supported publications involved clinical/health sciences, for example, neurology and pediatrics; life sciences, for example, neuroscience and immunology; or a combination of the two. Publications supported by multihub collaborations had distinct content emphasis, stronger citation influence, and greater interdisciplinary overlap. This study characterizes the CTSA consortium’s contributions to clinical and translational science, identifies content areas of strength, and provides evidence for the success of multihub collaborations. These methods lay the foundation for future investigation of the best policies and priorities for fostering translational science and allow hubs to understand their progress benchmarked against the larger consortium.

Keywords

translation CTSA bibliometrics publications research areas

For over a decade, considerable resources have been invested in building and maintaining a consortium of 64 Clinical and Translational Science Awards (CTSA) hubs across the United States (Califf & Berglund, 2010; Leshner, Terry, Schultz, & Liverman, 2013). Funded by the National Center for Advancing Translational Sciences (NCATS) of the National Institutes for Health (NIH) and sustained by the efforts and expertise of scientists, clinicians, educators, and community members at local research centers, the consortium’s common goal is to accelerate the translation of scientific discoveries into health practice. CTSA hubs, each with their own strengths and priorities, have been charged with taking a disease-agnostic approach to support many disciplines of high-quality research, emphasizing collaborative team science (Leshner et al., 2013; Reis et al., 2010). Given the time, work, and money invested in the consortium, it is necessary to evaluate the scope and influence of CTSA activities and outputs and their success in advancing clinical and translational science (Pincus, Abedin, Blank, & Mazmanian, 2013). Diverse evaluation methods have been used in the past to measure the impact of the CTSA program, including return on investment at various levels (Grazier, Trochim, Dilts, & Kirk, 2013), social network analysis of collaboration (Dozier et al., 2014), and mixed-methods approaches to assessing outcomes and processes (Wooten et al., 2014). In order to evaluate how CTSA-supported research has been shared and utilized across the academic community, one useful evaluation method is bibliometric analysis, or studying the publication and citation portfolios of a program. Bibliometrics can elucidate dissemination, linkages, and impact arising from publications, a narrow but pivotal step in the process of translating scientific discoveries into clinical use.

Publication data can be used to understand the topic areas and disciplinary scope of disseminated research. This includes defining the expected or intended audience by assessing characteristics of the journals in which the research is published. Citation data can be used to understand the utilization and influence of those publications across the academic literature. Although bibliometric analyses do not provide a complete picture of all scientific communication necessary for translation, the vast majority of biomedical knowledge that lays the foundation for clinical practice likely passes through the academic literature at some point. Therefore, measuring the qualities of supported publications is one method of characterizing the scope and influence of the work generated by research organizations such as the CTSA consortium.

Thus far, descriptions of the CTSA consortium’s overall publication and citation activity over the first 5 and 10 years of the program have been reported (Llewellyn, Carter, Rollins, & Nehl, 2018; Steketee, Frechtling, Cross, & Schnell, 2012). These studies showed that the quantity of publications supported by CTSA hubs is robust and accelerating over time and that their citation influence is consistently well above global averages for their fields and years of publication. The current study seeks to deepen and expand upon previous research by describing the research area content of the updated publication portfolio. The aim is to provide new exemplars for contextualizing publication productivity and impact according to different methods of content classification. The results of this article will first speak to the scope and diversity of the portfolio via the cumulative distribution and heterogeneity of research areas. Second, the results section will elaborate on the relative citation influence of the most frequently represented research areas using state-of-the-art citation impact metrics. Third, the results section will shed light on the collaborative, interdisciplinary structure of the publication portfolio by examining overlapping disciplinary designations to the same publications and by comparing the characteristics of publications supported by a single hub to those supported by multiple hubs. These complementary approaches to describing the portfolio, including illuminating areas of strength and opportunities for growth, relative influence across areas, and interdisciplinary collaboration, are intended to provide a multifaceted characterization of the bibliometric output associated with the CTSA program from its inception in 2006 through 2017.

Diversity of Scope Across Research Areas

The CTSA program aims to deliver resources to a diverse cross section of biomedical research endeavors, agnostic to any particular diseases or special interests. Coverage of research support and commensurate yields in published products are expected to be wide and far-reaching, especially when measured across hubs established at complementary research institutions nationwide. The collective interests and skill sets positioned at these hubs are likely to generate a publication set that is varied in scope and reasonably even in emphasis across research areas and across the translational spectrum (Leshner et al., 2013). Identifying predominant research areas addressed by CTSA-supported research and the balance of subjects in the publication portfolio is key to assessing the extent to which the CTSA consortium is providing unbiased support to a variety of relevant research endeavors. In this study, we examine the distributions of both narrow (individual subject area) and broad (general branch of research) research content categorizations, as well as publishing journals, to determine the diversity of scope in the publication set.

Relative Influence of Research Areas

Measuring citation activity around a publication portfolio is important for understanding the reach and influence of that research. Indeed, the influence of research publications has previously been proposed as an appropriate common metric for assessing products of the CTSA program (Rubio, 2013). CTSA-wide citation influence has been reported in the past (Frechtling, Raue, Michie, Miyaoka, & Spiegelman, 2012; Leshner et al., 2013; Steketee et al., 2012) but not with respect to disciplinary area. Because current citation influence metrics are benchmarked against other publications from the same research areas (Hutchins, Yuan, Anderson, & Santangelo, 2016; Llewellyn et al., 2018; Reuters, 2014), it is important to understand the disciplinary context within which articles are compared. This study utilizes multiple article- and journal-level citation metrics to triangulate aggregate and field-specific citation influence, allowing for a nuanced consideration of influence within and across research areas.

Interdisciplinarity and Collaboration Across CTSA Hubs

Translational science relies upon interdisciplinary research to combine complementary knowledge and skills that advance science into new frontiers (Hara, Solomon, Kim, & Sonnenwald, 2003; Leshner et al., 2013; Shi et al., 2011; Sosa, 2011). Scientists from different specialties may work in concert to conduct collaborative projects, or specialists in one field may draw upon or contribute to the knowledge base of other specialties. It is common today for innovative science to span multiple complementary disciplines rather than remain siloed within one area. By examining the intersecting disciplinary areas represented by publications, it is possible to understand how often findings are relevant to multiple areas and which areas most often overlap with one another. In this study, we utilized co-designations of broad disciplinary categories to each supported publication to quantify the extent of overlap between disciplines.

In addition to interdisciplinary overlap, another way to evaluate cooperative efforts is to examine publications resulting from the support of multiple CTSA hubs. As a consortium, it is possible to pool resources in support of ambitious projects that may not be possible without far-reaching or large-scale support. For instance, NCATS sponsors the Trial Innovation Network (Shah et al., 2017), which links CTSA hubs across the country to facilitate efficient, impactful, and pioneering multisite research that may merge disciplines that have not traditionally worked together. Multihub research of this type may have the advantages of reducing redundancies and uniting scholars with different perspectives, methods, and resources; enhancing collaboration across disciplinary silos; and accelerating progress (Bikard, Murray, & Gans, 2015; Cummings & Kiesler, 2007; Hara et al., 2003). We expected that multihub research may stand out in several ways: First, because certain types of research may lend themselves more readily to a multisite organizational structure, we suspected that a distinct distribution of research areas may be represented in multihub publications. Second, previous research suggests that multi-institutional hubs, made up of more than one separate research institution, support research that is higher in productivity (more publications) and influence (more citations; Llewellyn et al., 2018). Therefore, research conducted across multiple hubs, combining resources from multiple states or regions, likewise may be higher in citation influence. Third, we thought that multihub research may be more likely to come about from a need for larger, broader scope, and cooperative undertakings and thus may more often be crosscutting in disciplinary content.

Overview

In sum, we aim to characterize the CTSA-supported publication portfolio at the end of 11 years of the program, describing the distributional scope of research areas represented, placing citation influence within disciplinary context, and assessing interdisciplinary collaboration. This evaluation will aid in understanding the CTSA program’s cumulative contribution to clinical and translational science thus far and inform strategic management of resources moving forward. This methodology is also intended to be useful for examining other large, nationwide research and disease networks to reveal their relative value and influence to the wider research enterprise.

Method

Publication Data Collection

Expanding upon methods described in Llewellyn, Carter, Rollins, and Nehl (2018), in December 2017, we compiled CTSA grant numbers using the NIH Research Portfolio Online Reporting Tools (https://projectreporter.nih.gov/reporter.cfm). Comprehensive queries included all past and present CTSA funding mechanisms, including hub cooperative UL1 awards, smaller associated KL2/TL1 training awards, and supplemental awards funded by NCATS (where Institute/Center Code was TR) or its predecessor the National Center for Research Resources (where Institute/Center Code was RR). We linked each grant number to the corresponding hub and then queried all of the grant numbers we found in PubMed (https://www.ncbi.nlm.nih.gov/pubmed/) using the following search: “Grant #1” [Grant Number] OR “Grant #2” [Grant Number], and so on. This retrieved lists of publications that had formally cited each hub grant as of that time, including ePubs indexed ahead of print and publications that were or were not indexed in PubMed Central (NIH-funded publications are required to be publicly accessible via PubMed Central, but recent publications may not yet be indexed and older publications may predate the 2008 requirement [NIH Grant Notice Number: NOT-OD-08-033]). From these lists, we removed a small number of publications that were published before the first year of the cited grant as we assumed those grant citations to be erroneous. These grant citations may be attributable to General Clinical Research Center (Califf & Berglund, 2010) grants, which were precursors to the CTSA program, but would be considered outside the scope of this project. Using grant citations systematically identifies a significant proportion of publications resulting from research that was supported by CTSA resources that serve to facilitate translational research. It does not, however, account for publications which resulted from CTSA support but did not cite the appropriate grant, and it does not account for work conducted by translational researchers who did not utilize CTSA services (Feeney, Johnson, & Welch, 2014).

Next, for all publications identified in PubMed, we retrieved citation information in Clarivate Analytics’ Web of Science (WoS; https://webofknowledge.com/) and InCites (https://incites.thomsonreuters.com/) applications in January 2018. The WoS InCites data sets included 89% of the publications found in PubMed, with some publications most likely unindexed because these were too recently published (not yet imported into WoS’s database, which updates at varying rates depending on the journal; 42% of the unindexed articles were published in 2016 or later) or were from journals not indexed by WoS. We collected InCites citation and content area information for all 64 CTSA hubs, yielding a data set that included the following for each matched publication as of the time of data retrieval: reference information, citation impact measures, and research area according to two classification schemes detailed below.

Research Area Classification Measures

The InCites application includes multiple schemes for classifying publications according to research content area. For each publication in our data set, we examined two classification measures with complementary properties and uses: the narrow WoS research area (WoSRA) scheme and the broad Global Institutional Profiles Project (GIPP) category scheme (Reuters, 2014).

WoSRA Scheme

The WoSRA scheme, the narrowest and most descriptive categorization available from InCites, comprises 252 subject areas across natural and social science, engineering, arts, and humanities. Large fields such as psychology are represented with smaller subfields (e.g., psychology, applied and psychology, biological). This granular definition of subjects is an important characteristic of the scheme as citation behavior may vary significantly among even narrow subfields. The scheme is generally considered to be best for detailed bibliometric analysis, enabling users to objectively measure performance against articles that are most similar in scope and citation characteristics.

The WoSRA scheme is created by first assigning journals to one or more relevant subject areas with a maximum of six designations. It is often not possible to assign journals to a single research area, and overlapping coverage of areas is common. In most cases, individual publications inherit all research areas assigned to the parent journal; however, multidisciplinary journals (e.g., Nature and New England Journal of Medicine) publish articles on a wide array of topics, with individual articles in those journals usually focusing on particular areas of research. Clarivate Analytics reclassifies publications in multidisciplinary to their own most relevant subject areas using an algorithm based on their cited references. This process allows articles to be appropriately compared with articles of similar citation characteristics and topic focus.

GIPP Scheme

In contrast to the WoSRA scheme, the GIPP scheme is a more general system of categorization comprised of six broad branches covering all fields of scholarly research: arts & humanities, clinical, pre-clinical & health, engineering & technology, physical sciences, life sciences, and social sciences. This system is useful for describing large and diverse data sets in succinct terms. The GIPP scheme is based on an aggregation of WoSRAs and thus also includes up to six overlapping designations per publication. A mapping of WoSRAs represented by CTSA-supported publications to their corresponding GIPP categorization is provided in Online Appendix A. A few WoSRAs that cross disciplinary lines within their fields are assigned to multiple GIPP categories (e.g., oncology is assigned to both clinical, pre-clinical & health and life sciences). We utilized the GIPP categories to approximate broad distinctions in translational science, including the basic sciences, the clinical/health sciences, and technological applications.

Citation Impact Measures

The Category Normalized Citation Impact (CNCI), a recently developed metric from WoS InCites, is an adjusted index of citation impact, normalized to publication year and research category (Reuters, 2014). The CNCI score reflects the ratio of the observed number of citations attributed to an article to the expected number of citations for a typical article of that research area and publication year. A score of 3, for instance, means that an article was cited 3 times more frequently than average or 3 times what would be expected, for an article from that year and discipline. CNCI scores were available for all publications indexed in InCites.

The InCites journal impact factor (JIF) is an unadjusted measure of typical citation rates for journals in which articles are published (Reuters, 2014). For example, a JIF of 3 means that articles published in that journal in the past 2 years were cited, on average, 3 times in the metric year. The JIF percentile reflects the percentile ranking of the JIF of each journal by field of research, serving as an adjusted index of journal-level impact within a given discipline.

Complementary Citation Data Collection

We carried out a complementary citation data collection using the previously identified PubMed IDs to query the NIH Office of Portfolio Analysis iCite application (https://icite.od.nih.gov/). An alternative to the WoS InCites application, iCite yields article-level citation information compared against a slightly different reference group of NIH-funded research rather than WoS-indexed research. The iCite application aims to index all NIH-funded articles, and thus, all CTSA-supported publications found in PubMed, meaning that new articles and articles from smaller journals, are usually found in iCite. iCite does not provide information about research content areas, but it can be considered more comprehensive than InCites with regard to raw citation counts for NIH-supported articles. For our sample, 99.6% of publications found in PubMed were matched in iCite. For each matched publication as of the time of data retrieval, the iCite data set included the number of times cited and the relative citation ratio (RCR; Hutchins et al., 2016). The RCR is a field-normalized metric, similar to the CNCI, which approximates the citation impact of an article relative to similar NIH-funded articles. The RCR score is a ratio of the observed number of citations to the expected number of citations for articles within a given co-citation network. RCR data are only available for publications that are at least 1 calendar year old (92% of publications identified in our PubMed queries).

Data Analysis

First, to provide context for content area analysis, we conducted overall bibliometric benchmark analyses, including an assessment of the total numbers of publications supported by the consortium, journals, research areas, and citations, as well as grand mean citation influence indices. Next, in order to characterize the scope of the articles arising from CTSA support, we examined the frequency distributions of journals, WoSRAs, and GIPP categories represented in the portfolio, and the number and percent of articles represented by each journal and area. To quantify the relative diversity of the publication portfolio, we assessed two diversity indices for the WoSRA distribution. We calculated the Blau’s (1977) heterogeneity index by subtracting the sum of the squared proportions of each research area from 1, providing an index between 0 and 1, with numbers closer to 1 indicating greater diversity. We calculated the Shannon equitability index by dividing the Shannon’s (1948) diversity index (the proportion of publications in each research area multiplied by the natural log of that proportion) by the logarithm of the total number of research areas, providing an equitability index between 0 and 1, with 1 being a completely even distribution (Shannon, 1948). Next, in order to characterize the relative influence of the content areas in the CTSA’s publication set, we investigated citation indices for journals and WoSRAs. We examined the JIFs and JIF percentiles of the most frequently publishing journals, and the number and percent of citations garnered, mean CNCI, and RCR scores of publications representing each WoSRA. Then, to examine interdisciplinarity as pertains to translation, we used the GIPP classifications of each article to describe the distribution of articles across three areas of translational science: (1) an aggregation of physical sciences, life sciences, and social sciences reflecting basic science research; (2) clinical/pre-clinical & health reflecting clinical/health sciences; and (3) engineering & technology reflecting health applications in engineering and technology. Because articles are designated to all areas to which the research applies, we are able to quantify the degree of overlap across these three groups by describing the percentages of articles that reflect just one of the three groups listed above or span interdisciplinary boundaries across the three groups. Last, in order to explore bibliometric qualities associated with cross-institutional collaboration, we examined a subset of articles that formally cited support from more than one CTSA hub. This may be reflective of investigators using CTSA resources at one hub coauthoring publications with investigators using CTSA resources at different hubs, or it may occur when an investigator at one hub uses resources or support offered from more than one hub; further, it may arise when researchers draw on programs, data, resources, or infrastructure supplied by more than one hub. We describe the WoSRA and GIPP distributions and examine differences in mean citation indices and interdisciplinary overlap between multihub and single-hub publications. We used SPSS Version 24 (IBM SPSS Inc., Armonk, NY) to calculate all descriptive and inferential statistics.

Results

Overall Bibliometric Benchmarks

Results revealed that from 2006 through 2017, the CTSA consortium supported the research behind at least 69,436 articles, published in nearly 5,000 journals, covering 189 WoSRAs and all 6 GIPP categories. As of early 2018, the publications available for citation analysis in iCite (n = 69,139) were cited 1,503,098 times. On average, publications were cited more than 21 times apiece, with high variability (range: 0–4,689). Among the publications available for content and citation analysis in WoS InCites (n = 61,677), the overall mean CNCI score was 1.94 (SD: 4.38, range: 0–359.43), indicating that, on average, these publications were cited almost twice as often as comparable articles from the same years and research areas. The CNCI results were supported by converging RCR results: The overall mean RCR score of 2.02 (SD: 5.41, range: 0–472.71) similarly indicated that these articles were cited, on average, approximately twice as often as comparable to NIH-funded articles. Consistent with prior research (Schneider et al., 2017), CNCI and RCR scores were highly and significantly correlated with one another in this data set (r = .91, p < .001).

Diverse Scope of CTSA Publication Content

As of data collection, CTSA-supported articles were published in 4,927 journals with an overall mean weighted JIF of 5.64 (SD: 7.00, range: 0.04–187.04), meaning that the journals in which the research was published received an average of 5.64 citations per article per year (over the prior 2 years). Because it is difficult to infer relative impact of journals across widely varying research fields, we also assessed the rank percentiles of JIFs within their respective fields. The mean weighted JIF percentile across all CTSA-supported publications was 75.08% (SD: 21.15%, range: 0.17–99.81%), indicating that the research was published in journals that ranked, on average, in the top quartile in their field for citation influence. The 25 journal titles most frequently publishing CTSA-supported research and their associated journal impact indices are provided in Table 1. CTSA-supported publications are frequently published in the open-access journal PLoS One as well as multidisciplinary journals such as Proceedings of the National Academy of Sciences, Scientific Reports, and The Journal of the American Medical Association. Discipline-specific journals including The Journal of Clinical Endocrinology & Metabolism, Journal of Pediatrics, and Diabetes Care are also frequently represented. Some of the most influential journals that frequently publish CTSA-supported publications (those among the top 25 with the highest JIF percentiles) include American Journal of Obstetrics and Gynecology, Circulation, and New England Journal of Medicine.

Table 1.

Most Frequently Represented Journals in the 2006–2017 CTSA Publication Portfolio.

Rank	Journal Title	Number of Articles in Journal	Journal Impact Factor	Journal Impact Factor Percentile
1	PLoS One	1,955	2.81	77.34
2	The Journal of Clinical Endocrinology & Metabolism	615	5.45	85.87
3	Journal of Pediatrics	453	3.87	94.63
4	Proceedings of the National Academy of Sciences	447	9.66	94.53
5	Pediatrics	416	5.70	97.93
6	Diabetes Care	404	11.86	96.74
7	Journal of Allergy and Clinical Immunology	370	13.08	97.22
8	Journal of Biological Chemistry	367	4.13	74.65
9	Neurology	366	8.32	95.62
10	Obesity	324	3.87	75.25
11	Journal of the American Geriatrics Society	305	4.39	90.00
12	Journal of Acquired Immune Deficiency Syndromes	297	3.93	73.33
13	American Journal of Obstetrics and Gynecology	297	5.23	98.13
14	Clinical and Translational Science	295	0.86	11.33
15	American Journal of Respiratory and Critical Care Medicine	286	13.20	96.46
16	Scientific Reports	283	4.26	85.16
17	Journal of the American Medical Association	275	44.41	98.39
18	Blood	275	13.16	97.86
19	Journal of Immunology	257	4.86	77.81
20	Diabetes	256	8.68	93.84
21	Clinical Journal of the American Society of Nephrology	254	4.78	88.96
22	Circulation	253	19.31	98.61
23	Clinical Cancer Research	243	9.62	94.70
24	New England Journal of Medicine	241	72.41	99.68
25	AIDS	239	5.02	84.64

Note. Top 25 journals include 14% of all portfolio publications. CTSA = Clinical and Translational Science Awards.

We next calculated the distribution of all WoSRAs represented at the end of 11 years. There was reasonable coverage across many subject areas without heavy concentration in any particular areas. Consistent with prior analysis of an early CTSA-supported publication portfolio (Steketee et al., 2012), the Blau’s (1977) heterogeneity index score was >.99, confirming the considerable diversity of the WoSRA distribution. Together with the Shannon’s (1948) equitability index of .80, these indicators show that coverage is wide and fairly even; no one area is represented by more than 7.5% of articles. The 189 WoSRAs represented cover essentially all health science and clinical/translational fields. Table 2 presents a ranking of the 25 most frequently represented WoSRAs (representing 66% of all designations). Because articles are designated up to six WoSRAs and corresponding GIPP categories based on interdisciplinary content, many articles overlap multiple areas and totals sum to greater than 100%. In terms of publication productivity, the most prevalent content areas were neurosciences (7.5% of publications matched in InCites), oncology (6.7%), and clinical neurology (6.6%). Examples of WoSRAs not covered in this portfolio include humanities such as literature, film, and theater and nonhuman sciences and technologies such as horticulture, geology, and marine engineering.

Table 2.

Most Frequently Represented Web of Science Research Areas in the 2006–2017 CTSA Publication Portfolio.

Rank	WoSRA Designation	Number of Articles With Designation	Percentage of Portfolio	Citations for Articles With Designation	Percentage of Portfolio Citations	Mean CNCI	Mean RCR
1	Neurosciences	4,646	7.53	115,516	8.03	1.82	2.15
2	Oncology	4,157	6.74	115,487	8.03	1.98	1.95
3	Clinical neurology	4,066	6.59	98,066	6.82	1.90	2.15
4	Endocrinology & metabolism	4,022	6.52	101,283	7.04	1.80	2.09
5	Immunology	3,983	6.46	108,657	7.56	1.89	1.98
6	Public, environmental & occupational health	3,624	5.88	46,407	3.23	1.56	1.49
7	Pediatrics	3,568	5.78	55,471	3.86	1.90	1.66
8	Psychiatry	2,989	4.85	70,143	4.88	1.91	2.16
9	Biochemistry & molecular biology	2,975	4.82	86,003	5.98	2.03	2.07
10	Cardiac & cardiovascular systems	2,835	4.60	71,519	4.97	2.04	2.17
11	Surgery	2,578	4.18	43,448	3.02	1.80	1.84
12	Genetics & heredity	2,538	4.11	107,928	7.51	2.42	2.46
13	Pharmacology & pharmacy	2,405	3.90	40,133	2.79	1.46	1.48
14	Cell biology	2,362	3.83	78,721	5.47	2.26	2.39
15	Infectious diseases	2,261	3.67	44,973	3.13	1.52	1.74
16	Medicine, research & experimental	2,225	3.61	54,532	3.79	2.01	1.83
17	Urology & nephrology	1,795	2.91	41,558	2.89	2.17	2.05
18	Obstetrics & gynecology	1,773	2.87	27,685	1.93	1.53	1.57
19	Radiology, nuclear medicine & medical imaging	1,763	2.86	31,931	2.22	1.70	1.82
20	Health care sciences & services	1,708	2.77	25,789	1.79	1.72	1.56
21	Respiratory system	1,679	2.72	39,498	2.75	1.81	1.97
22	Hematology	1,656	2.68	37,209	2.59	1.64	1.68
23	Peripheral vascular disease	1,648	2.67	43,547	3.03	2.13	2.37
24	Gastroenterology & hepatology	1,574	2.55	42,864	2.98	2.08	2.36
25	Critical care medicine	1,416	2.30	36,966	2.57	1.94	2.41

Note. Top 25 ranked WoSRAs represent 66% of all portfolio designations. Percent of portfolio is percent of publications with available data for WoSRA; 11.17% of the original portfolio was not indexed in WoS. WoS = web of science; WoSRA = WoS research area; CNCI = category normalized citation index; RCR = relative citation ratio; CTSA = Clinical and Translational Science Awards.

We next assessed the distribution of publications across GIPP categories. Results show that clinical, pre-clinical & health (79.3% of articles) and life sciences (54.5%) are the most frequently represented categories in this portfolio. All 47 WoSRAs delineated as clinical, pre-clinical & health in the GIPP scheme were represented; however, areas in this category with the lowest representation include legal medicine, integrative & complementary medicine and audiology & speech-language pathology. Qualitative analysis of the clinical, pre-clinical & health and life sciences areas revealed that individual topics were broadly distributed within these areas, consistent with the diverse distribution of WoSRAs. Qualitative analysis of the topics represented within the remaining four categories revealed the following: Very few articles (0.1%) were designated as arts & humanities, the majority of which were in the linguistics field. Few articles (3%) were designated as physical sciences, most of which were in the fields of statistics, chemistry, and nanoscience. Few articles (3%) were designated as engineering & technology, most in the fields of biomedical engineering, computer/information science, and materials science. A greater number but still relatively few articles (8%) were designated as social sciences, most in the fields of psychology, health policy, and gerontology.

CTSA Publication Content and Article Citation Influence

In order to quantify the relative influence of the research areas, Table 2 presents impact indices for the most frequently represented WoSRAs. In terms of total citation sums, areas with the highest citation footprints include neurosciences (8.0% of all citations), oncology (8.0%), and immunology (7.6%). Some frequently represented WoSRAs with the highest mean CNCI and RCR scores (highest among the top 25) were genetics & heredity (mean CNCI = 2.42/mean RCR = 2.46), cell biology (2.26/2.39), and peripheral vascular disease (2.13/2.37). Some WoSRAs had higher mean CNCI/RCR scores but were representative of only a small number of articles. Overall, almost all research areas (covering 99% of the portfolio) had mean CNCI or RCR scores greater than 1, indicating above-average citation rates within their disciplines.

CTSA Publication Content and Interdisciplinarity

Because each journal and each article may be designated to up to six different WoSRAs and corresponding GIPP categories, multiple designations reflect content or journal audiences/readership that cross multiple disciplines. Examination of multiple classifications show that 46.1% of the overall portfolio was designated to two or more GIPP categories. We examined the interdisciplinary overlap among the three groups of GIPP categories which we intend to approximate three areas relevant to translational science: (1) basic sciences (social/life/physical sciences), (2) clinical sciences (clinical/pre-clinical & health science), and (3) technological applications (engineering & technology). Arts & humanities was excluded from this analysis due to extremely low representation and nonrelevance to the translational spectrum. The majority of CTSA-supported publications are designated as clinical sciences only (35.7%), basic science only (19.3%), or a combination of the two (42.4%). That is, research supported by the CTSA consortium most frequently addresses clinical and basic science disciplines jointly, followed by clinical or basic science alone. Among the basic sciences, life science is the most predominant category in the portfolio (included in 87.0% of basic science articles). A small percentage of articles (2.7% total) are classified as either engineering & technology alone or in conjunction with the other two groups.

Multihub Collaboration

Finally, to examine characteristics of articles supported by collaborations among hubs, we focused our analyses on articles that cited 2 or more of the 64 CTSA hubs as having contributed to their research and compared these to articles supported by one hub only. A total of 3,505 articles (5% of the portfolio) acknowledged support from multiple hub grants, ranging from 2 to 44 hubs cited (average of 3 hubs per multihub article). Some of these articles address activities specific to the CTSA consortium, such as a catalog of CTSA assets for translational and clinical health research (Shirey-Rice et al., 2014) and a review of accrual and recruitment practices at CTSA institutions (Kost et al., 2014), but many articles address other types of multisite research, for example, the Systolic Blood Pressure Intervention Trial (SPRINT) (Drawz et al., 2017). Multihub articles appeared in 723 journals, covering 115 WoSRAs and all 6 GIPP categories. These articles were cited a total of 111,501 times with an average of 32 citations per article, much greater than the rate for single-hub articles (21 citations/article). With regard to overall citation influence, independent samples t tests revealed that mean CNCI, RCR, JIF, and JIF percentile scores were significantly higher among multihub publications compared to single-hub publications (see Table 3).

Table 3.

Comparison of Publication Content, Citation Influence, and Interdisciplinarity for All 2006–2017 CTSA-Supported Publications, Publications Supported by a Single CTSA Hub, and Publications Supported by Multiple Hubs.

Measure	All CTSA Publications	Single-Hub Publications	Multihub Publications
Number of articles	69,463	65,958	3,505
Number of web of science research areas	189	189	115
Number of citations	1,503,098	1,391,597	111,501
Citation influence indices mean (SD)				t	df	p
Category normalized citation index	1.94 (4.38)	1.88 (4.11)	3.02 (7.73)	8.26	3289.76	<.001
Relative citation ratio	2.02 (5.41)	1.97 (5.23)	2.96 (8.08)	7.25	3308.05	<.001
JIF	5.64 (5.64)	5.52 (6.72)	7.78 (10.62)	11.86	3293.12	<.001
JIF percentile	78.1% (21.2)	74.8% (21.1)	80.1% (20.9)	14	3518.91	<.001
Articles with interdisciplinary GIPP designations	46.1%	45.8%	50.6%
GIPP category designations % of publications				χ²	df	p
Clinical/pre-clinical & health	79.3%	78.8%	87.8%	163.96	1	<.001
Life sciences	54.5%	54.5%	54.1%
Social sciences	8.0%	8.1%	6.4%
Physical sciences	3.1%	3.2%	1.8%
Engineering & technology	2.6%	2.6%	1.3%
Arts & humanities	0.1%	0.2%	0.0%
GIPP interdisciplinary overlap % of publications				χ²	df	p
Basic and clinical/health science	42.3%	42.1%	48.3%	52.38	1	<.001
Clinical, pre-clinical & health only	35.7%	35.6%	38.4%
Basic science only	19.3%	19.7%	11.4%
Engineering & technology and others	2.7%	2.6%	1.8%

Note. GIPP = Global Institutional Profiles Project; JIF = journal impact factor; CTSA = Clinical and Translational Science Awards.

Examination of the publications within this subset that were indexed in InCites (n = 3,192) revealed a distinct pattern of research area distribution compared to single-hub publications. Among WoSRAs represented in this subsample, there was some divergence in prevalence rankings compared to the larger portfolio, with the most frequently represented areas being immunology (10.0% of multihub publications), pediatrics (9.6%), and endocrinology & metabolism (8.6%). WoSRAs with the largest citation footprint included genetics & heredity (13.0% of multihub publication citations), immunology (8.1%), and pediatrics (6.9%). Examination of GIPP categories showed that although the overall distribution across the six categories did not significantly differ, χ²(5) = 1, ns, a significantly larger proportion of publications were designated as clinical, pre-clinical & health among multihub publications (9% more, Table 3). In terms of interdisciplinarity with respect to translational research, multihub publications showed more evidence of cutting across relevant disciplines. Patterns of multiple classification show that 50.6% of the portfolio was designated to two or more GIPP categories. The majority were again designated as clinical/pre-clinical & health only (38.4%), one of the three basic sciences only (11.4%), or a combination of the two (48.3%); a very small percentage of publications (<2%) represented engineering & technology alone or in combination with the other groups. The distribution across these groups differed significantly between multihub and single-hub publications, χ²(3) = 148.95, p < .01, and among multihub publications, a significantly larger proportion of articles overlapped clinical and basic sciences (6.2% more, Table 3).

Discussion

This study builds upon past evaluations of CTSA-supported publication productivity and impact (Llewellyn et al., 2018; Steketee et al., 2012), shedding light on the previously little-known content and interdisciplinary/collaborative structure of publication portfolio amassed over the past 11 years. We found that publications resulting from CTSA support are well distributed across research areas, without substantial concentration in any specific areas. Given the goal of the CTSA program to advance the translational research process in clinical and biomedical science, we confirmed our expectation that a large majority of supported publications involve clinical/health sciences such as neurology and pediatrics or life sciences such as neuroscience and immunology. Because the CTSA program serves to promote the translation of basic science to clinical medicine, it is notable that supported research more often spans basic and clinical science disciplines and secondarily supports clinical science alone. Given that another goal of the CTSA program is to accelerate translation of clinical research into new technologies and innovations, we were surprised that representation in engineering/technology fields was comparatively very low (<3% of the portfolio).

In terms of citation influence, we add to previous research on overall citation influence (Llewellyn et al., 2018) by delineating research areas with the largest citation footprint (total sum of citations, which is positively associated with number of publications and time since publication) and the highest relative citation influence (citation impact scores, which adjust for number of publications, time since publication, and discipline). The fields of neuroscience, oncology, and genetics & heredity stand out as areas in which the CTSA program has had exceptional impact, in terms of productivity and citation influence. However, a large percentage of the overall publication portfolio had mean citation influence indices and were published in journals with impact indices that were well above average.

Regarding interdisciplinarity, we found significant overlap between clinical/health sciences and basic sciences, as measured by the number of publications with designations to both clinical/pre-clinical & health and life, social, or physical science GIPP categories (driven largely by life science). This suggests that a significant portion of CTSA-supported research is serving the goal of bridging early phases of the translational process. In the future, additional emphasis on bridging the gap between clinical and applied/engineering fields may be advisable to further accelerate translation. We did not discover a significant percentage of PubMed-indexed publications in the engineering and technology fields using our methodology. However, CTSA-supported products in these fields may often be shared via other avenues (e.g., conference presentations, patent applications). These would not be considered to be within the scope of our study of CTSA-supported publications (which must be indexed in PubMed to be considered attributable publication products) but may nonetheless represent important contributions from those fields and targets for future evaluations. Still, the sharing of research across engineering, basic, and health science fields via publically accessible publications and collaborations may be an important goal for facilitating the later phases of the translational process. It may be useful to promote translation to technology via grant supplements, calls to research, or new infrastructure such as career training that would enable more/better links to technological innovation.

Analysis of publications supported by two or more CTSA hubs revealed distinct content emphasis, stronger citation influence, and greater interdisciplinary overlap compared to publications supported by a single hub. Publications arising from the support of multiple CTSA hubs were (1) differently distributed across WoSRAs and more likely to be designated as clinical/pre-clinical & health, (2) published in significantly higher impact journals and cited at significantly higher rates, and (3) significantly more likely to be designated to both basic science and clinical categories. Results suggest that the work and costs associated with multihub collaborative efforts may be rewarded by the production of research that is more influential and more in line with CTSA goals than other work at large. This evidence supports the idea that interhub collaborations are ambitious endeavors that are worthy of their investment.

Strengths, Limitations, and Future Directions

Strengths of this study include a comprehensive and nuanced perspective on the CTSA consortium’s publication portfolio, including a nationwide analysis of supported publications and multiple complementary methods of assessing research content over 11 years. We described this portfolio with both the journal- and article-level metrics, with both narrowly defined and broadly descriptive research categories and with both aggregate benchmarks and delineations by research area and collaboration status. Although our study is not comprehensive of all publication products by translational researchers (Feeney, Johnson, & Welch, 2014), it demonstrates methodology for assessing outputs of a long-standing and expanding program that supports translational research.

We were limited in our large-scale analysis by our inability to perform in-depth qualitative analysis on article content. We elected to use existing systems of classification available from WoS, due to their accessibility, ease of use, and reproducibility. This method is not as detailed as analysis of Medical Subject Headings (MeSH) terms or article abstracts, which have been done in the past (Sayavedra, Hogle, & Moberg, 2017; Surkis et al., 2016) and may be feasible for smaller subsets of publications in the future. Additionally, using GIPP categories to approximate translational phases is a general and imperfect delineation. For instance, the GIPP category of clinical, pre-clinical & health, while primarily representing the early phases of translation, includes categories such as public, environmental & occupational health (5.6% of clinical, pre-clinical & health designations) and health care sciences & services (1.1%), which should in many cases fall later on the translational spectrum. Other research has used clinical trial designations to summarize the overall quantity of translational research in a publication set (Han, Williams, & Zuckerman, 2018), but it is unlikely that any existing scheme for classifying publications into research content areas would align with translational phase designations. Individual fields of research often cross translational phases depending on how and from what perspective they are studied. We suggest that content research of this type is best served by focusing on the overlap and intersections among research areas rather than the partitioning of research into discreet bins. We favor schemes such as WoSRAs/GIPP, which allow for the multiple designations that are inherent to many types of clinical and translational research. Whereas the current study examined the relatively digestible overlap among three groups of GIPP categories as an initial step, future research may go further by investigating the intricate overlap across more nuanced classification systems like the WoSRAs.

We provided a collective summary of publication characteristics from the nationwide CSTA consortium, but it is not yet understood how individual hubs differ in terms of these metrics and analyses. For instance, the overall CTSA program is exceptionally diverse in research area content, but it is unknown how this diversity is distributed at local levels. There may be areas of prioritized focus or opportunities for growth in research support at individual hubs or regions of the country. Areas that we defined as having little to no representation relative to the CTSA consortium as a whole may be areas of strength or potential strength at hubs with relevant resources and talent. Conversely, areas we describe as predominant among the consortium may be areas of deliberate de-emphasis at certain hubs that lack local resources or expertise. Individual hubs can utilize the CTSA-wide benchmarks presented in this study to better understand how their organizations are positioned within the national program.

In this study, we describe the current, cumulative content of CTSA-supported research. However, it would be beneficial for future research to explore longitudinal patterns of growth and expansion in various areas, in order to understand how this portfolio came to be and where it may be heading. It would be informative to investigate growth trends for research area prevalence and how trends may be associated with factors such as collaboration, translation, and interdisciplinarity. Past collaboration research has examined ties among institutions or investigators (Bian et al., 2014; Okamoto, 2015; Yu & Wang, 2016), but future research could move beyond this by examining patterns of co-disciplinarity among publications using longitudinal approaches to network analysis (Krivitsky et al., 2017; Ripley, Snijders, Boda, Voros, & Preciado, 2018; Snijders, 2017). Evaluating the trajectories of growth in interdisciplinary ties could aid in decisions related to managing government and local resources around clinical and translational science.

Although we found desirable characteristics associated with publications resulting from multihub collaborations, it does not necessarily follow that all types of research would benefit from multihub collaboration. Research that can be accomplished more quickly and efficiently with the resources of a single hub, especially hubs that are multi-institutional and bring varied resources together in one geographical location, may not see advantages in pursuing costly multihub collaborations. Future research can expand upon our findings by examining mechanisms by which multisite collaboration fosters the most valuable, impactful, and interdisciplinary work, which may help in identifying ways that the NCATS Trial Innovation Network (Shah et al., 2017) can foster the highest impact multihub work.

Conclusion

Findings of this study demonstrate the diverse scope of subject matter and audiences reached by CTSA-supported efforts, the range of influence across top disciplines, and the considerable interdisciplinarity of the portfolio, especially between basic and clinical sciences. Characterizing this portfolio illustrates the landscape of research promoted by CTSA hubs across the United States, including main areas of strength and opportunities for growth. Findings help to recognize the value of the CTSA program, inform resource management as the national program continues to grow, and demonstrate methods for internal evaluation of similar programs. Findings suggest that publications supported by resources at multiple hubs, which likely entail greater effort and coordination costs (Bikard et al., 2015; Cummings & Kiesler, 2007), may be worthwhile investments as evidenced by heightened impact and interdisciplinary reach. This study underscores the utility and importance of using complementary bibliometric methods to evaluate large health science programs. Currently, CTSA hubs are only formally categorized based upon NIH funding amounts (as being small, medium, or large in size), but we assert that the methods described in this study go further by providing quantitative and qualitative ways of contextualizing hubs’ activities and contributions to specific areas of research.

Supplemental Material

Supplemental Material, CTSA_Content_Manuscript_EHP_appendix - Scope, Influence, and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017

Supplemental Material, CTSA_Content_Manuscript_EHP_appendix for Scope, Influence, and Interdisciplinary Collaboration: The Publication Portfolio of the NIH Clinical and Translational Science Awards (CTSA) Program From 2006 Through 2017 by Nicole Llewellyn, Dorothy R. Carter, Deborah DiazGranados, Clara Pelfrey, Latrice Rollins and Eric J. Nehl in Evaluation & the Health Professions

Footnotes

Acknowledgments

The authors wish to thank Kimberly Powell, MLIS, Emory Life Sciences Informationist, for her invaluable assistance in collecting and interpreting publication and citation data. In addition, the authors would like to thank Marie Manguerra, MPP, and Andrew West, MBA, MHA, from Emory University School of Medicine for their much appreciated feedback on this report.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Numbers UL1 TR002378, UL1 TR002649, and UL1 TR002548. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

ORCID iD

Nicole Llewellyn

Supplemental Material

Supplemental material for this article is available online.

References

Bian

Xie

Topaloglu

Hudson

Eswaran

Hogan

(2014). Social network analysis of biomedical research collaboration networks in a CTSA institution. Journal of Biomedical Informatics, 52, 130–140. doi:10.1016/j.jbi.2014.01.015

Bikard

Murray

Gans

J. S.

(2015). Exploring trade-offs in the organization of scientific work: Collaboration and scientific reward. Management Science, 61, 1473–1495.

Blau

(1977). Inequality and heterogeneity: A primitive theory of social structure. New York, NY: The Free Press.

Califf

R. M.

Berglund

(2010). Linking scientific discovery and better health for the nation: The first three years of the NIH’s Clinical and Translational Science Awards. Academic Medicine, 85, 457–462. doi:10.1097/ACM.0b013e3181ccb74d

Clarivate. (2018). InCites. Retrieved from https://incites.thomsonreuters.com/

Clarivate. (2018). Web of Science. Retrieved from https://webofknowledge.com/

Cummings

J. N.

Kiesler

(2007). Coordination costs and project outcomes in multi-university collaborations. Research Policy, 36, 1620–1634. doi:10.1016/j.respol.2007.09.001

Dozier

A. M.

Martina

C. A.

O’Dell

N. L.

Fogg

T. T.

Lurie

S. J.

Rubinstein

E. P.

Pearson

T. A.

(2014). Identifying emerging research collaborations and networks: Method development. Evaluation & the Health Professions, 37, 19–32. doi:10.1177/0163278713501693

Drawz

P. E.

Pajewski

N. M.

Bates

J. T.

Bello

N. A.

Cushman

W. C.

Dwyer

J. P.

… Rahman

(2017). Effect of intensive versus standard clinic-based hypertension management on ambulatory blood pressure: Results from the SPRINT (Systolic Blood Pressure Intervention Trial) ambulatory blood pressure study. Hypertension, 69, 42–50. doi:10.1161/hypertensionaha.116.08076

10.

Feeney

M. K.

Johnson

Welch

E. W.

(2014). Methods for identifying translational researchers. Evaluation & the Health Professions, 37, 3–18. doi:10.1177/0163278713504583

11.

Frechtling

Raue

Michie

Miyaoka

Spiegelman

(2012). The CTSA national evaluation final report. Rockville, MD: Westat.

12.

Grazier

K. L.

Trochim

Dilts

D. M.

Kirk

(2013). Estimating return on investment in translational research: Methods and protocols. Evaluation & the Health Professions, 36, 478–491. doi:10.1177/0163278713499587

13.

Han

Williams

S. R.

Zuckerman

B. L.

(2018). A snapshot of translational research funded by the National Institutes of Health (NIH): A case study using behavioral and social science research awards and Clinical and Translational Science Awards funded publications. PLoS One, 13, e0196545. doi:10.1371/journal.pone.0196545

14.

Hara

Solomon

Kim

S. L.

Sonnenwald

D. H.

(2003). An emerging view of scientific collaboration: Scientists’ perspectives on collaboration and factors that impact collaboration. Journal of the American Society for Information Science and Technology, 54, 952–965.

15.

Hutchins

B. I.

Yuan

Anderson

J. M.

Santangelo

G. M.

(2016). Relative citation ratio (RCR): A new metric that uses citation rates to measure influence at the article level. PLoS Biology, 14, e1002541. doi:10.1371/journal.pbio.1002541

16.

Kost

R. G.

Mervin-Blake

Hallarn

Rathmann

Kolb

H. R.

Himmelfarb

C. D.

… Schuff

K. G.

(2014). Accrual and recruitment practices at Clinical and Translational Science Award (CTSA) institutions: A call for expectations, expertise, and evaluation. Academic Medicine, 89, 1180–1189. doi:10.1097/acm.0000000000000308

17.

Krivitsky

P. N.

Handcock

M. S.

Hunder

D. R.

Goodreau

S. M.

Morris

Bohme

… Kirk

(2017). Fit, stimulate, and diagnose models for network evolution based on exponential-family random graph models. Retrieved from http://www.statnet.org

18.

Leshner

A. I.

Terry

S. F.

Schultz

A. M.

Liverman

C. T.

(2013). The CTSA program at NIH: Opportunities for advancing clinical and translational research. Washington, DC: National Academies Press.

19.

Llewellyn

Carter

D. R.

Rollins

Nehl

E. J.

(2018). Charting the publication and citation impact of the NIH Clinical and Translational Science Awards (CTSA) Program from 2006 through 2016. Academic Medicine, 93, 1162–1170. doi:10.1097/acm.0000000000002119

20.

National Institutes of Health NIH RePORTER. (2015). Research portfolio online reporting tools. Retrieved from https://projectreporter.nih.gov/reporter.cfm

21.

NIH Grant Notice Number: NOT-OD-08-033. Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-033.html

22.

NIH Office of Portfolio Analysis. (2018). iCite. Retrieved from https://icite.od.nih.gov

23.

Okamoto

(2015). Scientific collaboration and team science: A social network analysis of the centers for population health and health disparities. Translational Behavioral Medicine, 5, 12–23. doi:10.1007/s13142-014-0280-1

24.

Pincus

H. A.

Abedin

Blank

A. E.

Mazmanian

P. E.

(2013). Evaluation and the NIH Clinical and Translational Science Awards: A “top ten” list. Evaluation & the Health Professions, 36, 411–431. doi:10.1177/0163278713507061

25.

Reis

S. E.

Berglund

Bernard

G. R.

Califf

R. M.

Fitzgerald

G. A.

Johnson

P. C.

(2010). Reengineering the national clinical and translational research enterprise: The strategic plan of the National Clinical and Translational Science Awards Consortium. Academic Medicine, 85, 463–469. doi:10.1097/ACM.0b013e3181ccc877

26.

Reuters

(2014). InCites indicators handbook. Retrieved from http://researchanalytics.thomsonreuters.com/m/pdfs/indicators-handbook.pdf

27.

Ripley

R. M.

Snijders

T. A. B.

Boda

Voros

Preciado

(2018). Manual for Siena version 4.0. R package version 1.2-12. Retrieved from https://www.cran.r-project.org/web/packages/RSiena/

28.

Rubio

D. M.

(2013). Common metrics to assess the efficiency of clinical research. Evaluation & the Health Professions, 36, 432–446. doi:10.1177/0163278713499586

29.

Sayavedra

Hogle

J. A.

Moberg

D. P.

(2017). Using publication data to evaluate a Clinical and Translational Science Award (CTSA) career development program: Early outcomes from KL2 scholars. Journal of Clinical and Translational Science, 1, 352–360. doi:10.1017/cts.2018.1

30.

Schneider

Kane

Rainwater

Guerrero

Tong

Desai

Trochim

(2017). Feasibility of common bibliometrics in evaluating translational science. Journal of Clinical and Translational Science, 1, 45–52.

31.

Shah

Culp

Gersing

Jones

Purucker

Urv

… Kaufmann

(2017). Early vision for the CTSA program trial innovation network: A perspective from the National Center for Advancing Translational Sciences. Clinical and Translational Science, 10, 311–313. doi:10.1111/cts.12463

32.

Shannon

(1948). A mathematical theory of communication. Bell System Technical Journal, 27, 379–423, 623–656.

33.

Shi

Xiao

Wang

(2011). Diversity of social ties in scientific collaboration networks. Physica A: Statistical Mechanics and its Applications, 390, 4627–4635. doi:10.1016/j.physa.2011.06.072

34.

Shirey-Rice

Mapes

Basford

Zufelt

Wehbe

Harris

… Pulley

(2014). The CTSA Consortium’s Catalog of Assets for Translational and Clinical Health Research (CATCHR). Clinical and Translational Science, 7, 100–107. doi:10.1111/cts.12144

35.

Snijders

T. A. B.

(2017). Stochastic actor-oriented models for network dynamics. Annual Review of Statistics and Its Application, 4, 343–363.

36.

Sosa

M. E.

(2011). Where do creative interactions come from? The role of tie content and social networks. Organization Science, 22, 1–21. doi:10.1287/orsc.1090.0519

37.

Steketee

Frechtling

Cross

Schnell

(2012). Final report on CTSA-supported publications: 2006-2011. Rockville, MD: Westat.

38.

Surkis

Hogle

J. A.

DiazGranados

Hunt

J. D.

Mazmanian

P. E.

Connors

… Mueller

(2016). Classifying publications from the clinical and translational science award program along the translational research spectrum: A machine learning approach. Journal of Translational Medicine, 14, 235.

39.

U.S. National Library of Medicine. (2018). PubMed. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed

40.

Wooten

K. C.

Rose

R. M.

Ostir

G. V.

Calhoun

W. J.

Ameredes

B. T.

Brasier

A. R.

(2014). Assessing and evaluating multidisciplinary translational teams: A mixed methods approach. Evaluation & the Health Professions, 37, 33–49. doi:10.1177/0163278713504433

41.

S. Y.

Wang

H. M.

(2016). Scientific collaboration: A social network analysis based on literature of animal-derived regenerative implantable medical devices. Regenerative Biomaterials, 3, 197–203. doi:10.1093/rb/rbw019

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.30 MB