Ten-Year Publication Trajectories of Health Services Research Career Development Award Recipients

Abstract

This study’s purpose was to identify distinct publishing trajectories among 442 participants in three prominent mentored health services research career development programs (Veterans Affairs, National Institutes of Health, and Agency for Healthcare Research & Quality) in the 10 years after award receipt and to examine awardee characteristics associated with different trajectories. Curricula vitae (CVs) of researchers receiving awards between 1991 and 2010 were coded for publications, grants, and awardee characteristics. We found that awardees published at constant or increasing rates despite flat or decreasing rates of first-author publications. Senior-author publications rose concurrently with rates of overall publications. Higher overall publication trajectories were associated with receiving more grants, more citations as measured by the h-index, and more authors per article. Lower trajectory groups were older and had a greater proportion of female awardees. Career development awards supported researchers who generally published successfully, but trajectories varied across individual researchers. Researchers’ collaborative efforts produced an increasing number of articles, whereas first author articles were written at a more consistent rate. Career development awards in health services research supported the careers of researchers who published at a high rate; future research should further examine reasons for variation in publishing among early career researchers.

Keywords

career development VA health care system NIH AHRQ publications trajectory analysis

Early in their careers, researchers face the challenging tasks of establishing a research identity, building a professional network, and learning to obtain funding independently, all while working to publish at a consistently high rate (Johnson, 2011). To support junior researchers early in their careers, mentored career development awards such as the U.S. Department of Veterans Affairs Health Services Research & Development (VA HSR&D) Service’s Career Development Awards (CDAs), the National Institute of Health’s (NIH’s), and Agency for Healthcare Research & Quality’s (AHRQ’s) K Awards in health services research (HSR) have been established. These awards offer several years of mentored research experience and salary support to aid in the transition to an independent research career.

Publications comprise the principal medium used by researchers to disseminate their findings and to impact their respective scientific fields, and examining awardees’ early career publishing trajectories and associated characteristics could help award-granting institutions explore reasons why some awardees publish more than others. Although a singular focus on number of publications may obscure other important markers of career progress, publication rate does correlate with other important metrics of research productivity such as research funding and academic promotion (Batshaw, Plotnick, Petty, Woolf, & Mellits, 1988; Svider et al., 2013). The observation period in the present study was too short to detect variability in academic promotions, but we explored associations between publication trajectories and grant funding.

Existing evidence suggests that participants in research career development programs publish at a higher rate than both unfunded applicants and nonparticipants (Discovery Logic, 2011; Lichtman & Oakes, 2001; Mavis & Katz, 2003; Pion, 2001; Smith, Rogers, Hansen, & Smith, 2009), yet how participants’ publishing rates develop over time remain largely unexamined. Furthermore, evaluations of research career development programs have focused primarily on end point, whole-group analyses of individual programs, and determining the average number of publications at a follow-up point, thereby lumping together subsets of participants who publish at especially high or low rates. Some research which examines variation in economists’ publishing suggests that productivity in a research career is not constant over time and that groups of researchers follow distinct trajectories (Buchmueller, Dominitz, & Hansen, 1999; Rauber & Ursprung, 2006).

To address these gaps in the literature, we examined early career publication trajectories over a 10-year period of recipients of VA HSR&D, NIH, and AHRQ HSR career development awards. We also examined publishing trajectories for (a) first-authored articles, as these most directly represent the researchers’ ideas and contributions and (b) last- or senior-authored articles as these represent substantial contributions, possibly including securing funding for the project and mentoring younger investigators (Tscharntke, Hochberg, Rand, Resh, & Krauss, 2007). To further explore trends in publishing, we studied associations between publishing trajectories and h-index (a measure based both on citations and the number of articles) and the number of authors per article.

We furthermore sought to explore and describe differences in the characteristics of researchers with distinct publishing trajectories. These exploratory analyses examined factors potentially linked to productivity including gender (Jagsi, Motomura, Griffith, Rangarajan, & Ubel, 2009), age (Gingras, Larivière, Macaluso, & Robitaille, 2008), race/ethnicity (Palepu et al., 1998), research ranking of a degree-granting university (Drew & Karpf, 1981), and type of degree (MD vs. PhD). We also examined grants obtained during the observation period, publications prior to award, and award-granting agency (VA, NIH, or AHRQ). Understanding characteristics of awardees in different trajectory groups may aid in identifying awardees or applicants who are likely to publish at especially high rates.

Although the primary aims of the current study were largely exploratory, we hypothesized that distinct publishing trajectories would emerge, highlighting a subset of awardees who published at rates that increased over time and another set that plateaued. We further hypothesized that these groups would differ on important, identifiable personal characteristics, although due to the exploratory nature of our analyses we had no specific a priori hypotheses.

Method

Participants

Publications were coded from CVs, which were requested from 561 VA HSR&D CDA, NIH K, and AHRQ K career development award recipients who received their awards between 1991 and 2010; 442 (79%) sent their CVs. Our sample included 219 HSR&D CDAs, 154 NIH K awards, and 69 AHRQ K awardees. All received early career mentored research awards which lasted between 3 and 5 years.

Data Source

Complete rosters of VA HSR&D CDA program awardees were obtained from VA Central Office, and NIH and AHRQ K awardees were identified using the NIH Research Portfolio Online Reporting Tools—Expenditures & Results (RePORTER) database. NIH and AHRQ K awardees whose award topics were comparable to those in VA HSR were selected. The CVs were coded by a member of the project team according to a coding manual developed by the team (available from the authors upon request).

Measures

Publications

From each CV, full citations, years, and author positions for all publications through the end of 2010 were coded. The primary outcome measure used for trajectory analyses was the number of major journal publications–defined as reports of empirical research, literature reviews, extended commentaries or essays, guideline panel reports, or case studies—published in a given year. We also examined trajectories of first-authored and last- or senior-authored major journal publications.

To calculate publishing trajectories, we counted major journal publications for all years starting from the year of the award and ending either 10 years later or in 2010 (whichever came first). As a consequence, awardees who were funded closer to 2010 had fewer years of publication data. The mean number of years of publication data available per awardee was 6.3. At least 5 years of publication data were available for 68% of awardees and 10 years of publication data were available for 28% of awardees. Missing data for awardees who received their awards later were imputed in the trajectory model using a maximum likelihood estimator (Haviland, Jones, & Nagin, 2011).

Awardee characteristics

Gender and race/ethnicity were self-reported by participants via e-mail. Race/ethnicity was coded as a dichotomous variable, dividing the categories into White non-Hispanic and minority. Age at award was estimated assuming an age of 22 when a bachelor’s degree was conferred. Type of terminal degree (MD vs. PhD) and university attended were obtained from participants’ CVs. NIH research rankings for the top 100 universities (lower scores represent better rankings) from 1991 to 2010 were used to rank awardees’ terminal degree-granting universities. Participants with both an MD and a PhD were analyzed as MDs due to differences in salary mechanisms between MDs and PhDs. Total number of major journal articles before award receipt and the total number of first-authored major journal articles published before award receipt were also examined.

Other awardee productivity measures

We calculated the number of grants awarded during the 10-year observation window and the Hirsch h-index for each awardee. The h-index is a measure which combines information on the number of publications and citations of those publications defined as follows: “A scientist has index h if h of his/her N_p papers have at least h citations each, and the other (N_p − h) papers have no more than h citations each.” In practice, an investigator with an h-index of 10 has published 10 articles which have each been cited 10 or more times (Hirsch, 2005, p. 16569). To examine collaboration in publishing, we utilized a text parsing program to calculate the average number of authors per article for each awardee.

Analysis Plan

SAS 9.2 (SAS Institute, Cary, NC) was used to conduct statistical analyses. Using PROC TRAJ, a custom macro for semi-parametric group-based modeling in SAS, we identified subgroups of awardees whose publication records followed distinct trajectories (Jones, Nagin, & Roeder, 2001; Nagin, 1999). PROC TRAJ estimates a discrete mixture model to fit trajectory data. We performed trajectory analyses using three outcomes: (a) number of major journal publications, (b) number of first-authored major journal publications, and (c) number of last- or senior-authored major journal publications post-award. Model fit was assessed using the Bayesian Information Criterion (BIC), for which values closer to 0 indicate better fit. Differences in BIC were judged to be significant based on criteria set out in Jones, Nagin, and Roeder (2001). Akaike’s Information Criterion (AIC) values are also presented for comparison. The number of groups in the final model was chosen by balancing model fit with explanatory parsimony. Following the approach used by Franklin et al. (2013), we only tested up to six groups as the results of a solution with more than six groups would be too difficult to interpret. Next, we investigated and tested differences in demographic and educational characteristics across trajectory subgroups.

Results

Sample Characteristics

The study sample was 48% female. The majority of researchers had a highest degree that was either an MD or a joint MD/PhD (68%), whereas most of the remaining researchers (30%) had a PhD as their highest degree and only 2% had another highest degree. Minority awardees comprised 19% of the sample.

Trajectories for All Publications

We considered model fit for increasing numbers of groups from 1 to 6 using the BIC (BIC_{1 group} = −8,289, BIC_{2 group} = −6,990, BIC_{3 group} = −6,695, BIC_{4 group} = −6,670, BIC_{5 group} = −6,600, BIC_{6 group} = −6,547; AIC_{1 group} = −8,280, AIC_{2 group} = −6,970, AIC_{3 group} = −6,662, AIC_{4 group} = −6,625, AIC_{5 group} = −6,543, and AIC_{6 group} = −6,479). The six-group solution fits best (see Figure 1) and included (a) low publishers (n = 69; 17% of the sample), (b) low-medium publishers (101; 25%), (c) high publishers (49; 11%), (d) low-increasing publishers (140; 27%), (e) medium-increasing publishers (56; 13%), and (f) high-increasing publishers (27; 7%). The full sample of awardees published an average of 3.1 articles in the first year of an award and 6.2 articles in the final year of observation. The low, low medium, and high groups published at consistent rates throughout the awards (ps for linear trend > .05), whereas the low-increasing, medium-increasing, and high-increasing groups displayed a significantly increasing slope (p < .001). Average posterior probabilities for group membership were all greater than 0.72, indicating that the six-group model was moderately stable in its assignments of individuals to trajectory groups.

Figure 1.

Major journal publications per year by the trajectory group.

Awardee Characteristics by Trajectory Group

Characteristics for awardees by the trajectory group for all major journal publications are presented in Table 1. There were proportionately fewer women in the higher trajectory groups and more women in the lower trajectory groups (χ² = 18.05, p < .01), ranging from 59% of the low group to 26% of the high-increasing group. The higher trajectory groups also contained younger awardees, on average, than did the lower trajectory groups, ranging from a mean age of 39.2 in the low-increasing group to 35.0 in the high-increasing group (F = 3.46, p < .01). No significant differences were found across trajectory groups in the distribution of minority awardees, the awardees’ type of highest degree, or in the NIH ranking of an awardee’s terminal degree-granting university (ps > .05). Trajectory group membership was positively associated with the number of major journal articles published prior to award (F = 19.87, p < .001) as well as the first-authored major journal articles published prior to award (F = 22.25, p < .001).

Table 1.

Awardee Characteristics by Trajectory Group.

	Low (N = 69)	Low medium (N = 101)	High (N = 49)	Low increasing (N = 140)	Medium increasing (N = 56)	High increasing (N = 27)	F or χ²	η² or V
Gender, % female	59	52	49	50	29	26	18.05**	.20
Race, % minority	27	12	22	20	21	15	6.16	.12
Age at award, years	39.2	37.6	38.1	37.3	37.8	35.0	3.46**	.04
Highest degree							11.68	.11
MD, %	64	58	76	71	71	70
PhD, %	30	38	20	26	29	29
Other, %,	6	4	4	3	0	1
NIH research ranking of degree-granting university	50.2	38.5	39.2	37.3	38.5	31.9	1.67	.02
Type of career development award							2.35	.05
VA HSR&D CDA, %	44	52	43	49	61	52
NIH K, %	41	38	27	34	36	26
AHRQ K, %	16	11	31	17	4	22
Major journal articles before award	6.2	9.5	20.2	8.0	12.7	20.4	19.87***	.19
First-authored major journal articles before award	3.0	5.0	9.7	4.2	6.2	11.8	22.25***	.20
Number of authors per publication	3.8	4.2	4.9	4.4	4.9	4.7	6.67***	.07
h-index	6.9	13.0	20.9	14.3	21.4	26.0	67.22***	.44
Grants (PI) in 10 years since award	2.6	3.3	3.5	3.7	4.1	6.2	3.41**	.04

Note. NIH = National Institute of Health; VA HSR&D CDA = Veterans Affairs Health Services Research & Development Service Career Development Award; AHRQ = Agency for Healthcare Research & Quality; PI = Principal Investigator.

*p < .05, **p < .01, ***p < .001.

Other Awardee Productivity Measures

Awardees in higher trajectory groups had obtained more grants over the observation period (F = 3.41, p < .01). Controlling for years since award, awardees in higher trajectory groups had higher h-indices (F = 67.22, p < .001) and had more authors on an average publication (F = 6.67, p < .001).

Trajectories for First-Authored Publications

We considered model fit for increasing numbers of groups from 1 to 5 using the BIC (BIC_{1 group} = −4,798, BIC_{2 group} = −4,478, BIC_{3 group} = −4,413, BIC_{4 group} = −4,413, BIC_{5 group} = −4,422; AIC_{1 group} = −4,787, AIC_{2 group} = −4,458, AIC_{3 group} = −4,378, AIC_{4 group} = −4,369, and AIC_{5 group} = −4,367). The three-group solution fits best (see Figure 2) and included (a) low-decreasing first-author publishers (n = 239; 52% of sample), (b) medium first-author publishers (174; 40%), and (c) high first-author publishers (29; 8%). The full sample of awardees published an average of 1.4 first-authored articles in the first year of an award and 1.1 in the final year of observation. The low-decreasing first-author group displayed a significant negative linear trend (p < .001), whereas the medium first-author and the high first-author group displayed flat trends (ps > .05). Average posterior probabilities for group membership were all greater than .80.

Figure 2.

First-authored major journal publications per year by first-author trajectory group.

Trajectories for Last- or Senior-Authored Publications

We considered model fit for increasing numbers of groups from 1 to 6 using the BIC (BIC_{1 group} = −4,347, BIC_{2 group} = −3,592, BIC_{3 group} = −3,449, BIC_{4 group} = −3,411, BIC_{5 group} = −3,384, BIC_{6 group} = −3,385; AIC_{1 group} = −4,341, AIC_{2 group} = −3,578, AIC_{3 group} = −3,426, AIC_{4 group} = −3,380, AIC_{5 group} = −3,346, and AIC_{6 group} = −3,338). The five-group solution fits best (see Figure 3) and included (a) low last-author publishers (n = 193; 44% of sample), (b) low-increasing last-author publishers (122; 28%), (c) medium first-author publishers (57; 13%), (d) medium-increasing last-author publishers (53; 12%), and (e) high-increasing last-author publishers (17; 4%). The full sample of awardees published an average of 0.4 last-author articles in the first year of an award and 1.9 in the final year of observation. The low-increasing, medium-increasing, and high-increasing last-author groups displayed a significant positive linear trend (p < .001), whereas the low and medium last-author groups displayed flat trends (ps > .05). Average posterior probabilities for group membership were all greater than .69.

Figure 3.

Last- or senior-authored major journal publications per year by last- or senior-author trajectory group.

Discussion

The current analysis adds to the literature on research careers by identifying and describing several distinct publishing trajectories for recipients of three highly regarded mentored career development awards. Because the awardees of these programs did not differ significantly on their membership in these trajectory groups, our evaluation may represent more general early career publication trajectories for high-achieving health services researchers. Some publishing trajectories were more elevated than others, yet overall trajectories either remained flat or increased, indicating that awards support the careers of researchers who publish successfully. Future research should examine the extent to which trajectories which follow the same general trend (e.g., increasing vs. constant) are substantively distinct from one another. Awardees with higher publication trajectories also tended to obtain more grants, to have higher h-indices, and to have more authors per publication. The current research is, to our knowledge, the first to explore publication trajectories for health services researchers and career development awardees.

Examining trajectories for first-authored publications, last-authored publications, and the number of authors per article provide some insights into the amount of collaborative work investigators engage in across their early careers. Although awardees’ total publishing rates and their publishing rates as last or senior author generally increased over time, their first-author publishing rates declined slightly or remained the same. Researchers may face challenges in taking the lead on manuscripts as their time becomes increasingly taxed with grant writing, administrative, clinical, and mentoring duties. Collaborative publications with other researchers, however, have been found in other work to increase over time (Drenth, 1998; Endersby, 1996). Taken together with the finding that awardees in higher trajectory groups tended to have more authors on their articles, our results are consistent with the explanation that collaboration on articles contributes to a greater overall number of publications. This finding suggests a role for networking and collaboration in generating publications as one’s career progresses.

In exploratory analyses of awardee characteristics, statistically significant age and gender differences were found across groups. The higher trajectory groups included younger awardees than the lower trajectory groups. This effect was not large and may be an artifact of the natural arc of publishing careers (Gingras et al., 2008; Jonsson, Tollback, Gonzales, & Borg, 2000). Female career development awardees were more likely to be in the lower trajectory groups and less likely to be in the higher trajectory groups. We urge caution in interpreting this finding as it indicates a small difference in just one measure of career success. Still the finding is consistent with a body of research on gender in academic careers across disciplines and around the world which finds women with fewer grants, publications, and administrative positions than men (Amering, Schrank, & Sibitz, 2011; DesRoches, Zinner, Rao, Iezzoni, & Campbell, 2010). Institutional and social barriers still exist to women achieving parity in the research setting (Katila & Meriläinen, 1999), but substantial efforts are currently being made to “level the playing field” with regard to gender bias (Budden et al., 2008; Fried et al., 1996).

Limitations

The current study has limitations. Results are likely representative of HSR career development award recipients or other high-achieving researchers but may be less generalizable to the research population at large. Specifically, the study focused on mentored awards. The extent to which results generalize to nonmentored career awards is a potential topic for further study. Although first and senior authorship were considered, and a measure of citation frequency, the h-index, was explored, publications were not given different weights according to their length, journal impact factor, or field. Relatedly, the h-index, although a commonly used measure of scientific impact, can be prone to distortions and at times produces inconsistent rankings (Waltman & van Eck, 2012). More generally, citation-based measures of career success can be problematic as they are gameable and, on a broader scale, can disincentivize doing novel and groundbreaking work (Alberts, 2013; Stroobants, Godecharle, & Brouwers, 2013). We recognize the potential pitfalls of using impact-based measures to judge productivity and believe they should be just one of many measures used to assess academic advancement. In that regard, we found that publication trajectories were linked to grants obtained. Finally, the causal effects of career development awards on publishing careers could not be examined due to the lack of a sample of unfunded applicants. Despite these limitations, the current study delves more deeply into both early career publishing trajectories and the accomplishments of career development awardees than prior evaluations have.

Conclusions

Career development awards are intended to support a scientist’s career and enhance his or her contributions to a scientific field; in this case, to improve the delivery of health care services. Career development award recipients generally follow consistent or increasing trajectories in publishing major journal articles, suggesting that the awards support researchers who contribute substantially to scientific knowledge in HSR. Future research should investigate the reasons why some awardees publish at higher rates than others and the strength of these associations. Identifying mechanisms that support researchers’ development and barriers that inhibit their success will lead to a greater scientific impact.

Footnotes

Authors’ Note

The views expressed in this article are those of the authors and do not necessarily reflect the position nor the policy of the Department of VA or the U.S. government. The VA had no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the article for publication.

Acknowledgments

The authors wish to thank awardees for their participation in the evaluation.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The data presented are part of an evaluation funded by VA HSR&D Service Directed Research Project 10-182. The work presented is supported in part by VA Office of Academic Affairs. Dr. Finlay was supported by the VA Office of Academic Affiliations Advanced Fellowship in Health Services Research and Development. Dr. Timko was funded as a Senior Research Career Scientist (RCS 00-001).

References

Alberts

(2013). Impact factor distortions. Science, 340, 787. doi:10.1126/science.1240319

Amering

Schrank

Sibitz

(2011). The gender gap in high-impact psychiatry journals. Academic Medicine, 86, 946–952.

Batshaw

M. L.

Plotnick

L. P.

Petty

B. G.

Woolf

P. K.

Mellits

E. D.

(1988). Academic promotion at a medical school. Experience at Johns Hopkins University School of Medicine. N Engl J Med, 318, 741–747.

Buchmueller

T. C.

Dominitz

Lee Hansen

(1999). Graduate training and the early career productivity of Ph.D. economists. Economics of Education Review, 18, 65–77. doi: https://dx-doi-org.web.bisu.edu.cn/10.1016/S0272-7757(98)00019-3

Budden

A. E.

Tregenza

Aarssen

L. W.

Koricheva

Leimu

Lortie

C. J.

(2008). Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23, 4–6.

DesRoches

C. M.

Zinner

D. E.

Rao

S. R.

Iezzoni

L. I.

Campbell

E. G.

(2010). Activities, productivity, and compensation of men and women in the life sciences. Academic Medicine, 85, 631–639.

Discovery Logic. (2011). National Institutes of Health individual mentored career development awards program. Retrieved December 2, 2012, from http://grants.nih.gov/training/K_Awards_Evaluation_FinalReport_20110901.pdf

Drenth

J. H.

(1998). Multiple authorship: The contribution of senior authors. JAMA, 280, 219–221. doi:10.1001/jama.280.3.219

Drew

Karpf

(1981). Ranking academic departments: Empirical findings and a theoretical perspective. Research in Higher Education, 14, 305–320. doi:10.1007/bf00976681

10.

Endersby

J. W.

(1996). Collaborative research in the social sciences: Multiple authorship and publication credit. Social Science Quarterly (University of Texas Press), 77, 375–392.

11.

Franklin

J. M.

Shrank

W. H.

Pakes

Sanfelix-Gimeno

Matlin

O. S.

Brennan

T. A.

Choudhry

N. K.

(2013). Group-based trajectory models: A new approach to classifying and predicting long-term medication adherence. Medical Care, 51, 789–796. doi:10.1097/MLR.0b013e3182984c1f

12.

Fried

L. P.

Francomano

C. A.

MacDonald

S. M.

Wagner

E. M.

Strokes

E. J.

Carbone

K. M.

… Stobk

J. D.

(1996). Career development for women in academic medicine: Multiple interventions in a department of medicine. Journal of American Medical Association, 276, 898–905. doi:10.1001/jama.1996.03540110052031

13.

Gingras

Larivière

Macaluso

Robitaille

J.-P.

(2008). The effects of aging on researchers' publication and citation patterns. PLoS One, 3, e4048. doi:10.1371/journal.pone.0004048

14.

Haviland

A. M.

Jones

B. L.

Nagin

D. S.

(2011). Group-based trajectory modeling extended to account for nonrandom participant attrition. Sociological Methods & Research, 40, 367–390. doi:10.1177/0049124111400041

15.

Hirsch

J. E.

(2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences U S A, 102, 16569–16572. doi:10.1073/pnas.0507655102

16.

Jagsi

Motomura

A. R.

Griffith

K. A.

Rangarajan

Ubel

P. A.

(2009). Sex differences in attainment of independent funding by career development awardees. Annals of Internal Medicine, 151, 804–811. doi:10.7326/0003-4819-151-11-200912010-00009

17.

Johnson

A. M.

(2011). Charting a course for a successful research career A guide for early career researchers. Retrieved from http://cdn.elsevier.com/assets/pdf_file/0019/111835/charting-a-course-for-a-successful-research-career.pdf

18.

Jones

B. L.

Nagin

D. S.

Roeder

(2001). A SAS procedure based on mixture models for estimating developmental trajectories. Sociological Methods & Research, 29, 374–393.

19.

Jonsson

Tollback

Gonzales

Borg

(2000). Inter-rater reliability of the 1992 international standards for neurological and functional classification of incomplete spinal cord injury. Spinal Cord, 38, 675–679.

20.

Katila

Meriläinen

(1999). A serious researcher or just another nice girl?: Doing gender in a male-dominated scientific community. Gender, Work & Organization, 6, 163–173. doi:10.1111/1468-0432.00079

21.

Lichtman

M. A.

Oakes

(2001). The productivity and impact of the Leukemia & Lymphoma Society Scholar Program: the apparent positive effect of peer review. Blood Cells, Molecules, and Diseases, 27, 1020–1027. doi:10.1006/bcmd.2001.0476

22.

Mavis

Katz

(2003). Evaluation of a program supporting scholarly productivity for new investigators. Academy of Medicine, 78, 757–765.

23.

Nagin

D. S.

(1999). Analyzing developmental trajectories: A semiparametric, group-based approach. Psychological Methods, 4, 139.

24.

Palepu

Carr

P. L.

Friedman

R. H.

Amos

Ash

A. S.

Moskowitz

M. A.

(1998). MInority faculty and academic rank in medicine. Journal of American Medical Association, 280, 767–771. doi:10.1001/jama.280.9.767

25.

Pion

G. M.

(2001). The early career progress of NRSA predoctoral trainees and fellows (NIH Publication No. 00-4900). Bethesda, MD: National Institutes of Health.

26.

Rauber

Ursprung

H. W.

(2006). Evaluation of researchers: A life cycle analysis of German academic economists. CESifo, Working Paper no. 1673. Munich: CESifo Group.

27.

Smith

W. H.

Rogers

J. G.

Hansen

T. N.

Smith

C. V.

(2009). Early career development in academic pediatrics of participants in the APS-SPR Medical Student Research Program. Pediatric Research, 65, 474–477. doi:10.1203/PDR.0b013e3181975f85

28.

Stroobants

Godecharle

Brouwers

(2013). Research evaluation: Flanders overrates impact factors. Nature, 500, 29. doi:10.1038/500029b

29.

Svider

P. F.

Mauro

K. M.

Sanghvi

Setzen

Baredes

Eloy

J. A.

(2013). Is NIH funding predictive of greater research productivity and impact among academic otolaryngologists? Laryngoscope, 123, 118–122. doi:10.1002/lary.23659

30.

Tscharntke

Hochberg

M. E.

Rand

T. A.

Resh

V. H.

Krauss

(2007). Author sequence and credit for contributions in multiauthored publications. PLoS Biology, 5, e18.

31.

Waltman

van Eck

N. J.

(2012). The inconsistency of the h-index. Journal of the American Society for Information Science, 63, 406–415. doi:10.1002/asi.21678