The Career Award in Biomedical Sciences (CABS) was launched by the Burroughs Wellcome Fund to assist talented scientists to cross the “bridge” that connects postdoctoral training with independent investigator status. This study examines the causal effects of CABS, using both conventional methods and a newer method (propensity score analysis) in attempting to account for selection biases (e.g., selecting the “best and the brightest”) in estimating the award's impact. The results indicated that grantees outperformed unsuccessful applicants in obtaining a faculty position, competing for an R01 grant, and publishing in prestigious journals. However, it was nearly impossible to remove all the effects of selection bias from estimates of program effects. Because selection bias is a pervasive problem in assessing these programs, recommendations for improving evaluations of similar merit-based programs are offered.
Armstrong, P.W., Caverson, M.M., Adams, L., Taylor, M., & Olley, P.M. (1997). Evaluation of the Heart and Stroke Foundation of Canada Research Scholarship Program: Research productivity and impact. Canadian Journal of Cardiology, 13, 507-516.
2.
Boruch, R.F. (2003). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Sage.
3.
Boruch, R. F., & Mosteller, F. (Eds.). (2001). Evidence matters . Washington, DC: Brookings Institution Press.
4.
Burroughs Wellcome Fund (BWF). (2005). Annual report of the Burroughs Wellcome Fund. Chapel Hill, NC: Author.
5.
Chin, S. (2000). A simple method for converting an odds ratio to effect size for use in meta-analysis. Statistical Medicine , 19, 3127-3131.
6.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
7.
Cordray, D.S., & Morphy, P. (in press). Meta-analysis and public policy. In H. Cooper, L. V. Hedges, & J. Valentine (Eds.), Handbook of research synthesis (2nd ed.). New York: Russell Sage.
8.
Cottingham, P., Maynard, R., & Stagner, M. (2005). Generating and using evidence to guide public policy and practice: Lessons form the Campbell test-bed project. Journal of Experimental Criminology, 1, 279-294.
9.
Dehejia, R.H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. Review of Economics and Statistics, 84, 151-161.
10.
Dirk, L. (1999). A measure of originality: The elements of science . Social Studies of Science, 29, 765-776.
11.
Fanzo, J.C., & Gallin, E.K. (2006). The Doris Duke Clinical Scientist Development Award: A seven-year retrospective and summary. In G. Reinhart (Ed.), Enhancing philanthropy's support of biomedical sciences: Proceedings of a workshop on evaluation (pp. 11-20). Washington, DC: The National Academies Press.
12.
Fishman, N. (2006). Research program evaluation at the American Heart Association. In G. Reinhart (Ed.), Enhancing philanthropy's support of biomedical sciences: Proceedings of a workshop on evaluation (pp. 52-65). Washington, DC: The National Academies Press.
13.
Glowinski, I., Julian, C., Onken, J., Zimmerman, C., & Pion, G. (1998). The careers and professional activities of the NIGMS Medical Scientist Training Program (NIH Publication 98-4363). Bethesda, MD: National Institutes of Health.
14.
Grant, J., & Allen, L. (1999). Evaluating high-risk research: An assessment of the Wellcome Trust's Sir Henry Wellcome Commemorative Awards for Innovative Research. Research Evaluation, 8, 201-204.
15.
Heinze, T., Shapira, P., Senker, J., & Kuhlmann, S. (2007). Identifying creative research accomplishments: Methodology and results for nanotechnology and human genetics. Scientometrics, 70, 125-152.
16.
Hooper, C. (1994). Chutes and ladders: NIH scientists discuss the art and strategy of biomedical publishing. NIH Catalyst, 2(1), 8-10.
Lichtman, M.A., & Oakes, D. (2001). The productivity and impact of the Leukemia & Lymphoma Society Scholar Program: The apparent positive effect of peer review . Blood Cells and Molecular Diseases, 27, 1020-1027.
19.
Malik, K., & Pion, G. (2003). Reorganization of peer review in the neurosciences: A retrospective evaluation. Bethesda, MD: National Institutes of Health.
20.
Mavis, B., & Katz, M. (2003). Evaluation of a program supporting scholarly productivity for new investigators. Academic Medicine, 78, 757-765.
21.
National Institutes of Health (NIH). (2006 ). NIH announces program to foster the independence of new investigators . Retrieved from http://grants.nih.gov/grants/new_investigators/index.htm
22.
National Research Council. (1998). Trends in the early careers of life scientists. Washington, DC: The National Academies Press.
23.
National Research Council. (2005a). Biomedical and behavioral research personnel needs. Washington, DC: The National Academies Press.
24.
National Research Council. (2005b). Bridges to independence: Fostering the independence of new investigators in biomedical research. Washington, DC: The National Academies Press.
25.
National Research Council. (2006). Evaluation of the Markey Scholars Program. Washington, DC: The National Academies Press.
26.
Pion, G., & Ionescu-Pioggia, M. (2003). Bridging postdoctoral training and a faculty position: Initial outcomes of the Burroughs Wellcome Fund Career Awards in the biomedical sciences. Academic Medicine , 78, 177-186.
27.
Pion, G., Schaffer, W., & Seder, P. (1999). Customer satisfaction among applicants for R01 and R29 grants. Bethesda, MD: NIH .
28.
Rosenbaum, P.R., & Rubin, D.B. (1984). Reducing bias in observational studies subclassification on the propensity score. Journal of the American Statistical Association, 387, 516-524.
29.
Rubin, D.B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services Outcomes Research and Methods, 2, 169-188.
30.
Rubin, D.B., & Thomas, N. (1996). Matching using estimated propensity scores: Relating theory to practice. Biometrics, 52, 249-264.
31.
Sanchez-Meca, J., Marin-Martinez, F., & Chacon-Moscoso, S. (2003). Effect-size indices for dichotomized outcomes in meta-analysis. Psychological Methods , 8, 448-467.
32.
Stephan, P., & Levin, S. (1992). Striking the mother lode in science: The importance of age, place, and time. New York: Oxford University Press.
33.
U.S. Department of Education. (2007.) Report of the Academic Competitiveness Council. Washington, DC: Author.
34.
Whitehurst, G. (2003, April 21-25). The Institute of Education Sciences: New wine, new bottles. Paper presented at the Annual Meeting of the American Education Research Association, Chicago.
35.
Willard, R., & O'Neil, E. (1998). Trends among biomedical investigators at top-tier research institutions: A study of the Pew scholars. Academic Medicine, 73, 783-789.
