Over the past few decades there has been a rise in the number of federally funded large scientific research initiatives, with increased calls to evaluate their processes and outcomes. This article describes efforts to evaluate such initiatives in one agency within the U.S. federal government. The authors introduce the Evaluation of Large Initiatives (ELI) project, a preliminary effort to explore how to accomplish such evaluation. They describe a pilot effort of this project to evaluate the Transdisciplinary Tobacco Use Research Center (TTURC) initiative of the National Cancer Institute. They present a summary of this pilot evaluation including the methods used (concept mapping, logic modeling, a detailed researcher survey, content analysis and systematic peer-evaluation of progress reports, bibliometric analysis and peer evaluation of publications and citations, and financial expenditures analysis) and a brief overview of results. Finally, they discuss several important lessons and recommendations that emerged from this work.
Anderberg, M.R. (1973). Cluster analysis for applications. New York: Academic Press.
2.
Brainard, J. (2002a, March 29). New science measures released by OMB . Chronicle of Higher Education, 48, p. A25.
3.
Brainard, J. (2002b, June 7). New NIH director asks researchers' help to provide accountability. Chronicle of Higher Education, 48. Available at http://chronicle.com/daily/2002/06/2002060702n.htm .
4.
Compton, D. W., Baizerman, M., & Stockdill, S. H. (Eds.). (2002). The art, craft, and science of evaluation capacity building (Vol. 93). San Francisco : Jossey-Bass.
5.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis for field settings. Boston: Houghton Mifflin.
Davison, M.L. (1983). Multidimensional scaling. New York: John Wiley.
8.
Everitt, B. (1980). Cluster analysis (2nd ed.). New York: Halsted Press, A Division of John Wiley.
9.
Funkhouser, E.T. (1996). The evaluative use of citation analysis for communication journals. Human Communication Research, 22(4), 563—574.
10.
Garfield, E. (1994a). The impact factor. Current Contents , 25, 3-7.
11.
Garfield, E. (1994b). Using the impact factor. Current Contents, 29, 3-5.
12.
Garfield, E. (1995). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122(3159), 108-111.
13.
Godlee, F., & Jefferson, T. (1999). Peer review in health sciences. London: BMJ Books.
14.
Greene, J. C., & Caracelli, V. J. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. New directions for program evaluation (Vol. 74). San Francisco: Jossey-Bass.
15.
Greene, J.C., Caracelli, V.J., & Graham, W.F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis , 11, 255—274.
16.
Hood, W.W., & Wilson, C.S. (2001). The literature of bibliometrics, scientometrics, and informetrics. Scientometrics, 52(2), 291—314.
17.
Institute of Medicine. (2004). NIH extramural center programs. Washington, DC: National Academies Press.
18.
Kane, M., & Trochim, W. (2006). Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage.
Kostoff, R.N. (1994b). Quantitative and qualitative federal research impact evaluation practices. Technological Forecasting and Social Change, 45(2), 189—205.
21.
Kostoff, R.N. (1995). Research requirements for research impact assessment . Research Policy, 24(6), 869—882.
22.
Krippendorf, K. (2004). Content analysis: An introduction to its methodology (2nd ed.). Thousand Oaks, CA : Sage.
23.
Kruskal, J.B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage.
24.
Lane, N.F. (1995). What is the future of research—The science policy of the USA. Interdisciplinary Science Reviews, 20(2), 98—103.
25.
McCullough, J. (1992). Draft report of the NSF/Program Evaluation Staff Workshop on Methods for Evaluating Programs of Research Centers. Washington, DC: National Science Foundation.
26.
Molas-Gallart, J., & Davies, A. (2006). Toward theory-led evaluation: The experience of European science, technology, and innovation policies. American Journal of Evaluation, 27(1), 64-82.
27.
Nass, S.J., & Stillman, B.W. (2003). Large-scale biomedical science: Exploring strategies for future research. Washington, DC: National Academies Press.
28.
National Academy of Sciences, National Academy of Engineering, & Institute of Medicine. (1996). An assessment of the National Science Foundation's Science and Technology Centers Program . Washington, DC: National Academies Press.
29.
National Cancer Institute. (2007). Greater than the sum: Systems thinking in tobacco control (Smoking and Tobacco Control Monograph Series). Bethesda, MD: U.S. Department of Health and Human Services, National Institutes of Health .
30.
National Institutes of Health. (2006). FY 2007 president's budget request. NIH Office of Budget. Retrieved August 17, 2006, from http://officeofbudget.Od.Nih.Gov/fy07/supporting%20information.Pdf .
31.
National Research Council. (1999). Evaluating federal research programs: Research and the Government Performance and Results Act. Washington, DC: National Academies Press.
32.
Nye, M. (1996). Before big science: The pursuit of modern chemistry and physics, 1800-1940. New York: Twayne Publishers.
33.
Office of Management and Budget. (1993). Government Performance Results Act of 1993. Washington, DC: Executive Office of the President of the United States. Retrieved August 17, 2006, from http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html.
34.
Office of Management and Budget. (2007a). Expectmore.Gov. Retrieved August 12, 2007, from http://www.whitehouse.gov/omb/expectmore/.
35.
Office of Management and Budget. (2007b). Program assessment: NSF's federally funded research and development centers . Retrieved August 12, 2007, from http://www.whitehouse.gov/omb/expectmore/summary/10004401.2005.html .
36.
Office of Management and Budget. (2007c). Program Assessment Rating Tool (PART). Retrieved August 12, 2007, from http://www.whitehouse.gov/omb/part/index.html .
37.
Osareh, F. (1996a, September). Bibliometrics, citation analysis and co-citation analysis: A review of literatureI. Libri, 46, 149-158.
38.
Osareh, F. (1996b, September). Bibliometrics, citation analysis and co-citation analysis: A review of literature II. Libri, 46, 217-225.
39.
Rasmussen, N. (2002). Of “small men,” big science and bigger business: The Second World War and biomedical research in the United States . Minerva, 40(2), 115—146.
40.
Rosenberg, S., & Kim, M.P. (1975). The method of sorting as a data gathering procedure in multivariate research. Multivariate Behavioral Research , 10, 489—502.
41.
Rosenfield, P.L. (1992). The potential of transdisciplinary research for sustaining and extending linkages between the health and social sciences. Social Science and Medicine, 35, 1343—1357.
42.
Schoepflin, U., & Glanzel, W. (2001). Two decades of “scientometrics”—An interdisciplinary field represented by its leading journal. Scientometrics, 50(2), 301—312.
43.
Seglen, P.O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal, 314, 498-502.
44.
Skoie, H. (1999). Bibliometrics—Some warnings from the north . Scientometrics, 45(3), 433—437.
45.
Smith, R. (2001). Measuring the social impact of research: Difficult but necessary [Editorial]. British Medical Journal, 323, 528.
46.
Stokols, D., Fuqua, J., Gress, J., Harvey, R., Phillips, K., Baezconde-Garbanati, L., et al. (2003, December). Evaluating transdisciplinary science. Nicotine and Tobacco Research, (5, Suppl. 1), S21—S39.
47.
Trochim, W. (1984). Research design for program evaluation: The regression-discontinuity approach. Beverly Hills, CA: Sage .
48.
Trochim, W. (1985). Pattern matching, validity, and conceptualization in program evaluation. Evaluation Review, 9(5), 575—604.
49.
Trochim, W., & Linton, R. (1986). Conceptualization for planning and evaluation . Evaluation and Program Planning, 9(4), 289—308.
50.
U.S. Department of Health and Human Services. ( 2007). Institutional Clinical and Translational Science Award . Retrieved August 12, 2007, from http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-07—002.html .
51.
U.S. General Accounting Office. (2000). NIH research: Improvements needed in monitoring external grants (No. GAO-HEHS-AIMD-00—139) . Washington, DC: Author.
52.
U.S. General Accounting Office. (2005). OMB's part reviews increased agencies' attention to improving evidence of program results (No. GAO-06—67). Washington, DC : Author.
53.
Weber, R.P. (1990). Basic content analysis (Vol. 49, 2nd ed.). Newbury Park, CA : Sage.
54.
Weller, S.C., & Romney, A.K. (1988). Systematic data collection. Newbury Park, CA: Sage.
55.
Whitmore, E. (Ed.). (1998). Understanding and practicing participatory evaluation (Vol. 80). San Francisco: Jossey-Bass.
56.
Williams, B., & Imam, I. (Eds.). (2006). Systems concepts in evaluation: An expert anthology. Point Reyes CA: EdgePress.
57.
W. K. Kellogg Foundation. (2001). Logic model development guide: Using logic models to bring together planning, evaluation and action. Battle Creek, MI: W. K. Kellogg Foundation.
