This article reviews the literature and practice concerned with the evaluation of science, technology, and innovation (STI) policies and theway these relate to theories of the innovation process. Referring to the experience of the European Union (EU), the authors review the attempts to ensure that the STI policy theory is informed by advances in the authors'understanding of the innovation process. They argue, however, that the practice of policy evaluation lags behind advances in innovation theory. Despite the efforts to promote theory-led evaluations of STI policies based on new theories of the systemic nature of innovation, evaluation practice in the EU continues to favor the development of methods implicitly based on outdated linear views of the innovation process. This article examines the reasons why this is the case and suggests that STI policy evaluation should nevertheless be supported by the evolving theoretical understanding of the innovation process.
Arnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17.
2.
Barré, R. (1999). Public research programmes: Socio-economic impact assessment and user needs. IPTS Report Special Issue: Evaluation and Research Activities, 20, 5-9.
3.
Betz, F. (1994). Basic research and technology transfer. International Journal of Technology Management, 9(5, 6, 7), 784-796.
4.
Boden, R., Gummett, P., Cox, D., & Barker, K. (1998). Men in white coats.... Men in grey suits: New Public Management and the funding of scientific research in the UK. Journal of Auditing, Accounting and Accountability, 11(3), 267-291.
5.
Bush, V. (1945). Science, the endless frontier(a report to the president by Vannevar Bush, director of the Office of Scientific Research and Development). Washington, DC: U.S. Government Printing Office.
6.
Campbell, D., & Julian, S. (1963). Experimental and quasi-experimental designs for research on teaching. Chicago: Rand McNally.
7.
Dalpe, R., & Anderson, F. (1993). Evaluating the industrial relevance of public R&D laboratories. In B. Bozeman & J. Melkers (Eds.), Evaluating R&D impacts: Methods and practice (1st ed., pp. 207-228). Norwell, MA: Kluwer Academic.
8.
Dietz, J. S. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira & S. Kuhlmann (Eds.), Learning for science and technology policy evaluation. experiences from the United States and Europe (pp. 204-222). Cheltenham, UK: Edward Elgar.
9.
Fahrenkrog, G., Polt, W., Rojo, J., Tubke, A., & Zinocker, K. (Eds.). (2002). RTD evaluation toolbox: Assessing the socio-economic impact of RTD policies. Seville, Spain: Joint Research Centre, European Commission.
10.
Feller, I. (2002). Performance measurement redux. American Journal of Evaluation, 23(4), 435-452.
11.
Freeman, C., & Soete, L. (1997). The economics of industrial innovation. Cambrdige, MA: MIT Press.
12.
Georghiou, L. (1995). Assessing the Framework Programmes—A meta-evaluation. Evaluation, 1(2), 171-188.
13.
Georghiou, L. (2003). Evaluation of research and innovation policy in Europe—New policies, new frameworks? In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 65-79). Cheltenham, UK: Edward Elgar.
14.
Georghiou, L., Ribgy, J., & Cameron, H. (2002). Assessing the socio-economic impacts of the Framework Programme. Manchester, UK: Policy Research in Engineering Science and Technology (PREST), University of Manchester.
15.
Gibbons, M., & Georghiou, L. (1987). Evaluation of research: A selection of current practices. Paris: Organization for Economic Co-operation and Development.
16.
Grupp, H. (2000). R&D evaluation. Research Evaluation, 8(1), 87-99.
17.
Guy, K. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira& S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 174-203). Cheltenham, UK: Edward Elgar.
18.
Guy, K., & Arnold, E. (1998). Strategic options for the evaluation of the R&DProgrammes of the European Union(final report). Brussels, Belgium: European Parliament, Directorate General for Research, Scienctific and Technological Options Assessment.
19.
Guy, K., & Nauwelaars, C. (2003). Benchmarking STI policies in Europe: In search of good practice. Institute for Prospective and Technological Studies Report, 71, 20-27.
20.
Hawkins, R., Montalvo-Corral, C., van Audenhove, L., & Molas-Gallart, J. (2004). Analysis via case studies of the instruments used in the IST programme, submitted to European Commission, DG Information Society, Evaluation and Monitoring (017.31126/01/01).
21.
Henry, G., & Mark, M. (2003). Beyond use: Understanding evaluation's influence on attitudes and actions. American Journal of Evaluation, 24(3), 293-314.
22.
Hobday, M. (1988). Evaluating collaborative R&Dprogrammes in information technology: The case of the U.K. Alvey programme. Technovation, 8, 271-298.
23.
Jaffe, A. B. (1998). Measurement issues. In L. M. Branscomb& J. H. Keller (Eds.), Investing in innovation (pp. 64-84). Cambridge, MA: MIT Press.
24.
Karatzas, I., & Fayl, G. (1999). Editorial. Institute for Prospective and Technological Studies Report Special Issue: Evaluation and Research Activities, 40, 2-4.
25.
Kastrinos, N. (1994). Evaluating the impact of the EC Framework Programme. Technovation, 14(10), 679-688.
26.
Kline, S. J. (1985). Innovation is not a linear process. Research Management, 28(4), 36-45.
27.
Kline, S. J., & Rosenberg, N. (1986). An overviewof innovation. In R. Landau& N. Rosenberg (Eds.), The positive sum strategy: Harnessing technology for economic growth (pp. 275-305). Washington, DC: National Academy Press.
28.
Kostoff, R. N. (1997). The handbook of research impact assessment (No. DTICADA296021). Arlington, VA: Office of Naval Research.
29.
Kuhlmann, S. (1999). Distributed intelligence: Combining evaluation, foresight and technology assessment. Institute for Prospective and Technological Studies Report, 40, 16-22.
30.
Kuhlmann, S. (2003). Evaluation as a source of ‘strategic intelligence.’ In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 352-375). Cheltenham, UK: Edward Elgar.
31.
Laredo, P. (1998). The networks promoted by the framework programme and the questions they raise about its formulation and implementation. Research Policy, 27(6), 589-598.
32.
Laredo, P., Kahane, B., Meyer, J. B., & Vinck, D. (1992). The research networks built by the MHR4 Programme. Brussels, Belgium: Commission of the European Communities.
33.
Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24(1), 5-20.
34.
Lundvall, B.-A. (1992). National systems of innovation: Towards a theory of innovation and interactive learning (1st ed.). London: Pinter.
35.
Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27(6), 599-610.
36.
Luukkonen, T. (2002). Research evaluation in Europe: State of the art. Research Evaluation, 11(2), 81-84.
37.
Mayne, J. (2001). Attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16(1), 1-24.
38.
Meyer-Krahmer, F., & Reiss, T. (1992). Ex ante evaluation and technology assessment—Two emerging elements of technology policy evaluation. Research Evaluation, 2(1), 47-54.
39.
Molas-Gallart, J., Salter, A., Patel, P., Scott, A., & Duran, X. (2002). Measuring third stream activities. Brighton, UK: SPRU.
40.
Molas-Gallart, J., Tang, P., Sinclair, T., Morrow, S., & Martin, B. (1999). Assessing research impact on non-academic audiences. Brighton, UK: SPRU.
41.
Mowery, D. C., & Rosenberg, N. (1979). The influence of market demand upon innovation: A critical review of some recent empirical studies. Research Policy, 8, 105-153.
42.
O'Doherty, D., & Arnold, E. (2003). Understanding innovation: The need for a systemic approach. Institute for Prospective and Technological Studies Report, 71, 29-36.
43.
Office of Science and Technology. (1996). Winning through foresight. A strategy taking the foresight programme to the millenium. London: Department of Trade and Industry.
44.
Organization for Economic Co-operation and Development. (2002). Benchmarking industry-science relationships. Paris: Author.
45.
Policy Research in Engineering Science and Technology (PREST). (1990). The impact and utility of European Commission research programme evaluation reports (EUR13098 EN). Brussels, Belgium: European Commission.
46.
Policy Research in Engineering Science and Technology (PREST), AUEB, BETA, ISI, Joanneum Research, IE HAS, et al. (2002). Assessing the socio-economic impact of the framework programme. Manchester, UK: University of Manchester.
47.
Rip, A. (2003). Societal challenges for R&Devaluation. In P. Shapira& S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 32-53). Cheltenham, UK: Edward Elgar.
48.
Roessner, J. D. (2002). Outcome measurement in the USA: State of the art. Research Evaluation, 11(2), 85-93.
49.
Rothwell, R. (1992). Successful industrial innovation: Critical factors for the 1990s. R&D Management, 22(3), 221-239.
50.
Rothwell, R. (1993, May). Systems integration and networking: The fifth generation innovation process. Paper presented at the Chaire Hydro-Quebec Conference en Gestion de al Technologie, Montreal, Quebec, Canada.
51.
Sanz Menéndez, L., Fernández Carro, J. R., & García, C. E. (1999). Centralidad y cohesión en las redes de colaboración empresarial en la I+D subsidiada [Centrality and cohesion in interfirm collaborative networks participating in publicly funded R&D programs]. Papeles de Economía Española, 81, 219-241.
52.
Schmookler, J. (1966). Invention and economic growth. Boston: Harvard University Press.
53.
Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagné, & M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39-83). Chicago: Rand McNally.
54.
Shapira, P. (2003). Evaluating manufacturing extension services in the United States: Experiences and insights. In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 260-292). Cheltenham, UK: Edward Elgar.
55.
Sherwin, C. W., & Isenson, R. S. (1967, March). Estimating the science and technology components of anR&Dbudget. Paper presented at the Second Cost-Effectiveness Symposium of the Washington Operations Research Council, Washington, DC.
56.
Soete, L., & Corpakis, D. (2003). Editorial: R&D for competitiveness and employment—The role of benchmarking. Institute for Prospective and Technological Studies Report, 71, 2-12.
57.
Stame, N. (2004). Theory-led evaluation and types of complexity. Evaluation, 10(1), 58-76.
58.
STRATA-ETAN Expert Working Group. (2002). Benchmarking national research policies: The Impact of RTD on Competitiveness and Employment (IRCE).Brussels, Belgium: European Commission Directorate General Research.
59.
van Raan, A. F. J. (2000). R&D evaluation at the beginning of the new century. Research Evaluation, 8(2), 81-86.
60.
von Hippel, E. (1988). The sources of innovation. New York: Oxford University Press.
