Evaluation Review

Three types of semiquantitative methods used by the federal government in research impact assessment are presented. These include the classic retrospective method (Project Hindsight), another retrospective approach (Project TRACES andfollow-ons), and accomplishments books used by selected research-sponsoring organizations (Office of Naval Research, Air Force Office of Scientific Research, Department of Energy Office of Health and Environmental Research, Department of Energy High Energy Physics Program, Defense Advanced Research Projects Agency). The strengths and weaknesses of each approach are discussed. One goal of all the studies presented was to identify the products of research and some of their impacts. In addition, the Hindsight, the technology in retrospect and critical events in science (TRACES), and the Defense Advanced Research Projects Agency (DARPA) studies tried to identify factors that influenced the productivity and impact of research. The following general conclusions about the role and impact of basic research were reached: (1) the majority of basic research events that directly affected technologies or systems were nonmission oriented and occurred many decades before the technology or the system emerged; (2) the cumulative indirect impacts of basic research were not accounted for by any of the retrospective approaches published; (3) an advanced pool of knowledge must be developed in many fields before synthesis leading to an innovation can occur; and (4) allocation of benefits among researchers, organizations, and funding agencies to determine economic returns from basic research is very difficult and arbitrary, especially at the microlevel.

Grants peer review is a family of ex ante methods used by federal agencies to select research proposalsforfunding. This article draws on Chubin and Hackett's 1990 book, Peerless Science: Peer Review and U.S. Science Policy, to examine both the assumptions and theory underlying peer review as an evaluation methodology and the issues that arise in the operation of peer-based systems at the National Science Foundation (NSF) and the National Institutes of Health (NIH). The article discusses criticisms of peer review, the various criteria that agencies must balance in interpreting evaluations of expert peers, and outstanding issues to be addressed in the refinement and reform of peer review systems. The article concludes with suggestions on how to redress the shortcomings of peer review in decision making, especially the allocation of scarce public monies.

This article describes the use of peer review for federal research impact evaluation. For selected agencies, it covers peer review practices for proposed and existing programs. It shows that nearer-term research impacts typically play a more important role in the review outcome than longer-term impacts; however, they do not have quite the importance of team quality, research approach, or the research merit. Although advanced review processes can improve the efficiency of a review, three of the most important intangible factors for a high-quality peer review are motivation of the review leader, and competence and independence of the review team members. Although peer review in its broadest sense is the most widely used method in research selection, review, and expost assessment, it has its deficiencies, and there is no single method that provides a complete impact evaluation. The use of quantitative techniques such as bibliometrics to supplement peer review is an area ripe for exploitation.

In Europe, impact assessment is an essential element of evaluation. Impact cannot usually be measured directly but has to be explored in view of a set of both quantitative and qualitative impact dimensions. The impact needs also to be placed in the relevant context. At present, a range of established methods is available for use by evaluators. The methods can be grouped into three categories: those providing framework for assessment, those concerned with data collection, and those concerned with data analysis. Panels are used extensively to complement the other methods. European evaluations have so far been mainly concerned with quality and direct impact. Assessments of the medium- or long-term impact are still rare.

U.S. industry's interest in and implementation of research and development (R&D) program evaluation techniques has been evolving throughout the twentieth century. Due in large part to the amount of funds corporations are now investing in R&D programs, evaluation techniques continue to become increasingly important. In this article, the history of R&D and its evaluation within the 3M Company will be used as an example of this progression. The article discusses how U.S. industrial R&D has moved from individual or few-person efforts earlier this century to today's high-investment, large-scale team approaches. A review of the most frequently used U.S. industrial R&D program evaluation methods is included. Among these are scoring methods, cost/benefit analysis, and risk analysis. The largest part of the article is a description of the 3M technical audit process. In addition to discussing the procedural and scoring elements of the technical audit, emphasis is placed on how 3M attains a variety of benefits from the process. The article concludes with a summary of other activities that take place in 3M's Technology Analysis Department. These projects involve measuring R&D productivity and assessing the technological health (present and future) of 3M.

This article presents the theory behind modern evaluative bibliometric techniques at three levels. Policy applications, which characterizes the scientific and technological output of nations or regions; strategic analyses, which deals with articles and patents at the level of a university or company; and tactical analyses, which addresses questions concerning a single subject. The article explains the newer techniques that have been developed at each level, as well as the more important limitations.

This article reviews the principal methods economists and benefitlcost analysts use in evaluating research. These methods usually involve computing impacts using market information, mone tizing the impacts, and then comparing the value of the impacts with the cost of research. Two principal measures are common in the literature: surplus measures (consumers' surplus, producers'surplus, and combinations thereof) and productivity measures (average or marginal contributions of research and development [R&D] at firm, industry, national, or international levels). Given known time streams of benefits and costs, internal rates of return to R&D investments are then computed The article notes both the standard technical difficulties with these approaches and the political and organizational difficulties in using them.

The mission of a corporate research and development laboratory is to supply leadership technology that keeps the company's businesses winning in the global marketplace. Evaluation efforts must be simple and low in cost, must contribute directly to fulfilling that mission, must not distract researchers from their main tasks, and, most important, must help build credibility in the eyes of the lab's customers, the operating businesses of the company. These points are illustrated with special reference to General Electric's Research and Development Center.

This article presents basic principles and examples of spatial representations derived from the analysis of co-occurrence frequency data pertaining to bibliographic information elements, such as key words and citations, in research publications and patents. These bibliometric maps provide a means for communicating information on relational features of the science and technology (S&T) system—either for analytical or representational purposes. Characteristics of the main types ofbibliometric maps are outlined, and their potential for practical applications in S&T policy and research and development management are discussed. An emphasis is placed on more recent developments, in particular bibliometric maps produced by the Centre for Science and Technology Studies (CWTS) for depicting temporal changes in the S&T system. Three empirical examples ofsuch maps are presented with a focus on their application for impact assessment in both scientific as well as technological fields: (1) the emergence of new research topics in worldwide research on manufacturing technology, (2) changes in patterns of (inter)na tional collaboration within Dutch research on coal and coal products, and (3) the role of instruments in materials science.