Evaluation Review

This article examines a group of defenses of qualitative evaluation methods that are based upon a hard relativistic interpretation of the work of Thomas Kuhn. Hard relativism holds that (1) truth is a function of framework, and (2) all frameworks are equally legitimate because they are only to be criticized on the basis of internal criteria. Soft relativism accepts the first point but denies the second. It is argued, following Karl Popper, that hard relativism is an indefensible position. A defense of qualitative evaluation based upon this position, therefore, will be equally shaky. Kuhn, however, is a soft—not a hard—relativist. Thus, a more promising defense of qualitive evaluation may be found in a soft-relativist interpretation of Kuhn's analysis of the nature of scientific discovery.

A controlled study of the impact of a juvenile education program on the recidivism rates of juveniles was performed. The program involved introducing the juveniles to the realities of prison life. No significant differences between experimental and control groups in the mean number of self-reported status or criminal offenses committed during the premeasure or follow-up period was found. A finding that youths categorized as more delinquent were affected differently by program attendance compared to youths categorized as less delinquent is offered as a tentative explanation of the conflicting results of prior studies of this type of intervention.

Emergency programs, which often call for exceptions to ordinary rules and procedures and flexible, short-term, reactive activities that preempt attention from overall ends, pose special problems for both process and goal-attainment evaluations. An emergency is defined as: (1) a change in life circumstances so that individuals who could ordinarily cope can no longer do so without additional resources, and (2) a situation from which life-threatening or dangerous consequences are thought to flow. Relaxed standards are suggested in areas of targeting and coverage, equity, and efficiency. Such important secondary gains as development of community resources, establishing and reinforcing service networks, problem identification, and advocacy should be included in any general evaluation.

The 1981 Omnibus Budget Reconciliation Act (OBRA) called for important changes in the Aid to Families with Dependent Children (AFDC) program. This article reports the findings of a study that tracked two nationally representative samples of AFDC cases to learn about caseload dynamics prior to and following OBRA's implementation. Simple comparisons and the use of life-table techniques revealed that amendments to the AFDC program had a significant impact on program participation and costs that was not offset by any change in work behavior. Various features of the evaluation design and their relationship to the study objectives are described.

During this period of reductions in government social programs, there is great value in developing program evaluation methods that are low cost and timely. This article explores the use of past evaluation data to develop statistical models useful in predicting future program performance. Model data requirements are much lower than those of evaluations, and models can yield policy results much more quickly. The modeling approach is used to develop a low-cost monitoring scheme for the Bonneville Power Administration Residential Weatherization Program. Three years of evaluation data (1981, 1982, and 1983) were utilized. The results suggest that the modeling approach is most effective when program characteristics and actors, and the external environment are stable over time. In our application, modeling results changed between 1982 and 1983. The article concludes with a discussion about the defensibility ofpredictions yielded by streamlined evaluations.

This article evaluates the effectiveness of motorcycle helmets in accident situations. A latent variable model is developed and estimated. It is concluded that (1) motorcycle helmets have no statistically significant effect on the probability of fatality; (2) helmets reduce the severity of head injuries; and (3) past a critical impact speed, helmets increase the severity of neck injuries. Further analysis establishes the qualitative and quantitative nature of the head-neck injury trade-off.

The relationship between satisfaction with a residential energy audit and reported subsequent energy conservation behavior is investigated in a stratified random sample of 500 Illinois audit recipient. Results demonstrate a weak relationship between audit satisfaction and energy conservation behavior, a finding consistent with similar satisfac tion-behavior relationships studied in the literature of social and organizational psychology. Although impacts upon energy conservation behavior do not justify concern with participant satisfaction with energy conservation programs, other theoretical justifications for such concern are identified. It is concluded, therefore, that satisfaction should be measured as part of the comprehensive evaluation of an energy conservation program.

The high cost of one of the more expensive versions of the Class A home energy audit prompted a major utility company to examine two questions that have to date received surprisingly little careful attention: (1) Does receiving an audit result in more energy savings than not receiving an audit? (2) Can the energy savings resulting from more expensive versions of the Class A audit be achieved by less expensive versions? Both self-reported conservation behavior data and more objective kilo Watt hour consumption data were collected from subjects over a 12-month period following the audits. These data indicate that the answer to both of these questions is yes.

This article describes a study that was designed to test the ordinal and interval properties of rating scales, the like of which are commonly employed in evaluation research. Because there is a scarcity of available techniques, testing the interval properties of the scales required a novel adaptation of Mosteller's (1951) "test for internal consistency." The analysis is based upon data generated by the use of paired comparisons and direct scales, and the conclusion reached is that rating scales such as those outlined in this article appear to possess metric properties.