Evaluation Review

We describe an evaluation undertaken on contract for the New Zealand State Services Commission of a major project (the Administrative Decision-Making Skills Project) designed to produce a model of administrative decision making and an associated teaching/learning packagefor use by government officers. It describes the evaluation of a philosophical model of decision making and the associated teaching/learning package in the setting of the New Zealand Public Service, where a deliberate attempt has been initiated to improve the quality of decision making, especially in relation to moral factors.

Ten steps are identified and explained to assist the performance of a cost-effectiveness analysis (CEA) study. The article is presentedfrom the standpoint of an important social problem that can be elucidated by the CEA procedure. The ten steps outlined generally follow the scientific method, beginning with stating the problem and defining objectives, continuing through identifying alternatives, establishing measures of effectiveness, and formulating and testing a model program. A discussion of selecting alternatives based on different approaches used to conduct the CEA evaluation itself concludes the article.

Among the rationales used in promoting juvenile diversion is cost savings. This rationale is based on the assumption that diversion handling is less expensive than formal justice processing and that diversion reduces the number of cases handled by the formal system. To date, the few available results concerning diversion's cost savings are fragmented and inconclusive. Although diversion handling has been found to be less costly, it has also been shown that diversion increases the overall number of youth subject to court supervision and control. Diversion's capacity to "widen the net " has been ignored in previous cost studies. This article assesses the feasible cost savings associated with statewide imple mentation of diversionary arbitration programs in Florida and includes consideration of the programs' capacities to wide the client net.

Policies shaping evaluation of psychological and educational interventions reflect a naive understanding of the broad nature and function of evaluative research. As a result, such policies are overemphasizing the evaluation of immediate intervention outcomes and undermining comprehensive intervention research. The presentation highlights the danger of this trend and illustrates how the focus of evaluation should be expanded not only to make a greater contribution to practice but to aid in the development of basic knowledge about the nature of interventions. To these ends, evaluation is explored as one of four basic intervention problems; specific examples are offered to underscore the type of basic intervention questions currently ignored and in need of scholarly attention.

Program evaluation requiring postprogram follow-up with clients is often complicated by systematic inability to contact certain types of clients. In-person contact can be extremely expensive and mail follow-up is prone to error attributable to nonresponse. Problems in contacting and obtaining the cooperation of former clients often bring evaluation results into question. Employing a split-half design, this research examines the effects of prior letters, in conjunction with a follow-up telephone survey, on three factors affecting evaluation results: contact rates, response rates, and respondent cooperation. Contrary to what would be expected based upon the results of experimentation with general population surveys, the prior letter did not increase contact, improve cooperation, or decrease refusals. Discussion focuses on the reasons for the differences between the findings of this experiment and previous research and suggests that additional procedures for increasing contact in postprogram follow-up surveys be examined.

This article presents a study of the use of diaries to measure discretionary behavior. These are behaviors such as viewing television where personal choice plays a predominant role. Diary data collected upon two occasions five months apart were compared with automatically recorded data. The results show that use of diaries apparently caused an increase in thefrequency of viewing broadcast teletext on the part of those who normally used it infrequently. The data indicate that diaries are likely to be inaccurate in measuring the durations of behavioral events. It is concluded that diaries should not be used to estimate the population frequencies for discretionary behaviors.

The reliability of criterion-referenced tests (CRTs) used in health program evaluations can be conceptualized in different ways. Classical conceptualizations of reliability have limited usefulness when applied to health-related CRTs. When a cut-score is set, the test's reliability can be represented as the consistency of mastery/nonmastery classifications. When a cut-score is not set, the size of the standard error of measurement (SEM) of a domain score estimate is a reflection of the test's reliability. Formulas are presented for estimating appropriate SEMsfor CRTs. The SEM can be used in computing confidence intervals for domain score estimates andfor a cut-score.

This study presents a data-based examination of program evaluation in community mental health centers from two perspectives. First, survey data from 111 centers are analyzed according to a hierarchical model of evaluation performance proposed in previous research. Second, these data are reanalyzed using multidimensional scaling to examine empirical relationships among types of evaluation activities. Although evaluation patterns are consistent with the hierarchical model, the empirical model suggests important new relationships that refine prior conceptualizations. Results are discussed in the context of research on evaluation and professional issues for evaluators.