National Evaluation of the Emergency School Aid Act (ESAA): a Review of Methodological Issues

Large-scale field evaluations of education programs typically present complex and competing design requirements that can rarely be satisfied by ideal, textbook solutions. This paper uses a recently completed national evaluation of the federally-funded Emergency School Aid Act (ESAA) Program to illustrate in concrete fashion some of the problems often encountered in major program evaluations, and traces the evolution of efforts in that three-year longitudinal study—both in the original design conceptualization and in the actual implementation and data analysis phases—to resolve competing demands and to provide as much methodological rigor as possible under field conditions. Issues discussed here include the selection of experimental versus quasi-experimental designs; the development of sampling procedures to provide maximum precision in treatment-control comparisons; the selection of achievement tests and difficulties in developing and administering other, non-cognitive outcome measures; and the importance of ascertaining whether the underlying assumptions of a true experimental design have been met before conclusions about program impact are drawn on the basis of treatment-control comparisons.