One of the most important stages of a meta-analysis is specifying the inclusion criteria. In other
words, what studies will be included in or excluded from the quantitative review? How are these
decisions made? The author presents problems and illustrations from this first phase in an
ongoing meta-analysis of crime reduction programs. The eight criteria for including studies in
the crime reduction meta-analysis are specified, problematic studies confronted using the
criteria are listed, and rules for handling those studies to retain consistency throughout the
meta-analysis are discussed. The article concludes with three recommendations for future
meta-analyses of this type.
Get full access to this article
View all access options for this article.
References
1.
Abrami, P.C., P.A. Cohen, and S. d'Apollonia.1988. Implementation problems in meta-analysis . Review of Educational Research58(2): 151-79.
2.
Adams, R., and H.J. Vetter.1971. Probation caseload and recidivism rate . British Journal of Criminology11:390-3.
3.
Alexander, J.F., and B.V. Parsons.1973. Short-term behavioral intervention with delinquent families. Journal of Abnormal Psychology81(3): 219-25.
4.
Andrews, D.A., I. Zinger, R.D. Hoge, J. Bonta, P. Gendreau, and F.T. Cullen.1990. Does correctional treatment work? A clinically relevant and psychologically informed meta-analysis. Criminology28(3): 369-404.
5.
Antonowicz, D.H., and R.R. Ross.1993. Essential components of successful rehabilitation programs for offenders. Unpublished manuscript, University of Ottawa, Ontario, Canada.
6.
Boruch, R.F., A.J. McSweeney, and E.J. Soderstrom .1978. Bibliography: Illustrative field experiments. Evaluation Quarterly4:655-95.
7.
Bryant, F.B., and P.M. Wortman.1984. Methodological issues in the meta-analysis of quasi-experiments In Issues in data synthesis. New directions for program evaluation, No. 24, edited by W. H. Yeaton and P. M. Wortman, 5-24. San Francisco : Jossey-Bass.
8.
Bullock, R.J., and D.J. Svyantek.1985. Analyzing meta-analysis: Potential problems, an unsuccessful replication and evaluation criteria. Journal of Applied Psychology70(1): 108-11.
9.
Campbell, D.T., and J.L. Stanley.1966. Experimental and quasi-experimental designs for research. Chicago. Rand McNally.
10.
Carranza, E., M. Houved, and L.P. Mora.1994. Release on personal recognizance in Costa Rica: An experimental research study. In Alternatives to imprisonment in comparative perspective, edited by U. Zvekic, 439-61. Chicago: Nelson-Hall .
11.
Chelimsky, E., and L.G. Morra.1984. Evaluation synthesis for the legislative user. In Issues in data synthesis. New directions in program evaluation, No. 24, edited by W. H. Yeaton and P. M. Wortman, 75-89. San Francisco: Jossey-Bass.
12.
Cook, T.D., H. Cooper, D.S. Cordray, H. Hartmann, L.V. Hedges, R.I. Light, T.A. Louis, and F. Mosteller.1992Meta-Analysis for Explanation. New York: Russel Sage Foundation.
13.
Cooper, H.C.1989. Integrating research. A guide for literature reviews. 2d ed. Beverly Hills, CA: Sage.
14.
Cordray, D.S.1990. An assessment from the policy perspective. In The Future of Meta-Analysis, edited by K. Wachter and M. L. Straf, 99-119. Beverly Hills, CA: Sage.
15.
Davidson, W.S., R. Gottschalk, L. Gensheimer, and J. Mayer.1984. Interventions with juvenile delinquents: A meta-analysis of treatment efficacy. Unpublished manuscript, Psychology Department, Michigan State University.
16.
Dennis, M.L.1988. Implementing randomized field experiments: An analysis of criminal and civil justice research. Ph.D. diss., Northwestern University, Ann Arbor, MI (university microforms).
17.
Dole, V.P., J.W. Robinson, J. Orraca, and E. Towns.1969. Methadone treatment of randomly selected criminal addicts. New England Journal of Medicine280(25): 1372-75.
18.
Dunford, F.W.1990Random assignment: Practical considerations from field experiments. Evaluation and Program Planning13:125-32.
19.
Fagan, J.1990. Natural experiments in criminal justice. In Measurement issues in criminology, edited by K. Kempf, 108-37. New York: Springer-Verlag .
20.
Farrington, D.P.1983. Randomized experiments on crime and justice. In Crime and justice: An annual review of research. Vol. 4. Edited by M. Tonry and N. Morris, 257-308. Chicago: University of Chicago Press.
21.
Farrington, D.P., L.E. Ohlin, and J.Q. Wilson.1986. Understanding and controlling crime . New York: Springer-Verlag.
22.
Freidman, A.S.1989. Family therapy versus parent groups: Effects on adolescent drug users. American Journal of Family Therapy17(4): 335-47.
23.
Furby, L., M.R. Weinrott, and L. Blackshaw.1989. Sex offender recidivism: A review. Psychological Bulletin105(1): 3-30.
24.
Garrett, C.J.1985. Effects of residential treatment on adjudicated delinquents: A meta-analysis. Journal of Research in Crime and Delinquency22(4): 287-308.
25.
Glass, G.V. , B. McGaw, and M.L. Smith.1981. Meta-analysis in social research. London: Sage.
26.
Green, G.S.1985. General deterrence and television cable crime: A field experiment in social control. Criminology23(4): 629-45.
27.
Hamm, M.S., and J.C. Kite.1992. The role of offender rehabilitation in family violence policy: The Batterers' Anonymous experiment. Criminal Justice Review16(2): 227-48.
28.
Hedges, L.V., and I. Olkin.1985. Statistical methods for meta-analysis . New York: Academic Press.
29.
Hunter, J.E., and F.L. Schmidt.1990. Methods of meta-analysis. Correcting error and bias in research findings. Beverly Hills, CA : Sage.
30.
Kaufman, P.1985. Meta-analysis of juvenile delinquency prevention programs . Unpublished paper, Claremont Graduate School, Claremont, CA.
31.
Kelling, G.L., A. Pate, D. Dieckman, and C.E. Brown.1974. The Kansas City Preventive Patrol Experiment. A technical report. Washington, DC: Police Foundation.
32.
Kobrin, S., and M.W. Klein.1983. Community treatment of juvenile offenders: The DSO experiments. Beverly Hills, CA: Sage.
33.
Lamb, H.R., and V. Goertzel.1974. Ellsworth House: A community alternative to jail. American Journal of Psychiatry131(1): 64-68.
34.
Lerner, A.1953. An experiment in group counseling with male alcoholic inmates. Federal Probation17(3): 32-8.
35.
Lipsey, M.W.1992. Juvenile delinquency treatment: A meta-analytic inquiry into the variability of effects. In Meta-analysisfor explanation , edited by T. D. Cook, H. Cooper, D. S. Cordray , H. Hartmann, L. V. Hedges, R. J. Light, T. A. Louis, and F. Mosteller , 83-127. New York: Russell Sage Foundation.
36.
Lipsey, M.W., and D.B. Wilson.1993. The efficacy of psychological, educational, and behavioral treatment. Confirmation from meta-analysis. American Psychologist48(12): 1181-1209.
37.
Martin, S., S. Annan, and B. Forst.1986. Deterring the drunk driver. Washington, DC: Police Foundation.
38.
Massimo, J.L., and M.F. Shore.1963. The effectiveness of a comprehensive, vocationally-oriented psychotherapeutic program for adolescent delinquent boys. American Journal of Orthopsychiatry33:634-42.
39.
Matt, G.1989. Decision rules for selecting effect sizes in meta-analysis: A review and reanalysis of psychotherapy outcome studies. Psychological Bulletin105(1): 106-15.
40.
Morris, N.1974. Thefuture of imprisonment. Chicago : University of Chicago Press.
41.
Palmer, T.1992. The re-emergence of correctional intervention. Beverly Hills, CA: Sage.
42.
Police Foundation.1981. The Newark Foot Patrol Experiment. Washington, DC: Author.
43.
Rosenthal, R.1991. Meta-analytic procedures for social research, rev. ed. Beverly Hills, CA: Sage.
44.
Ross, H.A., and M. Blumenthal.1974. Sanctions for the drinking driver: An experimental study. Journal of Legal Studies3:53-61.
45.
Schmidt, F.L.1992. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist4(10): 1173-81.
46.
Schwartz, R.D., and S. Orleans.1967. On legal sanctions. University of Chicago Law Review34:274-300.
47.
Sechrest, L., S.O. White, and E.D. Brown.1979. The rehabilitation of criminal offenders: Problems and prospects. Washington, DC: National Academy of Sciences.
48.
Sherman, L.W.1988. Randomized experiments in criminal sanctions. In Lessons from selected program and policy areas. New directions for program evaluation, No. 37, edited by H. S. Bloom, D. S. Cordray, and R. J. Light, 85-98. San Francisco : Jossey-Bass.
49.
—. 1992. Policing domestic violence. Experiments and dilemmas. New York: Free Press.
50.
Sherman, L.W., and D.L. Weisburd.1987. The experimental effects of criminal sanctions. Unpublished grant proposal to the National Institute of Justice, Crime Control Institute, Washington, DC.
51.
Shore, M.F., and J.L. Massimo.1966. Comprehensive vocationally oriented psychotherapy for adolescent delinquent boys: A follow-up study. American Journal of Orthopsychiatry36(4): 609-15.
52.
—. 1979. Fifteen years after treatment: A follow-up study of comprehensive vocationally-oriented psychotherapy. American Journal of Psychotherapy49(2): 240-5.
53.
Smith, M.L.1980. Publication bias and meta-analysis. Evaluation in Education4:22-4.
54.
Smith, M.L., and G.V. Glass.1977. Meta-analysis of psychotherapy outcome studies. American Psychologist4 (September): 7532-760.
55.
Twain, D.1983. Creating change in social settings. Beverly Hills, CA: Sage.
56.
Vigdal, G.L., D.W. Stadler, D.D. Goodrick, and D.J. Sutton.1980. Skills training in a program for problem drinking offenders. A one year follow-up evaluation. Journal of Offender Counseling, Services and Rehabilitation5(2): 61-73.
57.
Wachter, K.W., and M.L. Straf.1990. Thefuture of meta-analysis. Newbury Park, CA: Sage.
58.
Wanous, J.P., S.E. Sullivan, and J. Malinak.1989. The role of judgment calls in meta-analysis . Journal of Applied Psychology74(2): 259-64.
59.
Weisburd, D.L., L. Maher, and L. Sherman, with M.E. Buerger, E. Cohn, and A.J. Petrosino.1993. Crime-specific versus crime-general theory: The case of hotspots. Advances in Criminological Theory4:44-70.
60.
Weisburd, D.L., L. Sherman, and A. Petrosino.1990. Registry of randomized experiments in criminal sanctions, 1950-1983. Los Altos, CA: Sociometics Corporation, Data Holdings of the National Institute of Justice .
61.
Whitehead, J.T., and S.P. Lab.1989. A meta-analysis of juvenile correctional treatment. Journal of Research in Crime and Delinquency26:276-95.
62.
Wilson, J.Q.1975. Thinking about crime. New York: Basic Books.
63.
Yeaton, W.H., and P.M. Wortman.1993. On the reliability of meta-analytic reviews. Evaluation Review17(3): 292-309.
64.
Zeisel, H.1973. Reflections on experimental techniques in the law. The Journal of Legal Studies2(1): 107-24.
