Objective: The goal of this study is to advance an approach to the assessment of the quality of studies considered for inclusion in systematic reviews of the effects of social-care interventions. Method: To achieve this objective, quality is defined in relation to the widely accepted validity typology; prominent approaches to study quality assessment are evaluated as to their adequacy. Results: Problems with these approaches are identified. Conclusion: A formal, yet explicit, multidimensional approach to assessment grounded in substantive issues relevant to the intervention and the broader context in which it is embedded is promoted. Uncritical and exclusive use of indicators of study quality such as publication status, reporting quality, and single summative quality scores are rejected.
Get full access to this article
View all access options for this article.
References
1.
Aos, S., Phipps, P., Barnoski, R., & Lieb, R. (2001). The comparative costs and benefits of programs to reduce crime (Version 4.0, Doc. No. 01-05-1201). Olympia : Washington State Institute for Public Policy. http://www.wa.gov.wsipp.
2.
Begg, C.B. (1994). Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 399-409). New York: Russell Sage Foundation.
3.
Begg, C., Cho, M., Eastwood, S., Horton, R., Moher, D., Olkin, I., et al. (1996). Improving the quality of reporting of randomized controlled trials. The CONSORT statement. Journal of the American Medical Association, 276, 637-639.
4.
Boruch, R., Petrosino, A., & Chalmers, I. (1999). The Campbell collaboration: A proposal for systematic, multinational, and continuous reviews of evidence. In P. Davis, A. Petrosino, & I. Chalmers (Eds.), The effects of social and educational interventions: Developing an infrastructure for international collaboration to prepare, maintain and promote the accessibility of systematic reviews of relevant research (pp. 1-22). London: London University College, LondonSchool of Public Policy.
5.
Bossuyt, P.M., Reitsma, J.B., Bruns, D.E., Gatsonis, C.A., Glasziou, P.P., Irwig, L.M., et al. (2003). Towards complete and accurate reporting of studies of diagnostic accuracy: The STARD initiative. British Medical Journal, 326, 41-44.
6.
Brestan, E.V., & Eyberg, S.M. (1998). Effective psychosocial treatments of conduct-disordered children and adolescents: 29 years, 82 studies, and 5,272 kids. Journal of Clinical Child Psychology, 27, 180-189.
7.
Bushman, B.J., & Wells, G.L. (2001). Narrative impressions of literature: The availability bias and the corrective properties of meta-analytic approaches. Personal and Social Psychology Bulletin, 27, 1123-1130.
8.
Caetano, P. (2004). Standards for reporting non-randomized evaluations of behavioral and public health interventions: The TREND statement. Society for the Study of Addiction, 99, 1075-1080.
9.
Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally .
10.
Chalmers, T.C., Smith, H., Jr., Blackburn, B., Silverman, B., Schroeder, B., Reitman, D., & Ambroz, A. (1981). A method for assessing the quality of a randomized control trial. Controlled Clinical Trials, 2, 31-49.
11.
Chan, A.W., Hrobjartsson, A., Haar, M.T., Cotzsche, P.C., & Altman, D.G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles . JAMA, 291, 2457-2465.
12.
Cook, T., & Campbell, D. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally.
13.
Cooper, H. (1998). Synthesizing research: A guide for literature reviews (3rd ed.). Thousand Oaks, CA: Sage.
14.
Cooper, H., & Hedges, L. (Eds.). (1994). The handbook of research synthesis. New York: Russell Sage Foundation.
15.
Deeks, J.J., Dinnes, J., D'Amico, R., Sowden, A.J., Sakarovitch, C., Song, F., et al. (2003). Evaluating non-randomised intervention studies. Health Technology Assessment, 7(27). http://www.hta.nhsweb.nhs.uk/fullmono/mon727.pdf
16.
Dickersin, K. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments. Chichester, UK: John Wiley.
17.
Downs, S.H., & Black, N. (1998). The feasibility of creating a check-list for the assessment of the methodological quality both of randomized and nonrandomized studies of health care interventions. Journal of Epidemiology & Community Health, 52, 377-384.
18.
Farrington, D., Gottfredson, D., Sherman, L., & Welsh, B. (2002). The Maryland Scientific Methods Scale. In L. Sherman, D. Farrington, B. Welsh, & D. MacKenzie, Evidence-based crime prevention (pp. 13-21). London: Routledge .
19.
Flay, B.R., Biglan, A., Boruch, R.F., Castro, F.G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151-175.
20.
Gartlehner, G., Hansen, R., Nissman, D., Lohr, K., & Carey, T. (2006). Criteria for distinguishing effectiveness from efficacy trials in systematic reviews (Tech. Report 12 prepared by the RTI-International-University of North Carolina Evidence-based Practice Center under Contract No. 290-02-0016, AHRQ Publication No. 06-0046). Rockville, MD: Agency for Healthcare Research and Quality.
21.
Glazerman, S., Levy, D., & Myers, D. (2002). Nonexperimental replications of social experiments: A systematic review. Washington, DC: Mathematica Policy Research.
22.
Grayson, L. (2002). Evidence-based policy and the quality of evidence: Rethinking peer review. London: ESRC UK Centre for Evidence Based Policy and Practice, Department of Politics, Queen Mary, University of London. Retrieved December 3, 2006, from http://www.evidencenetwork.org .
23.
Harbour, R., & Miller, J. (2001). A new system for grading recommendations in evidence based guidelines. British Medical Journal, 323(7308), 334-346.
24.
Hawe, P., Shiell, A., & Riley, T. (2004). Complex interventions: How "out of control" can a randomised controlled trial be? British Medical Journal , 328, 1561-1563.
25.
Herbison, P., Hay-Smith, J., & Gillespie, W.J. (2006). Adjustment of meta-analyses on the basis of quality scores should be abandoned. Journal of Clinical Epidemiology, 59, 1249-1256.
26.
Higgins, J., & Green, S. (Eds.). (2006). Cochrane handbook for systematic reviews of interventions 4.2.6 (The Cochrane Library, Issue 4). Chichester, UK: John Wiley. Retrieved December 3, 2006, from http://www.cochrane.org/Resourceshandbook/handbookpdf
27.
Hopewell, S., McDonald, S., Clarke, M., & Egger, M. (2006). Grey literature in meta-analyses of randomized trials of health care interventions. In The Cochrane database of systematic reviews, 2006, Issue 2. Chichester, UK: John Wiley.
28.
Jüni, P., Altman, D.G., & Egger, M. (2001). Assessing the quality of controlled clinical trials . British Medical Journal, 323(7303), 42-46.
29.
Jüni, P., Witschi, A., Bloch, R., & Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282(11), 1054-1061.
30.
Kunz, R., Vist, G., & Oxman, A.D. (2002). Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Methodology Reviews , No. 4. Art. No.: MR000012. DOI: 10.1002/14651858.MR000012.
31.
Larzelere, R.E., Kuhn, B.R., & Johnson, B. (2004). The intervention selection bias: An underrecognized confound in intervention research. Psychological Bulletin , 130, 289-303.
Littell, J.H. (2008). How do we know what works? The quality of published reviews of evidence-based practices. In D. Lindsey & A. Shlonsky (Eds.), Child welfare research: Advances for practice and policy (pp. 66-93). New York: Oxford University Press.
34.
Littell, J.H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. New York: Oxford University Press.
35.
Mahoney, M.J. (1997). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2), 161-175.
36.
Miller, W., Andrews, N., Wilbourne, P., & Bennet, M. (1998). A wealth of alternatives: Effective treatments for alcohol problems. In W. Miller & N. Heather (Eds.), Treating addictive behaviors (2nd ed., pp. 203-216). New York: Plenum.
37.
Miller, W., & Wilbourne, P. (2002). Mesa grande: A methodological analysis of clinical trials of treatments for alcohol use disorders. Addiction , 97, 265-277.
38.
Moher, D., Cook, D.J., Eastwood, S., Olkin, I., Rennie, D., Stroup, D.F., et al. (1999). Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. The Lancet, 354, 1896-1900.
39.
Moher, D., Jadad, A., Nichol, G., Penman, M., Tugwell, P., & Walsh, S. (1995). Assessing the quality of randomized controlled trials: An annotated bibliography of scales and checklists. Controlled Clinical Trials, 16, 62-73.
40.
Moher, D., Schulz, K., & Altman, D. (2001). The CONSORT statement: Revised recommendations for improving the quality of parallel-group randomized trials. Annals of Internal Medicine, 134(8), 657-662.
41.
Moher, D., Tetzlaff, J., Tricco, A.C., Sampson, M., & Altman, D.G. (2007). Epidemiology and reporting characteristics of systematic reviews. PLoS Med, 4(3), e78.
42.
Morgan, D.L. (2007). Paradigms lost and pragmatism regained: Methodological implications of combining qualitative and quantitative methods. Journal of Mixed Methods Research, 1(1), 48-76.
43.
Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Malden, MA: Blackwell.
44.
Rosenbaum, P. (1995). Observational studies. New York: Springer.
45.
Rothstein, H., Sutton, A. J., & Bornstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Chichester, UK: Wiley.
46.
Scherer, R.W., Langenberg, P., & von Elm, E. (2007). Full publication of results initially presented in abstracts. Cochrane Database of Systematic Reviews, Issue 2.
47.
Schultz, K.F., Chalmers, I., Hayes, R.J., & Altman, D.G. (1995). Empirical evidence of bias: Dimensions of methodological quality associated with estimates of treatment effects in controlled trials . Journal of the American Medical Association, 273, 408-412.
48.
Schulz, K.F., & Grimes, D.A. (2002). Allocation concealment in randomised trials: Defending against deciphering. The Lancet, 359, 614-618.
49.
Shadish, W.R., & Baldwin, S.A. (2005). Effects of behavioral marital therapy: A meta-analysis of randomized controlled trials. Journal of Consulting and Clinical Psychology, 73, 6-14.
50.
Shadish, W., Cook, T., Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.
51.
Shadish, W., & Myers, D. (2004). Campbell Collaboration research design policy brief. http://www.campbellcollaboration.org/MG/ResDesPolicyBrief.pdf
52.
Shea, B.J., Grimshaw, J.M., Wells, G.A., Boers, M., Andersson, N., Hamel, C., et al. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7. Retrieved March 22, 2008, from http://www.biomedcentral.com/content/pdf/1471-2288-7-10.pdf.
53.
Sherman, L., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1998, July). Preventing crime: What works, what doesn't, what's promising. National Institute of Justice Research in Brief. Washington, DC: U.S. Department of Justice, Office of Justice Programs, National Institute of Justice.
54.
Smedslund, G., Hagen, K.B., Steiro, A., Johme, T., Dalsbø, T.K., & Rud, M.G. (2006). Work programmes for welfare recipients. The Campbell Collaboration Library. Retrieved March 22, 2008, from http://www.ncchta.org/fullmono/mon219.pdf
55.
Song, F., Eastwood, A.J., Gilbody, S., Duley, L., & Sutton, A.J. (2000). Publication and related biases. Health Technology Assessment, 4(10), 1-115.
56.
Sutton, A.J. (2005). Evidence concerning the consequences of publication and related biases. In H. Rothstein, A. J. Sutton, & M. Bornstein (Eds.), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 175-192). Chichester, UK: Wiley.
57.
Sutton, A.J., Abrams, K.R., Jones, D.R., Sheldon, T.A., & Song, F. (1998). Systematic reviews of trials and other studies . Health Technolology Assessment, 2(19). Retrieved March 22, 2008, from http://www.ncchta.org/fullmono/mon219.pdf.
58.
Torgerson, C.J. (2006). Publication bias: The Achilles' heel of systematic reviews?British Journal of Educational Studies, 54, 89-102. U.S. Department of Education. (2006). What works clearinghouse evidence standards for reviewing studies. Retrieved June 1, 2007, from http://www.samhsa.gov/review-criteria.htm.
59.
U.S. Department of Health and Human Services. (n.d.). National registry of evidence-based programs and practices (NREPP) review criteria. Retrieved June 1, 2006, from http://www.whatworksclearinghouse.org/reviewprocess/standards.html
60.
West, S., King, V., Carey, T., Lohr, K., McKoy, N., Sutton, S., et al. (2002). Systems to rate the strength of scientific evidence (Evidence Report/Technical Assistance No. 47, prepared by the Research Triangle Institute-University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011, Agency for Healthcare Research and Quality Publication No. 02-E016). Rockville, MD: Agency for Healthcare Research and Quality. Retrieved August 25, 2006, from http://www.ahrq.gov/clinic/epcsums/strengthsum.htm#contents
61.
Wilson, D., & Lipsey, M. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods , 6, 413-429.
62.
Wilson, D.B., MacKenzie, D.L., & Mitchell, F.N. (2008). The effects of correctional boot camps on offending. The Campbell Collaboration Library. Retrieved March 22, 2008, from http://www.campbellcollaboration.org/doc-pdf/Wilson_bootcamps_review__abstract.pdf
63.
Wortman, P. (1994). Judging research quality. In H. Cooper & L. Hedges (Eds.), The handbook of research synthesis (pp. 97-109). New York: Russell Sage Foundation.
64.
Zaza, S., Briss, P.A., & Harris, K.W. (2005). The guide to community preventive services: What works to promote health?New York: Oxford University Press. Retrieved June 1, 2007, from http://www.thecommunityguide.org/library/book/default.htm
