This article raises some questions about the usefulness of meta-analysis as a means of reviewing quantitative research in the social sciences. When a meta-analytic model for SAT coaching is used to predict results from future studies, the amount of prediction error is quite large. Interpretations of meta-analytic regressions and quantifications of program and study characteristics are shown to be equivocal. The match between the assumptions of the meta-analytic model and the data from SAT coaching studies is not good, making statistical inferences problematic. Researcher subjectivity is no less problematic in the context of a meta-analysis than in a narrative review.
Alderman, D. L., and D. E. Powers. 1980. The effects of special preparation on SAT-verbal scores. American Educational Research Journal17:239-253.
2.
Anastasi, A.1981. Coaching, test sophistication, and developed abilities. American Psychologist36 (10): 1086-1093.
3.
Becker, B. J.1988. Synthesizing standardized mean-change measures. British Journal of Mathematical and Statistical Psychology41:257-278.
4.
Becker, B. J.1990. Coaching for the Scholastic Aptitude Test: Further synthesis and appraisal. Review of Educational Research60 (3): 373-417.
5.
Berk, R. A.2004. Regression analysis: A constructive critique. Thousand Oaks, CA: Sage.
6.
Berk, R. A., and D. A. Freedman. 2003. Statistical assumptions as empirical commitments. In Law, punishment, and social control: Essays in honor of Sheldon Messinger, 2nd ed., ed. T. G. Blomberg and S. Cohen, 235-254. http://www.stat.berkeley.edu/census/berk2.pdf.
7.
Bond, L.1989. The effects of special preparation on measures of scholastic ability. In Educational measurement, ed. R. L. Linn, 429-444. New York: American Council on Education/Macmillan.
8.
Bonde, L. A., and A. S. Magistrate. 1987. Writer’s guide: Psychology. Lexington, MA: D. C. Heath.
9.
Briggs, D. C.2001. The effect of admissions test preparation: Evidence from NELS:88. Chance14 (1): 10-18.
10.
Briggs, D. C.2002a. Comment: Jack Kaplan’s A new study of SAT coaching. Chance15 (1): 7-8.
11.
Briggs, D. C. 2002b. SAT coaching, bias and causal inference. Ph.D. diss., University of California, Berkeley.
12.
Briggs, D. C.2004a. Causal inference and the Heckman model. Journal of Educational and Behavioral Statistics29 (4): 397-420.
13.
Briggs, D. C. 2004b. Evaluating SAT coaching: Gains, effects and self-selection. In Rethinking the SAT: Perspectives based on the November 2001 conference at the University of California, Santa Barbara, ed. by R. Zwick, 217-234. New York: Routledge Falmer.
14.
Burke, K. B. 1986. A model reading course and its effect on the verbal scores of eleventh and twelfth grade students on the Nelson Denny Test, the Preliminary Scholastic Aptitude Test, and the Scholastic Aptitude Test. Ph.D. diss., Georgia State University, Atlanta.
15.
Coffin, G. C. 1987. Computer as a tool in SAT preparation. Paper presented at the Florida Instructional Computing Conference, Orlando, FL.
16.
Coffman, W. E., and M. E. Parry. 1967. Effects of an accelerated reading course on SAT-V scores. Personnel and Guidance Journal46:292-296.
17.
Cole, N.1982. The implications of coaching for ability testing. In Ability testing: Uses, consequences, and controversies. Part II: documentation section, ed. A. K. Wigdor and W. R. Gardner. Washington, DC: National Academy Press.
18.
Curran, R. G. 1988. The effectiveness of computerized coaching for the Preliminary Scholastic Aptitude Test (PSAT/NMSQT) and the Scholastic Aptitude Test (SAT). Ph.D. diss., Boston University.
19.
Cronbach, L. J.1982. Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
20.
Dear, R. E.1958. The effect of intensive coaching on SAT scores. Princeton, NJ: Educational Testing Service.
21.
DerSimonian, R., and N. M. Laird. 1983. Evaluating the effect of coaching on SAT scores: A meta-analysis. Harvard Educational Review53:1-15.
22.
Dyer, H. S.1953. Does coaching help?College Board Review19:331-335.
23.
Evans, F., and L. Pike. 1973. The effects of instruction for three mathematics item formats. Journal of Educational Measurement10 (4): 257-272.
24.
Federal Trade Commission, Boston Regional Office. 1978. Staff memorandum of the Boston Regional Office of the Federal Trade Commission: The effects of coaching on standardized admission examinations. Boston: Federal Trade Commission, Boston Regional Office.
25.
Federal Trade Commission, Bureau of Consumer Protection. 1979. Effects of coaching standardized admission examinations: Revised statistical analyses of data gathered by the Boston Regional Office of the Federal Trade Commission. Washington, DC: Federal Trade Commission, Bureau of Consumer Protection.
26.
Fraker, G. A.1987. The Princeton Review reviewed. In The Newsletter. Deerfield, MA: Deerfield Academy.
27.
Frankel, E.1960. Effects of growth, practice, and coaching on Scholastic Aptitude Test scores. Personnel and Guidance Journal38:713-719.
28.
French, J. W.1955. The coach ability of the SAT in public schools. Princeton, NJ: Educational Testing Service.
29.
Glass, G. V. 2000. Meta-analysis at 25. http://glass.ed.asu.edu/gene/papers/meta25.html.
30.
Glass, G. V, B. McGaw, and M. L. Smith. 1981. Meta-analysis in social research. Beverly Hills, CA: Sage.
31.
Harvey, K. S. 1988. Videotaped versus live instruction as a coaching method for the mathematics portion of the scholastic aptitude test. Ph.D. diss., University of Georgia, Athens.
32.
Hedges, L. V.1990. Directions for future methodology. In The future of meta-analysis, ed. K. W. Wachter and M. L. Straf, 11-26. New York: Russell Sage Foundation.
33.
Hedges, L. V., and I. Olkin. 1985. Statistical methods for meta-analysis. New York: Academic Press.
34.
Holmes, C. T., and R. Keffer. 1995. A computerized method to teach Latin and Greek root words: Effect on verbal SAT scores. Journal of Educational Research89 (1): 47-50.
35.
Hopmeier, G. H. 1984. The effectiveness of computerized coaching for Scholastic Aptitude Test in individual and group modes. Ph.D. diss., Florida State University, Tallahassee.
36.
Jackson, R.1980. The Scholastic Aptitude Test: A response to Slack and Porter’s critical appraisal. Harvard Educational Review50 (3): 382-391.
37.
Johnson, S. T.1984. Preparing Black students for the SAT—Does it make a difference? An evaluation report of the NAACP Test Preparation Project. New York: National Association for the Advancement for Colored People.
38.
Kaplan, J.2002. A new study of SAT coaching. Chance14 (4): 1-6.
39.
Keefauver, L. W. 1976. The effects of a program of coaching on Scholastic Aptitude Test scores of high school seniors tested as juniors. Ph.D. diss., University of Tennessee at Knoxville.
40.
Kintisch, L. S.1979. Classroom techniques for improving Scholastic Aptitude Test scores. Journal of Reading22:416-419.
41.
Kulik, C.-L., and J. A. Kulik. 1988. Meta-analysis: Historical origins and contemporary practice. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
42.
Kulik, J. A., R. L. Bangert-Drowns, and C.-L. Kulik. 1984. Effectiveness of coaching for aptitude tests. Psychological Bulletin95:179-188.
43.
Laschewer, A. 1985. The effect of computer assisted instruction as a coaching technique for the scholastic aptitude test preparation of high school juniors. Ph.D. diss., Hofstra University, Hempstead, NY.
44.
Marron, J. E. 1965. Preparatory school test preparation: Special test preparation, its effect on College Board scores and the relationship of affected scores to subsequent college performance. West Point, NY: Research Division, Office of the Director of Admissions and Registrar, United States Military Academy.
45.
McClain, B. 1999. The impact of computer-assisted coaching on the elevation of twelfth-grade students’ SAT scores. Ph.D. diss., Morgan State University, Baltimore.
46.
Messick, S.1980. The effectiveness of coaching for the SAT: Review and reanalysis of research from the fifties to the FTC. Princeton, NJ: Educational Testing Service.
47.
Messick, S.1982. Issues of effectiveness and equity in the coaching controversy: Implications for educational and testing practice. The Educational Psychologist17 (2): 67-91.
48.
Messick, S., and A. Jungeblut. 1981. Time and method in coaching for the SAT. Psychological Bulletin89:191-216.
49.
Oakes, M.1986. Statistical inference: A commentary for the social and behavioral sciences. New York: John Wiley.
50.
Pallone, N. J.1961. Effects of short- and long-term developmental reading courses upon the S.A.T. verbal scores. Personnel and Guidance Journal39:654-657.
51.
Petitti, D. B.2000. Meta-analysis, decision analysis and cost-effectiveness analysis. 2nd ed.New York: Oxford University Press.
52.
Pike, L. W.1978. Short-term instruction, testwiseness, and the Scholastic Aptitude Test: A literature review with research recommendations. Princeton, NJ: Educational Testing Service.
53.
Powers, D. E.1993. Coaching for the SAT: A summary of the summaries and an update. Educational Measurement: Issues and Practice12 (2): 24-39.
54.
Powers, D. E., and D. A. Rock. 1999. Effects of coaching on SAT I: Reasoning test scores. Journal of Educational Measurement36 (2): 93-118.
55.
Reynolds, A., and G. O. Oberman. 1987. An analysis of a PSAT preparation program for urban gifted students. Paper presented at the annual meeting of the American Educational Research Association, Washington, DC, April.
56.
Roberts, S. O., and D. B. Openheim. 1966. The effect of special instruction upon test performance of high school students in Tennessee. Princeton, NJ: Educational Testing Service.
57.
Rock, D.1980. Disentangling coaching effects and differential growth in the FTC commercial coaching study. Princeton, NJ: Educational Testing Service.
58.
Schroeder, B. 1992. Problem-solving strategies and the mathematics SAT: A study of enhanced performance. Ph.D. diss., Teacher’s College, Columbia University, New York.
59.
Sesnowitz, M., K. Bernhardt, and M. D. Knain. 1982. An analysis of the impact of commercial test preparation on SAT scores. American Educational Research Journal19 (3): 429-441.
60.
Shadish, W., T. Cook, and D. Campbell. 2002. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton-Mifflin.
61.
Shaw, E. 1992. The effects of short-term coaching on the Scholastic Aptitude Test. Ph.D. diss., University of La Verne, La Verne, CA.
62.
Slack, W. V., and D. Porter. 1980. The Scholastic Aptitude Test: A critical appraisal. Harvard Education Review50: 154-175.
63.
Smyth, F. L.1989. Commercial coaching and SAT scores. Journal of College Admissions123:2-9.
64.
Smyth, F. L.1990. SAT coaching: What really happens and how we are led to expect more. Journal of College Admissions129:7-17.
65.
Snedecor, P. J.1989. Coaching: Does it pay? revisited. Journal of College Admissions125:15-18.
66.
Stroud, T. W. F.1980. Reanalysis of the Federal Trade Commission study of commercial coaching for the SAT. Princeton, NJ: Educational Testing Service.
67.
Wachter, K. W.1988. Disturbed by meta-analysis?Science241 (4872): 1407-1408.
68.
Whitla, D. K.1962. Effect of tutoring on Scholastic Aptitude Test scores. Personnel and Guidance Journal41:32-37.
69.
Whitla, D. K.1988. Coaching: Does it pay? Not for Harvard students. College Board Review148 (Summer): 32-35.
70.
Wing, C. W., R. A. Childs, and S. E. Maxwell. 1989. Some field observations of the impact of test preparatory programs on high school students’ Scholastic Aptitude Test scores. A report to the Awards Committee for Education and Wake Forest University, Winston-Salem, NC.
71.
Wrinkle, G. W. 1996. A Scholastic Assessment Test preparation class and its effect on Scholastic Assessment Test scores. Ph.D. diss., University of Houston, TX.
72.
Zuman, J. P. 1988. The effectiveness of special preparation for the SAT: An evaluation of a commercial coaching school. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA, April.
