Most K–12 evaluations are designed to make inferences about how a program implemented at the classroom or school level affects student learning outcomes and such inferences inherently involve hierarchical data structure. One methodological challenge for evaluators is linking program implementation factors typically measured at the classroom or teacher level with student outcomes measured at the individual student level. Hierarchical linear modeling (HLM) is ideal for K–12 program evaluations because it can appropriately handle hierarchical data while allowing for a nuanced conceptualization of evaluation questions. The authors illustrate the advantages of HLM with an example of the evaluation of a technology-based reading program for elementary students. HLM enabled evaluators to have a deeper understanding of the relationship between program implementation and program outcomes by (a) providing a better and more proper estimate of the average program outcome, (b) providing an estimate of variation in student outcomes within classrooms, between classrooms, and between schools, and (c) most importantly, by providing a framework for probing what factors are related to variations in student outcomes. Through illustrating the potentials of HLM, the authors aim to advance and expand evaluative tools for establishing empirical evidence regarding program effectiveness in K–12 settings. The authors also hope to bring balance and additional insight into the current methodological discussions about the relationship between program implementation and program impact, rather than merely demonstrating the average effect of a program or policy intervention.
Berry, T., & Eddy, R. ( 2008). Consequences of No child left behind for educational evaluation. New Direction for Evaluation, 117. San Francisco : CA Jossey-Bass.
2.
Bloom, H.S., & Riccio, J.A. ( 2002). Using place-based random assignment and comparative interrupted time-series analysis to evaluate the Jobs-Plus Employment Program for public housing residents. New York, NY: MDRC.
3.
Bryk, A.S., & Raudenbush, S.W. (1992). Hierarchical linear models for social and behavioral research: Applications and data analysis methods . Newbury Park, CA: SAGE.
4.
Burstein, L. ( 1980). The analysis of multi-level data in educational research and evaluation. Review of Educational Research in Education , 8, 158-233.
5.
Campbell, D.T., & Stanley, J.C. ( 1963). Experimental and quasi-experimental designs for research . Chicago, IL: RandMcNally.
6.
Cronbach, L. ( 1963). Course improvement through evaluation. Teachers College Record, 64, 672-683.
7.
Cronbach, L. ( 1976). Research on classrooms and schools: Formulation of questions, design and analysis. Occasional paper of the Stanford Evaluation Consortium, School of Education, Stanford University .
8.
Cronbach, L.J. ( 1982). Designing evaluations of educational and social programs . San Francisco, CA: Jossey-Bass .
9.
De Leeuw, J. ( 2002). Series editor’s introduction to hierarchical linear models. In Stephen W. Raudenbush & Anthony S. Bryk (Eds.), Hierarchical linear models: Applications and data analysis method (2nd ed.). Thousand Oaks, CA: SAGE .
10.
Donaldson, S., Christie, T.C., & Mark, M.M. ( 2009). What counts as credible evidence in applied research and evaluation practice?Thousand Oaks, CA: SAGE.
11.
Edlin, A., Gelman, A., & Kaplan, N. ( 2007). Voting as a rational choice. Rationality and Society, 19, 293-314.
12.
Hansen, E.E., Llosa, L., & Slayton, J. ( 2004). Evaluation of the Waterford Early Reading Program as a supplementary program in the Los Angeles Unified School District 2002-2003. Planning, Assessment and Research Division Publication No. 177. Program Evaluation and Research Branch, Los Angeles Unified School District .
13.
Hopkins, K. ( 1982). The unit of analysis: Group means versus individual observations . American Educational Research Journal, 19, 5-18.
14.
Institute for Education Sciences. (2009). Request for applications: Evaluation of state and local education programs and policies (CFDA Number: 84.305E). Retrieved from http://ies.ed.gov/funding/pdf/2010_84305E.pdf
15.
Johnson, R.S. ( 2002). Using data to close the achievement gap: How to measure equity in our schools?Thousand Oaks, CA: Corwin Press, Inc.
16.
Kothari, A., Edwards, N., Yanicki, S., Hansen-Ketchum, P., & Kennedy, M.A. ( 2007). Socioecological Models: Strengthening intervention research in tobacco control. Drogues, santé et société , 6, 1-24.
17.
Lee, V.E., Loeb, S., & Lubeck, S. ( 1998). Contextual effects of prekindergarten classrooms for disadvantaged children on cognitive development: The case of Chapter 1. Child Development, 69, 479-494.
18.
Lin, C.Y., Gelman, A., Price, P.N., & Krantz, D.H. ( 1999). Analysis of local decisions using hierarchical modeling, Applied to home radon measurement and remediation. Statistical Science, 14, 305-328.
19.
Llosa, L., & Slayton, J. ( 2008). Using program evaluation to inform and improve the education of young English language learners in US schools. Language Teaching Research, 12, 497-516.
20.
Mark, M.M., Cooksy, L.J., & Trochim, W.M.K. (2009). Evaluation policy: An introduction and overview. In W. M. K. Trochim, M. M. Mark, & L. J. Cooksky (Eds.), Evaluation policy and evaluation practice. New Direction for Evaluation (Vol. 123, pp. 3-11). Hoboken, NJ: Wiley Periodicals, Inc.
21.
Mason, W.M., Wong, G.M., & Entwistle, B. ( 1983). Contextual analysis through the multilevel linear model . In S. Leinhardt (Ed.), Sociological methodology (pp. 72-103). San Francisco, CA: Jossey-Bass.
22.
Muijs, D., & Reynolds, D. ( 2000). School effectiveness and teacher effectiveness in mathematics: Some preliminary findings from the evaluation of the mathematics enhancement programme (Primary). School Effectiveness and School Improvement , 11, 273-303.
23.
Murray, D.M. ( 1998). Design and analysis of group-randomized trials. New York, NY: Oxford University Press.
24.
Nastasi, B.K., & Hitchcock, J. ( 2009). Challenges of evaluating multilevel interventions. American Journal of Community Psychology, 43, 360-376.
25.
Nye, B., Konstantopoulos, S., & Hedges, L.V. ( 2004). How large are teacher effects?Educational Evaluation and Policy Analysis, 26, 237-257.
Popham, J. ( 1988). Educational evaluation (2nd ed.), Chapter 1: Educational evaluation: A history without mystery. Englewood, NJ: Prentice Hall.
28.
Rabe-Hesketh, S., & Skrondal, A. ( 2008). Multilevel and longitudinal modeling using Stata (2nd edition). College Station, Texas: Stata Press.
29.
Raudenbush, S., & Bryk, A.S. ( 1989). Methodological advances in analyzing the effects of schools and classrooms on student learning. Review of Research in Education, 15, 423-475.
30.
Raudenbush, S.W., & Bryk, A.S. ( 2002). Hierarchical linear models: Application and data analysis methods (2nd ed.). Newbury Park, CA: SAGE.
31.
Schneider, B., Carnoy, M., Kilpatrick, J., Schmidt, W.H., & Shavelson, R.J. (2007). Estimating causal effects: Using experimental and observational designs (report form the Governing Board of the American Educational Research Association Grants Program). Washington, DC: American Educational Research Association .
32.
Seltzer, M. ( 1994). Studying variation in program success: A multilevel modeling approach. Evaluation Review, 18, 342-361.
33.
Seltzer, M. ( 2003). The use of hierarchical models in analyzing data from experiments and quasi-experiments conducted in field settings. Los Angeles, CA: University of California.
34.
Sikkema, K.J. ( 2005). HIV prevention among women in low-income housing developments: Issues and intervention outcomes in a place-based randomized controlled trial . The Annals of the American Academy of Political and Social Science, 599, 52-70.
35.
Singer, J.D., Fuller, B., Keiley, M.K., & Wolf, A. ( 1998). Early child-care selection: Variation by Geographic location, maternal characteristics, and family structure. Development Psychology, 34, 1129-1144.
36.
Slavin, R.E. ( 2008). Perspectives on evidence-based research in education-What works? Issues in synthesizing educational program evaluations. Educational Researcher, 37, 5-14.
37.
Slayton, J., & Llosa, L. ( 2002). Evaluation of the Waterford early reading program 2001-2002: Implementation and student achievement. Planning, Assessment and Research Division Publication No. 144. Program Evaluation and Research Branch, Los Angeles Unified School District.
38.
Slayton, J., & Llosa, L. ( 2005). The use of qualitative methods in large-scale evaluation: Improving the quality of the evaluation and the meaningfulness of the findings . Teachers College Record, 107, 2543-2565.
39.
Sloan, F. ( 2008). Through the looking glass: Experiments, quasi-experiment, and the medical model. Educational Researcher, 37, 41-46.
40.
Spybrook, J. ( 2008). Are power analyses reported with adequate detail? Evidence from the first wave of group randomized trials funded by the institute of education sciences. Journal of Research on Educational Effectiveness , 1, 215-235.
41.
Teruel, G.M., & Davis, B. ( 2000). Final report: An evaluation of the impact of PROGRESA cash payments on private inter-household transfers. Washington, DC: International Food Policy Research Institute.
42.
Trickett, E.J. ( 2009). Multilevel community-based culturally situated interventions and community impact: An ecological perspective. American Journal of Community Psychology, 43, 257-266.
43.
Tyler, R. ( 1991). General statement on program evaluation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education at quarter century: Ninetieth yearbook of the national society for the study of education (pp. 3-17). IL: University of Chicago Press.
44.
Weisburd, D. ( 2005). Hot spots policing experiments and criminal justice research: Lessons from the field. The Annals of the American Academy of Political and Social Science, 599, 220-245.
45.
Wiley, D. ( 1970). Design and analysis of evaluation studies. In M. C. Wittrock & D. Wiley (Eds.), The evaluation of instruction: Issues and problems (pp. 259-271). New York, NY: Holt, Rinehart & Winston.
46.
Woodcock, R.W. ( 1987). Woodcock reading mastery tests-revised: Forms G and H . Circles Pines, MN: American Guidance Service.
