Shoestring Evaluation: Designing Impact Evaluations under Budget,Time and Data Constraints

Abstract

The paper discusses two common scenarios in which evaluators must conduct impact evaluations when working under budget, time, or data constraints. Under the first scenario the evaluator is not called in until the project is already well advanced, and there is a tight deadline for completing the evaluation, frequently combined with a limited budget and without access to baseline data. Under the second scenario the evaluator is called in early, but for budget, political, or methodological reasons it is not possible to collect baseline data on a control group and sometimes not even on the project population. As a result of these constraints, many of the basic principles of impact evaluation design (comparable pretest-posttest design, control group, instrument development and testing, random sample selection, control for researcher bias, thorough documentation of the evaluation methodology, etc.) are often sacrificed. We describe the “Shoestring Evaluation” approach which is being developed to provide tools for ensuring the highest quality evaluation possible under constraints of limited budget, time and data availability. While most of the data collection and analysis techniques will be familiar to experienced evaluators, what is new is the combination of these techniques into an integrated six-step approach which covers: (1) planning and scoping the evaluation, (2-4) options for dealing with constraints related to costs, time and data availability (which could include reconstructing baseline conditions and control groups), (5) identifying the strengths and weaknesses (threats to validity and adequacy) of the evaluation design, and (6) taking measures to address the threats and strengthen the evaluation design and conclusions. When necessary, many of these corrective measures can be introduced at a very late stage, even when the draft evaluation report has already been produced. Although the approach was designed to assist evaluators working in developing countries, the principles are equally applicable for conducting evaluations under similar constraints in industrial nations. Case studies are included to illustrate the different Shoestring approaches, drawing from the authors’ experience in Eastern Europe, Africa, Asia, and Latin America.

Get full access to this article

View all access options for this article.

References

Baker, J. (2000). Evaluating the impacts of development projects on poverty: A handbook for practitioners. Directions in Development Series. Washington, DC: The World Bank.

Bamberger, M. (Ed.). (2000a). Integrating quantitative and qualitative research in development projects. Directions in Development Series. Washington, DC: The World Bank

Bamberger, M. (2000b). The evaluation of international development programs: A view from the front. American Journal of Evaluation, 21, 95-102.

Bamberger, M. , Blackden, M. , Fort, L. , & Manoukian, V. (2001). Gender. In J. Klugman (Ed.), A sourcebook for poverty reductions strategies (Vol. 1, pp. 333-376). Washington, DC: The World Bank. Available on the web at www.worldbank.org/poverty.

Caldwell, J. (1985). Strengths and limitations of the survey method approach for measuring and understanding fertility change: Alternative possibilities. In J. Cleland & J. Hobcraft (Eds.), Reproductive change in developing countries: Insights from the World Fertility Survey (pp 45-63). Oxford: Oxford University Press.

Cook, T. , & Campbell, D. (1978). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally

Dayal, R. , van Wijk, C. , & Mukherjee, N. (2000). Methodology for participatory assessments with communities, institutions and policy makers: Linking sustainability with demand, gender and poverty. Water and sanitation program. Washington, DC: The World Bank.

Deaton, A. , & Grosh, M. (2002). Consumption. In M. Grosh & P. Glewwe (Eds.), Designing household survey questionnaires for developing countries: Lessons from 15 years of the Living Standards Measurement Study. Washington, DC: World Bank.

Dimitrov, T. (in press). Enhancing the performance of a major environmental project through a focused mid-term evaluation: The Kombinat Za Cvetni Metali environmental improvement project in Bulgaria.

10.

Gomez, L. (2000). Gender analysis of two components of the World Bank transport projects in Lima. Peru: Bikepaths and busways. www.worldbank.org/gender/transport/grants/Peru1GP/handout.

11.

Grosh, M. , & Glewwe, P. (Eds.). (2002). Designing household survey questionnaires for developing countries: Lessons from 15 years of the living standards measurement study. Washington, DC: World Bank.

12.

Guba, E. , & Lincoln, Y. (1989) Fourth generation evaluation. Thousand Oaks, CA: Sage.

13.

Hashemi, S. , Schuler, S. R. , & Riley, A. P. (1996). Rural credit programs and women’s empowerment in Bangladesh. World Development, 24, 635-653.

14.

Hentschel, J. (1999). Contextuality and data collection methods: A framework and application to health service utilization. The Journal of Development Studies, 35(4), 64-94.

15.

Justice, J. (1986). Policies, plans and people: Culture and health development in Nepal. Berkeley: University of California Press.

16.

Kim, J. , Alderman, H. , & Orazem, P. (1999). Can private school subsidies increase schooling for the poor? The Quetta Urban Fellowship Program. World Bank Economic Review, 13, 443-466.

17.

Khandker, S. (1998). Fighting poverty with microcredit: Experience in Bangladesh. Oxford: Oxford University Press.

18.

Kumar, K. (Ed.). (1993). Rapid appraisal methods: Regional and sectoral studies. Washington, DC: The World Bank.

19.

Leeuw, F. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24, 5-20.

20.

Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage Publications.

21.

Lipsey, M. W. (1993). Theory as method: Small theories of treatment. In L. Sechrest & A. Scott (Eds.), Understanding causes and generalizing about them. New Directions for Program Evaluation, No. 57. San Francisco: Jossey-Bass.

22.

Lipsey, M. W. , & Wilson, D. (1993). The efficacy of psychological, educational and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48, 1181-1209.

23.

Miles, M. , & Huberman, M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage.

24.

Moore, D. , & McCabe, G. (1999). Introduction to the practice of statistics. New York: Freeman.

25.

Newman, C. (2001, February). Gender, time use, and change: Impacts of agricultural export employment in Ecuador. Policy research report on gender and development working paper series no. 18. Poverty Reduction and Economic Management Network/Development Research Group. The World Bank. Available on the web at www.worldbank.org/gender/prr.

26.

Patton, M. Q. (1997). Utilization-focused evaluation (3rd ed.). Thousand Oaks, CA: Sage.

27.

Patton, M. Q. (2002). Qualitative research and evaluation methods. Thousand Oaks, CA: Sage.

28.

Paul, S. (2002). Holding the state to account: Citizen monitoring in action. Bangalore: Books for Change ActionAid.

29.

Pebley, A. , Noreen, N. , & Choe, M. K. (1986). Evaluation of contraceptive history data in the Republic of Korea. Studies in Family Planning, 17, 22-35.

30.

Pratt, C. , McGuigan, W. , & Katzev, A. (2002). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-350.

31.

Rietberger-McCracken, J. , & Narayan, D. (1997). Participatory rural appraisal. Module III of Participatory tools and techniques: A resource kit for participation and social assessment. Environment Department. Washington, DC: The World Bank.

32.

Rogers, P. , Hacsi, T. , Petrosino, A. , & Huebner, T. (Eds.). (2000). Program theory in evaluation: Challenges and opportunities. New Directions for Evaluation, No. 87. San Francisco: Jossey-Bass.

33.

Rogers, P. , Petrosino, A. , Huebner, T. , & Hacsi, T. (2000). Program theory evaluation: Practice, promise and problems. In P. Rogers , T. Hacsi , A. Petrosino , & T. Huebner (Eds.), Program theory in evaluation: Challenges and opportunities. New Directions for Evaluation, No. 87. San Francisco: Jossey-Bass.

34.

Schwandt, T.A. (1990). On judging trustworthiness in interpretivist methodologies. Paper presented at the Annual Meeting of the American Educational Research Association, Boston.

35.

Schwarz, N. , & Oyserman, D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. American Journal of Evaluation, 22, 127-160.

36.

Shadish, W. , Cook, T. D. , & Campbell, D. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Miflin.

37.

Shadish, W. , Newman, D. , Scheirer, M. A. , & Wye, C. (1995). Guiding principles for evaluators.New Directions for Program Evaluation, No. 66. San Francisco: Jossey-Bass.

38.

Valadez, J. , & Bamberger, M. (1994). Monitoring and evaluating social programs in developing countries: A handbook for policymakers, managers and researchers. Washington, DC: World Bank.

39.

Weiss, C. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21, 501-524.

40.

Weitzman, B. , Silver, D. , & Dillman, K. N. (2002). Integrating a comparison group design into a theory of change evaluation: The case of the Urban Health Initiative. American Journal of Evaluation, 23, 371-386.

41.

World Bank . (1993). Early experience with involuntary resettlement: Impact evaluation on India: Maharashtra Irrigation Project. Operations Evaluation Department. Washington, DC: The World Bank.

42.

Yin, R. (2003) Case study research: Design and methods (3rd ed.). Thousand Oaks, CA: Sage.