A new generation of public programs emerges, which specifically addresses complex societal problems we witness today. Programs for these types of complex issues—in this article, we consider more closely the challenge of sustainable development—are characterized by emergent design, learning processes between diverse actors, and adaptive management. Managers of these kinds of programs have new demands for evaluation and evaluators. This article describes prevailing evaluation methods for sustainable development (progress assessment, goal-oriented program evaluation, and program theory evaluation) and the challenges they meet when confronted with the complexity of designing and conducting systemic intervention programs for sustainable development. The evaluation framework that we propose offers guiding principles to assist evaluators in evaluating complex programs.
Abma, T. ( 2001). Variaties in beleidsevaluaties: Meten, beoordelen en onderhandelen. In T. Abma & R. J. In ’t Veld (Eds.), Handboek beleidswetenschap (pp. 311-320). Amsterdam: Boom.
2.
Argyris, C., & Schön, D. (1974). Theory in practice: Increasing professional effectiveness. San Francisco, CA: Josey-Bass.
3.
Argyris, C., & Schön, D.A. (1978). Organizational learning: A theory of action perspective. Reading, MA: Addison Wesley.
4.
Armitage, D., Marschke, M., & Plummer, R. ( 2008). Adaptive co-management and the paradox of learning. Global Environmental Change, 18, 86-98.
5.
Attenborough, K. ( 2007). Soft systems in a hardening world: Evaluating urban regeneration . In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 75-88). Point Reyes, CA: Edge-Press/American Evaluation Association.
6.
Beck, U. ( 1986). Risikogesellschaft - Auf dem Weg eine andere Moderne . Frankfurt: Suhrkamp Verlag.
7.
Becker, E., Jahn, T., Stiess, I., & Wehling, P. ( 1997). Sustainability: A cross-disciplinary concept for social transformations (Policy Papers No. 6). Paris: UNESCO.
8.
Bell, S., & Morse, S. ( 2004). Experiences with sustainability indicators and stakeholder participation: A case study relating to a ‘blue plan’ project in Malta. Sustainable Development, 12, 1-14.
9.
Broerse, J.E.W. ( 1998). Towards a new development strategy. How to include small-scale farmers in the biotechnological innovation process. Delft : Eburon.
10.
Bunders, J. F. G. (Ed.). (1990). Biotechnology for small-scale farmers in developing countries. Analysis and assessment procedures. Amsterdam: VU University Press.
11.
Bunders, J.F.G., & Broerse, J.E.W. (1991). Appropriate biotechnology in small-scale agriculture: How to orient research and development. Wallingford: CAB International.
12.
Burns, D. ( 2007). Evaluation in complex governance arenas: The potential of large system action research. In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 181-196). Point Reyes, CA: EdgePress/ American Evaluation Association.
13.
Cabrera, D., Colosi, L., & Lobdell, C. ( 2008). Systems thinking. Evaluation and Program Planning, 31, 299-310.
14.
Cartland, J., Ruch-Ross, H.S., Mason, M., & Donohue, W. ( 2008). Role sharing between evaluators and stakeholders in practice . American Journal of Evaluation, 29, 460-477.
15.
Chen, H. ( 1990). Issues in constructing program theory. In L. Bickman (Ed.), Advances in program theory: New directions for evaluation (Vol. 47, pp. 7-18). San Francisco, CA: Jossey-Bass.
16.
Christie, C.A., & Alkin, M.C. ( 2003). The user-oriented evaluator’s role in formulating a program theory: Using a theory-driven approach. American Journal of Evaluation, 24, 373-385.
17.
Churchman, C.W. ( 1970). Operations research as a profession. Management Science, 17, B37-B53.
18.
Collins, H.M., & Evans, R. ( 2002). The third wave of science studies: Studies of expertise and experience. Social Studies of Science, 32, 235-296.
19.
Czarniawska, B., & Joerges, B. ( 1996). Travels of ideas. In B. Czarniawska & G. Sevon (Eds.), Translating organizational change (pp. 13-48). Berlin, New York: Walter de Gruyter.
20.
Dart, J., & Davies, R. ( 2003). A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24, 137-155.
21.
De Lancer Julnes, P. ( 2006). Performance measurement: An effective tool for government accountability? Evaluation, 12, 219-235.
22.
Dyehouse, M., Bennet, D., Harbor, J., Childress, A., & Dark, M. ( 2009). A comparison of linear and systems thinking approaches for program evaluation illustrated using the Indiana Interdisciplinary GK-12 . Evaluation and Program Planning, , doi:10.1016/j.evalprogplan.2009.03.001.
23.
Eoyang, G.H. ( 2007). Human systems dynamics: Complexity-based approach to a complex evaluation. In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 123-140). Point Reyes, CA: EdgePress/American Evaluation Association.
24.
Forrest, J. ( 2007). Evolutionary and behavioral characteristics of systems . In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 197-210). Point Reyes, CA: EdgePress/American Evaluation Association.
25.
Friedman, V.J. ( 2001). Designed blindness: An action science perspective on program theory evaluation. American Journal of Evaluation , 22, 161-181.
26.
Friedman, V.J., Rothman, J., & Withers, B. ( 2006). The power of why: Engaging the goal paradox in program evaluation. American Journal of Evaluation, 27, 201-218.
27.
Funtowicz, S.O., & Ravetz, J.R. ( 1993). Science for the post-normal age. Futures , 25, 739-755.
28.
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. ( 1994). The new production of knowledge: The dynamics of science and research in contemporary societies. London: Sage.
29.
Giddens, A. ( 1984). The constitution of society. Outline of the theory of structuration. Cambridge: Polity .
30.
Gore, A. ( 2006). An Inconvenient Truth: The Planetary Emergency of Global Warming and What We Can Do About It. Rodale Press in Emmaus, Pennsylvania.
31.
Greene, J.C., & Abma, T.A. ( 2001). Editors’ notes, special issue: Responsive evaluation . New Directions for Evaluation, 2001, 1-6.
32.
Guba, E.G., & Lincoln, Y. ( 1989). Fourth generation evaluation. Newbury Park, London, New Delhi: Sage.
33.
Hajer, M.A. ( 2003). Policy without polity? Policy analysis and the institutional void. Policy Sciences, 36, 175-195.
34.
Hendriks, C.M., & Grin, J. ( 2007). Contextualizing reflexive governance: The politics of dutch transitions to sustainability. Journal of Environmental Policy & Planning, 9, 333-350.
35.
Hildén, M., & Rosenström, U. (2008). The use of indicators for sustainable development. Sustainable Development, 16, 237-240.
36.
Hisschemöller, M., & Hoppe, R. ( 1996). Coping with intractable controversies: The case for problem structuring in policy design and analysis. Knowledge and Policy , 8, 40-60.
37.
Hoes, A.-C., Regeer, B.J., & Bunders, J.F.G. (2008). Transformers in knowledge production: Building science-practice collaborations. Action Learning: Research & Practice, 5, 207-220.
38.
Imam, I., LaGoy, A., & Williams, B. ( 2007). Introduction. In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 3-10). Point Reyes, CA: EdgePress/American Evaluation Association.
39.
Jasanoff, S. (Ed.). (2004). States of knowledge. The co-production of science and social order. London, New York: Routledge .
40.
Kemmis, S., & McTaggert, R. ( 1988). The action research planner. Victoria : Deadin University Press.
41.
Kemp, R., Parto, S., & Gibson, R.B. ( 2005). Governance for sustainable development: Moving from theory to practice. International Journal of sustainable development , 8, 12-30.
42.
King, J.A. ( 2008). Bringing evaluative learning to life. American Journal of Evaluation, 29, 151-155.
43.
Kleiner, A., & Roth, G. ( 1996). Field manual for the learning historian. MIT, Center for Organizational Learning.
44.
Kunneman, H. ( 2006). Voorbij het dikke-ik. Bouwstenen voor een kritisch humanisme. HuministicsUniversity Press.
Loorbach, D. ( 2007). Transition management. New mode of governance for sustainable development. Utrecht: International Books.
47.
Marra, M. ( 2004). The contribution of evaluation to socialization and externalization of tacit knowledge. Evaluation, 10, 263-283.
48.
McCoy, M., & Hargie, O.D.W. (2001). Evaluating evaluation: Implications for assessing quality. International Journal of Health Care Quality Assurance, 14, 317-327.
49.
Meadows, D., Meadows, D., & Randers, J. ( 2004). Limits to growth. The 30-year update. White River Jct., Vermont: Chelsea Green Publishing .
50.
Meadows, D.H., Meadows, D.L., Randers, J., & Behrens, W.W., III. (1972). The limits to growth. New York: Universe Books.
51.
Midgley, G. ( 2007). Systems thinking for evaluation. In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 11-34). Point Reyes, CA: EdgePress/ American Evaluation Association.
52.
Nowotny, H., Scott, P., & Gibbons, M. ( 2001). Re-thinking science. Knowledge and the public in an age of uncertainty. Oxford: Polity.
53.
Parsons, B.A. ( 2007). The state of methods and tools for social systems change . American Journal of Community Psychology, 39, 405-409.
Perrin, B. ( 2002). How to - and how not to - evaluate innovation. Evaluation, 8, 13-28.
56.
Preskill, H., & Catsambas, T.T. (2006). Reframing evaluation through appreciative inquiry. Thousand Oaks, CA, London, New Delhi: Sage.
57.
Reason, P., & Bradbury, H. ( 1990). Handbook of action research. London : Sage.
58.
Regeer, B.J., & Bunders, J.F.G. (2007/2009). Knowledge cocreation: Interaction between science and society. A transdisciplinary approach to complex societal issues [translated from Dutch]. Den Haag: RMNO/COS.
59.
Regeer, B.J., Mager, S., Beekman, V., & Bunders, J.F.G.New approaches to the sustainable development of agriculture. TransForum - The case of a mode-2 intermediary. Manuscript submitted for publication 2009.
60.
Reynolds, M. ( 2007). Evaluation based on critical systems heuristics. In B. Williams & I. Imam (Eds.), Systems concepts in evaluation. An expert anthology (pp. 101-122). Point Reyes, CA: EdgePress/ American Evaluation Association.
61.
Rosas, S.R. ( 2005). Concept mapping as a technique for program theory development: An illustration using family support programs. American Journal of Evaluation, 26, 389-401.
62.
Rotmans, J., Kemp, R., & Asselt, M. v. (2001). More evolution than revolution. Transition management in public policy. Foresight, 3, 15-31.
63.
SenterNovem. (2007, June 05). Bsik: Investing in knowledge infrastructure. Retrieved October 15, 2008, from http://www.senternovem.nl/english/products_services/encouraging_innovation/bsik_investing_in_knowledge_infrastructure.asp
64.
Stake, R.E. ( 1967). The countenance of educational evaluation. Teachers College Record, 68, 523-540.
65.
Veldkamp, A., Altvorst, A. C. v., Eweg, R., Jacobsen, E., Kleef, A. v., Latesteijn, H. v., et al. (2009). Triggering transitions towards sustainable development of the Dutch agricultural sector: TransForum’s approach . Agronomy for Sustainable Development, 29, 87-96.
66.
Voß, J.-P., Bauknecht, D., & Kemp, R. ( 2006). Reflexive governance for sustainable development. Cheltenham UK: Edward Elgar.
67.
Voß, J.-P., Kemp, R., & Bauknecht, D. ( 2006). Reflexive governance: A view on an emerging path. In J.P. Voss, D. Bauknecht & R. Kemp (Eds.), Reflexive governance for sustainable development. Cheltenham, UK, Northampton, MA, USA: Edward Elgar.
68.
Vrom. (2001). Een wereld en een wil: Werken aan duurzaamheid: Nationaal milieubeleidsplan 4. Den Haag: Ministerie Volkshuisvesting, Ruimtelijke Ordening en Milieubeheer, Centrale Directie Communicatie.
69.
Walker, G., & Kubisch, A.C. ( 2008). Evaluating complex systems-building initiatives: A work in progress. American Journal of Evaluation, 29, 494-499.
70.
Wced. (1987). Our common future . UK: Oxford University Press.
71.
Williams, B., & Imam, I. (Eds.). (2007). Systems concepts in evaluation. An expert anthology. Point Reyes, CA: EdgePress/American Evaluation Association.
72.
Yampolskaya, S., Nesman, T.M., Hernandez, M., & Koch, D. ( 2004). Using concept mapping to develop a logic model and articulate a program theory: A case example. American Journal of Evaluation , 25, 191-207.
73.
Zweekhorst, M.B.M. ( 2004). Institutionalising an interactive approach to technological innovation. The case of the Grameen Krishi Foundation. Amsterdam: Vrije Universiteit.
