Abstract
Drawing on their extensive experience in large-scale project management, Patricia Burch and Carolyn Heinrich bring their respective skills in policy and economic analysis together to articulate an agenda for program and policy evaluation—an agenda designed to ensure that actionable evidence of the effectiveness of a program and its implementation is generated for all stakeholders. The examples they provide, while not minimizing the challenges of employing a mixed methods approach, will convince readers of the practicality and value of iteratively integrating both qualitative and quantitative approaches throughout the entire evaluation process, from initial design through data integration and analysis to dissemination of research findings.
The authors begin with a useful positioning chapter, making clear that their emphasis will be on “‘tightly integrated” mixed methods research and evaluation. Assuming readers will have a basic understanding of quantitative and qualitative methods, they see their work as building on basic “how-to” designs and associated mixed methods literature by “intensifying the focus on policy and program evaluation and further probing areas of qualitative and quantitative integration” (p. 6). Throughout their work they emphasize flexibility in design and the importance of being prepared to revise plans as a result of learning from the data and of changes in the situation being investigated in order to meet the study’s purpose. In Chapter 2, they point out that, at the same time, flexibility needs to be supported by having a well thought out study framework. While design types might be a guide for this, they are not a prescription. With their insistence on simultaneous (interactive and iterative) integration of qualitative and quantitative approaches from the very beginning of the research process (asking questions) through every stage until the end (reporting), they are however advancing a very particular model of fully integrated mixed methods research for their specific application to policy and evaluation studies. They are not discussing integration in mixed methods research more generally, although of course, much of what they have to say applies more generally. And yet, despite their emphasis on “fully integrated” mixed methods and references to working iteratively, in describing their example project they describe separate qualitative and quantitative components, storage, and teams, which seems somewhat contradictory.
Burch and Heinrich then turn, in Chapters 3 and 4, to the issues of project design, instrumentation, and administration as polemic turns to practice. They recommend using a logic model or conceptual model based on a theory of action as a basis for planning a study. In the instrumentation design and data collection stage of a project, qualitative methods are seen as useful for identifying limitations in quantitative data from sample anomalies and attrition, and in developing survey and assessment instruments. For the data analysis phase, they raise the issue of data integration, but rather than describe directly linking qualitative and quantitative data sets, they present qualitative methods as contributing primarily in assisting interpretation of quantitative data through feedback loops. In their final phase, they suggest drawing on both text and numbers in dissemination of results is of benefit for diverse stakeholders. In these chapters, as well as in the later project-based chapters (5, 6, and 7), Burch and Heinrich emphasize the importance of both sound administrative support for large projects and a strong collaborative ethos (suggesting practical strategies for enhancing this) between dispersed and/or differently skilled team members. They note the necessity for having at least one person who is over both sets of analyses for coordination purposes.
The four stages of integrated mixed methods are then grounded in a range of case examples—evaluation projects covering educational and social policy interventions in which the authors have had a key role—in the following chapters. Although the focus of these chapters tends to be as much on the nature of the projects as on the practical details of the methods used to investigate them, the discerning reader will find valuable practical clues therein. The final chapter reflects on three aspects of context in shaping research, and identifies broader lessons from the case studies for the field of mixed methods. From the authors’ experiences, fully integrated mixed methods contributes (1) to enhancing researcher credibility and stakeholder confidence, with a gradual reframing of questions from “what works?” to “under what conditions are gains most likely?” leading to more actionable results; (2) to improved transparency in how evidence is generated and validated, which, in turn, helps replication and transferability; and (3) to more effective use of research resources.
Burch and Heinrich have attempted to walk a difficult balance throughout the book between specifying action steps for integrating mixed methods research while also maintaining their emphasis on dynamic flexibility and providing guidance rather than prescription. This tension between guidance and prescription could be frustrating for the novice mixed methods researcher who is looking for more specific strategies to employ. Similarly, the lone PhD researcher might find the projects provided as examples less relevant to their immediate situation, given they are generally longitudinal projects that employ large and often dispersed research teams. The authors recognize these issues in their emphasis themselves, beginning their final, reflective chapter thus We recognize that some of our readers may come to this concluding chapter feeling as though we have not delivered a prescription of a series of steps for how to conduct mixed methods research. In fact, we intentionally steered clear of a stance that would have suggested there is a single best approach to undertaking mixed methods research. (p. 187)
This, therefore, is a book for the evaluation researcher-practitioner who, as noted earlier, has enough methodological training and understanding to be able to absorb and benefit from the primary message of the book, especially as it is illustrated through the projects described. Other readers could end up keen to implement an integrated mixed methods project, having been thoroughly convinced of the benefits, but still not sure how to do it and carry it through. Overall, however, the book makes a valuable contribution to the mixed methods literature, primarily through the extensive case studies that are provided for readers to study and learn from. Practitioners involved in large scale evaluation projects especially will find therein some useful guidelines to improve their practice.