Book Review: Mixed Methods Research and Culture-Specific Interventions: Program Design and Evaluation

Audience

This book, while designed very much like a textbook with “reflective questions and exercises” after each chapter, is not a how-to guide—at least not an easily accessible one. Its intended audience is extremely broad. In the authors’ own words,

The intended audience for the book spans graduate students to experienced researchers who seek guidance on how to apply MMR to develop culturally specific programs and conduct subsequent evaluations. Specific types of professionals who should be interested include psychologists, particularly school and community psychologists; social workers; educational interventionists; and program evaluators. (p. 16)

I was sometimes surprised at the many definitions and explanations of rudimentary data collection methods and terms, such as “focus group,” “key informant interview,” and so on. In contrast, other areas of the book are highly theoretical, such as the discussion on causal inference. It may be because the book is written by two authors that the styles and levels are markedly different at times. The occasional heavy use of theoretical jargon may turn off some readers, especially if they are new to evaluation. This indicates to me that the authors were attempting to appeal to seasoned theoreticians, while not wanting to exclude readers who are entirely new to social science research and program evaluation. This is both a strength, in that it may appeal to a wide audience, and a weakness, in that it may lose both types of reader at some point for being either over- or underchallenging.

The appeal to psychologists and educators is undeniable since many of the examples come from educational psychology, and more precisely, from a multiyear research project in Sri Lanka on the mental health and well-being of students in which both authors were involved. Bonnie Kaul Nastasi is a professor in the Department of Psychology at Tulane University, where she codirects a trauma specialization in the School Psychology PhD Program. John H. Hitchcock is an associate professor of instructional technology at Indiana University’s School of Education, where he directs the Center for Evaluation and Education Policy. The heavy leaning on psychological theory is also apparent in the authors’ strong grounding in the work of Bronfenbrenner (1989), whose ecological systems theory has been highly influential for understanding human behavior and development. The authors suggest that getting to know the culture in which the intervention and evaluation take place should involve a similar inquiry as a psychologist’s approach to understanding a patient’s or client’s mental state through studying that individual’s background and environment. The authors apply Bronfenbrenner’s system “as a structure for exploring the complexity and ever changing nature of the cultural and contextual factors that influence design, implementation, and evaluation of interventions” (p. 28).

Legitimization

The authors remind readers that randomized controlled trials (RCTs), while very highly valued in terms of their capacity to demonstrate the effects or outcomes of program interventions, require restrictive conditions that cannot always be applied to a program situation. Monitoring an intervention and a control group is usually cost prohibitive and often not feasible. This makes it hard for program evaluators to apply what is still considered by many to be the gold standard in research. The authors argue that a design that does not use a control group but which combines quantitative and qualitative research can rival RCT in accuracy. In addition, they emphasize the potential of MM designs—with their strong emphasis on the cultural implications of research, program design, and evaluation—to bridge the gap between the culture of the researcher or evaluator and that of the research subject or evaluand.

MM has been viewed critically by some due to the dominant role of quantitative data over qualitative data in scientific circles, at least of the past. In RCTs, for instance, quantifiable results are what count most, and there is an assumption that in controlled conditions, differences between the intervention and control groups will provide optimal proof of whether the intervention worked. Yet the authors show that RCTs are not foolproof. For example, intervention and control groups are never identical, thus introducing potential bias into the comparisons. Similarly, since participation in a trial is always voluntary, there is always potential for selection issues to arise in the practical implementation of a control group design. For these and other reasons, the authors make the case for qualitative data to be used for purposes of triangulation. Observations, interviews, focus groups, document reviews, and other qualitative techniques are excellent ways of complementing quantitative data, helping to establish internal and external validity, and strengthening potential causal interpretations. The book provides numerous examples of these syntheses.

There are also some evaluation questions that simply require qualitative data. For instance, qualitative data may be indicated for questions about human decision-making, as in policy evaluation studies where the evaluator attempts to understand the thinking patterns (or lack thereof) of policy makers. Other evaluation situations may require qualitative data to support analysis of an individual’s or group’s contextual framework. In one example, the authors make the case for MM in the context of a proposal for the closing of small schools in a district undergoing school consolidation:

Consolidation entails complex interplay between budgets, community politics and education theory/programming…data that most people would see as being qualitatively oriented can help illuminate program effects…we see quantitative and qualitative applications as the norm in much of program evaluation and social science. To us, methodological choices tend to focus not on whether to use mixed methods but rather how to use them in a planned, systematic manner in ways supported by models such as CMMPE [Comprehensive Mixed Methods Participatory Evaluation]. (p. 109)

Culture

The authors focus strongly on the application of evidence from scientific research to interventions, and in turn, the adaptation of evidence-based interventions to various cultural contexts. By studying the literature on translational science and program adaptation, they propose “adaptations…based on an inductively derived understanding of cultural narratives that reflect population beliefs, values, and norms relevant to the intervention—what we refer to as cultural construction,” and they believe that “evidence-based cultural grounding is best facilitated by an iterative, reflexive, and participatory research process that relies on qualitative methods” (p. 35). This may translate best into total cultural immersion in order to understand a culture from the inside out. It also requires that researchers are competent facilitators who can elicit relevant information from others in a nonthreatening way. Moreover, the new insights need to be applied to the theory that underlies the intervention approach. It may lead to questioning the approach altogether, but more likely it will lead to adaptations that make the intervention workable in a specific cultural context.

Those evaluators who have worked in unfamiliar cultural environments know the concept of the “cultural broker,” a person with insights into both the studied culture and the researcher’s culture, who becomes the liaison between the two and helps to conduct research and evaluation in a culturally appropriate way. The authors also work with cultural brokers, but they extend the notion of cultural adaptation by essentially conducting an ethnographic study in the same way that Bronfenbrenner recommends studying the ecological systems of a child (peers, family, neighborhood, economic, social, and cultural environments) before attempting an intervention. They call this process “cultural co-construction” and distinguish it from cultural competence in that the latter is seen merely as the result of the co-construction. This means, however, that to do justice to the authors’ recommendations, the first phase of a research project may take a very long time (in the authors’ own research in Sri Lanka, approximately 2 years). Many kinds of data will be collected during this phase, from document gathering and review to participant observation, surveys, interviews, and so on. The length of this study phase is, of course, prohibitive for many local programs and evaluators, whose budgets usually do not allow for such extensive cultural investigation before adapting the program and beginning the intervention. This may frustrate some readers. For more elaborate, long-term studies, however, the recommendations are quite useful and detailed.

The book does a good job in demonstrating the interplay of quantitative and qualitative data during the entire process of a research project in order to do justice to cultural contexts. The authors discuss the role of cultural difference more as an aspect of intercultural communication than as an aspect of power dynamics, even though they also highlight the fact that one purpose of cultural co-construction is to prevent imposing one’s own cultural norms on others. However, in an era of shifting world power dynamics, one may ask (and I am asking myself this question in my own work) whether our resources should not in the future be focused away from enormous investments in cross-cultural understanding for the purpose of interventions by a group from one culture in the setting of another culture. Instead, we might focus more heavily on growing research and evaluation capacity within cultures, so that, for instance, a Sri Lankan can do this valuable research with Sri Lankan schoolchildren without having to invite U.S. academics to do so. This is not to minimize the enormous effort and successful research of Nastasi and Hitchcock and their valuable contribution to the MM discussion but rather as something to think about in future discussions about cultural difference, competence, and co-construction. In the meantime, Nastasi and Hitchcock have provided a thought-provoking and interesting approach to research and evaluation in unfamiliar cultural contexts. Moreover, their book illustrates the usefulness of MM across a wide range of research and evaluation settings.