Book Review: Cost-Inclusive Evaluation: Planning It,Doing It,Using It

A Strong Reference Manual

By carefully reviewing this text, a reader with a solid foundation in policy evaluation could feasibly conduct a cost-analysis with straightforward parameters, perhaps with limited oversight from a more experienced cost researcher. As mentioned above and indicated by the book's subtitle, Cost-Inclusive Evaluation is focused on facilitating the use of the techniques it describes. In some places, the text does an excellent job walking the reader through action steps, however, as is the plight of so many methods texts, in other places the book presents a clear charge to do better and to consider complex situations carefully without giving the reader a clear set of action items. For example, in Chapter one, the authors make the astute point that cost-inclusive evaluation is best conducted using a mixed methods approach. Further information on best practices for mixed methods work, including a list of resources, would help readers put action to intent. Similarly, in Chapter 2, which focuses on types of costs and outcomes, the authors suggest the consideration of statistical significance without providing enough detail to allow for researchers to meaningfully plan significance testing.

Although there are topics for which more specifics would be helpful, this book stands in contrast with similar resources in its ability to clearly state foundational concepts while striking a balance between breadth and depth. This effort is well supported by the strong and frequent use of examples. For instance, the authors define direct and indirect costs with the aid of a hypothetical example in which three community center programs share a set of indirect costs (pp. 28–29). At the same time, this book tends to rely heavily on business jargon, such as “profit” and “competitor,” that many in the social services would reasonably find uncomfortable as these terms rightfully are not precisely applicable to their work. It is also important not to assume tools from private enterprise necessarily work perfectly for social service application, rather new tools and vocabulary may need to be adapted that are reflective of the nuance and sensitivity of human services.

Two Approaches

This book covers two primary approaches to cost analysis; a financial analysis approach in which budgetary and accounting data are gathered, organized, and analyzed to estimate costs and an economics-based approach where information on the quality and quantity of program inputs are collected and then matched with standardized market prices. While the accounting approach is dominant, both approaches are covered. The choice to present both is practical and wise given that there are clear applications for both dependent on the evaluation's goals and audience. However, the decision to discuss these approaches in separate chapters means the authors miss the opportunity to contrast the two methods concisely. This contrast is important, as the goals and audiences differ across the two approaches. Financial analysis is well suited to a relatively narrow audience, for example a set of involved agencies seeking financial evidence on a given program. An economic approach is helpful when the evaluation seeks to be generalizable to external audiences or when the goal is to document the market value of the inputs that were leveraged to generate observed effects. It is laudable that the authors present a broad range of methodologies to suit a range of potential applications, however, greater attention could have been paid to the goals and audiences appropriate for each so that evaluators could select the best fit for their needs.

Data Sources

The authors present an array of potential data sources, including budgets, activity logs, interviews, and focus groups. By introducing the sources in passing in the sections where they are most relevant, the book fits its role as a reference guide. However, the text would have been strengthened by a chapter devoted to data sources for cost-inclusive evaluations, which would have allowed a more comprehensive review of methods. Currently, some important sources, such as program observations, cost surveys, and document analysis, are missed. Potential cost analysts would do well to understand the menu of data options available to them as well as general tips for designing both individual measurement tools. Additional discussion on how to design a data collection effort that allows for the triangulation of multiple data points on important cost drivers would also be warranted.

In keeping with an accounting approach to cost analysis, Persaud and Yates make frequent and liberal mention of budgetary and accounting data as a primary data source for prices in cost analysis. On one hand, this is pragmatic, as budgetary data are widely available and inexpensive sources for cost evaluation purposes. On the other hand, budgets offer a host of pitfalls of which the cost evaluator should be aware. The authors include a brief and admirable section on the challenges of budgets in Chapter 3. For instance, the book points out that budgets capture incomplete information at best and that the information may not be granular enough or may even be inaccurate (pp. 62–63). To these concerns, I would also add that budgets merely represent a plan rather than what actually happened and that they pick up too much site-specific nuance to be fully reliable for producing generalizable cost estimates (see Levin et al., 2018).

To fully cover the alternatives to budgetary data, the authors could have taken more time to explicitly discuss the application of standardized market prices. Although this approach is briefly discussed throughout (see, for example, pages 202–203 in Chapter 8), it is never fully developed. I am a strong advocate for a market price approach, that is, for data on resource use to be collected separately from data on prices, which should come from standardized local or national sources rather than a program's financial records. Using standardized market prices promotes generalizable cost estimates, in contrast to the documentarian effort of recording what was paid from budgetary or accounting data, which can pick up site-level noise and nuance that does not always translate cleanly to a broader audience.

Novel Approachs

While I wished for more discussion of some topics, I found the presentation of novel approaches to cost analysis valuable. In Chapter 8, “Breaking Down Cost by Activity for Better Cost-Inclusive Evaluations,” I was intrigued by the resource-activity matrix presented as a part of Yates’ resources-activities-processes-outcomes analysis (RAPOA) model. Traditionally, information on resources and activities has been presented in list form. In contrast, the resource-activity matrix places a program's activities along column headers and the program's resources along row headers, so an analyst or program manager can categorize collected data within the resulting grid. For example, in a prenatal support program, a social worker's time (resource) is allocated to client meetings, group sessions, and running a monthly provider training meeting (activities). The authors could do more to underscore the benefit of this way of organizing information in a matrix, that is the cost analyst is compelled to think about each cell in the grid. By directing the focus to where resources both are and are not allocated, it seems that there is less risk of missing important resource allocations. Similar to how evidence gap maps are used in evidence synthesis (Polanin et al., 2022; White et al., 2020), these matrices also have the potential to indicate gaps in resource allocation that may be more opaque in other formats.

Additionally, I find the full approach to “cost-inclusive” evaluation refreshing. The authors write “in the past, cost-inclusive evaluation tended to use economic appraisal methods (Persaud, 2021). However, the object of this book is to illustrate to program evaluators and program administrators that there are many other ways to analyze accounting data that can really enlighten decision making.” (p. 149). I appreciate how the authors suggest costs be thoroughly integrated into an evaluation, rather than a tacked-on component at the end. While this alone is not novel, the level of attention to the integration of cost analysis into an evaluation and into subsequent planning and decision-making paid in this book is refreshing.

Equity and Ethics

One under-emphasized element from the otherwise strong running narrative around the rationale for cost analysis is the importance of information on resource use and cost results for questions about equity and reallocation. This omission is notable because cost analysis can be a powerful tool for understanding how and to which groups’ resources are allocated and identifying potential equity gaps. Relatedly, the role of cost analysis as an equity tool is an important counterpoint to a common fear that cost analysis simplifies complex human interventions into cold economic data. In fact, the opposite is true: Cost can be complementary to evaluations that describe the human contributions to an intervention and an asset to studies that seek to improve equity.

However, in the section titled “Ethics,” the role of cost analysis in the pursuit of a better world does shine through. The authors write:

Cost-inclusive evaluators have a moral obligation to do their work well in compliance with high ethical standards to ensure that scarce resources are optimized for social good … Yet the consequences of doing our job well can involve challenging the status quo and political dynamics that could lead to retribution. (p. 65)

In this section, the authors point out that conducting cost analysis ethically (that is, without giving in to pressure to influence results one way or another, yielding to pressures to omit facts, or relying too heavily on incomplete data) can be quite challenging. The authors charge the would-be cost analyst to maintain their ethics and produce work that is rigorous and accurate to the best possible degree. While this is admirable, this important topic warrants further discussion and guidance. The authors mention “high ethical standards,” yet in the field of cost analysis in social sciences, these standards remain somewhat unarticulated. As other arenas of evaluation grapple with what it means to be rigorous, in an age where evidence is questioned more than ever before and where science has become an open target, the cost field must consider concrete and applicable guidance to help new and experienced scholars navigate the shifting currents in the discussions of standards.