Negotiating Measurement

Measurement as Simplification

The sociologist Hubert Blalock (1979, 1982), writing about the relationship between theory and measurement, noted that measurement in social science, by its nature, is a process of simplification and ordering of the world. Simplification serves the purpose of advancing the generalizability of concepts across units, so that claims may be made about groups rather than individuals. For example, a questionnaire measuring weekly exercise and physical activity may use a single metric, such as minutes per week of vigorous exercise, to classify individuals along a single dimension to establish comparability, even though their actual patterns of exercise will vary from each other in multiple ways that the metric ignores. This simplification process, however helpful, will necessarily entail assumptions and potential biases. Blalock noted that social science theories are, explicitly or implicitly, accompanied by “auxiliary measurement theories” that underlie the use of whatever specific measures have been chosen. “In short,” he wrote, “we must become much more attentive to the need for stating explicit auxiliary measurement theories and for examining comparability of measurement, just as we must also be concerned about the generalizability of our substantive theories… If either process lags too far behind the other, we shall find ourselves stymied” (Blalock, 1982, p. 31).

Blalock’s view of measurement theory places importance on the ways that measures reflect their underlying constructs, and thus it has obvious parallels to construct validity, which many psychometricians consider the most fundamental form of validity in the measurement literature, incorporating aspects of both criterion and content validity (Kane, 2006). “Arguably in its most common current use, construct validity refers to the degree to which inferences can be made legitimately from the observed scores to the theoretical constructs about which these observations are supposed to contain information… The term ‘construct validity’ has therefore evolved to be shorthand for the expression ‘an articulated argument in support of the inferences made from scores.’” (Zumbo, 2009, pp. 68–69). In other words, construct validity is concerned with how well desired outcomes are represented by the scales, tests, survey items, observation forms, scoring rubrics, and other measures that evaluations employ.

The Specification Process

The measurement specification process generally begins with the broadly conceived target construct, which reflects, often in everyday language, the issue that the program is designed to address. Many of the stakeholders who come together to create social programs express their goals for the desired anticipated outcomes in broad terms. For example, they may want to reduce obesity, increase student interest and enrollment in science courses, or promote safe sexual practices and reduce incidence of sexually transmitted diseases. In stakeholders’ conversations about program goals, their conceptions might not be more precise than this. These general constructs need to be interpreted and operationalized before the program and the evaluation begin. This process presents a chain of specific decision points, each of which can present a wide array of options for the evaluation planning team.

Figure 1 presents a model and illustration of this process in four phases, mapping a chain of potential decision options for a program focused on nutrition education and healthy eating. The phases are marked by increasing specificity, and the expansion of options at each of the steps is readily apparent. The figure’s identification of those options is necessarily incomplete, as many more options can be suggested for each phase.

OPEN IN VIEWER

Figure 1.

Illustration of the possible progression from construct to instrument (some paths left incomplete).

The Choice of Variables

A variable is a specific characteristic of a unit—person, family, and so on—that has differing values within a population of those units. The variables selected for an evaluation must be more precisely specified and measurable than the constructs from which they are drawn. One aspect of rigor in this step of the translation process will be the presence of a clear and logical link between the construct and the variable. Ideally there will be general agreement among the evaluation team and the primary stakeholders that the variable is a satisfactory representation of the guiding construct.

In small-scale, low-resource evaluations, one common scenario in which this link may be open to challenge can occur when a program aimed at producing behavioral change does not have the resources to follow participants for the long term, and must end data collection concurrently with the conclusion of the program. In such cases, the staff may feel constrained to measure a short-term outcome identified in a program’s logic model and claim it as a proxy for a long-term outcome further down the outcome chain (see, e.g., Funnell & Rogers, 2011; Weiss, 2000). For example, the behavioral construct of greatest interest, healthy eating, may be replaced by a presumed psychological mediator, dietary intentions, as the evaluation’s primary outcome variable. This substitution may or may not be justifiable in terms of the overall goal of the evaluation, and there may or may not be reasonable alternatives given the available resources. But program staff, evaluators, and other stakeholders need to be aware of the ways that the substitution weakens the ability to draw conclusions about the program. In this example, the program might produce impressive increases in behavioral intentions, but the original target construct—eating behavior—will remain unexamined.

The Choice of Approach

The options for a measurement approach will vary depending on the program context and the nature of the variable. For variables that reflect individual behaviors, the available approaches might include self-report by the participant, direct observation by the evaluation team, reports by others (e.g., parents or teachers on behalf of children), and physiological or biochemical measures, among others. For variables that reflect participants’ skills and aptitudes, the options might include direct tests, self-assessment by the individual, assessments by others, or existing indicators (e.g., standardized test scores). For variables that reflect internal psychological states (e.g., motivation, interest in a topic, satisfaction with a service), there may not be viable alternatives to self-report.

To measure variables that reflect presumably objective facts or status, such as body mass index, weight, or grade in math class, there might seem to be little choice of appropriate measures. But even in these cases, decisions must be made that will affect the reliability and accuracy of the results. In the case of weight, for example, evaluation planners will need to decide how carefully to control the time of day of the measurements, and whether the weight should be self-reported or recorded by a member of the evaluation team. In the case of grades, they may decide to ask the student or may seek more direct access to grading information.

A convergent validity study by Sussman and Stacy (1994) provides a valuable illustration of the divergences that can exist across different measurement strategies for the same conceptual variable. They compared alternative strategies to derive estimates of two institutional-level variables: school-level student cigarette use and school-level student alcohol use. For cigarettes, the five strategies were students’ self-reports of their own use, students’ estimates of school prevalence, staff estimates of school prevalence, structured direct observation of student behavior at locations near the school, and refuse evidence observed at selected sites such as parking lots and sports fields. For alcohol, they used the same strategies except direct observation. (The strategies of student self-reports, student prevalence estimates, and staff prevalence estimates entailed interviews with small samples of students and staff.) The researchers found the correlations between these measures to be strikingly divergent, ranging between .01 and .54 for cigarette use and between .09 and .63 for alcohol use. One is impressed not only by the lack of correlation in some of these pairings but also by the moderate limits on the high end: none of the correlations exceeded .63, indicating that the two most strongly related strategies (student and staff estimates of alcohol use prevalence) shared only 40% of their variance. Considering that these strategies were all intended to assess the same variables, this example illustrates the dramatic implications that the choice of measurement strategy can have on the apparent levels of an evaluation’s outcome variables.

The Selection or Development of Instruments

Finally, when the variable and measurement approach have been decided, the evaluation planning team must decide the structure, format, and (if appropriate) wording of the instrumentation. Recommendations about instrument design are beyond the scope of this article, but a few examples can illustrate the decisions to be considered:

Ideal and Adequate Rigor: Considerations of Cost for Alternative Measurement Strategies

In cases of well-funded, high-stakes evaluation studies, there is typically consensus that the evaluation plan needs to be as rigorous as possible in order to produce strong, credible evidence and a high level of confidence for the study’s interpretations, inferences, and conclusions. This may mean, for example, very close monitoring of the program’s operation, careful attention to sample selection and retention, multiple measurement points that can track the linkages specified in the program theory, and selection of measures that are the most valid and comprehensive representations of the program theory’s constructs.

However, rigor will involve costs of various kinds, and in smaller local evaluation contexts it may be productive to ask what may be an adequate level of rigor for measurement or other aspects of the evaluation study. Chatterji (2007) argued along similar lines in discussing the strength of evidence afforded by different kinds of research designs for purposes of inferring causality. She maintained that the quality of evidence should not be judged simplistically as either strong or weak, but rather in terms of “grades of evidence” that vary along certain dimensions.

A concern for identifying criteria for acceptable levels of rigor, as well as for the strength and credibility of evidence, can result in useful standards or benchmarks as the planning proceeds. These determinations will be situated within the local context of what the program is trying to do. The primary stakeholders, and the forms of information that they need and expect, must be clarified. Program stakeholders can be drawn into preliminary discussions in which they clarify their own intended uses for the information and the strength of evidence they require—questions they may not at first be readily able to answer. The importance of engaging primary stakeholders is widely acknowledged, but matters of evaluation methodology are not always considered in these discussions.

The Rigor-Feasibility Dynamic

Braverman and Arnold (2008) enumerated several categories of costs associated with increases in evaluation rigor—money and other material resources, time, and participant burden. Time can constitute a cost in multiple respects, including preparation and planning time (which may translate to money), the time available with participants during data collection, and the time horizon available for tracking change in the target groups. Participant burden can take several forms. Filling out questionnaires takes time and effort, and in some evaluation studies the questionnaires can become quite long. There may be multiple measures, as well as items that ask for revealing or sensitive information. The data collection may need to be completed at specific points in time, with which participants need to comply. If the program’s participants decide that they do not wish to tolerate these infringements, they will decline, and the evaluation will suffer accordingly.

In sum, these increased costs of rigorous methodology may lead to the judgments that some measurement strategy options are more feasible for the evaluation than others. These considerations may persuade program staff, target audiences, and others involved in the planning to reject the more demanding approaches in favor of the simpler ones.

Here is a thought experiment: If we can conceive of both the rigor and feasibility concepts as single, linear dimensions, one may imagine the relationship between them as depicted in Figure 2. This is a highly simplified representation, because there is no reason to assume that either of these concepts must be either unidimensional or linear. Nevertheless, the figure is offered to illustrate a basic conceptual point. Rigor in evaluation is typically associated with increased demand on people and/or resources. To the extent that the feasibility of any approach implies certain resource requirements and acceptance by affected audiences, feasibility and rigor are probably negatively related, more often than not, so that high levels of one will be associated with low levels of the other. As Figure 2 illustrates, an overemphasis on one of these dimensions may result in the other being so low that the evaluation becomes unacceptable or unworkable. (The existence of a discrete point signifying the change from acceptability to unacceptability is another oversimplification for purposes of illustration.) Thus, if rigor is too low, the level of confidence will be so compromised as to render the evaluation not worth doing. On the other hand, if feasibility is too low, there will be uncertainty about whether the evaluation can be completed.

OPEN IN VIEWER

Figure 2.

Hypothesized (and simplified) conceptual representation of the relationship between evaluation rigor and implementation feasibility.

This leaves a range of options between the two thresholds, in which both rigor and feasibility may be relatively higher or lower, in opposite relation to each other, but nevertheless acceptable. This intermediate range, however the concepts may be operationalized in terms of individual evaluation decision options, is the territory for negotiation and for attentive, thoughtful decision making—a “sweet spot” for planning, so to speak. In my experience, this deliberative decision-making process about methodology does not take place as often as it should during evaluation planning—at least not in a way that is explicit, conscious, and inclusive of diverse stakeholder influence.

A final consideration is how the thresholds for acceptability get set. In the local program context, this also can derive from discussion and negotiation, guided by the expertise and methodological sophistication of the evaluator in advancing arguments to help parties understand the bases for credibility of evidence. Of course, as the scale, complexity, stakes, visibility, and resources for an evaluation increase, questions about acceptable rigor quickly become considerably more complicated, and, perhaps, less negotiable.

When feasibility issues are taken into consideration, it may be determined that a moderate level of rigor—that is, something less than ideal—may be acceptable for the primary stakeholders and planned uses of the evaluation evidence. In the next section, I turn to the interpersonal aspects of the evaluator’s role, particularly the processes of guiding discussion and building consensus. In some cases, it may be appropriate for the evaluator to make the case that the rigor, and the ensuing evidence base, should be stronger.

Establishing the Standards for Evidence

I have participated in numerous evaluation planning meetings in which the program staff’s primary questions center on their concerns about the ease of implementing data collection. They wish to know how long various instruments may take to administer, whether the data collection might generate boredom or frustration, how much time the data collection will take away from program delivery, and so on. These are legitimate local concerns, which can justifiably be the basis for accepting some measurement options and rejecting others. However, the concerns can often obscure the longer term aims of the evaluation to produce strong and credible evidence that can support useful conclusions about the level of program success.

Thus, one of the evaluator’s tasks is to make sure that the challenges inherent in conducting the evaluation are balanced against the evaluation’s potential long-term utility and value. The planning group and the program stakeholders who are being consulted must take explicit account of the broader utilization context. The evaluator will also be able to remind the group about the parties not in the room who will nevertheless be important audiences for the study, which might include—depending on the individual case—program funders, target audience members, state- or county-level legislators, clearinghouse committees, journal editors, and so on. The evidence standards for these audiences will need to be identified or estimated, ideally through direct consultation but, at a minimum, through knowledgeable appraisal.

Patton (2008) has described procedures and exercises in which the evaluator can lead stakeholders through a futuring process, to anticipate potential findings and what the utilization response will be in each case. Issues about the credibility and strength of evidence need to be part of these discussions, the result of which will be guidance for determining how rigorous the evaluation needs to be. Thus, taking into account all of the reasons for which the evaluation is being done, the group will determine the criteria for the strength of the evidence base resulting from the evaluation, in order to support the conclusions they may wish to draw with a desired level of confidence.

This process suggests that the adequacy of evidence—in terms of its relevance, credibility, and strength (Schwandt, 2009)—lies in the eyes of the beholders, who in this case are the eventual audiences for the evaluation. This is consistent with the view that evaluation is, indeed, a process of argument. Consider the example of a smoking cessation program, for which the primary outcome of interest is postprogram smoking levels. Given an expected social desirability bias that may lead to underreporting of smoking, the planning team must decide whether participant self-report, by itself, will be a sufficiently rigorous approach to address the questions and concerns of the evaluation’s primary audiences. If those audiences consist of community members and organizations, the answer may be yes. However, if the primary audiences also include public health researchers, editors, or critics who have already expressed an interest in defunding the program, the answer may be no. In the latter case, the evaluation planning team will need to consider supplementing the self-report with biochemical validations such as saliva tests for cotinine or expired air samples for carbon monoxide (Hukkanen, Jacob, & Benowitz, 2005). The general point is that evidence cannot be judged as sufficiently or insufficiently credible in the absence of the context in which it is being used, the audiences who review it, and the purposes to which it is being put.

Weighing Alternatives

Based on the evaluator’s training, acquired knowledge, and experience, he or she will probably be in the best position to educate and advise about the strengths and shortcomings of various kinds of evidence. Consider the example presented earlier, in which the long-term outcome variable of primary interest is healthy eating behavior (e.g., 6 months postprogram), but a potential short-term proxy under consideration is dietary intentions, measured at the program’s immediate conclusion. The difference in costs between these alternatives will be considerable and will favor the shorter term option. A determination must be made with respect to the net benefit of the longer term (behavioral) option, whether it is worth those added costs, and how much credibility would be lost if only the intentions variable were used. In this case, the evaluator might need to describe the relationship that has been found previously between intentions and behaviors in various domains. In general, the relationship is positive and significant, but certainly limited (e.g., Fishbein, 2008; Webb & Sheeran, 2006). More specifically, the findings have been mixed for the relationship between intentions and behaviors in the specific area of eating behavior (e.g., Lohse, Wall, & Gromis, 2011; Shaikh, Yaroch, Nebeling, Yeh, & Resnicow, 2008; Zoellner, Estabrooks, Davy, Chen, & You, 2012). Depending on the audiences for the evaluation and the discussions about this research base, the use of intentions as an outcome variable may or may not be sufficiently convincing to enable the evaluation study to achieve its intended purposes.

The evaluator can also lead the planning team through the process of identifying and weighing uncertainties regarding the consequences of alternative decision options. For example, the benefits for data analysis inherent in using a more comprehensive and detailed set of measures may be offset by issues of sample representativeness if substantial numbers of participants refuse to participate in the evaluation. However, the effects of each of the alternatives on participation rates may not be easy to predict beforehand. If a high participant-demand measurement strategy results in lowering participation by, say, 20–30% compared to less demanding options—for example, smokers declining to participate in a study that includes saliva tests compared to only self-report—the study’s rigor, as a whole, will not be enhanced by the ostensibly more rigorous option. The variety of consequences associated with each option cannot be predicted with precision, thus introducing degrees of uncertainty into the mix. Examining the experiences of prior evaluation studies will be useful in reaching reasoned predictions, and the evaluator is probably in the best position to identify these precedents and assess their relevance to the context at hand.

Negotiating the Selection of Measures

Bringing the planning team to a set of final decisions about which measures to use in the evaluation will incorporate judgments about the most critical variables for capturing the program theory’s central constructs, the most appropriate measurement strategies, the required strength of evidence, and the anticipated feasibility of the approach. In Blalock’s (1982) terminology, these decisions will reflect the auxiliary measurement theory that supports the primary program theory driving the evaluation. If there is an initial consensus about the measures, the team will be fortunate. If not, the group will need to resolve its differences in some way that is reasonably acceptable to all. The evaluator will play a key role in this consensus-building process, but may or may not be the individual who leads it.

Consider again the scenario mentioned earlier in this section, in which the program staff’s immediate focus is the ease of implementing the data collection procedures. It may be that the evaluator believes that more rigorous measurement is needed, compared to what the staff and/or other members of the planning team are advocating. The evaluator can remind the group that if they are fortunate enough to wind up with an evaluation that shows program impact, the demonstrable rigor of the study’s plan and its implementation will be of tremendous political benefit for the program. It may be that the feasibility concerns still carry the day. In that case, the decision will at least have been well considered, and the long-term consequences, in terms of relative reductions in the validity or persuasiveness of the study’s conclusions, will presumably have been anticipated.

Explaining the Measurement Requests to Program Participants

When describing the measurement requirements of impact evaluations to program participants or, in the case of children, to their parents, it may be helpful to stress to them the value of the information that is being sought about the program and/or the social condition being addressed. To the extent that the participants whom it serves value the program, an argument that the program needs the opportunity to convincingly demonstrate its effectiveness may be persuasive.

A colleague of mine, Dr. Katherine Gunter, studies bone health in children and has developed physical activity programs that emphasize high-impact exercises such as jumping and running. These exercises, when performed with sufficient frequency, produce long-lasting benefits in bone density and mineral content in the hip, spine, and other skeletal areas (Gunter & Kasianchuk, 2011; Gunter et al., 2008). In evaluations of these programs, the measurement of the primary outcome variable of bone mass entails the use of dual-energy X-ray absorptiometry (DXA), a scan that requires an extremely low dose of radiation, approximately the amount that an individual would receive from background environmental radiation exposure in a normal day’s activity. In the course of program evaluations over several years, Dr. Gunter has spoken to many parents of the children who participate in the programs. She explains the procedure, in accord with her university’s Institutional Review Board (IRB) protocols, and the parents make an informed decision about whether to let their children undergo the DXA scans. The great majority of parents and children do provide their consent, but, in many cases, only after considerable discussion to address the parents’ concerns and questions.

This example illustrates a case in which the presentation of the evaluation’s measurement procedures to prospective participants requires significant knowledge and expertise. In the case of DXA scans, there are no alternative procedures that can address the primary outcome variable of bone mass. The evaluator, researcher, or other individuals charged with this task must communicate with honesty and objectivity so that participants can reach an informed decision.

In some program settings, the ethical issues involved in discussions with participants will be particularly challenging. Mitchell, Nakamanya, Kamali, and Whitworth (2002) report on a process evaluation they conducted to examine the local acceptability of a randomized trial, implemented in several dozen villages in rural Uganda, of an intervention aimed at reducing HIV/AIDS prevalence through education and the improved management of other sexually transmitted diseases. The outcome measures for the randomized controlled trial included, in addition to psychometric and behavioral variables assessed through questionnaires, the collection of blood samples to assess HIV prevalence. Mitchell et al. describe the intensive efforts that were needed to dispel rumors and misinformation about the serological survey. One widespread rumor was that the blood was being collected, not for its stated purpose, but for resale abroad. In addition, they note, “There was evidence that enthusiastic supporters of the survey within the community sometimes used the promise of an impending cure to cajole neighbours and peers into providing blood” (p. 1085). With continuing communication efforts as the study progressed, the rumors and misstatements subsided over time. Nevertheless, faced with these substantial problems, the question arises as to whether the blood collection was necessary or advisable for ensuring the validity of the study’s findings. The authors believe that it was: “One might argue that alternative methods of measuring HIV (such as urine or saliva) may have engendered less controversy. The drawback of these alternative methods is that they do not allow for so many tests for other STD’s” (which constituted an important component of their program theory). “Furthermore, earlier pilot studies conducted in the study population found neither urine nor saliva to be any more acceptable than blood” (p. 1088). This example dramatically illustrates the difficulties that can exist in explaining to participants the need for a particular choice of measures.

The utilization of a measurement protocol that entails high participant burden is frequently addressed by offering compensation to participants in the form of cash or other incentives. Compensation can increase participation in evaluation studies and communicate to participants that their time and good will are valued. Federal research regulations provide latitude for IRBs to set their own standards about compensation, but they direct IRBs to insure against “undue inducement” (Dunn & Gordon, 2005; Permuth-Wey & Borenstein, 2009; Wertheimer & Miller, 2008), a term that is generally interpreted to mean a level of compensation so disproportionate that prospective participants might be induced to act against their own best interests. Ultimately, for any individual evaluation study, the decision about whether to compensate participants, and in what amounts, will need to be resolved in consultation with the IRB that is overseeing the project.

Defense of the Measurement Decisions

A final function that may fall to the evaluator is defending the decisions that were made during the evaluation planning. When the study has been completed, there may be skeptics, critics, or simply interested stakeholders who ask why a more rigorous approach was not taken in some area of the study, such as the choice of variable, the timing of measurement, or the technical characteristics of instruments. If the choice can be defended in terms of the balance it provides across critical planning considerations, it can make an important difference in the credibility afforded to the study and the extent to which the study is eventually used. Criticisms regarding disputable levels of rigor can be countered by identifying the significant feasibility challenges that existed. A further strength in this debate would be the opportunity (if true) to report that the measurement choices reflect the input gathered from multiple stakeholders.