A Framework for Understanding Modifications to Measures for Diverse Populations

Reasons for Considering Modifications to a Measure

There are a number of reasons why investigators might consider modifying a measure. In health disparities research, the most common reason is that the population group(s) being studied differs substantially from the one in which the original measure was developed. Essentially, the motivation for modifying measures is the concern that racial and ethnic or generational differences might adversely affect the meaning, reliability, or validity of the original measure (information on these basic measurement concepts is available elsewhere; for example, McDowell & Newell, 2006; Nunnally & Bernstein, 1994). Some key reasons why measures developed in one group may not be suitable include: (a) a concept or dimension may be missing from a measure, (b) the meaning of the concepts or items may differ, (c) items/phrases may not be interpreted as intended, (d) the process of responding to the questions is complex or difficult, and (e) the study context or mode of administration may differ from the original. Any of these reasons could result in findings that measures do not meet minimal psychometric criteria in these new groups or that the measures do not demonstrate invariance across groups (Gregorich, 2006; Stewart & Nápoles-Springer, 2000, 2003).

Basis for Modifications: Source of Information Used to Decide How to Modify

Once a decision is made to modify an existing measure, investigators need information on which to base those modifications. Such information is often available in the same studies that identified the need for modifications. Three sources of information can be used as a basis for modifications: (a) qualitative research on a concept or measure, (b) literature reviews of the adequacy of a measure, and (c) investigator experience or judgment. We review these three information sources and provide examples of how the information source was used to make modifications.

Types of Modifications

The types of modifications that can be made range from simple format changes such as improving the contrast or increasing the font size to extensive changes such as adding new subscales or changing item wording. To facilitate thinking about the various types of modifications, we have classified them into three broad categories: content, context, and format. For each of these, in Table 1, we define the various specific modifications and provide examples.

OPEN IN VIEWER

Table 1.

Organizing Framework of Types of Possible Modifications with Definitions and Examples

Category/type of modification	Definition	Examples
Content modifications
Dimensions
Add dimension	Add a set of items hypothesized to measure a new dimension (subscale)	Two dimensions were added to an existing social support measure: “language support” and “financial support” (Wong et al., 2007) in response to qualitative research that these were important to older Chinese and Korean immigrants (Wong et al., 2005). Items were written based on the same qualitative studies.
		A new “discrimination/stereotyping” dimension was added to an existing patient satisfaction measure in response to qualitative studies indicating its importance to older African Americans (Fongwa et al., 2006).
		An “integration” dimension (providing various types of support to others in their social environment) was added to a social support measure to make it more appropriate for older adults (Krause & Markides, 1990).
Drop dimension	Omit all items from one dimension (subscale)	The SF-36 physical functioning scale was omitted in a study of spinal cord injury patients because patients considered the items demeaning, given that walking and climbing stairs were prominent items (Mattson-Prince, 1997).
Item stems
Drop items	Drop items from an existing scale	Four items from the Cancer Behavior Inventory referring to treatment were dropped in a study of Latina breast cancer survivors who had completed treatment (Nápoles-Springer et al., 2011).
		A few items from the McNeilly Perceived Racism Scale pertaining to racism in academic settings were dropped as not applicable to older African American patients (Moody-Ayers, Stewart, Covinsky, & Inouye, 2005).
		Several items from the CES-D were dropped as being inappropriate for older Asian groups (reported by Mui et al., 2001, p. 141).
		Eight items were omitted from the CES-D in a study of older American Indians because they were determined to be inappropriate (Chapleski, Lamphere, Kaczynski, Lichtenberg, & Dwyer, 1997).
Add items	Add items to an existing scale to fill in a missing element	The Insulin Resistance Atherosclerosis Study food frequency questionnaire was modified to add foods typically eaten by Chinese persons for a multiethnic study of atherosclerosis (Bild et al., 2002).
		The CHAMPS Physical Activity Questionnaire was modified for a church-based intervention for African Americans. Based on focus groups and pilot testing, “praise dancing” was added because it was common and contributed a substantial amount of energy expenditure (Resnicow et al., 2003).
Modify items	Substitute new term, modify wording without changing meaning, split complex item into two	Item stems in the Family Peer Relationship Questionnaire were modified to be appropriate for Arab Muslim women, based on information from five cultural informants (Arab immigrant women from the Middle East) with experience providing health and social services to Arab Muslim families (Aroian et al., 2008).
		Hays and colleagues (2002) suggest modifying some SF-36 items for studies of patients with disabilities by changing “does your health now limit you in these activities” to “does your health or a disability now limit you in these activities.”
		Several items in a patient satisfaction survey were reworded or split into two items based on problems identified with original items in qualitative research with older African Americans (Fongwa et al., 2006).
		To adapt CES-D items for verbal administration to low-literate respondents, based on cognitive pretesting, statements were modified to be questions and a time frame was added to each item (from only being in the initial instructions). For example, “I felt depressed” was modified to “During the past week, how many days did you feel depressed?” (Rosal et al., 2003).
Response options
Replace response option set	Replace an entire set of response options with another.	To adapt a measure for older Thai adults, Thiamwong and colleagues (Thiamwong et al., 2009) replaced a bidirectional true/false response scale (1 = definitely true to 5 = definitely false) with a unidirectional response scale (1 = not at all true to 5 = definitely true).
		To improve variability of two role functioning scales from the SF-36 in the Veterans Health Study, the yes/no response options were replaced with a 5-level ordinal scale (ranging from yes, all of the time to no, none of the time; Kazis et al., 1999).
Modify response options	Split response options into two levels to reduce the number of options presented at any one time	For a study of low-literate individuals, a 4-point Likert-type scale response option ranging from 4 = very important to 1 = not important was modified to be administered in a nested format. A 2-point scale was first administered (important, not important); for those responding important, three options were offered to indicate the strength of the choice (very important, important, somewhat important; Rosal et al., 2003).
	Add verbal labels to all response options rather than having only endpoints labeled	For a study of low-literate individuals, response options asking how confident they are they can do something with endpoints anchored (1 = not at all, 4 = completely) were modified to provide a verbal anchor for each level (1 = not confident, 2 = a little confident, 3 = somewhat confident, 4 = completely confident; Rosal et al., 2003).
	Drop a response option, for example, to reduce confusion	The SF-36 Version 1 had six response options for the mental health and vitality items (1 = all of the time, 2 = most of the time, 3 = a good bit of the time, 4 = some of the time, 5 = a little of the time, 6 = none of the time). In Version 2, “a good bit of the time” was dropped because respondents consistently had difficulty understanding its meaning, leaving five options (McDowell & Newell, 2006; Ware, 2000).
	Remove numbering from response scales	In adapting the Spanish Brief Religious Coping scale, the original items were asked on a 4-point response scale: 0 = not at all to 3 = a great deal. Because of difficulty with Likert-type scales, response options were modified to omit the numbers (leaving the phrasing: not all, somewhat, quite a bit, a great deal; Martinez & Sousa, 2011).
Study-specific context modifications
Modify recall period	Different time frame applies	The CHAMPS Physical Activity Questionnaire was modified for an intervention for African Americans; the 1-month recall period for activities was changed to 2 weeks, based on formative research indicating that respondents could more accurately recall activity during a shorter period (Resnicow et al., 2003).
Change referent	Change referent in instructions or item stems	To adapt the Everyday Discrimination Scale for use in a health care setting, instructions asking whether they “ever had any of the following things happen . . .” were modified to “when getting health care, have you ever had. . .” (Bird & Bogart, 2001).
		An item in the Experiences of Racial Discrimination Scale referring to discrimination “while getting medical care” was modified to refer to getting care in a Veteran’s Administration (VA) health care setting (Hausmann, Kressin, Hanusa, & Ibrahim, 2010).
Format or presentation modifications
Modify format for responding	Modify way in which respondents answer questions, for example, check box instead of circle number	Instead of asking respondents to circle the number in front of each response, for each response choice, they were asked to check a box (Walter et al., 2008).
Modify appearance of page	Make skip patterns more obvious, reduce crowding, increase font, improve contrast	Because reading performance declines for older adults and persons with impaired vision when questionnaires use small fonts, questionnaires were designed using 14-point font (Vanderplas & Vanderplas, 1981).
Add visual aids for response options	Use a card with the response options to aid remembering the options	An interviewer-administered survey in a study of lower-income, minority adults used a large card containing response options for each set of questions. Interviewers showed the appropriate card to respondents which aided of the options and allowed respondents to point to a response (Santoyo-Olsson et al., 2011).
	Add visual aid to illustrate increasing intensity	A Visual Analogue Pain Scale for Spanish-speaking individuals was modified from a 10-cm line with end points labeled (Dixon & Bird, 1981) to a histogram; bars increased in size from 0 to 10 visually depicted increasing pain intensity and a number appeared below each bar (González et al., 1995).

Content modifications can be made at the level of dimensions, item stems, or response options, all of which can be added, dropped, modified, or replaced. Adding dimensions or items may be indicated when additional components are found to be needed. Dropping dimensions or items might be done when either is found to be unsuitable for a particular group. Replacing items might be done when an item is unsuitable and a comparable alternative has been suggested by respondents during cognitive interviews.

Context modifications are those made primarily because of study-specific differences, for example, due to a different referent such as nurses instead of doctors. This can include changes to instructions for a self-administered measure to be used with verbal or web-based administration. For example, to modify a Consumer Assessment of Healthcare Providers and Systems (CAHPS®) survey for an American Indian health service setting, items were modified to substitute the term “doctor or nurse” for “doctor,” and to substitute “health professional” for “health provider” (Weidmer-Ocampo et al., 2009).

Format and presentation modifications include changes in appearance or the way of responding to reduce respondent burden or enhance readability. For older adults this might include the use of audio or touch screens and the addition of response aids; it might also include simplifying instructions and increasing the font size. Mullin and colleagues (2000) have reviewed “state-of-the-art” formatting methods for self-administered questionnaires that are known to improve data quality, on the basis of an extensive literature review. They describe a variety of formatting and presentation ideas that can reduce errors in responding, reduce burden, and enhance motivation for respondents to complete questionnaires. For example, a study focused on improving self-reported questionnaires for older people with multiple sclerosis, inconsistent formatting was distracting; thus, questionnaires were reformatted for consistency (e.g., response choices printed below each question; Ploughman, Austin, Stefanelli, & Godwin, 2010).

Recommendations for Assessing Modified Measures

Once a measure is modified it is essential that the new measurement be assessed for its reliability and validity. For some minor modifications such as modifying font typeface, it may not be necessary to conduct a formal assessment of the psychometric properties of the instrument. However, for changes such as adding or deleting a number of question items or modifying response categories, investigators should seriously consider more formal assessment of the new measure.

One classification system for determining the magnitude or extensiveness of a modification was developed by Coons and colleagues, which is based on defining potential effects of modification on “the content, meaning, or interpretation of the measure’s items and/or scales” (Coons et al., 2009, pp. 422-423). The system was originally conceived for electronic data collection methods, but it is likely equally applicable to paper-and-pencil or orally administrated instruments. The three-level classification system includes the following:

One of the biggest challenges faced by health disparities researchers is the time and cost that is involved in a full-scale psychometric assessment of a measure that was extensively modified. As Coons and colleagues (2009) note, the rigor of equivalence testing probably should vary with the extent of modification. However, regardless of the level of modification, it is judicious to test the new measure for psychometric adequacy as well as equivalence with the original measure. For minor modifications, a small-scale pretest would suffice to assess that the changes are working as expected. For moderate modifications, a more thorough assessment of the psychometric adequacy of the measure or the extent to which its properties are similar to the original measure should be undertaken. For substantial modifications, where there will be little in the way of prima facie evidence for measurement adequacy or equivalence, a full-scale psychometric assessment is probably needed. In many cases, it is impractical to conduct a detailed assessment and the investigators must balance the requirements of a time-limited research program with having a modified measure without fully assessing its reliability and validity. We offer some thoughts on this tricky situation from our own and others’ experiences.

One approach we recommend is to conduct an in-depth pretest on a small sample to determine whether the modifications are “working” prior to going into the field with the main study. While likely not definitive about measurement performance, this kind of information can be invaluable in determining whether there are major problems with the modified measure prior to mounting a major study that includes it. Once the investigative team decides to move forward with pretesting the measure, it is possible to assess its psychometric properties. Typically, these assessments would include item-scale correlations and internal consistency reliability. For example, Fongwa and colleagues field-tested a modified patient satisfaction questionnaire in a sample of African Americans and Whites and reported results of extensive psychometric analysis (Fongwa, Hays, Gutierrez, & Stewart, 2006). Ideally, one would evaluate the adequacy of the original and the modified measure in the new sample (Hays, Hahn, & Marshall, 2002), but this requires administering the original measure and the modified items. To allow for this possibility, we strongly suggest that researchers do not drop any items and instead add new or modified items to the established measure (Aroian, Hough, Templin, & Kaskiri, 2008; Hays et al., 2002); only in this way will it be possible to compare the original and modified measures. For example, González and colleagues analyzed the construct validity of their modified Visual Analogue Pain Scale in relation to the original Visual Analogue Pain Scale for Hispanics recruited from several U.S. communities; the correlation of the modified scale and the original was .72, and the modified scale had less missing data (6% compared to 24%; González, Stewart, Ritter, & Lorig, 1995). There are several other examples of the benefits of comparing the original and modified measures in the same study (Aroian et al., 2008; Kazis et al., 2004; Tucker et al., 1998). However, including both old and new versions in the same study may not be practical and can introduce context effects.

Investigators should also consider how they might assess the validity of the modified measure. One approach is to conduct validity tests to parallel those done with the original measure; thus, administration of the same indicators of validity used with the original measure is necessary. The expectation is that the modified measure is an improvement over the original measure in the new context or population (Hays et al., 2002). For example, in a study to adapt a patient satisfaction instrument for lower-literacy population, two versions of a “new” instrument, one with cartoon/pictorial enhancements and one a computer-assisted telephone delivery, the original self-report text instrument, and new versions were compared head to head in a randomized trial, with each arm using the same validity indicators (Shea et al., 2008). Another approach is to use new indicators of validity to assess the performance of the modified measure. While this approach may produce strong evidence of validity, it makes it difficult to compare the validity assessment with previous validity results using the original measure.

Recommendations for Reporting Results of Modifications

If assessment studies of measure modifications were routinely published, we would gain a tremendous amount of information on how various modifications affect the reliability and validity of measures in new populations. Such results could also point to new strategies and methods for test and measurement assessment. Currently, reporting appears to be missing; for example, in a review of measures of spirituality for use in palliative care, changes in the content of instruments after cultural adaptation were poorly reported (Selman, Harding, Gysels, Speck, & Higginson, 2011).

One approach would be to provide details of the modification and its assessment in a separate methods paper. Investigators reported on an adaptation of the CAHPS Clinician and Group Survey for use in the American Indian Health Service, including methods for deciding to modify the measure, the types of modifications, and the psychometric characteristics (Weidmer-Ocampo et al., 2009). Similarly, in reporting modifications to the CHAMPS Physical Activity Questionnaire (Stewart et al., 2001) for a church-based lifestyle intervention for African Americans, Resnicow and colleagues (Resnicow et al., 2003) clearly described the entire process of modifications, including tests of validity. Other examples include a study adapting a patient satisfaction instrument for low-literacy individuals by modifying the format (Shea et al., 2005) and one adapting a patient satisfaction survey for low-literacy VA (Veteran Affairs) patients by adding illustrations (Weiner et al., 2004).

Another approach is to report details of the modification and assessment process within the methods section of a substantive paper. For example, Nápoles-Springer and colleagues (Nápoles-Springer, Ortiz, O’Brien, Sereno, & Kaplan, 2011) reported their modifications of the Cancer Behavior Inventory in the methods section of a paper on coping resources and self-rated health among Latina breast cancer survivors, including results of psychometric tests of the modified measure.

When publishing papers that include modified measures, we recommend at a minimum reporting: (a) features of the original measure that required modification, (b) source of information on the basis for modifications, (c) specific type of modification made, and (e) how the modified measure was tested for psychometric adequacy and results. It might also be necessary to report on whether permission was obtained to modify it or permission was granted on measure or website. Incorporating this information adds to accumulated knowledge on the particular measure as applied in diverse groups and helps our understanding of broader issues involved in adapting measures for diverse populations. These reporting guidelines may not always be practical, though, given space limitations in published papers and typically brief treatment of measurement issues. Especially when this information would be disproportional to the substantive portion of a paper, we encourage taking advantage of options to provide supplemental methodological information on the websites of journal. This is becoming more common in some journals that discourage extensive methodological information in articles (Tobin, 2000).

Discussion

We have addressed a gap in the literature pertaining to use of measures in health disparities and minority aging studies that are appropriate for the population group. There are virtually no published papers that discuss methodological issues in modifying measures. Our article becomes a first step toward understanding the issues. As the field of measurement in diverse populations evolves, we will increasingly find more information on such modifications. It was very difficult to find examples in the literature on modifications to existing measures for use in a diverse population group. Modifications were typically described in a brief paragraph in the measures section, but it was not possible to find these modifications through keyword searches. Thus, by calling attention to the importance of providing details, we hope to promote better reporting in publications that use modified measures.

Many modifications are designed to improve the ability of the measure to answer specific, contemporary research questions. Thus, often modifications are made to update terminology or to reflect historical changes (Krause, 2006). In this case, modifications are an integral part of the evolution of a concept or measure, with modifications improving an existing measure for use by other researchers. The SF-36 is a good example of evolution in measures of health-related quality of life. As noted by Ware (2000), the SF-36 version 2 includes several modifications to the original SF-36, and the original SF-36 in turn was a modification or subset of several longer-form Medical Outcomes Study (MOS) measures of “functioning and well-being.” The MOS measures in turn were adapted and developed based on measures of health status developed for the Health Insurance Experiment (HIE) (Stewart and Ware, 1992) and HIE measures were based on even earlier measures such as the General Psychological Well-Being Index (Dupuy et al., 1984). Thus, the SF-36 Version 2 reflects numerous iterations of prior measurement instruments, each involving modifications and adaptations of earlier versions to improve the measures.

Other examples of how measures evolve can illustrate this process. One is the adaptation of the Mini-Mental State examination by Teng and Chui (1987) to sample a broader variety of cognitive functions, include a broader range of difficulty levels, and improve reliability and validity. Chatters and colleagues reviewed the evolution of concepts and measures of religiosity for research on the role of religiosity in the lives of older African Americans (Chatters, Taylor, & Lincoln, 2001). They noted a progression from single measures of church attendance to multidimensional concepts and measures, including prayer, private religious practices, daily spiritual experiences, and organizational religiousness. As the concept evolved through research on how spirituality affects health and well-being, the measures also evolved to reflect the changing concepts.

Consistent with the concept of measurement evolution is the perspective that the validity of the measure is not a property of a test or measure but of a measure tested under a particular set of conditions (Messick, 1995; Sechrest, 2005). This perspective holds that because validity pertains to understanding the meaning of scores, construct validity can only be established incrementally based on the accumulation of evidence on how the measure relates to other measures (Sechrest, 2005, p. 1596). Well-designed modifications thus contribute to enhancing the validity of measures in new and diverse populations.

The scientific study of the adequacy of measures in diverse populations is still emerging, but research promoting understanding of how to modify measures and test those modifications is even newer. By laying out some of the issues in understanding how to modify and test measures and by describing some possible solutions, we hope to guide future research efforts to advance the field of measurement in diverse populations. By highlighting the value of modifications in the ongoing evolution of concepts and measures in studies of minority aging, we also hope to encourage authors of published measures to be flexible in imposing limitations on use of their measures.