A Framework for Understanding Modifications to Measures for Diverse Populations

Abstract

Objectives:

Research on health disparities and determinants of health disparities among ethnic minorities and vulnerable older populations necessitates use of self-report measures. Most established instruments were developed on mainstream populations and may need adaptation for research with diverse populations. Although information is increasingly available on various problems using these measures in diverse groups, there is little guidance on how to modify the measures. A framework of issues to consider when modifying measures for diverse populations is presented.

Methods:

The authors describe reasons for considering modifications, the types of information that can be used as a basis for making modifications, and the types of modifications researchers have made. Recommendations are made to test modified measures to assure their appropriateness, and suggestions are provided on reporting modifications in publications using the measures.

Discussion:

The issues open a dialogue about what appropriate guidelines would be for researchers adapting measures in studies of ethnically diverse populations.

Keywords

measurement modifying measures adapting measures minority aging health disparities

Eliminating health disparities among ethnic minorities and vulnerable older populations is a national priority (Smedley, Stith, & Nelson, 2003; U.S. Department of Health and Human Services, 2000). To accomplish this goal, it is essential to understand factors that contribute to these disparities (National Institute on Aging, 2011). Research on health disparities and its determinants among ethnic minorities and vulnerable older populations necessitates use of self-report measures. However, many of these widely used measures were developed and tested on mainly white, young, and middle-aged well-educated samples. These measures may have limitations when used in studies of minority or lower-socioeconomic status (SES) older adults included in health disparities and minority aging research.

A substantial amount of research has attempted to address these issues over the past 15 years, including systematic efforts by the Resource Centers for Minority Aging Research (RCMARs; Stahl & Hahn, 2006; also see preface for an overview of RCMAR contributions in measurement in diverse older populations). There are published guidelines on methods for examining the conceptual and psychometric adequacy of measures in ethnically diverse populations (Collins, 2003; Hahn & Cella, 2003; Johnson, 2006; Nápoles-Springer, Santoyo-Olsson, O’Brien, & Stewart, 2006; Nápoles-Springer & Stewart, 2006; Stewart & Nápoles-Springer, 2003; Teresi, Stewart, Morales, & Stahl, 2006). There also are systematic reviews examining the conceptual or psychometric adequacy of particular concepts and measures in diverse populations (Coates & Monteilh, 1997; Stewart & Nápoles-Springer, 2000), including older adults (Mui, Burnette, & Chen, 2001; Mutran, Reed, & Sudha, 2001).

However, when problems are found with measures in diverse population groups, there are no guidelines on what to do next. Once a measure has been determined to be inappropriate for minority or lower-SES participants, researchers have three options. One is to use the measure “as is” without modification and articulate the limitations. This abides by the tenet of administering measures as published, which some believe preserves score reliability and validity (Juniper, 2009). However, if a measure is not suitable for a population, scientific inferences derived from it may be compromised.

A second option is to create a new measure de novo. Developing a new measure is fraught with challenges, including having the expertise, time, and resources to develop and test a new measure, which may not be practical for most health disparities researchers. Methods for developing new measures to be culturally sensitive involve a mixed-methods approach that includes concept development, writing items, pretesting, revising, field testing, and conducting psychometric analysis to derive final measures. Although there are no general guidelines for developing new measures in minority populations, several publications provide detailed examples of these steps (Jackson, 1996; Krause, 2006; Stewart, Nápoles-Springer, Gregorich, & Santoyo-Olsson, 2007).

The third option is to modify or adapt an existing measure. There is a delicate balance between trying to retain the strength of an existing measure, which may have undergone extensive development and testing, but clearly will be problematic, and making modifications that may or may not work. Furthermore, there are very few practical guidelines on how to go about making those modifications. In multinational research, measures developed in English must be translated for use in non–English-speaking countries. In guidelines for multinational studies, adaptations are an integral part of the translation process; for example, items may be modified to achieve “semantic” equivalence (Aaronson et al., 1992; Beaton, Bombardier, Guillemin, & Ferraz, 2000; Bullinger et al., 1998). However, these guidelines give little attention to issues in modifying measures when no language translation is needed.

This article attempts to address this gap by providing a framework for modifying measures to improve their reliability and validity in health disparities research involving ethnically and racially diverse populations. The goal of such modifications would be to increase the likelihood that the modified measure has comparable meaning, reliability, and validity as the original measure, but in a new population group. The issues raised here pertain primarily to addressing differences from mainstream populations (on which original measures were tested) in SES, race and ethnicity, language, and literacy. Issues of modifications based on these types of group differences are relevant to health disparities studies of adults of all ages. Occasionally, modifications are made to adapt a measure specifically to be more appropriate for older adults (Thiamwong, Stewart, & Warahut, 2009; Vanderplas & Vanderplas, 1981). We describe three issues: (a) Reasons for considering modifications—why a modification would be needed; (b) the basis for modifications—information that can be used to make the modifications; and (c) possible types of modifications. We conclude with recommendations for assessing modified measures and for reporting results of these assessments.

Reasons for Considering Modifications to a Measure

There are a number of reasons why investigators might consider modifying a measure. In health disparities research, the most common reason is that the population group(s) being studied differs substantially from the one in which the original measure was developed. Essentially, the motivation for modifying measures is the concern that racial and ethnic or generational differences might adversely affect the meaning, reliability, or validity of the original measure (information on these basic measurement concepts is available elsewhere; for example, McDowell & Newell, 2006; Nunnally & Bernstein, 1994). Some key reasons why measures developed in one group may not be suitable include: (a) a concept or dimension may be missing from a measure, (b) the meaning of the concepts or items may differ, (c) items/phrases may not be interpreted as intended, (d) the process of responding to the questions is complex or difficult, and (e) the study context or mode of administration may differ from the original. Any of these reasons could result in findings that measures do not meet minimal psychometric criteria in these new groups or that the measures do not demonstrate invariance across groups (Gregorich, 2006; Stewart & Nápoles-Springer, 2000, 2003).

Concept or dimension is missing

Many concepts used in health disparities research are complex and multidimensional. As researchers delve into existing measures, it often is apparent that key dimensions relevant to a diverse group may be missing. For example, in focus groups exploring social support concepts for older Chinese and Korean immigrants, getting help with English in conducting business or in health care visits and aspects of financial support were identified as important but were not included in standard measures of social support (Wong, Yoo, & Stewart, 2005).

The meaning or appropriateness of concepts may differ

The meaning of the existing concepts that are the target of a measure may differ by race or ethnicity or by generation. Published measures reflect the population, place, and time in which they were developed. To the extent that a new population group may define or perceive the concept differently, the original measure may lack content validity for the new group. One example is the use of food frequency questionnaires to assess intake of foods and nutrients. The validity of food frequency instruments is partly dependent on the list of foods; thus, use of a food frequency questionnaire developed for a general population in a study of a minority population may result in underestimates of intake unless it is adapted for the new population (Tucker, Bianchi, Maras, & Bermudez, 1998). Investigators aiming to determine energy and nutrient sources of older Hispanics modified a questionnaire developed by the National Cancer Institute to add Southwestern regional foods such as chile rellenos and tamales (Mayer-Davis et al., 1999). Without these modifications, researchers would not have learned that chile sauces were major sources of Vitamins A and C for Hispanic older adults.

The new group may not interpret item terms/phrases as intended

A measure might include terms and phrases that may be misinterpreted due to unfamiliar language, idioms, or colloquialisms. Terms may be so general as to generate a variety of interpretations, and words may be confusing. There is a greater likelihood of unfamiliar terminology when working with low-literate or nonnative English-speaking populations. For example, several items in the Beck Depression Inventory (Beck, Steer, & Garbin, 1988) were misunderstood in a sample of older and low-literacy patients, leading to changes to make the items clearer (Sentell & Ratcliff-Baird, 2003). In a pretest of diabetes knowledge in low-literate Spanish-speaking patients with diabetes, the concept “blood sugar drop” was not understood; similarly in the same study the phrase “bothered by” in the CES-D item, “I was bothered by things that usually don’t bother me,” was interpreted as having a physical complaint (Rosal, Carbone, & Goins, 2003).

The new group may use different styles of responding

Systematic differences may exist across racial and ethnic groups in the way they respond to survey question items. For example, Blacks and Latinos are more likely than Whites to choose the extreme response options rather than middle options of Likert-type response scales (Bachman & O’Malley, 1984; Hui & Triandis, 1989; Warnecke et al., 1997). In contrast, Asian Americans appear to be more likely than Whites to avoid extremely positive options in surveys of health services (Murray-Garcia, Selby, Schmittdiel, Grumbach, & Quesenberry, 2000; Ngo-Metzger, Legedza, & Phillips, 2004; Taira et al., 1997).

Process of responding is complex or difficult

For older adults who might have limited vision, or for those with limited English proficiency or low literacy, question formatting and the response task may impact the validity of a measure. Complex instructions and formatting may work with younger White populations but could be challenging for older minority groups or persons with low literacy (Mullin, Lohr, Bresnahan, & McNulty, 2000; Rosal et al., 2003). Scientific studies have examined ways to present and format questions to maximize respondent ability to answer the questions accurately (Stone et al., 2000).

The study context or mode of administration may differ from that in which the original measure was developed

Studies need to be sensitive to the appropriateness of different modes of data collection. For example, pencil and paper administration may pose no difficulties in middle-aged populations but might be inappropriate for older populations with low vision where oral administration might work better. The growth in use of technologies to accommodate low-literacy populations, such as audiovisual computer-based platforms (Hahn & Cella, 2003; Hahn et al., 2004) as well as other electronic modes of administration (web-based surveys, touch-screen computers, handheld computers, interactive voice response, and automated telephone assessment), has led to a call for extensive testing to assure the comparability of electronic and paper-based measures (Coons et al., 2009).

Basis for Modifications: Source of Information Used to Decide How to Modify

Once a decision is made to modify an existing measure, investigators need information on which to base those modifications. Such information is often available in the same studies that identified the need for modifications. Three sources of information can be used as a basis for modifications: (a) qualitative research on a concept or measure, (b) literature reviews of the adequacy of a measure, and (c) investigator experience or judgment. We review these three information sources and provide examples of how the information source was used to make modifications.

Qualitative research on concept or measure

The two most common qualitative methods for exploring concepts and measures in diverse populations include cognitive interview pretests (Collins, 2003; Drennan, 2003; Nápoles-Springer et al., 2006; Rosal et al., 2003; Willis, 2005) and focus groups (Fuller, Edwards, Vorakitphokatorn, & Sermsri, 1993; Hughes & DuMont, 1993; Nápoles-Springer, Santoyo-Olsson, Houston, Perez-Stable, & Stewart, 2005; Vogt, King, & King, 2004). In addition to identifying specific problems, transcripts often include dialogue that can be used to help make modifications such as alternative phrasing. Qualitative research on the adequacy of a specific measure in a new population group often identifies items that are not culturally appropriate or relevant. The following are some examples of how qualitative research provided information by which to modify a measure:

Cognitive interviewing to test several diabetes-related measures for low-literate, Spanish-speaking older adults with diabetes resulted in several modifications (Rosal et al., 2003). Responses to probes regarding the suitability of specific words, clarity of response options, and clarity of instructions provided suggestions for modifications.

To develop new dimensions of language support and financial support for older Chinese and Korean immigrants, transcripts of the focus group dialogue provided specific ideas and phrasing for writing new items (Wong, Yoo, & Stewart, 2007).

To explore the concept of cultural sensitivity in health care, Nápoles-Springer and colleagues conducted focus groups with African Americans, Latinos, and non-Latino Whites (Nápoles-Springer et al., 2005), querying about culture and what cultural factors influenced the quality of their medical encounters. Analysis of transcripts identified several dimensions of cultural sensitivity, forming the basis for a new multidimensional measure based in part on participants’ phrases and comments (Nápoles-Springer, Santoyo-Olsson et al., 2012).

Literature reviews

Literature reviews of the adequacy or appropriateness of measures of particular concepts for racial and ethnic groups can be done by individual investigators or by convening an expert panel (Mutran et al., 2001). For example, literature on measures of park and recreation environments was reviewed for adequacy in studies of how parks and recreation settings contribute to physical activity in low-income communities of color. The review provided suggestions for improving measures, for example, the need to reflect the concerns and preferences of residents in these communities and to “explicitly reflect inequality in the availability and quality of parks and recreation areas” (Floyd, Taylor, & Whitt-Glover, 2009). Another example is from a review of physical activity questionnaires for appropriateness for middle-aged and older minority women (Masse et al., 1998). The authors suggested a number of alterations: clarify physical activity–related words like exercise and physical activity, improve definitions of phrases like moderate physical activity, modify instructions about walking to capture walking in multiple contexts, modify unsuitable phrases such as leisure-time activities, and substitute culturally relevant activities such as dancing for items such as playing golf or tennis. Other examples of how literature reviews of measures provided specific information by which to modify measures include reviews of measures of SES for minority aging studies (Rudkin & Markides, 2001), measures of acculturation (Salant & Lauderdale, 2003; Skinner, 2001), and food frequency questionnaires for minority populations (Coates & Monteilh, 1997).

Investigator experience

Investigator experience and knowledge can provide ideas for modifications, based on long-term programs of research with diverse populations. For example, on the basis of their experience conducting research with persons with disabilities, Meyers and Andresen (2000) recommended shortening the recall period in health-related quality of life instruments for patients with disabilities. For instance, changes may occur over very short periods of time in multiple sclerosis patients, and in patients with stroke, recall periods need to be compressed due to short-term memory loss.

Types of Modifications

The types of modifications that can be made range from simple format changes such as improving the contrast or increasing the font size to extensive changes such as adding new subscales or changing item wording. To facilitate thinking about the various types of modifications, we have classified them into three broad categories: content, context, and format. For each of these, in Table 1, we define the various specific modifications and provide examples.

Table 1.

Organizing Framework of Types of Possible Modifications with Definitions and Examples

Category/type of modification	Definition	Examples
Content modifications
Dimensions
Add dimension	Add a set of items hypothesized to measure a new dimension (subscale)	Two dimensions were added to an existing social support measure: “language support” and “financial support” (Wong et al., 2007) in response to qualitative research that these were important to older Chinese and Korean immigrants (Wong et al., 2005). Items were written based on the same qualitative studies.
		A new “discrimination/stereotyping” dimension was added to an existing patient satisfaction measure in response to qualitative studies indicating its importance to older African Americans (Fongwa et al., 2006).
		An “integration” dimension (providing various types of support to others in their social environment) was added to a social support measure to make it more appropriate for older adults (Krause & Markides, 1990).
Drop dimension	Omit all items from one dimension (subscale)	The SF-36 physical functioning scale was omitted in a study of spinal cord injury patients because patients considered the items demeaning, given that walking and climbing stairs were prominent items (Mattson-Prince, 1997).
Item stems
Drop items	Drop items from an existing scale	Four items from the Cancer Behavior Inventory referring to treatment were dropped in a study of Latina breast cancer survivors who had completed treatment (Nápoles-Springer et al., 2011).
		A few items from the McNeilly Perceived Racism Scale pertaining to racism in academic settings were dropped as not applicable to older African American patients (Moody-Ayers, Stewart, Covinsky, & Inouye, 2005).
		Several items from the CES-D were dropped as being inappropriate for older Asian groups (reported by Mui et al., 2001, p. 141).
		Eight items were omitted from the CES-D in a study of older American Indians because they were determined to be inappropriate (Chapleski, Lamphere, Kaczynski, Lichtenberg, & Dwyer, 1997).
Add items	Add items to an existing scale to fill in a missing element	The Insulin Resistance Atherosclerosis Study food frequency questionnaire was modified to add foods typically eaten by Chinese persons for a multiethnic study of atherosclerosis (Bild et al., 2002).
		The CHAMPS Physical Activity Questionnaire was modified for a church-based intervention for African Americans. Based on focus groups and pilot testing, “praise dancing” was added because it was common and contributed a substantial amount of energy expenditure (Resnicow et al., 2003).
Modify items	Substitute new term, modify wording without changing meaning, split complex item into two	Item stems in the Family Peer Relationship Questionnaire were modified to be appropriate for Arab Muslim women, based on information from five cultural informants (Arab immigrant women from the Middle East) with experience providing health and social services to Arab Muslim families (Aroian et al., 2008).
		Hays and colleagues (2002) suggest modifying some SF-36 items for studies of patients with disabilities by changing “does your health now limit you in these activities” to “does your health or a disability now limit you in these activities.”
		Several items in a patient satisfaction survey were reworded or split into two items based on problems identified with original items in qualitative research with older African Americans (Fongwa et al., 2006).
		To adapt CES-D items for verbal administration to low-literate respondents, based on cognitive pretesting, statements were modified to be questions and a time frame was added to each item (from only being in the initial instructions). For example, “I felt depressed” was modified to “During the past week, how many days did you feel depressed?” (Rosal et al., 2003).
Response options
Replace response option set	Replace an entire set of response options with another.	To adapt a measure for older Thai adults, Thiamwong and colleagues (Thiamwong et al., 2009) replaced a bidirectional true/false response scale (1 = definitely true to 5 = definitely false) with a unidirectional response scale (1 = not at all true to 5 = definitely true).
		To improve variability of two role functioning scales from the SF-36 in the Veterans Health Study, the yes/no response options were replaced with a 5-level ordinal scale (ranging from yes, all of the time to no, none of the time; Kazis et al., 1999).
Modify response options	Split response options into two levels to reduce the number of options presented at any one time	For a study of low-literate individuals, a 4-point Likert-type scale response option ranging from 4 = very important to 1 = not important was modified to be administered in a nested format. A 2-point scale was first administered (important, not important); for those responding important, three options were offered to indicate the strength of the choice (very important, important, somewhat important; Rosal et al., 2003).
	Add verbal labels to all response options rather than having only endpoints labeled	For a study of low-literate individuals, response options asking how confident they are they can do something with endpoints anchored (1 = not at all, 4 = completely) were modified to provide a verbal anchor for each level (1 = not confident, 2 = a little confident, 3 = somewhat confident, 4 = completely confident; Rosal et al., 2003).
	Drop a response option, for example, to reduce confusion	The SF-36 Version 1 had six response options for the mental health and vitality items (1 = all of the time, 2 = most of the time, 3 = a good bit of the time, 4 = some of the time, 5 = a little of the time, 6 = none of the time). In Version 2, “a good bit of the time” was dropped because respondents consistently had difficulty understanding its meaning, leaving five options (McDowell & Newell, 2006; Ware, 2000).
	Remove numbering from response scales	In adapting the Spanish Brief Religious Coping scale, the original items were asked on a 4-point response scale: 0 = not at all to 3 = a great deal. Because of difficulty with Likert-type scales, response options were modified to omit the numbers (leaving the phrasing: not all, somewhat, quite a bit, a great deal; Martinez & Sousa, 2011).
Study-specific context modifications
Modify recall period	Different time frame applies	The CHAMPS Physical Activity Questionnaire was modified for an intervention for African Americans; the 1-month recall period for activities was changed to 2 weeks, based on formative research indicating that respondents could more accurately recall activity during a shorter period (Resnicow et al., 2003).
Change referent	Change referent in instructions or item stems	To adapt the Everyday Discrimination Scale for use in a health care setting, instructions asking whether they “ever had any of the following things happen . . .” were modified to “when getting health care, have you ever had. . .” (Bird & Bogart, 2001).
		An item in the Experiences of Racial Discrimination Scale referring to discrimination “while getting medical care” was modified to refer to getting care in a Veteran’s Administration (VA) health care setting (Hausmann, Kressin, Hanusa, & Ibrahim, 2010).
Format or presentation modifications
Modify format for responding	Modify way in which respondents answer questions, for example, check box instead of circle number	Instead of asking respondents to circle the number in front of each response, for each response choice, they were asked to check a box (Walter et al., 2008).
Modify appearance of page	Make skip patterns more obvious, reduce crowding, increase font, improve contrast	Because reading performance declines for older adults and persons with impaired vision when questionnaires use small fonts, questionnaires were designed using 14-point font (Vanderplas & Vanderplas, 1981).
Add visual aids for response options	Use a card with the response options to aid remembering the options	An interviewer-administered survey in a study of lower-income, minority adults used a large card containing response options for each set of questions. Interviewers showed the appropriate card to respondents which aided of the options and allowed respondents to point to a response (Santoyo-Olsson et al., 2011).
	Add visual aid to illustrate increasing intensity	A Visual Analogue Pain Scale for Spanish-speaking individuals was modified from a 10-cm line with end points labeled (Dixon & Bird, 1981) to a histogram; bars increased in size from 0 to 10 visually depicted increasing pain intensity and a number appeared below each bar (González et al., 1995).

Content modifications can be made at the level of dimensions, item stems, or response options, all of which can be added, dropped, modified, or replaced. Adding dimensions or items may be indicated when additional components are found to be needed. Dropping dimensions or items might be done when either is found to be unsuitable for a particular group. Replacing items might be done when an item is unsuitable and a comparable alternative has been suggested by respondents during cognitive interviews.

Context modifications are those made primarily because of study-specific differences, for example, due to a different referent such as nurses instead of doctors. This can include changes to instructions for a self-administered measure to be used with verbal or web-based administration. For example, to modify a Consumer Assessment of Healthcare Providers and Systems (CAHPS®) survey for an American Indian health service setting, items were modified to substitute the term “doctor or nurse” for “doctor,” and to substitute “health professional” for “health provider” (Weidmer-Ocampo et al., 2009).

Format and presentation modifications include changes in appearance or the way of responding to reduce respondent burden or enhance readability. For older adults this might include the use of audio or touch screens and the addition of response aids; it might also include simplifying instructions and increasing the font size. Mullin and colleagues (2000) have reviewed “state-of-the-art” formatting methods for self-administered questionnaires that are known to improve data quality, on the basis of an extensive literature review. They describe a variety of formatting and presentation ideas that can reduce errors in responding, reduce burden, and enhance motivation for respondents to complete questionnaires. For example, a study focused on improving self-reported questionnaires for older people with multiple sclerosis, inconsistent formatting was distracting; thus, questionnaires were reformatted for consistency (e.g., response choices printed below each question; Ploughman, Austin, Stefanelli, & Godwin, 2010).

Recommendations for Assessing Modified Measures

Once a measure is modified it is essential that the new measurement be assessed for its reliability and validity. For some minor modifications such as modifying font typeface, it may not be necessary to conduct a formal assessment of the psychometric properties of the instrument. However, for changes such as adding or deleting a number of question items or modifying response categories, investigators should seriously consider more formal assessment of the new measure.

One classification system for determining the magnitude or extensiveness of a modification was developed by Coons and colleagues, which is based on defining potential effects of modification on “the content, meaning, or interpretation of the measure’s items and/or scales” (Coons et al., 2009, pp. 422-423). The system was originally conceived for electronic data collection methods, but it is likely equally applicable to paper-and-pencil or orally administrated instruments. The three-level classification system includes the following:

Minor modifications that are not expected to change content or meaning. This would include changing from paper-and-pencil format into a screen text or using touch response on screen instead of circling a response on paper.

Moderate modifications may change the meaning of the items but in small, subtle ways. There are many examples such as splitting a single item into two, changes in item wording, changing the order of item presentation, and changing the mode of administration from self-administration to interactive voice.

Substantial modifications are more extensive and almost always change the content or meaning of the measure. These more aggressive modifications might include dropping items and changes in item wording or response options.

One of the biggest challenges faced by health disparities researchers is the time and cost that is involved in a full-scale psychometric assessment of a measure that was extensively modified. As Coons and colleagues (2009) note, the rigor of equivalence testing probably should vary with the extent of modification. However, regardless of the level of modification, it is judicious to test the new measure for psychometric adequacy as well as equivalence with the original measure. For minor modifications, a small-scale pretest would suffice to assess that the changes are working as expected. For moderate modifications, a more thorough assessment of the psychometric adequacy of the measure or the extent to which its properties are similar to the original measure should be undertaken. For substantial modifications, where there will be little in the way of prima facie evidence for measurement adequacy or equivalence, a full-scale psychometric assessment is probably needed. In many cases, it is impractical to conduct a detailed assessment and the investigators must balance the requirements of a time-limited research program with having a modified measure without fully assessing its reliability and validity. We offer some thoughts on this tricky situation from our own and others’ experiences.

One approach we recommend is to conduct an in-depth pretest on a small sample to determine whether the modifications are “working” prior to going into the field with the main study. While likely not definitive about measurement performance, this kind of information can be invaluable in determining whether there are major problems with the modified measure prior to mounting a major study that includes it. Once the investigative team decides to move forward with pretesting the measure, it is possible to assess its psychometric properties. Typically, these assessments would include item-scale correlations and internal consistency reliability. For example, Fongwa and colleagues field-tested a modified patient satisfaction questionnaire in a sample of African Americans and Whites and reported results of extensive psychometric analysis (Fongwa, Hays, Gutierrez, & Stewart, 2006). Ideally, one would evaluate the adequacy of the original and the modified measure in the new sample (Hays, Hahn, & Marshall, 2002), but this requires administering the original measure and the modified items. To allow for this possibility, we strongly suggest that researchers do not drop any items and instead add new or modified items to the established measure (Aroian, Hough, Templin, & Kaskiri, 2008; Hays et al., 2002); only in this way will it be possible to compare the original and modified measures. For example, González and colleagues analyzed the construct validity of their modified Visual Analogue Pain Scale in relation to the original Visual Analogue Pain Scale for Hispanics recruited from several U.S. communities; the correlation of the modified scale and the original was .72, and the modified scale had less missing data (6% compared to 24%; González, Stewart, Ritter, & Lorig, 1995). There are several other examples of the benefits of comparing the original and modified measures in the same study (Aroian et al., 2008; Kazis et al., 2004; Tucker et al., 1998). However, including both old and new versions in the same study may not be practical and can introduce context effects.

Investigators should also consider how they might assess the validity of the modified measure. One approach is to conduct validity tests to parallel those done with the original measure; thus, administration of the same indicators of validity used with the original measure is necessary. The expectation is that the modified measure is an improvement over the original measure in the new context or population (Hays et al., 2002). For example, in a study to adapt a patient satisfaction instrument for lower-literacy population, two versions of a “new” instrument, one with cartoon/pictorial enhancements and one a computer-assisted telephone delivery, the original self-report text instrument, and new versions were compared head to head in a randomized trial, with each arm using the same validity indicators (Shea et al., 2008). Another approach is to use new indicators of validity to assess the performance of the modified measure. While this approach may produce strong evidence of validity, it makes it difficult to compare the validity assessment with previous validity results using the original measure.

Recommendations for Reporting Results of Modifications

If assessment studies of measure modifications were routinely published, we would gain a tremendous amount of information on how various modifications affect the reliability and validity of measures in new populations. Such results could also point to new strategies and methods for test and measurement assessment. Currently, reporting appears to be missing; for example, in a review of measures of spirituality for use in palliative care, changes in the content of instruments after cultural adaptation were poorly reported (Selman, Harding, Gysels, Speck, & Higginson, 2011).

One approach would be to provide details of the modification and its assessment in a separate methods paper. Investigators reported on an adaptation of the CAHPS Clinician and Group Survey for use in the American Indian Health Service, including methods for deciding to modify the measure, the types of modifications, and the psychometric characteristics (Weidmer-Ocampo et al., 2009). Similarly, in reporting modifications to the CHAMPS Physical Activity Questionnaire (Stewart et al., 2001) for a church-based lifestyle intervention for African Americans, Resnicow and colleagues (Resnicow et al., 2003) clearly described the entire process of modifications, including tests of validity. Other examples include a study adapting a patient satisfaction instrument for low-literacy individuals by modifying the format (Shea et al., 2005) and one adapting a patient satisfaction survey for low-literacy VA (Veteran Affairs) patients by adding illustrations (Weiner et al., 2004).

Another approach is to report details of the modification and assessment process within the methods section of a substantive paper. For example, Nápoles-Springer and colleagues (Nápoles-Springer, Ortiz, O’Brien, Sereno, & Kaplan, 2011) reported their modifications of the Cancer Behavior Inventory in the methods section of a paper on coping resources and self-rated health among Latina breast cancer survivors, including results of psychometric tests of the modified measure.

When publishing papers that include modified measures, we recommend at a minimum reporting: (a) features of the original measure that required modification, (b) source of information on the basis for modifications, (c) specific type of modification made, and (e) how the modified measure was tested for psychometric adequacy and results. It might also be necessary to report on whether permission was obtained to modify it or permission was granted on measure or website. Incorporating this information adds to accumulated knowledge on the particular measure as applied in diverse groups and helps our understanding of broader issues involved in adapting measures for diverse populations. These reporting guidelines may not always be practical, though, given space limitations in published papers and typically brief treatment of measurement issues. Especially when this information would be disproportional to the substantive portion of a paper, we encourage taking advantage of options to provide supplemental methodological information on the websites of journal. This is becoming more common in some journals that discourage extensive methodological information in articles (Tobin, 2000).

Discussion

We have addressed a gap in the literature pertaining to use of measures in health disparities and minority aging studies that are appropriate for the population group. There are virtually no published papers that discuss methodological issues in modifying measures. Our article becomes a first step toward understanding the issues. As the field of measurement in diverse populations evolves, we will increasingly find more information on such modifications. It was very difficult to find examples in the literature on modifications to existing measures for use in a diverse population group. Modifications were typically described in a brief paragraph in the measures section, but it was not possible to find these modifications through keyword searches. Thus, by calling attention to the importance of providing details, we hope to promote better reporting in publications that use modified measures.

Many modifications are designed to improve the ability of the measure to answer specific, contemporary research questions. Thus, often modifications are made to update terminology or to reflect historical changes (Krause, 2006). In this case, modifications are an integral part of the evolution of a concept or measure, with modifications improving an existing measure for use by other researchers. The SF-36 is a good example of evolution in measures of health-related quality of life. As noted by Ware (2000), the SF-36 version 2 includes several modifications to the original SF-36, and the original SF-36 in turn was a modification or subset of several longer-form Medical Outcomes Study (MOS) measures of “functioning and well-being.” The MOS measures in turn were adapted and developed based on measures of health status developed for the Health Insurance Experiment (HIE) (Stewart and Ware, 1992) and HIE measures were based on even earlier measures such as the General Psychological Well-Being Index (Dupuy et al., 1984). Thus, the SF-36 Version 2 reflects numerous iterations of prior measurement instruments, each involving modifications and adaptations of earlier versions to improve the measures.

Other examples of how measures evolve can illustrate this process. One is the adaptation of the Mini-Mental State examination by Teng and Chui (1987) to sample a broader variety of cognitive functions, include a broader range of difficulty levels, and improve reliability and validity. Chatters and colleagues reviewed the evolution of concepts and measures of religiosity for research on the role of religiosity in the lives of older African Americans (Chatters, Taylor, & Lincoln, 2001). They noted a progression from single measures of church attendance to multidimensional concepts and measures, including prayer, private religious practices, daily spiritual experiences, and organizational religiousness. As the concept evolved through research on how spirituality affects health and well-being, the measures also evolved to reflect the changing concepts.

Consistent with the concept of measurement evolution is the perspective that the validity of the measure is not a property of a test or measure but of a measure tested under a particular set of conditions (Messick, 1995; Sechrest, 2005). This perspective holds that because validity pertains to understanding the meaning of scores, construct validity can only be established incrementally based on the accumulation of evidence on how the measure relates to other measures (Sechrest, 2005, p. 1596). Well-designed modifications thus contribute to enhancing the validity of measures in new and diverse populations.

The scientific study of the adequacy of measures in diverse populations is still emerging, but research promoting understanding of how to modify measures and test those modifications is even newer. By laying out some of the issues in understanding how to modify and test measures and by describing some possible solutions, we hope to guide future research efforts to advance the field of measurement in diverse populations. By highlighting the value of modifications in the ongoing evolution of concepts and measures in studies of minority aging, we also hope to encourage authors of published measures to be flexible in imposing limitations on use of their measures.

Footnotes

Acknowledgements

The authors appreciate the very helpful suggestions on earlier drafts of this paper from Steven Gregorich, Ron D. Hays. Anna M. Nápoles, and Irene H. Yen.

Authors’ Note

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute on Aging or the National Institutes of Health.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Supported by Grant Numbers P30-AG015272, P30-AG015292, and P30-AG031043 under the Resource Centers for Minority Aging Research program of the National Institute on Aging.

References

Aaronson

N. K.

Acquadro

Alonso

Apolone

Bucquet

Bullinger

. . . Ware

J. E.

Jr. (1992). International Quality of Life Assessment (IQOLA) project. Quality of Life Research, 1, 349-351.

Aroian

K. J.

Hough

E. S.

Templin

T. N.

Kaskiri

E. A.

(2008). Development and psychometric evaluation of an Arab version of the Family Peer Relationship Questionnaire. Research in Nursing and Health, 31, 402-416.

Bachman

J. G.

O’Malley

P. M.

(1984). Yea-saying, nay-saying, and going to extremes: Black-white differences in response style. Public Opinion Quarterly, 48, 491-509.

Beaton

D. E.

Bombardier

Guillemin

Ferraz

M. B.

(2000). Guidelines for the process of cross-cultural adaptation of self-report measures. Spine, 25, 3186-3191.

Beck

A. T.

Steer

R. A.

Garbin

M. G.

(1988). Psychometric properties of the Beck Depression Inventory: Twenty-five years of evaluation. Clinical Psychology Review, 8, 77-100.

Bild

D. E.

Bluemke

D. A.

Burke

G. L.

Detrano

Diez Roux

A. V.

Folsom

A. R.

. . . Tracy

R. P.

(2002). Multi-ethnic study of atherosclerosis: Objectives and design. American Journal of Epidemiology, 156, 871-881.

Bird

S. T.

Bogart

L. M.

(2001). Perceived race-based and socioeconomic status (SES)-based discrimination in interactions with health care providers. Ethnicity and Disease, 11, 554-563.

Bullinger

Alonso

Apolone

Leplege

Sullivan

Wood-Dauphinee

. . . Ware

J. E.

Jr. (1998). Translating health status questionnaires and evaluating their quality: The IQOLA Project approach. Journal of Clinical Epidemiology, 51, 913-923.

Chapleski

E. E.

Lamphere

J. K.

Kaczynski

Lichtenberg

P. A.

Dwyer

J. W.

(1997). Structure of a depression measure among American Indian elders: Confirmatory factor analysis of the CES-D scale. Research on Aging, 19, 462-485.

10.

Chatters

L. M.

Taylor

R. J.

Lincoln

K. D.

(2001). Advances in the measurement of religiosity among older African Americans: Implications for health and mental health researchers. Journal of Mental Health and Aging, 7, 180-200.

11.

Coates

R. J.

Monteilh

C. P.

(1997). Assessments of food-frequency questionnaires in minority populations. American Journal of Clinical Nutrition, 65, 1108-1115.

12.

Collins

(2003). Pretesting survey instruments: An overview of cognitive methods. Quality of Life Research, 12, 229-238.

13.

Coons

S. J.

Gwaltney

C. J.

Hays

R. D.

Lundy

J. J.

Sloan

J. A.

Revicki

D. A.

. . . Basch

(2009). Recommendations on evidence needed to support measurement equivalence between electronic and paper-based patient-reported outcome (PRO) measures: ISPOR ePRO good research practices task force report. Value Health, 12, 419-429.

14.

Dixon

J. S.

Bird

H. A.

(1981). Reproducibility along a 10 cm vertical visual analogue scale. Annals of the Rheumatic Diseases, 40, 87-89.

15.

Drennan

(2003). Cognitive interviewing: Verbal data in the design and pretesting of questionnaires. Journal of Advanced Nursing, 42, 57-63.

16.

Dupuy

H. J.

(1984). The psychological general well-being (PGWB) index. In Wenger

N. K.

Mattson

M. D.

Furberg

C. D.

Elinson

(Eds.), Assessment of quality of life in clinical trials of cardiovascular therapies (pp. 170-83). New York: Le Jacq Publishing, Inc.

17.

Floyd

M. F.

Taylor

W. C.

Whitt-Glover

(2009). Measurement of park and recreation environments that support physical activity in low-income communities of color: Highlights of challenges and recommendations. American Journal of Preventive Medicine, 36, 156-160.

18.

Fongwa

M. N.

Hays

R. D.

Gutierrez

P. R.

Stewart

A. L.

(2006). Psychometric characteristics of a patient satisfaction instrument tailored to the concerns of African Americans. Ethnicity and Disease, 16, 948-955.

19.

Fuller

T. D.

Edwards

J. N.

Vorakitphokatorn

Sermsri

(Eds.). (1993). Using focus groups to adapt survey instruments to new populations. Newbury Park, CA: Sage.

20.

González

V. M.

Stewart

Ritter

P. L.

Lorig

(1995). Translation and validation of arthritis outcome measures into Spanish. Arthritis and Rheumatism, 38, 1429-1446.

21.

Gregorich

S. E.

(2006). Do self-report instruments allow meaningful comparisons across diverse population groups? Testing measurement invariance using the confirmatory factor analysis framework. Medical Care, 44, 78-94.

22.

Hahn

E. A.

Cella

(2003). Health outcomes assessment in vulnerable populations: Measurement challenges and recommendations. Archives of Physical Medicine and Rehabilitation, 84, 35-42.

23.

Hahn

E. A.

Cella

Dobrez

Shiomoto

Marcus

Taylor

S. G.

. . . Webster

(2004). The talking touchscreen: A new approach to outcomes assessment in low literacy. Psycho-Oncology, 13, 86-95.

24.

Hausmann

L. R.

Kressin

N. R.

Hanusa

B. H.

Ibrahim

S. A.

(2010). Perceived racial discrimination in health care and its association with patients’ healthcare experiences: Does the measure matter? Ethnicity and Disease, 20, 40-47.

25.

Hays

R. D.

Hahn

Marshall

(2002). Use of the SF-36 and other health-related quality of life measures to assess persons with disabilities. Archives of Physical Medicine and Rehabilitation, 83, S4-9.

26.

Hughes

DuMont

(1993). Using focus groups to facilitate culturally anchored research. American Journal of Community Psychology, 21, 775-806.

27.

Hui

C. H.

Triandis

H. C.

(1989). Effects of culture and response format on extreme response style. Journal of Cross-Cultural Psychology, 20, 296-309.

28.

Jackson

J. S.

(1996). Concepts and measures in the National Survey of Black Americans. In Jones

R. L.

(Ed.), Handbook of tests and measurements for black populations (Vol. 2, pp. 535-540). Hampton, VA: Cobb & Henry.

29.

Johnson

T. P.

(2006). Methods and frameworks for crosscultural measurement. Medical Care, 44, S17-20.

30.

Juniper

E. F.

(2009). Validated questionnaires should not be modified. European Respiratory Journal, 34, 1015-1017.

31.

Kazis

L. E.

Miller

D. R.

Clark

J. A.

Skinner

K. M.

Lee

Ren

X. S.

. . . Ware

J. E.

Jr. (2004). Improving the response choices on the veterans SF-36 health survey role functioning scales: Results from the Veterans Health Study. Journal of Ambulatory Care Management, 27, 263-280.

32.

Kazis

L. E.

Ren

X. S.

Lee

Skinner

Rogers

Clark

Miller

D. R.

(1999). Health status in VA patients: Results from the Veterans Health Study. American Journal of Medical Quality, 14, 28-38.

33.

Krause

(2006). The use of qualitative methods to improve quantitative measures of health-related constructs. Medical Care, 44, S34-38.

34.

Krause

Markides

(1990). Measuring social support among older adults. International Journal of Aging and Human Development, 30, 37-53.

35.

Martinez

N. C.

Sousa

V. D.

(2011). Cross-cultural validation and psychometric evaluation of the Spanish Brief Religious Coping Scale (S-BRCS). Journal of Transcultural Nursing, 22, 248-256. doi:10.1177/104365961140442610.1177/1043659611404426

36.

Masse

L. C.

Ainsworth

B. E.

Tortolero

Levin

Fulton

J. E.

Henderson

K. A.

Mayo

(1998). Measuring physical activity in midlife, older, and minority women: Issues from an expert panel. Journal of Womens Health, 7, 57-67.

37.

Mattson-Prince

(1997). A rational approach to long-term care: Comparing the independent living model with agency-based care for persons with high spinal cord injuries. Spinal Cord, 35, 326-331.

38.

Mayer-Davis

E. J.

Vitolins

M. Z.

Carmichael

S. L.

Hemphill

Tsaroucha

Rushing

Levin

(1999). Validity and reproducibility of a food frequency interview in a multi-cultural epidemiology study. Annals of Epidemiology, 9, 314-324.

39.

McDowell

I. Y.

Newell

(2006). Measuring health: A guide to rating scales and questionnaires (3rd ed.). New York, NY: Oxford University Press.

40.

Messick

(1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

41.

Meyers

A. R.

Andresen

E. M.

(2000). Enabling our instruments: Accommodation, universal design, and access to participation in research. Archives of Physical Medicine and Rehabilitation, 81, S5-9.

42.

Moody-Ayers

S. Y.

Stewart

A. L.

Covinsky

K. E.

Inouye

S. K.

(2005). Prevalence and correlates of perceived societal racism in older African American adults with type 2 diabetes mellitus. Journal of the American Geriatrics Society, 53, 2202-2208.

43.

Mui

A. C.

Burnette

Chen

L. M.

(2001). Cross-cultural assessment of geriatric depression: A review of the CES-D and the GDS. Journal of Mental Health and Aging, 7, 137-164.

44.

Mullin

P. A.

Lohr

K. N.

Bresnahan

B. W.

McNulty

(2000). Applying cognitive design principles to formatting HRQOL instruments. Quality of Life Research, 9, 13-27.

45.

Murray-Garcia

J. L.

Selby

J. V.

Schmittdiel

Grumbach

Quesenberry

C. P.

Jr. (2000). Racial and ethnic differences in a patient survey: Patients’ values, ratings, and reports regarding physician primary care performance in a large health maintenance organization. Medical Care, 38, 300-310.

46.

Mutran

E. J.

Reed

P. S.

Sudha

(2001). Social support: clarifying the construct with applications for minority populations. Journal of Mental Health and Aging, 7, 67-78.

47.

Nápoles-Springer

A. M.

Santoyo-Olsson

Farren

Olmstead

Cabral

Ross

. . . Stewart

A. L.

(2012). The patient-reported Clinicians’ Cultural Sensitivity Survey: A field test among older Latino primary care patients. Health Expectations, 15, 63-77.

48.

Nápoles-Springer

A. M.

Ortiz

O’Brien

Sereno

A. B.

Kaplan

C. P.

(2011). Coping resources and self-rated health among latina breast cancer survivors. Oncology Nursing Forum, 38, 523-531. doi:10.1188/11.ONF.523-53110.1188/11.ONF.523-531

49.

Nápoles-Springer

A. M.

Santoyo-Olsson

Houston

Perez-Stable

E. J.

Stewart

A. L.

(2005). Patients’ perceptions of cultural factors affecting the quality of their medical encounters. Health Expectations, 8, 4-17.

50.

Nápoles-Springer

A. M.

Santoyo-Olsson

O’Brien

Stewart

A. L.

(2006). Using cognitive interviews to develop surveys in diverse populations. Medical Care, 44, S21-30.

51.

Nápoles-Springer

A. M.

Stewart

A. L.

(2006). Overview of qualitative methods in research with diverse populations: Making research reflect the population. Medical Care, 44(Suppl. 3), 5-9.

52.

National Institute on Aging. (2011). Health disparities strategic plan: Fiscal years 2009-2013. Baltimore, MD: Author.

53.

Ngo-Metzger

Legedza

A. T.

Phillips

R. S.

(2004). Asian Americans’ reports of their health care experiences. Results of a national survey. Journal of General Internal Medicine, 19, 111-119.

54.

Nunnally

J. C.

Bernstein

I. H.

(1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.

55.

Ploughman

Austin

Stefanelli

Godwin

(2010). Applying cognitive debriefing to pre-test patient-reported outcomes in older people with multiple sclerosis. Quality of Life Research, 19, 483-487.

56.

Resnicow

McCarty

Blissett

Wang

Heitzler

Lee

R. E.

(2003). Validity of a modified CHAMPS physical activity questionnaire among African-Americans. Medicine and Science in Sports and Exercise, 35, 1537-1545.

57.

Rosal

M. C.

Carbone

E. T.

Goins

K. V.

(2003). Use of cognitive interviewing to adapt measurement instruments for low-literate Hispanics. Diabetes Educator, 29, 1006-1017.

58.

Rudkin

Markides

K. S.

(2001). Measuring the socioeconomic status of elderly people in health studies with special focus on minority elderly. Journal of Mental Health and Aging, 7, 53-66.

59.

Salant

Lauderdale

D. S.

(2003). Measuring culture: A critical review of acculturation and health in Asian immigrant populations. Social Science and Medicine, 57, 71-90.

60.

Santoyo-Olsson

Cabrera

Freyre

Grossman

Alvarez

Mathur

. . . Stewart

A. L.

(2011). An innovative multiphased strategy to recruit underserved adults into a randomized trial of a community-based diabetes risk reduction program. Gerontologist, 51(Suppl. 1), 82-93.

61.

Sechrest

(2005). Validity of measures is no simple matter. Health Services Research, 40, 1584-1604.

62.

Selman

Harding

Gysels

Speck

Higginson

I. J.

(2011). The measurement of spirituality in palliative care and the content of tools validated cross-culturally: A systematic review. Journal of Pain and Symptom Management, 41(4), 728-753.

63.

Sentell

T. L.

Ratcliff-Baird

(2003). Literacy and comprehension of Beck Depression Inventory response alternatives. Community Mental Health Journal, 39, 323-331.

64.

Shea

J. A.

Aguirre

A. C.

Sabatini

Weiner

Schaffer

Asch

D. A.

(2005). Developing an illustrated version of the Consumer Assessment of Health Plans (CAHPS). Joint Commission Journal on Quality and Patient Safety, 31, 32-42.

65.

Shea

J. A.

Guerra

C. E.

Weiner

Aguirre

A. C.

Ravenell

K. L.

Asch

D. A.

(2008). Adapting a patient satisfaction instrument for low literate and Spanish-speaking populations: Comparison of three formats. Patient Education and Counseling, 73, 132-140.

66.

Skinner

J. H.

(2001). Acculturation: Measures of ethnic accommodation to the dominant American culture. Journal of Mental Health and Aging, 7, 41-52.

67.

Smedley

B. D.

Stith

A. Y.

Nelson

A. R.

(Eds.). (2003). Unequal treatment: Confronting racial and ethnic disparities in health care. Washington, DC: The National Academies Press.

68.

Stahl

S. M.

Hahn

A. A.

(2006). The National Institute on Aging’s Resource Centers for Minority Aging Research. Contributions to measurement in research on ethnically and racially diverse populations. Medical Care, 44, 1-2.

69.

Stewart

A. L.

Mills

K. M.

King

A. C.

Haskell

W. L.

Gillis

Ritter

P. L.

(2001). CHAMPS physical activity questionnaire for older adults: Outcome for interventions. Medicine and Science in Sports and Exercise, 33, 1126-1141.

70.

Stewart

A. L.

Nápoles-Springer

(2000). Health-related quality of life assessments in diverse population groups in the United States. Medical Care, 38, II102-124.

71.

Stewart

A. L.

Nápoles-Springer

A. M.

(2003). Advancing health disparities research: Can we afford to ignore measurement issues? Medical Care, 41, 1207-1220.

72.

Stewart

A. L.

Nápoles-Springer

A. M.

Gregorich

S. E.

Santoyo-Olsson

(2007). Interpersonal processes of care survey: Patient-reported measures for diverse groups. Health Services Research, 42, 1235-1256.

73.

Stewart

A. L.

Ware

J. E.

Jr. (Eds.). (1992). Measuring functioning and well-being: The medical outcomes study approach. Durham, NC: Duke University Press.

74.

Stone

A. A.

Turkkan

J. S.

Bachrach

C. A.

Jobe

J. B.

Kurtzman

H. S.

Cain

V. S.

(2000). The science of self-report: Implications for research and practice. Mahwah, NJ: Lawrence Erlbaum.

75.

Taira

D. A.

Safran

D. G.

Seto

T. B.

Rogers

W. H.

Kosinski

Ware

J. E.

. . . Tarlov

A. R.

(1997). Asian-American patient ratings of physician primary care performance. Journal of General Internal Medicine, 12, 237-242.

76.

Teng

E. L.

Chui

H. C.

(1987). The Modified Mini-Mental State (3MS) examination. Journal of Clinical Psychiatry, 48, 314-318.

77.

Teresi

J. A.

Stewart

A. L.

Morales

Stahl

(2006). Measurement in a multi-ethnic society: Overview to the special issue. Medical Care, 44(Suppl. 3), 2-3.

78.

Thiamwong

Stewart

A. L.

Warahut

(2009). Development, reliability and validity of the Thai Healthy Aging Survey. Walailak Journal of Science and Technology, 6, 167-188.

79.

Tobin

M. J.

(2000). Authors, authors, authors—follow instructions or expect delay. American Journal of Respiratory and Critical Care Medicine, 162, 1193-1194.

80.

Tucker

K. L.

Bianchi

L. A.

Maras

Bermudez

O. I.

(1998). Adaptation of a food frequency questionnaire to assess diets of Puerto Rican and non-Hispanic adults. American Journal of Epidemiology, 148, 507-518.

81.

U.S. Department of Health and Human Services. (2000). Healthy people 2010 (Conference Edition, in Two Volumes). Washington, DC: Author.

82.

Vanderplas

J. H.

Vanderplas

J. M.

(1981). Effects of legibility on verbal test performance of older adults. Perceptual and Motor Skills, 53, 183-186.

83.

Vogt

D. S.

King

D. W.

King

L. A.

(2004). Focus groups in psychological assessment: Enhancing content validity by consulting members of the target population. Psychological Assessment, 16, 231-243.

84.

Walter

M. J.

Castro

Kunselman

S. J.

Chinchilli

V. M.

Reno

Ramkumar

T. P.

. . . Wechsler

M. E.

(2008). Predicting worsening asthma control following the common cold. European Respiratory Journal, 32, 1548-1554.

85.

Ware

J. E.

Jr. (2000). SF-36 health survey update. Spine, 25, 3130-3139.

86.

Warnecke

R. B.

Johnson

T. P.

Chaavez

Sudman

O’Rourke

D. P.

Lacey

Horm

(1997). Improving question wording in surveys of culturally diverse populations. Annals of Epidemiology, 7, 334-342.

87.

Weidmer-Ocampo

Johansson

Dalpoas

Wharton

Darby

Hays

R. D.

(2009). Adapting CAHPS for an American Indian population. Journal of Health Care for the Poor and Underserved, 20, 695-712.

88.

Weiner

Aguirre

Ravenell

Kovath

McDevit

Murphy

. . . Shea

J. A.

(2004). Designing an illustrated patient satisfaction instrument for low-literacy populations. The American Journal of Managed Care, 10, 853-860.

89.

Willis

G. B.

(2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage.

90.

Wong

S. T.

Yoo

G. J.

Stewart

A. L.

(2005). Examining the types of social support and the actual sources of support in older Chinese and Korean immigrants. International Journal of Aging and Human Development, 61, 105-121.

91.

Wong

S. T.

Yoo

G. J.

Stewart

A. L.

(2007). An empirical evaluation of social support and psychological well-being in older Chinese and Korean immigrants. Ethnicity & Health, 12, 43-67.