Abstract
HIV prevention programs targeting men who have sex with men, Blacks, and young adults commonly use measures of HIV knowledge as an important component of demonstrating overall program effectiveness. These scales, however, are rarely subjected to repeated analysis to confirm reliability and validity and the results of psychometric analysis rarely include subpopulation variations. In this study, we administered an adapted version of a previously validated HIV knowledge scale to participants of a large, city-wide HIV prevention program (n = 5,027) and performed psychometric analysis to determine if differences existed across populations. Analysis showed that the HIV knowledge scale performed poorly for men who have sex with men, but very well for transgenders. Results were similar for Blacks, Hispanics, and Whites, very poor for 30- to 39-year-olds, but very well for 60+ year olds. Findings underscore the need for further research on the measurement of HIV knowledge among high-risk populations and the importance of culturally appropriate survey items tailored to each population.
Keywords
Introduction
Burden of HIV in Priority Populations
The Centers for Disease Control and Prevention (CDC) estimates that approximately 1.2 million people in the United States are living with HIV infection and nearly 20% of them are unaware that they are infected (CDC, 2011a). The latest data available demonstrate that from 2006 to 2009, the rates of annual diagnoses of HIV infection remained fairly stable (CDC, 2011b), in spite of increased federal funding for HIV/AIDS (Kaiser Family Foundation, 2009).
Men who have sex with men (MSM) continue to be the population at highest risk for HIV infection. Although MSMs only represent approximately 2% of the U.S. population (CDC, 2011a), they accounted for nearly 57% of all infections in 2009 (CDC, 2011b). In addition, between 2006 and 2009, the rate of HIV infection of MSMs has increased by 14.22% (CDC, 2011b).
Even though most HIV infections in the United States occur in males, women are also at risk. The rate of new infections among Black women was 15 times that of White women and the rate for Hispanic/Latina women was more than 3 times the rate of White women (CDC, 2011c). Of particular concern is the death rate for HIV infection among women. From 2000 to 2007, HIV infection was among the top 10 leading causes of death among Black females aged 10 to 54 years and Hispanic/Latina females aged 15 to 54 years (CDC, 2011c).
In 2009, the largest percentage (15% of all diagnoses) and the highest rate (36.9 per 100,000 population) for new HIV infections were among people aged 20 to 24 years and these numbers have increased since 2006 (CDC, 2011b). New HIV infection rates among 15- to 19-year-olds and 55- to 59-year-olds have also increased from 2006 to 2009 (CDC, 2011b), highlighting the need for more targeted intervention programs at lower and higher ends of the age distribution.
Finally, a growing body of evidence suggests that transgender populations are at extremely high risk for HIV infection. Currently, the CDC does not collect information on the HIV incidence and prevalence among transgenders, so the actual number of HIV-infected transgenders is unknown; however, a recent study published in 2008 has shed light on this population. A systematic review of published studies between 1990 and 2003 conducted by Herbst et al. (2008) documented high rates of HIV infection among male-to-female (MTF) transgenders. Meta-analysis of the data revealed that 27.7% of MTFs tested positive for HIV infection and 11.8% of MTFs self-reported being HIV positive and these rates were higher for Black MTFs.
Measures of HIV Knowledge
A PubMed search of the scientific literature returns thousands of articles reporting various aspects of HIV prevention program evaluations, including increases in HIV knowledge documented by self-administered paper-and-pencil survey items. However, few report the psychometric properties of survey items used to measure HIV knowledge and even fewer use scales that have been subjected to repeated measurement to confirm factor structure and generalizability. We did identify one scale, however, that has been subjected to multiple administrations to confirm reliability and validity.
In 1997, Carey, Morrison-Beedy, and Johnson published a 45-item survey to measure HIV knowledge based on 10 substudies involving a literature review, expert input, focus groups with low-income women, item and factor analysis, internal consistency and test–retest reliability, several validity analyses, and reading level analysis, and time completion analysis. In 2002, Carey and Schroder reduced the 45-item survey to 18 items and confirmed that the reduced scale exhibited strong internal consistency (α =.75-.89), test–retest reliability (r = .76-.94), and association with the longer, previously validated instrument (r = .93-.97). The analysis, however, was reported in the aggregate, thus masking potential variations in the psychometric properties of the HIV knowledge scale across subpopulations.
Purpose of the Study
The purpose of this research is to determine if the psychometric properties of a measure of HIV knowledge differ by gender, sexual orientation, race/ethnicity, and age.
Method
Source of Data
The data used in this research were extracted from a larger data set containing survey responses for an evaluation of a city-wide, HIV prevention program. The prevention program reached multiple high-risk populations in a large, urban Midwestern city from 2008 to 2011. Because the purpose of the research was to better understand HIV knowledge among high-risk populations without the influence of intervention, the data that were extracted for this analysis only contained pretest responses. Finally, because of data considerations for the specific analyses being performed, only complete responses (no missing values) were extracted. Each person participating in the evaluation completed a pencil-and-paper intake form. Responses were anonymous to the evaluators. The form was translated from English into Spanish for those who preferred this option. The study was approved by the Kent State University Institutional Review Board.
Measures
The measures used in this analysis include race, ethnicity, gender, sexual orientation, age, and HIV knowledge.
Race and ethnicity
Respondents were presented nine racial categories from which to select one: Black/African American, Asian, American Indian/Alaskan Native, Native Hawaiian/Other Pacific Islander, Asian and White, Black and White, American Indian/Alaskan Native and White, and Other Multiracial. A separate question asked respondents to indicate if they were of Hispanic or Latin descent. Because of very low cell sizes in many of the racial categories and because the purpose of the research was to examine high-risk populations (and not all populations), only three racial/ethnic categories were used in the analysis (Black, Hispanic, and White).
Gender and sexual orientation
Respondents were asked to select one category from a list that included male, female, transgender male-to-female, and transgender female-to-male. Sexual orientation was determined by comparing the response to the gender item with another question asking respondents if they had sex with men, women, both, or have never had sex. The gender item was condensed into Male and Female, and a new Sexual Orientation variable was created to include the categories of Lesbian, Gay, Bisexual, and Transgender.
Age
Rather than ask respondents to enter their exact age, they were asked to select one of several age categories: ≤18, 19 to 24, 25 to 29, 30 to 39, 40 to 49, 50 to 59, and ≥60 years old. The response options correspond to the age categories used in HIV surveillance reports issued by the CDC and by our state and local health departments.
HIV knowledge
HIV knowledge was measured by a set of 10 items drawn from the HIV Knowledge Questionnaire (Carey & Schroder, 2002). We reduced the items from 18 to 10 because of concerns of respondent burden. Retained items included statements about transmission from oral, anal, and vaginal sex; myths and stereotypes; condom efficacy; and vaccine availability. After reading each statement, respondents could select “True,” “False,” or “I Don’t Know.” Correct responses to each item had a value of 1, incorrect responses had a value of 0, and responses of I Don’t Know had a value of 0. Seven of the statements presented to respondents were correct and three were incorrect, requiring recoding before analysis.
Analysis Procedures
Since the data used in this study were dichotomous (correct/incorrect), the statistics used must be capable of handling categorical data. Thus, the two statistical procedures used to analyze the psychometric properties of the HIV knowledge scale were Cronbach’s alpha (Cronbach, 1951) and categorical principal components analysis (CATPCA; Linting, Meulman, Groenen, & van der Koojj, 2007). All statistical procedures were performed with SPSS v19 for Mac (Statistical Package for the Social Sciences, 2010) using the base software and the Categories add-on module.
Cronbach’s alpha
When performed on dichotomous data, Cronbach’s alpha is identical to the Kuder–Richardson formula 20, which is designed for dichotomous data (Nunnally & Bernstein, 1994). The resulting alpha coefficient is an upper limit estimate of the scale’s internal consistency reliability, which reflects a scale’s ability to produce similar results in repeated administrations over time (Nunnally & Bernstein, 1994). Cronbach’s alpha values near .70 to .90 indicate moderate to high internal consistency reliability (Nunnally & Bernstein, 1994). Values lower than .60 are generally regarded in the social science fields as undesirable.
Categorical principal component analysis
Collectively, as a scale, the 10 items we used to test respondents’ knowledge about HIV should measure one construct: HIV knowledge. To test if the HIV knowledge scale in fact does measure just one construct, we used a factor analytic technique called principal components analysis (PCA). PCA is a common tool used in survey research to statistically determine if a scale is measuring more than one phenomenon (called components). However, because the responses in the HIV knowledge scale are dichotomous (correct/incorrect) and not continuous, we used a special type of PCA developed by Linting et al. (2007) called categorical principal components analysis (CATPCA).
In our CATPCA analysis, we gave all variables the same weight so that each item could contribute equally in the model. We also used the nominal scaling method because the data were not ordinal. Because we were interested in obtaining the correlations between variables and principal components (component loadings), we used the variable principal normalization method, which is conceptually equivalent to the orthogonal (VARIMAX) rotation in traditional PCA. The use of this normalization method also eases the interpretation of component loadings. Missing values were excluded from analysis. Based on the overall component loading matrices, the variance accounted for (VAF) for each high-risk population, and for theoretical reasons, we considered component loadings less than .399 to be questionable (see Nunnally & Bernstein, 1994 and Stevens, 1986, for a more extensive discussion of where to establish cutoffs).
Results
Study Sample
As shown in Table 1, there were a total of 5,027 persons included in the analysis. Fifty-eight percent of the data were collected during the 2008-2009 grant year, 21.5% in 2009-2010, and 20.6% in 2011-2012. Slightly more than half of the sample (54.7%) were female and the majority (71.6%) were heterosexual. Among LGBT respondents, there were roughly equal numbers of lesbian, gay, and bisexual persons (24.6%, 31.5%, and 38.4%, respectively) and only 5.5% were transgender. By race/ethnicity, 66.8% were Black, 13.1% were Hispanic, 15.6% were White, and 4.6% were other race/ethnicities or missing. The age distribution of the sample skewed positively toward younger age-groups. Persons 18 years old or younger constituted 34.1% of the sample, 22.5% were 19 to 24 years old, 10.5% were 25 to 29 years old, 10.4% were 30 to 39 years old, 8.9% were 40 to 49 years old, 5.7% were 50 to 59 years old, 2.6% were 60 years or older, and 5.4% did not provide an age.
Demographic Characteristics of the Sample (n = 5,027)
Internal Consistency
Cronbach’s alpha was used to measure the scale’s internal consistency reliability, which is a measure of the scale’s statistical ability to produce consistent results on repeated uses. To examine the degree to which the HIV knowledge scale could produce differing results for each high-risk population, Cronbach’s alpha coefficients for the entire HIV knowledge scale were obtained for each group. Additionally, to identify which items perform poorly for each high-risk population, analyses were performed to obtain the Cronbach’s alpha if each item were deleted (αdel).
As noted on Table 2, the overall Cronbach’s alpha for all high-risk populations was .762. The alpha for males (.753) was slightly lower than females (.766). By sexual orientation (Table 3), the scale was most internally consistent for transgender (α = .860) and least for lesbians (α = .603). The alpha for all heterosexuals was .720 and the alpha for all LGBT respondents was slightly lower (.699). The alphas for gay and bisexual respondents were .698 and .693, respectively. Among the three racial/ethnic categories used in the analysis (Table 2), the HIV knowledge scale was most internally consistent for Hispanics (α = .808) and slightly less for Whites (α = .759) and Blacks (α = .745). Last, among all age-groups (Table 4), the HIV knowledge scale was most internally consistent for respondents 60 years and older (α = .892) and least for 30- to 39-year-olds (α =.524).
Cronbach’s Alpha, Alpha if Item Deleted, and Factor Loadings for All Respondents and by Gender and Race/Ethnicity
NOTE: αdel denotes Cronbach’s alpha (α) if item deleted. am denotes component loading for one dimension using categorical principal components analysis (CATPCA).
Cronbach’s Alpha, Alpha if Item Deleted, and Factor Loadings by Sexual Orientation
NOTE: αdel denotes Cronbach’s alpha (α) if item deleted. am denotes component loading for one dimension using categorical principal components analysis (CATPCA).
Cronbach’s Alpha, Alpha if Item Deleted, and Factor Loadings by Selected Age-Groups
NOTE: αdel denotes Cronbach’s alpha (α) if item deleted. am denotes component loading for one dimension using categorical principal components analysis (CATPCA).
Tables 2-4 show Cronbach’s alphas if individual items were deleted from the scale. If Item 1 were removed from the HIV knowledge scale, it would increase internal consistency for LGBT (αdel = .700 vs. α = .699), Lesbians (αdel = .612 vs. α = .603), and bisexuals (αdel = .697 vs. α = .693), but only by small margins (.001, .009, and .004, respectively). If Item 4 were removed, it would slightly increase internal consistency for 30- to 39-year-olds (αdel = .526 vs. α = .524). Finally, if Item 5 were removed, the scale would slightly increase internal consistency for 50- to 59-year-olds (αdel = .745 vs. α = .742).
Unidimensionality
To measure the unidimensionality of the HIV knowledge scale, we used a special form of PCA, called categorical PCA. To examine the degree to which unidimensionality differed across each high-risk population, we performed CATPCA on the HIV knowledge scale for each group. The resulting analyses produced a statistic indicating the overall percentage of variation accounted for by the scale (VAF) and component loadings for each item (am). Before conducting CATPCA, frequencies were run on all items for each high-risk population to identify any potential ceiling effects that could compromise the analysis. No items were answered correctly more than 95% of the time by any subpopulation, alleviating concerns for potential ceiling effects in further analysis (Clark & Watson, 1995).
As noted in Table 2, the overall VAF for all high-risk populations was 34.00. The VAF was slightly higher for males than females (33.80 and 33.64, respectively). By sexual orientation (Table 3), the VAF was highest for transgender (VAF = 39.05) and lowest for lesbians (VAF = 30.44). The VAF for heterosexuals was 30.70 and the VAF for all LGBT respondents was higher (VAF = 34.60). The VAF for gay and bisexual respondents was 36.31 and 32.08, respectively. Among the three racial/ethnic categories used in the analysis (Table 2), the VAF was highest for Whites (VAF = 37.13), slightly lower for Hispanics (VAF = 37.06), and lowest for Blacks (VAF = 32.43). Last, among all age-groups (Table 4), the VAF, in descending order, was 49.59 for 50- to 59-year-olds, 48.37 for 60+-year-olds, 34.24 for ≤18-year-olds, 30.69 for 30- to 39-year-olds, and 30.09 for 19- to 24-year-olds.
Also noted in Tables 2-4 are the component loadings (am) for each item in the HIV knowledge scale. In all but two high-risk populations (Whites and Hispanics), Item 5 had component loadings below the established cutoff of .399. In fact, the lowest component loadings for Item 5 occurred among several of the LGBT populations. The lowest was among transgender respondents, where Item 5’s component loading was slightly negative (am = −.017). There was only one other occurrence of component loading below the established cutoff. Item 7 for 30- to 39-year-olds had a component loading of .375.
Discussion
Overall, the internal consistency for our adapted 10-item HIV knowledge scale was .762, which is comparable to findings from Carey and Schroder (2002) who found alphas between .75 and .89 with their 18-item scale. In addition, our analysis for unidimensionality also confirmed the presence of one dimension, even with the relatively poorer performance of Item 5, which Carey and Schroder also found. But, when the psychometric analysis is performed on individual subpopulations, considerable variability is revealed.
The disparity in HIV infection is highest for MSMs, underscoring the importance of prevention programs targeting this population, including appropriate measurements for research and program evaluation. Our analysis revealed that the internal consistency of the HIV knowledge scale for MSMs was .698, which is slightly below the generally accepted cutoff of .7 and below the overall alpha of .762 for all populations. However, as one unidimensional construct, the percentage variance explained (VAF) by the HIV knowledge scale for MSMs was higher than for the group overall (36.31 and 34.00, respectively), in spite of the poor component loading on Item 5. These inconsistent findings are evidence of instability in the HIV knowledge scale for this population, indicating the need for more precise measurement construction for this population.
Although not yet confirmed by systematic epidemiologic surveillance, a growing body of evidence is demonstrating alarmingly high rates of HIV infection among people who identify as transgender, particularly MTF transgenders. Interestingly, the internal consistency of the HIV knowledge scale was very high for transgenders (α = .860), indicating that this scale could be a useful tool for measuring HIV knowledge in this population. Readers should be cautioned, however, that the relatively low number of transgenders in the sample could compromise the generalizability of the findings.
Among all racial/ethnic groups, African Americans continue to experience disproportionate rates of HIV infection. Our analysis did reveal a difference in internal consistency and unidimensionality by racial/ethnic category but the difference was minimal. These findings suggest that the HIV knowledge scale does not differ significantly based on race/ethnicity.
As mentioned above, the largest percentage and highest rate of new HIV infections occur among people aged 20 to 24 years and the numbers have risen over the past few years (CDC, 2011b). Persons aged 15 to 19 years and 55 to 59 years have also seen increases in HIV infection in recent years (CDC, 2011b). Our analysis by age-group revealed that the HIV knowledge scale had unacceptably low internal consistency for 30- to 39-year-olds (α = .524), marginally low internal consistency for 19- to 24-year-olds (α = .696), and low levels of VAF (30.69 and 30.09, respectively). Based on these findings, we assert that more work is needed to develop a better scale to measure HIV knowledge in younger age-groups, particularly among 19- to 39-year-olds.
It was interesting to note that internal consistency and unidimensionality metrics for respondents 60 years and older were among the highest observed (α = .892 and VAF = 48.37). In follow-up analysis (not described here), we determined these respondents were proportionately distributed across sexual orientation and race/ethnicity categories, ruling out the possibility that these high scores were due to clustering in other demographic categories with high internal consistency and VAF (i.e., transgendered Hispanics). We also determined that these respondents were scattered across most of the organizations participating in the city-wide prevention program, which significantly reduces the possibility that selection bias accounted for these high scores. For these reasons, we believe that the scale used in this research is a useful tool for measuring HIV knowledge in this population.
Finally, it should also be noted that the items used to measure HIV knowledge used terms that are more applicable to heterosexual populations. Statements about sexual transmission used terms associated with opposite sex partners (e.g., risk resulting from man inserting penis into a woman’s vagina), which do not measure coupling and sexual behaviors of same-sex partners (e.g., risk resulting from man inserting penis into another man’s anus). This is also a likely contributor to the low internal consistency and VAF of LGBT respondents, particularly lesbians, gays, and bisexuals.
Study Limitations
Although this research was conducted on a relatively large sample size (n = 5,027), it was focused exclusively in one urban geographic area in the Midwest. In addition, the data used in this analysis were extracted from a dataset containing program evaluation data for a city-wide HIV prevention program and not a stratified random sample of residents representative of all the populations under study. Readers should consider these factors when interpreting results.
Conclusions
Increasing knowledge about HIV transmission is an important component of effective HIV prevention programs, thus precise measurement of HIV knowledge is necessary to document these program gains. Using one instrument to measure HIV knowledge in all high-risk populations, however, may not be as effective. As we presented here, items commonly used to assess HIV knowledge can demonstrate acceptable levels of internal consistency and unidimensionality as a whole, but can mask important variations across subpopulations, particularly those at highest risk for HIV infection.
Footnotes
Authors’ Note:
This project was partially funded by the City Cleveland Department of Public Health Community Development Block Grant.