Factor Structure and Administration Measurement Invariance of the Beliefs Toward Mental Illness Scale in Latino College Samples: Paper

Abstract

The psychometric properties of the paper–pencil and online versions of the Beliefs Toward Mental Illness Scale (BTMI) were examined in two studies with Latina/o individuals. In Study 1, 316 Latina/o participants completed the BTMI in a paper–pencil mode. The original three-factor model was found to be a poor fit model for the sample. Subsequent exploratory and confirmatory factor analyses identified a four-factor model as the best fitting model for the sample. The identified factors were Dangerousness, Social Dysfunction, Incurability, and Embarrassment. In Study 2, the identified best fit model was tested with 280 Latina/o participants who completed the BTMI online. The four-factor model had adequate fit. A series of measurement invariance tests on the fit model supported equal factor loadings, but rejected equivalent intercepts across paper–pencil and online administration methods, though partially equivalent intercepts and residuals were found. Consequently, modality-specific norms are recommended, depending on whether paper–pencil or online venues are utilized for administration.

Keywords

negative beliefs factor structure Latinos online measurement invariance

Recent studies have suggested that prevalence rates of psychological disorders among Latinas/os are comparable or higher than rates for White or other ethnic minority groups (e.g., Asnaani, Richey, Dimaite, Hinton, & Hofmann, 2010; Hernandez, Plant, Sachs-Ericsson, & Joiner, 2005; Woodward et al., 2012). Thus, the promotion of psychological well-being in Latinas/os is particularly important. Yet a growing body of research suggests that Latino’s negative beliefs about psychological disorders predict unwillingness to seek mental health services (e.g., Hirai, Vernon, Popan, & Clum, 2015; Nadeem et al., 2007; Rojas-Vilches, Negy, & Reig-Ferrer, 2011). Two studies to date (Hirai et al., 2015; Rojas-Vilches et al., 2011) investigated negative beliefs toward psychological disorders in Latinas/os by administering the Beliefs Toward Mental Illness Scale (BTMI; Hirai & Clum, 2000). Rojas-Vilches et al. (2011) reported correlations between negative beliefs about psychological disorders on the BTMI and less positive attitudes toward conventional psychological treatment options among Puerto Rican and Cuban American young adults and their parents. A more recent study by Hirai et al. (2015) found that Mexican American undergraduates expressing more negative attitudes toward psychological disorders on the BTMI had a stronger preference for no psychological treatment. These results suggest that understanding negative beliefs toward psychological disorders among Latinos/as may be a necessary step toward the goal of promoting their psychological well-being.

The BTMI has been gaining attention among theorists and researchers interested in measuring stigma toward psychological disorders (e.g., Mey et al., 2014; Stone & Finlay, 2008). In the United States, the BTMI has been used to assess stigma toward psychological disorders among a predominantly White student sample (Segal, Coolidge, Mincic, & O’Riley, 2005), in White community samples (Pepin et al., 2015; Royal & Thompson, 2013; Segal et al., 2005), and among Latinas/os (Hirai et al., 2015; Rojas-Vilches et al., 2011). The BTMI has also been used by researchers in Turkey (e.g., Yildirim, Demirbuken, Balci, & Yurdalan, 2015), Portugal (de Jesus Loureiro, Dias, & Aragão, 2008), and India (Yadav, Arya, Kataria, & Balhara, 2012). The growing use of the BTMI among a range of populations including Latinas/os illustrates the importance of assessment of stigma toward psychological disorders.

The original BTMI (Hirai & Clum, 2000) was developed and validated separately among Asian international student and predominantly White American student samples. The factor structure was identified by separate exploratory factor analyses (EFAs) for each ethnic group. Hirai and Clum (2000) found, with both groups, three BTMI factors assessing three aspects of stigma toward psychological disorders: Dangerousness, Poor Social Skills, and Incurability. A recent study by Royal and Thompson (2013), however, concluded that a single-factor model best described the BTMI in a White American sample from a church community. These results raise the question of which factor structure best defines the construct measured by the BTMI, particularly when utilized to measure attitudes toward mental illness in Latinas/os.

It is also important to determine whether negative beliefs about psychological disorders are differentially influenced by responding in different assessment modalities. Evidence exists that unproctored, online surveys may elicit more endorsement on sensitive items when compared with paper–pencil venues (e.g., Kays, Gathercoal, & Buhrow, 2012; Wood, Nosko, Desmarais, Ross, & Irvine, 2006). Online assessments provide private environments for the respondent, which may facilitate more honest and accurate reporting about private matters, and also likely have less missing data than paper–pencil surveys (e.g., Kays et al., 2012; Wood et al., 2006). Given the increasing popularity of online assessment and research and the benefits noted above, it is important to establish the psychometric properties of assessment instruments using this medium (e.g., Buchanan, 2003).

The present study sought to determine the factor structure of the BTMI for Latina/o individuals in two different assessment modalities. Specifically, by applying a series of EFAs and confirmatory factor analyses (CFAs), the original three-factor model and empirically informed models of the BTMI were tested using both paper–pencil (Study 1) and online (Study 2) assessment venues. It was hypothesized that the original three-factor model would be identified as the best fitting model for the paper–pencil administration and for the online administration. It was also hypothesized that measurement invariance would be found between the two administration modalities.

Study 1

The aim of this study was to test the original three-factor model and the recently identified one-factor model of the BTMI for a paper–pencil administration with a Latina/o sample.

Method

Participants

At a state university in the border region of Texas, 316 Latino American undergraduate students, 70 males and 246 females, were recruited from various psychology courses and participated in the study. The mean age was 23.3 years (SD = 6.11) ranging from 18 to 48 years for males and 23.2 years (SD = 6.19) ranging from 18 to 57 years for females. All participants were English–Spanish bilingual, with 51% endorsing English as their primary language. The majority, 95.3%, of participants, identified themselves as a Latina/o with Mexican heritage and the remaining individuals as a Latina/o with Central or South American heritage.

Measures

All measures were administered in English.

A demographic questionnaire asked participants’ age, sex, ethnicity, ancestral descent, primary language, and level of education. Age was measured in an open-ended format and the remaining items were measured in a closed-ended response format.

Beliefs Toward Mental Illness (Hirai & Clum, 2000)

The BTMI was utilized as a measure of stigma associated with mental illness. The BTMI consists of 21 items that assess negative stereotypical views of psychological disorders, such as viewing mentally ill people as dangerous (e.g., “A mentally ill person is more likely to harm others than a normal person”) and socially dysfunctional (e.g., “It might be difficult for mentally ill people to follow social rules such as being punctual or keeping promises”), and viewing mental illness as incurable (e.g., “Psychological disorders are recurrent”). The original selection of items was atheoretical, but culled from a variety of studies on this topic (e.g., Fabrega, 1991; Gaw, 1993; Johnson & Orrell, 1995; Ng, 1997; Raguram, Weiss, Channabasavanna, & Devins, 1996). The items are rated on a 6-point Likert-type scale ranging from 0 (completely disagree) to 5(completely agree). Higher scores reflect more stigma toward psychological disorders. The original study (Hirai & Clum, 2000) reported moderate to good alphas for three factors (.74 < α < .85).

Procedure

The current study was part of a larger assessment study of attitudes toward psychological disorders that attempted to obtain psychometric properties of English and Spanish versions of the BTMI. The current study used data from participants completing the English language BTMI before the Spanish language BTMI. The participants were invited to a large lab room, signed the consent form, and completed the battery of questionnaires in paper–pencil format. Participants received research credit for a course as compensation.

Results

No missing data for the BTMI items were found.

Factor Structure

Analyses were performed utilizing Mplus 6.11 (Muthén & Muthén, 2007). A series of CFAs were performed to test the original three-factor model and the one-factor model. Maximum likelihood parameter estimates were used. For CFA, the current study utilized Hu and Bentler’s (1999) cutoff scores in assessing model fit: >.95 for comparative fit index (CFI), <.06 for root mean square error of approximation (RMSEA), and <.08 for standardized root mean square residual (SRMR). Results are presented in Table 1. Neither the three-factor model nor the one-factor model was a good fit for the current data, based on the fit indices.

Table 1.

CFA for the One-Factor Model and the Original Three-Factor Model for the Entire Sample and the Original Three-Factor Model and Newly Identified Four-Factor Model for a Random Half of the Sample.

Group	Model	χ²	df	p	CFI	RMSEA [90% CI]	SRMR	χ² difference test
Group	Model	χ²	df	p	CFI	RMSEA [90% CI]	SRMR	Δχ²	Δdf	p
Entire sample (n = 316)	One-factor model	1053.950	189	<.001	.687	.120 [.113, .127]	.093	—	—	—
	Three-factor model	621.923	186	<.001	.842	.086 [.079, .094]	.081	—	—	—
Random half of the sample (n = 158)	Three-factor model	534.97	186	<.001	.85	.052 [.074, .090]	.070
	Four-factor model	243.416	183	.002	.953	.046 [.029, .060]	.059	291.55	3	<.001
	With four freed covariates^a	231.256	179	.005	.959	.043 [.025, .058]	.058	12.160	4	<.05

Note. df = degrees of freedom; CFI = comparative fit index; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual.

Error variances of Item 1 and Item 6, Item 4 and Item 8, Item 9 and Item 10, and Item 16 and Item 20 covaried.

To identify a factor structure fitting the data, an EFA with Oblimin rotation was performed, utilizing Mplus 6.11 (Muthén & Muthén, 2007), on a randomly selected subset of the sample (n = 158). The random selection was conducted through SPSS Version 21.0 (IBM, 2012). One to five factors were extracted. Eigenvalues of the first four factors were 1.25 and larger, which were above the cutoff score of 1.0. A parallel analysis was also performed to further determine the number of factors, by using the parallel analysis engine (Patil, Singh, Mishra, & Donavan, 2007). The eigenvalues of the first three extracted factors were larger than the first three 95th percentile random data eigenvalues. The fourth extracted factor had an eigenvalue of 1.25, which was lower than the fourth 95th percentile random data eigenvalue of 1.47. The eigenvalue of the fifth extracted factor was 0.83, which was much lower than the fifth 95th percentile random data eigenvalue of 1.37.

Based on these results, both the three-factor model and the four-factor model were considered. The fit indices for the three-factor model were mediocre: χ²(150) = 303.93 (p < .01), CFI = .90, RMSEA = .081 [.068, .094]; SRMR = .05. The fit indices for the four-factor model were improved: χ²(132) = 185.40 (p < .05), CFI = .97, RMSEA = .05 [.032, .067]; SRMR = .03. The chi-square results of both models were significant, which was likely due to the sample size. No negative residual variances were found for either model. All of the four factor loadings were significant. The descriptive and model fit values only for the four-factor model largely met conditions and cutoff scores recommended by Muthén and Muthén (2009). Examination of the extracted factors of these models revealed that a factor of the three-factor solution was a combination of two distinct factors of the four-factor solution. Based on the above results, the four-factor model was selected as the best.

Factor loadings are presented in Table 2. These four factors were titled: (a) Dangerousness, (b) Social Dysfunction, (c) Incurability, and (d) Embarrassment. The first three factors retained the majority of their original items. Interfactor correlations were fairly high: .49 for Dangerousness and Social Dysfunction; .43 for Dangerousness and Incurability; .44 for Dangerousness and Embarrassment; .46 for Social Dysfunction and Incurability; .43 for Social Dysfunction and Embarrassment; and .33 for Incurability and Embarrassment.

Table 2.

Factor Structure.

Factors		Factor loadings
Dangerousness
1	A mentally ill person is more likely to harm others than a normal person.	.93	.02	−.01	−.05
3	It may be a good idea to stay away from people who have psychological disorders because their behavior is dangerous.	.39	.26	.09	.07
6	Mentally ill people are more likely to be criminals than nonmentally ill people.	.58	.01	.16	.10
13	I am afraid of people who are suffering from psychological disorders because they may harm me.	.67	.05	.02	.21
Social Dysfunction
5	A person with a psychological disorder should have a job with minor responsibilities.	.07	.69	−.03	−.02
11	It might be difficult for mentally ill people to follow social rules such as being punctual or keeping promises.	.14	.60	−.05	.03
14	A person with a psychological disorder is less likely to function well as a parent.	.00	.54	.13	.21
17	Mentally ill people are unlikely to be able to live by themselves because they are unable to assume responsibilities.	−.04	.75	.09	.03
18	Most people would not knowingly be friends with a mentally ill person.	.01	.54	.05	.15
19^a	The behavior of people who have psychological disorders is unpredictable.	.05	.81	.05	−.11
21	I would not trust the work of a mentally ill person assigned to my work team.	.00	.61	−.07	.24
Incurability
2^b	Mental disorders would require a much longer period of time to be cured than would other general diseases.	.15	−.06	.62	.04
7	Psychological disorders are recurrent.	.12	−.16	.59	.07
9	Individuals diagnosed as mentally ill will suffer from its symptoms throughout their life.	.00	.06	.80	.00
10	People who have once received psychological treatment are likely to need further treatment in the future.	−.10	−.03	.75	.15
16	I do not believe that psychological disorders are ever completely cured.	−.04	.09	.73	−.10
20	Psychological disorders are unlikely to be cured regardless of treatment.	.12	.14	.64	−.10
Embarrassment
4	The term “psychological disorder” makes me feel embarrassed.	.14	−.09	.07	.61
8	I am afraid of what my boss, friends, and others would think if I were diagnosed as having a psychological disorder.	.10	−.03	.04	.68
12	I would be embarrassed if people knew that I dated a person who once received psychological treatment.	.01	.17	.00	.68
15	I would be embarrassed if a person in my family became mentally ill.	−.02	.11	.01	.64

Note. Decimals in boldface are primary factor loadings.

The original study (Hirai & Clum, 2000) reported this item loaded on Incurability. ^bThe original study reported this item loaded on Dangerousness.

The remaining data (n = 158) were used to perform CFA to test the four-factor model in comparison with the three-factor model. Results are presented in Table 1. The four-factor model was an acceptable fit for the data based on the fit indices. The chi-square difference test comparing the three-factor model with the four-factor model demonstrated a significant improvement, supporting the four-factor model. Alpha reliability coefficients and 90% confidence intervals for the identified four factors are .84 [.81, .87] for Dangerousness, .86 [.84, .88] for Social Dysfunction, .85 [.83, .88] for Incurability, and .77 [.73, .81] for Embarrassment. All alphas are adequately high. Correlations among the factors ranged from .33 to .53 (ps < .01).

Discussion

The current study found that a refinement of the original factor structure of the BTMI was optimal in the current Latino sample. EFA results showed that the majority of the items loaded on the original three factors and were retained in the current factor structure. However, a fourth factor, formed with four Embarrassment items (originally loaded on the Poor Social Skills factor) emerged. EFA and CFA identified and confirmed four factors: Dangerousness, Social Dysfunction, Incurability, and Embarrassment. The three-factor model and one-factor model were a poor fit for the current sample. CFA results support the current four-factor model for the Latino group, as evidenced by fit indices better than the recommended cutoff values (Hu & Bentler, 1999). The significant chi-square difference test results further supported the conclusion. All of the factor loadings on the four factors of the identified model were significant. Alpha reliability coefficients of the factors in the present Latina/o sample ranged from .77 to .86.

Based on analyses with the current Latina/o samples, two items were moved from their originally loaded factor to another factor. Specifically, the current Incurability factor consists of five original items and an item describing the need for lengthy treatments that originally loaded on the Dangerousness factor. An item originally loaded on the Incurability factor describing unpredictable behavior among the mentally ill was moved to the current Social Dysfunction factor. The content of these moved items appears to better match their current factors as opposed to their original factors. Fit statistic values suggest that constraining cross-loadings to zero do not dramatically affect model fit. The newly identified Embarrassment factor consists of items that were originally loaded on the Poor Social Skills factor of the three-factor model. Three of the four items include the word “embarrassed” and the fourth item concerns what others would think of a psychological disorder diagnosis. These results suggest that for Latinas/os, the social dysfunction and role failure concerns expressed in the Social Dysfunction subscale items may be distinct from the potential for embarrassment due to stigma. This contrasts with other ethnic groups, including Asian and White Americans, who responded to these items in a similar manner, thus generating a factor that combines Social Dysfunction and Embarrassment items.

The discrepancy between the factor structures of the BTMI obtained by previous studies and that found in the current study with a Latina/o sample may be attributed to the different sample characteristics of these studies. A one-factor solution was found among White Americans from a church community (Royal & Thompson, 2013), a three-factor solution was found with a predominantly White American student sample (Hirai & Clum, 2000), and a three-factor solution was also found with an Asian international student sample (Hirai & Clum, 2000). These results suggest that cultural and religious values and beliefs influence individuals’ views and interpretations of statements concerning stigma toward psychological disorders. In addition, statistical methodologies for identifying a factor structure differed across the three studies. Hirai and Clum (2000) applied only EFA, whereas the current study employed EFA and CFA for separate subsamples. Royal and Thompson (2013) utilized an item response theory technique. Different analysis techniques might also have resulted in different factor structures.

Study 2

The aim of this study was to test the newly identified four-factor model of the BTMI (Study 1) in an online format in a Latino student sample. The BTMI’s measurement invariance between the paper–pencil administration in Study 1 and the online administration in Study 2 was examined. It was hypothesized that the four-factor model would provide a good fit for the online data. It was also hypothesized that strict invariance would be found between the paper–pencil and online administrations of the BTMI.

Method

Participants

Of 280 Latino American undergraduate students, 65 were males (23.5%), 212 females (75.7%), and 3 unidentified (1.1%). Participants were recruited from various psychology courses at a state university in the border region of Texas. The mean age was 23.0 years (SD = 5.37, range: 18-41) for males and 23.2 years (SD = 5.79, range: 18-47) for females. All participants were English–Spanish bilingual, with 57% endorsing English as their primary language. Most participants (95.0%) were Latino of Mexican descent.

Measures and Procedure

The same demographic questionnaire and the BTMI (Hirai & Clum, 2000) were used. Study 2 was the online version of Study 1. The platform of the survey was Qualtrics (Qualtrics, Provo, UT). The same set of assessment instruments was administered. Participants were invited to a large lab room to sign the consent form, and then were asked to complete the battery of questionnaires online, in a private setting (e.g., at home), within a designated 12-hour period. Participants received research credit for a course as compensation.

Analysis Plan

CFA was used to examine the four-factor model identified in Study 1. Multiple-group measurement invariance testing of the respecified model was performed to examine modality invariance between the paper–pencil and the online data.

Results

Participants were given a “prefer not to answer” option for all items, but no one selected this choice for any items, including the BTMI.

Factor Structure

A series of CFAs were performed utilizing Mplus 6.11 (Muthén & Muthén, 2007). Maximum likelihood parameter estimates were used. The one-factor, three-factor, and four-factor models were examined. Results are presented in Table 3. The four-factor model was a reasonably acceptable fit for the data based on the fit indices, meeting the requirements for RMSEA and SRMR, but obtaining a CFI of .93 when the cutoff score suggested by Hu and Bentler (1999) is >.95. Modification indices indicated that including free error covariances might improve the fit of the model; those covariances that made theoretical sense (e.g., among constructs that were conceptually related to one another) were added to the model (two sets of Incurability items [Items 9 and 10; Items 16 and 20], and two sets of Embarrassment items [Items 1 and 6; Items 4 and 8]) and the fit of the model was reevaluated. Chi-square difference tests were also performed to examine model improvement. Results are presented in Table 3. The fit indices for the model were improved and met all cutoff scores. The chi-square difference test results showed significant improvement toward the four-factor model with the freed covariates. Alpha reliability coefficients and 90% confidence intervals for the identified four factors are .84 [.80, .87] for Dangerousness, .84 [.80, .86] for Social Dysfunction, .83 [.80, .86] for Incurability, and .70 [.64, .76] for Embarrassment. Correlations among the factors ranged from .34 to .64 (ps < .01).

Table 3.

Online Study: CFA for All Models of the BTMI.

Group	Model	χ²	df	p	CFI	RMSEA [90% CI]	SRMR	χ² difference test
Group	Model	χ²	df	p	CFI	RMSEA [90% CI]	SRMR	Δχ²	Δdf	p
Entire sample (n = 280)	One-factor model	746.391	189	<.001	.764	.103 [.095, .110]	.078
	Three-factor model	534.969	186	<.001	.853	.082 [.074, .092]	.070	211.42	3	<.001
	Four-factor model	349.659	183	<.001	.930	.057 [.048, .066]	.051	185.31	3	<.001
	With four freed covariates^a	293.005	179	<.001	.952	.048 [.038, .057]	.048	56.65	4	<.001

Note. BTMI = Beliefs Toward Mental Illness Scale; df = degrees of freedom; CFI = comparative fit index; RMSEA = root mean square error of approximation; CI = confidence interval; SRMR = standardized root mean square residual.

Error variances of Item 1 and Item 6, Item 4 and Item 8, Item 9 and Item 10, and Item 16 and Item 20 covaried.

Measurement Invariance

CFA measurement invariance tests were performed and evaluated by the steps described in the recent literature (e.g., Meredith, 1993; Meredith & Teresi, 2006). Once the configure invariance was established, the paper–pencil administration group from Study 1 and the online administration group from Study 2 were compared across the following models: (a) equal factor loadings assumed (metric invariance); (b) in addition to the previous assumption, equal item intercepts assumed (scalar invariance); and (c) in addition to the previous assumptions, equal variances of the residuals assumed. A minimum of metric invariance is required for basic research, as this invariance confirms that the identified factor structure is applicable to different groups (Meredith & Teresi, 2006). To examine invariance, the current study examined chi-square differences.

First, the two modality groups were compared on demographic variables (e.g., age, education level, and primary language), no group differences were found.

The four-factor model with the four free error covariates was used to examine measurement invariance across the paper–pencil and online administrations. Before performing invariance tests, the four-factor model with the four free error covariates was examined for the paper–pencil data, revealing some slightly improved fit indices. Results are presented in Table 1.

Results of invariance tests are presented in Table 4. The fit indices demonstrated configural invariance for the two administration modalities. The next step examined metric invariance, comparing the configural model with the metric model. The chi-square difference between the models was close to significant (p = .09). Yet the significance level still suggests equal factor loadings on the four-factor model with four free error covariates across the two modalities. Because the model had four freed error covariates, the next step tested equal error covariances across the two modalities, comparing the metric model and the metric model with equal error covariances. The chi-square difference was not significant, suggesting equal error covariances across the two modalities. The next step examined scalar invariance, testing equal intercepts across the two modalities. The chi-square difference was significant, indicating unequal intercepts between the two groups. The modification indices suggested four items to be freed to attain partial scalar invariance. Specifically, constraints for one item from Incurability, two items from Social Dysfunction, and one item from Embarrassment were lifted. As a result, the chi-square difference became nonsignificant. The next step examined residual invariance across the two modalities. The chi-square difference was significant and the modification indices suggested two error variances, one for an item of Social Dysfunction and the other for an item of Embarrassment, to be lifted. Without constraints on the two error variances, the chi-square difference became nonsignificant.

Table 4.

Measurement Invariance Test for the Four-factor Model With Four Free Error Variances of the BTMI.

Invariance	χ²	df	CFI	RMSEA [90% CI]	SRMR	χ² difference test
Invariance	χ²	df	CFI	RMSEA [90% CI]	SRMR	Δχ²	Δdf	p
Configural^a	524.260	358	.954	.046 [.037, .054]	.051
Metric^a	549.499	375	.952	.046 [.037, .054]	.056	25.239	17	.09
Metric with equal error covariances^a	552.972	379	.952	.046 [.037, .054]	.056	3.473	4	.48
Scalar: Four freed intercepts^b	576.459	396	.95	.046 [.037, .054]	.058	23.487	17	.13
Residual: Two freed errors^c	601.849	415	.95	.045 [.037, .053]	.059	25.390	19	.15

Error variances of Item 1 and Item 6, Item 4 and Item 8, Item 9 and Item 10, and Item 16 and Item 20 covaried. ^bIntercepts of Item 5, Item 12, Item 18, and Item 20 were freed. ^cError variances of Item 4 and Item 18 freed.

Because the four intercepts were found variant across the two administration modalities, comparisons on these item scores between the modalities were performed. A series of t tests revealed that the paper–pencil administration group scored significantly higher on the Incurability item than the online administration group (paper–pencil mean = 1.96; online mean = 1.58; p < .01) and the online administration group scored marginally significantly higher on one of the Social Dysfunction items than the paper–pencil administration group (paper–pencil mean = 1.77; online mean = 2.03; p = .06). No significant group difference was found for the Embarrassment item (paper–pencil mean = 1.14; online mean = .99; p = ns) and the other Social Dysfunction item (paper–pencil mean = 1.92; online mean = 2.14; p = ns).

Discussion

This study tested the one-factor, three-factor, and four-factor models of the BTMI online with a Latina/o student sample. The four-factor model with four sets of freed error variances was found to be the best fitting model, as evidenced by fit indices better than the recommended cutoff values (Hu & Bentler, 1999). The significant chi-square difference test results further supported the conclusion. All of the factor loadings on the four factors of the model were significant and all factors had adequate alpha reliability coefficients, ranging from .70 to .84.

Similar to results from Study 1, this study rejected the one-factor and the original three-factor model. Freeing the four sets of error variances improved the four-factor model for the online data. In fact, these freed covariances improved the fit indices for the paper–pencil data as well. These convergent results support the assertion that among a Latina/o sample the BTMI can be best represented by four factors.

The hypothesis that strict invariance would be found between paper–pencil and online administrations of the BTMI was only partially supported. The measurement invariance tests provided evidence for metric invariance of the four-factor model of the BTMI across the two administrations. These results indicate that the items, when administered in different modalities, assess negative attitudes factors similarly. That is, the relationship between the observed items and the latent constructs (factors) are similar across the administration modalities, suggesting the identified model of the BTMI can assess the four factors similarly via paper–pencil and online. The measurement invariance tests failed to support full scalar invariance. Evidence for partial scalar invariance was attained when constraints on four intercepts were removed. Administration group comparisons on the four items did not show a specific pattern, suggesting differential effects of scores of these items on the factor scores between the administration modalities. Importantly, the lack of full scalar invariance suggests different item means across different administration modalities. That is, when the BTMI is administered paper–pencil and online, modality-specific norms may need to apply. The results of partial residual invariance suggest that score differences between the modalities may be attributed to results of the four factors as well as some uniqueness existing across the modalities.

Overall, given that the metric invariance was close to significant (p = .09) and limited evidence for scalar and residual invariance across the modalities, further invariance studies on the current factor structure are warranted.

Additional Analyses

Higher Order Structure

Although testing the original three-factor model and identifying a better fit model for Latinas/os was the primary purpose of the current studies, given the high interfactor correlations, a higher order model was explored. Based on the recent finding that supported a unidimensional model of the BTMI (Royal & Thompson, 2013) and the similar magnitudes of the interfactor correlation coefficients across the four factors, a single second-order factor model with the previously freed covariates was developed and examined. The fit indices for the paper–pencil group were χ²(181) = 235.01, p = .004; CFI = .96; RMSEA = .043 [.025, .058]; SRMR = .060. The fit indices for the online group were χ²(181) = 303.06, p < .001; CFI = .95; RMSEA = .049 [.039, .059]; SRMR = .049. The single second-order factor model was an acceptable fit for both data based on the fit indices. The second-order factor was labeled as General Negative Beliefs. Alpha reliability coefficients were .91 [.895, .924] for the paper–pencil group and .91 [.894, .924] for the online group.

Group Comparisons on the Factors

It may be informative to explore potential influences of administration modalities and sample characteristics on BTMI scores. All factor scores were compared between the two administration modality groups and second-order factor scores (i.e., total scores) were compared between the current groups and the samples from the original study (Hirai & Clum, 2000). Results are presented in Table 5. A series of t tests revealed that modality effects were found only for the Incurability factor of the four-factor model. The paper–pencil administration group yielded a significantly higher mean for the factor than the online group. Both the Asian international student sample and the predominantly White American student sample reported higher scores on the total factor outside the confidence intervals for the total scores for the current study.

Table 5.

Total and Factor Scores of the BTMI.

Factor	Hirai and Clum (2000)		Current study
	Asian	American^a	Hispanic (paper–pencil)				Hispanic (online)				t Test
	M (SD)	M (SD)	M (SD)	Lowest	Highest	95% CI	M (SD)	Lowest	Highest	95% CI	t Test
BTMI total (General Negative Beliefs)	56.1 (19.22)	43.1 (15.33)	39.0 (16.3)	0	77	[37.2, 40.8]	38.0 (16.86)	0	86	[36.0, 40.0]	ns
Three factors
Dangerousness	12.6 (5.04)	10.2 (4.53)	—	—	—	—	—	—	—	—	—
Poor Social Skills	24.6 (9.07)	16.7 (7.09)	—	—	—	—	—	—	—	—	—
Incurability	15.0 (5.76)	13.9 (5.50)	—	—	—	—	—	—	—	—	—
Four factors
Dangerousness	—	—	6.0 (4.35)	0	19	[5.5, 6.5]	6.1 (4.42)	0	17	[5.6, 6.7]	ns
Dysfunction	—	—	13.4 (6.83)	0	29	[12.6, 14.1]	13.3 (6.77)	0	32	[12.5, 14.1]	ns
Incurability	—	—	15.2 (5.93)	0	30	[14.5, 15.9]	14.0 (5.99)	0	30	[13.3, 14.7]	p < .05
Embarrassment	—	—	4.4 (3.88)	0	15	[4.0, 4.8]	4.6 (3.77)	0	18	[4.1, 5.0]	ns

Note. BTMI = Beliefs Toward Mental Illness Scale; CI = confidence interval.

Predominantly White American.

General Discussion

The current studies performed a series of EFA and CFAs to examine and compare several possible factor structures of the BTMI in Latina/o samples. Converging results from the two independent studies, one using the paper–pencil version of the BTMI and the other using the online version, supported a four-factor model in Latina/o student samples. Both the original three-factor model found with separate White student samples (Hirai & Clum, 2000) and the one-factor model found with a White church community sample (Royal & Thompson, 2013) were rejected in the current two studies. The discrepancies between the identified factor structures across the five studies may be attributed to different sample characteristics. Specifically, the negative beliefs described by the BTMI may be interpreted differently across White American, Asian, and Latina/o individuals, and college students relative to a church community, all of whom may have different cultural values, religious beliefs, and education. In addition, different analysis methods across studies (EFA, CFA, item response theory method) might have been another factor differentiating findings across the studies.

Three of the four factors identified in the present studies (Dangerousness, Social Dysfunction, and Incurability) essentially replicated the three factors found by previous research with different samples (Hirai & Clum, 2000). Yet there are several notable differences between the original one-factor model and the current four-factor model. Two items moved to load on a different factor and the content and factor loadings (and lack of cross-loadings) of the moved items seem to fit their new factor. The Embarrassment factor identified by the current Latino sample was part of the original Poor Social Skills factor found in the three-factor models of White student samples (Hirai & Clum, 2000) and the unidimensional model found with a White church community (Royal & Thompson, 2013). These findings are interesting, as the current results suggest that Latinos may differentiate the socially dysfunctional aspects and embarrassing aspects of stigma, whereas Asian and White Americans may respond to these aspects similarly. This result suggests the importance of considering the two aspects of stigma toward psychological disorders separately for Latinas/os.

Both of the current studies produced similar and adequate alphas for the four factors. Specifically, alphas for the Dangerousness, Social Dysfunction, and Incurability factors were above .80 and alphas for the Embarrassment factor were above .70. Relatively low alphas for the Embarrassment factor can be explained by the small number of items loaded on the factor (four items). Overall, these alphas suggest that the BTMI is a reasonably reliable measure for assessing negative beliefs toward psychological disorders. One interesting source of comparison for the BTMI is the Social Perceptions Questionnaire (SPQ; Stone & Finlay, 2008), which was developed in the United Kingdom to assess negative beliefs about schizophrenia. The SPQ consists of subscales of Negative Social Status, Dangerousness, Trustworthiness, Abnormality, and Curability. While the SPQ targets specifically stigma attached to schizophrenia, the BTMI addresses stigma toward psychological disorders in general. Both measures assess stigma addressing dangerousness and incurability of symptoms similarly. Yet each measure offers its unique domains of stigma, such as abnormality by the SPQ and embarrassment by the BTMI. These similarities and differences between the two self-report measures suggest the validity and incremental utility of the BTMI, particularly when assessing negative beliefs toward psychological disorders in general.

The measurement invariance test revealed a metric invariance between the paper–pencil and online administrations, which meets the minimum requirement for research (Meredith & Teresi, 2006) and suggests that the four-factor model is applicable to both modalities. Only a partial scalar invariance and a partial residual invariance were found between the two administration modalities. Differential effects of four items on the latent factors as well as residual uniqueness across the modalities were evident. The presence of metric invariance and lack of full scalar invariance and full residual invariance across the administration modalities suggest that the BTMI assesses the four factors similarly across the modalities, yet the instruments need modality-specific norms. This is important because the literature has suggested that unproctored, online surveys have less missing data than paper–pencil surveys and elicit more open reports on sensitive items (e.g., Kays et al., 2012; Wood et al., 2006).

Interestingly, the current results did not replicate these advantages of online administrations of questionnaires. Modality effects on missing data and mean scores were minimal to none. Given that participants in the current studies were well-educated, relatively older college students (mean age = 23 for both studies), and receiving class point compensation, it is possible that they may have been especially motivated to complete the questionnaires, producing no missing data. Education effects might also explain the lack of modality effects on scores. Specifically, the current participants may have less stigma toward psychological disorders from their advanced college education (the majority were upper division students), and thus may have been less influenced by social desirability concerns regardless of administration modality. Their low levels of stigma were evident, based on additional analyses revealing lower scores than those in the original study (Hirai & Clum, 2002), whose participants were relatively young college students (mean age = 19).

An attempt was made to examine a possible second-order factor structure. This model reasonably fit the current data. Because this was exploratory, measurement invariance was not sought. Yet the presence of the second-order single-factor structure in the current Latino samples and the unidimensional model of the BTMI identified by the White American community sample (Royal & Thomson, 2011) together suggest cross-cultural similarity of negative beliefs toward psychological disorders at the second-order level. BTMI total scores might offer cross-cultural comparisons on such stigma. Cross-cultural divergence in domains of stigma attached to psychological disorders appears to exist at the lower order level. Further research should examine the second-order structure in diverse samples to confirm this possibility.

Descriptive information about factor scores from the current studies is informative, revealing within-culture differences as well as within-individual differences in levels of specific types of stigma attached to psychological disorders. Specifically, some Latina/o participants reported elevated stigma scores, while others reported minimal to no stigma. Further correlations among factor scores indicate individuals likely have different levels of stigma for each of the four dimensions. Understanding both within-culture and within-individual differences in levels of stigma could facilitate the development of individualized strategies for diminishing negative attitudes toward psychological disorders. For such efforts, the BTMI is a useful instrument.

The current findings should be evaluated in light of the limitations of the two studies. Since participants were mostly upper division students recruited from psychology courses and thus may have been educated about psychological disorders, the findings may have limited generalizability to those less familiar with psychological disorders. Furthermore, because most participants were Latinas/os with Mexican cultural backgrounds, the findings may not be fully applicable to individuals with different cultural backgrounds including other subgroups of Latinas/os. Future studies should investigate the effects of culture, ethnicity, and education on the factor structure and measurement invariance of the BTMI using varying community and clinical populations.

The current study was the first to examine the factor structure of the paper–pencil and online versions of the BTMI with two large Latina/o samples (ns = 316 and 280). Given that Latinas/os have equal or higher prevalence rates of psychological disorders relative to their non-Latino counterparts (e.g., Asnaani et al., 2010; Hernandez et al., 2005; Woodward et al., 2012), improving our understanding of attitudes toward psychological disorders, which in turn affect Latinos’ willingness to seek treatment for psychological disorders (Hirai et al., 2015), is particularly important. Information gained through the use of a sound measure of such attitudes may have clinical implications. For example, approaches to reducing different types of negative beliefs might be designed to improve attitudes about psychological disorders, which would in turn likely facilitate treatment-seeking behaviors. The present study also provided evidence that the online version of the BTMI is a sound measure, which is useful given the benefits of online administration of questionnaires. In addition, given international utilization of the BTMI, the current findings have potential to further facilitate investigations in and prevention of stigma attached to psychological illnesses worldwide.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Asnaani

Richey

J. A.

Dimaite

Hinton

D. E.

Hofmann

S. G.

(2010). A cross-ethnic comparison of lifetime prevalence rates of anxiety disorders. Journal of Nervous and Mental Disease, 198, 551-555. doi:10.1097/NMD.0b013e3181ea169f

Buchanan

(2003). Internet-based questionnaire assessment: Appropriate use in clinical contexts. Cognitive Behaviour Therapy, 32, 100-109. doi:10.1080/16506070310000957

de Jesus Loureiro

L. M

Dias

C. A. A.

Aragão

R. O.

(2008). Crenças e Atitudes acerca das doenças e dos doentes mentais Contributos para o estudo das representações sociais da loucura [Beliefs and attitudes toward mental ill and illness Contributions for the study of social representations of madness]. Revista de Enfermagem Referência II^a Série, 2008(8), 33-44.

Fabrega

(1991). Psychiatric stigma in non-Western societies. Comprehensive Psychiatry, 32, 534-551. doi:10.1016/0010-440X(91)90033-9

Gaw

A. C.

(1993). Culture, ethnicity, and mental illness. Washington, DC: American Psychiatry Press.

Hernandez

Plant

E. A.

Sachs-Ericsson

Joiner

T. J.

(2005). Mental health among Hispanics and Caucasians: Risk and protective factors contributing to prevalence rates of psychiatric disorders. Journal of Anxiety Disorders, 19, 844-860. doi:10.1016/j.janxdis.2004.11.002

Hirai

Clum

G. A.

(2000). Development, reliability, and validity of the Beliefs toward Mental Illness Scale. Journal of Psychopathology and Behavioral Assessment, 22, 221-236. doi:10.1023/A:1007548432472

Hirai

Vernon

L. L.

Popan

J. R.

Clum

G. A.

(2015). Acculturation and enculturation, stigma toward psychological disorders, and treatment preferences in a Mexican American sample: The role of education in reducing stigma. Journal of Latina/o Psychology, 3, 88-102. doi:10.1037/lat0000035

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.

10.

IBM. (2012). IBM SPSS Statistics for Windows (Version 21.0). Armonk, NY: Author.

11.

Johnson

Orrell

(1995). Insight and psychosis: A social perspective. Psychological Medicine, 25, 515-520. doi:10.1017/S0033291700033432

12.

Kays

Gathercoal

Buhrow

(2012). Does survey format influence self-disclosure on sensitive question items? Computers in Human Behavior, 28, 251-256. doi:10.1016/j.chb.2011.09.007

13.

Meredith

(1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525-543.

14.

Meredith

Teresi

J. A.

(2006). An essay on measurement and factorial invariance. Medical Care, 44, S69-S77.

15.

Mey

Fowler

J. L.

Knox

Shum

D. K.

Fejzic

Hattingh

. . . Wheeler

(2014). Review of community pharmacy staff educational needs for supporting mental health consumers and carers. Community Mental Health Journal, 50, 59-67. doi:10.1007/s10597-012-9580-4

16.

Muthén

L. K.

Muthén

B. O.

(2007). Mplus user’s guide (6th ed.). Los Angeles, CA: Author.

17.

Muthén

L. K.

Muthén

B. O.

(2009). Mplus short courses. Topic 1: Exploratory factor analysis, confirmatory factor analysis, and structural equation modeling for continuous outcomes. Retrieved from https://www.statmodel.com/download/Topic%201.pdf

18.

Nadeem

Lange

J. M.

Edge

Fongwa

Belin

Miranda

(2007). Does stigma keep poor young immigrant and U.S.-born Black and Latina women from seeking mental health care? Psychiatric Services, 58, 1547-1554. doi:10.1176/appi.ps.58.12.1547

19.

C. H.

(1997). The stigma of mental illness in Asian cultures. Australian & New Zealand Journal of Psychiatry, 31, 382-390. doi:10.3109/00048679709073848

20.

Patil

V. H.

Singh

S. N.

Mishra

Donavan

(2007). Parallel analysis engine to aid determining number of factors to retain [Computer software]. Retrieved from http://smishra.faculty.ku.edu/parallelengine.htm

21.

Pepin

Segal

D. L.

Klebe

K. J.

Coolidge

F. L.

Krakowiak

K. M.

Bartels

S. J.

(2015). The Barriers to Mental Health Services Scale Revised: Psychometric analysis among older adults. Mental Health & Prevention, 3, 178-184. doi:10.1016/j.mhp.2015.09.001

22.

Raguram

Weiss

M. G.

Channabasavanna

S. M.

Devins

G. M.

(1996). Stigma, depression, and somatization in South India. American Journal of Psychiatry, 153, 1043-1049.

23.

Rojas-Vilches

A. P.

Negy

Reig-Ferrer

(2011). Attitudes toward seeking therapy among Puerto Rican and Cuban American young adults and their parents. International Journal of Clinical and Health Psychology, 11, 313-341.

24.

Royal

K. D.

Thompson

J. M.

(2013). A psychometric validation of the Beliefs toward Mental Illness Scale. Journal of Nursing Measurement, 21, 516-524. doi:10.1891/1061-3749.21.3.516

25.

Segal

Coolidge

Mincic

O’Riley

(2005). Beliefs about mental illness and willingness to seek help: A cross-sectional study. Aging & Mental Health, 9, 363-367. doi:10.1080/13607860500131047

26.

Stone

Finlay

W. L.

(2008). A comparison of African-Caribbean and White European young adults’ conceptions of schizophrenia symptoms and the diagnostic label. International Journal of Social Psychiatry, 54, 242-261. doi:10.1177/0020764008089616

27.

Wood

Nosko

Desmarais

Ross

Irvine

(2006). Online and traditional paper-and-pencil survey administration: Examining experimenter presence, sensitive material and long surveys. Canadian Journal of Human Sexuality, 15, 147-155.

28.

Woodward

A. T.

Taylor

R. J.

Bullard

K. M.

Aranda

M. P.

Lincoln

K. D.

Chatters

L. M.

(2012). Prevalence of lifetime DSM-IV affective disorders among older African Americans, Black Caribbeans, Latinos, Asians and non-Hispanic White people. International Journal of Geriatric Psychiatry, 27, 816-827. doi:10.1002/gps.2790

29.

Yadav

Arya

Kataria

Balhara

Y. P. S.

(2012). Impact of psychiatric education and training on attitude of medical students towards mentally ill: A comparative analysis. Industrial Psychiatry Journal, 21, 22-31. doi:10.4103/0972-6748.110944

30.

Yildirim

Demirbuken

Balci

Yurdalan

(2015). Beliefs towards mental illness in Turkish physiotherapy students. Physiotherapy Theory and Practice, 31, 461-465. doi:10.3109/09593985.2015.1025321

Factor Structure and Administration Measurement Invariance of the Beliefs Toward Mental Illness Scale in Latino College Samples: Paper–Pencil Versus Internet Administrations

Abstract

Keywords

Study 1

Method

Participants

Measures

Beliefs Toward Mental Illness (Hirai & Clum, 2000)

Procedure

Results

Factor Structure

Discussion

Study 2

Method

Participants

Measures and Procedure

Analysis Plan

Results

Factor Structure

Measurement Invariance

Discussion

Additional Analyses

Higher Order Structure

Group Comparisons on the Factors

General Discussion

Footnotes

Declaration of Conflicting Interests

Funding

References