Abstract
To assess the effectiveness of China's recent moves to increase community mental health literacy and decrease stigma, we developed the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ). Preliminary versions were assessed in pilot studies and revised versions were included in an interviewer-administered community survey of a representative sample of 2425 adult residents of Ningxia Province and a re-test survey in 188 individuals. Internal consistency, factor structure and test-retest reliability were assessed for three measures: (a) the 25-item MHKQ (alpha = .71, 6 factors accounting for 51% of variance identified in exploratory factor analysis of one-half of the sample, and intraclass correlation coefficient [ICC] for total score of .40); (b) the 14-item attitudinal subscale of MHAQ (alpha = .69, 3 factors accounting for 42% of variance, ICC = .47); and (c) the 7-item causal attribution subscale of MHAQ (alpha = .60, 3 factors accounting for 60% of variance, ICC = .26). Confirmatory factor analysis assessed fitness of modified models of the measures using chi-squared, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA): (a) for the MHKQ, RMSEA = .037 (90% CFI = .033, .040), CFI = .86, TLI = .84, χ2 = 682.86 (df = 260); (b) for the attitudinal subscale of MHAQ, RMSEA = .045 (CI = .039, .052), CFI = .94, TLI = .92, χ2 = 226.67 (df = 66); and (c) for the causal attribution subscale of MHAQ, RMSEA = .054 (.039, .069), CFI = .97, TLI = .94, χ2 = 49.13 (df = 11). We conclude that the internal consistency and factor structure of the new measures are satisfactory, but further work is needed to improve the scales' stability and to assess the construct validity and responsiveness of the scales.
Keywords
Introduction
As low- and middle-income countries (LMICs) undergo the epidemiological transition and chronic diseases replace infectious diseases as the most important health problems, the role of mental health in the overall public health agenda for these countries is increasing (GBD 2013 DALYs and HALE Collaborators, 2015). Recognizing this change, the WHO released its first-ever global mental health action plan for 2013–2020 (WHO, 2013). One of the key goals of this plan is to reduce the stigma associated with mental disorders.
Improving mental health literacy, defined as “knowledge and beliefs about mental disorders which aid their recognition, management or prevention” (Jorm, 2000), is widely recognized as an essential component of the development of mental health services (Furnham & Lousley, 2013; Jorm, 2000; Meng, Yao, Zhu, & Zhang, 2002). Increasing a community's overall mental health literacy and reducing the stigma associated with mental disorders (Gao & Phillips, 2001; Griffiths et al., 2006; Kurihara, Kato, Sakamoto, Reverger, & Kitramura, 2000) potentially has the dual benefit of increasing care-seeking among individuals who suffer from these disorders and of facilitating the reintegration of persons with chronic mental disorders into the community (Dietrich, Mergl, Freudenberg, Althaus, & Hegerl, 2010; Dumesnil & Verger, 2009; Hickie, 2004; Joa et al., 2008; Pinto-Foltz, Logsdon, & Myers, 2011).
In line with these goals, China has addressed the issue of stigma and mental health literacy in national health plans since 2002 (Ministry of Health et al., 2003; Ministries of Health et al., 2008; Xiong & Phillips, 2016) and in its first national mental health law, released in 2013 (Chen et al., 2012). The most recent mental health plan, covering 2015–2020, specifies that by 2020 the “level of mental health awareness in the general public should reach 70% in urban areas and 50% in rural areas” (Xiong & Phillips, 2016).
There is, however, no consistent method of assessing mental health knowledge and stigma about mental illnesses, a limitation that seriously undermines any attempt to evaluate the effectiveness of public education programs aimed at increasing mental health literacy or at decreasing stigma. Moreover, given major differences in the mental health care systems between countries and cultural differences in attitudes about the causes and management of “aberrant” behaviors, the methods of assessing mental health literacy and stigma in high-income countries may not be suitable for LMICs, particularly LMICs like China where the family (not the individual) remains the most important social unit and where “social harmony” (a Confucian value) often trumps individual autonomy.
Development of reliable and valid measures that accurately reflect the level of mental health literacy and the degree and characteristics of stigmatization of mental disorders in different communities is crucial to achieving the goals of China's 2013 national mental health law and of the 2015–2020 national mental health plan. China has previously employed a variety of ad hoc measures to assess mental health awareness (Huang, Xu, Yin, & Tian, 2015; Li et al., 2009; Meng et al., 2002; J. Y. Wang et al., 2013) and stigma (Gao et al., 2005; W. Wang et al., 2011; Xu, Yin, Yang, & Tian, 2014), but the reliability and validity of these measures have not been assessed.
This paper reports on the internal consistency and test-retest reliability of two questionnaires about mental health literacy and attitudes related to mental disorders—the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ)—that were included in a community-based survey of a representative sample of residents of the Ningxia Hui Autonomous Region (hereafter, “Ningxia”), a province in northwest China in which 37% of the population belongs to the Muslim Hui ethnic group.
Methods
This report is part of the Epidemiological Investigation and Health System Interventions for Mental Health in rural Western China conducted by the School of Public Health at Ningxia Medical University in collaboration with the Shanghai Mental Health Center.
Development of the knowledge and attitude questionnaires used in the study
As an initial step in developing the overall survey, investigators from the Shanghai Mental Health Center conducted focus groups about mental health-related issues in Ningxia with government officials in the Provincial Bureau of Health, leaders of the main psychiatric hospitals in the province, and collaborators at the Ningxia Medical University. As a part of these discussions it became clear that understanding both the level of mental health literacy and community attitudes about mental health-related issues would be essential to developing targeted mental health promotion campaigns. Based on these focus groups it was evident that most community members did not have a clear idea about the types of conditions that should be considered mental illnesses; about the causes of mental disorders; about the characteristics of depression, alcohol abuse, and suicide; and about the appropriate prevention and treatment of psychological problems and mental illnesses. It was also clear that community members simultaneously held negative, stigmatizing beliefs about the mentally ill and supportive, caring intentions towards individuals with mental illnesses.
In 2010 China's Ministry of Health (renamed the National Health and Family Planning Commission of China in 2013) posted self-completion questionnaires about mental health literacy and attitudes about mental illness on their website that they recommended local providers use to assess the status of community mental health work after implementation of the 2002–2010 national mental health plan (Ministry of Health, 2010). 1 No information is provided on the website about the origin and development of the questionnaires; the items used in the questionnaires were, apparently, recommended by a small group of psychiatric experts. These scales have been used in a couple of studies reported in Chinese (Huang et al., 2015; J. Y. Wang et al., 2013; Yao et al., 2013; Zhang, 2011), but the test-retest reliability and validity of the scales have not been evaluated. Review of these scales by our research group found substantial problems with their face validity. For example, four of the knowledge questions related to whether or not the respondent had heard about World Mental Health Day, World Drug Abuse Day, World Suicide Prevention Day, and World Sleep Day. Based on this assessment, we decided to develop our own scales, rather than use those proposed by China's Ministry of Health.
The preliminary versions of the two scales used in the current study, the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ), used some of the items from the Ministry of Health scales, but most items were developed after extensive discussions between nine members of the research team at the Shanghai Mental Health Center. The goal was to develop a set of items that could assess community members’ knowledge and attitudes about the issues identified in the focus groups with stakeholders in Ningxia. Unlike the Ministry of Health questionnaires, these scales are not self-completion instruments; they are administered by trained interviewers. We did this to ensure that illiterate respondents (who constitute 36% of adults in Ningxia) could be included in the study sample. Standardized instructions read before reading the questionnaire items included the statement “There are no ‘right’ or ‘wrong’ answers to these statements; if you are unsure how to respond, please make your best guess.”
Up until the recent past, public perceptions of “mental illnesses” in Chinese culture only included psychotic illnesses; mood disorders, alcohol and substance abuse, dementia, and other common mental disorders were considered social or moral problems that were largely unrelated to mental illness (Phillips et al., 2009). The social dysfunction of individuals with psychosis and their inability to conform to social norms resulted in a high level of stigmatization of individuals with a “mental illness,” both for the individual and for his/her family. It is only relatively recently (over the last 10–15 years) that the health system has expanded its focus to include the treatment of depression and other less serious conditions as important targets for community mental health (Xiong & Phillips, 2016); in many parts of the country (including Ningxia) this effort is only just beginning, so the treatment rates for non-psychotic conditions are very low (Phillips et al., 2009). Many community members in China continue to have a very narrow idea of what conditions are classified as mental illnesses, so we decided to use a more inclusive term in the items—“psychological/psychiatric problems” (
/


)—to indicate the types of issues we wanted participants to consider when responding to the items. (For some items where it was necessary to have a narrower definition of the target condition, the term “serious psychological/psychiatric problems” was employed.) In an initial pilot study conducted in May 2013, the preliminary versions of the scales were administered to 254 community member respondents from three urban and two rural locations in Ningxia. The final versions of the instruments (used in the current study) were created using the results of the pilot study.
Characteristics of residents who completed the survey and of the subgroup of residents who completed the repeat survey.
Responses of a representative sample of 2425 adult community residents from Ningxia to the 25 items in the Mental Health Knowledge Questionnaire (MHKQ) a .
aThese items are the English translation of the original Chinese-language version of the scale.
Responses of a representative sample of 2424 adult community residents from Ningxia to the 21 items in the Mental Health Attitude Questionnaire (MHAQ) a .
aThese items are the English translation of the original Chinese-language version of the scale. bThe interpretation of this item in Chinese was ambiguous so it was dropped from the analysis.
The overall score for the causal attribution subscale was computed by first reverse scoring the five non-biological causes items and then converting the sum of all seven items into a score ranging from 0 to 100 (overall score = 100*[sum of seven item scores -number of missing items]/[21 -number of missing items]), with higher scores representing stronger belief in biological causes of mental illnesses. Similarly, the overall score of the attitude subscale was computed by first reverse coding the seven items about negative attitudes and then converting the sum of all 14 items in the subscale into a score ranging from 0 to 100 (overall score = 100*[sum of 14 item scores -number of missing items]/[42 -number of missing items]), with higher scores representing more positive and supportive attitudes about mental illness. The magnitude of negative and positive attitudes about mental illnesses were also separately assessed by reverse coding the seven negative attitude items and converting the combined score of the seven revised item scores into a scale of 0 to 100, and by converting the combined score of the seven positive attitude items to a scale of 0 to 100 (score = 100*[sum of seven item scores -7]/21), with higher scores representing stronger negative or positive attitudes.
These two questionnaires—MHKQ and MHAQ—were included as part of the survey instrument used in the Epidemiological Investigation and Health System Interventions for Mental Health in Ningxia. Demographic characteristics of the respondents collected as part of the overall survey (shown in Table 1) were used in this analysis.
Sampling and conduct of survey
The main investigation was conducted from 18 July 2013 to 26 October 2013. As shown in Figure 1, 10 of the 22 counties in Ningxia Province were selected using probability proportionate to size (PPS) methods, and then 9 urban neighborhoods and 11 rural villages (20 primary sampling sites [PSUs]) were selected from these counties using PPS methods. Using simple random methods, 7032 households were selected from these 20 PSUs. In 3051 (43%) of these households residents were not located or not eligible. Among the 3981 households with potentially eligible residents, the household screening interview was completed and a potential respondent for the full survey was randomly selected from 3054 (76.7%) households. Among these selected respondents, 2425 (79.4%) were located, provided written consent and completed the survey, including the MHKQ and MHAQ (1 of these respondents did not finish the MHAQ). After finishing the survey, the participants were given a gift worth 20 Renminbi (about $US3) for their time.
Flowchart of the study.
Four teams were involved in completing the surveys at the 20 PSUs, which were conducted two to three days after the sampling was completed. Each team consisted of a supervisor, a coordinator, 6 to 12 investigators (students at the School of Public Health of the Ningxia Medical University) and 1 to 5 local guides (local administrators, doctors, women's cadres, etc.). The supervisors, coordinators, administrators, and guides were trained by the research team from the Shanghai Mental Health Center.
Among individuals who completed the main survey, 299 were randomly selected to repeat the survey: 188 (62.9%) of them repeated the survey (including the MHKQ and MHAQ) an average of 2.5 days after the first administration of the survey. The research assistant who completed the repeat survey was blind to the result of the first administration of the survey.
Details of the study procedures are provided in a prior report (Chen, Wang, Phillips, Sun, & Cheng, 2014).
Statistical methods
The data were prepared using double entry verification in EpiData 3.1. SPSS 20.0 was used for the analysis. Reliability analysis and test-retest analysis were done in all participants and in subgroups of respondents based on residence (urban versus rural), ethnicity (Han versus Hui), and educational level (above and below the median level of education). Cronbach's alpha was used to assess internal consistency of the questionnaires. Intra-class correlation coefficients (ICC) were used to assess the test-retest reliability of continuous variables and Kappa was used to assess the test-retest reliability of categorical variables.
Exploratory factor analysis and confirmatory factor analysis were used to assess the factor structure of the two questionnaires with Mplus Version 7. The sample was first divided randomly into two groups. Exploratory factor analysis was performed on the first group (n = 1173) using Varimax rotation. Eigenvalues, a scree plot, and item factor loadings were used to identify the most appropriate number of factors. The stability of the identified factors was then assessed in a confirmatory factor analysis using the second group of participants (n = 1252). The model fitness indexes considered included chi-squared, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA). Values of CFI and TLI >.95 indicate excellent model fit and values >.90 but ≤.95 indicate good model fit; RMSEA values < .05 indicate optimal model fit, values of ≥.05 but < .08 indicate suboptimal model fit, and values of ≥.08 indicates that modification of the model is needed (Hu & Bentler, 1999).
Results
Demographic characteristics and item-response pattern of respondents
Table 1 shows the demographic characteristics of the 2425 respondents who completed the survey and of the subgroup of 188 respondents who repeated the survey in the test-retest part of the study. Participants had a mean (SD) age of 45.7 (15.3) years; those who repeated the survey had a mean age of 47.9 (15.4) years. There was a higher proportion of women among the 188 individuals who repeated the survey than among the 2237 who did not repeat the survey (64% vs. 56%, χ2 = 4.24, p = .039), but there were no statistically significant differences in any of the other demographic characteristics between those who did and did not repeat the survey.
Tables 2 and Table 3 show respondents' answers to the 25 items in the MHKQ and to the 21 items in the MHAQ.
Internal consistency
Internal consistency of participants' responses to the two scales (only using data from the first administration of the scale) was assessed using the mean inter-item correlation and Cronbach's alpha. For the MHKQ, the mean inter-item correlation for the 25 items was .91. The alpha value for the full sample was .71; it was .70 in respondents of Han ethnicity and .74 in respondents of Hui ethnicity; .71 in urban respondents and .72 in rural respondents; and .70 in respondents with 9 years of formal education or more and .71 in respondents with less than 9 years of formal education.
The mean inter-item correlation for the 14 attitudinal items in the MHAQ (seven items about negative attitudes and seven items about positive attitudes related to providing the mentally ill with social support) was .127. The alpha value for the full sample was .67; it was .68 among Han respondents and .65 among Hui respondents; .66 in urban respondents and .66 in rural respondents; and .65 in respondents with 9 years of formal education or more and .65 in respondents with less than 9 years of formal education.
For the seven items in the causal attribution subscale of MHAQ, the mean inter-item correlation was .181. The alpha value for the full sample was .60; it was .60 in Han respondents and .60 in Hui respondents; .61 in urban respondents and .60 in rural respondents; and .55 in respondents with 9 years of formal education or more and .63 in respondents with less than 9 years of formal education.
Overall, based on cut-off scores for alpha suggested by Carmines and Zeller (1979) and by Kline (2000), the internal consistency of the MHKQ is “good” but for the two subscales of MHAQ it is only “fair,” though the higher mean inter-item correlation of the latter scales suggests that the lower alpha values are due to the smaller number of items in these scales (Cortina, 1993). The internal consistency of the scales does not appear to vary by ethnicity, urban versus rural residence, or level of education.
Exploratory factor analysis
Eigenvalues and percent of total variance explained in principal component exploratory factor analysis of one-half of the sample (n = 1173) for the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ).
For the 14-item subscale of the MHAQ about attitudes towards the mentally ill, four eigenvalues were greater than 1. Two constructs were considered when constructing the subscale (negative attitudes about mental illnesses and positive attitudes about providing social support to the mentally ill) so a three latent factor model was considered optimal: one for the overall attitude subscale, and one for each of the two theoretical constructs.
For the 7-item causal attribution subscale of the MHAQ, two eigenvalues were greater than 1. Two constructs were considered when constructing the subscale (biological and non-biological causes of mental illnesses), so a three latent factors model was considered optimal: one for the overall causal attribution subscale and one for each of the two theoretical constructs.
Confirmatory factor analysis
The six-factor model of MHKQ was examined in the second half of the sample using confirmatory factor analysis. As shown in panel A of Figure 2, factor loadings of the 25 items ranged from 0.58 to 1.32 and those of the five latent items ranged from 0.36 to 0.51. The CFI was .756 and the TLI was .734; since both values were less than .900, this indicates that the goodness of f it of the model needs to be improved. However, the RMSEA was .047 (90% CI = .044, .051), which indicated optimal fit (χ2 = 1015.36, df = 275). With the goal of modifying the model to improve its performance, we considered the correlations between pairs of items in each factor and the correlations between the five latent factors; we then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 2, the correlations between the items that were included in the revised model ranged from .03 to .34, and the correlations between the latent variables that were included in the revised model ranged from −.19 to −.34. After modifying the model in this manner, factor loadings of the 25 items ranged from 0.53 to 1.38 and those of the five latent items ranged from 0.37 to 0.55; the CFI (.861) and TLI (.839) improved, though they remained below the target value of .900; the RMSEA also improved (.037, 90% CI = .033, .040), indicating optimal model fit (χ2 = 682.86, df = 260).
Pathway diagram of the confirmatory factor analysis of the six-factor model for the Mental Health Knowledge Questionnaire (MHKQ) as assessed when using the 25 item scores and five latent factors as observed variables (panel A) and when using the 25 item scores and the correlations of items and the correlations of latent factors as observed variables (panel B).The factors shown in the ovals are the latent factors relating to mental health literacy: factor 1, awareness of the treatment and prevention of psychological problems; factor 2, awareness of alcohol-related problems; factor 3, awareness of types of problems considered mental health problems; factor 4, awareness of the problem of suicide in China; factor 5, knowledge about depression and its relationship to physical health; and factor 6, composite measure of mental health literacy. The items shown in the rectangles are the observed variables. The variance of factor 6 and the first item of each of the latent factors are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from latent factor 6 to the five other latent factors and from these latent factors to the observed variables (“items”). (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable). The “unique factors” (which relate to a single observed variable and to a single latent variable), the fractions to the left of the rectangles and below the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In the bottom panel the correlations of items and those of latent variables are added as new observed variables in the model.
The three-factor model for the 14-item attitude subscale of MHAQ was examined in the second half of the sample using confirmatory factor analysis. As shown in the Panel A of Figure 3, factor loadings of the 14 items ranged from 0.21 to 1.41 and those of the two latent items were 0.40 and 0.47. The CFI was .804 and the TLI was .768, both less than .900, indicating that the goodness of fit of the model needs improvement. The RMSEA was .075 (90% CI = .070, .081) which indicated suboptimal fit (χ2 = 601.81, df = 77). To modify the model, we first considered the correlations between pairs of items in each factor and the correlations between the two latent factors and then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 3, the item correlations included in the model ranged from .05 to .24 and the correlation of the two latent items (included in the model) was −.05. After modifying the model in this manner, factor loadings of the 14 items ranged from 0.14 to 1.40 and those of the two latent items were 0.43 and 0.48; all fit indices of the revised model suggested excellent fit (RMSEA = .045, 90% CI = .039, .052; CFI = .940 and TLI = .917; χ2 = 226.67, df = 66).
Pathway diagram of the confirmatory factor analysis of the three-factor model for the attitude subscale of the Mental Health Attitude Questionnaire (MAKQ) as assessed when using the 14 item scores and two latent factors as observed variables (panel A) and when using the 14 item scores and the correlations of items as observed variables (panel B). The factors shown in the ovals are the latent factors for the model, factor 1 is the latent variable of negative attitudes about mental illnesses and persons with mental illnesses, factor 2 is the latent variable of positive attitudes about providing social support to persons with mental illnesses, and factor 3 is the latent variable of the attitude to mental illness. The items shown in the rectangles are the observed variables. The variance of factor 3 and the first item of each latent factor are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from the latent variables to the observed variables and from the latent factor 3 to the other latent factors. (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable or the variance of other latent variable explained by the latent variable 3.) The “unique factors” (which relate to a single observed variable and to a single latent variable) to the left of the rectangles and the bottom of the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In panel B the correlations of Items and the correlations of two first-order latent factors are added as new observed variables in the model.
The three-factor model for the MHAQ causal attribution subscale was also assessed in the second half of the sample using confirmatory factor analysis. As shown in Panel A of Figure 4, factor loadings of the seven items ranged from 0.981 to 1.31 and those of the two latent items were 0.47 and 0.41. The CFI was .936 and the TLI was .904; both were greater than .900, indicating optimal model goodness of fit. The RMSEA was .068 (90% CI = .055, .081), which indicated suboptimal fit (χ2 = 91.43, df = 14). To improve the model, we first considered the correlations between pairs of items in each factor and the correlations between the two latent factors and then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 4, the item correlations included in the model ranged from −.07 to .15 and the correlation of the two latent items was not included in the model. After modifying the model in this manner, factor loadings of the seven items ranged from 1.00 to 1.48 and those of the two latent items were 0.37 and 0.48; the RMSEA improved but remained suboptimal (RMSEA = .054, 90% CI = .039, .069); and CFI (.968) and TLI (.940) both indicted optimal model fit (χ2 = 49.13, df = 11).
Pathway diagram of the confirmatory factor analysis of the three-factor model for the causal attribution subscale of the Mental Health Attitude Questionnaire (MAKQ) as assessed when using the 7-item scores and two latent factors as observed variables (panel A) and when using the 7-item scores and the correlations of items as observed variables (panel B). The factors shown in the ovals are the latent factors for the model, factor 1 is the latent variable of biomedical causes of mental illnesses, factor 2 is the latent variable of non-biomedical causes of mental illness, and the factor 3 is the latent variable of beliefs about the causes of mental illness. The items shown in the rectangles are the observed variables. The variance of factor 3 and the of first item of each first-order latent variable are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from the latent variables to the observed variables and from the latent factor 3 to the other latent factors. (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable or the variance of other latent variable explained by the latent variable 3.) The “unique factors” (which relate to a single observed variable and to a single latent variable) to the left of the rectangles and the bottom of the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In panel B the correlations of items are added as new observed variables in the model.
Test-retest reliability
Mean (SD) total scores of Mental Health Knowledge Questionnaire (MHKQ) and Mental Health Attitude Questionnaire (MHAQ) and intraclass correlation coefficient (ICC) in different groups of participants who completed the instrument twice.
aHigher scores represent greater knowledge about mental illnesses; paired t-test of before versus after total score in all participants = 0.57 (p = .566).
bHigher scores represent more positive, supportive attitudes about the mentally ill; paired t-test of before versus after total score in all participants = 2.29 (p = .023).
cHigher scores represent greater support for biological versus non-biological causes of mental illnesses; paired t-test of before versus after total score in all participants = 2.59 (p = .010).
Discussion
Assessing the influence of mental health literacy and stigmatization of people with mental illness on the development and utilization of mental health services is an important component of understanding the effectiveness (or lack of effectiveness) of policies aimed at improving mental health. Given the very different social, economic, and health care environments of high-income countries and low- and middle-income countries (LMICs), LMICs must develop their own culture-sensitive methods for assessing mental health literacy and stigmatization. This involves identifying the specific types of information and attitudes about mental health that are most pertinent to the target culture and deciding which stakeholders in the setting hold the “correct” knowledge about these topics. This is a complex task that requires the integration of high-quality qualitative and quantitative methods, so few LMICs have developed fully validated instruments for doing this. A recent systematic review of the quality of instruments developed to assess mental health knowledge (Wei, McGrath, Hayden, & Kutcher, 2016) identified very few high-quality studies in Western countries and almost none in non-Western countries.
This study reports on one attempt to do this in China. We started with focus groups among key stakeholders, generated preliminary versions of two instruments, conducted a pilot test of the preliminary instruments, and then conducted a relatively large community-based survey with revised versions of the instruments. The instruments assessed knowledge about mental illnesses in five dimensions (classification of specific conditions as psychological problems, characteristics of depression, alcohol use and addiction, suicide, and treatment/prevention of psychological problems), attitudes about the mentally ill in two dimensions (negative and positive attitudes), and causal attributions of mental illnesses in two dimensions (biological and non-biological causes). We found that the internal consistency of the items in the MHKQ and in the two subscales of the MHAQ were satisfactory (though only marginally so for the two MHAQ subscales) and that the indices of model fit for the three measures were generally strong (though not optimal in all cases). However, the test-retest reliability of the total scores for the three measures—particularly the causal attribution subscale of the MHAQ—was relatively poor.
The few studies that have assessed test-retest reliability of mental health knowledge and attitudes in Western countries (Svensson et al., 2011) also report poor test-retest reliability, suggesting that this may be a general weakness of instruments assessing knowledge and attitudes in the general public. In our study the very short 2- to 3-day time between administrations (dictated by the practicalities of travelling to 20 study sites around the province) may have been a factor. The learning effect of the first administration may have persisted at the time of the second administration, resulting in higher scores for knowledge, more positive attitudes, and an increase preference for biological causal models of mental illnesses. An interval of 2 to 3 weeks would have been preferable.
Other limitations need to be considered when evaluating these results. We assessed the instruments in a large community-based sample of adults from one province in China so the results are relevant for this population, but they may not be relevant for other parts of China or for adolescents. Due to the low educational level of respondents we had interviewers administer the scales; self-completion versions of the scales may have different psychometric properties. The very low test-retest reliability of the MHAQ for Hui respondents (ICC = .00) may have been a statistical outlier, but this needs to be repeated in a separate sample to clarify the reason for this surprising finding. We have combined two different dimensions into a summary measure of attitudes about mental illnesses and two different dimensions into a summary measure of causal attributions of mental illnesses; this assumes that the dimensions are conceptually related, which may not be the case. The modifications of the final models for the three scales were based on the characteristics of responses in the second half of the sample used for the confirmatory factor analysis; they should be validated by testing the modified models in an independent sample. Finally, we did not assess the construct validity of the scales (e.g., by comparing mental health literacy between community members and mental health providers) or the responsiveness of the scale scores to changes in knowledge or attitudes (e.g., before and after an intensive public mental health education campaign).
Despite these limitations, the satisfactory internal consistency and generally strong indices of model fit for the MHKQ and the two subscales of the MHAQ support further development of the scales. Subsequent work with the scales should (a) reassess test-retest reliability at longer intervals; (b) assess construct validity and sensitivity to change; (c) compare self-completion versus interview-completed versions of the scales; (d) conduct detailed analyses of the relationship of the different dimensions of each scale; and (e) expand the use of the instruments to other parts of China and to younger respondents. After completion of these steps, final versions of the scales (with somewhat different content for different types of respondents) can be employed to help develop and assess the outcome of targeted mental health campaigns in China that have the following objectives: expanding perceptions about what constitutes a “mental illness”; increasing the care-seeking of individuals with these problems; decreasing the stigma associated with mental illnesses; and increasing the willingness of community members, institutions, and government agencies to expend the resources required to provide mentally ill individuals with the help they need.
Footnotes
Acknowledgments
The authors thank the students in the School of Public Health at Ningxia Medical University and staff members of the Ningxia Center for Disease Control and Prevention for their assistance in the conduct of the study. Hanhui Chen and Zhizhong Wang are co-first authors.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the China Medical Board (project number: 11-063).
Ethics approval
The study was approved by the institutional review board of Ningxia Medical University (No. 2013-167).
