Assessing knowledge and attitudes about mental illness in Ningxia,China

Abstract

To assess the effectiveness of China's recent moves to increase community mental health literacy and decrease stigma, we developed the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ). Preliminary versions were assessed in pilot studies and revised versions were included in an interviewer-administered community survey of a representative sample of 2425 adult residents of Ningxia Province and a re-test survey in 188 individuals. Internal consistency, factor structure and test-retest reliability were assessed for three measures: (a) the 25-item MHKQ (alpha = .71, 6 factors accounting for 51% of variance identified in exploratory factor analysis of one-half of the sample, and intraclass correlation coefficient [ICC] for total score of .40); (b) the 14-item attitudinal subscale of MHAQ (alpha = .69, 3 factors accounting for 42% of variance, ICC = .47); and (c) the 7-item causal attribution subscale of MHAQ (alpha = .60, 3 factors accounting for 60% of variance, ICC = .26). Confirmatory factor analysis assessed fitness of modified models of the measures using chi-squared, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA): (a) for the MHKQ, RMSEA = .037 (90% CFI = .033, .040), CFI = .86, TLI = .84, χ² = 682.86 (df = 260); (b) for the attitudinal subscale of MHAQ, RMSEA = .045 (CI = .039, .052), CFI = .94, TLI = .92, χ² = 226.67 (df = 66); and (c) for the causal attribution subscale of MHAQ, RMSEA = .054 (.039, .069), CFI = .97, TLI = .94, χ² = 49.13 (df = 11). We conclude that the internal consistency and factor structure of the new measures are satisfactory, but further work is needed to improve the scales' stability and to assess the construct validity and responsiveness of the scales.

Keywords

China factor analysis mental health literacy psychometric assessment stigma test-retest reliability

Introduction

As low- and middle-income countries (LMICs) undergo the epidemiological transition and chronic diseases replace infectious diseases as the most important health problems, the role of mental health in the overall public health agenda for these countries is increasing (GBD 2013 DALYs and HALE Collaborators, 2015). Recognizing this change, the WHO released its first-ever global mental health action plan for 2013–2020 (WHO, 2013). One of the key goals of this plan is to reduce the stigma associated with mental disorders.

Improving mental health literacy, defined as “knowledge and beliefs about mental disorders which aid their recognition, management or prevention” (Jorm, 2000), is widely recognized as an essential component of the development of mental health services (Furnham & Lousley, 2013; Jorm, 2000; Meng, Yao, Zhu, & Zhang, 2002). Increasing a community's overall mental health literacy and reducing the stigma associated with mental disorders (Gao & Phillips, 2001; Griffiths et al., 2006; Kurihara, Kato, Sakamoto, Reverger, & Kitramura, 2000) potentially has the dual benefit of increasing care-seeking among individuals who suffer from these disorders and of facilitating the reintegration of persons with chronic mental disorders into the community (Dietrich, Mergl, Freudenberg, Althaus, & Hegerl, 2010; Dumesnil & Verger, 2009; Hickie, 2004; Joa et al., 2008; Pinto-Foltz, Logsdon, & Myers, 2011).

In line with these goals, China has addressed the issue of stigma and mental health literacy in national health plans since 2002 (Ministry of Health et al., 2003; Ministries of Health et al., 2008; Xiong & Phillips, 2016) and in its first national mental health law, released in 2013 (Chen et al., 2012). The most recent mental health plan, covering 2015–2020, specifies that by 2020 the “level of mental health awareness in the general public should reach 70% in urban areas and 50% in rural areas” (Xiong & Phillips, 2016).

There is, however, no consistent method of assessing mental health knowledge and stigma about mental illnesses, a limitation that seriously undermines any attempt to evaluate the effectiveness of public education programs aimed at increasing mental health literacy or at decreasing stigma. Moreover, given major differences in the mental health care systems between countries and cultural differences in attitudes about the causes and management of “aberrant” behaviors, the methods of assessing mental health literacy and stigma in high-income countries may not be suitable for LMICs, particularly LMICs like China where the family (not the individual) remains the most important social unit and where “social harmony” (a Confucian value) often trumps individual autonomy.

Development of reliable and valid measures that accurately reflect the level of mental health literacy and the degree and characteristics of stigmatization of mental disorders in different communities is crucial to achieving the goals of China's 2013 national mental health law and of the 2015–2020 national mental health plan. China has previously employed a variety of ad hoc measures to assess mental health awareness (Huang, Xu, Yin, & Tian, 2015; Li et al., 2009; Meng et al., 2002; J. Y. Wang et al., 2013) and stigma (Gao et al., 2005; W. Wang et al., 2011; Xu, Yin, Yang, & Tian, 2014), but the reliability and validity of these measures have not been assessed.

This paper reports on the internal consistency and test-retest reliability of two questionnaires about mental health literacy and attitudes related to mental disorders—the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ)—that were included in a community-based survey of a representative sample of residents of the Ningxia Hui Autonomous Region (hereafter, “Ningxia”), a province in northwest China in which 37% of the population belongs to the Muslim Hui ethnic group.

Methods

This report is part of the Epidemiological Investigation and Health System Interventions for Mental Health in rural Western China conducted by the School of Public Health at Ningxia Medical University in collaboration with the Shanghai Mental Health Center.

Development of the knowledge and attitude questionnaires used in the study

As an initial step in developing the overall survey, investigators from the Shanghai Mental Health Center conducted focus groups about mental health-related issues in Ningxia with government officials in the Provincial Bureau of Health, leaders of the main psychiatric hospitals in the province, and collaborators at the Ningxia Medical University. As a part of these discussions it became clear that understanding both the level of mental health literacy and community attitudes about mental health-related issues would be essential to developing targeted mental health promotion campaigns. Based on these focus groups it was evident that most community members did not have a clear idea about the types of conditions that should be considered mental illnesses; about the causes of mental disorders; about the characteristics of depression, alcohol abuse, and suicide; and about the appropriate prevention and treatment of psychological problems and mental illnesses. It was also clear that community members simultaneously held negative, stigmatizing beliefs about the mentally ill and supportive, caring intentions towards individuals with mental illnesses.

In 2010 China's Ministry of Health (renamed the National Health and Family Planning Commission of China in 2013) posted self-completion questionnaires about mental health literacy and attitudes about mental illness on their website that they recommended local providers use to assess the status of community mental health work after implementation of the 2002–2010 national mental health plan (Ministry of Health, 2010).¹ No information is provided on the website about the origin and development of the questionnaires; the items used in the questionnaires were, apparently, recommended by a small group of psychiatric experts. These scales have been used in a couple of studies reported in Chinese (Huang et al., 2015; J. Y. Wang et al., 2013; Yao et al., 2013; Zhang, 2011), but the test-retest reliability and validity of the scales have not been evaluated. Review of these scales by our research group found substantial problems with their face validity. For example, four of the knowledge questions related to whether or not the respondent had heard about World Mental Health Day, World Drug Abuse Day, World Suicide Prevention Day, and World Sleep Day. Based on this assessment, we decided to develop our own scales, rather than use those proposed by China's Ministry of Health.

The preliminary versions of the two scales used in the current study, the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ), used some of the items from the Ministry of Health scales, but most items were developed after extensive discussions between nine members of the research team at the Shanghai Mental Health Center. The goal was to develop a set of items that could assess community members’ knowledge and attitudes about the issues identified in the focus groups with stakeholders in Ningxia. Unlike the Ministry of Health questionnaires, these scales are not self-completion instruments; they are administered by trained interviewers. We did this to ensure that illiterate respondents (who constitute 36% of adults in Ningxia) could be included in the study sample. Standardized instructions read before reading the questionnaire items included the statement “There are no ‘right’ or ‘wrong’ answers to these statements; if you are unsure how to respond, please make your best guess.”

Up until the recent past, public perceptions of “mental illnesses” in Chinese culture only included psychotic illnesses; mood disorders, alcohol and substance abuse, dementia, and other common mental disorders were considered social or moral problems that were largely unrelated to mental illness (Phillips et al., 2009). The social dysfunction of individuals with psychosis and their inability to conform to social norms resulted in a high level of stigmatization of individuals with a “mental illness,” both for the individual and for his/her family. It is only relatively recently (over the last 10–15 years) that the health system has expanded its focus to include the treatment of depression and other less serious conditions as important targets for community mental health (Xiong & Phillips, 2016); in many parts of the country (including Ningxia) this effort is only just beginning, so the treatment rates for non-psychotic conditions are very low (Phillips et al., 2009). Many community members in China continue to have a very narrow idea of what conditions are classified as mental illnesses, so we decided to use a more inclusive term in the items—“psychological/psychiatric problems” (/)—to indicate the types of issues we wanted participants to consider when responding to the items. (For some items where it was necessary to have a narrower definition of the target condition, the term “serious psychological/psychiatric problems” was employed.) In an initial pilot study conducted in May 2013, the preliminary versions of the scales were administered to 254 community member respondents from three urban and two rural locations in Ningxia. The final versions of the instruments (used in the current study) were created using the results of the pilot study.

The preliminary version of the MHKQ used in the pilot study included 20 items with “yes” or “no” responses. Six of the items were revised versions of items from the Ministry of Health scale; the remaining 14 items were generated by the research team to assess respondents' knowledge about the range of conditions considered psychological or psychiatric problems, and about the causes, appropriate treatment, and prevention of common mental illnesses and suicide. We specifically chose items for the MHKQ that addressed differences of opinion between Chinese mental health professionals and community members (to increase the discriminating power of the items) and we used the opinions of Chinese mental health professionals—primarily psychiatrists—as the “correct” response. The alpha value for all 20 items in the pilot study (representing the internal consistency of the items) was acceptable (alpha = .74), but the response patterns for six items were imbalanced (either too easy or too hard), so they were removed from the final scale. The use of sentences with double negatives in nine of the items was confusing for some respondents in the pilot study, so the wording of these questions was simplified in the final scale. When creating the final version of the scale, 11 new items were added: five items about alcohol use, two items about psychological problems in the elderly, and one item each about sleep, physical symptoms as a sign of psychological problems, effectiveness of treatment for psychological problems, and the level of insight persons with psychological problems have about their condition. The final 25-item MHKQ (see Table 2) included five groups of items: five items about treatment and prevention of psychological problems (items 3, 5, 11, 21, 25), six items about alcohol use and addiction (items 2, 8, 12, 14, 19, 22), six items about whether or not specific conditions should be considered psychological/psychiatric problems (items 1, 6, 10, 16, 18, 24), four items about suicide (items 7, 13, 20, 23), and four items related to depression (items 4, 9, 15, 17). The correct response is “yes” for items 3, 4, 6, 9, 10, 12, 13, 14, 15, 17, 19, 22 and 23; and “no” for items 1, 2, 5, 7, 8, 11, 16, 18, 20, 21, 24 and 25. Each correct response is given a score of “1,” so multiplying the total score of all 25 items by 4 generated a 0–100 mental health literacy score, with higher scores representing greater mental health literacy.

Table 1.

Characteristics of residents who completed the survey and of the subgroup of residents who completed the repeat survey.

	Residents who completed the survey (n = 2425)		Subsample of residents who completed the repeat survey (n = 188)
Characteristic	n	%	N	%
Gender
Female	1347	55.5	120	63.8
Male	1078	44.5	68	36.2
Residence
Urban	1207	49.8	95	50.5
Rural	1218	50.2	93	49.5
Ethnicity
Han	1846	76.1	152	80.9
Hui	561	23.1	35	18.6
Other	18	0.8	1	0.5
Educational level
Less than 9 years of education	1373	56.6	117	62.2
≥9 years of education	1052	43.4	71	37.8
Marital status
Never married	191	7.9	12	6.4
Married	2106	86.8	165	87.8
Divorced, separated or widowed	128	5.3	11	5.9
Religious affiliation
None	1583	65.3	124	66.0
Buddhist	211	8.7	23	12.2
Islamic	551	22.7	37	19.7
Other	80	3.3	4	2.1
Employment status
Agricultural laborer	877	36.2	72	38.3
Factory worker/service personnel	286	11.8	26	13.8
Professional/government cadre	337	13.9	19	10.1
Self-employed/temporary worker	238	9.8	15	8.0
Retired/on sick leave	237	9.8	17	9.0
Unemployed	144	5.9	11	5.9
Student/housewife	293	12.1	28	14.9
Other	13	0.6	0	0.0

Table 2.

Responses of a representative sample of 2425 adult community residents from Ningxia to the 25 items in the Mental Health Knowledge Questionnaire (MHKQ)^a.

Item	Yes (%)	No (%)
1. Most psychological problems are due to morally incorrect (inappropriate) thinking	61.9	38.1
2. The main reason for alcoholism is that the will is not firm	53.4	46.6
3. For some psychological/psychiatric problems it is necessary to constantly use medication	39.7	60.3
4. Depression is one of the common psychological problems in the elderly	58.3	41.7
5. Most people who are aware of their psychological problems are willing to seek professional help	61.2	38.8
6. Heavy drinking is a psychological problem	48.9	51.1
7. Suicide is much more common in rural China than in urban China	28.1	71.9
8. Medications used to treat psychological problems are harmful or addictive	69.4	30.6
9. Persons with diabetes are much more likely to develop depression than others	65.5	34.4
10. Long-term sleep problems should be considered a psychological problem	74.1	25.9
11. Most psychological problems are not treatable	32.5	67.5
12. Drinking is addictive	71.1	28.9
13. Male and female suicide rates are similar in China	28.6	71.4
14. Long-term excessive drinking causes liver cirrhosis	88.6	11.4
15. A variety of physical discomforts are common symptoms of psychological problems	64.1	35.9
16. Old-age dementia is NOT a psychological/psychiatric problem	54.4	45.6
17. Persons with depression are much more likely than other people to have a heart attack	70.2	29.8
18. Illicit drug abuse and dependence are NOT psychological/psychiatric problems	48.6	51.4
19. Accidents are more likely to happen after drinking	93.9	6.1
20. Suicide is more common in the young than in the elderly	84.8	15.2
21. There is no way to prevent psychological/psychiatric problems	47.8	52.2
22. Persons with alcohol dependence need systematic treatment	62.7	37.3
23. Compared to other people, those with a prior suicide attempt are more likely to die of suicide	71.1	28.9
24. Most people with psychological problems are not aware of their psychological problem	86.0	14.0
25. Old people are very unlikely to develop psychological problems	48.1	51.9

^aThese items are the English translation of the original Chinese-language version of the scale.

The preliminary version of the MHAQ used in the pilot study included 14 items (four of which were revised versions of items in the Ministry of Health scale) with 4-point Likert scale responses (“completely agree” = 4, “basically agree” = 3, “basically disagree” = 2, “completely disagree” = 1) that assessed respondents' beliefs about the causes of psychological and psychiatric problems, their adoption of negative attitudes about mental illnesses, and their ideas regarding the types of social supports that should be provided to individuals with mental health problems. After the pilot study, 12 of the original items were retained, two items were substantially reworded, and nine items were added, including four items about attitudes related to alcohol use, two items about causes of mental disorders, two items about social support for the mentally ill, and one item about stigma of the mentally ill. The 22-item version of the MHAQ (see Table 3) used in the main study included two subscales: a 7-item causal attribution subscale with two items about biological factors (items 1, 17) and five items about non-biological factors (items 4, 7, 10, 14, 21); and a 15-item attitude subscale with seven items that assess negative attitudes about mental illness (items 2, 3, 5, 12, 15, 19, 22) and eight items that assess positive attitudes about providing social support to persons with mental illnesses (items 6, 8, 9, 11, 13, 16, 18, 20). However, one of the newly added items about the social acceptability of heavy drinking (item 8) proved ambiguous (i.e., different respondents interpreted the meaning of the item differently), so it was dropped from the scale, resulting in a final 21-item version of MHAQ, with seven items in the causal attribution subscale and 14 items in the attitude subscale.

Table 3.

Responses of a representative sample of 2424 adult community residents from Ningxia to the 21 items in the Mental Health Attitude Questionnaire (MHAQ)^a.

Item	Completely agree (%)	Basically agree (%)	Basically disagree (%)	Completely disagree (%)
1. Genetic factors are the main cause of most psychological and psychiatric problems	16.6	24.9	42.8	15.7
2. Family members should take the responsibility for any harm caused by persons with psychological or psychiatric problems	13.4	19.5	39.6	27.5
3. Psychological and psychiatric problems that have serious effects on functioning can't be cured	19.9	33.7	31.3	15.1
4. External stimuli and stress are the main causes of most psychological and psychiatric problems	3.7	9.3	52.7	34.4
5. Being treated in a psychiatric hospital is a sign of personal failure	41.6	31.5	17.3	9.5
6. All doctors should provide services for persons with psychological or psychiatric problems	4.9	11.2	35.4	48.5
7. Social factors are the main cause of most psychological and psychiatric problems	4.9	15.2	53.3	26.6
8. Excessive drinking that has the purpose of establishing good interpersonal relationships is acceptable^b	–	–	–	–
9. The government reimbursement ratio for treatment of psychological and psychiatric problems should be higher than that for other diseases	7.3	22.6	44.5	25.6
10. Bad family relationships are the main cause of most psychological and psychiatric problems	8.4	16.0	47.9	27.7
11. The total cost for treating alcoholism should be covered by the government	47.0	28.9	15.3	8.8
12. Persons with psychological or psychiatric problems are a heavy burden on the society	14.7	23.6	37.5	24.2
13. Companies should employ persons who have completely recovered from mental illnesses	7.7	12.6	44.8	34.9
14. Bad luck/fate is the main cause of most psychological and psychiatric problems	20.9	29.8	36.5	12.9
15. Persons with psychological or psychiatric problems must accept involuntary treatment	7.5	20.7	40.0	31.8
16. The intelligence of persons who have been treated at a psychiatric hospital is similar to that of the general public	15.8	28.5	41.9	13.7
17. Environment pollution, toxic substances, unhealthy diet, trauma, physical illnesses, and other factors are the main causes of most psychological and psychiatric problems	10.4	23.0	47.2	19.5
18. All types of medical facilities should provide diagnostic and treatment services for persons with psychological or psychiatric problems	3.2	9.0	45.2	42.7
19. Persons with psychological or psychiatric problems often threaten others' safety	8.3	20.9	43.7	27.1
20. Everyone has responsibility for preventing suicides among persons with psychological or psychiatric problems	8.6	13.6	42.1	35.7
21. Individual personality is the main cause of most psychological and psychiatric problems	5.4	12.6	55.5	26.4
22. Persons with alcoholism pose a heavy burden on the society	15.5	26.1	32.5	25.8

^aThese items are the English translation of the original Chinese-language version of the scale. ^bThe interpretation of this item in Chinese was ambiguous so it was dropped from the analysis.

The overall score for the causal attribution subscale was computed by first reverse scoring the five non-biological causes items and then converting the sum of all seven items into a score ranging from 0 to 100 (overall score = 100*[sum of seven item scores -number of missing items]/[21 -number of missing items]), with higher scores representing stronger belief in biological causes of mental illnesses. Similarly, the overall score of the attitude subscale was computed by first reverse coding the seven items about negative attitudes and then converting the sum of all 14 items in the subscale into a score ranging from 0 to 100 (overall score = 100*[sum of 14 item scores -number of missing items]/[42 -number of missing items]), with higher scores representing more positive and supportive attitudes about mental illness. The magnitude of negative and positive attitudes about mental illnesses were also separately assessed by reverse coding the seven negative attitude items and converting the combined score of the seven revised item scores into a scale of 0 to 100, and by converting the combined score of the seven positive attitude items to a scale of 0 to 100 (score = 100*[sum of seven item scores -7]/21), with higher scores representing stronger negative or positive attitudes.

These two questionnaires—MHKQ and MHAQ—were included as part of the survey instrument used in the Epidemiological Investigation and Health System Interventions for Mental Health in Ningxia. Demographic characteristics of the respondents collected as part of the overall survey (shown in Table 1) were used in this analysis.

Sampling and conduct of survey

The main investigation was conducted from 18 July 2013 to 26 October 2013. As shown in Figure 1, 10 of the 22 counties in Ningxia Province were selected using probability proportionate to size (PPS) methods, and then 9 urban neighborhoods and 11 rural villages (20 primary sampling sites [PSUs]) were selected from these counties using PPS methods. Using simple random methods, 7032 households were selected from these 20 PSUs. In 3051 (43%) of these households residents were not located or not eligible. Among the 3981 households with potentially eligible residents, the household screening interview was completed and a potential respondent for the full survey was randomly selected from 3054 (76.7%) households. Among these selected respondents, 2425 (79.4%) were located, provided written consent and completed the survey, including the MHKQ and MHAQ (1 of these respondents did not finish the MHAQ). After finishing the survey, the participants were given a gift worth 20 Renminbi (about $US3) for their time.

Figure 1.

Flowchart of the study.

Four teams were involved in completing the surveys at the 20 PSUs, which were conducted two to three days after the sampling was completed. Each team consisted of a supervisor, a coordinator, 6 to 12 investigators (students at the School of Public Health of the Ningxia Medical University) and 1 to 5 local guides (local administrators, doctors, women's cadres, etc.). The supervisors, coordinators, administrators, and guides were trained by the research team from the Shanghai Mental Health Center.

Among individuals who completed the main survey, 299 were randomly selected to repeat the survey: 188 (62.9%) of them repeated the survey (including the MHKQ and MHAQ) an average of 2.5 days after the first administration of the survey. The research assistant who completed the repeat survey was blind to the result of the first administration of the survey.

Details of the study procedures are provided in a prior report (Chen, Wang, Phillips, Sun, & Cheng, 2014).

Statistical methods

The data were prepared using double entry verification in EpiData 3.1. SPSS 20.0 was used for the analysis. Reliability analysis and test-retest analysis were done in all participants and in subgroups of respondents based on residence (urban versus rural), ethnicity (Han versus Hui), and educational level (above and below the median level of education). Cronbach's alpha was used to assess internal consistency of the questionnaires. Intra-class correlation coefficients (ICC) were used to assess the test-retest reliability of continuous variables and Kappa was used to assess the test-retest reliability of categorical variables.

Exploratory factor analysis and confirmatory factor analysis were used to assess the factor structure of the two questionnaires with Mplus Version 7. The sample was first divided randomly into two groups. Exploratory factor analysis was performed on the first group (n = 1173) using Varimax rotation. Eigenvalues, a scree plot, and item factor loadings were used to identify the most appropriate number of factors. The stability of the identified factors was then assessed in a confirmatory factor analysis using the second group of participants (n = 1252). The model fitness indexes considered included chi-squared, comparative fit index (CFI), Tucker-Lewis index (TLI), and root mean square error of approximation (RMSEA). Values of CFI and TLI >.95 indicate excellent model fit and values >.90 but ≤.95 indicate good model fit; RMSEA values < .05 indicate optimal model fit, values of ≥.05 but < .08 indicate suboptimal model fit, and values of ≥.08 indicates that modification of the model is needed (Hu & Bentler, 1999).

Results

Demographic characteristics and item-response pattern of respondents

Table 1 shows the demographic characteristics of the 2425 respondents who completed the survey and of the subgroup of 188 respondents who repeated the survey in the test-retest part of the study. Participants had a mean (SD) age of 45.7 (15.3) years; those who repeated the survey had a mean age of 47.9 (15.4) years. There was a higher proportion of women among the 188 individuals who repeated the survey than among the 2237 who did not repeat the survey (64% vs. 56%, χ² = 4.24, p = .039), but there were no statistically significant differences in any of the other demographic characteristics between those who did and did not repeat the survey.

Tables 2 and Table 3 show respondents' answers to the 25 items in the MHKQ and to the 21 items in the MHAQ.

Internal consistency

Internal consistency of participants' responses to the two scales (only using data from the first administration of the scale) was assessed using the mean inter-item correlation and Cronbach's alpha. For the MHKQ, the mean inter-item correlation for the 25 items was .91. The alpha value for the full sample was .71; it was .70 in respondents of Han ethnicity and .74 in respondents of Hui ethnicity; .71 in urban respondents and .72 in rural respondents; and .70 in respondents with 9 years of formal education or more and .71 in respondents with less than 9 years of formal education.

The mean inter-item correlation for the 14 attitudinal items in the MHAQ (seven items about negative attitudes and seven items about positive attitudes related to providing the mentally ill with social support) was .127. The alpha value for the full sample was .67; it was .68 among Han respondents and .65 among Hui respondents; .66 in urban respondents and .66 in rural respondents; and .65 in respondents with 9 years of formal education or more and .65 in respondents with less than 9 years of formal education.

For the seven items in the causal attribution subscale of MHAQ, the mean inter-item correlation was .181. The alpha value for the full sample was .60; it was .60 in Han respondents and .60 in Hui respondents; .61 in urban respondents and .60 in rural respondents; and .55 in respondents with 9 years of formal education or more and .63 in respondents with less than 9 years of formal education.

Overall, based on cut-off scores for alpha suggested by Carmines and Zeller (1979) and by Kline (2000), the internal consistency of the MHKQ is “good” but for the two subscales of MHAQ it is only “fair,” though the higher mean inter-item correlation of the latter scales suggests that the lower alpha values are due to the smaller number of items in these scales (Cortina, 1993). The internal consistency of the scales does not appear to vary by ethnicity, urban versus rural residence, or level of education.

Exploratory factor analysis

Results of the exploratory factor analysis using data from one-half of the total sample are shown in Table 4. For the MHKQ, seven of 25 components' eigenvalues were greater than 1. Five theoretical concepts were considered when constructing the scale (treatment and prevention, alcohol use and addiction, psychological/psychiatric conditions, suicide, and depression), so a model with six latent factors was considered optimal; one latent factor for the overall scale and one latent factor for each of the five theoretical constructs.

Table 4.

Eigenvalues and percent of total variance explained in principal component exploratory factor analysis of one-half of the sample (n = 1173) for the Mental Health Knowledge Questionnaire (MHKQ) and the Mental Health Attitude Questionnaire (MHAQ).

	MHKQ (25 items)			14-item attitudinal subscale of MHAQ			7-item causal attribution subscale of MHAQ
Component	Eigen- value	% of variance	Cumulative %	Eigen- value	% of variance	Cumulative %	Eigen- value	% of variance	Cumulative %
1	4.91	19.6	19.6	3.21	22.9	22.9	2.28	32.5	32.5
2	2.74	11.0	30.6	1.41	10.1	33.0	1.00	14.3	46.8
3	1.58	6.3	36.9	1.30	9.3	42.3	0.90	12.9	59.6
4	1.28	5.1	42.0	1.05	7.5	49.8	0.81	11.5	71.2
5	1.18	4.7	46.8	0.98	7.0	56.8	0.72	10.3	81.5
6	1.13	4.5	51.3	0.85	6.1	62.9	0.68	9.7	91.2
7	1.04	4.2	55.5	0.81	5.8	68.7	0.62	8.8	100.0
8	1.00	4.0	59.4	0.76	5.4	74.1
9	0.90	3.6	63.1	0.72	5.1	79.2
10	0.84	3.4	66.4	0.70	5.0	84.2
11	0.81	3.2	69.7	0.61	4.4	88.6
12	0.80	3.2	72.9	0.58	4.1	92.7
13	0.74	3.0	75.8	0.53	3.8	96.5
14	0.69	2.7	78.6	0.45	3.5	100.0
15	0.65	2.6	81.2
16	0.62	2.5	83.7
17	0.59	2.3	86.0
18	0.56	2.2	88.3
19	0.55	2.2	90.5
20	0.52	2.1	92.5
21	0.51	2.0	94.6
22	0.44	1.8	96.3
23	0.40	1.6	97.9
24	0.33	1.3	99.2
25	0.19	0.8	100.0

For the 14-item subscale of the MHAQ about attitudes towards the mentally ill, four eigenvalues were greater than 1. Two constructs were considered when constructing the subscale (negative attitudes about mental illnesses and positive attitudes about providing social support to the mentally ill) so a three latent factor model was considered optimal: one for the overall attitude subscale, and one for each of the two theoretical constructs.

For the 7-item causal attribution subscale of the MHAQ, two eigenvalues were greater than 1. Two constructs were considered when constructing the subscale (biological and non-biological causes of mental illnesses), so a three latent factors model was considered optimal: one for the overall causal attribution subscale and one for each of the two theoretical constructs.

Confirmatory factor analysis

The six-factor model of MHKQ was examined in the second half of the sample using confirmatory factor analysis. As shown in panel A of Figure 2, factor loadings of the 25 items ranged from 0.58 to 1.32 and those of the five latent items ranged from 0.36 to 0.51. The CFI was .756 and the TLI was .734; since both values were less than .900, this indicates that the goodness of f it of the model needs to be improved. However, the RMSEA was .047 (90% CI = .044, .051), which indicated optimal fit (χ² = 1015.36, df = 275). With the goal of modifying the model to improve its performance, we considered the correlations between pairs of items in each factor and the correlations between the five latent factors; we then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 2, the correlations between the items that were included in the revised model ranged from .03 to .34, and the correlations between the latent variables that were included in the revised model ranged from −.19 to −.34. After modifying the model in this manner, factor loadings of the 25 items ranged from 0.53 to 1.38 and those of the five latent items ranged from 0.37 to 0.55; the CFI (.861) and TLI (.839) improved, though they remained below the target value of .900; the RMSEA also improved (.037, 90% CI = .033, .040), indicating optimal model fit (χ² = 682.86, df = 260).

Figure 2.

Pathway diagram of the confirmatory factor analysis of the six-factor model for the Mental Health Knowledge Questionnaire (MHKQ) as assessed when using the 25 item scores and five latent factors as observed variables (panel A) and when using the 25 item scores and the correlations of items and the correlations of latent factors as observed variables (panel B).The factors shown in the ovals are the latent factors relating to mental health literacy: factor 1, awareness of the treatment and prevention of psychological problems; factor 2, awareness of alcohol-related problems; factor 3, awareness of types of problems considered mental health problems; factor 4, awareness of the problem of suicide in China; factor 5, knowledge about depression and its relationship to physical health; and factor 6, composite measure of mental health literacy. The items shown in the rectangles are the observed variables. The variance of factor 6 and the first item of each of the latent factors are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from latent factor 6 to the five other latent factors and from these latent factors to the observed variables (“items”). (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable). The “unique factors” (which relate to a single observed variable and to a single latent variable), the fractions to the left of the rectangles and below the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In the bottom panel the correlations of items and those of latent variables are added as new observed variables in the model.

The three-factor model for the 14-item attitude subscale of MHAQ was examined in the second half of the sample using confirmatory factor analysis. As shown in the Panel A of Figure 3, factor loadings of the 14 items ranged from 0.21 to 1.41 and those of the two latent items were 0.40 and 0.47. The CFI was .804 and the TLI was .768, both less than .900, indicating that the goodness of fit of the model needs improvement. The RMSEA was .075 (90% CI = .070, .081) which indicated suboptimal fit (χ² = 601.81, df = 77). To modify the model, we first considered the correlations between pairs of items in each factor and the correlations between the two latent factors and then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 3, the item correlations included in the model ranged from .05 to .24 and the correlation of the two latent items (included in the model) was −.05. After modifying the model in this manner, factor loadings of the 14 items ranged from 0.14 to 1.40 and those of the two latent items were 0.43 and 0.48; all fit indices of the revised model suggested excellent fit (RMSEA = .045, 90% CI = .039, .052; CFI = .940 and TLI = .917; χ² = 226.67, df = 66).

Figure 3.

Pathway diagram of the confirmatory factor analysis of the three-factor model for the attitude subscale of the Mental Health Attitude Questionnaire (MAKQ) as assessed when using the 14 item scores and two latent factors as observed variables (panel A) and when using the 14 item scores and the correlations of items as observed variables (panel B). The factors shown in the ovals are the latent factors for the model, factor 1 is the latent variable of negative attitudes about mental illnesses and persons with mental illnesses, factor 2 is the latent variable of positive attitudes about providing social support to persons with mental illnesses, and factor 3 is the latent variable of the attitude to mental illness. The items shown in the rectangles are the observed variables. The variance of factor 3 and the first item of each latent factor are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from the latent variables to the observed variables and from the latent factor 3 to the other latent factors. (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable or the variance of other latent variable explained by the latent variable 3.) The “unique factors” (which relate to a single observed variable and to a single latent variable) to the left of the rectangles and the bottom of the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In panel B the correlations of Items and the correlations of two first-order latent factors are added as new observed variables in the model.

The three-factor model for the MHAQ causal attribution subscale was also assessed in the second half of the sample using confirmatory factor analysis. As shown in Panel A of Figure 4, factor loadings of the seven items ranged from 0.981 to 1.31 and those of the two latent items were 0.47 and 0.41. The CFI was .936 and the TLI was .904; both were greater than .900, indicating optimal model goodness of fit. The RMSEA was .068 (90% CI = .055, .081), which indicated suboptimal fit (χ² = 91.43, df = 14). To improve the model, we first considered the correlations between pairs of items in each factor and the correlations between the two latent factors and then decided whether or not to include each specific correlation in the model based on the size of the correlation and the conceptual similarity of the correlated items or the correlated latent factors. As shown in panel B of Figure 4, the item correlations included in the model ranged from −.07 to .15 and the correlation of the two latent items was not included in the model. After modifying the model in this manner, factor loadings of the seven items ranged from 1.00 to 1.48 and those of the two latent items were 0.37 and 0.48; the RMSEA improved but remained suboptimal (RMSEA = .054, 90% CI = .039, .069); and CFI (.968) and TLI (.940) both indicted optimal model fit (χ² = 49.13, df = 11).

Figure 4.

Pathway diagram of the confirmatory factor analysis of the three-factor model for the causal attribution subscale of the Mental Health Attitude Questionnaire (MAKQ) as assessed when using the 7-item scores and two latent factors as observed variables (panel A) and when using the 7-item scores and the correlations of items as observed variables (panel B). The factors shown in the ovals are the latent factors for the model, factor 1 is the latent variable of biomedical causes of mental illnesses, factor 2 is the latent variable of non-biomedical causes of mental illness, and the factor 3 is the latent variable of beliefs about the causes of mental illness. The items shown in the rectangles are the observed variables. The variance of factor 3 and the of first item of each first-order latent variable are set at 1 to allow the factor loadings to be freely estimated. The standardized factor loadings are shown on the arrows from the latent variables to the observed variables and from the latent factor 3 to the other latent factors. (The square of each factor loading is the “communality,” the proportion of the variance of the observed variable explained by the latent variable or the variance of other latent variable explained by the latent variable 3.) The “unique factors” (which relate to a single observed variable and to a single latent variable) to the left of the rectangles and the bottom of the ovals account for measurement error and any other sources of variance not accounted for by the latent variable. In panel B the correlations of items are added as new observed variables in the model.

Test-retest reliability

The results of the 25 items in MHKQ, of the 14-item attitude subscale in MHAQ, and of the 7-item causal attribution subscale in MHAQ were converted into overall scores ranging from 0 to 100. The test-retest reliability of the three overall scores in the 188 participants who completed the scale twice and for different demographic subgroups of these respondents are shown in Table 5. The ICC of the MHKQ was .40 in all participants and—with the exception of the unexpected ICC of .00 (p = .981) for the 36 Hui respondents who completed the test-retest assessment—ranged from .31 to .47 in the different subgroups of respondents. In the total sample and in most (but not all) subgroups the mean MHAQ score increased slightly at the time of the second assessment, suggesting a slight increase in awareness about mental illness, but the difference was not statistically significant (paired t-test for all participants was .57, p = .566). The ICC of the attitude subscale of MHAQ was .48 in all participants and ranged from .41 to .66 in the different subgroups of respondents; for the total sample and all subgroups the mean score increased over time, indicating more positive attitudes at the time of the second administration (paired t-test for all participants was 2.29, p = .023). The ICC of the causal attribution subscale of MHAQ was .26 in all participants and ranged from .12 to .35 in the different subgroups of respondents; in the total sample and most subgroups the mean score increased slightly over time, suggesting that respondents were more likely to attribute mental illnesses to biological causes at the time of the second assessment (paired t-test for all participants was 2.59, p = .010).

Table 5.

Mean (SD) total scores of Mental Health Knowledge Questionnaire (MHKQ) and Mental Health Attitude Questionnaire (MHAQ) and intraclass correlation coefficient (ICC) in different groups of participants who completed the instrument twice.

	Score for the first assessment		Score for the repeat assessment
Subgroup of respondents	Mean	SD	Mean	SD	ICC
25-item MHKQ^a
All participants (n = 188)	55.4	10.3	55.8	10.0	.40
Male participants (n = 69)	56.5	9.7	56.2	9.5	.41
Female participants (n = 119)	54.8	10.6	55.6	10.2	.39
Han participants (n = 150)	56.3	10.3	56.6	10.2	.45
Hui participants (n = 36)	51.0	8.9	52.7	8.3	.00
Participants living in urban counties (n = 95)	56.9	9.6	57.1	10.0	.47
Participants living in rural counties (n = 93)	53.9	10.8	54.5	9.8	.31
Participants with 9 + years of education (n = 76)	57.2	9.7	55.5	9.0	.38
Participants with < 9 years of education (n = 112)	54.3	10.5	56.0	10.6	.41
14-item attitudinal subscale of MHAQ^b
All participants (n = 188)	79.3	9.5	80.9	9.3	.48
Male participants (n = 69)	78.4	10.5	79.7	8.5	.51
Female participants (n = 119)	79.8	8.9	81.7	9.7	.46
Han participants (n = 150)	79.8	9.4	80.8	9.6	.44
Hui participants (n = 36)	77.9	9.7	81.5	7.7	.64
Participants living in urban counties (n = 95)	78.5	9.9	80.4	9.5	.45
Participants living in rural counties (n = 93)	80.2	9.1	81.5	9.2	.51
Participants with 9 + years of education (n = 76)	77.0	9.5	77.9	8.8	.47
Participants with < 9 years of education (n = 112)	80.7	9.3	83.0	9.1	.41
7-item causal attribution subscale of MHAQ^c
All participants (n = 188)	74.8	10.6	76.5	10.3	.26
Male participants (n = 69)	75.0	9.7	77.8	8.4	.12
Female participants (n = 119)	74.8	11.1	75.8	11.3	.32
Han participants (n = 150)	74.7	10.1	77.0	9.9	.24
Hui participants (n = 36)	74.8	12.3	74.5	11.7	.28
Participants living in urban counties (n = 95)	75.2	9.2	76.7	9.3	.19
Participants living in rural counties (n = 93)	74.4	11.8	76.4	11.3	.31
Participants with 9+ years of education (n = 76)	76.3	10.0	76.8	9.3	.35
Participants with <9 years of education (n = 112)	74.0	10.9	76.4	11.0	.21

^aHigher scores represent greater knowledge about mental illnesses; paired t-test of before versus after total score in all participants = 0.57 (p = .566).

^bHigher scores represent more positive, supportive attitudes about the mentally ill; paired t-test of before versus after total score in all participants = 2.29 (p = .023).

^cHigher scores represent greater support for biological versus non-biological causes of mental illnesses; paired t-test of before versus after total score in all participants = 2.59 (p = .010).

Discussion

Assessing the influence of mental health literacy and stigmatization of people with mental illness on the development and utilization of mental health services is an important component of understanding the effectiveness (or lack of effectiveness) of policies aimed at improving mental health. Given the very different social, economic, and health care environments of high-income countries and low- and middle-income countries (LMICs), LMICs must develop their own culture-sensitive methods for assessing mental health literacy and stigmatization. This involves identifying the specific types of information and attitudes about mental health that are most pertinent to the target culture and deciding which stakeholders in the setting hold the “correct” knowledge about these topics. This is a complex task that requires the integration of high-quality qualitative and quantitative methods, so few LMICs have developed fully validated instruments for doing this. A recent systematic review of the quality of instruments developed to assess mental health knowledge (Wei, McGrath, Hayden, & Kutcher, 2016) identified very few high-quality studies in Western countries and almost none in non-Western countries.

This study reports on one attempt to do this in China. We started with focus groups among key stakeholders, generated preliminary versions of two instruments, conducted a pilot test of the preliminary instruments, and then conducted a relatively large community-based survey with revised versions of the instruments. The instruments assessed knowledge about mental illnesses in five dimensions (classification of specific conditions as psychological problems, characteristics of depression, alcohol use and addiction, suicide, and treatment/prevention of psychological problems), attitudes about the mentally ill in two dimensions (negative and positive attitudes), and causal attributions of mental illnesses in two dimensions (biological and non-biological causes). We found that the internal consistency of the items in the MHKQ and in the two subscales of the MHAQ were satisfactory (though only marginally so for the two MHAQ subscales) and that the indices of model fit for the three measures were generally strong (though not optimal in all cases). However, the test-retest reliability of the total scores for the three measures—particularly the causal attribution subscale of the MHAQ—was relatively poor.

The few studies that have assessed test-retest reliability of mental health knowledge and attitudes in Western countries (Svensson et al., 2011) also report poor test-retest reliability, suggesting that this may be a general weakness of instruments assessing knowledge and attitudes in the general public. In our study the very short 2- to 3-day time between administrations (dictated by the practicalities of travelling to 20 study sites around the province) may have been a factor. The learning effect of the first administration may have persisted at the time of the second administration, resulting in higher scores for knowledge, more positive attitudes, and an increase preference for biological causal models of mental illnesses. An interval of 2 to 3 weeks would have been preferable.

Other limitations need to be considered when evaluating these results. We assessed the instruments in a large community-based sample of adults from one province in China so the results are relevant for this population, but they may not be relevant for other parts of China or for adolescents. Due to the low educational level of respondents we had interviewers administer the scales; self-completion versions of the scales may have different psychometric properties. The very low test-retest reliability of the MHAQ for Hui respondents (ICC = .00) may have been a statistical outlier, but this needs to be repeated in a separate sample to clarify the reason for this surprising finding. We have combined two different dimensions into a summary measure of attitudes about mental illnesses and two different dimensions into a summary measure of causal attributions of mental illnesses; this assumes that the dimensions are conceptually related, which may not be the case. The modifications of the final models for the three scales were based on the characteristics of responses in the second half of the sample used for the confirmatory factor analysis; they should be validated by testing the modified models in an independent sample. Finally, we did not assess the construct validity of the scales (e.g., by comparing mental health literacy between community members and mental health providers) or the responsiveness of the scale scores to changes in knowledge or attitudes (e.g., before and after an intensive public mental health education campaign).

Despite these limitations, the satisfactory internal consistency and generally strong indices of model fit for the MHKQ and the two subscales of the MHAQ support further development of the scales. Subsequent work with the scales should (a) reassess test-retest reliability at longer intervals; (b) assess construct validity and sensitivity to change; (c) compare self-completion versus interview-completed versions of the scales; (d) conduct detailed analyses of the relationship of the different dimensions of each scale; and (e) expand the use of the instruments to other parts of China and to younger respondents. After completion of these steps, final versions of the scales (with somewhat different content for different types of respondents) can be employed to help develop and assess the outcome of targeted mental health campaigns in China that have the following objectives: expanding perceptions about what constitutes a “mental illness”; increasing the care-seeking of individuals with these problems; decreasing the stigma associated with mental illnesses; and increasing the willingness of community members, institutions, and government agencies to expend the resources required to provide mentally ill individuals with the help they need.

Footnotes

Acknowledgments

The authors thank the students in the School of Public Health at Ningxia Medical University and staff members of the Ningxia Center for Disease Control and Prevention for their assistance in the conduct of the study. Hanhui Chen and Zhizhong Wang are co-first authors.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the China Medical Board (project number: 11-063).

Ethics approval

The study was approved by the institutional review board of Ningxia Medical University (No. 2013-167).

Note

Hanhui Chen, M.D. is a psychiatrist at the Shanghai Mental Health Center affiliated with the Shanghai Jiao Tong University School of Medicine. His academic and research publications focus on global mental health, addiction, and group therapy. He is currently helping to coordinate a study funded by an RO-1 grant about cognitive functioning in persons with serious mental disorders and helping in the translation and adaptation of the Chinese version of the SCID-5.

Zhizhong Wang, PhD., is Associate Professor and Vice Director of the Department of Epidemiology and Statistics at the School of Public Health, Ningxia Medical University. He completed medical school at the Fourth Military Medical University in China and received his PhD degree in 2009. His academic interests and publications are focused on social psychiatry, community mental health, and the relationship of religion and ethnicity to mental health in mainland China.

Michael R. Phillips, C.M., M.D., M.A., M.P.H., is a Professor of Psychiatry at the Shanghai Mental Health Center of the Shanghai Jiao Tong University School of Medicine. His research and publications focus on mental health epidemiology and suicide. He runs a number of research training courses each year, supervises Chinese and foreign graduate students, and helps coordinate WHO mental health activities in China.

References

Carmines

E. G.

Zeller

R. A.

(1979) Reliability and validity assessment, Beverly Hills, CA: Sage.

Chen

H. H.

Phillips

M. R.

Cheng

Chen

Q. Q.

Chen

X. D.

Fralick

Bueber

(2012) Mental health law of the People's Republic of China [English translation with annotations]. Shanghai Archives of Psychiatry 24(6): 305–321.

Chen

H. H.

Wang

Z. Z.

Phillips

M. R.

Sun

Y. L.

Cheng

G. H.

(2014) Internal consistency and test-retest reliability of the Chinese version of the 5-item Duke University Religion Index. Shanghai Archives of Psychiatry 26(4): 300–309.

Cortina

J. M.

(1993) What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology 78: 98–104.

Dietrich

Mergl

Freudenberg

Althaus

Hegerl

(2010) Impact of a campaign on the public's attitudes towards depression. Health Education Research 25: 135–150.

Dumesnil

Verger

(2009) Public awareness campaigns about depression and suicide: A review. Psychiatric Services 60: 1203–1213.

Furnham

Lousley

(2013) Mental health literacy and the anxiety disorders. Health 5(3): 521–531.

Gao

S. Y.

Phillips

M. R.

(2001) Attitudes about mental illness of different types of respondents in Beijing. Chinese Mental Health Journal 15(2): 107–109. [Chinese].

Gao

S. Y.

Phillips

M. R.

Wang

X. Q.

Jia

Z. M.

Gao

W. C.

D. G.

(2005) Experience of stigma among patients with schizophrenia and their family members. Chinese Mental Health Journal 19: 82–85. [Chinese].

10.

GBD 2013 DALYs and HALE Collaborators (2015) Global, regional, and national disability-adjusted life years (DALYs) for 306 diseases and injuries and healthy life expectancy (HALE) for 188 countries, 1990–2013: Quantifying the epidemiological transition. Lancet 386: 2145–2191.

11.

Griffiths

K. M.

Nakane

Christensen

Yoshioka

Jorm

A. F.

Nakane

(2006) Stigma in response to mental disorders: A comparison of Australia and Japan. BMC Psychiatry 6: 21.

12.

Hickie

(2004) Can we reduce the burden of depression? The Australian experience with beyond blue: The national depression initiative. Australasian Psychiatry 12: S38–S46.

13.

Bentler

P. M.

(1999) Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 6(1): 1–55.

14.

Huang

G. M.

Yin

H. F.

Tian

H. J.

(2015) The awareness of mental health and its associates: A cross-sectional survey in Tianjin community inhabitants. Journal of International Psychiatry 42(4): 23–26.

15.

Joa

Johannessen

J. O.

Auestad

Friis

McGlashan

Melle

Larsen

T. K.

(2008) The key to reducing duration of untreated first psychosis: Information campaigns. Schizophrenia Bulletin 34: 466–472.

16.

Jorm

A. F.

(2000) Mental health literacy: Public knowledge and beliefs about mental disorders. British Journal of Psychiatry 177: 396–401.

17.

Kline

(2000) The handbook of psychological testing, 2nd. London, UK: Routledge.

18.

Kurihara

Kato

Sakamoto

Reverger

Kitramura

(2000) Public attitudes towards the mentally ill: A cross- cultural study between Bali and Tokyo. Psychiatry and Clinical Neuroscience 54: 547–552.

19.

Xiao

S. Y.

Huang

Z. P.

Shi

J. G.

Zheng

Z. H.

Luo

W. F.

et al. (2009) Mental health literacy in three cities of China: A survey study. Chinese Mental Health Journal 23(12): 883–887. [Chinese].

20.

Meng

G. R.

Yao

X. W.

Zhu

Z. Q.

Zhang

M. Y.

(2002) Investigation of knowledge about mental health in Shanghai residents: Analysis with 2697 respondents. Shanghai Archives of Psychiatry 14: 56–57. [Chinese].

21.

Ministry of Health. (2010). Notification of survey indicator for mental health assessment by general office of the Ministry of Health. Retrieved from http://www.moh.gov.cn/jkj/s5888/201003/c6487d3f272e4e7b88d30347ff6a2299.shtml [Chinese].

22.

Ministry of Health, Ministry of Civil Affairs, Ministry of Public Security, and China Disabled Persons Federation (2003) Mental Health Plan of China for 2002–2010. Shanghai Archives of Psychiatry 15(2): 125–128. [Chinese].

23.

Ministries of Health, Publicity Department of the Communist Party of China, National Development and Reform Commission, Ministry of Education, Ministry of Public Security, Ministry of Civil Affairs, et al. (2008). Instruction summary on mental health system development of China for 2008–2015. Retrieved from http://www.gov.cn/gzdt/2008-02/02/content_879701.htm [Chinese].

24.

Phillips

M. R.

Zhang

J. X.

Shi

Q. C.

Song

Z. Q.

Ding

Z. J.

Pang

S. T.

Wang

Z. Q.

(2009) Prevalence, associated disability and treatment of mental disorders in four provinces in China, 2001–2005: An epidemiological survey. Lancet 373: 2041–2053.

25.

Pinto-Foltz

M. D.

Logsdon

M. C.

Myers

J. A.

(2011) Feasibility, acceptability, and initial efficacy of a knowledge-contact program to reduce mental illness stigma and improve mental health literacy in adolescents. Social Science & Medicine 72: 2011–2019.

26.

Svensson

Markstrom

Bejerholm

Bjorkman

Brunt

Eklund

Ostman

(2011) Test-retest reliability of two instruments for measuring public attitudes towards persons with mental illness. BMC Psychiatry 11: 11.

27.

Wang

J. Y.

Y. L.

Jiang

Cai

Wang

W. L.

Zheng

Q. Z.

Zhang

(2013) Mental health literacy among residents in Shanghai. Shanghai Archives of Psychiatry 25(4): 224–234.

28.

Wang

Kuang

Chen

J. M.

Gan

Zeng

et al. (2011) Prevalence of social distance and restriction among college students with suicide attempts in Chongqing. Chinese Journal of Epidemiology 32(4): 341–345. [Chinese].

29.

Wei

Y. F.

McGrath

P. J.

Hayden

Kutcher

(2016) Measurement properties of tools measuring mental health knowledge: A systematic review. BMC Psychiatry 16: 297.

30.

WHO (2013) Mental Health Action Plan 2013–2020, Geneva: Switzerland: Author.

31.

Xiong

Phillips

M. R.

(2016) Translated and annotated version of the 2015–2020 National Mental Health Work Plan of the People's Republic of China. Shanghai Archives of Psychiatry 28(1): 4–17.

32.

G. M.

Yin

H. F.

Yang

G. F.

Tian

H. J.

(2014) Survey of mental illness stigma in community population in Tianjin City. Chinese Mental Health Journal 28(2): 97–103. [Chinese].

33.

Yao

M. J.

Yan

X. F.

Chang

Z. J.

Liu

Yan

Gao

Q. J.

(2013) Awareness rate of mental health knowledge in residents in Zhengzhou City. Chinese Mental Health Journal 27(9): 682–685. [Chinese].

34.

Zhang

H. W.

(2011) Investigation on the awareness rate for mental knowledge, attitude and treat method to psychosis patients. China Journal of Health Psychology 19(11): 1307–1309. [Chinese].