Abstract
Early detection of child mental health problems in schools is critical for implementing strategies for prevention and intervention. The development of an effective measure of mental health and well-being for this context must be both empirically sound and practically feasible. This study reports the initial validation of a brief self-report measure for child mental health suitable for use with children as young as 8 years old (“Me & My School” [M&MS]). After factor analysis, and studies of measurement invariance, 2 subscales emerged: emotional difficulties and behavioral difficulties. These 2 subscales were highly correlated with corresponding constructs of the Strengths and Difficulties Questionnaire (SDQ) and showed correlations with attainment, deprivation, and educational needs similar to ones obtained between these demographic measures and the SDQ. Results suggest that this school-based self-report measure is psychometrically sound, and has the potential of contributing to school mental health surveys, evaluation of interventions, and recognition of mental health problems within schools.
Early detection and assessment of child mental health problems are critical for implementing strategies for prevention and intervention (Weist, Rubin, Moore, Adelsheim, & Wrobel, 2007). Schools provide a key setting, common across children, for early recognition and intervention (Massey, Armstrong, Boroughs, Henson, & McCash, 2005; Roeser, Eccles, & Strobel, 1998). This focus on the school as a setting for intervention necessitates routine and widespread collection of information about child mental health and well-being (Levitt, Saka, Romanelli, & Hoagwood, 2007). The development of an effective measure of mental health and well-being for this context relies heavily on two key principles: it must be both empirically sound and practically feasible.
Two key elements contribute to the feasibility: who provides the information and compromises between breadth and brevity. In terms of who provides information, child self-report may be the most viable because there are practical constraints on the repeated use of teachers’ time and the ability to gain sufficient response rates from parents. Furthermore, both parents and teachers tend to be less accurate in their assessment of emotional than behavioral difficulties (e.g., Tremblay, Vitario, Gagnon, Piche, & Royer, 1992).
A systematic review identified 113 child mental health assessment measures (Wolpert et al., 2008), but no self-report assessments of general mental health (emotional and behavioral) were identified for children below the age of 11 that could be used as a brief community-based screening measure.
There are mixed views in relation to children’s self-ratings of well-being, with particular concern in the literature that younger children may not be reliable informants about their own mental health (Van Roy, Veenstra, & Clench-Aas, 2008). However, analysis of extant measures (Wolpert et al., 2008) indicated that a likely reason for the limitations of self-report measures for this age group may be that measures developed for older children use inappropriate language for younger children or those with low levels of reading and/or language skills. Evidence suggests that children as young as 6 years can reliably self-report if an age appropriate measure is used (Riley, 2004), especially where measures are developed specifically for this age group and used in community settings (Muris, Meesters, Eijkelenboom, & Vincken, 2004). An underutilized area in the administration of these types of measures to children, which may facilitate use with younger children, is the use of computers with audiofeeds to reach those whose reading skills are poor.
In order to capture community-wide prevalence and trends in common mental health and well-being difficulties, measures need to be sufficiently brief to allow for routine use, yet be sufficiently wide ranging to cover broad categories of the most common psychological difficulties. The most commonly reported mental health difficulties are either behavioral or emotional (Green, McGinnity, Meltzer, Ford, & Goodman, 2005; Levitt et al., 2007). These are also the domains most commonly covered by existing measures of mental health and well-being such as the Achenbach System of Behaviour Assessment (ASEBA; Achenbach & Rescorla, 2001) and the Strengths and Difficulties Questionnaire (SDQ; Goodman, 1997). In terms of empirical soundness, the scores derived from a range of measures of child mental health and well-being have shown good reliability in both clinical and community settings and there is evidence for the possibility of valid interpretation (e.g., SDQ, Goodman, 1997; ASEBA, Achenbach & Rescorla, 2001). Some of these have even adopted approaches to validation using new, more advanced psychometric techniques, such as Item Response Theory (IRT; Chorpita et al., 2010). However, few of these measures have been developed specifically for the school setting and none of these existing instruments, to the authors’ knowledge, offer a self-report measure for children under the age of 11 that is both brief and free.
Aims
The aim of this article is to present the development and initial validation of a brief self-report measure for child mental health and well-being, as indexed by emotional and behavioral difficulties, suitable for use with children as young as 8 years old. Specifically it describes the constructs underlying responses to the measure, score reliability, studies of measurement invariance in regard to several subgroups, construct validity, as well as the development of clinical cutoff scores based on an established mental health measure.
Materials and Method
Sample
Data were collected in 2008 from pupils who attended state schools in 25 local areas across England. The analyses reported are based on surveys completed by 9,814 pupils aged 8 to 9 years (school year 4; 51.4% male) from 311 primary schools and 9,881 pupils aged 11 to 12 years (school year 7; 49.8% male) from 82 secondary schools. The initial sample was not drawn to be representative of all school children in England; it was based on each local area’s own selection of schools to be involved in a wider national program of child mental health provision in schools (Department for Children, Schools and Families(DCSF, 2008)).
The average academic attainment for children in the sample was slightly lower than the national average (primary schools: national average = 15.30, sample average = 14.,84, SD = 3.63; secondary schools: national average = 27.70, sample average = 27.24, SD = 4.52). They also had a slightly elevated level of deprivation, as measured by the Income Deprivation Affecting Children Index (IDACI) scores, compared to the national figures (primary schools: national average = 0.24, sample average = 0.291, SD = 0.19; secondary schools: national average = 0.22, sample average = 0.28, SD = 0.19). Finally, there were similar proportions of children belonging to the “White British” ethnic category compared to national figures (primary schools: national percentage = 73.8%, sample percentage = 74.1%; secondary schools: national percentage = 77.3%, sample percentage = 76.8%; Department for Education, 2010).
Procedure
Children completed questionnaires using a secure online system during their usual school day with parent consent. Teachers explained to participating children what the questionnaire was about, the confidentiality of their answers and their right to decline participation. The online system was designed to be easy to read and child-friendly; the font size was large and the instructions and individual questions were presented slowly to allow for less accomplished readers. For younger children, recorded spoken accompaniment for all instructions, questionnaire items and response options was also provided. Parents were also invited to complete a questionnaire about their child.
Measures
Me & My School (M&MS)
The M&MS measure was developed to cover two broad domains: emotional difficulties and behavioral difficulties. Based on a review of outcome measures (Wolpert et al., 2008) and an analysis of key concepts covered by the emotional and behavioral scales of other measures a large pool of items was generated keeping in mind lower reading age and usability by younger age groups. This larger pool of items was piloted in focus groups with children to establish which terms and concepts younger children used and understood. This process resulted in an initial pool of 24 items included in the online questionnaire (see Table 1). Items consisted of short behavioral statements to which children responded using the response options “never,” “sometimes,” or “always.” The items were converted to an online survey, designed to be visually clear and appealing to children.
EFA Rotated Loadings, CFA Standardized Loadings, and Correlations Between Factors.
Note. Both EFA and CFA were performed on polychoric correlations using diagonally weighted estimator. Rotation method in EFA: Geomin. All loadings in CFA and factor correlations in both EFA and CFA are significant at p < .000.
The Strengths and Difficulties Questionnaire
The SDQ (Goodman, 1997) is a widely used and well-validated mental health measure with subscales including behavioral and emotional problems and validated cut-offs indicating thresholds distinguishing children who are likely to require clinical intervention and those who are not (Goodman, Meltzer, & Bailey, 1998). As this measure is for children aged 11 years and over, it was only completed by children who were 11-12 years in the current sample.
Additional Variables
Additional variables were drawn from the English National Pupil Dataset, a government dataset holding records on all pupils in England.
Academic attainment
Academic attainment was recorded based on children’s performance in nationally mandated assessments and the scores achieved for the most recent assessment were used for each age group. Scores were based on each child’s attainment in three core subject areas (English, mathematics, and science), high scores representing high attainment.
Deprivation
Deprivation scores were defined using the deprivation score associated with the child’s home postcode, known as the IDACI (McLennan, Barnes, Noble, Davies, Garratt & Dibben, 2011). IDACI scores are on a metric between 0 and 1 with higher scores reflecting higher levels of deprivation.
Special Educational Needs (SEN)
SEN were based on the school’s assignment of a child to a level of special educational needs. These were coded on a four-point scale as follows: 0 = No statement of SEN, 1 = School Action, 2 = School Action Plus, and 3 = Statement of SEN. Therefore, the higher the score, the greater the educational needs.
Analyses
Analyses were carried out in several stages. First, exploratory factor analysis (EFA) was performed on a randomly selected half of the total sample, in order to determine which items constituted coherent subscales of emotional and behavioral difficulties. The items with most salient factor loadings and least cross-loadings were selected and confirmatory factor analysis (CFA) confirming this item assignment was performed on the other half of the sample. Second, differential item functioning (DIF) analyses were used to confirm whether the items were suitable for use across a range of demographic groupings. Third, Cronbach’s α was used to assess internal consistency of the derived scores based on the retained items. Fourth, validity of the scores was investigated by correlating the emotional and behavioral subscales with preexisting mental health scales and other theoretically related constructs, such as academic attainment and deprivation. Finally, cutoffs indicating those at high risk of mental health problems were established for the new measure by equipercentile equating to cutoffs on a preexisting well validated instrument (SDQ).
Results
Factor Structure
EFA and parallel analysis (Hoyle & Duvall, 2004) carried out on the polychoric correlations of the three-category items (Muthén & Kaplan, 1985) on half of the sample (n = 9,837) suggested presence of three factors (first four eigenvalues were 6.87, 2.86, 1.81, and 1.02). Goodness of fit of the three-factor solution was good (CFI = 0.955; TLI = 0.940; RMSEA = 0.044; SRMR = 0.035), and substantially better than a two-factor solution (CFI = 0.898; TLI = 0.877; RMSEA = 0.064; SRMR = 0.058). An oblique rotation of the factors yielded a solution presented in Table 1. The items designed to measure behavioral difficulties clustered together yielding high loadings (Factor III). The emotional difficulties items also largely loaded on a single factor (Factor I), although some were more clearly linked to the second factor. The second factor (Factor II) was made up of a small set of items related to social aspects of child’s life such as friendships, which were initially developed to form part of the emotional difficulties subscale. The correlations between factors were positive but low, confirming that they represented related but conceptually distinct constructs.
CFA was carried out on the second half of the sample (n = 9,858). We tested a model with two correlated factors indicated by the items identified in EFA as belonging to the emotional and behavioral difficulties (bold items under Factors I and III in Table 1). The items were hypothesized to indicate only one factor (independent clusters structure; McDonald, 1999). The model’s goodness of fit (CFI = 0.934; TLI = 0.924; RMSEA = 0.060) was acceptable (Hu & Bentler, 1999). The correlation between the two latent factors was 0.42 (p < .001). The standardized factor loadings from this model are presented in Table 1.
Differential Item Functioning (DIF)
DIF analysis has been used to inform the selection of suitable items that operate equivalently across a range of different subgroups of school children. Five grouping criteria were examined: gender, SEN, whether English was the child’s second language (EAL), whether the child received free school meals (FSM), and whether the child was in care. Girls (49.5%), children with SEN (1.9% statemented), nonnative English speakers (17.5%), children receiving FSM (21%), and children in care (0.5%) were the focus of these investigations (formed the focal groups in the DIF analyses). DIF analyses compare the item endorsement rates in the focal group compared to the reference group (e.g., children in care vs. all other children), conditioning on the test score. An item is said to display DIF if children with the same test score but belonging to different groups have different probabilities of endorsing the item.
The statistical approach taken was the Liu–Agresti common log odds ratio (L-A LOR; Liu & Agresti, 1996), a nonparametric Mantel and Haenszel-type estimator. The L-A LOR relies on the log odds ratio of one group selecting a particular response option relative to the other group, stratified by overall level of the measured construct. Both constructs (emotional and behavioral difficulties) were examined using this method. Table 2 shows results of DIF analyses performed with DIFAS 5.0 (Penfield, 2005). Positive L-A LOR values indicate the item is more difficult to endorse for the focal group; negative values indicate that the item is easier to endorse, given the same level of the construct. L-A LOR values are printed in the table only if (1) they are statistically significant at the 0.05 level; and (2) LOR is at least moderate in size (|L-A LOR| > 0.43; Penfield, 2007).
Differential Item Functioning for the M&MS Scales.
Note. EAL = English as an additional language; FSM = free school meals; SEN = special educational needs.
Very few cases are identified as “In Care”; results presented are significant but might be unstable.
Analysis indicated that “Other children tease me” and “I bully others” were the most problematic items, both in terms of the magnitude and the number of groupings showing DIF. Gender differences were identified for some emotional items but were not considered sufficient alone to warrant removal of items since gender differences in presentation of emotional distress are well known (Rutter, Caspi, & Moffitt, 2003). Boys, those with SEN and those in care, were more likely to agree that other children teased them than were others with similar levels of emotional difficulties. Also, those with SEN, EAL, and in care were more likely to agree that they bullied others than were others with similar levels of behavioral difficulties. Based on this, both of these items were removed from the final scale, resulting in a 10-item scale of emotional difficulties and a 6-item scale of behavioral difficulties (Table 2).
Internal Consistency
Cronbach’s α for the two resulting scales was good (behavioral difficulties: α = 0.78 and 0.80 for Years 4 and 7, respectively; emotional difficulties: α = 0.72 and 0.77). In comparison, the α for the corresponding SDQ subscales in the Year 7 sample was slightly lower (emotional symptoms: α = 0.72; conduct problems: α = 0.68).
Validity
The construct validity of the scores derived from the M&MS measure was assessed by considering the convergent and divergent validity of each subscale against the relevant subscales of the SDQ. Additional validity evidence was gained by considering behavioral and emotional difficulties with theoretically related constructs of academic attainment, deprivation, and SEN.
Construct Validity
As the self-report SDQ can only be used for children aged 11 years or over, correlations between the M&MS scales and the self-report SDQ subscales could only be computed for the older age group (Table 3). In this group, correlations between the emotional and behavioral M&MS scales, and the corresponding self-report SDQ subscales were high (r = .67, p < .001; r = .70, p < .001). It is also notable that the correlations between the M&MS subscales and the noncorresponding self-report SDQ subscale were much lower (r = .22, p < .001; r = .24, p < .001), suggesting good discriminant validity.
Correlations Between M&MS and SDQ Subscales for Children Aged 11 to 12 (Year 7).
Note. *p < .05. **p < .01.
Sample size for child–parent/parent–parent assessments varied from 579 to 655, sample size for child–child assessments varied from 8,138 to 9,324.
Correlations between M&MS scales and the corresponding parent report SDQ subscales (Tables 3 and 4) were lower but remained statistically significant and were comparable to correlations observed between the SDQ self-report and parent-report, where these data were available (i.e., for the older group).
Correlations Between M&MS and SDQ Subscales for Children Aged 8 to 9 (Year 4).
Note. *p < .05. **p < .01.
Sample size for child–parents/parent–parent assessments varied from 609 to 675, sample size for child–child assessments was 8,300.
Additional Validity Evidence
Three variables documented to have a moderately strong relationship with emotional and behavioral difficulties were considered: academic attainment, extent of SEN, and deprivation (Masten et al., 2005; Meltzer, Gatward, Goodman, & Ford, 2003; Reijneveld et al., 2010). Consistent with the literature, each of these variables correlated more highly with the two behavioral scales than the emotional scales (see Table 5). In each instance higher levels of emotional or behavioral difficulties were associated with lower academic attainment, greater extent of SEN, and higher deprivation. Correlations between these variables and the M&MS subscales were statistically significant due to large samples used, but small in magnitude. These relationships are, however, consistent with correlations observed for the SDQ subscales (see Table 5).
Correlations Between M&MS and Conceptually Related Variables.
Note. *p < .05. **p < .01.
Sample size varied from 8,476 to 9,246.
Establishing Clinical Cutoffs
Clinical cutoffs were established for the M&MS measure against the already established cutoffs for the SDQ using equipercentile equating (Kolen & Brennan, 2004). For the SDQ emotional symptoms subscale, a score of 6 is borderline and scores 7 and above are clinically significant (Goodman et al., 1998). In our SDQ sample of Year 7 pupils, these scores corresponded to percentile ranks 91.9 and 95.7, respectively. Percentiles indicate the percentage of children obtaining this score or lower on the scale. These percentile ranks translate into cutoff scores of 10 to 11 for borderline and 12 and above for clinically significant for the M&MS emotional difficulties subscale. For the SDQ conduct problems subscale the recommended cut-offs of 4 (borderline) and 5 (clinically significant) corresponded to percentile ranks 85.5 and 91.9 in our SDQ sample of Year 7 pupils, translating into cutoff scores of 6 for borderline and 7 and above for high risk for the M&MS behavioral difficulties subscale.
The M&MS high-risk cutoff identified 12% of younger children (Year 4) and 12.4% of older children (Year 7) in the sample as having behavioral difficulties (i.e., obtained a score of 7 or above), and 10.2% of younger children and 5.8% of older children in the sample as having emotional difficulties (i.e., obtained a score of 12 or above).
Discussion
Initial validation suggests this is a psychometrically sound brief self-report measure of mental health and well-being for young children, functioning equivalently across described groups.
The final M&MS subscales showed strong relationships with the self-report SDQ, suggesting that they measure similar underlying constructs. The discrepancy between the high magnitude of the conceptually similar subscales and the relatively low magnitude of the conceptually distinct subscales provides support for the construct validity of the scores (John & Benet-Martínez, 2000). Correlations between M&MS subscales and measures of academic attainment, SEN, and deprivation were low but statistically significant and consistent with correlations observed between these measures and the SDQ subscales.
In terms of limitations, it should be noted that the current analyses were based on a web-based delivery of measures, responses on paper versions are still to be tested. A limitation with regards to establishing clinical cutoffs is that the derived scores have not yet been validated with clinical populations. Stability of scores (test–retest reliability) will also need to be established. All of these areas will be explored in future research.
While further work is needed, the M&MS appears to fill a number of gaps. It was designed specifically for use in schools and allows even young children to report their own experience of mental health and well-being. It has potential for use as a broad screening tool to aid detection of child mental health problems and evaluation of school-based child mental health interventions.
Footnotes
Acknowledgements
The authors thank other members of the research group: Professor Neil Humphrey, Professor Norah Frederickson, Pam Meadows, Professor Anthony Fielding, Dr. Rob Coe, Mike Cuthbertson, Neville Hallam, Dr. John Little, Dr. Andrew Lyth, Professor Sir Michael Rutter, Professor Bette Chambers and Professor Alastair Leyland. The authors thank the English Department for Children Schools and Families (now the Department for Education) for funding the research.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research reported here was supported financially by the English Department for Children Schools and Families (now the Department for Education).