Children’s mental health in different contexts: Results from a multi-informant assessment of Estonian first-grade students

Abstract

The current study sought to investigate the agreement between teachers and parents about the mental health of first-grade students, the factors that affected this agreement and the associations between measures completed by students, parents and teachers. The investigation used baseline data collected during the PAX Good Behavior Game (PAX GBG) effectiveness study in 42 Estonian schools (N = 708). Information was collected about externalizing and internalizing difficulties and prosocial behaviour, and about inhibitory control. Our research showed higher agreement between parents and teachers about externalizing behaviour and lower agreement regarding prosocial behavior and emotional problems. Inhibition was correlated with teacher-rated questionnaires, but not with parents’ responses. Sociodemographic factors influenced the agreement between teachers and parents somewhat differently. This study highlights the importance of a multi-informant approach in students’ mental health assessments, as some problems might be less observable in certain environments or by some respondents. The practical implications of these findings are discussed and suggestions are provided for the development of a school-based mental health screening system.

Keywords

mental health assessment children multiple informants parent-teacher agreement

Introduction

According to Kessler et al. (2005) and Kieling et al. (2011), 10–20% of children experience mental health problems, with half of these difficulties beginning before 14 years of age. A number of studies indicate that mental health problems in childhood can have profound consequences in adulthood, including increased contact with the criminal justice system, reduced levels of employment and personal relationship difficulties (Chen et al., 2006; Fergusson et al., 2005; Zechmeister et al., 2008). At the same time, most children and adolescents with mental health needs are not identified and do not receive the help they need (Lempinen et al., 2019).

Several researchers have suggested the application of universal mental health screening at schools, as schools offer a valuable opportunity for assessing the risks and early symptoms of mental health problems among young people (Humphrey & Wigelsworth, 2016; National Research Council and Institute of Medicine, 2009). There are several mental health screening tools available that can be used in the school setting. Previous research has suggested that individual reports often yield inconsistent conclusions, while using multiple informants can provide a richer understanding of a child’s mental health status (De Los Reyes et al., 2015; Humphrey & Wigelsworth, 2016). At the same time, researchers have found that there is low or moderate agreement between raters, emphasizing the impact of the subjectivity of the persons who make the assessments as well as the contextual variations that affect the display of mental health concerns (Cheng et al., 2018; De Los Reyes et al., 2015).

Informers’ assessments can differ due to various reasons, as contextual variations in the severity of problems, informants’ background, perspectives and abilities as well as rater biases can all have an effect on them (De Los Reyes et al., 2015, 2009). Factors such as socioeconomic status, family structure, number of children in the family, parents’ ages, education and employment can influence the magnitude of informant discrepancies (Harvey et al., 2013). The amount of discrepancies can also vary across countries and cultures (Achenbach et al., 2008; Cheng et al., 2018; Rescorla et al., 2014), highlighting the importance of repeating the studies in various contexts to better understand the magnitude of contextual and cultural factors that affect these discrepancies. For example, a study in Swedish primary schools demonstrated that informant discrepancies were related to the parental education and foreign background (Swedish-born parent vs non-Swedish-born parent) and suggested that these might affect norms of the “normal” child behaviour, which might in turn be a risk factor as there might be a conflict of norms between parent and teacher perceptions (Boman et al., 2016). Another study carried out in seven diverse European countries showed significant country differences in parent–teacher agreement for both internalizing and externalizing problems, which can have an impact on mental health diagnosis and cross-cultural research (Cheng et al., 2018).

Various mental health problems are associated in the research literature with deficits in executive functions that can be evaluated via neuropsychological testing (Johnson, 2012; Vuontela et al., 2013; Wright et al., 2014). At the same time, there are only few studies in which data is collected about elementary school children’s executive functions, such as inhibitory control, and mental health symptoms rated by different informants (Utendale et al., 2011). Torralva et al. (2013) have suggested that real-life deficits often remain undetected in performance-based assessments - while some studies have demonstrated relationship between inhibition deficits and students’ mental health problems rated by parents and teachers (Vuontela et al., 2013), others have shown that performance-based executive functioning tests have lower predictive value compared with behavioural questionnaires (Tan, Delgaty et al., 2018). Studies examining the link between children’s executive functions and mental health problems rated by different informants have provided inconsistent findings as there is heterogeneity of executive function abilities among children with mental health issues (Teivaanmäki et al., 2020). Children with mental health problems might perform within normal range on measures of executive functions due to differences in the severity of impairment or because additional mechanisms might underlie the mental health issues (Tan et al., 2018). More information is needed to understand how inhibitory control abilities are related to emotional or behavioural issues at home and/or school.

This study draws information from another study carried out in Estonian mainstream schools and explores the subjectivity of multi-informant assessments of children’s mental health. In sum, this research seeks to answer the following questions:

How strong is the inter-rater agreement between parent- and teacher-rated Strengths and Difficulties Questionnaires (SDQs) among Estonian first graders (RQ1)?

Is there a relationship between parent- and teacher-rated questionnaires and behavioural tasks measuring inhibition (Go/No-Task) (RQ2)?

Which sociodemographic factors predict parent-teacher agreement for externalizing and internalizing behaviour (RQ3)?

Estonian context

Estonia has a population of 1.3 million with Estonian as the official language of the country. There is cultural and language diversity among the population, with ethnic Estonians making up to 70% of the population, Russians representing about 25% of the population and other ethnic groups about 5%. Estonia is a developed country which performs well in measurements of education but lags behind with health indicators (Santiago et al., 2016). For example, according to the Programme for International Student Assessment (PISA), performance of Estonian students is among the highest compared with other participating countries and socio-economic status has low impact on students’ performance (OECD, 2020). At the same time, Estonian students have one of the lowest school satisfaction and psychological problems have become more frequent in time (Havik, 2020; Inchley et al., 2020).

The compulsory education in Estonia begins at age 7, but about 90% of 3-year-olds are enrolled in early childhood education and care. Majority of children attend public schools and participation in schooling is almost universal (Santiago et al., 2016)

Method

Participants and procedure

This study used baseline data collected in 2016 for the PAX Good Behavior Game (PAX GBG) effectiveness study, which was conducted in 2016–2018 in Estonia. It was a cluster-randomized waitlist-controlled trial, which aimed to evaluate the impact of PAX GBG on students' mental health and behaviour as well as teachers’ self-efficacy in Estonia compared with the waiting list control condition (Streimann et al., 2017, Streimann, Selart, & Trummal, 2020).

All mainstream schools were invited to participate if their instruction language was Estonian and if their classroom had at least 13 pupils (average class size in Estonia is 19 students in primary education (OECD, 2019)). Schools that focused solely on children with special educational needs, schools with single-sex classrooms and schools that already implemented some evidence-based prevention programs were not suitable to participate in the study. Invitations to take part in the study were sent to 164 schools across Estonia, of which 55 signed up to participate. Three schools were excluded due to not meeting the inclusion criteria and 6 withdrew their participation, hence the final sample consisted of 42 primary schools, each with one first-grade classroom.

Ethical approval for the study was received from the Tallinn Medical Research Ethics Committee in June 2016. Informed consent forms were signed by all participants. As signatories, schools confirmed that they understood the conditions for participation, and parents provided an all-encompassing, opt-in consent by which they approved data collections from children, teachers and themselves. In addition, each child’s verbal consent was obtained before each completion of the Go/No-Go task. More detailed information about the procedures can be found in the study protocol (Streimann et al., 2017).

Of the 796 eligible children, 708 (88.9%) had parental consent to participate, 46 (5.8%) had parents who declined and 42 (5.3%) had parents who did not respond. Baseline data from teachers was available for about 703 children (99.3%), and from parents, for about 576 children (81.4%). The completeness of the measures varied – approximately 80% of children had complete mental health data available from parents and teachers, while about 75% had sociodemographic information available as well.

Baseline data was collected from the children themselves across 24 randomly selected schools. As the task that measures inhibitory control was conducted individually with each child at that child’s school, and given the fact that the procedure was time consuming, it was not possible to include all children in the sample. The initial sample consisted of 413 children with average age of 7 years, and final data was available for 359 of them (86.9%). The data was missing from children who were not at school during data collection, who refused to participate or quitted the participation.

Measures

The research uses the data collected from multiple informants (teachers, parents, children) by means of different tools (questionnaires and behavioural tasks) and explores agreement between raters, explanatory factors for agreement and correlations between different measures in Estonia. Information was collected about:

Teachers’ and parents’ perspectives on children’s overall mental health and behavioural and emotional difficulties, as measured by the total difficulties score and subscales of the Strengths and Difficulties Questionnaire (SDQ); and

Children’s abilities in inhibitory control, as measured by the visual computerized Go/No-Go task completed by the children.

Strengths and Difficulties Questionnaire (SDQ)

SDQ is a short screening instrument that measures both the problem behaviours and prosocial skills of children between 4–16 years of age (Goodman & Goodman, 2009; Goodman, 1997, 1999). While the questionnaire can be completed by teachers, parents or the adolescents themselves (starting from 11 years of age), only teachers and parents did so in the current study, as the children were too young for self-reporting. In this regard, it has demonstrated adequate validity and reliability across populations for the parent and teacher versions (Stone et al., 2010). The psychometric properties of the Estonian version of the SDQ has been studied previously by Trummal and Kukk (2018), who found the internal consistency to be between moderate and satisfactory (0.5 – 0.8) depending on the subscale. The construct and the concurrent validity of the questionnaire were also explored in the same study and SDQ total difficulties and externalizing behaviour aspects were correlated strongly with the Eyberg Child Behavior Inventory (ECBI) questionnaire’s intensity scale.

The instrument consists of 25 statements that comprise five subscales: emotional symptoms, conduct problems, hyperactivity/inattention, peer relationship problems and prosocial behaviour. The first four subscales lead to a total difficulties score, which is a valid measure of overall child mental health problems (Goodman & Goodman, 2009). All statements are rated on a three-point Likert scale, ranging from 0 (not true) to 2 (certainly true).

In addition to using SDQ scores as continuous variables, cut-off scores can be used to identify likely cases of mental health disorders (Meltzer et al., 2000). This study used an SDQ three-band categorization and grouped children to either a ‘not-at-risk’ group (‘normal’ on SDQ categorization) or an ‘at-risk’ group (‘borderline’ and ‘abnormal’ on SDQ categorization). The Estonian cut-offs were used for parent-rated SDQs (Trummal & Kukk, 2018), the original cut-offs originating from the United Kingdom (Meltzer et al., 2000) were used for teacher-rated SDQs due to the lack of normative teacher-rated data. In this study, teachers and parents filled out the Estonian version of the SDQs.

The visual computerized Go/No-Go task

Go/No-Go task, a sensitive measure of impulsivity, was used to assess children's inhibitory control. Developed for the purposes of the present study, the task registered the number of commission and omission errors, together with reaction times, for both go and no-go tasks. These markers reflect the level of a child’s self-control, problems in behavioural inhibition and attentional difficulties. More information on the development, piloting and application of the task is available in the study protocol (Streimann et al., 2017). Less research has been done to define the psychometrics of behavioural tests of impulsivity (Bari & Robbins, 2013; Perales et al., 2009), but studies have shown moderate to good test-retest reliability of the Go/No-Go task and an association between performance on the task and other executive-functioning tests or measures of psychopathology, e.g. teacher ratings of ADHD symptoms (Kuntsi et al., 2005; Langenecker et al., 2007).

Sociodemographic data

Additional data was collected from parents about their child’s gender and age, parental gender and age, parental nationality and home language, family structure, number of children and household members, financial situation of the household, current employment status and parental education.

Statistical analysis

First, the non-response bias was assessed with the aim to explore whether there was a relationship between a child’s overall mental health as rated by teachers (total difficulties score rated on SDQ) and the absence or presence of a parental response. A mixed effects model of teacher-rated total difficulties score with school effects was compared with a model including a dummy variable that indicated the presence or absence of a parent-rated total difficulties score on SDQ. A likelihood ratio test between the two models estimated that there was no statistically significant improvement in model performance (χ²[1] = 1.16, p = 0.282), indicating that the teacher’s responses about children with or without parental responses did not differ significantly and missing data was unrelated to the child’s mental health status. Hence, the impact of missing data was relatively benign and the available-data analysis approach was applied.

Per each SDQ subscale, summary statistics were calculated. SDQ scores can be scaled up pro-rata if at least 3 items are completed out of five per each subscale, thus even if the respondent did not reply to some questions, it was still possible to use the data for the analysis. Inferential statistics (paired t-test) were used to test whether there were significant differences between teacher and parent ratings. Internal consistency was retrieved by calculation of Cronbach’s alpha coefficients. Cronbach’s alpha values for the teacher-rated SDQ subscales ranged from 0.60 (Peer Problems) to 0.89 (Hyperactivity–Inattention), whereas for the parent ratings, values ranged from 0.54 (Peer Problems) to 0.78 (Hyperactivity–Inattention). For the total difficulties score, the alpha value for the teacher-rated score was 0.86 and for the parent-rated score, 0.80. These values indicate good reliability.

Parent-teacher agreement on the SDQ subscales was measured continuously and categorically, using Pearson correlation and kappa statistic. For kappa statistic, a two by two comparison of parent and teacher-reported SDQ categorization (e.g. ‘at risk’, ‘not at risk’) was used. Proportions of the ratings were also calculated to find out how many children were assessed to be at risk by only the parents’ rating, only the teachers’ rating or both.

Associations between different measures (SDQs and Go/No-Go task) were explored by Pearson correlation, making the Bonferroni adjustment to the calculated significance levels.

Finally, multiple logistic regression analyses were performed to explore which sociodemographic factors predict parent-teacher agreement about a child’s risk for externalizing problems (conduct problems and hyperactivity/inattention subscales on SDQ), internalizing problems (emotional problems and peer problems on SDQ) and overall mental health (total difficulties score on SDQ). Cut-offs for the risk of externalizing and internalizing problems were defined by the sum of respective subscales’ scores – the cut-off for externalizing problems was greater or equal to 8 (the sum of conduct and hyperactivity subscales), the cut-off for teacher-rated internalizing problems was greater or equal to 8 and for parent-rated internalizing problems greater or equal to 7 (the sum of emotional and peer problems). A binary variable was created that described either form of agreement on the child’s status – i.e. both teacher and parent assessed the child either to be at risk or not to be at risk for mental health problems. Disagreement reflected how one respondent thought the child to be at risk, while the other did not. Results from previous studies were taken into account when deciding on possible predicting factors (e.g. Cheng et al., 2018; Harvey et al., 2013) and child’s gender, family structure, financial situation of the household, parents’ employment, parents’ education and number of children were added to the model. Models were expressed as odds ratios (OR) with a 95% confidence interval (CI). As the parent-teacher agreement can be affected by the school factors, the generalized estimating equation (GEE) approach that allows for clustering was used (Table S4, available online) to control the results of the simpler one level model. The intraclass correlation was trivial (less than 1% of the total variance in the outcome, ranging between −0.018–0.0086) and results remained valid in both models, hence the one level model is presented in the article.

All statistical analysis were performed with Stata version 14.2 (StataCorp, 2015).

Results

Descriptive statistics

Descriptive data are presented in Supplementary Table 1 (available online). Children’s average age was 7.1 (SD = 0.3) years; 50.1% of the sample were girls, and 49.9% were boys; 66.3% of children lived in a nuclear family, 25.9% lived in single-parent family and 7.8% lived in a blended family.

Most children in the study lived in households comprising 3–4 members (60.6%). Most parents or caregivers who filled out the parental questionnaire were women (92.1%), and their average age was 35.9 years (SD = 6.3), the youngest of whom was 24 years old and the oldest, 72 years old.

The number of students per classroom ranged from 13 to 28 (mean of 16.9, SD = 3.3). The age of the teachers ranged from 25 to 64 years, the average being 46 years (SD = 8.3). All teachers except one were women.

Mean scores of the measures are described in Table S2 (available online). Parents reported significantly higher mean scores than teachers in all SDQ subscales (indicating more difficulties) except prosocial behaviour, where parents reported higher prosocial behaviour than teachers (p < 0.001).

Agreement between parent and teacher-rated SDQs (RQ1)

Pearson correlations between parent and teacher subscale scores on SDQ ranged from 0.22 (prosocial behaviour) to 0.47 (hyperactivity-inattention) (Table 1), indicating a small to medium association between parent-teacher responses.

Table 1.

Parent-teacher correlations for SDQ, kappa statistics and agreement rates for risk status (n = 573).

	Subscale score r	Kappa (95% ci)	Agreeing on risk status		Not agreeing on risk status
SDQ domains	Subscale score r	Kappa (95% ci)	Not at risk	At risk	At risk by parent only	At risk by teacher only
TOTAL DIFFICULTIES	0.44	0.26 (0.17–0.34)	348 (60.7%)	64 (11.2%)	94 (16.4%)	67 (11.7%)
EMOTIONAL SYMPTOMS	0.28	0.08 (0.00–0.16)	399 (69.5%)	19 (3.3%)	127 (22.1%)	29 (5.1%)
CONDUCT PROBLEMS	0.44	0.34 (0.25–0.42)	378 (66.0%)	61 (10.6%)	97 (16.9%)	37 (6.5%)
HYPERACTIVITY-INATTENTION	0.47	0.32 (0.24–0.41)	359 (62.5%)	70 (12.2%)	72 (12.5%)	73 (12.7%)
PEER PROBLEMS	0.35	0.27 (0.16–0.38)	449 (78.2%)	28 (4.9%)	54 (9.4%)	43 (7.5%)
PROSOCIAL BEHAVIOUR	0.22	0.16 (0.07–0.25)	367 (63.9%)	43 (7.5%)	88 (15.3%)	76 (13.2%)
EXTERNALIZING PROBLEMS	0.51	0.36 (0.28-0.45)	353 (61.6%)	79 (13.8%)	83 (14.5%)	58 (10.1%)
INTERNALIZING PROBLEMS	0.34	0.14 (0.05-0.24)	428 (74.6%)	21 (3.7%)	93 (16.2%)	32 (5.6%)

There was a slight agreement on emotional symptoms (k = 0.08) and fair agreement on other SDQ subscales using kappa statistic. A greater level of agreement was found on externalizing behaviours (conduct problems and hyperactivity/inattention), where parents and teachers agreed more often on the risk status of the child.

Across raters, agreeing on risk status was found for more than 70% of children for all subscales; the highest agreement was found on the peer problems subscale, where 83.1% of parents and teachers agreed in terms of the risk status of the child. The largest differences for not agreeing on risk status were related to the emotional symptoms of children – 22.1% of parents considered their child to have a risk for emotional problems that were not noticed by teachers.

Associations between parent- and teacher-rated SDQs and child-performed go/No-Go task (RQ2)

Table S3 (available online) describes the correlations between parent- and teacher-rated SDQs and Go/No-Go tasks completed by children. Weak, but significant correlations were found between Go/No-Go tasks and hyperactivity/inattention as measured by teacher-rated SDQs. The task correlated to some extent with conduct and peer problems as well as with overall mental health status observed by teachers. Inhibitory abilities were practically not related with parent-rated measures.

Factors predicting parent-teacher agreement on child’s mental health (RQ3)

Logistic regression analysis demonstrated that different sociodemographic factors were associated with parent-teacher agreement on externalizing problems, internalizing problems and children’s overall mental health status (Table 2). A child’s gender was associated with agreement about externalizing problems, internalizing problems and overall mental health, with the odds for agreement being higher for girls than for boys.

Table 2.

Logistic regression model of factors predicting parent-teacher agreement on child’s externalizing, internalizing and total difficulties categories.

	External difficulties (N = 499)		Internal difficulties (N = 500)		Total difficulties (N = 499)
Sociodemographic characteristics	OR	P	OR	P	OR	P
CHILD’S GENDER (REF.GROUP: BOY)	2.12(1.38–3.27)	0.001	1.97 (1.25–3.12)	0.004	1.62 (1.07–2.45)	0.022
MOTHER’S EDUCATION (REF. GROUP: BASIC EDUCATION)
VOCATIONAL EDUCATION	1.15 (0.66–2.01)	0.616	1.41 (0.77–2.59)	0.263	1.17 (0.69–1.98)	0.568
HIGHER EDUCATION	1.43 (0.79–2.59)	0.236	1.12 (0.60–2.09)	0.717	1.90 (1.07–3.38)	0.029
FATHER’S EDUCATION (REF. GROUP: BASIC EDUCATION)
VOCATIONAL EDUCATION	0.87 (0.53–1.44)	0.594	0.78 (0.46–1.33)	0.371	0.93 (0.58–1.50)	0.769
HIGHER EDUCATION	0.72 (0.38–1.35)	0.306	0.79 (0.40–1.53)	0.479	0.78 (0.42–1.44)	0.424
FAMILY TYPE (REF. GROUP: NUCLEAR FAMILY)
BLENDED FAMILY	0.45 (0.23–0.92)	0.027	0.85 (0.39–1.86)	0.683	1.11 (0.52–2.38)	0.782
SINGLE-PARENT FAMILY	0.60 (0.35–1.05)	0.072	1.33 (0.70–2.51)	0.387	0.86 (0.50–1.47)	0.575
GOOD FINANCIAL SITUATION OF HOUSEHOLD (REF. GROUP: POOR OR AVERAGE)	0.81 (0.49–1.33)	0.405	1.84 (1.12–3.02)	0.015	1.98 (1.26–3.11)	0.003
NUMBER OF CHILDREN IN FAMILY (REF. GROUP: 1 CHILD)
2 CHILDREN	1.67 (0.87–3.19)	0.123	1.39 (0.68–2.85)	0.367	1.96 (1.03–3.73)	0.040
3 CHILDREN	1.82 (0.91–3.67)	0.091	1.82 (0.84–3.98)	0.130	1.67 (0.85–3.28)	0.135
4 OR MORE CHILDREN	2.36 (1.02–5.46)	0.045	1.30 (0.55–3.08)	0.546	3.66 (1.58–8.49)	0.003
AT LEAST ONE PARENT UNEMPLOYED (REF. GROUP: BOTH EMPLOYED OR ACTIVE)	0.75 (0.40–1.40)	0.365	0.81 (0.43–1.55)	0.524	0.68 (0.38–1.22)	0.199
PSEUDO R2	0.05		0.04		0.06

While a mother’s higher level of education significantly increased the odds for agreement on her child’s overall mental health status (OR = 1.90, 95% CI = 1.07–3.38), it was not statistically significant for internalizing or externalizing problems. Living in a blended or single-parent family decreased the odds for parent-teacher agreement about externalizing problems; the results were significant for blended families (OR = 0.45, 95% CI = 0.23–0.92), and a similar but insignificant trend was visible also for single parent families (OR = 0.60, 95% CI = 0.35–1.05).

A greater number of children in the family increased the odds for agreement about externalizing behaviour and overall mental health, but not about internalizing behaviour. Children growing up in families with 4 or more children had 3.7 times higher odds for parent-teacher agreement about the children’s total difficulties (OR = 3.66, 95% CI = 1.58–8.49) and 2.4 times higher odds for agreement about externalizing problems (OR = 2.36, 95% CI = 1.02–5.46), compared with children who had no siblings.

A household’s good financial situation significantly increased the odds for agreement about internalizing behaviour (OR = 1.84, 95% CI = 1.12–3.02) as well as the child’s overall mental health (OR = 1.98, 95% CI = 1.26–3.11), but not about the child’s externalizing behaviour.

The father’s level of education and parents’ employment status did not predict the agreement.

Discussion

This study aimed to examine the value of the multiple informant approach and to explore how different mental health measures are associated with each other, the agreement between different raters about elementary school children’s mental health and the factors predicting this agreement.

The results demonstrated that parents and teachers disagreed the most about the child’s risk for low prosocial behaviour and for emotional symptoms, indicating that these issues are more contextual and related to the perspective of the informant. Previous studies have concluded that parents are more likely to identify internalizing behaviours, as these might be more observable in the home context (De Los Reyes et al., 2015; Goodman et al., 2003). It has also been concluded that prosocial behaviour might be more difficult to observe and is thus susceptible to inferences in ratings (Stone et al., 2010).

Agreement across raters was found for more than 70% of the children for all SDQ subscales. The correlations and inter-rater agreements between parent- and teacher-rated SDQs were larger for externalizing behaviours, which is similar to the findings of previous research (Cheng et al., 2018). The agreement between parents and teachers has been even higher in previous studies; for example, in Sweden, the concordant risk status between teachers and parents was over 80% among primary school children (Boman et al., 2016). One possible explanation for this can be the lack of Estonian norms for the teacher-rated SDQ, which might somewhat influence the results.

The subscales of hyperactivity and overall mental health in teacher-rated SDQs were significantly correlated with inhibition, but, in parent-rated SDQs, mostly were not. This can imply that teachers are better equipped to detect externalizing behaviours, as they have the opportunity to compare their reports with normative classroom behaviours (De Los Reyes et al., 2015). Inhibition abilities are more visible in the school environment, where the skills to inhibit impulses and to focus on tasks directly affect students’ performance (St Clair-Thompson & Gathercole, 2006).

Several sociodemographic factors predicted inter-rater agreement about children’s mental health. Teachers and parents agreed significantly more about the mental health status of a girl compared with that of boys, indicating that mental health issues are exhibited more similarly for girls in both home and school contexts. This finding differs from that of a previous study, in which the child’s gender influenced only the ratings about externalizing behaviours (Cheng et al., 2018). It might indicate that the cultural context affects the inter-rater agreement about mental health problems among girls and boys differently.

A mother’s higher level of education increased the agreement about her child’s overall mental health almost two-fold, but it did not influence agreement about specific problems (e.g. externalizing or internalizing problems). A father’s level of education did not predict agreement between raters, which might be caused by the fact that most of the raters were mothers. It can be hypothesized that if the data were collected from fathers, then their education might better predict agreement with teachers.

Family structure was an important factor that was associated with the agreement between parents and teachers about externalizing behaviour, but not about internalizing behaviour or a child’s overall mental health. If the child was living in a blended or single-parent family, then the agreement about his behaviour was lower compared with that for children growing up in nuclear families. This might imply that children living in nuclear families have similar behavioural expectations at home and at school, compared to other types of families. One earlier study also hypothesized that single parenthood can result in less cohesiveness and control as well as higher parental stress, leading to lower parent–teacher agreement (Cheng et al., 2018). Based on this study, it is not possible to assess whether family structure results in a child’s contrasting behaviour in different contexts or if the rater’s perspective is affected by family type; hence, this is an important area for further studies.

A higher number of children in the family predicted stronger agreement between raters about externalizing problems and also about a child’s overall mental health. On the other hand, good financial situation of the household was significantly related with agreement about internalizing behaviour and a child’s overall mental health. Previous research has suggested that parents and teachers may have differing perceptions of a child’s externalizing behaviours due to differences in the reference group to which they compare the child (King et al., 2018). That can explain why the higher number of children in the household can reduce the discrepancies between raters about externalizing behaviour. Socioeconomic status of the family, on the other hand, affects parental emotional well-being and parenting practices (Bøe et al., 2014), which, as a result, can increase the discrepancies between raters.

Practical implications

Internationally, the importance of early identification, intervention and prevention have been highlighted as necessary responses to reduce the incidence and prevalence of children’s mental health problems (National Research Council and Institute of Medicine, 2009; World Health Organization, 2014). However, there are no universal recommendations about either school-based screening measures or informants from whom to collect information, both of which depend on the resources available as well as on the child’s age.

Based on the results of this study, teachers are well-equipped to provide information about children’s externalizing behaviour and to recognize deficits in children’s self-control skills as early as the beginning of the first grade. Parents seemed to notice the emotional issues of children that might be missed in the school environment. This suggests that both parents and teachers should provide information about the mental health of children in their charge, as one informant’s contribution is unique vis-à-vis that of the other (De Los Reyes et al., 2015).

Starting screening at the beginning of school life will support the early detection of those issues that not only affect children’s mental health, but also their overall school performance during later years (Guzman et al., 2011; Murphy et al., 2015). It is suggested to collect data from teachers about externalizing problems and from parents about internalizing problems at the beginning of the school year among first graders. It is also possible that when children have been at school for longer period, teachers are better equipped to notice emotional problems as the accuracy of teacher-rated SDQ has been validated in various other studies.

The study also demonstrated that several factors affect the inter-rater agreement about a child’s mental health status, which emphasizes the value of collecting data from different social agents while simultaneously considering the determinants of discrepancies.

In addition, children should also be involved in the process of screening, as parents and teachers can still miss the issues and struggles that children are experiencing. While teachers and parents might complete brief standardized measures, such as SDQs, in the course of universal screening for first graders, children can be active participants using measures and methods that are age-appropriate when risks are detected by either teachers or parents. For example, a child-centred outcome measure PSYCHLOPS Kids has been developed for children aged 7–13 years, which includes both the child’s perspective on issues of concern (qualitative data) and helps to measure mental health intervention outcome scores (quantitative data) (Godfrey et al., 2019).

Strengths and limitations of the study

The different measures used in this study helped to clarify what aspects of mental health are associated with one another, and what aspects are better noticed in different contexts or by different informants. In addition, as data was collected at the beginning of school life for Estonian students, this study provides important insight into what to consider when designing a universal school-based screening system in Estonia.

This study also has limitations. First, the measures that were used provided insight into only some aspects of mental health. The selection of questionnaires was partially related to the fact that only a few reliable and valid measures have been adapted to Estonian. As many of the available adapted instruments still lack Estonian norms, the original cut-off scores were used, which can influence the results.

Further, the study explored how family sociodemographic factors are related to inter-rater agreement. At the same time, teachers as informants carry their own agency, which can also affect the results. The current focus was on parental factors affecting agreement between the raters, but further studies could explore the relationship between teachers’ background and parent-teacher agreement. Another limitation is that the parents who responded were mostly mothers, which can also affect the results concerning parent-teacher agreement. Further studies could assess the level of agreement between fathers, mothers and teachers.

Additionally, the non-representative sample of 42 schools that participated in this study might influence the results. Involving schools with Russian instruction language, small Estonian schools or schools focusing on children with special educational needs can impact the results as well.

Most of the data was collected from parents and teachers; however, children are the key informants and experts on their own lives (Casas, 2019), and their opinion could be asked when assessing their mental health needs. This study did not collect comprehensive data from children; including the children’s own perspectives on their mental health needs and competencies might broaden our understanding as to whether screening tools such as SDQs can identify children who struggle with their emotions or behaviours.

Conclusions

This research contributes to a better understanding of the subjectivity of assessing first-grade students’ mental health, and it highlights how the involvement of multiple informants from different contexts in assessing children’s mental health provides a more comprehensive picture about a child’s difficulties and strengths. In sum, several sociodemographic factors are related to the agreement between raters, and these should be considered when evaluating the mental health difficulties and competencies of children.

Supplemental Material

sj-pdf-1-spi-10.1177_01430343211000414 - Supplemental material for Children’s mental health in different contexts: Results from a multi-informant assessment of Estonian first-grade students

Supplemental material, sj-pdf-1-spi-10.1177_01430343211000414 for Children’s mental health in different contexts: Results from a multi-informant assessment of Estonian first-grade students by Karin Streimann, Merike Sisask and Karmen Toros in School Psychology International

Footnotes

Acknowledgements

The authors would like to express their gratitude to Kirsti Akkermann and the Center for Cognitive Behavior Therapy for developing the visual computerized Go/No-Go task used in this study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was funded by the European Social Fund and Ministry of the Interior, Estonia.

Author biographies

Karin Streimann works in the Children and Youth Unit, National Institute for Health Development. She is involved with mental health promotion and substance use prevention research and development activities. She is also a doctoral student in Tallinn University, focusing on the prevention of mental health problems in elementary schools.

Merike Sisask, PhD, Professor of Social Health Care at Tallinn University, School of Governance, Law and Society, Board Member of the Estonian-Swedish Mental Health and Suicidology Institute. Educational background in law, psychological counselling, public health, and sociology. Research interest is public health view on mental health, well-being, and suicide prevention.

Karmen Toros is Professor of Social Work in School of Governance, Law and Society, Tallinn University. She is actively involved in social work education and child protection training. Her research focuses on child welfare and protection, strengths-perspective in child protection practice and is particularly interested in assessment of child well-being.

References

Achenbach

T. M.

Becker

Döpfner

Heiervang

Roessner

Steinhausen

H. C.

Rothenberger

(2008). Multicultural assessment of child and adolescent psychopathology with ASEBA and SDQ instruments: Research findings, applications, and future directions. Journal of Child Psychology and Psychiatry, and Allied Disciplines, 49, 251–275. https://doi.org/10.1111/j.1469-7610.2007.01867.x

Bari

Robbins

T. W.

(2013). Inhibition and impulsivity: Behavioral and neural basis of response control. Progress in Neurobiology, 108, 44–79. https://doi.org/10.1016/j.pneurobio.2013.06.005

Bøe

Sivertsen

Heiervang

Goodman

Lundervold

A. J.

Hysing

(2014). Socioeconomic status and child mental health: The role of parental emotional well-being and parenting practices. Journal of Abnormal Child Psychology, 42, 705–715. https://doi.org/10.1007/s10802-013-9818-9

Boman

Stafstrom

Moghadassi

Ostergren

P. O.

Lundin

Tornhage

C. J.

(2016). Comparing parent and teacher assessments of mental health in elementary school children. Scandinavian Journal of Public Health, 44, 168–176. https://doi.org/10.1177/1403494815610929

Casas

(2019). Are all children very happy? An introduction to children’s subjective well-being in international perspective. In: Kutsar, D. & Raid, K. (eds) Children’s subjective well-being in local and international perspectives (pp. 6–18). Statistikaamet.

Chen

Cohen

Kasen

Johnson

J. G.

Berenson

Gordon

(2006). Impact of adolescent mental disorders and physical illnesses on quality of life 17 years later. Archives of Pediatrics & Adolescent Medicine, 160, 93–99. https://doi.org/10.1001/archpedi.160.1.93

Cheng, S., Keyes, K. M., Bitfoi, A., Carta, M. G., Koç, C., Goelitz, D., Otten, R., Lesinskiene, S., Mihova, Z., Pez, O., & Kovess-Masfety, V. (2018). (2018). Understanding parent–teacher agreement of the strengths and difficulties questionnaire (SDQ): comparison across seven European countries. International Journal of Methods in Psychiatric Research, 27, 1–10. https://doi.org/10.1002/mpr.1589

De Los Reyes

Augenstein

T. M.

Wang

Thomas

S. A.

Drabick

D. A. G.

Burgers

D. E.

Rabinowitz

(2015). The validity of the multi-informant approach to assessing child and adolescent mental health. Psychological Bulletin, 141, 858–900. https://doi.org/10.1037/a0038498

De Los Reyes

Henry

D. B.

Tolan

P. H.

Wakschlag

L. S.

(2009). Linking informant discrepancies to observed variations in young children’s disruptive behavior. Journal of Abnormal Child Psychology, 37, 637–652. https://doi.org/10.1007/s10802-009-9307-3

10.

Fergusson

D. M.

John Horwood

Ridder

E. M.

(2005). Show me the child at seven: The consequences of conduct problems in childhood for psychosocial functioning in adulthood. Journal of Child Psychology and Psychiatry, 46, 837–849. https://doi.org/10.1111/j.1469-7610.2004.00387.x

11.

Godfrey, E., Aubrey, M., Crockford, S., Haythorne, D., Kordowicz, M., & Ashworth, M. (2019). The development and testing of PSYCHLOPS Kids: a new child-centred outcome measure. Child and adolescent mental health, 24(1), 54–65. https://doi.org/10.1111/camh.12271

12.

Goodman

(2009). Strengths and difficulties questionnaire as a dimensional measure of child mental health. Journal of the American Academy of Child and Adolescent Psychiatry, 48, 400–403. https://doi.org/10.1097/CHI.0b013e3181985068

13.

Goodman

(1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586. https://doi.org/10.1111/j.1469-7610.1997.tb01545.x

14.

Goodman

(1999). The extended version of the strengths and difficulties questionnaire as a guide to child psychiatric caseness and consequent burden. Journal of Child Psychology and Psychiatry, 40, 791–799. https://doi.org/10.1111/1469-7610.00494

15.

Goodman

Ford

Simmons

Gatward

Meltzer

(2003). Using the strengths and difficulties questionnaire (SDQ) to screen for child psychiatric disorders in a community sample. International Review of Psychiatry (Abingdon, England), 15, 166–172. https://doi.org/10.1080/0954026021000046128

16.

Guzman, M. P., Jellinek, M., George, M., Hartley, M., Squicciarini, A. M., Canenguez, K. M., Kuhlthau, K. A., Yucel, R., White, G. W., Guzman, J., & Murphy, J. M. (2011). (2011). Mental health matters in elementary school: First-grade screening predicts fourth grade achievement test scores. European Child & Adolescent Psychiatry, 20, 401–411. https://doi.org/10.1007/s00787-011-0191-3

17.

Harvey

E. A.

Fischer

Weieneth

J. L.

Hurwitz

S. D.

Sayer

A. G.

(2013). Predictors of discrepancies between informants’ ratings of preschool-aged children’s behavior: An examination of ethnicity, child characteristics, and family functioning. Early Childhood Research Quarterly, 28, 668–682. https://doi.org/10.1016/j.ecresq.2013.05.002

18.

Havik

(2020). Eesti kooliõpilaste vaimne tervis rahvusvahelises kontekstis. Eesti Arst, 99, 16–23.

19.

Humphrey

Wigelsworth

(2016). Making the case for universal school-based mental health screening. Emotional and Behavioural Difficulties, 21, 22–42. https://doi.org/10.1080/13632752.2015.1120051

20.

Inchley

Currie

Budisavljevic

Torsheim

Jåstad

Cosma

(2020). Spotlight on adolescent health and well-being. Findings from the 2017/2018 Health Behaviour in School-aged Children (HBSC) survey in Europe and Canada. International report. Volume 1. Key findings. Copenhagen. www.euro.who.int

21.

Johnson

M. H.

(2012). Executive function and developmental disorders: The flip side of the coin. Trends in Cognitive Sciences, 16, 454–457. https://doi.org/10.1016/j.tics.2012.07.001

22.

Kessler

R. C.

Berglund

Demler

Jin

Merikangas

K. R.

Walters

E. E.

(2005). Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the national comorbidity survey replication. Archives of General Psychiatry, 62, 593–602. https://doi.org/10.1001/archpsyc.62.6.593

23.

Kieling, C., Baker-Henningham, H., Belfer, M., Conti, G., Ertem, I., Omigbodun, O., Rohde, L. A., Srinath, S., Ulkuer, N., & Rahman, A. (2011). (2011). Child and adolescent mental health worldwide: Evidence for action. The Lancet, 378, 1515–1525. https://doi.org/10.1016/S0140-6736(11)60827-1

24.

King

K. M.

Luk

J. W.

Witkiewitz

Racz

McMahon

R. J.

(2018). Externalizing behavior across childhood as reported by parents and teachers: A partial measurement invariance model. Assessment, 25, 744–758. https://doi.org/10.1177/1073191116660381

25.

Kuntsi

Andreou

Börger

N. A.

van der Meere

J. J.

(2005). Testing assumptions for endophenotype studies in ADHD: Reliability and validity of tasks in a general population sample. BMC Psychiatry, 5, 40–12. https://doi.org/10.1186/1471-244X-5-40

26.

Langenecker

S. A.

Zubieta

J. K.

Young

E. A.

Akil

Nielson

K. A.

(2007). A task to manipulate attentional load, set-shifting, and inhibitory control: Convergent validity and test-retest reliability of the parametric go/No-Go test. Journal of Clinical and Experimental Neuropsychology, 29, 842–853. https://doi.org/10.1080/13803390601147611

27.

Lempinen

Luntamo

Sourander

(2019). Changes in mental health service use among 8-year-old children: A 24-year time-trend study. European Child & Adolescent Psychiatry, 28, 521–530. https://doi.org/10.1007/s00787-018-1218-9

28.

Meltzer

Gatward

Goodman

Ford

(2000). Mental health of children and adolescents in Great Britain. The Stationery Office.

29.

Murphy, J. M., Guzmán, J., McCarthy, A. E., Squicciarini, A. M., George, M., Canenguez, K. M., Dunn, E. C., Baer, L., Simonsohn, A., Smoller, J. W., & Jellinek, M. S. (2015). 2015). Mental health predicts better academic outcomes: A longitudinal study of elementary school students in Chile. Child Psychiatry and Human Development, 46, 245–256. https://doi.org/10.1007/s10578-014-0464-4

30.

National Research Council and Institute of Medicine. (2009). Preventing mental, emotional, and behavioral disorders among young people. National Academies Press. https://doi.org/10.17226/12480

31.

OECD. (2019). Education at a Glance: OECD indicators. Estonia.

32.

OECD. (2020). Education policy outlook: Estonia. http://www.oecd.org/education/policy-outlook/country-profile-Estonia-2020.pdf

33.

Perales

J. C.

Verdejo-García

Moya

Lozano

Ó.

Pérez-García

(2009). Bright and dark sides of impulsivity: Performance of women with high and low trait impulsivity on neuropsychological tasks. Journal of Clinical and Experimental Neuropsychology, 31, 927–944. https://doi.org/10.1080/13803390902758793

34.

Rescorla, L. A., Bochicchio, L., Achenbach, T. M., Ivanova, M. Y., Almqvist, F., Begovac, I., Bilenberg, N., Bird, H., Dobrean, A., Erol, N., Fombonne, E., Fonseca, A., Frigerio, A., Fung, D. S., Lambert, M. C., Leung, P. W., Liu, X., Marković, I., Markovic, J., Minaei, A., … Verhulst, F. C. (2014). (2014). Parent–teacher agreement on children’s problems in 21 societies. Journal of Clinical Child and Adolescent Psychology, 53, 43, 627–642. https://doi.org/10.1080/15374416.2014.900719

35.

Santiago, P., Levitas, A., Radó, P. & Shewbridge, C. (2016). OECD Reviews of School Resources: Estonia 2016, OECD Reviews of School Resources. OECD Publishing, Paris. https://dx-doi-org.web.bisu.edu.cn/10.1787/9789264251731-en

36.

St Clair-Thompson

H. L.

Gathercole

S. E.

(2006). Executive functions and achievements in school: Shifting, updating, inhibition, and working memory. Quarterly Journal of Experimental Psychology, 59, 745–759. https://doi.org/10.1080/17470210500162854

37.

StataCorp. (2015). Stata statistical software: Release 14. StataCorp LLC.

38.

Stone

L. L.

Otten

Engels

R. C. M. E.

Vermulst

A. A.

Janssens

J. M. A. M.

(2010). Psychometric properties of the parent and teacher versions of the strengths and difficulties questionnaire for 4- to 12-year-olds: A review. Clinical Child and Family Psychology Review, 13, 254–274. https://doi.org/10.1007/s10567-010-0071-2

39.

Streimann, K., Selart, A., & Trummal, A. (2020). Effectiveness of a universal, classroom-based preventive intervention (PAX GBG) in Estonia: a cluster-randomized controlled trial. Prevention Science, 21(2), 234–244. https://doi.org/10.1007/s11121-019-01050-0

40.

Streimann, K., Trummal, A., Klandorf, K., Akkermann, K., Sisask, M., Toros, K., & Selart, A. (2017). Effectiveness of a universal classroom-based preventive intervention (PAX GBG): A research protocol for a matched-pair cluster-randomized controlled trial. Contemporary Clinical Trials Communications, 8(August), 75–84. https://doi.org/10.1016/j.conctc.2017.08.013

41.

Tan

Delgaty

Steward

Bunner

(2018). Performance-based measures and behavioral ratings of executive function in diagnosing attention-deficit/hyperactivity disorder in children. ADHD Attention Deficit and Hyperactivity Disorders, 10, 309–316. https://doi.org/10.1007/s12402-018-0256-y

42.

Teivaanmäki

Huhdanpää

Kiuru

Aronen

E. T.

Närhi

Klenberg

(2020). Heterogeneity of executive functions among preschool children with psychiatric symptoms. European Child & Adolescent Psychiatry, 29, 1237–1249. https://doi.org/10.1007/s00787-019-01437-y

43.

Torralva

Gleichgerrcht

Lischinsky

Roca

Manes

(2013). “ Ecological” and highly demanding executive tasks detect real-life deficits in high-functioning adult ADHD patients. Journal of Attention Disorders, 17, 11–19. https://doi.org/10.1177/1087054710389988

44.

Trummal

Kukk

(2018). Lapse käitumise ja vanemliku kompetentsuse küsimustike psühhomeetrilised omadused ning normatiivsed andmed. Tallinn.

45.

Utendale

W. T.

Hubert

Saint-Pierre

A. B.

Hastings

P. D.

(2011). Neurocognitive development and externalizing problems: The role of inhibitory control deficits from 4 to 6 years. Aggressive Behavior, 37, 476–488. https://doi.org/10.1002/ab.20403

46.

Vuontela

Carlson

Troberg

A. M.

Fontell

Simola

Saarinen

Aronen

E. T.

(2013). Working memory, attention, inhibition, and their relation to adaptive functioning and behavioral/emotional symptoms in school-aged children. Child Psychiatry & Human Development, 44, 105–122. https://doi.org/10.1007/s10578-012-0313-2

47.

World Health Organization. (2014). The case for investing in public health. A public health summary report for EPHO 8. http://www.euro.who.int/__data/assets/pdf_file/0009/278073/Case-Investing-Public-Health.pdf

48.

Wright

Lipszyc

Dupuis

Thayapararajah

S. W.

Schachar

(2014). Response inhibition and psychopathology: A meta-analysis of go/No-Go task performance. Journal of Abnormal Psychology, 123, 429–439. https://doi.org/10.1037/a0036295

49.

Zechmeister, I., Kilian, R., McDaid, D. (2008). Is it worth investing in mental health promotion and prevention of mental illness? A systematic review of the evidence from economic evaluations. BMC Public Health, 8, 20–12. https://doi.org/10.1186/1471-2458-8-20

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.20 MB