Who benefits most? Predicting the effectiveness of a social and emotional learning intervention according to children’s emotional and behavioural difficulties

Abstract

School-based social and emotional learning (SEL) programmes represent a practical method of improving social and emotional well-being in students. To date, however, what is less well understood is why a theoretically sound, appropriately administered, engaging universal SEL programme may be more effective for some children over others. In the present study, an established SEL programme, KooLKIDS Whole of Class, was delivered to 524, 8 to 12-year-old Australian primary school students. Classroom teachers completed pre- and post-measures regarding children’s social-emotional competence and emotional and behavioural difficulties. Hierarchical linear modelling was used to examine which characteristics predicted differential improvement from the programme. Findings revealed that the pre-programme total score on emotional and behavioural difficulties was the strongest predictor of improvements in SEL, independent of age, sex or socioeconomic status. This implies that while universal SEL programmes represent an effective and relatively low-cost method of improving social competence for all children, they may be particularly effective for improving social and emotional competence in those with greater emotional and behavioural difficulties.

Keywords

social and emotional learning emotional and behavioural difficulties emotion regulation intervention predictors of outcome programme effectiveness

Introduction

Social and emotional learning (SEL) has been conceptualised as the development of a set of inter-related core competencies that allow individuals to recognise and manage their emotions, set and achieve positive goals, make responsible, thoughtful decisions to overcome obstacles and generally get along well with others through perspective-taking, conflict management and positive relationship skills (Elias et al., 2015; Zins & Elias, 2006). The five core competencies of SEL identified by the Collaborative for Academic, Social, and Emotional Learning (CASEL, 2013) are self-awareness (e.g. recognising emotions, strengths, limitations and values); self-management (e.g. regulating emotions and behaviours); social awareness (e.g. perspective-taking and empathising with others); relationship skills (e.g. establishing and maintaining healthy relationships) and responsible decision-making (e.g. making helpful choices across varied situations. These are all skills considered to be important for success in school, work and life (Taylor, Oberle, Durlak, & Weissberg, 2017; Weissberg, Durlak, Domitrovich, & Gullotta, 2015). Overall, there is a growing belief that SEL is of at least equal importance for children as the learning of educational curricula content in the educational setting (Durlak, Domitrovich, Weissberg, & Gullotta, 2015; Taylor et al., 2017).

Various SEL programmes have been developed, and these range from whole-class or universal programmes to targeted programmes for children at-risk or identified as having skill deficits. Evidence across many studies support the many benefits conferred to children and adolescents from participating in SEL programmes including increases in self-concept and achievement, improvements in relationships and well-being and decreases in behaviour problems, drug use and emotional distress (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Durlak et al., 2015; Taylor et al., 2017). Further evidence from longitudinal studies suggests that significant positive social, emotional and behavioural benefits may persist for up to 15 years following the completion of a programme (Low, Smolkowski, Cook, & Desfosses, 2019; Taylor et al., 2017).

Meta-analytic reviews have assessed SEL programme effectiveness based on practical aspects of programme design. Results suggest that effective programmes remain consistent with proven therapeutic approaches (e.g. cognitive behavioural therapy), have a high degree of interactivity, involve small group work and have community support (Diekstra, 2008; Durlak et al., 2011). In addition, the most effective delivery timeframe (between up to one month and more than one year) was found to be between three and six months of weekly lessons, accompanied by booster sessions to ensure maintenance and long-term benefits (Diekstra, 2008). Where SEL programmes were delivered by teachers in schools (compared to external professional delivery), the outcomes were found to be superior (Diekstra, 2008; Durlak et al., 2011).

Within-subject variables and SEL programme effectiveness

What is less well understood regarding SEL programme effectiveness are the within-subject variables that influence why a theoretically sound, appropriately administered, engaging programme may be more effective for some children than others. Evidence suggests that children who have the most positive outcomes following the completion of a programme are those who experience the greatest long-term benefit (Taylor et al., 2017). However, research is yet to identify which children, prior to participating in the programme, are likely to respond most positively. This is of critical importance if programme effectiveness is to be optimised. Although previous research has attempted to explore within-child characteristics including the age and gender of the participating children and their socioeconomic context, the extent to which conclusions can be drawn has been limited due to poor reporting of sample characteristics or inconsistent outcomes (Diekstra, 2008; Taylor et al., 2017; Wilson & Lipsey, 2007).

With reference to age, effects are not commonly found, with researchers suggesting that children and adolescents of all ages appear to benefit equally (e.g. Diekstra, 2008; Hahn et al., 2007). Similarly, the effectiveness of SEL programmes based on gender is limited due to under-reporting, prompting calls for future studies to rectify this in order for stronger conclusions to be drawn (Diekstra, 2008). Another within-child factor that has received inconsistent results in previous research is socioeconomic status (SES). The prevailing belief is that children living in low SES areas face a higher risk for social-emotional or behavioural difficulties than their peers from middle or high SES areas (Diekstra, 2008; Hahn et al., 2007; Wilson & Lipsey, 2007). However, meta-analytic results based on a limited number of studies have reported no significant differences between children from different SES areas (Taylor et al., 2017).

Overall, these mixed results on within-subject variables suggest that the influence of age, gender and SES on the effectiveness of universal SEL interventions remains unclear, and more studies are needed to better understand their influence in programme effectiveness.

Emotional and behavioural difficulties and programme effectiveness

Children who show persistent emotional and behavioural difficulties are often regarded as being low in social-emotional competence (see Conklin, Kamps, & Wills, 2017; Gresham, 2015). A recent large-sample national survey of Australian 5- to 11-year-olds (Lawrence, Dawson, Houghton, Goodsell, & Sawyer, 2019) highlighted the differential patterns of internalising (e.g. anxiety and depression) and externalising disorders (e.g. oppositional and behavioural problems) which arise in mainstream schools. Overall, 18.3% of males and 12.4% of females had experienced a mental health issue. Specifically, 7.9% of males and 7.1% of females had experienced an anxiety disorder, 1.3% of males and 1.5% of females a major depressive disorder, 6.1% of males and 4.9% of females oppositionality and 2.7% of males and 1.7% of females conduct disorder. Such problems reflect deficits in social-emotional skills that most SEL programmes specifically aim to enhance.

Many SEL programmes target children with emotional and behavioural difficulties in particular because of the adverse impact they can have on peers and teachers (e.g., Carroll et al., 2017) and the assumption that an intensive individualised or small group approach is more efficacious (see Murphy, Hawkins, & Nabors, 2019). However, the recognition of the heterogeneity of emotional and behavioural difficulties within classrooms has witnessed a number of SEL programmes moving towards a universal (whole-of-class or whole-of-school) approach to promote strong and long-lasting positive outcomes for all children (Wilson & Lipsey, 2007). In addition, many SEL programmes now incorporate relevant adaptations in order to meet student need (e.g. cultural, age and developmental differences). While these are positive moves, one question remains unanswered. That is, whether differential levels of emotional and behavioural difficulties with which children present pre-programme predict differential outcomes post-programme. Answering this question may reveal whether pre-programme assessments have the capacity to predict which children will gain the greatest benefits and, in doing so, significantly enhance programme efficacy. For example, do children who already show more serious emotional or behavioural difficulties than typically developing peers (e.g. externalising or internalising disorders) benefit more from universal programmes than their typically developing peers?

The present study

The present study seeks to address the limitations identified in the current research literature by utilising the pre-programme intervention scores of students to identify those who benefit most from an established universal SEL intervention. It achieves this by examining the predictive influence of within-subject variables such as age, gender and SES. It also builds on the established research findings by examining the outcomes of participation in a universal SEL programme based on children’s differential profiles of emotional and behavioural difficulties. This study examines student outcomes following participation in the Australian-developed KooLKIDS Whole of Class programme (Carroll & Houghton, 2018). A recent evaluation of the KooLKIDS Whole of Class programme using a waitlist-control design showed significant improvements in internalising and externalising problems as well as in social and emotional competencies for children who completed the programme compared to a control group (see Carroll, McCarthy, Houghton & Sanders O'Connor, 2020). This SEL programme adheres to previous research recommendations by utilising a programme that incorporates best design elements and addresses the five core SEL competencies (Taylor et al., 2017; Weissberg et al., 2015). Delivered by classroom teachers, the KooL KIDS Whole of Class Program aims to assist all children aged 8 to 12 years to recognise their strengths, understand and manage their emotions, be aware of others’ emotions and develop their social and friendship skills. It utilises an interactive, multimedia approach and aligns with the Australian Curriculum and Reporting Authority’s personal and social capabilities of the national Australian curriculum.

The present study is particularly relevant to school psychologists, because not only do they work closely with teachers to identify children who require additional support with their SEL but they also collaborate in the delivery, monitoring and evaluation of universal SEL programmes (e.g. Poulou, Reddy, & Dudek, 2019; Reddy, Dudek, & Lekwa, 2017). Furthermore, school psychologists may utilise standardised test scores, especially those measuring social, emotional or conduct behaviours, for assessing the benefits that children accrue as a result of participating in these programmes. It is yet to be determined, however, whether children with a range of emotional or behavioural difficulties differentially benefit from participating in universal SEL programmes. School psychologists possess specialist knowledge and skills to best utilise this new information to advocate the importance of SEL programmes and to maximise outcomes for schools, children and their families.

Therefore, the current study addressed the following research questions:

Do the within-subject variables of student age, gender and SES predict improvements on measures of social and emotional competence following participation in a universal school-based SEL programme?

Do differential pre-programme profile scores of emotional and behavioural difficulties (i.e. normal, abnormal and borderline) predict improvements on measures of social and emotional competence following participation in a universal school-based SEL programme?

Method

Participants and setting

The sample comprised 524 primary school students (46% males, N = 234) who completed the KooL KIDS Whole of Class Program in a large metropolitan city of Queensland, Australia as well as the teachers who facilitated the programme with their classes (N = 21). The sample for this study is essentially the intervention group sample from a larger evaluation of the KooL KIDS Whole of Class Program (Carroll et al.,2020), comprising only students who were exposed to the SEL programme and who had complete measures for both before and after the delivery of the SEL programme. The students in this sample were recruited from 21 classrooms across six Catholic Education schools. Currently, 19.7% of primary school-aged children in Australia attend Catholic schools (Australian Bureau of Statistics (ABS), 2018). Students ranged in age from 8 to 12 years (average age = 9.7 years; SD = 0.84), with 268 students in grade 4 (8 to 9 years of age), 158 students in grade 5 (10 years) and 98 students in grade 6 (11 to 12 years).

SES was measured using students’ residential postcode. This is based on the Index of Relative Socioeconomic Disadvantage (IRSD that was developed by the ABS, 2016). Scores are generated for postcodes based on a range of information about the economic and social conditions of people within those areas (ABS, 2016). These are then ranked and grouped into deciles and quintiles to indicate levels of socioeconomic disadvantage. In the data analysis of the present study, IRSD quintiles have been used to indicate the relative socioeconomic disadvantage of the areas in which students reside, with one representing the most disadvantage and five representing the least disadvantage. Approximately 11% of the sample resided in quintile 1, 8% in quintile 2, 4% in quintile 3, 31% in quintile 4 and 46% in quintile 5.

SEL intervention programme

A full description of the KooL KIDS Whole of Class Program and evaluation has been described elsewhere (Carroll et al.,2020). This whole-of-class, teacher-facilitated, 13-session weekly programme is designed for 8- to 12-year-old children. Encompassing the set of inter-related core SEL competencies (Taylor et al., 2017; Weissberg et al., 2015), the programme aims to enhance children’s self-worth and self-awareness, emotional understanding, capacity for empathy, prosocial behaviours and friendship skills. The initial 12 sessions cover four programme modules, each comprising three sessions. The final session concludes the programme with a review and celebration. The four modules follow the acronym KooL: ‘Know yourself’, which focusses on strengths and self-esteem; ‘Understanding Our needs and emotions’, which focusses on emotion regulation; ‘Understanding Others needs and emotions’, which focusses on empathy; and ‘Live well with others’, which focusses on friendship and social skills.

In order to enhance engagement with the programme, eight of the programme sessions include watching a series of an animated story that features the character ‘Okki the Octopus’, who, while highly likeable, often engages in problematic behaviour and social interactions at school and has difficulty managing his emotions. Participants watch Okki’s stories through both a first-person perspective at times (i.e. seen through Okki’s eyes) and a second-person perspective at other times (i.e. scenes are shown with Okki in full-view). Through these stories and associated exercises, the skills and core concepts of the programme are reinforced to children. In addition to these videos, session activities are varied, teaching social skills, emotion and behaviour regulation through individual and group work, games, drawing and writing tasks, role plays, narrative exercises, calming exercises and behavioural challenges. These activities are informed by three therapeutic frameworks: cognitive behavioural therapy, solution-focused therapy and a strengths-based approach.

Measures

Strengths and Difficulties Questionnaire (SDQ)

The 25-item SDQ (Goodman, 1997) has been used extensively as a screening measure for emotional and behavioural difficulties in children and adolescents. The teacher version of the SDQ for 4- to 12-year-old children has been found to have strong psychometric properties (Stone, Otten, Engles, Vermulst, & Janssens, 2010). Teachers rate the 25 items on a three-point Likert scale ranging (0 = almost never, 1 = sometimes, 2 = often). Although the SDQ is designed to produce five subscales (i.e. emotional problems, conduct problems, hyperactivity, peer relationship problems and prosocial behaviour), a three-subscale division is recommended when assessing non-clinical populations (Goodman, Lamping, & Ploubidis, 2010). These three subscales comprise an internalising subscale (emotional problems and peer problems summed), an externalising subscale (conduct problems and hyperactivity summed) and a prosocial scale. More significant emotional and behavioural difficulties are indicated by higher scores on the SDQ subscales, except for the prosocial subscale where higher scores indicate greater prosocial behaviour. Scores on the SDQ can be compared to norms to indicate the relative severity of symptoms and classify scores within the ‘normal’, ‘borderline’ or ‘abnormal’ range. The SDQ is not intended to diagnose but rather to screen for behaviours that could indicate clinical levels of symptoms in the present or risk for them in the future.

Social and Emotional Competence Questionnaire (SEC)

The American Institute of Research (AIR) and the CASEL developed the SEC as a measure of the five core SEL competencies: self-awareness, self-management, social awareness, relationship skills and responsible decision-making (CASEL & AIR, 2013). This teacher-completed measure has 20 items measured on a five-point Likert scale, with higher scores on the SEC indicating higher levels of social and emotional regulation.

Teacher satisfaction

In teacher-led SEL programmes, a teacher’s satisfaction with the programme has the potential to influence how effectively they deliver the programme to students. This may also affect variability in student outcomes across classes. For this reason, teacher satisfaction with the programme was used as a control variable and included as a covariate in the analyses. Following completion of the programme, one teacher from each class (N = 21) completed a single item that related to their overall level of satisfaction with the programme in assisting students to manage their emotions and behaviour. Response options for this single item ranged from 1 (not at all satisfied) to 5 (extremely satisfied).

Procedure

Approval for the study was obtained from the Human Research Ethics Committee at the administering institution and the Catholic Education Office in the city in which the study was conducted. Following this, the principals of schools who had previously expressed interest in the KooL KIDS Whole of Class Program were invited to participate in the study. Those who consented to participate subsequently received information to share with teachers of grades 4 to 6, along with parent information and consent forms for distribution to students in each class. School psychologists (one per school) were also invited to participate in the programme and assist with programme facilitation as needed. Written informed consent was received from all participating teachers and school psychologists. Parental and self-consent was received from all participating students.

Participating teachers and school psychologists received a standardised full-day facilitator training session conducted by the research team as well as a package containing all the materials needed to facilitate the programme, including a fully scripted facilitator manual, children’s workbooks, web-based animated stories and a range of resources for the 13 sessions. Using these materials, participants became familiar with the development and philosophy of KooLKIDS; the theoretical basis of its approach (e.g. cognitive behavioural, strengths-based and solution-focused therapy models); the implementation of strategies with students and its method of delivery. Following training, participant feedback was obtained using an eight-item scale, yielding a mean overall facilitator satisfaction with training score of 4.4 (SD = 0.66) out of 5. Following training, teachers delivered the programme to their class groups across a school term.

Intervention integrity was maintained in a number of ways. Facilitators followed a fully scripted intervention manual with detailed session plans to guide them in programme delivery. Prior to the intervention, the intervention manual was reviewed by several academics with experience in child and adolescent psychopathology and clinical psychology. Throughout the intervention, the length of sessions was monitored, and facilitators received ongoing support from the research team in the form of weekly check-ins about how the sessions were progressing. Facilitators were also requested to complete a session-by-session checklist marking off each of the steps completed within each session. For further details, see Carroll et al. (2020).

This present study employed a within-subjects repeated measures design. Questionnaires were completed by participating teachers prior to commencing the KooL KIDS Whole of Class Program and again immediately after completion of the 13-session weekly programme. Teachers completed questionnaires in relation to each child in their class, taking approximately 2 hours at each time point.

Data Analysis

Hierarchical linear modelling, specifically a multilevel generalised linear model, was used to examine which within-subject variables and differential profiles of emotional and behavioural difficulties predicted improvements in social and emotional competence. Multilevel models can handle clustered or nested data, such as data on students who also belong to classes and schools. Multilevel modelling captures common variance within nested observations (such as measures of student behaviour within classes) and produces appropriate standard errors that account for this common variance (Gelman & Hill, 2007). It also enables the estimation of both the fixed and random effects (observed and unobserved effects) of groups or clusters on individual observations (Gelman & Hill, 2007).

In this analysis, within-subject variables (i.e. age, gender and SES); differential emotional and behavioural difficulties profiles (i.e. norm classification on the SDQ total difficulties scale) and teacher satisfaction scores with the programme were entered as fixed effects in the model. Class (N = 21) was entered as a random effect to account for any differential impacts on student outcomes that class membership may have. School was not included as a random effect due the low number of school observations in this study (N = 6) which can result in poorly estimated between-group variance (Gelman & Hill, 2007; Maas & Hox, 2005).

Results

Elevated SDQ scores have been shown to be valid predictors of psychological disorders (Goodman, 2001). In the current study, pre-programme SDQ norms were used to classify students as either in the normal range (0–15) or in the borderline/abnormal range (16–40) in terms of their total difficulties score. This classification allowed for examination of differential profiles of emotional and behavioural difficulties. Nearly four in five students (79%) were in the normal range on the SDQ and 21% in the borderline/abnormal range, which is in line with current rates identified by Lawrence et al. (2019) in the national Australian survey of child and adolescent mental health.

Pre- and post-intervention programme scores and teacher satisfaction

Table 1 shows the mean and standard deviations for scores on the SDQ and SEC subscales from Time 1 to Time 2 (pre- and post-SEL programme participation) and the resulting difference scores. Negative difference scores on the internalising, externalising and total difficulties SDQ subscales indicate reductions in emotional and behavioural problems over time, while positive difference scores on the SDQ prosocial subscale indicate increases in prosocial behaviour over time. For the SEC subscales, positive difference scores indicate increases in social and emotional competence.

Table 1.

Mean and standard deviations on the SDQ and SEC subscales from Time 1 (T1) to Time 2 (T2) and mean difference scores.

	Mean T1 (SD)	Mean T2 (SD)	Mean difference, T1 to T2 (SD)
SDQ subscales
Internalising	2.85 (3.72)	2.58 (3.37)	−0.25 (2.64)
Externalising	4.21 (4.46)	3.88 (4.37)	−0.33 (2.79)
Prosocial	7.73 (2.53)	7.89 (2.37)	0.14 (2.09)
Total difficulties	7.06 (6.92)	6.46 (6.54)	−0.58 (4.43)
SEC subscales
Self-awareness	11.85 (3.32)	12.69 (3.14)	0.80 (2.33)
Self-management	12.03 (3.42)	12.70 (3.27)	0.66 (2.14)
Social awareness	12.52 (3.11)	13.09 (2.94)	0.56 (2.23)
Relationship skills	12.38 (3.21)	13.10 (2.98)	0.70 (2.18)
Responsible decision-making	12.52 (3.27)	13.17 (3.07)	0.63 (2.13)
Total SEC score	61.32 (15.50)	64.77 (14.59)	3.34 (9.65)

Note: SDQ = Strengths and Difficulties Questionnaire; SEC = Social and Emotional Competence Questionnaire.

In addition to the pre- and post-student programme scores, teachers’ satisfaction with the programme was measured at post-programme on a scale ranging from 1 (not at all satisfied) to 5 (extremely satisfied). Forty-six percent of teachers had a satisfaction score of 5, indicating that they were extremely satisfied; 46% had a mean score of 4; and 9% had a mean score of 3, indicating that they were somewhat satisfied. No teachers scored their satisfaction with the programme as 1 or 2, thereby indicating high levels of satisfaction (90.6%) with the programme.

Predicting reductions in emotional and behavioural difficulties

Multilevel modelling was used to assess the influence of a range of characteristics on changes in emotional and behavioural difficulties, as measured by the SDQ subscales, following participation in the KooL KIDS Whole of Class Program. Student’s age, gender, SES, their classification on the SDQ total difficulties scale (normal vs. borderline/abnormal) and teacher satisfaction with the programme were entered as fixed effects in the model, while class membership (N = 21) was entered as a random effect. The dependent variable was students’ change scores on the total difficulties subscale, prosocial subscale, externalising subscale and internalising subscale, from T1 to T2 (pre- to post-KooL KIDS Whole of Class Program). For each of the subscales, an initial null model was run, followed by a full model examining the impact of each of the predictors on change in SDQ subscale scores, to determine whether the key explanatory factors provided a better fit for the changes in emotional and behavioural problems than the null model. The results of the multilevel modelling can be seen in Table 2.

Table 2.

Results of multilevel modelling predicting changes in emotional and behavioural problems following exposure to the SEL programme.

	Total difficulties		Prosocial subscale		Externalising subscale		Internalising subscale
	Null model	Full model	Null model	Full model	Null model	Full model	Null model	Full model
Parameter	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)
Intercept	−0.718 (0.415)	2.539 (4.123)	0.206 (0.195)	−0.579 (1.964)	−0.410 (0.228)	0.952 (2.418)	−0.298 (0.214)	1.126 (2.363)
Fixed effects
Age		−0.295 (0.312)		−0.054 (0.151)		−0.119 (0.192)		−0.148 (0.187)
Gender (male = 0, female = 1)		−0.343 (0.368)		−0.121 (0.180)		−0.159 (0.240)		−0.180 (0.233)
SES		0.049 (0.141)		0.006 (0.069)		0.068 (0.091)		0.013 (0.089)
SDQ norms		−3.835 (0.475)***		0.571 (0.232)*		−1.855 (0.309)***		−1.982 (0.299)***
Teacher satisfaction		0.089 (0.607)		0.291 (0.285)		0.002 (0.328)		0.095 (0.322)
Random effect: Class	2.913 (1.119)	2.642 (1.018)	0.646 (0.244)	0.567 (0.218)	0.797 (0.337)	0.683 (0.295)	0.698 (0.299)	0.663 (0.287)
Residual	16.717 (1.062)	14.477 (0.948)	3.671 (0.233)	3.470 (0.227)	6.952 (0.441)	6.209 (0.406)	6.302 (0.400)	5.829 (0.382)
Model fit
ICC (class)	0.148 (0.050)	0.154 (0.051)	0.150 (0.049)	0.140 (0.048)	0.103 (0.040)	0.099 (0.039)	0.100 (0.039)	0.102 (0.040)
AIC	2963.88	2733.96	2180.34	2036.56	2503.46	2313.49	2452.23	2283.37
BIC	2976.63	2767.47	2193.09	2070.06	2516.21	2346.99	2464.97	2316.87
N	517	487	517	487	517	487	517	487

Note: AIC = Akaike information criterion; BIC = Bayesian information criterion; ICC = intraclass correlation coefficient; SDQ = Strengths and Difficulties Questionnaire; SEC = Social and Emotional Competence Questionnaire.

p < .05, ***p < .001.

Results indicate that the only characteristic associated with significant changes in emotional and behavioural difficulties in students was their classification on the SDQ total difficulties subscale. Students classified as borderline/abnormal on the SDQ total difficulties subscale at T1 demonstrated a significantly greater reduction in emotional and behavioural problems on the total difficulties subscale, the internalising subscale and the externalising subscale at T2. In addition, significantly greater improvements were evident on the prosocial subscale for students classified as borderline/abnormal, compared to those students classified as normal on the SDQ. There was no effect of age, gender, SES or teacher satisfaction with the SEL programme, on changes in the SDQ subscales from pre- to post-SEL programme.

Predicting increases in social and emotional competence

To assess the influence of characteristics on changes in social and emotional competence, age, gender, SES, SDQ total difficulties norm classification and teacher satisfaction were entered as fixed effects in a multilevel generalised linear model, with class (N = 21) entered as a random effect. For each of the SEC subscales, an initial null model was run. This was followed by a full model examining the impact of each of the predictors on change in SDQ subscale scores. This sought to determine whether the key explanatory factors provided a better fit for the changes in emotional and behavioural problems than the null model (see Table 3.)

Table 3.

Results of multivariate linear regression, predicting changes in social and emotional competence following exposure to the SEL programme.

	Total SEC		Self-aware subscale		Self-manage subscale		Social awareness subscale		Relationship skills subscale		Responsible decision-making subscale
	Null model	Full model	Null model	Full model	Null model	Full model	Null model	Full model	Null model	Full model	Null model	Full model
Parameter	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)	Estimate (SE)
Intercept	3.699 (1.256)**	−0.479 (11.016)	0.875 (0.261)**	0.042 (2.503)	0.718 (0.244)**	−0.166 (2.342)	0.634 (0.274)*	−0.420 (2.43)	0.779 (0.254)**	0.935 (2.336)	0.691 (0.262)**	0.120 (2.34)
Fixed effects
Age		−0.076 (0.714)		0.050 (0.174)		0.049 (0.164)		−0.078 (0.166)		−0.101 (0.165)		−0.083 (0.159)
Gender		−1.29 (0.771)		−0.255 (0.191)		−0.069 (0.182)		−0.319 (0.183)		−0.289 (0.183)		−0.361 (0.176)*
SES		0.049 (0.298)		0.118 (0.074)		0.644 (0.238)		0.015 (0.071)		−0.026 (0.071)		0.034 (0.068)
SDQ norms		2.49 (1.007)*		0.521 (0.249)*		0.645 (0.238)**		0.564 (0.239)*		0.395 (0.239)		0.330 (0.230)
Teacher satisfaction		1.096 (1.892)		−0.011 (0.406)		0.166 (0.376)		0.414 (0.401)		0.224 (0.373)		0.304 (0.387)
Random effect: Class	30.413 (10.268)	29.172 (9.991)	1.257 (0.444)	1.296 (0.461)	1.102 (0.389)	1.106 (0.397)	1.426 (0.488)	1.278 (0.447)	1.202 (0.422)	1.082 (0.391)	1.299 (0.446)	1.189 (0.416)
Residual	63.861 (4.085)	62.022 (4.110)	4.187 (0.267)	3.827 (0.253)	3.519 (0.225)	3.478 (0.230)	3.551 (0.227)	3.495 (0.231)	3.611 (0.231)	3.522 (0.233)	3.275 (0.209)	3.242 (0.214)
Model fit
ICC (class)	0.323 (0.075)	0.320 (0.076)	0.231 (0.064)	0.253 (0.069)	0.238 (0.065)	0.241 (0.067)	0.286 (0.071)	0.268 (0.070)	0.249 (0.067)	0.235 (0.066)	0.284 (0.071)	0.268 (0.070)
AIC	3626.02	3389.74	2236.33	2063.32	2148.15	2017.15	2157.39	2021.34	2162.41	2021.77	2115.68	1985.35
BIC	3638.72	3423.08	2249.04	2096.69	2160.86	2050.52	2170.11	2054.71	2175.12	2055.15	2128.39	2018.73
N	510	477	512	479	512	479	512	479	512	479	512	479

p < .05, **p < .01.

The most significant influence on improvements in social and emotional competence following participation in KooLKIDS was students’ pre-programme classification on the SDQ total difficulties subscale. Students classified as borderline or abnormal experienced significantly greater increases in total social and emotional competence, as well as self-awareness, self-management and social awareness subscales, compared to students classified as normal on the SDQ. Male students demonstrated significantly greater increases on the responsible decision-making subscale compared to female students; however, there were no other significant differences related to gender on the other subscales. There were no significant associations between student characteristics and changes on the relationship skills subscales. Additionally, age, SES and teacher satisfaction with the SEL programme did not significantly influence changes in any of the SEC subscales.

Discussion

This study aimed to examine whether particular student characteristics and teacher satisfaction were associated with student improvements from pre- to post-SEL programme, for a programme that has been found to be effective in improving social and emotional competence and emotional and behavioural problems (KooLKIDS Whole of Class, see Carroll et al.,2020). Results of the current study demonstrate that the strongest predictor of positive change in SEL following participation in the programme was the individual’s pre-programme score on emotional and behavioural difficulties. Despite the SEL programme being delivered in a universal/whole-of-class format with no additional sessions offered for high-risk children, students classified as borderline/abnormal on the teacher-rated SDQ total difficulties scale showed significantly greater reductions in emotional and behavioural difficulties and significantly greater improvements in social and emotional competence than students who were classified in the normal range (on the SDQ). Age, gender and SES did not predict improvements in students’ emotional and behavioural difficulties following participation in the SEL programme nor did teachers’ level of satisfaction with the programme. With the exception that male students showed greater improvements in responsible decision-making, student and teacher characteristics did not predict changes in social and emotional competence following participation in the programme. Last, there did not appear to be significant differences in improvements in relationship skills from pre- to post-programme according to student characteristics. In the evaluation study (Carroll et al.,2020), it was found that students who participated in the SEL programme demonstrated significantly greater mean improvements in relationship skills from pre- to post-programme compared to a control group. These findings collectively may indicate that this is an area of emotional and social competence where most students can benefit.

It could be argued that the greater benefits for students with higher ratings of emotional and behavioural difficulties at pre-programme could be a statistical artifact, produced by either regression to the mean or ceiling effects on the outcome measures. Regression to the mean can occur when changes in groups of individuals are examined over time, particularly when individuals are grouped according to baseline measures, with more extreme observations for individuals naturally moving or regressing towards the mean over time (Barnett, van der Pols, & Dobson, 2005). However, the current study sample was drawn from a previous study (Carroll et al.,2020) which used a waitlist-control design to examine student outcomes following participation in the SEL programme. The study found that intervention students showed significantly greater mean improvements in social and emotional competencies and emotional and behavioural problems following programme participation compared to the control group. The study also found that mean changes for the control and intervention groups were not occurring in the same direction, with intervention students in general improving and control students in general worsening over time on the same measures. This suggests regression to the mean alone is not a sufficient explanation for the positive changes that the intervention students with emotional and behavioural difficulties experienced as a result of participation in the programme. It is also possible that the relatively smaller changes among the students classified as normal on the SDQ may reflect ceiling effects, whereby high pre-programme ratings of emotional and behavioural functioning on the SDQ limit the opportunity for any improvements to be detected in emotional and behavioural functioning (SDQ). However, positive changes on the measures of social and emotional competence suggest that these findings are not simply reflective of ceiling effects, though this may have played a role particularly in the findings on the SDQ measure.

It is probable that a universal SEL programme may involve delivery of SEL to a reasonable number of students who have moderate to high social and emotional competencies along with little evidence of emotional and behavioural problems. However, universal SEL programmes have been found to have a broader range of benefits that may not be discernable through individual student measures alone, such as positive changes in the classroom or school social environment (Osher et al., 2008; Sprague & Walker, 2010). Furthermore, the results of this study suggest that a universal SEL programme can be associated with measurable improvements among students who may be identified as having more significant social and emotional regulation or internalising and externalising problems. These results reinforce the potential value of universal SEL programmes such as KooLKIDS. They also suggest that universal programmes can be appropriately specific to benefit children who may have emotional or behavioural problems in the clinical or sub-clinical range. Of importance is that by being delivered as a whole-class programme not only are all children taught the skills to manage difficult behaviours and emotions in themselves and others and learn a common language with which to discuss social and emotional concepts and manage peer interactions but also the time and stigma associated with being removed from class to attend individual SEL lessons is minimised (Carroll et al.,2020).

A finding that children with the greatest difficulties derived the greatest benefit from the KooL KIDS Whole of Class Program is in accordance with the results reported by Wilson and Lipsey (2007) in their comprehensive meta-analysis of SEL programmes that specifically aim to reduce aggressive and disruptive behaviours. In their analysis, larger treatment effects were achieved by higher-risk students in universal programmes and by those children already showing problematic behaviours in selected and/or indicated programmes. The current findings provide further evidence for this effect by showing that it is the child’s current degree of difficulties that specifically predicts greater change in relation to participating in a quality SEL programme; this effect is independent of other characteristics that may exist concurrent to the behavioural/emotional difficulties (e.g. age, gender and SES).

Consistent with a recent review of universal SEL programmes (Taylor et al., 2017), students’ age and SES in the present study did not affect their ability to respond positively to the KooL KIDS Whole of Class Program. With regard to gender, males and females showed similar improvements in all but one construct, responsible decision-making, whereby males showed greater improvement than females. Such results support the conclusions drawn by others (e.g. Diekstra, 2008; Taylor et al., 2017) that SEL programmes are beneficial for all children. Future research should be conducted to further investigate the outcomes of the present study, especially the unexpected but interesting finding that male students demonstrated significantly greater increases in responsible decision-making compared to female students.

Beyond the influence of within-subject student variables, the results indicate that teachers’ satisfaction with the KooL KIDS Whole of Class Program did not affect how well students responded to it. This suggests that teacher-delivered SEL programmes that effectively engage students may be robust to implementation effects related to teacher engagement or ‘buy-in’ to the SEL programme. However, the teacher satisfaction measure in this study had a limited range (i.e. satisfaction ratings ranged from somewhat satisfied to extremely satisfied), and this may have limited the ability to detect the influence of teachers’ satisfaction or ‘buy-in’ on outcomes from the SEL programme.

Implications for practice

The finding that KooLKIDS is particularly effective for children with the greatest difficulties has added translational value because the research indicates that approximately 50% of children who show severe behavioural problems during childhood carry these with them into adolescence and onwards into adulthood (Parsonage, Khan, & Saunders, 2014). These individuals are at 20 times greater risk of offending, are twice more likely to be early school-leavers and are at four times at greater risk of drug use. The lifetime costs of this on society are greater than £260,000 per case (Parsonage et al., 2014). Thus, although long-term effects were not examined in this study, a quality universal SEL programme may provide a low-cost method of reducing or preventing long-term negative outcomes for these individuals and, in doing so, reduce the associated costs for society. It should be noted, however, that some sociodemographic characteristics may moderate responsiveness to SEL programmes in children with emotional and behavioural difficulties, and this was not examined in the present study.

The findings of the present study provide important implications for school psychologists working with children with behavioural and emotional difficulties. School psychologists working closely alongside teachers within classroom settings to embed SEL strategies may find this collaboration most efficacious for their most vulnerable children (see Carroll, Bower, Chen, Watterston, & Ferguson, 2019). Moreover, the shared delivery, monitoring and evaluation of universal SEL programmes by teachers and school psychologists may be the best solution to providing universal preventive strategies while intervening and remediating for those children with emotional and behavioural difficulties. The use of positive peer role models would also be of advantage in these whole-of-class sessions. School psychologists have the knowledge and skills to closely monitor the social, emotional and behavioural outcomes pertaining to the differential benefits that children accrue as a result of participating in these universal SEL programmes.

Limitations of the study and future research directions

As with all research, there are limitations to acknowledge in the present study. First, the sample composition may limit the generalisability of the findings. That is, the sample was recruited exclusively from Catholic Education schools, and as fees, albeit relatively low ones, for enrolment are common, this may preclude students from low SES families. Hence, the sample of students may not be adequately representative of the full SES range. Future research should therefore seek to recruit samples from diverse areas so that measures on the effectiveness of interventions are more fully representative of the school-aged population. Future research should also attempt to ascertain whether these findings generalise to other SEL programmes.

Although this is the first known study to directly assess SEL programme effectiveness according to severity of a child’s emotional and behavioural difficulties, the method of classification used was not specific enough to determine the presence of a clinical diagnosis. A classification of borderline or abnormal on the SDQ can be an indication of sub-clinical or clinical externalising or internalising problems, but this is not diagnostic. To obtain a formal diagnosis a range of professionals, including a psychologist/psychiatrist, would have been necessary and this was beyond the scope of the study.

As noted, there is also the possibility of ceiling effects on the SDQ measures for children classified as normal, and this may have limited the scope for improvements to be displayed to the same extent as those for the children classified as borderline/abnormal. Nevertheless, consistent effects of SDQ classification across both the SDQ and the SEC support the findings of the impact of the presence of emotional and behavioural problems on responsiveness to a SEL programme. Finally, there is always the possibility that teachers may have inflated their satisfaction or engagement with the programme ratings for purposes of social desirability. This is a potential issue in all such research, and to address this in the present study, all teacher responses were de-identified.

Conclusion

Recognition of the importance of SEL during childhood and adolescence is growing, and schools provide an ideal location in which this learning can occur. An important facet in planning and implementing effective SEL programmes is understanding which characteristics of students might best influence their positive outcomes from participating in such programmes. The current study has contributed to the literature by identifying which students benefit from an established universal SEL intervention and confirm that universal programmes, such as KooLKIDS, can have sufficient specificity to benefit all children, even those with emotional and behavioural difficulties. As such, there are benefits for SEL skills being taught as a whole-of-class approach. Moreover, advocating for school psychologists and teachers to work collaboratively to embed SEL in whole-of-school curriculum should be supported.

Footnotes

Acknowledgements

The authors would like to thank the principals, teachers and students who participated in this study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: This work was supported by the Australian Research Council Special Research Initiative Science of Learning Research Centre (Project Number SR120300015).

ORCID iD

Kylee Forrest

Annemaree Carroll is an Associate Dean Research in the Faculty of Humanities and Social Sciences and Professor of Educational Psychology at The University of Queensland. Annemaree’s research is particularly focussed on understanding the impact of emotions, attention, and behaviour on learning throughout child and adolescent development and developing and implementing strategies that can be translated into educational outcomes.

Stephen Houghton is a Registered Psychologist and Director of the Centre for Child and Adolescent Related Disorders at the University of Western Australia. His research activities focus on childhood and adolescent psychopathology particularly in relation to developmental trajectories to delinquency and antisocial behaviour and the affective and interpersonal traits constituting juvenile psychopathy.

Kylee Forrest is a Clinical Psychologist in private practice and also a Research Officer in the School of Education at the University of Queensland. Kylee’s research interests have focused on childhood anxiety; the mental health and well-being of women, couples and babies during their ante- and post-natal periods; school-based interventions and mindfulness interventions.

Molly McCarthy is a Research Fellow at the Griffith Criminology Institute at Griffith University. Molly’s research takes a cross-disciplinary approach to understanding aggression and offending behaviour, and she is interested in examining prosocial and antisocial influences on child and adolescent development.

Emma Sanders-O’Connor is a Clinical Psychologist and a Research Officer in the School of Education at the University of Queensland. Emma’s research interests have focused on the mental health and well-being of children, school-based interventions, as well as mindfulness interventions and their application in schools.

References

Australian Bureau of Statistics. (2016). Technical paper: Socio-economic indexes for areas (SEIFA) (Cat 2033.0.55.001). Canberra, Australia: Author.

Australian Bureau of Statistics. (2018). Schools, Australia, 2017 (Cat 4221.0). Retrieved from https://www.abs.gov.au/AUSSTATS/abs@.nsf/Previousproducts/4221.0Main%20Features99992017?opendocument&tabname=Summary&prodno=4221.0&issue=2017&num=&view=.

Barnett

A. G.

van der Pols

J. C.

Dobson

A. J.

(2005) Regression to the mean: What it is and how to deal with it. International Epidemiological Association 34: 215–220. doi:10.1093/ije/dyh299.

Carroll, A., Bower, J., Chen, H., Watterston, J., & Ferguson, A. (2019). Towards a new pedagogical framework for social and emotional well-being: Focus group interviews with system and school stakeholders in Australia. Manuscript submitted for publication.

Carroll, A. & Houghton, S. (2018). KooLKIDS: A Self-Regulatory Intervention to Empower Children to Live Well with Themselves and Others. Whole of Class Teacher Manual. The University of Queensland: Brisbane.

Carroll, A., McCarthy, M., Houghton, S., & Sanders O’Connor, E. (2020). KooLKIDS: An interactive social emotional learning program for Australian primary school children. Psychology in the Schools. Advance online publication. doi:10.1002/pits.22352.

Carroll

Sanders O’Connor

Houghton

S. J.

Hattie

J. A.

Donovan

Lynn

(2017) A preliminary empirical evaluation of KooLKIDS: A school-based program to reduce early onset antisocial behaviour in children. Journal of Intellectual & Developmental Disability 42(1): 22–35. doi:10.3109/13668250.2016.1190004.

Collaborative for Academic, Social, and Emotional Learning (2013) 2013 CASEL guide: Effective social and emotional learning programs—Preschool and elementary school edition, Chicago, IL: Author Retrieved from http://www.casel.org/preschool-and-elementary-edition-casel-guide/.

Collaborative for Academic, Social and Emotional Learning and American Institutes for Research (2013) Teacher rating of student social and emotional competencies, Washington, DC and Chicago, IL: Author.

10.

Conklin

C. G.

Kamps

Wills

(2017) The effects of class-wide function-related intervention teams (CW-FIT) on students’ prosocial classroom behaviours. Journal of Behavioral Education 26: 75–100. doi:10.1007/s10864-016-9252-5.

11.

Diekstra, R. F. W. (2008). Effectiveness of school-based social and emotional education programmes worldwide. In C. Clouder, B. Dahlin, R. Diekstra, P. Fernandez-Berrocal, B. Heys, L. Lantieri & H.Paschen (Eds.) Social and emotional education: An international analysis (pp. 255–312). Santander, Spain: Fundación Marcelino Botin.

12.

Durlak, J. A., Domitrovich, C. E., Weissberg, R. P., & Gullotta, T. P. (Eds.). (2015). Handbook of social and emotional learning: Research and practice. New York, NY: Guilford.

13.

Durlak

J. A.

Weissberg

R. P.

Dymnicki

A. B.

Taylor

R. D.

Schellinger

K. B.

(2011) The impact of enhancing students’ social and emotional learning: A meta-analysis of school-based universal interventions. Child Development 82(1): 405–432. doi:10.1111/j.1467-8624.2010.01564.x.

14.

Elias

M. J.

Leverett

Duffell

J. C.

Humphrey

Stepney

Ferrito

(2015) Integrating SEL with related prevention and youth development approaches. In: Durlak

J. A.

Domitrovich

C. E.

Weissberg

R. P.

Gullotta

T. P.

(eds) Handbook for social and emotional learning: Research and practice, New York, NY: Guilford, pp. 33–49.

15.

Gelman

Hill

(2007) Data analysis using regression and multilevel/hierarchical models, New York, NY: Cambridge University Press.

16.

Goodman

(1997) The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry 38(5): 581–586. doi:10.1111/j.1469-7610.1997.tb01545.x.

17.

Goodman

(2001) Psychometric properties of the strengths and difficulties questionnaire. Journal of the American Academy of Child and Adolescent Psychiatry 40(11): 1337–1345. doi:10.1097/00004583-200111000-00015.

18.

Goodman

Lamping

D. L.

Ploubidis

G. B.

(2010) When to use broader internalising and externalising subscales instead of the hypothesised five subscales on the strengths and difficulties questionnaire (SDQ): Data from British parents, teachers and children. Journal of Abnormal Child Psychology 38(8): 1179–1191. doi:10.1007/s10802-010-9434-x.

19.

Gresham

F. M.

(2015) Evidence-based social skills interventions for students at risk for EBD. Remedial and Special Education 36(2): 100–104. doi:10.1177/0741932514556183.

20.

Hahn

Fuqua-Whitley

Wethington

Lowy

Crosby

Fullilove

. Task Force on Community Preventive Services (2007) Effectiveness of universal school-based programs to prevent violent and aggressive behavior: A systematic review. American Journal of Preventive Medicine 33(2, Suppl. 1): S114–S129. doi:10.1016/j.amepre.2007.04.012.

21.

Lawrence

Dawson

Houghton

Goodsell

Sawyer

M. G.

(2019) Impact of mental disorders on attendance at school. Australian Journal of Education. 63, 5–21.

22.

Low

Smolkowski

Cook

Desfosses

(2019) Two-year impact of a universal social-emotional learning curriculum: Group differences from developmentally sensitive trends over time. Developmental Psychology 55(2): 415. doi:10.1037/dev0000621.

23.

Maas, C. J. M., & Hox, J. J. (2005). Sufficient Sample Sizes for Multilevel Modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1(3), 86–92. doi: 10.1027/1614-2241.1.3.86.

24.

Murphy

J. M.

Hawkins

R. O.

Nabors

(2019) Combining social skills instruction and the good behavior game to support students with emotional and behavioral disorders. Contemporary School Psychology. Advance online publication. doi:10.1007/s40688-019-00226-3.

25.

Osher

Sprague

Weissberg

R. P.

Axelrod

Keenan

Kendziora

Zins

J. E.

(2008) A comprehensive approach to promoting social, emotional, and academic growth in contemporary schools. In: Thomas

Grimes

(eds) Best practices in school psychology V Vol. 4 Bethesda, MD: National Association of School Psychologists, pp. 1263–1278.

26.

Parsonage, M., Khan, L., & Saunders, A. (2014). Building a better future: The lifetime costs of childhood behavioural problems and the benefits of early intervention. Retrieved from http://www.centreformentalhealth.org.uk/pdfs/building_a_better_future.pdf.

27.

Poulou

M. S.

Reddy

L. A.

Dudek

C. M.

(2019) Relation of teacher self-efficacy and classroom practices: A preliminary investigation. School Psychology International 40(1): 25–48. doi:10.1177/0143034318798045.

28.

Reddy

L. A.

Dudek

C. M.

Lekwa

(2017) Classroom strategies coaching model: Integration of formative assessment and instructional coaching. Theory into Practice 56(1): 46–55. doi:10.1080/00405841.2016.1241944.

29.

Sprague

J. R.

Walker

H. M.

(2010) Building safe and healthy schools to promote school success: Critical issues, current challenges, and promising approaches. In: Shinn

M. R.

Walker

H. M.

Stoner

(eds) Interventions for achievement and behavior problems in a three-tier model including RTI, Bethesda, MD: National Association of School Psychologists, pp. 225–257.

30.

Stone

L. L.

Otten

Engels

R. C. M. E.

Vermulst

A. A.

Janssens

J. M. A. M.

(2010) Psychometric properties of the parent and teacher version of the strengths and difficulties questionnaire for 4- to 12-year-olds: A review. Clinical Child and Family Psychology Review 13: 254–274. doi:10.1007/s10567-010-0071-2.

31.

Taylor

R. D.

Oberle

Durlak

J. A.

Weissberg

R. P.

(2017) Promoting positive youth development through school-based social and emotional learning interventions: A meta-analysis of follow-up effects. Child Development 88(4): 1156–1171. doi:10.1111/cdev.12864.

32.

Weissberg

R. P.

Durlak

J. A.

Domitrovich

C. E.

Gullotta

T. P.

(2015) Social and emotional learning: Past, present, and future. In: Durlak

J. A.

Domitrovich

C. E.

Weissberg

R. P.

Gullotta

T. P.

(eds) Handbook for social and emotional learning: Research and practice, New York, NY: Guilford, pp. 3–19.

33.

Wilson

S. J.

Lipsey

M. W.

(2007) School-based interventions for aggressive and disruptive behavior: update of a meta-analysis. American Journal of Preventive Medicine 33(2 Suppl): S130–S143. doi:10.1016/j.amepre.2007.04.011.

34.

Zins

J. E.

Elias

M. J.

(2006) Social and emotional learning. In: Bear

G. G.

Minke

K. M.

(eds) Children’s needs III, Bethesda, MD: NASP, pp. 1–13.