Abstract
Objective:
This pilot study sought to better understand what can be achieved by an evidence-based classroom social and emotional education programme.
Design and Methods:
A 10-lesson, classroom-based programme that taught about emotional literacy, personal strengths, coping and problem-solving strategies, stress management, emotional regulation and support seeking was provided to 56 students in Years 7 (13 years) and 8 (14 years) in an Australian middle school. Teachers were trained to deliver the programme, with participatory modelling of each activity. Before and after delivery of the programme, students were surveyed for their social and emotional wellbeing using the Kessler 10 (K10) instrument for non-specific psychological distress; the ‘Internal Assets’, ‘School Resources’ and ‘Cooperation and Communication’ questions from the Resilience and Youth Development Module (RYDM) of the California Healthy Kids Survey (CHKS); and questions developed for this study on class connectedness and social and emotional skills. Subsequent to programme completion, focus groups were conducted with teachers and participating students to gauge programme fidelity, utility and engagement.
Results:
There was an improvement in psychological distress that approached significance (t = 2, df = 42, p = .053), although the symptomatic score remained in the range indicative of medium-level distress. Cooperation and communication improved significantly (t = −2.34, df = 42, p = .024) as did class connectedness (t = −2.46, df = 43, p = .018). There was no change in individual resilience factors, school protective factors, or social and emotional skills. The focus groups were generally positive about the programme, but indicated fidelity was compromised, mainly because the lesson periods were too short.
Conclusion:
While this small-scale pilot study has a number of limitations, it does indicate the need to improve the psychological wellbeing of middle school students. The findings also provide evidence that brief social and emotional education programmes can have some positive effects.
Introduction
Social and emotional wellbeing refers to the way a person thinks and feels about themselves and others. The development of social and emotional competence is important for children as it enables them to better navigate their way through life (Australian Institute of Health and Welfare, 2012). Socially and emotionally competent children are confident, have good interpersonal relationships, can communicate well, do better academically, persist with challenging tasks and have a sense of competence and self-worth (Pahl and Barrett, 2007). Such competencies provide resilience against stressors and help to prevent behavioural and emotional difficulties developing later in life (Garmezy, 1993).
Children spend a substantial portion of their time in schools, and while traditionally schools have been expected to focus on developing the academic proficiency of their students, they can also play a central role in providing educational opportunities that develop their social and emotional skills (Durlak et al., 2011). Positive and supportive learning and social environments help to build connectedness to schools and engagement in learning. In turn, this connectedness is associated with positive health, social and academic outcomes (Blum, 2005; Bond et al., 2007; Jose et al., 2012; Roffey, 2012; Sánchez et al., 2005). A sense of connectedness or belonging to school is a significant protective factor for young people and contributes to building their resilience (Resnick, 1997; Resnick et al., 1997).
The benefit of schools formally teaching social and emotional skills is supported by extensive research. In a meta-analysis of programmes that sought to enhance students’ social and emotional learning, Durlak et al. (2011) identified the most effective as those that provided lessons that were sequenced, used active learning strategies, devoted sufficient time to skill development and had explicit learning goals. Their meta-analysis indicated that students receiving such programmes felt more connected to their school and improved on measures of positive behaviour, such as classroom discipline and attendance (Durlak et al., 2011). These benefits have also been identified in a number of other studies of social and emotional learning programmes (Bond et al., 2007; Bond and Hauf, 2004; Goleman, 2008; Payton et al., 2008). In addition, students in these programmes were less likely to engage in anti-social behaviour, such as bullying, fighting and problematic substance use, and they evidenced a reduction in mental health problems such as depression, anxiety and alienation (Bond et al., 2007; Durlak et al., 2011; Goleman, 2008; Payton et al., 2008). In terms of empirical academic data, when schools offered programmes on social and emotional learning, the achievement scores of their students improved by approximately 11% (Durlak et al., 2011).
Social and emotional skills are beneficial in their own right, but they also interact with, and potentiate, cognitive skills to improve academic performance and enhance children’s likelihood of achieving positive outcomes in later life. The Organisation for Economic Co-operation and Development’s (OECD, 2015) Skills for Social Progress report considered that schools need to develop the ‘whole child’, with education that provides a balanced set of cognitive, social and emotional skills, so that they are better equipped to face life’s challenges. This recognition that schooling is not just about academic outcomes has gained increasing traction in Australia in recent years, with the National Ministerial Council on Education, Training and Youth Affairs stating that schools have a role in promoting the wellbeing of young Australians (Centre for Education Statistics and Evaluation, 2015; Ministerial Council on Education, Employment, Training and Youth Affairs, 2008). However, despite evidence of the wide-ranging benefits of social and emotional education, it is not routinely provided in secondary schools, not just in Australia but worldwide. Uptake, effective implementation and sustainability are such problems globally that the OECD (2015) report called for better translation of evidence from intervention research so that policymakers would be better informed about how to implement evidence-based social and emotional education programmes in schools.
This pilot study with Year 7 (average age 13 years) and Year 8 (average age 14 years) Australian, middle school students seeks to contribute to a better understanding of what can be achieved when the best evidence of effect forms the basis of a classroom social and emotional education programme capable of being accommodated in a school’s health curriculum. This will provide a practical example of research translation in the Australian context.
Social and emotional wellbeing is influenced by both risk and protective factors. Accordingly, this study will seek the measure of each. Wellbeing risk will be assessed through a measure of psychological distress (Gladstone et al., 2006; Masten, 2011). Wellbeing strength will be assessed using functional resilience concepts, where the presence of internal and environmental resilience factors is predictive of wellbeing (Olsson et al., 2003). In doing this, the study seeks to inform the development of large-scale efficacy and effectiveness research trials.
Methods
Study design
This is a small-scale pre–post pilot study, without a control group. An Australian middle school was recruited to the study and active consent was sought from all students and their parent/primary carer in Years 7 (average age: 13 years) and 8 (average age: 14 years). In total, 56 students agreed to participate in the study: 22 boys and 34 girls. No students refused to participate in the study. In July 2015, the second author (HC) led a 2-day workshop to train teachers in the delivery of the programme. Baseline survey data were collected later that month. The 10-lesson social and emotional education curriculum was taught weekly from late July to late September. Although designed for delivery in 40- to 50-minute sessions, the programme was truncated to fit in the half-hour period available as part of the school’s pastoral care time. Post-intervention, survey data were collected in December from 44 students who had completed the baseline survey: 20 boys and 24 girls. The same survey instrument was used at baseline and post-intervention. A student-generated code, based on easily remembered fragments of personal information, was used to maintain anonymity while allowing matching of individuals over the course of the study. Confidentiality was demonstrated by having participating students seal their completed survey forms in envelopes, which were then collected by researchers and immediately taken out of the school.
Subsequent to programme completion, three focus groups were conducted on a volunteer basis, respectively, with three teachers, eight Year 7 students and four Year 8 students. The purpose of the focus groups was to assess fidelity of implementation, student engagement and utility of the material taught.
The social and emotional education programme
The 10-lesson social and emotional curriculum was derived from the Victorian Education Department’s Social and Emotional Learning Resources (Cahill et al., 2014a). These resources were informed by research in the fields of resilience education and social and emotional learning that identify emotional competence, social skills, problem-solving, autonomy and a sense of purpose as key attributes of resilient young people. The resources also emphasise the importance of appropriate pedagogy, and highlight the importance of collaborative and developmentally appropriate learning strategies to develop social and emotional skills, while also engaging explicitly with the key content areas of emotional literacy, decision-making, problem-solving, positive self-regard, stress management, positive coping, help-seeking and peer support. The collaborative learning strategies included small-group problem-solving discussions, critical thinking tasks, skills-development exercises, role-play and simulation (Benard, 2004; Cahill et al., 2014a; Payton et al., 2008). The curriculum covered the following topics:
Emotional literacy;
Personal strengths;
Positive coping strategies;
Problem-solving strategies;
Stress management and emotional regulation;
Help-seeking and peer support.
The teachers delivering the programme were provided with 2 days of professional training. Grounding in the evidence base informing the research was provided, and interactive, student-centred pedagogy was emphasised by participatory modelling of each curriculum activity. This approach has its basis in implementation studies that have demonstrated these methods are crucial to the effectiveness of programmes seeking to build health decision-making capability (Cahill et al., 2014b; Midford et al., 2012, 2014).
The student survey and measurement of change
The student survey was designed to measure the social and emotional wellbeing of the student group over time. It comprised two elements: measurement of psychological wellbeing and measurement of resilience factors. The psychological wellbeing of students was assessed using the Kessler 10 (K10) instrument, which is a widely used, brief, standardised and validated screening tool for non-specific psychological distress (Australian Bureau of Statistics, 2012; Kessler and Mroczek, 1994; Rickwood et al., 2015). Students’ internal and external resilience factors were primarily measured using an abbreviated version of the Resilience and Youth Development Module (RYDM) of the California Healthy Kids Survey (CHKS) (California Department of Education, 2014; Hanson and Austin, 2003; Hanson and Kim, 2007). This instrument is one of the few to demonstrate conceptual adequacy by examining resilience using a multi-level approach (Windle et al., 2011). Additional items were added that measured students’ connection to their class and their social and emotional skills.
The K10 contains 10 questions about symptomatic feelings of anxiety and depression and is scored using a five-level frequency scale ranging from ‘none of the time’ to ‘all of the time’; 1 is the minimum score and 5 is the maximum score on each question, yielding a minimum total score of 10 and a maximum total score of 50. The psychological distress categorisation in this study was drawn from an amalgam of work by the Clinical Research Unit for Anxiety and Depression (CRUfAD) at the University of New South Wales and Andrews and Slade (2001). A K10 score of 10–15 indicates no or low psychological distress, 16–29 indicates a medium level of psychological distress and 30–50 indicates a high level of psychological distress (Australian Bureau of Statistics, 2012).
The RYDM contains 56 items designed to measure internal and environmental factors that have been linked to positive developmental outcomes (Benard and Slade, 2009).
As this was a small-scale pilot study that sought to identify the proximal impact of a social and emotional school curriculum, analysis has focused on construct groupings derived from the personal abilities of the students and the influence of their school environment. The RYDM scores were interpreted using the construct groupings reported by Furlong et al. (2009): ‘Internal Assets’ (12 items in 4 subscales: Self-efficacy, Empathy, Problem Solving, and Self-awareness) and ‘School Resources’ (14 items in 3 subscales: School Support, Meaningful Participation, and School Connectedness). In addition, the RYDM subscale, ‘Cooperation and Communication’ (3 items), and two non RYDM scales, developed for this study, ‘Class Connectedness’ (4 items) and ‘Social and Emotional Skills’ (6 items), were included because the curriculum emphasised these issues. The School Connectedness items from the RYDM ‘School Resources’ subscale and the ‘Class Connectedness’ scale used balanced, 5-point Likert scaling with item response options ranging from ‘strongly disagree’ to ‘strongly agree’. All other items used an unbalanced, 4-point Likert (1932) scale with response options ranging from ‘not at all true’ to ‘very much true’.
As treated analyses of the six measures were conducted using SPSS v23 (note the ‘School Resources’ measure was analysed as two separate scales because some items used a 5-point Likert scale, while others used a 4-point scale). Paired t-tests were used to assess whether there was a significant change in any of the measures between baseline and post-intervention.
The focus groups
The core questions put to the focus groups were as follows:
Was the programme useful?
Was the programme enjoyable?
What were the best elements of the programme?
Was the programme implemented as intended?
Did the programme make a difference in terms of feelings and behaviour?
Detailed notes were taken during the course of each focus group. The interviews were coded for reoccurring patterns and then analysed thematically to identify key issues of programme utility, engagement and fidelity of implementation.
Results
The mean baseline and post-programme scores and standard deviations for all measures are presented in Table 1, accompanied by their associated paired t-test values, degrees of freedom and p statistics.
Baseline and post-intervention student social and emotional wellbeing survey, component scores.
SD: standard deviation; K10: Kessler 10; RYDM: Resilience and Youth Development Module.
A total of 44 students provided data pre- and post-intervention. However, for one or two students, data were missing on some measures.
p < .05.
Student survey
K10 Psychological Distress Scale
Both the mean baseline (25.07) and post-intervention (23.09) symptomatic scores were in the range indicative of medium-level psychological distress. There was an improvement in the scores of students, subsequent to receiving the social and emotional education programme, of just under 8%. This approached significance (t = 2, df = 42, p = .053).
RYDM Internal Assets construct grouping
At baseline, the mean item score was 3 (pretty much true). The highest indication of support for each Internal Asset statement was 4 (very much true). There was no significant change in the Internal Assets measure from baseline to post-programme.
RYDM School Resources construct grouping
At baseline, the mean score for the 5-point scale items was 3.52. This was midway between neither disagreeing nor agreeing (score of 3) and agreeing (score of 4) with the School Resource statements. The mean score for the 4-point scale items was 2.69. This was just over midway between considering the School Resource statements a little true (score of 2) and pretty much true (score of 3). There was no significant change in the School Resources measures from baseline to post-programme.
RYDM Cooperation and Communication Sub-Scale
At baseline, the mean item score was 2.91, where 3 indicated the Cooperation and Communication statement was pretty much true. The highest indication of support for each statement was 4 (very much true). The mean score on this measure increased by just over 6% from baseline to post-programme (mean item score: 3.09), which was significant (t = −2.34, df = 42, p = .024).
Class Connectedness Scale
At baseline, the mean item score was 3.24. This was closest to the midpoint of neither disagreeing or agreeing (score of 3) with the class connectedness statements. The mean score on this measure increased by just under 10.5% from baseline to post-programme (mean item score: 3.57). This was closest to the agree statement (score of 4), and the change was significant (t = −2.46, df = 43, p = .018).
Social and Emotional Scale
At baseline, the mean item score was 3.02. This was marginally above considering the Social and Emotional statements pretty much true (score of 3). There was no significant change in the Social and Emotional measure from baseline to post-programme.
Focus groups
Teachers were generally positive about the social and emotional education materials provided but believed that longer lesson periods would have enabled them to deliver each lesson more interactively, rather than in some cases simply presenting information in order to cover the content. They reported that their students got more out of the programme when there was time for active participation.
Students reported enjoying the programme, particularly games, role-plays and group work opportunities. They saw the value in mixing with students they did not normally speak with. They also reported using skills in calming friends down during playground fights as a result of the programme. The students indicated that the programme was delivered with partial fidelity. They referred to the use of lectures by teachers, indicating some use of didactic methods, which were not part of the programme design. Year 8 students did not complete the help-seeking activities, and role-play activities were not provided for one class.
Discussion
The measure of psychological distress (K10) indicated that on average the participating Year 7 and 8 students were experiencing a medium level of psychological distress at baseline. Scores on this measure did improve post-programme by just under 8%, which approached significance at p = .05, but the average remained in the medium range. This suggests that the psychological state of Year 7 and 8 students, based on the K10 questions about anxiety and depressive symptoms, is less than ideal. These findings should not come as a surprise as a recent survey of 43,799 Australian school students found that approximately one-third reported feeling unhappy, depressed and constantly under strain (Fuller et al., 2015). This indicates that the psychological state of students in this study is not out of the ordinary and together with findings from Fuller et al. (2015) provides a strong argument for teaching school students social and emotional skills to better deal with their feelings and address the issues that underlie them.
Student scores on the RYDM measure of Internal Assets, which can be characterised as individual resilience factors associated with positive development, did not change over the course of the programme. Similarly, scores on the RYDM measure of School Resources, which can be characterised as protective factors within the school environment, did not change. This is understandable in terms of the short period between baseline and post-programme waves of data collection. Elias et al. (2003), in their examination of the factors that support the implementation of successful education programmes, found that reflection, insight and resultant development of different perspectives take time, and this was likely to have been the case in this study.
Interestingly, the Class Connectedness Scale, developed for this study, did evidence significant change from baseline to post-programme. This could be considered somewhat at odds with an unchanged School Resources score, but is probably best explained in terms of the programme’s proximal and personal influence. The programme was conducted by one teacher with the same class of student peers, and the focus group findings indicated the students enjoyed the activities and valued mixing with peers outside their normal friendship group. Blum (2005) considered that connection to school is less about policies and regulations and more about the individual actions of teachers and the classroom culture they create. Teachers build connectedness to classroom and through that to the school. Given the classroom focus of the study’s social and emotional education programme, it is understandable that change in terms of connection to school would first occur in the classroom context. The change also suggests that measurement of school connectedness is likely to be more sensitive if focused on particular situations involving personal interaction.
Student scores on the RYDM measure of Cooperation and Communication did improve significantly from baseline to post-intervention, and this was likely due to the interactive quality of the programme activities, which by their nature required cooperation and communication and an explicit focus on teaching cooperation and communication skills. Such skills have been identified in two meta-analyses by Durlak et al. (2010, 2011) as features of effective social and emotional education programmes. Additionally, because the skills were put into practice during the course of the programme, there was close link between programme activity and indication of effect. This immediacy of effect conforms to findings by Sklad et al. (2012) that social and emotional programmes had their greatest immediate effect on behavioural skill outcomes, such as social and emotional skills and anti-social behaviour, whereas the effect on more complex, less skill-related outcomes, such as mental health disorders and substance abuse, was much less. The implication is that programme activity that focuses on developing instrumental behavioural skills is likely to produce greater change in the short term.
The findings in relation to development of cooperation and communication skills suggest that similar beneficial change should have been achieved on the measure of Social and Emotional Skills, but this was not the case. Student scores on the measure of Social and Emotional Skills, developed for this study, did not change over the course of the programme, and explanation draws on the same understanding of what is likely to produce short-term change (Sklad et al., 2012). In terms of social and emotional skills, the programme sought to achieve change more indirectly through activities that required an abstract understanding and use of these skills, rather than specific actions, as was the case with communication and cooperation. Accordingly, the influence of the programme was less direct and immediate and less likely to produce short-term change. This suggests that the programme should be modified to provide more directly applicable social and emotional skill development activities, and measurement should allow more time for change to develop.
The positive focus group feedback as to engagement with the programme and the practical benefits it provided is encouraging, given the implementation limitations mentioned by both teachers and students. These fidelity breakdowns are consistent with those commonly reported in the implementation literature, which identifies that teachers unfamiliar with such teaching methods commonly omit the role-plays and critical thinking tasks (Herbert and Lohrmann, 2011). Breakdown in programme fidelity is of concern as there is a strong association between the fidelity of programme delivery and positive student outcomes. When programmes are not implemented with high fidelity, and specifically, when the collaborative learning tasks are not used, programmes do not deliver the same outcomes (Dusenbury et al., 2003; Ransford et al., 2009; Stead et al., 2007).
The results of this small-scale pilot study are mixed, but understandable in terms of its design, methodology and implementation constraints. The constraint of less time to conduct the lessons as intended affected fidelity, both directly in terms of limiting coverage of material and indirectly in terms of forcing teachers to use didactic rather than interactive delivery methods in some instances, so as to cover the set content. It also meant that the programme was less engaging for students because it reduced their opportunity for active participation. Feedback from both teachers and students indicated this structural element of the programme cannot be compromised if the programme is to be delivered for best effect.
A large-scale project, with more focus on direct skill development, greater student numbers and a longer interval between baseline and post-programme data collection waves, would likely have achieved greater measurable improvement in social and emotional skills (Durlak et al., 2010; Sklad et al., 2012). Additionally, the incorporation of a control group would have given greater confidence that the observed changes were in fact due to the influence of the programme. These are acknowledged limitations of the research. However, this small-scale pilot study has demonstrated that an evidence-based classroom social and emotional education programme for Year 7 and 8 students can be integrated into a school curriculum pastoral care structure, and can achieve significant change in some areas of social and emotional functioning, despite truncated delivery. This provides a sound basis for conducting a large-scale, multi-school control group study where the social and emotional programme is delivered in accordance with its design parameters. Such a study would give greater confidence as to programme effects and their generalisability to other students and other schools.
Footnotes
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was funded by Charles Darwin University through the Developing Areas of Research Strength (DARS) programme.
