A school-wide digital programme has context specific impacts on self-regulation but not social skills

Abstract

It is assumed that digital tools with ubiquitous classroom use have affordances for student agency and a range of social skills. However, few studies have explored the generalised impact of everyday digital classrooms on self-regulation and empathy, perspective taking and prosocial skills. Ten and 11 year old students’ (n = 115) ratings of self-regulation, social skills and personality were examined in relationship to school-wide practices and instructional foci in two groups of schools (n = 9) involved in a digital innovation serving low-SES culturally diverse communities. In an early adopting group, students had received a high dosage of three or 4 years of 1:1 digital pedagogy, and in a later adopting group of schools, students had received a low dosage of only 6 months. This natural experiment revealed a context specific effect where high dosage students rated their regulation in digital contexts higher, but not in more general non-digital contexts. However, personality scores particularly those related to self-regulation, were higher for the high dosage students. There were no differences in social skills. The differences were related to the strong focus in the digital innovation on aspects of self-regulation. There was less focus on social skills in the digital innovation. More deliberate teacher augmentation and instructional designs for social skills may be required to capitalise on the affordances of digital tools. School-wide practices, while necessary may not be sufficient to enable the generalisation of skills without this deliberate teacher focus.

Keywords

digital programme school-wide instruction social skills self-regulation

Introduction

The adoption of digital devices and use of the internet, have increased markedly, both generally (Chassiakos et al., 2016), and in educational settings (OECD, 2021). These tools are part of the increasing educational focus on the ‘21^st Century skills’ in schools (National Research Council [NRC], 2012), which contribute to important educational outcomes and to long-term personal and collective benefits (García, 2016). Among the valued skills are those which underpin interpersonal capabilities (e.g., perspective taking, empathy and prosocial skills) and those that are intrapersonal (e.g., self-regulation and persistence) skills. Knowing how the pervasive use of digital tools in educational settings best promotes or possibly undermines these valued skills, is a major research challenge.

Background

Ubiquitous digital usage creates both risks and opportunities for the development of self-regulation. Both positive and negative effects are possible depending on the conditions of use (Chassiakos et al., 2016). Digital multitasking, can have costs for early cognitive and brain development (Courage et al., 2015) and, in older children, increase distractibility (Courage et al., 2015) and disengagement (Liau et al., 2015). But positive impacts on self-regulation have also been documented in classrooms where digital tool use is linked with greater engagement and increased agency (Clark et al., 2016; Jabbar and Felecia, 2015; Karich et al., 2014; Zheng et al., 2016).

Similarly, usage can impact social skills both positively and negatively (Chassiakos et al., 2016). An increase in online bullying behaviour, reflecting weaknesses or undermining in both empathy and perspective-taking in young people, has been documented amongst students frequenting social media sites (Gardella et al., 2017), but increased social connectedness has also been found for some groups of users (Chassiakos et al., 2016). In addition to these variable effects, some relationships are not linear. Negative outcomes on adolescents’ mental health have been found at both extremes of low/no Internet usage and heavy usage (>2 hours/day) (Belanger et al., 2011). The OECD (2015) identified a cut point at 6 hours or more per weekday of internet use outside of school associated with lowered well-being on a range of measures

In addition to time, what matters to self-regulation and social skills are the activities in which students engage. Social media and games may be powerful ‘impulsogenic’ forces across contexts, at high levels undermining individual self-control and self-regulation (Duckworth and Steinberg, 2015). Aiken (2016) describes an online disinhibition effect and an online escalation effect associated with social media use. The former is exhibited in impaired judgements about appropriateness of actions and increased impulsivity, and the latter in amplification of problematic social and emotional skills, seen in online aggression and reduced empathy in messaging. Activities and content with these digital tools can promote dysfunctional stereotypes and inappropriate norms and values such as those for sexual behaviour which further undermine empathy and perspective taking (Chassiakos et al., 2016).

These findings suggest a range of possible impacts of digital usage, but are not specific to classroom usage. What happens in classrooms is more constrained in terms of time, content and activities than what might happen outside of school, and it is likely that effects are mediated by what teachers and schools do. In the following sections the non-digital evidence for conditions that support these skills not specific to ubiquitous digital use, is summarised followed by what is known for ubiquitous digital usage.

Two conditions for promoting self-regulation and social skills in schools

Two general conditions for promoting self-regulation and social skills in schools are identifiable in the research literature, not specific to digital usage. Specific forms of instruction (e.g., structured curriculum activities with feedback and contingent scaffolding) are associated with social and emotional skill development in young people (Durlak et al., 2011). Effective social and emotional learning interventions focus on a range of social and interpersonal skills such as self-awareness, self-management, social awareness, relationship skills, and responsible decision-making and have this instructional focus (Jones and Doolittle, 2017). The largest effects are found when teachers embed this instructional focus in day-to-day curriculum delivery (Durlak, et al., 2011).

But effective SEL programmes have an additional component, they change the wider school and classroom context to create caring teacher and student relationships with clear norms, expectations, and consistent school-wide practices (Durlak et al., 2011). Both components are identified in evidence about the development of executive function at school (Cumming et al., 2020) and they are a feature also of effective interventions focused on specific components of self-regulation and social skills (Jacob and Parkinson, 2015).

This second component is conceptualised as reflecting school climate, school discipline, or properties of communities of practice. School climate which comprises positive mutually trusting relationships, students feeling safe at school and health and disciplinary policies, is related to social and emotional outcomes (Charlton et al., 2020). School discipline, which has fair and conducive environments for learning and internalising rules, support for empathy and autonomy and positive teacher-student relationships, is linked to increased self-control (Li et al., 2021).

The significance of building the properties of communities of practice for social skills has been demonstrated in classroom studies of collaborative reasoning in science, or in collaborative and dialogic forms of comprehending. These instructional designs include a focus on the interpersonal and intrapersonal skills necessary to engage effectively in the practices of the face-to-face communities. Developing expertise in collaborative reasoning, for example, requires perspective taking and aspects of cognitive and emotional empathy. Experimental demonstrations in face-to-face ‘communities of learners’ show these skills can be used in reading and writing across different subject areas such as science and English language arts (Brown, 2016; (Rapanta et al., 2013).

These face-to-face communities demonstrate the significance of deliberately designing activities, and explicitly establishing values, beliefs, norms, knowledge and skills needed, for a functioning community. Well-designed communities have been associated with a snowball phenomenon through which students' learn from each other and skills grow in strength and generalise across different activities (Anderson et al., 2001; Reznitskaya et al., 2007).

Impacts of digital usage on self-regulation and social skills: teaching

The significance of teacher mediation is partly captured in an hypothesis of a teacher augmentation effect which increases the likelihood of valued outcomes in digital contexts (McNaughton et al., 2018). The evidence suggest that both overall effectiveness and the consistency of direct effects of the digital tools themselves, depends on teacher mediation which adds instructional power to the direct effects of digital technologies. For example, augmentation can be provided indirectly through personal tool and activity selection or more directly through deliberate instruction with teacher guidance and feedback (Jabbar and Felecia, 2015). Similarly, digital tools have affordances for enabling social skills for collaborative learning in classrooms, but different instructional designs to promote collaboration are associated with different levels of effects on peer interaction (Sung et al., 2017). Beyond this, the most recent reviews available indicate little is known about utilisation of affordances and the role of teachers for the full range of valued social skills (Clark et al., 2016).

Similarly, digital tools have affordances for promoting self-regulation (Heisawn and Hmelo-Silver, 2016; Lorena et al., 2017). Their use in classrooms has been are associated with greater engagement and increased agency, using measures of time on task (persistence) and task motivation (Clark et al., 2016; Jabbar and Felecia, 2015; Karich et al., 2014; Zheng et al., 2016). However, it is unclear whether ubiquitous adoption has a generalised effect on self-regulation or executive functioning (Karich et al., 2014). But, augmentation effects are clear. For example, in game-based learning, the strongest effects on intrapersonal skills are found when features of the game design are enhanced by teacher scaffolding (Clark et al., 2016).

School-wide promotion in digital environments

There is little research, apart from the specific teacher augmentation evidence, into the second component of a school-wide community of practice enhancing self-regulation and social skills with ubiquitous digital usage. It is likely aspects of the same conditions will apply as suggested by research by Yang et al. (2021). They examined the moderating role of school climate on Social and Emotional Leaarning (SEL) related to self-regulation and social skills and the degree to which students experienced cyberbullying. The more positive the school climate the greater the relationship between self-regulation competencies and lower reported rates of cyber victimisation. The impact was not specifically related to what happened during school time, which suggests a generalizable effect.

This study

More evidence is needed to design effective digital conditions within and across schools to promote self-regulation and social skills. The research evidence reviewed here, not specifically related to digital environments, shows they are malleable, but their expression may be context specific (Duckworth et al., 2014); Eisenberg et al., 2015). The following study capitalised on a natural experiment (Campbell and Stanley, 1963) to provide such evidence. It took the form of a ‘Difference in Differences’ quasi-experimental design where an intervention is experienced by one a group of students or schools, at a point in time but not others at the same point in time in a naturally occurring setting (Gopalan et al., 2020). While quasi experimental designs are increasingly used to answer questions about ‘what works’ in everyday school contexts they are still limited in terms of establishing causation. Gopalan et al. (2020) point out they could contribute more given greater robustness.

Among the ways to increase that robustness is to know more about the counterfactual. In the following study the comparison is based on the idea of dosage. One group of schools had extensive experience over several years of a whole school (ubiquitous) digital design and the other group was in the early stages of implementing that same design. The national context at the time this research took place was of widespread and rapid implementation of digital teaching and learning. The amount of time 15 year olds in Aotearoa New Zealand spent on the internet in a week doubled from 2012 to 2018 to 42 hours a week, with 29 hours outside of school and 13 hours inside school, the third highest of OECD (2021) countries. The nine schools were engaged in a research-informed, school-wide digital programme using an innovative pedagogy (described further below), specifically designed to raise the achievement of their mostly Māori (indigenous) and Pasifika children (from Pacific Islands), who were from low SES communities. In addition to personal digital devices and the innovative pedagogy, there was ubiquitous access at school and at home, and a community set of norms, values and practices for using the devices and the internet. A collaboration with researchers in a design-based partnership meant educationally significant gains in writing were being achieved, at rates that were greater than national patterns and at levels that are significantly higher than other low SES schools (Jesson et al., 2018). The measures reported here were collected towards the end of 2015 and the middle of 2016.

Given the paucity of evidence about the conditions in schools with ubiquitous digital environments for promoting self-regulation and social skills and the naturally occurring differences in implementation, two questions were posed.

1. What was the impact of extended experience (high dosage) versus limited experience (low dosage) of a school-wide digital programme on self-regulation and social skills in primary school-aged students?

2. How specific to the digital contexts were any effects on self-regulation and social skills?

Method

Participants were drawn from urban primary schools in a large city in Aotearoa New Zealand. and were designated ‘low decile’, serving low SES communities (see www.education.govt.nz/school/funding-and-financials/resourcing/operational/ funding/school-decile-ratings/). They were naturally divided into two groups who were at different stages of adopting the same digital learning programme at their respective sites. The digital programme was designed to address persistent issues with unequal and inequitable access to learning opportunities for students from these communities, and inequitable educational outcomes. The present study took place within an ongoing design based research partnership in which detailed classroom observations and students achievement measures were being collected (see Jesson et al., 2018).

School-wide digital environment: High and low dosage

A shared pedagogy with digital tools and infrastructure for ubiquitous access to tools and the internet had been adopted in the two clusters of schools. One group of schools had been engaged in the digital learning programme for more than 5 years, designing, testing and redesigning their pedagogy for all students from Years 3 (8 year olds) to Year 13 (18 year olds). They were termed the early adopting (EA – high dosage) group. The second cluster was at the beginning of the implementation phase, having been engaged with the programme for less than a year (at the time of this study). They were termed the later adopting (LA – low dosage) group. Two year groups were selected for this study, Year 5 (10 year olds) and Year 6 (11 year olds). Testing was at the end of the academic year for the EA students, which meant they had been in the programme for almost three or four full academic years starting as 8 year olds. The Year 5 and 6 students in the LA schools were tested at the middle of the school year after the schools had been implementing from the start of the academic year, which meant they received only two terms (in school year comprised of 4 terms).

The schools did not have an add-on school-wide programme for self-regulation and social skills but they had developed a ‘community of practice’ which incorporated attention to these skills and this was complemented by aspects of the classroom instruction. There was an explicit school-wide commitment to Digital citizenship (how to be a good citizen in the 1:1 environment and when online), socialised in each school, and at classroom and individual student levels through a set of norms, values and practices termed being cybersmart for “Being accountable for our actions.” The community was where students “learn to make smart decisions and [where they ] understand that every time they connect, collaborate and share online it combines to create their digital footprint”. Students sign a kawa (a Māori concept meaning protocols) of care agreement, which included detailed statements about the responsibility of the learner. These had a focus on managing learning and the responsible use of the equipment such as ‘respecting the equipment’, ‘being in the right place, at the right time when online’, learning new things, keeping the device safe, and ‘being responsible’ for the activities undertaken with device. One of the nine intentional statements referred to concern for others (“I will respect others by using this device to interact in a kind, positive and helpful way”). These norms, values and practices were upheld by teachers and parents, both of whom sign an agreement to support these skills.

Similarly, there wasn’t an added classroom instruction programme for self-regulation and social skills, but the instructional design had foci on self-regulatory strategies and social skills. Students researched topics under teacher guidance, engaged in composing multimodal digital creations, which explained their learning, and then posted these on their individual blogs for both in-class and wider audiences. This provided a template for teaching and learning using the devices and applications (e.g., Google Apps for Education). All core curriculum topics were designed around a learn phase (e.g., gathering and using textual resources); a create phase (e.g., using resources to develop an informed, digital response to a topic) and a share phase (e.g., creating a Digital Learning Object (DLO) to share with others).

In both the EA and LA clusters, observations show that nine out of 10 students used personal devices during lessons, engaging in activities associated with these in about half of each lesson (see McNaughton et al., 2022). Around half of the lesson time was spent in online collaboration and face-to-face collaboration, and in a quarter of the lesson time students created DLOs. Frequent feedback for collaboration and appropriate use was observed (Jesson et al., 2018). A focus on self-regulated learning occurred in between a third to one-half of observation intervals mostly prompting for independent strategies (e.g. “When you are reading why do you ask questions?”). In both the Create and the Share phases there was collaborative activity, either face to face or on line. The initial learn phase similarly required personal agency. More details of instruction were collected in this study and are reported below.

Participants

While the schools in each cluster had the common feature of serving the lowest income and employment communities, other features varied including the size of schools, ethnicity and age group distributions (schools with Years 7–13 were slower to join the LA cluster). To achieve as close a match as possible Year 5–6 students and teachers from both clusters were invited to participate in this study. Participation varied between clusters (see Table 1). There were more Māori students (from the indigenous community) in the LA cluster and more Pasifika students (from Pacific Islands communities) in the EA cluster. The students were matched as closely as possible on age and reading achievement, but fewer students in the EA cluster were reading at expected age-appropriate levels on the normed English language reading assessments (67% of students compared with 79%), reflecting in part that many had a first language other than English. Consistent with the national picture students in both schools were generally frequent users of the internet after school (self-described from a student questionnaire). The median frequency in each cluster was 3 (frequent) on a 3 point scale: limited (less than once a week); moderate (2–3 times a week); frequent (most days), and the typical duration was 4.4 hours (1.7 school related and 2.7 fun/recreational).

Table 1.

Year 5 and year 6 students in early adopting and later adopting clusters.

	Cluster
	Early adopting	Later adopting
Total (N)	76	39
Age (years)	10.1	9.9
Gender
Male	46%	59%
Female	54%	41%
Ethnicity
Māori	13%	62%
Pasifika	64%	18%
European	4%	13%
Asian	13%	8%
Other	5%	0%
At reading level	67%	79%
Home internet use (median)	Most days	Most days

Given the age of the student participants, legal guardians were required to consent to their participation. The study was approved by the ethics committee at the researchers’ home university.

Measures

Student details were gathered from school records. In addition, an online student questionnaire was used to gather demographic data such as internet use as well as three measures of student self-regulation and social skills. These provided a gradient of specificity to context of use. The Big Five Personality Inventory was the most general, being a measure of personality traits. In addition, more focused self-ratings of aspects of self-regulation and social skills were used, which were not specific to digital contexts. Finally, these same measures of components skills were redesigned for self-ratings specific to digital contexts. Observational measures of classroom instruction related to self-regulation and social skills were collected to provide evidence for instructional foci and the degree to which implementation was similar in both EA and LA schools, and teacher questionnaires probed beliefs and practices relating to the self-regulation and social skills. The student and teacher questionnaires were provided in electronic format and completed by students and teachers during class time. McNaughton et al., (2022) provide further details and a copy of the questionnaires.

The Big Five Personality Inventory

The widely used Big Five Personality Inventory (BFI - John et al., 2008) provided general information about dispositions related to self-regulation and social skills. The dimensions are: extraversion (sociability, engages in class activities), agreeableness (empathy, wants to help and sympathises), conscientiousness (self-regulation, perseveres at activities), neuroticism (framed positively as emotional stability) and openness to experience (curiosity, appreciates new experiences). The dimensions are relatively stable, responsive to interventions and dependent on situational factors. They account for a large part of individual differences in personality attributes and have robust predictive validity for a range of life outcomes including mortality, divorce, and occupational attainment (Kankaraš, 2017). Importantly, similar personality structures have been identified across a range of countries.

The BFI uses self-report ratings on general disposition statements using a five-point Likert scale (from strongly disagree to strongly agree). Our own reliability testing using Cronbach’s alpha illustrated issues with using the full BFI survey, particularly with items that used ‘double negatives.’ We removed those items and created a revised version (see McNaughton et al., 2022 for details) Confirmatory factor analysis (CFA) was used to evaluate the validity and reliability of the revised measurement. The final model fit indicates reasonable validity of the five dimensions (Comparative Fit Index CFI = 0.916; Tucker-Lewis Index TLI = 0.895; Root Mean Square Error of Approximation RMSEA (90% confidence interval) = [0.034, 0.063]; Standard Root Mean Square Residual SRMR = 0.061), providing satisfactory psychometric properties of the tool.

Self-regulation and social skills in non-digital contexts

Measures of two executive function constructs, attention focus and inhibitory control, were used which are basic processes in process models of self-control and self-regulation (Duckworth et al., 2014; Zimmerman, 2008). A 5 point self-rating scale with six items referred to specific (but non digital) activities from Does not describe me well to Describes me very well (Capaldi and Rothbart, 1992). Only one of the items referred to school-related activities (e.g., “It is easy for me to really concentrate on homework problems”). Four existing measures were used for social skills: affective and cognitive empathy (Jolliffe and Farrington, 2006); perspective taking (Davis, 1980); and prosocial behaviour (e.g., ‘I try to be nice to other people’; ‘I care about their feelings’; ‘I often volunteer to help other people’) using the subscale of the Strengths and Difficulties Questionnaire (SDQ, Goodman, 1997). The social items, 3 or 4 items for each construct, were not specific to school contexts (e.g.) “Before criticizing somebody, I try to imagine how I would feel if I were in their place” and “I have trouble figuring out when my friends are happy.” Each of these used a 5 point scale, except for the four prosocial items from the SDQ which used a 3 point scale (Not True to Certainly True). The validity and reliability of the measures were checked and items with a double-negative removed. This produced an adequate model fit (CFI = 0.93, TLI = 0.91, RMSEA (90% confidence interval) = [0.015, 0.047], SRMSR = 0.06).

Self-regulation and social skills in digital contexts

Each of the non-digital skill measures was adapted for use in the digital context, using the same rating scale. Revised statements for self-regulation included: “I can get off task easily by different online sites (i.e., emails, chat, games) or devices (e.g., mobile phone) while doing my school work on the computer”. These items were aggregated into one measure of self-regulation in digital contexts. Similarly, the social skill measures in non-digital contexts were adapted and then aggregated into one measure for digital contexts. Sample items included “Before criticizing someone’s game play, chat or posts, I try to imagine how I would feel if they did that to me”.

A simplified two factor model of social skills (cognitive empathy, affective empathy and prosocial skills) and self-regulatory skills (attention focus and inhibitory control) was found to be valid and reliable (CFI = 0.95, TLI = 0.93, RMSEA (90% confidence interval) = [0.009, 0.066], SRMSR = 0.052).

Classroom instruction

A purpose-built tool collected observational data about teacher practices in both EA and LA school classrooms. The tool was designed to complement the teacher and student questionnaires with data on instructional foci and everyday classroom patterns of self-regulation and specific social skills. The observer recorded descriptions of acts involving self-regulation and social skills and captured both instructional interactions by the teacher with students (one-to-one, group or whole class) and independent student (or peer-to-peer) activity, in an attempt to not only observe the direct teaching foci but also what students not directly engaged with the teacher were doing. In order to reduce reactivity, observations were undertaken of regularly occurring and scheduled lessons for English, science or mathematics, and the teacher questionnaires were completed after the observations. The observer alternated between observing the teacher for intervals of 3 minutes and then 3 minutes focussed on the learning resources and independent activities of all students not in individual or group instruction with the teacher. A typical observation for a whole lesson lasted 5 to 6 cycles of 6 minutes (3 min intervals observing the teacher and 3 min intervals on students not with the teacher). In each interval notes were taken to provide detailed examples.

An incident or event recording procedure was followed. Each interval was scored for the presence of any direct teacher instruction or explicit student behaviour related to self-regulation or social skills. Judgements were made about the presence of any focus or specific instance of features of self-regulation such as teacher support for effortful control, transfer of control or goal setting and focus. Examples from classrooms were: (eg maths: “This problem will be challenging for you, but if you persist, using your box of strategies, I know you will succeed”; (eg English: “Once you have finished checking and re-crafting your film review, can you explain in a couple of sentences why it would be important to get some peer review?”). Three features of explicit focus on social skills were observed: (a) perspective taking (eg maths: “I would like you to get your partner’s perspective on the approach to this problem and then decide on the best way forward.”; empathy/being in another’s shoes (eg English: “How does the author help us to think about what it would be like to be in Laila’s position as a woman in Afghanistan”). (c) pro-social behaviour (eg science: “When you are collaborating on your shared report, let’s comment on ways to use more scientific language, but do it in a respectful way.”).

In the EA cluster, 28 intervals were collected from five teachers and in the LA cluster 36 intervals were collected from six different teachers. A sample of 36 intervals across the two clusters was moderated by a second member of the research team who independently observed the same lessons. Average inter-rater agreement across all intervals was 96%.

Teacher questionnaire

Teachers across the age range in the EA (n = 36) and in the LA cluster (n = 10), completed an online questionnaire related to teaching in English, maths or science. Three questions provided data on the frequency: (“How often you get students to use digital technology to learn the (each) skill”), the method they used (“Which of the following teaching methods would you use: direct and explicit teaching, modelling, indirect”); and how they viewed opportunities for teaching (“From your experience does digital technology [reduce, make no difference, add a little, add a lot], opportunities to teach…?”)

Data analysis

Mean scores for the student measures of self-regulation and social skills were compared between EA and LA clusters. Given the small and unequal numbers involved, independent t-tests assuming unequal variances were used. p-values (with a corresponding statistical significance at 5%) were computed for each t-test and effect sizes were measured. A check on the contributions of student variables (age, ethnicity and gender) with cluster membership were then made in an analyses of covariance. Finally, chi-square tests were used to measure the differences in the frequency of teacher instruction and student engagement.

Results

Personality measures-BFI

Significant differences were found between students attending EA and LA schools on four of the BFI personality measures with large effect sizes (see Table 2). The EA students had higher ratings of extraversion (sociability), conscientiousness (grit/perseverance), agreeableness (both cognitive and emotional empathy) and lower ratings of neuroticism (the negative direction indicating lowered anxiety or increased levels of self-efficacy). They were also marginally higher on the remaining measure of openness (curiosity, d = 0.37).

Table 2.

EA and LA Students: Self-Regulation and Social Skills (non-digital contexts) and Personality (BFI).

Social skills, self-regulation	Early adopting			Later adopting			t-test (Note: Assumed unequal variances)
Social skills, self-regulation	N₁	M₁	SD₁	N₂	M₂	SD₂	M_diff	SEdiff	T	p values	Sig	ES
Cognitive empathy	76	3.52	0.59	39	3.85	0.60	−0.33	0.12	−2.81	0.01	**	−0.55
Affective empathy	76	3.20	0.47	39	3.22	0.49	−0.02	0.09	−0.23	0.82		−0.05
Perspective taking	76	3.09	0.63	39	3.04	0.63	0.05	0.12	0.43	0.67		0.09
Prosocial behaviour	76	4.07	0.76	39	4.14	0.82	−0.06	0.16	−0.41	0.68		−0.08
Attention focus	76	3.14	0.59	39	3.04	0.64	0.10	0.12	0.79	0.43		0.16
Inhibitory control	76	3.28	0.78	39	3.12	0.83	0.16	0.16	1.00	0.32		0.20
BFI
Extraversion	76	3.28	0.65	39	2.88	0.45	0.40	0.10	3.88	0.00	***	0.68
Agreeableness	76	3.46	0.61	39	3.06	0.27	0.39	0.08	4.78	0.00	***	0.76
Conscientious-ness	76	3.39	0.58	39	2.91	0.42	0.48	0.10	5.00	0.00	***	0.89
Neuroticism	76	2.86	0.58	39	3.10	0.49	−0.24	0.10	−2.35	0.02	*	−0.44
Openness	76	3.50	0.45	39	3.33	0.51	0.17	0.10	1.79	0.08		0.37

Note: Sig = Significance; Significance of p-values: *** <0.001; ** <0.01; * <0.05; ES = effect size (Cohen’s d).

Self-regulation and social skills: Non digital contexts

No significant differences emerged between EA and LA school students in five of the six social skill and self-regulation categories – affective empathy, perspective taking, prosocial behaviour, attention focus, and inhibitory control (see Table 2). LA students reported significantly higher scores for cognitive empathy than EA students (p < .01; d = 0.55). EA students rated self-regulation skills attention focus and inhibitory control higher with small Effect Sizes differences (d = 0.16; d = 0.20).

Self-regulation and social skills: Digital contexts

EA students rated themselves significantly higher than LA students on the aggregated measure of self-regulation in digital contexts, the effect size was medium to large (p < .01, d = .64). No significant difference (d = 0.09) was found between the two groups on the aggregated social skills in digital contexts measure (see Table 3).

Table 3.

Mean scores for LA and EA students on measures of intrapersonal and interpersonal social skill development in a digital context.

Aggregated measure –digital contexts	Early adopting			Later adopting			t-test (Note: Assumed unequal variances)
Aggregated measure –digital contexts	N₁	M₁	SD₁	N₂	M₂	SD₂	M_diff	SE_diff	t	p-values	Sig	ES
Self-regulation	76	3.29	0.86	39	2.71	1.00	0.58	0.19	3.08	0.00	**	0.64
Social skills	76	3.40	0.76	39	3.47	0.83	−0.07	0.16	−0.44	0.66		−0.09

Note: Sig = significance; Significance of p-values: *** <0.001; ** <0.01; * <0.05 . ES = effect size (Cohen’s d).

Student variables (age, ethnicity and gender) with school cluster membership (EA or LA) were then tested in analyses of covariance. The means of the two clusters remained similar on the social skills measure and no covariate reached significance (F = 0.889, p = 0.490), however, a significant difference was identified on the self-regulation measure (F = 4.534, p < 0.001). Both cluster membership and ethnicity were significant, with most variance attributed to the covariate of ethnicity. Pasifika students rated their self-regulation higher than Māori students (p = 0.025).

Classroom observations

There were no significant differences revealed between clusters in the observations (see Tables 4–5). The most frequently observed focus by teachers in both clusters was on self-regulation, an overt reference by the teachers occurred in nearly half the intervals (EA versus LA: X² 2, N = 64 = 1.2457, p > .05). It took the form of teaching effortful control, transferring locus of control, goal setting and self-management (examples: T: “What do you need to do to work effectively?”; “What do you think you need to do?”; “[use] the podlet…you can compare your ideas to our learning (intention). It will help with your forward planning.”; “I talk in my head all the time. If I don’t understand I think what question do I need to ask to help me understand.”). A self-regulation focus was observed in students’ engagement when away from the teacher, although with less frequency (EA versus LA: X² 2, N = 64 = 1.8445, p > .05); 32% of intervals in EA classrooms and 39% of intervals in LA classrooms (eg, Examples: when a student updated a Blog Log to track Blog goals; or when a group planned who was responsible for writing sections through a shared document).

Table 4.

Frequency of teacher focus and student engagement on self-regulation and social skills in EA and LA schools.

Teacher focus	Number of intervals		Chi-squared test (p = 0.05)
Teacher focus	EA (N = 28)	LA (N = 36)	Chi-squared test (p = 0.05)
Self-regulation	14	13	ns
Social skills	4	10	ns
Student engagement
Self-regulation	9	14	ns
Social skills	9	18	ns

Table 5.

Reported frequency, method of teaching and perceived opportunities to teach self-regulation and social skills: Percentage of teachers EA and LA schools.

	EA (n = 36)	LA (n = 10)
Frequency (weekly/daily)
Self-regulation	58% (21)	100% (10)
Social skills	75% (27)	80% (8)
Teaching method
Direct	78% (28)	90% (9)
Modelling	89% (32)	80% (8)
Indirect	50% (18)	60% (6)
Opportunities^a
Self-regulation	82%	87%
Social skills	67%	67%

^aAverage response by teachers across English, Maths and Science: ‘A little or a Lot more’.

Teachers in both clusters were similar with less of an overt focus on social skills; in EA classrooms 32% of intervals and 28% of intervals in LA classrooms (EA versus LA: X² 2, N = 64 = 0.1929, p > .05). The focus involved concern expressed for a student’s difficulties, but also the need to be able to take another’s perspective (T: “Who’s had a yelling match before? It’s not a debate!” goes on to explain the difference and how a ‘shouting match’ doesn’t take consideration of other’s perspectives; “What are people [characters in an English novel] saying? What are their different perspectives?). Students also had an observable overt focus on social skills, in around a third (EA) and a half (LA) of the intervals (EA versus LA: X² 2, N = 64 = 2.0592, p > .05). The observations almost always involved anticipating or perceiving a peer in difficulty and initiating helping. (For example: seeing a peer struggling with pasting a screen shot of evidence from a text on to a digital worksheet, a peer leans over and demonstrates how to use the app - “Here, it’s like this.” Or having accomplished one’s own next steps anticipating a peer might not know what to do and telling them “Now go and put it on your blog”).

Teacher beliefs about opportunities and teaching

The majority of teachers in each cluster indicated that they taught both sets of skills either weekly or daily, using a range of methods, but more teachers indicated they were aware of a ‘little or a lot’ more opportunities across the three subjects to teach self-regulation (EA M = 82%, LA M = 87%) than Social Skills (EA M = 67%, LA M = 67%).

Discussion

Students in the EA cluster had nearly 3 or 4 years of the digital programme with its school-wide practices and instructional foci, compared with students in the LA schools who had half a year, around a 10th or less of the time. The questions we asked were whether this ‘dosage’ difference was associated with differences in the students’ self-regulation and social skills, and whether this was an effect specific to the digital contexts or was more generalised.

There were differences systematically associated with dosage but they were limited to measures of self-regulation in digital contexts, and the evidence was unclear for a generalised effect. There was a significant difference, with a large effect size, on the personality dimension of conscientiousness, a dispositional measure reflecting aspects of self-regulation (e.g., “I am someone who perseveres until the task is finished”). This might be considered strong evidence for a generalised effect. However, two patterns make this questionable. One is the lack of a substantial effect in near transfer to components of executive function on activities in non-digital contexts (the effect sizes are positive but small). The second is that other personality dimensions, more related to social skills also were higher for the EA students, and yet there was no effect detected favouring EA students for social skills in either digital or non-digital contexts.

At this age, self-regulation is malleable and shows patterns of context specificity (Shulman et al., 2016; Zimmerman, 2008), and this is consistent with the conclusion here of a context effect of digital use on self-regulation. We have previously reported on declining self-ratings of self-regulation in digital contexts across the age range 9 years–12 year olds (McNaughton et al., 2022). The decline is noted widely in the general and non-digital evidence, but the decline in the ratings in digital contexts was systematically lower than more general ratings of self-regulation in non-digital contexts. In addition, the more often and longer students spent time online at home on ‘fun’ activities, the lower the ratings of self-regulation in digital contexts, also supporting a context specificity effect and how digital environments may be powerful ‘impulsogenic’ forces across contexts influencing self-regulation (Duckworth and Steinberg, 2015). The likely explanation for the lower ratings is that students become increasingly aware of difficulties for self-regulating in digital contexts and need support and strategies to regulate (Duckworth et al., 2014). We also reported that high-frequency parental monitoring mitigated the negative effects on self-regulation, contributing to the evidence that socialization agents can be influential (Duckworth et al., 2014; Zimmerman, 2008). This study adds a further finding that the more that students had experienced the overall digital school programme the higher the ratings, indicating the importance of the two components which effective interventions for social and emotional skills have identified as important; communities of practice and instructional foci (Durlak et al., 2011).

Importantly, this study extends previous research on associations between digital tool use and increased engagement or agency amongst students, based on observational measures and survey data (Zheng et al., 2016), to self-ratings of components of self-regulation. The previous research claims that the tools have affordances for increased agency and by extension self-regulation, coming from the capability to take control of and manage learning through archival, curation and creation functions (Clark et al., 2016; Jabbar and Felecia, 2015; Karich et al., 2014; Zheng et al., 2016).

But a similar digital context specific effect did not emerge for the social skills. This is despite similar claims to those for student agency being made about the affordances for social skills (Heisawn and Hmelo-Silver, 2016), especially those associated with collaboration in game-based learning (Clark et al., 2016) and cooperative peer interactions using mobiles computers (Sung et al., 2017). There is less evidence for these effects in the literature to date on the specific social skills measured here (Zheng et al., 2016; Sailer and Homner, 2020), and the evidence shows that any effects are likely to be greater with systematic teacher augmentation, either directly through how activities are designed in the classroom, or through the instructional conditions throughout the activities (McNaughton et al., 2018).

Opportunities for effects on increased social skills were present in the instructional design of learn - create – share (Jesson et al., 2018) and teachers were generally aware of them and reported teaching the skills through direct instruction and by modelling. This leaves the possibility that the teacher augmentation was not as effective, or was needed in different ways, or the affordances were not as strong for the social skills, compared with self-regulation. The SEL intervention literature does not typically look at differential effects on social skills separately from the intrapersonal skills, but it does show the need for step-by-step training, active forms of learning, sufficient time on skill development, and explicit learning goals (Durlak et al., 2011). A teacher focus on social skills was less frequently observed in classrooms in both clusters of schools compared with self-regulation and the form it took was mostly concern for student difficulties, rather than deliberate teaching of the skills, and this was coupled with fewer teachers seeing opportunities to teach these skills. Finally, the cybersmart protocols adopted within and across schools and families had more of a focus on personal responsibility, than on specific interpersonal skills.

The picture is complicated by finding that as well as conscientiousness, personality dimensions which are linked with social skills were higher for EA students, notably extraversion (sociability, engages in class activities), and agreeableness (empathy, wants to help and sympathises). There were much larger numbers (and a higher overall percentage) of Pasifika students attending schools in the EA cluster. It is unlikely that the higher ratings for EA students reflects pre-existing personality differences between Pasifika students and Māori students, as the available evidence indicates no differences in overall SDQ scores between Pasifika and Māori students (Black et al., 2010). However, the Difference in Difference design does not control for this possibility. A possible other explanation is that the digital intervention had a general effect on beliefs about dispositions or personality traits related to both self-regulation and to social skills, but these dispositions were in turn only weakly or not at all related to acts of self-regulation or sociability in general.

It was important to establish that the overall digital programme was similar in each cluster of schools to rule out differences due to different implementations. The classroom observations and teacher questionnaires indicated implementation was similar, if anything the LA teachers were more focused on self-regulation, and all the teachers, students and parents had signed the protocols for being a digital citizen and being cybersmart. The clusters had similar instructional foci on and a set of norms and values for learner agency.

Conclusion

The overall conclusion is that more experience of the community of practice and the instructional foci, conditions similar to those identified in school-wide social and emotional interventions (e.g., Durlak et al., 2011), together with the affordances of the tools, enabled higher self-regulation. But this was specific to digital contexts. The digital initiative added a school-wide emphasis on self-regulation which socialised a set of norms, values and practices regarding digital citizenship, and a strong instructional focus which the EA students had experienced over several years. These likely created conditions similar to those identified in school-wide social and emotional interventions (e.g., Durlak et al., 2011).

The additional suggestion is that the school-wide intervention did not have the same impact on social skills in digital contexts as on self-regulation. This could result from the affordances for these skills being different from those for self-regulation, the design of the activities, the manner and degree of augmentation, or the socialisation of norms and values through the communities of practice. Further research is needed to untangle these possible explanations. This would require experimental tests of the affordances for social skills with and without types of teacher augmentation and studies of how different components and content of school wide community functioned to socialise norms, values and practices.

Limitations

There are several caveats. The primary caution is that as a natural experiment there was no control over pre-existing conditions and it is not possible to rule out alternative explanations, let alone establish causation. Further, the classroom observations were a very light sampling of instruction at one point in time and hence weak indicators of possible teacher (or peer) augmentation over time. Additionally, only overt foci through actions or discourse were observed, which meant that any augmentation that was indirect, say through instructional artefacts and templates, or students’ internalisation of earlier teacher guidance, would not be picked up. The limitations underline the need for detailed understanding of the instructional conditions for the target skills.

Finally, this study illustrates the usefulness and the challenges of using quasi-experimental designs to understand the implementation and scaling up of educational innovations, in this case those using digital tools. Further research is needed which controls for pre-existing differences, has better matched groups, describes the conditions of the digital context both at school and at home in detail, and directly tests the effects of those contexts.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the New Zealand Ministry of Business Innovation and Employment Health and Society - Targeted Research Grant (UOAX1412).

Data availability

The data used in this report can be requested from the corresponding author. Ethics obligations and participants’ consents preclude sharing personalised data with third parties without consent. Fully anonymised and aggregated files may be made available.

ORCID iD

Stuart McNaughton

Author Biographies

Stuart McNaughton PhD Professor, Faculty of Education and Social Work, The University of Auckland, New Zealand.

Naomi Alexandra Rosedale, PhD Researcher, literacy facilitator and learning designer with the Manaiakalani Education Trust, Auckland, New Zealand.

Zhu Tong PhD Honorary Research Fellow in the Woolf Fisher Research Centre Faculty of Education and Social Work, The University of Auckland.

Lin Sophie Teng PhD Professor, Department of Linguistics, Zhejiang University, Hangzhou, China.

Rebecca Jesson PhD Associate Professor, Faculty of Education and Social Work, The University of Auckland, New Zealand.

Jacinta Oldehaver PhD lecturer Faculty of Education and Social Work, The University of Auckland, New Zealand.

Rashina Hoda PhD Associate Professor, Faculty of Information Technology, Monash University, Melbourne.

Rachel Williamson-Deane PhD Researcher, literacy facilitator and learning designer, Auckland, New Zealand.

References

Aiken

(2016) The Cyber Effect. London, UK: John Murray Publishers.

Anderson

Nguyen-Jahiel

McNurlen

, et al. (2001) The snowball phenomenon: spread of ways of talking and ways of thinking across groups of children. Cognition and Instruction 19(1): 1–46.

Belanger

Akre

Berchtold

, et al. (2011) A U-shaped association between intensity of internet use and adolescent health. Pediatrics 127: e330–e335. DOI: 10.1542/peds.2010-1235.

Black

Pulford

Christie

, et al. (2010) Differences in New Zealand secondary students’ reported strengths and difficulties. New Zealand Journal of Psychology 39(1): 19–23.

Brown

(2016) Classroom community and discourse: How argumentation emerges during a Socratic circle. Dialogic Pedagogy: An International Online Journal 4. http://dpj.pitt.edu/ojs/index.php/dpj1/article/view/160.

Campbell

Stanley

(1963) Experimental and Quasi-Experimental Designs for Research. Chicago, IL: Rand McNally & Company.

Capaldi

Rothbart

(1992) Development and validation of an early adolescent temperament measure. The Journal of Early Adolescence 12(2): 153–173. DOI: 10.1177/0272431692012002002.

Charlton

Moulton

Sabey

, et al. (2020) A systematic review of the effects of school-wide intervention programs on student and teacher perceptions of school climate. Journal of Positive Behavior Interventions 23(3): 185–200. DOI: 10.1177/1098300720940168.

Chassiakos

Radesky

Christal

, et al. (2016) Children and adolescents and digital media. Pediatrics 138(5): e20162593. DOI: 10.1542/peds.2016-2593.

10.

Clark

Tanner-Smith

Killingsworth

(2016) Digital games, design, and learning: a systematic review and meta-analysis. Review of Educational Research 86(1): 79–122. DOI: 10.3102/00346554315582065.

11.

Courage

Bakhtiar

Fitzpatrick , et al. (2015) Growing up multitasking: the costs and benefits for cognitive development. Developmental Review 35: 5–41. DOI: 10.1016/j.dr.2014.12.002.

12.

Cumming

Bettini

Pham

, et al. (2020) School-classroom-and dyadic-level experiences: a literature review of their relationship with students’ executive functioning development. Review of Educational Research 90(1): 47–94. DOI: 10.3102/0034654319891400.

13.

Davis

(1980) A multidimensional approach to individual differences in empathy. JSAS Catalog of Selected Documents in Psychology 10: 1–85.

14.

Duckworth

Gendler

Gross

(2014) Self-control in school-age children. Educational Psychologist 49(3): 199–217.

15.

Duckworth

Steinberg

(2015) Unpacking self-control. Child Development Perspectives 9(1): 32–37. DOI: 10.1111/cdep.12107.

16.

Durlak

Weissberg

Dymnicki

, et al. (2011) The impact of enhancing students’ social and emotional learning: A meta analysis of school based universal interventions. Child Development 82(1): 405–432. DOI: 10.1111/j.1467-8624.2010.01564.x.

17.

Eisenberg

Spinrad

Knafo-Noam

(2015) Prosocial development. In: Lamb

Lerner

(eds), Handbook of child psychology and developmental science, Vol. 3: Social, emotional and personality development. New Jersey, NJ: Wiley Publishers, pp. 610–656.

18.

García

(2016) The need to address non-cognitive skills in the education policy agenda. In: Khine

Areepattamannil

(eds), Non-cognitive skills and factors in educational attainment: Contemporary approaches to research in learning innovations. Sense Publishers, pp. 31–61.

19.

Gardella

Fisher

Teurbe-Tolon

(2017) A systematic review and meta-analysis of cyber-victimization and educational outcomes for adolescents. Review of Educational Research, 87(2), 283–308.

20.

Goodman

(1997) The strengths and difficulties questionnaire: a research note. Journal of Child Psychology and Psychiatry 38(5): 581–586.

21.

Gopalan

Rosinger

Ahn

(2020) Use of quasi-experimental research designs in education research: growth, promise, and challenges. Review of Research in Education, 44(1), 218–243.

22.

Heisawn

Hmelo-Silver

(2016) Seven affordances of computer-supported collaborative learning: how to support collaborative learning? How can technologies help? Educational Psychologist 51(2): 247–265.

23.

Jabbar

Felecia

(2015) Gameplay engagement and learning in game-based learning: A systematic review. Review of Educational Research 85(4): 740–779. DOI: 10.3102/0034654315577210.

24.

Jacob

Parkinson

(2015) The potential for school-based interventions that target executive function to improve academic achievement: A review. Review of Educational Research 85: 512–552.

25.

Jesson

McNaughton

Rosedale

Zhu

Cockle

(2018) A mixed-methods study to identify efective practices in the teaching of writing in a digital learning environment in low income schools. Computers & Education 119: 14–30. https://doi.org/10.1016/j.compedu.2017.12.005.

26.

John

Naumann

Soto

(2008) Paradigm shift to the integrative big-five trait taxonomy: history, measurement, and conceptual Issues. In: John

Robins

Pervin

(eds), Handbook of Personality: Theory and Research. Guilford Press, pp. 114–158.

27.

Jolliffe

Farrington

(2006) Development and validation of the basic empathy scale. Journal of Adolescence 29(4): 589–611. DOI: 10.1016/j.adolescence.2005.08.010.

28.

Jones

Doolittle

(2017) Social and emotional learning: introducing the issue. The Future of Children 27(1): 3–11. https://www.wallacefoundation.org/knowledge-center/Documents/FOC-Spring-Vol27-No1-Compiled-Future-of-Children-spring-2017.pdf.

29.

Kankaraš

(2017) Personality Matters: Relevance and Assessment of Personality Characteristics. OECD Publishing.

30.

Karich

Burns

Maki

(2014) Updated meta-analysis of learner control within educational technology. Review of Educational Research 84(3): 392–410. DOI: 10.3102/00346543314526064.

31.

J-B

S-S

Willems

Y-E

, et al. (2021) The association between school discipline and self-control from preschoolers to high school students: a three-level meta-analysis. Review of Educational Research 91(1): 73–111. DOI: 10.3102/0034654320979160.

32.

Liau

Choo

, et al. (2015) Pathological video-gaming among youth: a prospective study examining dynamic protective factors. Addiction Research & Theory 23(4): 301–308. DOI: 10.3109/16066359.2014.987759.

33.

Lorena

Argentin

Gyu

, et al. (2017) Is it the way they use it? Teachers, ICT and student achievement. Economics of Education Review 56: 24–39.

34.

McNaughton

Rosedale

Jesson

Hoda

Teng

(2018) How digital environments in schools might be used to boost social skills: Developing a conditional augmentation hypothesis. Computers and Education 126: 311–323. DOI: 10.1016/j.compedu.2018.07.018.

35.

McNaughton

Zhu

Rosedale

Jesson

Oldehaver

Williamson

(2022) In school and out of school digital use and the development of children’s self-regulation and social skills. British Journal of Educational Psychology 92(1): 236–257. DOI: 10.1111/bjep.12447.

36.

National Research Council (2012) Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. National Academies Press.

37.

OECD (2015) Students, Computers and Learning: Making the Connection. OECD Publishing.

38.

OECD (2021) 21st-Century Readers: Developing Literacy Skills in a Digital World. Paris: OECD Publishing.

39.

Rapanta

Garcia-Mila

Gilabert

(2013) What Is meant by argumentative competence? An integrative review of methods of analysis and assessment in education. Review of Educational Research 83(4): 483–520.

40.

Reznitskaya

Anderson

Kuo

(2007) Teaching and learning argumentation. The Elementary School Journal 107(5): 449–472. DOI: 10.1086/518623.

41.

Sailer

Homner

(2020) The gamification of learning: a meta-analysis. Educational Psychology Review 32: 77–112. DOI: 10.1007/s10648-019-09498-w.

42.

Shulman

Smith

Silva

, et al. (2016) The dual systems model: review, reappraisal, and reaffirmation. Developmental Cognitive Neuroscience 17: 103–117. DOI: 10.1016/j.dcn.2015.12.010.

43.

Sung

Yang

Lee

(2017) The effects of mobile-computer-supported collaborative learning: meta-analysis and critical synthesis. Review of Educational Research 87(4): 768–805. DOI: 10.3102/0034654317704307.

44.

Yang

Chen

Lin

, et al. (2021) School-wide social emotional learning and cyberbullying victimization among middle and high school students: moderating role of school climate. School Psychology 36(2): 75–85. DOI: 10.1037/spq0000423.

45.

Zheng

Warschauer

Lin

, et al. (2016) Learning in one-to-one laptop environments: a meta-analysis and research synthesis. Review of Educational Research 86(4): 1052–1084.

46.

Zimmerman

(2008) Investigating self-regulation and motivation: historical background, methodological developments, and future prospects. American Educational Research Journal 45(1): 166–183. DOI: 10.3102/0002831207312909.