Abstract
The development of social-emotional skills is crucial in early childhood. Behavior problems in early childhood are risk factors for difficulties throughout childhood and adolescence and beyond. Considering the importance of developing social and emotional skills during early childhood, this study introduced the Papilio-3to6 program into everyday early childhood education and care (ECEC) in Germany. The program combines measures of developmentally appropriate practice and measures of social-emotional learning with strategies of developmentally appropriate prevention of behavioral and emotional problems. The underlying theory, the components of the program, evaluation results, dissemination, and implementation into ECEC center in Germany are described. A total of 627 children (MAGE = 56.77 months at pretest; 49% girls) from 50 ECEC center groups participated in an effectiveness, randomized controlled trial. At the pre- and posttest, teachers completed questionnaires related to children’s behaviors. Teachers completed questionnaires measuring their levels of job satisfaction, self-efficacy (control variables) and program implementation. A multivariate-multi-level-analysis revealed that children in the intervention groups, compared to the control groups, showed a significant decrease in their hyperactivity/inattention symptoms, as well as emotional, peer relationship problems, and conduct problems. The results also showed a significant increase in children’s prosocial behavior. Teachers’ job satisfaction and self-efficacy had no influence on the effectiveness of the program. Results supported treatment fidelity and usability.
Keywords
While there are many preventive intervention programs aimed at enhancing children’s social and emotional skills and preventing behavioral problems (Kalvin et al., 2015), there are still not many comparable early childhood prevention and intervention programs in Europe, Germany respectively (Schell et al., 2015; for a review, see von Klitzing et al., 2011). Most programs target elementary school children, adolescents, or the programs are adaptations of elementary/secondary school programs for early childhood education and care (ECEC). Also, knowledge and best practice examples about managing the steps from an evaluated “lab” program to the successful implementation into everyday care are scarce. This results in a great need for practical measures and programs, and demonstrations of the effectiveness of preventive intervention programs in typical childcare settings are needed. The present paper contributes to the field of early childhood development by introducing the German preventive intervention program Papilio-3to6 (Scheithauer & Peter, 2022), its theoretical background (e.g., how the program combines measures of developmentally appropriate practice and measures of social-emotional learning with strategies of developmentally appropriate prevention of behavioral and emotional problems), results of an effectiveness study, and the large-scale implementation into everyday care.
Social-Emotional Skills in Early Childhood
In early childhood (age 3 to 6) children face social and emotional challenges, including, entering the social field of ECEC centers and meeting the complex demands of teachers and peers, establishing new relationships with other children, making new friends (for a review, see Bukowski et al., 2018; Pahl & Barrett, 2007; Rubin et al., 1998), evaluating their self-worth, competence, and view of the world as pleasant or hostile (Institute of Medicine, 2000). Consequently, early childhood is a crucial time for the development of various social and emotional skills and is a unique opportunity to lay the foundation for healthy development. These skills provide a basis for a child’s motivation, self-control, and perseverance during an activity and for the child to experience, express, and understand emotions (e.g., Denham et al., 2011; Rubin & Rose-Krasnor, 1992; Saarni, 1999).
Adaptive social and emotional competencies develop when children acquire various behaviors, skills, and beliefs that help them to function effectively in social interactions (Halberstadt et al., 2001). Social competence is defined as “the ability to achieve personal goals in social interaction while simultaneously maintaining positive relationships with others over time and across situations” (Rubin & Rose-Krasnor, 1992, p. 285). Specific constructs in the early childhood social competence domain include: prosocial skills, i.e.“voluntary behavior intended to benefit another” (Eisenberg et al., 2006, p. 646) such as friendly, cooperative, and helpful behaviors); self-control skills, such as problem-solving skills, negotiation skills, anger management, regulating your own emotions, communicating effectively, and developing positive peer relationships (Raver & Zigler, 1997). Emotional competence, on the other hand, is defined as the ability to understand the emotions of self and others, emotional signals, and the consequences of own emotional expressiveness; regulate your own emotions; and react to others emotions (Denham et al., 2002; Saarni, 1999). As children continue to develop social and emotional competencies, they gain the skills and abilities needed to regulate motions, build relationships across settings, and solve problems (Halle & Darling-Churchill, 2016; Rubin & Rose-Krasnor, 1992; Saarni, 1999).
Social and emotional skills have a critical role not only in supporting children’s wellbeing and positive mental health, but also improving academic performance and successful adjustment at school (e.g., Ashdown & Bernard, 2012; Cristóvão et al., 2017; Denham et al., 2015; Denham et al., 2012; Denham & Brown, 2010; Durlak et al., 2011), and later on into adulthood (e.g., Jones et al., 2015; Moffitt et al., 2011). Children who are socially and emotionally well-adjusted are more successful at school, cope better with challenges and emotions, gain the confidence and competence needed to build peer and adult relationships, and have good problem-solving skills (Institute of Medicine, 2000; for a review, see Darling-Churchill & Lippman, 2016). These children are also able to make positive peer relationships by playing and communicating cooperatively with peers, identifying feelings of self and others, regulating own emotions, inhibiting reactive aggression, and solving social problems by negotiating with peers (Kalvin et al., 2015). A study by Jones et al. (2015) found significant associations between early childhood social-emotional development and outcomes in education, mental health, employment, criminal activity, and substance abuse 13 to 19 years later. Results suggests that children who are better at understanding emotions, resolving conflicts with peers, and helping and cooperating with others, are more likely to become well-adjusted adults who have jobs and contribute positively to society. Higher self-control (i.e., the ability to regulate behavior and emotions) in early childhood was also found to be a predictor of better physical health, lower substance dependence, fewer finances struggles, and fewer criminal offenses in adulthood (Moffitt et al., 2011). Not being able to master the required skills might put children at risk for developing serious forms of behavioral problems (Izard, 2002). Early childhood behavioral problems such as aggressive behavior, hyperactivity, social withdrawal, and poor peer relationships are known risk factors for difficulties during adolescence and adulthood, including depression, antisocial behavior, school dropouts, substance use, and delinquency (for a review, see Herrenkohl et al., 2010). Thus, the acquisition of social and emotional skills associated with social-emotional competence during early childhood (age 3 to 6) affects development through middle childhood, adolescence, and beyond and social and emotional competence are recognized as important protective factors in early childhood, buffering children from stressors and preventing the development of various behavioral and emotional difficulties in later life (Pahl & Barrett, 2007).
Fostering Social-Emotional Competence in Early Childhood: The Role of ECEC
Since significant emotional and behavioral problem in early childhood may impede the acquisition of age-appropriate skills and adversely affect developmental trajectories - as described above -, the typical social-emotional development has been a growing concern for day care providers, child mental health services, family, and child welfare, such as kindergartens in Germany. In Germany, children aged about three to six, may attend the so called “Kindergarten”, “Kindertagesstätte” or in short “Kita” respectively, a type of preschool or daycare centre for children supervised by pedagogical professionals. The training, educational background respectively, of pedagogical professionals –in Germany called “Erzieher” (males) and “Erzieherinnen” (females), and in the following called “teachers” - can vary greatly (from unskilled to university of applied sciences degree). Since 2013, there is a legal entitlement to a subsidized or free place in a Kita in Germany. Thus, nearly all 3-6-year-olds attend child kindergarten institutions and can be reached with universal preventive interventions. Kindergarten teachers play very important role in the context of prevention and fostering in early childhood learning and development (for a review, see Schonert-Reichl, 2017): For example, kindergarten centers will be most successful in their educational mission when they integrate efforts to foster children’s cognitive, social, and emotional learning (SEL; Denham et al., 2012; Denham et al., 2015; Elias et al., 1997). Teachers create a classroom climate that promotes children’s SEL and as Edwards and Raikes (2002) describe it: “The teacher-child relationship is an extension of the primary parent-child relationship, and teachers invest in building supportive relationships with families around their common interest, the child” (p. 12). Apart from teacher support, positive peer interactions are highly important for healthy development from early childhood on, and preventive intervention programs should include elements that train skills to prevent peer rejection and promote peer acceptance (for a review, see Bukowski et al., 2018). Relevant skills are: cooperative play skills, language and communication skills, emotional understanding and regulation, aggression control and social problem-solving skills (Kalvin et al., 2015, p. 2).
Theoretical Background, Contents, and Training Approach of the Papilio-3to6 Program
Papilio-3to6 is a universal preventive intervention program that focuses on social-emotional skills, and cooperative peer relations in early childhood (age 3 to 6) to prevent the onset of emotional and behavioral problems. Research in the field of emotional development suggests a framework for conceptualizing the emotion related aspects of preventive interventions whose aim it is to enhance children’s socio-emotional competence to prevent behavior problems (Izard, 2002). According to a summary by Kalvin and colleagues (2015) children’s social-emotional development might profit from direct instruction, modelling behaviour, etc. to illustrate skill concepts. Additionally, it might profit from providing settings in the kindergarten classroom where acquired skills can be practiced with support and guidance, from specific feedback that increases self-monitoring and social awareness, and from providing situations where skills can be transferred to other settings (e.g., from a pretend play situation to a situation during joint crafting). As a developmentally appropriate program, the Papilio-3to6 program is founded upon empirically derived models of normal and abnormal developmental pathways (cf. Scheithauer et al., 2009). The program development followed principals of developmentally appropriate prevention of behavioral and emotional problems and developmentally appropriate practice in kindergarten centers (e.g., National Association for the Education of Young Children [NAEYC], 2009; Tremblay & Craig, 1995).
The program works with kindergarten teachers as program implementers, because they are “central change agents” with a consistent presence in the classroom enabling them to promote children’s social-emotional development. As a universal intervention program, Papilio-3to6 is directed toward the entire classroom to promote social-emotional learning and positive peer relations while indicated program elements focus on remediating skill deficits and reducing the existing problems of children e.g., with behavioral disturbances. A universal preventive approach - targeting all children within a respective kindergarten center –was chosen to avoid stigmatization and possible iatrogenic effect of high-risk interventions (i.e., an increase of problem behaviors; Dishion et al., 1999). Thus, all children are addressed equally, without stigmatizing individual children, e.g., with behavioral problems. However, during the training, teachers are instructed to better recognize and pay attention to very lively or very quiet children and to support them more intensively in the program units, or to take this into account when putting together groups in the program units, for example.
Program Contents
The development of the program components followed a “user-based” approach tailored to the specific need for an adequate preventive intervention in German kindergarten centers, because the education of teachers in kindergarten centers in Germany is not standardized and sometimes lacks specialized training and education. The program components were designed to fit an ethnically and socio-economically diverse population of children.
Teachers are trained by Papilio-3to6 coaches in the implementation of the following manualized and standardized program components: Child-focused intervention component: teachers regularly implement three educational procedures that - oriented to important developmental tasks of the preschool age (Elias et al., 1997) - are mainly meant to foster children’s social-emotional skills, prosocial behavior, and peer integration: “Toys on Holiday” (ToH). Children’s play takes on an important function in the development of social-emotional competencies (e.g., Creasey et al., 1998). Once a week, children play interactive games without toys to enhance their involvement in the peer group, cooperative play, peer interactions, and communication skills (Bukowski et al., 2018). “Puppet in the Box Story” (PIBS). Social and emotional skills can be seen as protective factors against behavioral and emotional problems, and emotional skills provide the basis for empathy, prosocial, and helping behavior (cf. Izard et al., 2002). PIBS includes skill presentation lessons (with modelling story, puppets, pictures) and consists of an interactive story dealing with four characters (puppets) representing the basic emotions of sadness, fear, anger, and happiness (see Fig. 1). PIBS aims at improving emotional skills (Saarni, 1999). It fosters self- and others perception of emotions (including perception of physical reactions as well as facial expressions and gestures) of oneself and others, emotion regulation, and problem-solving skills (e.g., children leave pictures in a box to give the puppets advice on how to regulate emotions). Related materials support the proper identification of emotions (e.g., pictures of the puppets’ faces, a CD with recordings of their voices). Papilio-3to6 cooperates with the puppet theatre “Augsburger Puppenkiste” and several materials have been developed, including for example a puppet play with wooden puppets, songs with emotion content, and picture books, a radio play, and a DVD (puppet play; see www.papilio.de, for further information). “Mine-Yours-Our-Game” (MYOG). To manage children’s social behavior in group situations, a modified version of the “Good Behavior Game”, a contingency management intervention, is implemented (cf. Dadakhodjaeva et al., 2020), rewarding children’s positive social behavior that is in line with agreed group rules. MYOG includes guided practice activities (e.g., in role plays and games, compliance with social rules is practiced) to teach social-emotional skills and reward positive social behavior in the classroom. It is rooted in Social Learning Theory (MacBlain, 2021) and considers the development of social norms (House, 2018). Teacher-focused intervention component: the main focus is to support daily interactions between teachers and children and construct a supportive educational atmosphere in the kindergarten centers. Teachers are trained in their group management and positive interaction skills (e.g., by emphasizing positive reinforcement, appropriate use of negative consequences), and kindergarten teachers are provided with knowledge about the social-emotional development of children aged 3 to 6 years. Child-focused interventions represent program components that are targeted to all children at specific times or on specific days and are implemented by the teachers. The teacher-focused interventions, on the other hand, are program components that are intended to establish themselves permanently in the sense of professionalizing the daily work of educators and the quality of interactions with children and parents. Parent-focused intervention component: informs parents about the background and implementation of the program, encourages them to transfer program elements into family routines and is rooted in Social Learning Theory, positive, and inductive parenting. The cooperation with parents is seen in the program in the sense of an “educational partnership” (e.g., Textor, 2020).
Protagonist “Zornibold”, representing the basic emotion “anger” of the interactive story ‘Puppet in the Box Story’ (the puppets were manufactured by the Augsburger Puppenkiste). *Image and copy rights: Papilio gGmbH
The Papilio-3to6 program differs from many preventive intervention programs in that it does not implement its components in a consecutive way (e.g., each individual measure once, in a defined sequence) but simultaneously and continuously in kindergarten centers. For example, the teacher-focused intervention components address daily interactions (e.g., giving positive feedback to children’s positive behavior) and teachers are encouraged to use them permanently or as often as possible, also when implementing the child-focused interventions. In addition, it uses easy-to-apply elements that can be integrated into everyday kindergarten practice. Thus, it resembles an integrated model of prevention as described by Domitrovich et al. (2010) by combining different strategies of intervention to build one coherent program that is easy to implement in every kindergarten center. Despite its standardization, the program implementation allows some level of flexibility while considering the key elements of the program. More details of the program development can be found in Scheithauer and Peter (2022).
Training Approach
Becoming an accredited Papilio-3to6 coach involves: (a) Attendance at a skills training approach (6 days), including didactic input, live demonstration of consultation skills, small-group exercises to practice skills, etc., and a final two-day workshop, (b) coaching, and (c) supervision during training of kindergarten teachers, as practitioners who access supervision and workplace support are more likely to implement the program as expected. Papilio-3to6 coaches have regular online and face-to-face exchange, including “quality days” to maintain training quality and implementation fidelity.
Becoming an accredited Papilio-3to6 teacher involves (a) attendance at basic and in-depth skills training (7 + 2 days) involving didactic input, small-group exercises to practice skills, and implementation of program components and quality assurance, and (2) supervision meetings during first implementation of the program. Finally, kindergarten center teachers are accredited following a concluding two-day workshop. After accreditation, teachers have access to online teacher networks, technical assistance, teacher meetings, and update sessions to maintain implementation fidelity. The content of the in-service training consists of information transfer, competence building, active practice, feedback and coaching, and self-awareness elements. In addition, there is professional support during the implementation of the program. The training follows a situation- and practice-oriented concept. The comprehensive training can be seen as an important contribution to the professional development of kindergarten teachers (cf. Schachter et al., 2019). Kindergarten teachers are supported in the training to establish an “educational partnership” with parents (Textor, 2020). Every training course is evaluated, and feedback is elicited on aspects such as course content.
Coaches are trained by pedagogical staff of Papilio gGmbH with many years of experience. Kindergarten teachers are trained by accredited coaches. The Papilio-3to6 coaches and teachers use standardized materials to ensure program integrity. Thus, the Papilio-3to6 coaches as well as teachers have to adhere to an ongoing quality-assurance process that includes mandatory updates of accreditation, and participation in a network with regular meetings.
Dissemination and Implementation
The program development followed the IOM model of the prevention research cycle as described by Wandersman and colleagues (2008) while combining step 3 (pilot studies) and 4 (large scale trial). Hereby difficulties in the transfer of an otherwise “lab”-approach to usual-care are avoided (cf. Rohrbach et al., 2006). Since 2003, Papilio-3to6 program was developed, evaluated, and disseminated by the beta Institut gGmbH, and since 2010 the social enterprise Papilio gGmbH is responsible for program dissemination and implementation in Germany, Belgium, and Finland in cooperation with local partners. Papilio gGmbH brings together regional partner organizations that are necessary for the implementation and its funding. Papilio gGmbH further trains the Papilio-3to6 coaches (usually employed by cooperating institutions, e.g., communities, prevention institutions, charities), and leads the quality management. The kindergarten teachers are usually employed by the cooperating kindergarten centers (in Germany these are usually regional charities-run, church-run, or state-run facilities).
According to a yearly report based on the Social Reporting Standard, currently the program is implemented in 14 out of 16 German Federal States by 234 Papilio-3to6 coaches and 7,745 kindergarten teachers. To achieve this nationwide implementation (Type II translation, that is the institutionalization of a program), different strategies (see Rohrbach et al., 2006) were realized: (a) attractive and user-friendly program materials were designed, (b) a diffusion system was established (e.g. since 2006 275 “tour days” all over Germany have been organized to disseminate the program by introducing the puppet play “Paula and the Box Puppets”), (c) adopting sites are supported to build their organizational capacity for program implementation (due to the federal system in Germany [with 16 federal states], different state ministries, charities, health insurance companies have to be involved and state specific “education plans” for kindergarten centers have to be taken into account), (d) training and assistance to program implementers have been provided (e.g., kindergarten teachers continuously receive feedback and assistance provided from the Papilio-3to6 coaches including on-site visits), and (e) a system for collecting and reporting data on program delivery has been established (e.g., feedback questionnaires immediately after finishing training sessions; documentation of implementation by teachers). There are regular supervision meetings of teachers and Papilio-3to6 coaches focusing on program implementation to guide the regular teachers in administering the program components correctly and fitting the needs of the specific group. In addition, participating teachers return weekly records about implementation progress and difficulties or deviations from the manualized measures to the Papilio-3to6 coaches enabling them to offer support in case of any implementation problems. To ensure an appropriate delivery of the program a standardized training and dissemination system was developed and tested. Experienced and well-educated kindergarten teachers or prevention professionals are eligible to become Papilio-3to6 coaches (cf. Wandersman et al., 2008).
The Present Paper
First aim of our study is to investigate the effectiveness of the Papilio-3to6 program in comparison to control groups, taking into account the multi-level structure of the data. We hypothesize that children in the intervention groups will show fewer behavioral and emotional problems and more prosocial behavior after the program is implemented. Data from a randomized controlled effectiveness trial will be presented, regarding the impact of the program on children’s behavior as rated by the teachers. Preventive intervention programs for kindergarten facilities “can only be effective if teachers are willing and able to implement them” (Baker et al., 2010, p. 210). Thus, certain variables such as job satisfaction or self-efficacy are discussed in the literature as important “influencing factors” (e.g., Won & Chang, 2020). Therefore, the second aim of our study is to consider job satisfaction and teacher self-efficacy as control variables in the analyses. We assume that these factors will have an impact on the effectiveness of the program. Our third aim is to provide data by kindergarten teachers about their impression and motivation regarding the implementation of the Papilio-3to6 as well as the frequency of realized program elements to inform about treatment integrity and fidelity.
Method
Participants
The present study was carried out in a medium-sized German city that is comparatively diverse (proportion of non-Germans=160% compared to the country average; Statistisches Bundesamt, 2015) and has a moderately high socio-economic background (GDP per inhabitant = 134% compared to the country average; Statistisches Bundesamt, 2015). In a stratified sample selection that controlled for relevant socio-demographics, a total of 25 kindergarten centers were drawn, who participated with two groups each. Of 1,231 eligible children those children that would not have been available for further assessments (due to transfer to regular elementary school) were excluded for the following analyses resulting in 796 children (age 3 to 6) that received regular childcare from 109 kindergarten teachers (per group: M = 2.18, SD = 0.60). Data are based on reports ranging from 92 to 101 teachers. Participation rate (after active informed consent by parents) was high (86%; while all parents agreed to their children’s participation in the program), which left a total of 687 children. Thereof, completed data was available from 667 children at the pretest and 646 children at the posttest, resulting in a total analytic sample of 627 children (M AGE = 56.77 months at pretest; 49% girls) with valid longitudinal data. Kindergarten centers in the selected region offered educational and care services usually for children aged 3 to 6 years. Missing-data analyses demonstrated that children with incomplete data (9%) differ only marginally from the rest regarding gender (45% vs. 49% girls; d = 0.13) and treatment participation (55% vs. 48% in intervention group; d = 0.16) but moderately regarding age (54.25 vs. 56.77 months; d = 0.32). This medium effect size difference in age, however, is based on a marginal magnitude (Δ=2.52 months), so that the overall attrition can largely be considered as unsystematic.
Design
The program was evaluated using a 1-year-randomized controlled trial design with an intervention (IG) and waiting-control group (CG) that provided data at the pretest (before teacher training), intermediate test (during program implementation 7-8 months later), and posttest (after full implementation one year after teacher training). The present study focuses on the pre-posttest design to report long-term effects after the program has been conducted. Program implementation in the CG started one year after implementation in the IG. After stratified selection based on relevant socio-demographics, recruitment and random assignment to IG and CG, a total of 12 kindergarten centers with 24 kindergarten groups were part of the IG (n = 301; 48%) and 13 kindergarten centers with 26 kindergarten groups were part of the waiting-CG (n = 326; 52%) that received the program after the evaluation period. Both groups are comparable, as supported by small effect sizes regarding relevant socio-demographics such as gender (49% girls in CG vs. 49% girls in IG; d = 0.00) and age (57.10 months in CG vs. 56.43 months in IG; d = 0.09). A detailed comparison between IG and CG can be found in Table 1 based on parent reports on socio-demographics at pretest.
Parent’s Report of Socio-Economic Background at Pretest
Note. *p<.10, **p<.01; IG = intervention group; CG = control group.
Instruments
The present study focused on the reports from kindergarten teachers (for results related to other information sources see Crayen et al., 2011; Scheithauer et al., 2016), which enable an unbiased external assessment of children’s behavior within their daily social setting. Winsler and Wallace (2002) for example found that teacher reports on kindergarten children’s behavior and skills, in contrast to parent’s report, were significantly associated with observations of children’s behavior in the classroom.
Teacher Reports on Children’s Behavior
At the pre- and posttest, kindergarten teachers completed the teacher form of the German version (Klasen et al., 2003) of the Strength and Difficulties Questionnaire (SDQ). This instrument includes 25 items that form five subscales with five items each: emotional symptoms (e.g. “many fears and easily scared”), conduct problems (e.g. “often fights with other children or bullies them”), hyperactivity/inattention (e.g. “restless, overactive, and cannot stay still for long”), peer relationship problems (e.g. “rather solitary and tends to play alone”), and prosocial behavior (e.g. “shares readily with other children”). Using a three-point scale (0 = not true, 1 = somewhat true, 2 = certainly true), kindergarten teachers evaluated every child of their kindergarten group on the SDQ separately. Subscales were created by calculating the sum score of relevant items. All subscales demonstrated satisfactory reliabilities at both the pre- and the posttest with an average reliability of Cronbach’s α=0.79 (range = 0.72 - 0.87). In order to evaluate the program effectiveness, we calculated a change score for each subscale by subtracting the pretest score from the posttest score, so that the resulting score represents the change during the evaluation period.
Teacher Self-Report
Teachers reported their motivation to implement Papilio-3to6, their work satisfaction, and prior education experiences. At the pretest, we assessed job satisfaction utilizing a validated scale that was specifically created for this workplace setting (Alsaker & Valkanover, 2000). On eight items (e.g. “Are you satisfied with the overall work situation?”), teachers rated their satisfaction with the workplace using a four-point scale (1 = no to 4 = yes). The mean score across all items and, in turn, across all teachers in a kindergarten group represents the respective scale with a reliability of Cronbach’s α=0.83. At the posttest, we additionally assessed the self-efficacy of kindergarten teachers utilizing a validated scale (Schmitz & Schwarzer, 2000) that we adapted to the kindergarten setting. On ten items (e.g. “When I try really hard, I am able to reach even the most difficult children.”), teachers rated their self-efficacy beliefs using a four-point scale (1 = not at all true to 4 exactly true). The mean score across all items and, in turn, across all teachers in a kindergarten group represents the respective scale with a reliability of Cronbach’s α=0.67.
At pre- (IG and CG) and post-test (CG only, to control for any training or continuing education experiences they may have had between the measurement occasions), teachers were asked to answer a questionnaire (self-development) about advanced training experiences regarding topics relevant to the content of the program, e.g., behavioral disorders and improvement of their pedagogical work, and their interest in the program. A comparison between daycare center teachers of the IG and CG revealed no (significant) differences e.g. in the content-related or professional reasons (example items: “because it is important to promote the social competence of children” or “because it is an opportunity to gain further professional qualifications”).
Teacher Reports on Program Implementation
IG teachers rated their satisfaction with different program components and aspects of the overall program. All IG teachers were asked to document the implementation of the program. IG teachers rated their respective satisfaction with the different program components (ToH with 8 items, PIBS with 11 items, MYOG with 7 items, kindergarten teacher component with 4 items) and different aspects of the program in general, based on 10 items at posttest using a five-point Likert-scale (1 = very bad to 5 = excellent). The questionnaire included ratings of age appropriateness of the components, easiness of day-to-day implementation, and comprehensibility of the tasks. To assure adherence to the treatment protocol various actions have been pursued. Assessment of program fidelity should include dose (amount of the program delivered), quality of program delivery, and participant reactions or acceptance (Dane & Schneider, 1998). Therefore, all teachers form the IG had to document the number of interventions accomplished through the course of the implementation phase as well as the number of positive and negative experiences with the single components. One documentation protocol for each kindergarten group was completed since different teachers from one group cooperated in documenting the implementation.
Analysis
We applied multilevel modeling (HMLM2) (Raudenbush & Bryk, 2002) for the following reasons: First, it allows considering the existing nested data structure (children nested in kindergarten groups); second, it allows testing the intervention effect on the contextual level, that is, where the program was conducted (i.e., kindergarten group); and third, it allows evaluating the program on multiple outcomes simultaneously. A multivariate multilevel model was performed with the change score in SDQ subscales on level 1 (intra-individual outcome level), children on level 2 (individual level), and kindergarten groups on level 3 (contextual level). This model predicted children’s change in SDQ subscales by testing the intervention effect on level 3, while controlling for socio-demographics and teacher variables on level 2 and 3, respectively. Furthermore, interaction effects between teacher variables and treatment participation were modeled in order to evaluate the extent to which the efficacy of the program depends on characteristics of the implementing teacher. All continuous predictors were z-standardized, separately on each level, to facilitate the interpretation of the regression coefficients. Analyses were carried out with full-information maximum likelihood (FIML) estimation. Estimation accounts for missing values at random (MAR) and includes all the available data.
Results
Analysis Strategy
Descriptive of study variables are presented in Tables 2 and 3. The upper part of Table 2 illustrates the means and standard deviations of all SDQ subscales at the pretest and posttest. Within the possible range of 0 to 10, children score, on average, low on all scales. Furthermore, children take in general (i.e., across both the IG and CG) the expected adaptive development and are, on average, perceived as less problematic and more prosocial by their kindergarten teacher one year later at posttest. Despite this positive trend, however, a considerably large number of children, especially in the CG, take a negative development from pre- to posttest as illustrated in the lower part of Table 2. Table 3 demonstrates that teachers report to have a high job satisfaction and self-efficacy independently of teacher’s treatment participation. At the same time, the illustrated histograms in the lower part of Table 3 indicate that teacher vary to some extent regarding these characteristics, which needs to be controlled for when evaluating the implemented program. Results about the motivation to implement the program and prior training experiences e.g. on how to prevent behavior problems, to handle conduct problems, and to improve educational skills are described in section “Rating of Program Elements and Treatment Fidelity”.
Children’s Socio-Emotional Competences and Behavioral Problems
Note. Effects sizes contrast the frequency of cases with a negative development in CG vs. IG.
Teacher’s Job Satisfaction and Self-Efficacy
Note. Effect sizes contrast job satisfaction and self-efficacy of teachers in CG vs. IG.
Program Effectiveness
The multivariate multilevel model analyzed 3,135 intra-individual change scores on level 1, nested in 627 children on level 2, nested in 50 kindergarten groups on level 3. Prior to the main model, we estimated the unconditional model without any predictors in order to determine the variance proportion in the outcome variable that is attributable to the individual versus contextual level. The unconditional model revealed intra-class coefficients (ICC) of .04,.05,.10,.02,.08, indicating that 2% to 10% of the variance in the five outcome variables is on the contextual level. This amount of variance differs, except for one ICC, significantly from zero (χ2(49)=77, p < 0.01; χ2(49)=84, p < 0.01; χ2(49)=114, p < 0.001; χ2(49)=59, p = 0.16; and χ2(49)=104, p < 0.001, respectively), which underscores the usefulness of the applied multilevel approach.
Table 4 summarizes the main model, which included socio-demographic controls on level 2 (sex and age), teacher controls on level 3 (job satisfaction and self-efficacy), treatment participation on level 3 (group), and the interaction effects between teacher and treatment participation on level 3 (group x job satisfaction and group x self-efficacy). Results support the efficacy of the kindergarten prevention program Papilio-3to6, as indicated by the “group” predictor that represents the change of the SDQ subscales in the IG in contrast to the CG. The effect of all five subscales points into the expected direction, which becomes significant for hyperactivity/inattention as well as prosocial behavior and in tendency for emotional symptoms. Moreover, the main model also reveals, consistently across all outcome variables, that all treatment effects are not affected by teacher characteristics, as indicated by the non-significant effects of teacher controls and their interaction with the treatment predictor. It is worthy to note that the main model can be replicated when controlling for additional socio-demographics that were found to differ between IG and CG.
Multivariate Multilevel Prediction of Children’s Change in Socio-Emotional Competences and Behavioral Problems
Note. 10 = girls and 1 = boys; 20 = control group and 1 = intervention group; one-tailed significance, *p<.05, †p<0.1.
Prior Training Experiences and Motivation to Implement the Program
The majority of teachers reported no prior experiences in the field of violence (77.3 %) and addiction (85.4 %) prevention. However, almost half of the teachers attended some kind of training dealing with conduct problems (45.3 %) and 60.9% attended training improving educational skills prior to the Papilio-3to6 training. Teachers from the CG had more prior training experiences in the field of conduct problems than teachers from the IG (exact Fisher-test=p<0.05). Fourteen CG teachers reported to have attended other advanced courses improving their educational skills during project implementation. The most important reasons for implementing the program reported by the teachers were: promoting children’s social competence (M = 2.90, SD = 0.30) and preventing severe behavioral problems (M = 2.87, SD = 0.34). Other reasons were: opportunities to improve working skills (M = 2.77, SD = 0.45), status of becoming a “Papilio-3to6-teacher” (M = 2.68, SD = 0.51), desire to integrate the Papilio-3to6 components into daily routines (M = 2.63, SD = 0.51), and scientific foundation of the program (M = 2.58, SD = 0.54). Teacher’s reasons for implementation did not differ between IG and CG (t-tests for independent samples) with two exceptions: Teachers from CG considered cost-free participation, t (83)=-2.35; p < 0.05 and cost absorption by program providers for play material, t (90)=-2.77; p < 0.01 more important than teachers from IG.
Rating of Program Elements, Treatment Fidelity, and Sustainability
The teacher report items administered to evaluate the program structure and its usability were summed up and averaged to an overall score due to high internal consistency (α=0.93). Accordingly, the program “as a whole” was rated as “rather good” (M = 4.23, SD = 0.70). In detail, kindergarten teachers rated the program to be useful for improving children’s social skills, to prevent behavior problems, they acknowledged the support provided by project staff, the promotion of interpersonal communication among teachers, and they judged the program and the associated training as a way to improve their own working skills (M from 4.15 to 4.49, SD from 0.66 to 0.95). Teacher’s reports regarding confidence with the program components can be summarized as follows (with items summed up and averaged to an overall score for each respective child measure due to high internal consistency [αToH=0.79; α PIBS = .81; αMYOG=.95]): ToH: M = 3.90 (SD = 0.65) indicating an overall rating as “rather good”; PIBS: M = 4.41 (SD = 0.44) indicating an overall rating as “rather good” to “very good”; MYOG: M = 3.65 (SD = 1.04) indicating an overall rating as “average” to “rather good”. The PIBS was rated more positive than the other components, t (47)=-6.01; p < 0.001, and some teachers rated the program components as too difficult for younger children or associated preparatory work as too time consuming. Teachers rated their confidence with the kindergarten teacher component of the program (with items summed up and averaged to an overall score due to high internal consistency [α=0.95]) as “rather good” to “very good” (M = 4.55, SD = 0.50). Teachers also rated this element as helpful to promote a positive group atmosphere (M = 4.58, SD = 0.50), to build a positive relationship with the children (M = 4.54, SD = 0.54), and to foster self-worth regarding own educational skills (M = 4.43, SD = 0.65).
Between pretest and intermediate test all child centered components were conducted with considerable variance based on documentation protocols from teachers. The PIBS was delivered on average eleven times during a six-week period (M = 11.05, SD = 3.65, range = 5 –18), the ToH component was delivered 15 times on average during a four-month period (M = 11.05, SD = 3.65, range = 11 –19), and the MYOG was delivered 17 times on average during a three-month period (M = 17.46, SD = 6.12, range = 6 –27). Following the intermediate test, the teachers continued to carry out the different components for six months. During this time the PIBS was executed on average 16 times during a six-week period (M = 16.08, SD = 6.56, range = 4 –30), the ToH component was delivered 19 times on average (M = 19.00, SD = 3.95, range = 11 –25), and the MYOG was delivered 23 times on average (M = 22.79, SD = 8.88, range = 8 –37). In general, number of positive experiences with all components was higher than number of negative experiences between pretest and posttest, t(21 –23)=4.55 –8.85, all p < .001, d.=0.63 –2.18, and between intermediate and posttest, t(23)=7.02 –8.14, all p < .001, d.=1.09 –1.50. Effect sizes were calculated based on a formula for dependent t-tests from Dunlap et al. (1996, p. 171). The component with the highest ratio between positive and negative experiences was the PIBS.
Discussion
The present paper introduced the Papilio-3to6 program, described its background, content, implementation, dissemination, and selected evaluation results. The main findings refer to the effectiveness of the program for preventing behavioral problems and fostering social skills in kindergarten aged children as well as its applicability for everyday childcare. Within a multilevel framework, the present evaluation study demonstrated the effectiveness of the program in comparison to the control group regarding multiple outcome variables while considering the nested data structure. It is worthy to note that this test of effectiveness was analyzed on the contextual level (i.e., the kindergarten), which is more conservative due to the limited statistical power when contrasting 24 versus 26 analytic units. At the same time, however, this analytic strategy represents the most accurate evaluation, because it allows matching the unit of intervention with the unit of evaluation. The lack of statistical power might account for the missing significance of some outcomes that merely point into the expected direction. As illustrated above, teachers from the CG had more prior training experiences in the field of conduct problems than teachers from the IG. This may have an influence on educational skills of teachers from the CG and therefore may have diminished the effectiveness of the program. It is also likely that the two non-significant outcomes (i.e., conduct problems and peer relationship problems) will show a delayed effect. That is, while the program starts to change children’s emotional symptoms, hyperactivity/inattention, and prosocial behavior immediately due to specific components that focus on strengthening self-regulation and emotional skills, conduct and peer relationship problems might need more time and decrease only after children became more emotionally competent, more prosocial, and less hyperactive. Domitrovich et al. (2007) reported similar results in that the “PATHS” Curriculum showed an immediate effect for adaptive and positive behavior but not regarding the reduction of externalizing behavior such as aggression. Based on Cohen’s benchmark (1988), the effects of the program, contrasting the frequency of cases with a negative development in CG vs. IG (see Table 2), reached a small magnitude, with nearly a medium sized effect regarding hyperactivity/inattention. According to Lipsey et al. (2012), even small effects of a universal intervention program in the education field could be considered large. The results on effectiveness of Papilio-3to6 are comparable to results of other programs in Germany, such as EFFEKT (Lösel et al., 2006), or FAUSTLOS (Schick & Cierpka, 2003).
All models were controlled for relevant socio-demographics and teacher variables. In particular, the non-significant effects of teacher controls and their interaction with the treatment participation indicate that the Papilio-3to6 program is effective independently of job satisfaction or self-efficacy beliefs of implementing kindergarten teachers. The inclusion of additional moderators, such as the implementation of the program that was found to vary to some extent among kindergarten teachers, requires more contextual units in order to conduct a meaningful interaction test. It is plausible to assume, however, that program implementation moderates and, realized in an optimal way, further increases the effectiveness of the program.
Implications for Systems-Wide Implementation
Successful programs need to translate evidential effectiveness into the real world. Following the IOM model of implementation (Wandersman et al., 2008) as well as pioneering programs (e.g., Incredible Years; Webster-Stratton & Herman, 2010), the Papilio-3to6 program applied established dissemination strategies. More importantly, this structure is perceived as applicable. Despite some variations in treatment adherence, our data in general reveal support for the treatment integrity and fidelity of the Papilio-3to6 program under real-life conditions. Kindergarten teachers reported that the program components were perceived as positive (especially the PIBS that was designed in cooperation with a puppet theatre) and could be integrated into day-to-day kindergarten routines in a straightforward way. The results can help to improve the professional development of kindergarten teachers in Germany, by implementing effectiveness-evaluated and quality-assured programs nationwide in the future or by reaching teachers already in their undergraduate education.
Limitations and Future Directions
Despite the proof of effectiveness concerning multiple outcome variables, we were unable to disentangle the contribution of specific program components to the overall effect. In the future, we believe that it might be of value to unravel the effect of different program components within a continuous process evaluation. Another shortcoming regards the restriction to one context (i.e., Germany), which limits the external validity of the present evaluation. In order to further develop a universal program that can be generalized to different contexts, the cross-national adaptation and implementation of the program has been tested in Finnish kindergartens (Koivula et al., 2020), and scholars from Finland are currently running a trial using this culturally adapted version. The role and involvement of center administrators in the program should be the subject of further study. Further analysis will look at the impact of the program at the teacher level (job satisfaction and self-efficacy). First results demonstrate that kindergarten teachers from the IG expressed more job satisfaction after completing the Papilio-3to6 program compared to teachers of the CG, which suggests that the program supports kindergarten teachers in their daily work and may even help to prevent negative developments such as work overload or burnout. Moreover, in addition to the fidelity evaluation regarding program implementation, it is also important to consider the fidelity of the teacher trainings or coaches. A final limitation represents the exclusive use of teacher reports in the present evaluation study. Winsler and Wallace (2002) found that teachers are a valid information source on kindergarten children’s behavior and skills, Hinshaw et al. (1992), however, found in their study that parent ratings of preschool aged children’s internalizing behavior better predicted independently observed isolation and withdrawal, while teacher ratings of children’s externalizing behavior better predicted play group observations. Thus, it is possible that in our study kindergarten teachers underrated children’s internalizing behavior. Although teachers are, in contrast to parents, able to provide an unbiased evaluation of children’s behavior within their daily social setting, a multi-methodological approach including observations or play-oriented measures to assess data from children would provide valuable additional insights that allow analyzing the effectiveness of the program in more detail. However, it should be mentioned that the positive effect of the program also appeared for example in a study using peer nominations based on Social Cluster Mapping: Scheithauer et al. (2016) reported that IG children (from pre- to intermediate and posttest) received more positive peer nominations and nominations as being a friend. These positive changes correlated positively with prosocial behavior and negatively with the total difficulties score of the SDQ.
In summary, the results presented indicate the positive impact and feasibility of the program. However, future studies should investigate the implementation in other contexts (or countries) and the exact mechanisms of change.
Author Note
The development of Papilio-3to6 and the evaluation study was conducted by beta Institut gGmbH für sozialmedizinische Forschung und Entwicklung, Augsburg, in cooperation with science partners at Universität Bremen (Franz Petermann, Ute Koglin, Herbert Scheithauer) and Universität Augsburg (Leonie Herwartz-Emden), since 2004 with science partners at Freie Universität Berlin (see authors). Program development and the evaluation study was supported by funding from Bayerisches Staatsministerium für Umwelt, Gesundheit und Verbraucherschutz, betapharm Arzneimittel GmbH, BMW Group, and Rotary District 1840. Preparation/writing of the manuscript was supported by a grant from the Academy of Finland to Maryam Zarra-Nezhad (grant number 330960). We would like to thank the former staff members at beta Institut gGmbH, especially Horst Erhardt and Beatriz Barquero-Santander, and all of the kindergarten teachers, participating children, and their parents. Heidrun Mayer, co-developer of the program Papilio-3to6 and co-founder of the association Papilio e.V., passed away unexpectedly in August 2016 as a result of a bicycle accident. We mourn the loss of Heidrun Mayer and thank her for the many years of trusting and cordial cooperation.
Footnotes
Bio Sketches
Dr. Herbert Scheithauer is Professor for Developmental and Clinical Psychology at Freie Universität Berlin, Berlin, Germany, and Head of the Unit “Developmental Science and Applied Developmental Psychology”. His research interests are on bullying, cyberbullying, and the development and evaluation of preventive interventions.
Dr. Markus Hess is Professor for Applied Developmental and Social Psychology at the German University of Health and Sports, Berlin, Germany. His research interests focus on the development, evaluation and implementation of preventive interventions promoting social-emotional and intercultural development and preventing aggressive and (cyber)bullying behavior in different contexts (e.g. sports clubs, child care centers).
Dr. Maryam Zarra-Nezhad is Postdoctoral researcher in the field of child development at School of Applied Education Science and Teacher Education Philosophical Faculty, University of Eastern Finland, Finland. Her research interests are early childhood social and emotional well-being, peer relations, and parenting relationships.
Dr. Charlotte Peter, psychologist, heads the area of responsibility of “Science and Development” at the non-profit, limited company Papilio, Augsburg, Germany. Her professional interest centers mainly around the development and evaluation of evidence-based, developmentally appropriate preventive intervention programs.
Dr. Ralf Wölfer is Professor for Social Psychology at the University of the Federal Public Administration, Berlin, Germany. His research interests include the study and intervention of prejudice, discrimination, and intergroup conflict.