Abstract
Objective:
The study had two objectives (a) to adapt for Chinese children an intervention designed to strengthen the social information–processing (SIP) skills of children in the United States, and (b) to pilot test the adapted intervention in China.
Methods:
Adaptation of the Making Choices program involved reviewing Chinese literature on child development, translating and back-translating a treatment manual, modifying intervention content, and engaging experts to review program materials. Children (n = 91), ages 8–10, in five after-school child care centers in Tianjin, China, received the program. After propensity score adjustments, the skills of children who received the program were compared to the skills of children (n = 123) recruited from neighborhood primary schools.
Results:
The adapted program appears to have strengthened encoding skills. Patterns for other information-processing skills were promising but mixed.
Conclusion:
The program has the potential to strengthen the SIP skills of children in China.
Keywords
In the United States, China, and other countries, both academic achievement and social competence in childhood predict developmental outcomes (Chen et al., 2002; Chen, Huang, Chang, Wang, & Li, 2010; Chen, Huang, Wang, & Chang, 2012; Cunha & Heckman, 2010). Social competence includes interpersonal problem solving and emotional regulation, which are viewed widely as skills needed to develop positive peer relationships, work cooperatively in groups, and maintain enduring social relationships (Chen & Rubin, 1992; Conner & Fraser, 2011, 2011; Siu & Shek, 2010; Webster-Stratton, Reid, & Hammond, 2001; Xu & Zhang, 2008; Zalewski, Lengua, Wilson, Trancik, & Bazinet, 2011; Zhou, Main, & Wang, 2010). Further, research suggests that social–emotional skills may be as important as academic skills in contributing to developmental and life course outcomes, such as acceptance by peers, the ability to sustain employment, and the capacity to participate meaningfully in society (Heckman & Kautz, 2012). The purpose of this article is to describe the results from a controlled pilot test of a U.S.-based social–emotional skills training program that was adapted for children in the People’s Republic of China.
In China, preparing children to participate meaningfully in society is complicated by dramatic social and economic changes. As an element of sweeping reforms associated with the adoption of a market-oriented economy and the decline of a government-centered, planned economy, family life and work are being transformed by modernization and migration (Biao, 2007; Duan & Zhou, 2005; Nielsen, Nyland, Smyth, Zhang, & Zhu, 2005; Yi, 1986; Zhang, 2004). Manufacturing and technology jobs have blossomed in urban areas, while agricultural work in rural areas has become increasingly mechanized. Seeking work, millions of young people have migrated with their families from rural to urban areas, and this migration has created new demands in virtually all cities for child care, housing, medical services, and education. Urban migration has produced a dramatic increase in nuclear families that live without the traditional support of grandparents and other relatives who have remained in rural areas (Duan & Zhou, 2005; Liu, Li, & Ge, 2009; Yang & Chen, 2013).
The majority of urban families have only one child (Feng, 2006; Su, 1994). Children in these single-child urban households face a variety of intertwined challenges. Parents often work long hours (Cang, 2010; Zhang, Chen, & Lin, 2006), and children spend long days in highly competitive schools. Schools tend to focus on academic achievement, with little emphasis on social development (Lu, Tang, & Luo, 2007; Qiang, 1999; Sun, 2002). Because health care and other services are accessed through the historic Hukou system of household registration (户籍/口) in a home province, children whose parents migrated from a rural to an urban area often do not benefit from China’s comprehensive social welfare and health services system (Luo, Wang, & Gao, 2011). Compared to children whose Hukou registration is in an urban area, children in migrant families—even if born in an urban area—hold rural registration and face special challenges in accessing services. They tend to live in substandard housing and receive poor health care (Hou, Zou, & Li, 2009; Huang & Yang, 2000; Ren, 2008; Wong, Fu, Li, & Song, 2007).
Although the government, media, and advocates have expressed concern about the condition of migrant children and, more broadly, the developmental effects of growing up in single-child households without siblings (Liu, 2013; State Council of the People’s Republic of China, 2011; Zhou, 2006), programs related to children have been designed to focus on problems after they have developed. Indeed, China has excelled at mobilizing resources after crises but, in spite of historic emphases on planning, it has developed few prevention programs for children (Jia, Zheng, Liu, & Ye, 2004; Yu & Seligman, 2002). Many children receive services only after they develop social or health problems, and interventions to create safe and supportive environments and to strengthen both the academic and social skills of children have only recently received attention. The National Commission on Family Planning and Population Control (NCFPPC) was among the first governmental authorities to recognize the need for prevention.
In this context, the NCFPPC funded a project to strengthen the social–emotional skills of elementary school children in China. The Commission provided for the cultural adaptation of the Making Choices program, a U.S.-based program shown to improve the social–emotional skills of elementary school children (Fraser et al., 2005; Fraser, Lee, Kupper, & Day, 2011; Fraser, Nash, Galinsky, & Darwin 2000; Fraser, Thompson, Day, & Macy, 2014; Smokowski, Fraser, Day, Galinsky, & Bacallao, 2004; Terzian, Li, Fraser, Rose, & Day, 2014). The adapted program was called Let’s Be Friends (Making Choices Research Group, 2011). Let’s Be Friends retained the core content of Making Choices; however, activities and exercises were adapted for a Chinese context and new content was developed to have relevance for both migrant children and children from single-child households. To test Let’s Be Friends, an evaluation was undertaken in five “Happy Camps” at neighborhood child care centers in the metropolitan Tianjin area from September 2011 to January 2012. In conjunction with the NCFPPC and Nankai University, the pilot test of Let’s Be Friends was funded by China’s Ministry of Education. The findings from this pilot test are described in this article. We ask the following question: Compared to the social–emotional skills of children who received routine after-school care, do the social–emotional skills of children who received Let’s Be Friends differ significantly?
Child Development in China: Single-Child Households and Rural-to-Urban Migration
In the West, the social and academic skills of children are widely recognized as predicting developmental outcomes (e.g., Heckman & Kautz, 2012; Rutter, 1989; Rutter, Kim-Cohen, & Maughan, 2006). As indicated earlier, improving social competence and academic achievement is related to a variety of long-term positive life course outcomes, including peer acceptance; high school completion; and lower risk for unwanted pregnancy, drug use, and delinquency (Hawkins, Kosterman, Catalano, Hill, & Abbott, 2005; Odgers et al., 2008; Temcheff et al., 2008). In China, studies suggest that social and academic skills also predict developmental outcomes (Chen, Cen, Li, & He, 2005; Chen, Rubin, & Li, 1997; Deng & Roosa, 2007). However, China’s population control policy and the rapid growth of a marketplace economy have created conditions that attenuate the relationship between developmental outcomes and predictors such as social competence and academic achievement.
Single-Child Households
The Chinese government recently relaxed the one-child policy to allow couples from single-child households to have a second child (Central Commission of China’s Communist Party, 2013). From its inception in 1979 to its revision in 2013, the effects of the one-child policy were of great interest to Chinese and other researchers. Some studies suggested that the absence of siblings in single-child households contributed to negative child developmental outcomes, including what was called the “little emperor” syndrome in which children in one-child families were purported to display arrogance, disobedience, and selfishness (Wan, Fan, & Lin, 1984; Wang, 1983; Xiao & Zhang, 1982; Zhang, Xiao & Cao, 1987). Other studies, however, suggested that children from single-child households differed little from children in multi-child households (Bai, 1992; Cao, Zhang, & Sun, 1994; Feng, 1993, 2006; Su, 1994).
Although the little emperor syndrome has obtained scant support in recent research, studies indicate that children raised without siblings may have a somewhat more individual than collective orientation (Chen & Goldsmith, 1991; Fan, Wan, Lin, & Jin, 1994; Jiao, Ji, Jing, & Ching, 1986; McLoughlin, 2005; Ouyang, 2005, 2006; Su, 1994). In Confucian tradition, family relationships in China have been governed by relatively structured role differentiation, and the family setting has provided opportunities for children to learn to work collectively and cooperatively in different kinds of roles. Indeed, the roles afforded to family members have shaped cognitive and behavioral patterns that have contributed to social harmony, an organizing theme of both historic and modern China.
The nature of childhood in single-child versus multi-child households is argued to contribute to comparatively poorer social skills among children from one-child families (Jiao et al., 1986; Liu, 2007; McLoughlin, 2005). Although children from one-child families might be expected to have strengths in communicating with adults and to have benefitted educationally from undifferentiated parental attention (Bian & Zhen, 1997; Ouyang, 2001), research suggests that children without siblings may be less skillful in collaborating with peers, sharing with others, and working toward collective goals (Jiao et al., 1986; McLoughlin, 2005; Ouyang, 2005). Because children in single-child households may not blend as easily into collective efforts, policy makers became concerned that the single-child policy may have inadvertently reinforced values that militate against social harmony (An & Jia, 2009).
Rural-to-Urban Migration and Peer Relationships
At the same time, family structure and migration status interact in China, and the problems observed in Chinese children may be related as much to migration as to the one-child policy. In contrast to parents in urban areas, parents in rural areas have long been allowed to have more than one child. When families migrate to urban areas, some children in these multi-child households remain with relatives in agricultural areas. Children who are left behind in rural areas—sometimes called home-staying children—and children who migrate with their parents to urban areas face hardships that were not encountered by earlier generations and for which local welfare and education systems were not developed (Biao, 2007; Nielsen et al., 2005; Wong et al., 2007; Zhou, Sun, Liu, & Zhou, 2005).
These hardships affect child development. For example, a study in Wan’an, Jiangxi, a rural area in eastern China, reported that of the 295 children who were separated from their parents, some 23% reported feeling lonely and depressed, 5% reported feeling discriminated against, 51% reported dreaming of their parents, 12% reported insomnia, and 12% reported being bullied (Zhang, 2007). Often living with grandparents, the home-staying children of parents who have emigrated to urban areas are reported to be more emotionally labile and anxious, to engage in greater self-blame, and to have lower self-esteem (Duan & Zhou, 2005; Hao & Cui, 2007; LuoWang, & Gao, 2011; Wang & Shi, 2010; Ye, Wang, Zhuang, & Lu, 2006).
The children who have emigrated with their parents from rural areas also face hardships. As their parents seek employment, they move from flat to flat and from school to school. Because of frequent moves, it becomes difficult to develop stable peer relationships (Zeng, 2010), and immigrant children are often excluded from the activities of children whose families have legal Hukou residence (Lu, Lu, & Wu, 2004; Ren, 2008). Children from migrant families may come to lack confidence and, fearing rejection, may become reluctant to assume responsibility (Wang & Shi, 2010; Yin & Wang, 2009). Moreover, because they tend to lack stable peer relationships, they may lose the learning opportunities, social support, and social control that are afforded by membership in normative peer groups.
Public Policies to Strengthen the Social Development of Children
The challenges faced by modern Chinese children are related to the rapid growth of marketplace employment in urban areas; however, they are exacerbated by public policies such as the Hukou system. Driven in part by economic concerns, these policies ignored de facto migration. The Hukou system required children and families to receive services in their home provinces regardless of the place of work of parents. This forced many parents to leave their children behind in rural areas or to take children with them and seek sub rosa services outside of China’s comprehensive social welfare and health systems. At the same time, schools were charged largely with promoting academic achievement and, based on traditional methods, cultivated an authoritative and test-oriented style of teaching (Lu et al., 2007; Qiang, 1999). They have tended to ignore the neighborhood and classroom social dynamics that placed children from single-child households and the children of migrant families at higher risk. At the policy level, the design and testing of prevention programs for children received little attention (Jia et al., 2004; Yu & Seligman, 2002).
In 2007, however, the Central Commission of China’s Communist Party (CCCCP) and China’s State Council issued new guidelines. In a watershed statement, the Commission called for new programs to promote human development. The 9-item policy explicitly invited a national effort “to strengthen educational and training programs that will affect single-child children’s social behavior” (CCCCP and State Council, 2007). The project on which this report is based arose as a response to the Council’s new guidelines.
Project Description
To begin to develop programs to strengthen the social behavior of children, a research project was funded jointly by the NCFPPC and the University of North Carolina at Chapel Hill. Over a 5-year period, the research team adapted a U.S.-based program and pilot tested the revised program, which was focused on strengthening the social–emotional skills of Chinese children. Developed in the United States over the past 20 years as the Making Choices program (Fraser et al., 2005, 2000; Terzian et al., 2014), the adapted program was called Let’s Be Friends.
A Social Development Perspective
The project was rooted in research and theory related to social development. Social perspectives on child development often emphasize interaction between life experiences and cognitions, such as schema and scripts (Huesmann & Guerra, 1997; Schank & Abelson, 1977; Yang, 2001). However, theories related to schema and scripts are not adept at explaining variation in interpersonal behaviors in specific situations (Fontaine, 2008); and as researchers have studied social cognition, behavior, and emotional processes, they have developed a series of new theories and models (Crick & Dodge, 1994; Wang & Hu, 2003). Among these, the most influential—arguably—is social information–processing (SIP) theory. The Making Choices program is based on an SIP perspective.
SIP
Strongly supported by cross-cultural research, SIP theory conceptualizes behavior as emerging from the interaction of social circumstances, basal arousal, emotional regulation, and cognitive processes (Lansford et al., 2006; Lansford, Malone, Dodge, Pettit, & Bates, 2010). Information processing is hypothesized to follow five steps: (a) encoding social and environmental cues, (b) interpreting cues, (c) setting social goals, (d) searching for alternative responses, and (e) making a response decision (Crick & Dodge, 1994; Pakaslahti, 2000). As sequenced skills, these five information-processing competencies are the organizing framework of the Making Choices program.
Although SIP is often described as a linear process, processing is viewed as iterative and nonlinear such that new information constantly has the potential to reformulate goals, responses, and decisions. Information processing is affected by emotional regulation. Although less is known about emotional processing, physiological arousal and unregulated emotions may short-circuit information processing (Leve, Pears, & Fisher, 2002). Deficits in emotional regulation and SIP skills are related to a variety of negative child developmental outcomes, including peer rejection and antisocial aggressive behavior (Crick, 1995; Crick & Werner, 1998; Delveaux & Daniels, 2000; Dodge, 1980; Lengua, 2003; Li, Fraser, & Wike, 2013; Nelson & Crick, 1999).
SIP theory provides a skill-oriented basis for understanding social competence in childhood; and, at the same time, SIP-related skills appear to be malleable in intervention. That is, training to promote social competence in children can be structured to follow the steps in information processing (Dodge, Bates, & Petit, 1990; Fraser et al., 2005; Guerra & Slaby, 1990; Hudley & Graham, 1993; Li et al., 2013; Martin, 2010; Rabiner & Coie, 1989). Research on children in China indicates that gaps in SIP skills are related to poor developmental outcomes, such as peer rejection and aggressive antisocial behavior (Choy, 2001; Fung, Gerstein, Chan, & Engebretson, 2013; Liao, Li, & Su, 2014; Liu, Yu, Yang, & Pei, 2011; Wang & Hu, 2003; Xu & Zhang, 2008; Zheng, 2002).
Making Choices: Skills Training to Strengthen Information-Processing Skills
Making Choices is designed to strengthen skills in processing information and regulating emotions. Intended for elementary school children, the program has been tested in a variety of settings by itself (Fraser et al., 2004, 2014; Terzian et al., 2014) and in conjunction with other interventions (Fraser, Day, Galinsky, Hodges, & Smokowski, 2004). In addition, it has been adapted for preschool-age children (Fraser et al., 2011).
Fully manualized, the Making Choices intervention is comprised of 22 sequential 45-min lessons. Delivered in the third grade, the program builds on children’s socioemotional knowledge and life experiences. The lessons teach emotion recognition, strategies for self-regulation, and effortful control. In addition, the curriculum focuses on the SIP skills of identifying and interpreting social cues, setting prosocial goals, formulating ways to reach goals, choosing which goal to pursue, and then carrying out choices. The problem-solving process is introduced to students, and then they practice skills through interactive play-related activities and rehearsal. Rehearsal is designed to lead to skill acquisition through repetition (Cartledge & Milburn, 1996). Participatory activities utilize a variety of instructional modes to engage students with different strengths and preferences, using both individual and group procedures. For example, children described physiological responses to emotions by coloring outlines of a human body and then discussing the feelings that inspired their drawings. Another group exercise invites students to imagine the intentions behind the actions of comic strip characters. Students also perform skits that dramatize the skills of self-talk and sequential problem solving. To promote generalization of the skills (see Gresham, 1997), activities provide for both individual- and group-setting practice. Making Choices includes instructional features designed to strengthen program implementation (see Mitchem & Young, 2001). For example, the lessons are relatively short, consistently structured, fit into the regular classroom routine, and are aligned with standard academic learning goals.
Prior evaluation studies of the Making Choices program found effects on SIP skills, peer acceptance, social competence, emotion regulation, and aggression (Fraser et al., 2005; Smokowski et al., 2004). A previous study using the same intervention cohorts but a different comparison cohort (N = 548) found positive effects on aggression and social competence (Fraser et al., 2005). The study also detected an interaction effect for gender, with boys demonstrating greater reductions in aggression than girls. Posttest-only measures of SIP skills suggested that intervention students in Making Choices were more skillful than comparison students in encoding social cues and setting prosocial goals. A final study comparing the efficacy of Making Choices using pre- and posttest SIP data (Fraser et al., 2014; Terzian et al., 2014) found the intervention associated with reduced levels of aggression and improved SIP skills. Gender-moderating effects were also observed; boys displayed significant reductions in aggressive behavior and significant increases in positive social goals from pretest to posttest, whereas girls’ aggressive behaviors and social goals showed no significant changes.
The cultural adaptation of Making Choices for children in China took place in two phases. Focused on revising the U.S. program manual, Phase 1 involved a multidisciplinary team of 12 researchers in China and the United States. The team read the Making Choices program manual and related materials, observed Making Choices being used in U.S. elementary schools, visited Chinese primary school classrooms, discussed the project with teachers and school administrators in both countries, and met with Chinese public officials to understand the needs of local stakeholders. In addition, Phase 1 activities included appraisal of the research literature on child development in China, translation and back-translation of the treatment manual, adaptation of program materials and vignette-based measures for assessing SIP skills in children, review of materials and measures by experts, and a public hearing by Chinese scholars and school officials in Beijing. Phase 2 was comprised of a quasi-experimental pilot test designed to assess the impact of the adapted program on the social skills of children. In this article, we report on the adaptation of Making Choices and the initial pilot test of Let’s Be Friends.
Method
Design
The pilot test was designed to determine whether the SIP skills of children exposed to Let’s Be Friends differed significantly from those of comparable children who were not exposed to Let’s Be Friends program. That is, focused on key proximal outcomes, the goal of the evaluation was to determine whether the Let’s Be Friends intervention succeeded in improving information processing, a set of skills related strongly to positive developmental outcomes in children. A pretest–posttest quasi-experimental design was used (Shadish, Cook, & Campbell, 2002).
The project was implemented in Happy Camp programs of the Hexi District of Tianjin, a major urban area in northeast China. Happy Camps provide after-school child care, a service that is ubiquitous and free for elementary school children across China. Although no survey data are available, the director of the Hexi District indicated that nearly all elementary school children attend Happy Camps. From 13 neighborhoods in the district, 5 neighborhoods were randomly selected. The Happy Camps in these five neighborhoods were invited to participate in the study and all agreed to adopt the program. At the same time, children who did not attend Happy Camps but who lived in the same neighborhoods were invited to participate in the study through recruitment at neighborhood schools. These children served as a nonequivalent comparison condition.
In the participating Happy Camp centers, Let’s Be Friends was adopted as a core element of the curriculum. It was explained to parents and children. Parents were invited to participate and, if they declined, they were given the option of sending their children to a convenient, nearby neighborhood center that did not offer the program. Following local educational research standards, the Chinese investigators did not recommend written parental informed consent or child assent for the adoption of Let’s Be Friends in the Happy Camps curriculum or for the collection of data. Data collected from the students were confidential, and the investigators did not reveal any identifiable information. All analyses were conducted using de-identified data.
Participants
The Let’s Be Friends program was delivered in the five Happy Camps by 10 trained program specialists who were masters of social work (MSW) graduate students at Nankai University. Each graduate student was assisted by a full-time Happy Camp staff member and an undergraduate student majoring in social work at the same university. As shown in Table 1, 135 students who were 7–12 years of age began the study. Of these, 44 (32.6%) children dropped out. Anecdotal evidence from interviews suggests that the parents of children who dropped out preferred Happy Camp time to focus on academic achievement.
Pretest and Posttest Sample Sizes in Intervention and Comparison Conditions (Names of Neighborhood are Masked).
The comparison group was comprised of 123 students who were not enrolled in Happy Camps that offered Let’s Be Friends. These children were from the same neighborhoods and attended the same primary schools as program participants. Data collection occurred through the local primary school. None of these students dropped out of the study.
The evaluation was based on the data of 214 participants, including both treated (n = 91) and comparison (n = 123) students. Over 90% of the children were from single-child households. In the intervention condition, 20 (22.0%) children were from immigrant families and in the comparison condition, 42 (34.1%) children were from immigrant families.
Adaptation of the Making Choices Program
Adapted from Making Choices, Let’s Be Friends was designed to help children solve interpersonal problems, cultivate positive social skills, and reduce peer rejection. As shown in Table 2, the manual included seven units, with the first unit helping children to identify and regulate emotions. The other six units corresponded to SIP-related skills. Emphasis was placed on applying skills in peer- and work-related social situations. Different from the exam-oriented education that predominates in Chinese schools, the lessons encouraged stepping into social situations and refining social skills through lively, enjoyable games and role-play. Also breaking with tradition, group leaders were asked to debrief students after the activities and to accept as correct many possible responses to each situation. The Let’s Be Friends program is available in Chinese from China Population Press.
Social Information-Processing Skills by Unit Content in Let’s Be Friends.
To adapt Making Choices, a multidisciplinary team of researchers at Nankai University and the University of North Carolina at Chapel Hill was formed. The team was comprised of faculty with expertise in social work, child development, education, psychology, and English. In a series of face-to-face meetings and phone calls, the two teams aimed to understand the SIP theory and interactive teaching methods of the Making Choices program, and the needs and culture of primary school children in Nankai. The manual and supporting research materials, including measures, were given to the Nankai team. Working in small units, they translated the manual section by section. As they worked, they flagged words, concepts, references, and activities that appeared to have different cultural meaning or had no equivalent in Chinese language or culture. The Nankai team peer reviewed each other’s work and suggested equivalent content and language. Members of the Making Choices team met with the Nankai group to review the adaptation and discuss the identified cultural incongruities, to describe the theory and intent of specific activities in detail, and to determine a Chinese equivalent or option. Three new lessons were added by the Nankai group to specifically address children of single-child households. A side-by-side draft manual in Chinese and English was sent back to the Making Choices group and was reviewed and corrected by the two bilingual/bicultural members of the group. A final revision was reviewed by an outside bilingual/bicultural reader who was not familiar with the Making Choices program. Through this process, dozens and dozens of incremental corrections were made to the translation and also to the conceptual adaptation of the draft manual and the measures.
Implementation of Let’s Be Friends Program
Training and supervision related to Let’s Be Friends followed the model used in prior U.S. studies. That is, organized by the lead author, a 3-day orientation and initial training was conducted for both program staff and graduate students in Nankai University. The third author attended this training in person in Tianjin. The other authors participated via teleconferencing. The training reviewed the core features of SIP theory and child development, lesson-by-lesson program content, and the evaluation plan. Role-play simulations were used to demonstrate teaching scenarios and lesson content. A key feature of this training involved demonstrating an interactive as opposed to a didactic style of group leadership.
The initial training was followed by weekly clinical supervision by the first author. Program staff and graduate students met with the first author to review and prepare “lessons.” During these meetings, implementation issues were discussed. These included group work challenges such as involving shy students and managing disruptions.
In addition, the lead author visited all five program sites and observed Let’s Be Friends sessions each week during the first 3 weeks of implementation. Subsequently, she visited every third week. For some of these meetings, the lead author was accompanied by the Hexi District Happy Camp director whose presence and interest reinforced the importance of program fidelity.
Halfway through the implementation, a 1-day retreat to discuss the project was held at Nankai University in Tianjin. Program staff, MSW students, officials from Hexi District, and school teachers participated. The purpose of the meeting was to review implementation issues and discuss the evaluation.
Measures
Because the intervention agents were not masked to the program and worker ratings of children could be affected by social desirability or other measurement artifacts, the outcome—SIP skill—was measured by a test of information-processing competence. That is, the SIP skills of children were assessed using the skill-level activity (SLA) test. The purpose of the SLA is to measure how children interpret and respond to specific interpersonal vignettes. Through pictures and stories, children are asked to imagine that they are in three different story situations involving joining a “dodgeball game,” wearing “new pants,” and losing “new magazine.” The scenarios were reviewed and selected to have cultural relevance for Chinese children. Children are instructed to pretend to play the leading roles in these stories and to make judgments about the situations in which they find themselves. Based on the responses of children in each hypothetical situation, competence in four SIP skills is assessed with a grading rubric. These four skills include encoding (α = .78), interpretation (hostile attribution; α = .52), goal formulation (α = .76), and response decision (α = .80).
The SLA is an adaptation of Dodge’s Home Interview for attributional bias (Dodge, 1980) modified for group administration as a pen-and-paper measure. Students listen to three short stories in which a peer interaction of ambiguous intent occurs. They are asked to put themselves in the place of the main character and answer the questions according to how they would respond in the given situation. As a measure of hostile intent, they are asked to attribute friendly, hostile, or unknown intent to the antagonist in the story. For the measures of goal formulation and response decision, students select from among aggressive and nonaggressive response options. For encoding cues, students search an accompanying illustration for objects, persons, and expressions that “help tell you what is happening in the story,” and they circle as many relevant cues as they can find. A coder compares circled objects with a rubric of acceptable cues, and the total number of acceptable cues is recorded. An independent assessment of the rubric-based scoring protocol found high agreement between two coders who scored three case scenarios on 414 child reports (κ = .96; Fraser et al., 2004).
Data Analysis
Because of the nonequivalent comparison group design and the attrition in the intervention condition, a rigorous analysis was needed to account for differences between the intervention and comparison groups (Guo & Fraser, 2010; Imbens, 2004; Sobel, 1996). We applied the Neyman–Rubin counterfactual framework as a conceptual model to guide the data analysis.
According to the logic of the counterfactual model, individuals in the treatment and the comparison groups have potential outcomes in both states, that is, the one in which they are observed and the one in which they are not observed. The Neyman–Rubin framework offers a practical way to evaluate counterfactuals, when random assignment is not used or when comparability is compromised by attrition or other factors. Working with data from a sample that represents the population of interest (i.e., using
We employed two propensity score models to analyze data. A propensity score is a conditional probability of a participant receiving treatment, given observed covariates (Rosenbaum & Rubin, 1983). The propensity score can be conceptualized as a balancing score representing a vector of covariates or “conditioning variables.” We used a binary logistic regression to estimate propensity scores for the probability of receiving the Let’s Be Friends program. Four child-related conditioning variables were used, namely, age at baseline, gender (coded 1 if male and 0 if female), birthplace (a dichotomous variable coded 1 if urban and 0 if rural), and community (four dichotomous variables indicating different neighborhood groups). Birthplace was used to control for migrant and Hukou status.
To balance the intervention and comparison groups on these measures and to provide a measure of the sensitivity of outcomes to different statistical adjustment procedures, we employed two propensity score models to estimate the treatment effect: (1) propensity score weighting (PSW) and (2) optimal full matching (OFM). The PSW approach treats propensity scores as sampling weights (Hirano & Imbens, 2001). Using PSW, we estimated two kinds of treatment effects, that is, ATE and the average treatment effect for the treated (ATT). Denoting the estimated propensity score as P, we calculated weights as follows: when estimating ATE, the weight for a treated participant is 1/P and for a control participant is [1/(1 − P)]; when estimating ATT, the weight for a treated participant is 1 and for a control participant is [P/(1 – P)]. After creating these weights, we ran a multiple regression model on each outcome variable and treated a specific propensity score weight (i.e., either the weight for ATE or that for ATT) as sampling weight. The outcome regression employed age at baseline, gender, birthplace, and treatment condition as predictors.
OFM
Optimal matching is the process of developing matched sets (A1,…As; B1,…Bs) with size of (α, β) in such a way that the total sample distance of propensity scores is minimized. Formally, optimal matching minimizes the total distance ▵ defined as
For this study, we employed OFM. In full matching, each treated participant matches to one or more comparisons, and similarly each comparison participant matches to one or more treated participants; formally, this is a stratification of (A1,…As; B1,…Bs) in which the minimum of (|AS|,|BS|) = 1 for each s.
The purpose of propensity score analysis is to rebalance data when intervention and comparison conditions differ. Conditions may differ for a variety of reasons. We used propensity score methods to control for differences due to selection bias and attrition. To test for covariate balance before and after weighting, we used the Wilcoxon rank-sum (Mann–Whitney) test, independent sample t-test, and χ2 test. The Hodges–Lehmann aligned rank test (Hodges & Lehmann, 1962) was used to assess differences in outcomes between intervention and comparison participants for the sample generated by the OFM. It also tests whether the ATE is statistically significant.
Treatment effects
Following previous work (Fraser et al., 2005, 2011), one-tailed directional hypothesis tests were used. This method is based on previous findings of positive effects and on the a priori hypothesis that the intervention improves outcomes (for a comparable example, see Raudenbush & Bryk, 2002, pp. 112–113).
Findings
Balance Tests Before and After Adjustment on Propensity Scores
As shown in Table 3, the two groups differed significantly at pretest, and PSW and OFM were used to rebalance the conditions. Had these pretest differences not been taken into consideration in evaluation, the estimates of treatment effects would have been biased. Using PSW and OFM, the two groups were rebalanced on the four covariates. After adjustment, all p values from the imbalance check using PSW were statistically nonsignificant. The imbalance check for OFM showed that on each covariate, dXm was much smaller than dX and below the 0.10 threshold, indicating that after full matching, the standardized differences between treated and nontreated groups were less than 10% of a standard deviation.
Sample Descriptive Statistics and Imbalance Check.
Note. PSW = propensity score weighting. Reference group for categorical variable is shown in parenthesis. Rank-sum test refers to Wilcoxon’s rank-sum (Mann–Whitney) test. dx is the measure of imbalance prior to matching and dXm is the measure of imbalance after matching.
As shown in Table 4, the estimated treatment effects based on the PSW and OFM models suggest the Let’s Be Friends program was effective. Conditioned on balance on the four covariates, positive intervention effects were observed. Of the 12 SLA outcomes, PSW found 10 ATEs that were directionally beneficial to study children; of the 10 beneficial changes, 3 were statistically significant (p < .01 for “dodgeball—encoding,” p < .05 for “dodgeball—response decision,” and p < .05 for “new magazine—hostile attribution”), and 3 showed statistical trends (p < .1 for “dodgeball—hostile attribution,” “dodgeball—goal formulation,” and “new magazine—encoding”). All significant effects except “new magazine—hostile attribution” had effect sizes greater than .2. Of the 12 SLA outcomes, PSW found nine ATTs favoring study children. Of the nine beneficial changes, four were statistically significant (p < .01 for “dodgeball—encoding,” p < .05 for “dodgeball—goal formulation,” p < .05 for “dodgeball—response decision,” and p < .05 for “new magazine—encoding”). Two showed statistical trends (p < .1 for “dodgeball–hostile attribution” and “new magazine–hostile attribution”).
Estimated Program Effects on Social Information–Processing Skills.
Note. PSW = propensity score weighting; ATE = average treatment effect; ATT = average treatment effect for the treated. Beneficial changes consistent with hypotheses are underlined. n differs slightly among different models due to missing data.
**p < .01. *p < .05. †p < .1, one-tailed test.
The OFM with a Hodges–Lehmann test found 10 ATEs directionally favoring study children. Of the 10 beneficial changes, 2 were statistically significant (p < .05 for “dodgeball–encoding” and for “dodgeball–response decision”). One showed a statistical trend (p < .1 for “new magazine–encoding”).
Discussion and Applications to Social Work
The findings contribute to a growing body of controlled trials of interventions in mainland China and Hong Kong (see, e. g., Leung, Tsang, Sin, & Choi, 2014). An evidence-supported, theory-based intervention was adapted for Chinese children. Using theory-derived measures, the adapted intervention was tested with a yoked-comparison group design. The yoking of a neighborhood comparison condition and the use of entire Happy Camps as intervention sites reduced the potential for postassignment confounds, such as compensatory rivalry. Comparison group children were unaware of the intervention condition, and children in Happy Camps regarded the Let’s Be Friends program as a routine feature of after-school care.
The results suggest that the Let’s Be Friends program had a positive impact on children’s social–emotional skills. Directional effects favored the intervention condition. Findings were strongest for encoding, where an effect size of .250–.300 was observed on two of the three assessments. Patterns for interpretation, goal formulation, and response decision-making skills were mixed. Provisionally, the pilot test data suggest that the Let’s Be Friends program did what it was intended to do. That is, it has the capacity to strengthen social–emotional skills that are related to—that mediate—a variety of positive developmental outcomes, including both academic achievement and long-term socioeconomic success.
The design and development of social work interventions for school-age children in China could influence the nature of Chinese social work. The Education Ministry in China does not currently recognize school social work; however, school social workers have been used in schools in Sichuan where students and teachers continue to recover from a disastrous earthquake. Lack of institutional support and understanding is a barrier to the development of school social work in China. The findings in this article provide the first empirical support for the use of social work services in after-school settings. This has the potential to influence, at least in part, the shape social work in China.
More generally, the findings contribute to an emerging definition of social work practice in China. In cities, nearly every community (neighborhood committees, 居委会) has a service center. Some neighborhoods also have “social work stations” (社工站), and they employ social workers to assist residents. As these programs take shape across China, the findings from Let’s Be Friends have the potential to influence program content in Happy Camps and social work stations.
Clearly, however, the translation of evidence-supported interventions from one cultural context to another requires deep understanding of the core concepts of interventions and deep understanding of culture. A cross-cultural team of researchers and practitioners from both China and the United States was involved in the adaptation of the Making Choices program. The first action undertaken by the team was to determine the applicability of the program’s deep structure—the core ingredients (i. e., SIP skills)—to child development in China. Careful examination of information-processing theory and research in China indicated that strengthening of Chinese children’s processing skills could be expected to produce positive developmental outcomes. That is, based on the Chinese research literature, the key features of Making Choices appeared relevant in China.
The Making Choices manual was then translated into Chinese and refined in a systematic process of translation, back-translation, and retranslation to ensure that the content was familiar to and comfortable for children in China. Professionals from both cultures were closely involved. They engaged in careful and critical analysis of program theory, intervention activities, teaching contexts, cultural relevance, and measures. This kind of examination of an intervention developed in one culture and adapted for use in a different culture appears to be an emerging challenge as evidence-supported interventions are adopted for use across a variety of populations and contexts (Sundell, Ferrer-Wreder, & Fraser, 2014).
The results are promising but the patterns in the findings suggest that the following two streams of development are needed: (a) refinement of the SLA and (b) reexamination of some program materials. Although pretest–posttest changes were in the predicted direction, only 6 of 18 tests related to goal formulation and response decision making had p values less than .10. This pattern could be due to features of the program or to limitations in the measurement protocol. Suggesting a measurement explanation, the new pants scenario produced no significant findings, while other scenarios appeared to have greater treatment sensitivity. Measuring latent variables such as cognitive skills in children is inordinately difficult, and the findings indicate that the new pants scenario lacked the ambiguous intent required to distinguish children with high SIP skills from those with low SIP skills. That is, regardless of SIP training, children were able to correctly encode and consistently interpret social cues in the new pants scenario. In short, the intent and meaning of actions in the new pants story appear to have been relatively clear regardless of SIP training exposure. The new pants scenario warrants revision prior to undertaking a larger trial.
From the anecdotal interviews with parents and based on the attrition rate, revisions of program materials and procedures are also warranted. Of the 135 children in the experimental group, only 91 (67.4%) completed the SIP training. Most of this attrition occurred as parents withdrew their children from the program. This level of attrition affects the generalization of the findings to other Happy Camps and similar settings. But perhaps more important, it suggests that features of the pilot program were not attractive to many Chinese parents. Parents appear to have misunderstood the purpose of the Let’s Be Friends program. Many regarded it as “games and fun,” having little to do with learning. They were leery of investing time into activities that seemed to have little relationship to school studies, and they preferred that their children spend camp time focused on academic achievement. Clearly, program recruitment and orientation materials failed to make a compelling case for social–emotional skills training. The pilot data—including, in this case, interviews with parents who chose to drop out of the program—suggest that the conceptual ties between academics and social development must be underscored in future trials.
In addition, the findings are conditioned by the limitations of the design. The balance or equivalence of the two conditions was affected both by the yoking of a comparison group and by differential attrition in the intervention condition. Without adjustments, the intervention and comparison conditions were significantly different at pretest. Using propensity scores based on four covariates (age, gender, birthplace [rural vs. urban], and community residence), the two groups were rebalanced prior to conducting intervention tests. Two different rebalancing methods produced equivalence on balance checks. However, though the adjustments appear to have been successful, balance cannot be inferred for unobserved covariates. It is possible that the findings were due, in part or in full, to unmeasured confounding variables. As a covariate, birthplace (rural vs. urban) represents a host of the social, demographic, and risk-related characteristics of children in China; however, other important covariates such as family income, parental education, and length of time in the urban area were not measured and controlled. Although statistical adjustments such as the use of propensity scores can improve internal validity, they do not afford the advantages of a randomized trial, where balance on unobserved covariates may be assumed when sample sizes are adequate.
Besides recruitment procedures and measurement, the format of intervention warrants attention. The manual uses an interactional teaching style that is different from the authority-oriented, exam-focused nature of much Chinese teaching. Although effective in conveying content, the unfamiliar style of the learning activities may have exacerbated the views of some parents that the program was insufficiently academic. At least initially, a more familiar educational format may have rendered the material more acceptable to parents.
In the same vein, other program features may have affected the findings. Similar to an early study of Making Choices (Fraser et al., 2004), MSW students, assisted by local staff, were used as intervention agents. The students followed the treatment manual carefully and were monitored in supervision, which included qualitative case monitoring. However, the use of beginning-level group leaders may have weakened the results.
Considering these limitations, the findings are promising. Further work on adapting the intervention to the Chinese population is warranted. The effect sizes were small but consistent with other research on social–emotional skills training (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Wilson & Lipsey, 2007). The data suggest that the Let’s Be Friends program has the potential to strengthen the SIP skills of children in after-school settings in China.
Footnotes
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors received funding from the China National Commission on Family Planning and Population Control (NCFPPC) and from the University of North Carolina at Chapel Hill. This funding was provided in order to design and develop the Let’s Be Friends program. This report summarizes the design and development of the program.
