Evaluation of the Project P.A.T.H.S. in Mainland China: Findings Based on Student Diaries

Abstract

Objectives:

Based on 859 student diaries, the present study evaluated a positive youth development program entitled “Tin Ka Ping (TKP) Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.) ” project implemented in mainland China during the 2015–2016 academic year.

Method:

To understand the perceived effectiveness of the students, the study analyzed quantitative as well as qualitative data derived from the student diaries.

Results:

The students held very positive views toward the program and the implementers. The narratives of the students also showed that they perceived improvement in the intrapersonal, interpersonal, familial, and societal domains after joining the program.

Conclusion:

In conjunction with the subjective outcome evaluation findings, the present findings suggest that the “TKP P.A.T.H.S.” project is able to promote holistic development of secondary school students in mainland China.

Keywords

reflective journal qualitative evaluation program evaluation positive youth development

Influenced by the evidence-based practice movement in the field of medicine starting from the 1970s, the number of evidence-based interventions has increased in many disciplines such as psychology, youth services, social work, and nursing (Catalano et al., 2012; Mesibov & Shea, 2010). For successful implementation of evidence-based intervention and prevention programs, evaluation is an indispensable element to provide evidence for program effectiveness and program improvement.

Quantitative and qualitative methods are two commonly adopted inquiries in program evaluation, with the former focusing on numbers and the latter using words and visuals (Lichtman, 2013). Quantitative methods enable researchers to carry out quick and effective program evaluation by involving a large sample size and performing sophisticated statistical analyses (Atieno, 2009). Moreover, quantitative methods can examine logico-deductive research questions, provide descriptive information, and assess generalizability of the findings to different populations. However, one notable weakness of quantitative methods is their inability to reveal subjective experiences of the participants behind the quantitative findings (Choy, 2014).

In contrast, although qualitative methods typically do not involve many respondents, which reduces the generalizability of the findings (Choy, 2014), qualitative methods have the advantage in providing in-depth information. Holloway and Biley (2011) suggested that “qualitative inquiry is still the most humanistic and person-centered way of discovering and uncovering thoughts and action of human beings” (p. 974). By using qualitative evaluation methods, hidden thoughts, ideas, and underlying beliefs could be captured, permitting researchers to have a more holistic perspective on the needs and changes of the target groups (Holloway & Biley, 2011). As Patton (2002) put it, “qualitative findings in evaluation illuminate the people behind the numbers and … to deepen understanding” (p. 10).

Although quantitative and qualitative evaluation methods have relative merits and weaknesses, they constitute alternative but not mutually exclusive evaluation strategies. Based on evidence-based approach, a better way to yield comprehensive evaluation findings is to combine both quantitative and qualitative evaluation data (i.e., mixed-methods evaluation; Patton, 2008; Shek, 2013). In addition, qualitative data can be analyzed using quantitative approaches such as assessing the proportion of positive responses out of the total responses (Shek, 2010; Shek & Sun, 2012a).

In line with the above notion, mixed-methods evaluation is increasingly applied in positive youth development (PYD) programs, which endeavor to prevent problematic behaviors in adolescents through promoting a wide range of psychosocial competences (Lerner et al., 2013). For example, in a large-scale curricula-based PYD program in Hong Kong entitled “Positive Adolescent Training through Holistic Social” programs (P.A.T.H.S. project), systematic evaluation utilizing quantitative evaluation (e.g., randomized group trial and subjective outcome evaluation via the client satisfaction approach) and qualitative evaluation (e.g., observations, focus groups, and student weekly diaries) was carried out (e.g., Ma & Shek, 2017; Shek, 2014; Shek & Sun, 2013). Likewise, when the P.A.T.H.S. project was transplanted to mainland China, mixed-methods evaluation was carried out. While the effectiveness the P.A.T.H.S. project in Hong Kong has been widely supported (Catalano et al., 2012; Shek & Wu, 2016; World Health Organization, 2016), evaluation findings of the project in mainland China have not been sufficiently reported. Hence, this article attempts to outline the evaluation findings based on student diaries for the P.A.T.H.S. project implemented in mainland China.

Student Diary as a Qualitative Evaluation Method

Reflective writing such as journals and diaries is one of the methods being commonly used in qualitative research. As stressed by Beauchamp (2006), reflection is a process (e.g., thinking and understanding) that concerning a particular object (e.g., experience, belief) in view of a particular goal or rationale (e.g., thinking differently or more clearly). Unlike other qualitative approaches such as individual interview and focus group, the private and confidential nature and spontaneous context of reflective writing allows authors to express themselves in an honest and open manner without hesitation of receiving comments from others (Travers, 2011).

Through disclosing one’s own feelings and thoughts in diaries, the writers can further explore themselves and their views of the world, hence enhancing their understanding of the target objectives and self-awareness (Pavlovich, Collins, & Jones, 2009). In view of its value in providing in-depth information, diary writing has been widely used in educational and clinical contexts to document human behavior. For example, Swim, Hyers, Cohen, and Ferguson (2001) used daily diary to investigate the incidence, nature, and impact of everyday sexisms reported by college women and men. Travers (2011) also used diary to explore final-year students’ stressors and coping skills within a UK university business school.

Reflective writing has also been increasingly utilized to evaluate evidence-based intervention and prevention programs. Scholars argued that the journal or diary written by the participants is an informative data source for evaluating program effectiveness and suggesting directions of future improvement (Shek, Sun, Lam, Lung, & Lo, 2008). In health-care field, Mak, Plant, and Toussaint (2006) used reflective journals to assess whether a 6-month training program for prevocational medical practitioners achieved its goals. Cohen, Leviton, Isaacson, Tallia, and Crabtree (2006) evaluated a preventive care program using interactive online diaries, which set an avenue for real-time communication between the evaluators and the program implementers.

In the field of social sciences, Schmitz and Wiese (2006) used diaries to evaluate the effectiveness of an intervention promoting university students’ self-regulated learning. Travers, Morisano, and Locke (2015) recently used students’ reflective diaries to explore how students benefited from a personal growth goal-setting program. With specific respect to social work, diaries or journals are also regarded as valuable evaluation materials that can reveal the views and impressions of participants and/or program implementers (Grinnell, Gabor, & Unrau, 2015; Robson, 2000). For instance, Siu, Chan, Poon, Chui, and Chan (2007) used the participants’ diaries to monitor their self-management behaviors and changes in people with chronic illness after participating in a 6-week program. Hasson, Blomberg, and Dunér (2012) also used diary data to investigate the implementation fidelity and its moderating factors of a complex care intervention for elderly people.

As a qualitative evaluation method, reflective diary is increasingly applied in evaluating PYD programs as well. For example, in the aforementioned P.A.T.H.S. project in Hong Kong, student weekly diary was adopted as one of the evaluation strategies (Shek, 2010; Shek & Sun, 2012a). Findings based on student diaries revealed that students held very positive views toward the program, instructors, and program benefits where students gained substantial improvement in self-confidence, emotional competence, communication skills, and social competence (Shek, 2010; Shek & Sun, 2012a, 2013). These results echo the quantitative evaluation findings (Shek & Sun, 2013; Shek & Wu, 2016).

Overview of the “Tin Ka Ping (TKP) P.A.T.H.S. Project”

Financially supported by the Hong Kong Jockey Club Charities Trust, the P.A.T.H.S. project has been implemented in Hong Kong over a decade since the 2005–2006 academic year. This project included two tiers of programs tailor made for adolescents with different levels of need. Specifically, the Tier 1 program was curricula based, designed for all junior secondary students, and built on the 15 PYD constructs (e.g., bonding, social competence, moral competence, self-efficacy, and spirituality) that are incorporated in successful PYD programs (Catalano, Berglund, Ryan, Lonczak, & Hawkins, 2004) with the adoption of experiential teaching and learning pedagogy. In contrast, the Tier 2 program was specifically designed for students having greater psychosocial needs (Shek & Wu, 2016). Findings based on different evaluation strategies consistently showed that the project promoted multiple psychosocial competences and reduced risk behaviors in adolescents (e.g., Ma & Shek, 2017; Shek & Sun, 2012b; Shek & Wu, 2016; Shek & Yu, 2013).

In view of the prominent success of the P.A.T.H.S. project in Hong Kong, the Tier 1 program was transplanted to mainland China and initially implemented in one secondary school in Shanghai from 2007 to 2010. As the result of the 3-year initial implementation was very positive (Shek, Han, & Ma, 2009), the Tin Ka Ping Foundation (TKPF) decided to launch a pilot implementation from 2011–2012 to 2013–2014 academic years in four secondary schools in East China which constituted the “TKP P.A.T.H.S. project.” During the pilot implementation, scholars from the P.A.T.H.S. research team and one university in mainland China collaborated with the four schools to adapt the junior secondary curricula used in Hong Kong for local use in mainland China. The fundamental adaptations included (a) traditional Chinese characters were changed to simplified Chinese characters, (b) Cantonese expressions were changed to Mandarin expressions, and (c) some of the teaching materials with Hong Kong characteristics were replaced by those conceptually similar yet locally relevant materials. The adapted junior secondary curricula used in the “TKP P.A.T.H.S. project” consisted of 26, 26, and 24 curricular units for Grades 7, 8, and 9, respectively.

The 3-year pilot implementation also yielded very encouraging results (Shek, Han, Lee, & Yu, 2013, 2014; Shek et al., 2014), which motivated the TKPF to support the project for another 3 years (i.e., 2015–2016 to 2017–2018 academic years as the full implementation phase). Notably, the year before the full implementation (i.e., 2014–2015 academic year) was the preparatory year when four training programs (69-hr training in total) were conducted for potential implementers (474 participants and 785 participation times in total). Furthermore, to satisfy the needs of senior secondary students, the research team started developing senior secondary curricula in 2015.

In the first year of the full implementation phase, a total of 30 secondary schools in more than 25 mainland China cities were selected as project schools by the TKPF based on following criteria: (a) the school principal supported the philosophy of PYD and program implementation, (b) the school established a P.A.T.H.S. team comprised of one school management member as the leader and at least five teachers as the implementers with each team member receiving at least 20-hr training in the preparatory year, and (c) the school would keep on participating in the training program.

These project schools implemented junior and/or senior secondary programs at one or more grade levels by incorporating the curricular units of the project into their formal school curricula. In total, 25 of the 30 project schools implemented the junior secondary program and 15,361 students participated in the program. Several measures were taken to facilitate program implementation. First, teaching manuals showing detail procedures of delivering the curriculum at each grade level were developed and offered to all implementers who were well-informed to follow the manuals. Second, one training workshop was conducted every 4 months to help the implementers better understand the rationales and concepts of the curricula and master experiential teaching skills. Third, a “cowalker” scheme was initiated to conduct school visit for teaching improvement and to facilitate regular online communication between the research team and the project schools on issues like school implementation progress and quality, difficulties encountered, and possible solutions. Fourth, each school was required to submit an implementation report every 4 months to reveal the number of students joining the project, the number of units being delivered, the degree to which the teachers delivered the curriculum as designed, and the implementers’ reflection. Based on our observations and school reports, the program was implemented smoothly with good fidelity in the 2015–2016 academic year.

It is widely agreed that the best way to evaluation a PYD program is to conduct experimental studies (Shek, 2013). However, given the financial and manpower constraints, such evaluation strategies were not used in the full implementation of the TKP P.A.T.H.S. project. Instead, subjective outcome evaluation via the client satisfaction approach, a widely used quantitative strategy in human services (Fraser & Wu, 2016), was employed to examine the perceptions of the students and implementers on three aspects of the program including program quality, instructor (i.e., implementer) quality, and program benefits. According to the findings in the 2015–2016 academic year, over 90% of the students and implementers reported high satisfaction with the program quality (e.g., the design and objectives of the curriculum), instructor quality (e.g., teaching skills and interactions with the students), and the benefits of the program (Shek, Lee, & Ma, in press; Shek, Wu, & Law, in press; Shek, Zhu, & Leung, 2018). In particular, most of the students and teachers believed that the program helped the students promote various competences such as cognitive competence, emotional competence, social responsibility, and self-confidence, which ultimately lead to student holistic development.

While the above evaluation findings provided considerable evidence for the effectiveness of the TKP P.A.T.H.S. project, the analyses were predominantly quantitative in nature. To triangulate the findings and to further understand the subjective experience of the participants, student diary was adopted as a qualitative evaluation approach in the full implementation phase.

The Present Study

The present article attempts to report evaluation findings based on diaries written by the participants in the first year of the full implementation phase. Analyses and interpretation of the data were guided by the frameworks identified in previous studies (Shek, 2010; Shek & Sun, 2012a). Specifically, themes emerged from the diaries were classified into three domains including “views about the program,” “views about the instructor,” and “perceived benefits of the program.” Moreover, “perceived benefits of the program” were further examined with reference to four domains, including the intrapersonal, interpersonal, familial, and societal domains (Shek, 2010; Shek & Sun, 2012a).

As a study employing student diary as a qualitative evaluation strategy, the “quality” of the study is an important issue to be considered. Shek, Tang, and Han (2005) highlighted that 12 principles should be upheld in a rigorous qualitative evaluation study. For example, the study should have a clear philosophical stand and the number as well as the nature of the participants should be clearly described and justified. Other principles included description of and methods taken to address the issues of the data collection procedures; exposition of researcher biases and preoccupations as well as ways to deal with them; examination of alternative explanations, triangulation, and audit trails; and acknowledgment of limitations of the study. In the present study, we adhered to these important principles as far as possible. Similar to previous evaluation research utilizing student diaries (Shek, 2010; Shek & Sun, 2012a), a general qualitative orientation under postpositivism incorporating qualitative principles (e.g., reliance on nonnumerical raw data and holistic emphasis) was upheld as the philosophical orientation in the current study.

Method

Participants

In April 2016, among the 25 project schools that implemented junior secondary program, 14 were randomly selected and invited to join the present qualitative study. Among the 14 selected schools, 6 implemented the program at all junior secondary grades (i.e., Grades 7–9), 5 implemented the program at Grades 7 and 8 only, while 3 implemented the program at Grade 7 only. The mean number of Grades 7, 8, and 9 students joining the program across these schools was 341.78 (range: 40–820), 286.00 (range: 80–480), and 218.00 (range: 50–392), respectively. In total, we received 859 valid diaries (Grade 7 = 758, Grade 8 = 92, and Grade 9 = 9), with 15.84%, 2.92%, and 0.69% of the students participating in the program at Grades 7, 8, and 9 submitted their diaries, respectively. Because the cases obtained were based on random sampling and the response rate was 100%, generalizability of the findings can be regarded as high. Besides, most of the students wrote more than 300 words in their diaries and only 7 of 859 students wrote less than 100 words. Therefore, we believed with confidence that these diaries could provide adequate information regarding the students’ feelings toward the program.

Procedures

We conducted the present study in accordance with the postpositivistic stand which highlights reliability and validity as two hallmarks of research quality (Shek et al., 2005). In each of the 14 selected schools, the leader of the program implementers was the contact person who randomly invited some junior secondary students from one or more grades at the whole grade level or in some classes to write and hand in diaries at the end of June 2016 when the academic year ended. This sampling method ensures the present study’s ecological validity because in China, students are commonly asked by their teachers to write weekly diaries to express their views and experiences toward recent or current events.

The invited students were informed to reflectively express their perceptions or feelings regarding their participation in the program. The students were well explained that (a) their participation was entirely voluntary in nature, (b) there was no need to reveal their identities in the diary, and (c) the information they provided would be kept confidential and only used for academic research. All selected students were willing to join the present study and their consent was obtained. After all students had handed in their diaries to the teachers in each school, teachers passed student works to the contact person, who was responsible for sending a scanned copy of students’ diaries to the research team.

Data Analyses

Student diaries received from the 14 project schools were firstly transcribed into separate cases by research assistants who are native Chinese speakers. The transcription was checked by two members of the research team before being used as the basis of data analyses, which included two steps. The first step concerned analyses of the degree of positivity or negativity of the student diaries. The analyses procedures used in this step were those adopted in Shek (2010) and Shek and Sun (2012a). Specifically, five well-trained research staff members classified the diaries into four categories: “positive” (i.e., responses are primarily positive), “negative” (i.e., responses are primarily negative), “neutral” (i.e., there were both positive as well as negative responses), or “undecided” (i.e., mixed responses are related to positivity or negativity of the responses). Then, each diary was rated on a 7-point Likert-type scale: 1 = strongly negative (all or almost all responses are negative), 2 = moderately negative (most of the responses are negative), 3 = slightly negative (both negative and positive responses are present, with negative responses more than positive responses), 4 = neutral (negative and positive responses are presented with similar intensity), 5 = slightly positive (both negative and positive responses are present, with positive responses more than negative responses), 6 = moderately positive (most of the responses are positive), and 7 = strongly positive (all or almost all responses are positive).

Among the raw diaries, 15% (i.e., 130 cases) were randomly selected for calculating intra- and interrater reliability. Specifically, to calculate intrarater reliability, these selected diaries were rerated by the five colleagues who were involved in initial rating process. For interrater reliability, two other research staff who did not participate in initial rating were invited to rate the 130 randomly selected diaries individually.

The second step was qualitative analyses in terms of grouping students’ reflections into different domains. At this step, narratives illustrations were guided by the general qualitative analyses guidelines (Wolcott, 1994) and “a priori scheme” identified in the past studies (Shek, 2010; Shek & Sun, 2012a).

First, a well-trained research assistant and the second author read and reviewed student diaries several times to conduct a pattern analysis to identify content themes (e.g., “rich program content” and “achievement in emotional competence”) and reached a consensus on the themes. Second, conceptually similar themes were grouped to one of the three domains established in previous studies (Shek, 2010; Shek & Sun, 2012a), including “views about the program,” “views about the instructor,” and “benefits of the program.” Specifically, themes related to students’ perceptions of the program and their overall impressions of participation in the program were categorized into the domain of “views about the program.” Besides, themes about students’ perceptions of the program implementers were grouped under the domain of “views about the instructor,” while themes relevant to students’ positive changes and what they had learnt and achieved after joining the program were included in the third domain of “benefits of the program.” Third, the program benefits were further analyzed using a framework developed in previous studies based on more than 1,100 diaries (Shek, 2010; Shek & Sun, 2012a), including the intrapersonal (e.g., resilience and psychosocial competences), interpersonal (e.g., cooperation, communication), familial (e.g., relationships with family members), and societal (e.g., social responsibility) domains.

Results

Reliability of the Analyses

Intra- and interrater reliability of coding (“positive,” “negative,” “neutral,” and “undecided”) reached 100% and 99.6%, respectively. Intrarater agreement percentage for the 7-point rating was 90.0% and interrater agreement percentage was 91.9%.

Categorization in Terms of Positivity or Negativity of the Cases

As shown in Table 1, a vast majority of the diaries was regarded as “positive” at all junior secondary grades. In total, 831 (96.74%) diaries were categorized as “positive,” while 12 (1.40%) and 16 (1.86%) diaries were classified as “negative” and “neutral,” respectively.

Table 1.
Classification of Student Diaries.

Grade Total Number Positive Negative Neutral Undecided

n % n % n % n %

7 758 732 96.57 12 1.58 14 1.85 0 0.00

8 92 90 97.83 0 0.00 2 2.17 0 0.00

9 9 9 100.00 0 0.00 0 0.00 0 0.00

Total 859 831 96.74 12 1.40 16 1.86 0 0.00

Similarly, ratings for diaries at Grade 7 (M = 6.72, SD = .81), Grade 8 (M = 6.78, SD = .63), and Grade 9 (M = 6.78, SD = .44) were also quite positive and did not significantly differ from each other, F(2, 856) = .26, p = .77. As depicted in Table 2, a total of 724 (84.28%) diaries were rated as “strongly positive.”

Table 2.
Rating of Student Diaries.

Grade Total Number M SD 1 = Strongly Negative 2 = Moderately Negative 3 = Slightly Negative 4 = Neutral 5 = Slightly Positive 6 = Moderately Positive 7 = Strongly Positive

n % n % n % n % n % n % n %

7 758 6.72 .81 2 0.26 6 0.79 4 0.53 14 1.85 16 2.11 79 10.42 637 84.04

8 92 6.78 .63 0 0.00 0 0.00 0 0.00 2 2.17 4 4.35 6 6.52 80 86.96

9 9 6.78 .44 0 0.00 0 0.00 0 0.00 0 0.00 0 0.00 2 22.22 7 77.78

Total 859 6.73 .79 2 0.23 6 0.70 4 0.47 16 1.86 20 2.33 87 10.13 724 84.28

Views About the Program

Most of the students enjoyed the participating experiences and expressed their appreciation for the program in diaries. Students often commented the program as educative and practical because the program content (e.g., elements of peer relationships and emotions) was closely related to problems and challenges they encountered in school and family life. Thus, the program largely satisfied their developmental needs. A considerable number of students indicated that the program was meaningful and insightful.

Some students even regarded their participation in the program as “the most unforgettable time” and “the color of life.” As mentioned by students in the diaries, the program had very useful and educative content, clear and well-designed structure, and inspirational learning goals. Besides, students expressed that they were able to think more critically from different perspectives and apply what they had learned in the class into daily life through participating in diverse in-class activities (e.g., group discussion, self-reflection, and role-play). Overall, students found the program interesting and enjoyable. The following narratives illustrate some of students’ overall impression of the program:
“Participating in this program will be the most memorable time in my life and I will never forget it.”

“This program was like a light, it gave me directions and guides.”

“This program was a life key to open the door inside my heart, it also colored my life.”

Views About the Instructors

Most students were satisfied with the instructors’ performance, their attitude, and good preparation for the lessons. Students mentioned that instructors with warmth and kindness were able to engage every student in the class. In addition, students highly recognized the creative and stimulating teaching method utilized by instructors. Specifically, instructors did not merely deliver knowledge using didactic teaching, instead, they cared about students’ needs, gave students a lot of encouragements, helped students find their life goals, and tried their best to assist students to deal with challenges and difficulties. It was strongly emphasized by students that their relationship with instructors was greatly improved because of the program. Many students showed their appreciation to the instructors and the following are some exemplary illustrations:
“I am eternally grateful for everything my teacher has taught me, thanks for all the hard work.”

“The teacher looked very nice, when the teacher gave us the lessons, he or she taught in a very organized manner so we can understand easily.”

However, a few students commented that their instructors were a little bit dull and not able to perform effective classroom management. Nevertheless, most of the students were satisfied with the instructors’ performance.

Benefits of the Program

As reflected in the students’ diaries, the program benefited them in several aspects. Generally speaking, the program expanded the scope of students’ knowledge and promoted their holistic development. The students’ positive changes and their improvement as a result of joining the program were presented in terms of four domains as mentioned before, including the intrapersonal, interpersonal, familial, and societal domains.

Benefits in the intrapersonal domain

Students had notable growth in a wide range of intrapersonal dimensions after joining the program. A significant number of students indicated improvement in their self-concept, self-awareness, self-understanding, and self-confidence. Many students wrote that they were not a confident person and afraid to stand out before participating in the project. However, after engaging in different interactive activities of the program, they became more comfortable and confident to interact with others and express their own opinions in front of their classmates. They also realized their own shortcomings (e.g., bad temper and disrespectful to others) and gained improvement with the help from the instructors and classmates. Some diaries showing benefits in this domain are depicted below:
“I had more confidence on myself. I did not dare to answer questions and speak loud in the class before, but I am different now, I speak louder to answer questions and I’m braver than before.”

“I became more confident and had a better understanding of my personal strengths and weaknesses.”

The students also had a higher level of resilience when facing adversity after joining the program. Some students remarked that they learnt to be optimistic and not to give up easily when facing difficulties. Furthermore, they learnt to be more grateful for everything they owned and explained that it is of the most importance to have a thankful heart. The following diary reflects one student’s awareness of cherishing one’s life:
“Some poor children living in rural areas should have the chance to go to school joyfully and study in a clean and tidy classroom, but they can only work hard in order to survive. Therefore, I think we should cherish our life. When we face difficulties, we should think of those poor children.”

The students also gained improvement in cognitive, emotional, and moral competences. Comprehensive group discussion and self-reflection activities embedded in the program allowed the students to think in a more rational, analytical, and critical way from different perspectives, which greatly enhanced their problem-solving skills. For example, some students elaborated that when facing a problem, they would stay calm, analyze the program thoroughly, and try to consider different solutions. Furthermore, many students mentioned that they learnt to control their temper and utilize different strategies to manage emotions such as talking to friends and writing diary. The students also highlighted their growth regarding moral competence and virtues. The students’ development in intrapersonal competences can be evidenced by the following examples:
“It taught us how to distinguish right and wrong. Not everything is going smoothly in our life but the teachers led us to do the right things, which inspired us a lot.”

“I realized that my emotions cannot be easily controlled. Because of the program, I can control my emotions, sometimes I had a good time in the class, which made me feel more relax and positive. Because of the program, we are able to stay positive.”

Last but not least, students learnt to do life planning. Some students expressed that the program had inspired them to think about their purposes of life and their values and beliefs. Guided by the instructors, the students set life goals and made plans to achieve their dreams:
“I have learnt to think more about my life purposes and dreams, also learnt to do planning for my life.”

“The program has taught me about some successful experience. Everyone has his or her own definition of success, we do not need to compare with others, just set our own goals which are attainable and meaningful.”

Benefits in the interpersonal domain

The students perceived that the program enhanced their interpersonal competence and strengthened team spirit among classmates. Primarily, communication among the students was promoted by the rich in-class activities and the pleasant and interactive learning environment. As such, the students had more opportunities to get to know and understand each other in depth, which facilitated effective teamwork and ultimately established a sense of belonging and team cohesion. Moreover, through participating in the program, the students learnt to respect others who have different opinions, be trustworthy and sincere to others. As a result, the students felt that they gradually established good friendship with peers characterized by mutual understanding and trust. Benefits perceived by the students in the interpersonal domain can be shown by the following narratives:
“After participating in the program, I made a lot more friends and got along with them so well.”

“I learnt how to communicate with others and I learnt that I need to ‘change my way of thinking’ and try to be more considerate of others.”

Benefits in the familial domain

A considerable number of students indicated that the program improved their competence in communicating with family members and dealing with issues within family. For example, some students illustrated that the program enabled them to understand their parents in an empathetic way and express gratitude to their parents. As being more capable of “putting themselves in their parents’ shoes,” the students had better relationship and less conflicts with their parents at home. Furthermore, the students were more willing to show their filial piety to their parents as a way to respect and return parents’ dedication in the family. Examples below demonstrate the benefits in the familial domain:
“I have never told my dad and mum that I love them. But after taking the program, I told my father and mother that I love them. I could not imagine their facial expression, but it seemed like the taste of marshmallow, a sweet taste.”

“This program taught me how to communicate with people, let me know what is empathy. I learnt not to lose my temper in getting along with my parents when I was in adolescent rebellious stage, and to think about my own behaviors from my parents’ perspective.”

Benefits in the societal domain

In this domain, the two primarily benefits perceived by the students were increased awareness of social norms and willingness to take social responsibility. Some students stated that they understood more about the negative impacts of behaviors not in line with social and moral standards. Due to the beneficial influence of the program, the students not only behaved in a good manner but also developed a higher sense of responsibility to serve their hometown and the country. For example, students wrote,
“What we have learnt in the program were very practical because it taught us the dos and don’ts, and it developed my sense of responsibility to serve the society.”

“I learnt to protect the environment after participating in the program. Some people like littering and throwing rubbish in the river causing water pollution, but it is very bad. We should protect our environment together and should stop other citizens from littering.”

Discussion and Applications for Practice

Rigorous evaluation including multiple strategies is a hallmark of the P.A.T.H.S. project implemented in Hong Kong (Shek, 2013). Adapted from this effective program, the “TKP P.A.T.H.S. project” in mainland China is also featured with scientific evaluation incorporating subjective outcome evaluation and qualitative evaluation approaches. In the present study, student diary was used as a qualitative technique to understand the in-depth views of the program participants. The present pioneer qualitative evaluation study in mainland China adopted a general postpositivistic qualitative orientation.

Specifically, the present study possesses the following critical attributes with reference to the principles highlighted in prior literature addressing reliability and validity of qualitative research (e.g., Miles, Huberman, & Saldaña, 2014; Shek et al., 2005). First, details of the program implementation; participants including sample size, their grades, sampling method; and the justifications for the number of participants were documented. In addition, the procedures of data collection and analyses were fully described. By reporting these details, an audit trial can be formed. Second, intra- and interrater reliability were calculated. Third, data were collected in a nonartificial way within a natural setting. Fourth, school teachers instead of the researchers collected the data, minimizing the impacts of biases and expectations of the researchers. Finally, multiple researchers were involved and both qualitative and quantitative approaches were employed in data analyses.

Based on the students’ diaries, the present study showed that the participants generally held positive views toward the program, the instructors, and the program benefits. The present findings echo the positive evaluation findings for the P.A.T.H.S. project in Hong Kong based on different evaluation strategies (Shek & Sun, 2013; Shek & Yu, 2013). The findings are also in line with the aforementioned quantitative findings based on the subjective outcome evaluation for the TKP P.A.T.H.S. project in mainland China (Shek, Wu, & Law, in press; Shek, Zhu, & Leung, 2018). As such, the present study provides additional evidence supporting the effectiveness of the project in mainland China. The consistency between the present findings and those obtained from other evaluation methods is also a sign showing the validity of the positive findings. In short, the effectiveness of the project is supported by evaluation findings obtained from different informants (e.g., implementers and participants) and through different strategies (e.g., quantitative and qualitative methods).

Several observations can be highlighted from the present findings. First, the students enjoyed participating in the project and appreciated the learning experience in the classes with their classmates and the instructors. Besides, the program incorporated a student-centered teaching pedagogy and used multiple interactive and reflective in-class activities to enhance students’ learning engagement. Second, the most frequently mentioned growth or change after participating in the program is the improvement in social and emotional competences, which has helped students build positive relationships with peers, teachers, and family members. This kind of benefits is particularly important for adolescents (Domitrovich, Durlak, Staley, & Weissberg, 2017), who undergo various changes (e.g., physiological and psychological changes as a result of puberty) and challenges (e.g., academic and peer pressures).

Third, the students have positive views toward the instructors. It is true that instructors in mainland China did not have much experience in performing student-centered teaching. While most of the instructors adapted effectively to the new teaching philosophy and acquired some experiential teaching skills through attending training workshops, a small proportion of the instructors failed to do so. Given the importance of the instructors for effective program implementation (Shek & Sun, 2008), there is a need to help these instructors shift their traditional “spoon-feeding” teaching approaches to interactive ones. To achieve this goal, more effective training workshops need to be provided as a platform for the instructors to observe, reflect, learn, and practice (Shek, Leung, & Wu, in press; Shek, Zhu, & Leung, 2017).

There are several implications of the present findings to social work practice. First, the present study underscores the value of PYD programs in promoting adolescent psychosocial competences. In the PYD literature, there are studies showing that adolescents equipped with social and emotional competences will be able to understand and manage emotions, set goals, make and carry out the plans to achieve the goals, make wise decisions and take responsibilities, show respect and empathy to others, and establish and maintain positive social relationships, all of which are critical for manifesting resilience and healthy development people (Domitrovich et al., 2017; Durlak, Domitrovich, Weissberg, & Gullotta, 2015). The present findings reinforce the usefulness of PYD programs.

Second, the present study also underscores the advantages of using student diaries to reveal views of program participants and investigate program effectiveness. First of all, writing diaries is more economic and convenient compared with other evaluation methods such as survey or interview. By writing diary, the respondents have a chance to revisit their learning experience and share their most significant impressions of the program. Because the participants have freedom to express their perceptions in their favorite time and environment without any pressures, researchers obtain not only rich and in-depth data from diaries but also the most reliable inner subjective experience of the participants. Therefore, diary writing can be easily used by social work researchers to triangulate evaluation findings from other evaluation means and suggest explanations for quantitative evaluation findings.

Of course, one possible query of using student diaries is whether the related findings are valid. According to LeCompte and Goetz (1982) and Miles, Huberman, and Saldaña (2014), internal validity refers to whether the observations are “authentic” representation of the reality, whereas external validity asks whether the constructs can be generalized across groups. Regarding internal validity, as the reflections are anonymous and there is trust between the teachers and students, internal validity can be considered to be high in the present study. Concerning external validity, as the participants were randomly selected, the probability that the related findings could be generalized to different groups is also high.

Finally, the current research demonstrates the positive impact of the school-based PYD program in mainland China where validated youth programs are grossly inadequate (Shek & Yu, 2011). With rapid economic development and growth of social issues in young people, the Chinese government has set a goal of training 3 million professional social workers by 2020 (Central Committee of the Communist Party of China & the State Council of the People’s Republic of China, 2010), aiming to promote social work practice to help address the issues and establish a health society. To promote the development of social work in China, launching PYD programs such as the TKP P.A.T.H.S. project can help social workers, particularly school social workers, to promote holistic development in adolescents in the school context and demonstrate the value of social work.

Despite its pioneer nature in China and in the field of PYD programs, this study has several limitations. First, as the students expressed their views in a one-way manner, it was not possible to have interactions with the informants to further clarify some of their views and explore reasons behind their views. Second, as the students needed to write down their perceptions, some students might not be able to express their views clearly and correctly due to the lack of writing skills, thus they only wrote very little. Nonetheless, this is not a big issue in the present study as such cases were very few and most of the participants wrote rich content in their diaries.

Third, alternative explanations of the present positive findings should be considered. For example, students may consciously express positive views to show their nice acts and cooperation, which may result in their teachers’ praise. However, the students did not need to take examinations in the P.A.T.H.S. classes, and their performance in and their evaluation of the program do not affect their grades in any aspects of academic study in the school. Thus, the students’ motivation to please their teachers by consciously giving favorable feedback for the program is not high. Besides, the students wrote in an anonymous way, and they were specifically encouraged by their teachers to express their views honestly. Thus, we believe that the problem of desirability bias is not strong in the present study. In fact, negative comments were also recorded. Despite these limitations, the present qualitative findings provide additional support for the effectiveness of the curricula-based PYD programs implemented in China mainland.

Grade	Total Number	Positive	Negative	Neutral	Undecided
7	758	732	96.57	12	1.58	14	1.85
8	92	90	97.83	0	0.00	2	2.17
9	9	9	100.00	0	0.00	0	0.00
Total	859	831	96.74	12	1.40	16	1.86

Grade	Total Number	M	SD	1 = Strongly Negative	2 = Moderately Negative	3 = Slightly Negative	4 = Neutral	5 = Slightly Positive	6 = Moderately Positive	7 = Strongly Positive
7	758	6.72	.81	2	0.26	6	0.79	4	0.53	14	1.85	16	2.11	79	10.42	637	84.04
8	92	6.78	.63	0	0.00	0	0.00	0	0.00	2	2.17	4	4.35	6	6.52	80	86.96
9	9	6.78	.44	0	0.00	0	0.00	0	0.00	0	0.00	0	0.00	2	22.22	7	77.78
Total	859	6.73	.79	2	0.23	6	0.70	4	0.47	16	1.86	20	2.33	87	10.13	724	84.28

Footnotes

Authors’ Note

This study was conducted with the approval from the ethics review from the Human Subjects Ethics Subcommittee of The Hong Kong Polytechnic University.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research and the Project P.A.T.H.S. in Mainland China are financially supported by Tin Ka Ping Foundation.

References

Atieno

O. P.

(2009). An analysis of the strengths and limitation of qualitative and quantitative research paradigms. Problems of Education in the 21st Century, 13, 13–18.

Beauchamp

(2006). Understanding reflection in teaching: A framework for analyzing the literature (doctoral dissertation). Montreal, Canada: McGill University.

Catalano

R. F.

Berglund

M. L.

Ryan

J. A. M.

Lonczak

H. S.

Hawkins

J. D.

(2004). Positive youth development in the United States: Research findings on evaluations of positive youth development programs. The Annals of the American Academy of Political and Social Science, 591, 98–124. doi:10.1177/0002716203260102

Catalano

R. F.

Fagan

A. A.

Gavin

L. E.

Greenberg

M. T.

Irwin

C. E.

Jr Ross

D. A.

Shek

D. T. L.

(2012). Worldwide application of prevention science in adolescent health. The Lancet, 379, 1653–1664. doi:10.1016/S0140-6736(12)60238-4

Central Committee of the Communist Party of China, & the State Council of the People’s Republic of China. (2010). 国家中长期人才发展规划纲要(2010-2020) [National medium- and long-term talent development plan (2010–2020)]. Beijing. Retrieved from http://www.mohrss.gov.cn/SYrlzyhshbzb/zwgk/ghcw/ghjh/201503/t20150313_153952.htm

Choy

L. T.

(2014). The strengths and weaknesses of research methodology: Comparison and complimentary between qualitative and quantitative approaches. IOSR Journal of Humanities and Social Science, 19, 99–104.

Cohen

D. J.

Leviton

L. C.

Isaacson

Tallia

A. F.

Crabtree

B. F.

(2006). Online diaries for qualitative evaluation: Gaining real-time insights. American Journal of Evaluation, 27, 163–184. doi: 10.1177/1098214006288448

Domitrovich

C. E.

Durlak

J. A.

Staley

K. C.

Weissberg

R. P.

(2017). Social-emotional competence: An essential factor for promoting positive adjustment and reducing risk in school children. Child Development, 88, 408–416. doi:10.1111/cdev.12739

Durlak

J. A.

Domitrovich

C. E.

Weissberg

R. P.

Gullotta

T. P.

(Eds.). (2015). Handbook of social and emotional learning: Research and practice. New York: Guilford Press.

10.

Fraser

M. W.

(2016). Measures of consumer satisfaction in social welfare and behavioral health: A systematic review. Research on Social Work Practice, 26, 762–776. doi:10.1177/1049731514564990

11.

Grinnell

R. M.

Jr Gabor

P. A.

Unrau

Y. A.

(2015). Program evaluation for social workers: Foundations of evidence-based programs (7th ed.). New York: Oxford University Press.

12.

Hasson

Blomberg

Dunér

(2012). Fidelity and moderating factors in complex interventions: A case study of a continuum of care program for frail elderly people in health and social care. Implementation Science, 7, 23. doi:10.1186/1748-5908-7-23

13.

Holloway

Biley

F. C.

(2011). Being a qualitative researcher. Qualitative Health Research, 21, 968–975. doi:10.1177/1049732310395607

14.

LeCompte

M. D.

Goetz

J. P.

(1982). Problems of reliability and validity in ethnographic research. Review of Educational Research, 52, 31–60. doi:10.3102/00346543052001031

15.

Lerner

J. V.

Bowers

E. P.

Minor

Boyd

M. J.

Mueller

M. K.

Schmid

K. L.

… Lerner

R. M

(2013). Positive youth development: Processes, philosophies, and programs. In Lerner

R. M.

Easterbrooks

M. A.

Mistry

Weiner

I. B.

(Eds.), Handbook of psychology: Developmental psychology (Vol. 6, pp. 365–392). Hoboken, NJ: John Wiley & Sons.

16.

Lichtman

(2013). Qualitative research for the social sciences. Thousand Oaks, CA: Sage.

17.

C. M. S.

Shek

D. T. L

. (2017). Objective outcome evaluation of a positive youth development program: The project P.A.T.H.S. in Hong Kong. Research on Social Work Practice. doi:10.1177/1049731517711246

18.

Mak

D. B.

Plant

A. J.

Toussaint

. (2006). “I have learnt … a different way of looking at people’s health”: An evaluation of a prevocational medical training program in public health medicine and primary health care in remote Australia. Medical Teacher, 28, e149–e155. doi:10.1080/01421590600776404

19.

Mesibov

G. B.

Shea

(2010). The TEACCH program in the era of evidence-based practice. Journal of Autism and Developmental Disorders, 40, 570–579. doi:10.1007/s10803-009-0901-6

20.

Miles

M. B.

Huberman

A. M.

Saldaña

(2014). Qualitative data analysis: A methods sourcebook (3rd ed.). Washington, DC: Sage.

21.

Patton

M. Q.

(2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

22.

Patton

M. Q.

(2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

23.

Pavlovich

Collins

Jones

(2009). Developing students’ skills in reflective practice: Design and assessment. Journal of Management Education, 33, 37–58. doi:10.1177/1052562907307640

24.

Robson

(2000). Small-scale evaluation: Principles and practice. Thousand Oaks, CA: Sage.

25.

Schmitz

Wiese

B. S.

(2006). New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data. Contemporary Educational Psychology, 31, 64–96. doi:10.1016/j.cedpsych.2005.02.002

26.

Shek

D. T. L.

(2010). Using students’ weekly diaries to evaluate positive youth development programs: Are findings based on multiple studies consistent? Social Indicators Research, 95, 475–487. doi:10.1007/s11205-009-9532-8

27.

Shek

D. T. L.

(2013). Evaluation of the project P.A.T.H.S. using multiple evaluation strategies. In Shek

D. T. L.

Sun

R. C. F.

(Eds.), Development and evaluation of Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.) (pp. 53–67). Singapore, Singapore: Springer.

28.

Shek

D. T. L.

(2014). Is subjective outcome evaluation related to objective outcome evaluation? Insights from a longitudinal study in Hong Kong. Journal of Pediatric and Adolescent Gynecology, 27, S50–S56. doi:10.1016/j.jpag.2014.02.012

29.

Shek

D. T. L.

Han

X. Y.

Lee

T. Y.

(2013). Impact of the project P.A.T.H.S. in China. In Shek

D. T. L.

Sun

R. C. F.

(Eds.), Development and evaluation of Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.) (pp. 291–304). Singapore, Singapore: Springer.

30.

Shek

D. T. L.

Han

X. Y.

Lee

T. Y.

(2014). Subjective outcome evaluation of a positive youth development program in China. International Journal on Disability and Human Development, 13, 275–283. doi:10.1515/ijdhd-2014-0313

31.

Shek

D. T. L.

Han

X. Y.

H. K.

(2009). 雙城記:滬港青少年正面成長計劃分享錄 [Experiences of joining the positive youth development program: Narratives from Shanghai and Hong Kong]. Shanghai, China: Academia Press.

32.

Shek

D. T. L.

Lee

T. Y.

L. K

. (in press). Subjective outcome evaluation of the Tin Ka Ping P.A.T.H.S. project in China: View of the students. International Public Health Journal.

33.

Shek

D. T. L.

Leung

J. T. Y.

. (in press). Subjective evaluation of a training program of the Tin Ka Ping P.A.T.H.S. project in mainland China. International Public Health Journal.

34.

Shek

D. T. L.

Sun

R. C. F.

(2008). Implementation of a positive youth development program in a Chinese context: The role of policy, program, people, process, and place. The Scientific World Journal, 8, 980–996. doi:10.1100/tsw.2008.120

35.

Shek

D. T. L.

Sun

R. C. F.

(2012a). Evaluation of the project P.A.T.H.S. based on students’ weekly diaries: Findings from eight datasets. The Scientific World Journal, 2012, 354254. doi:10.1100/2012/354254

36.

Shek

D. T. L.

Sun

R. C. F.

(2012b). Participants’ evaluation of the project P.A.T.H.S.: Are findings based on different datasets consistent? The Scientific World Journal, 2012, 187450. doi:10.1100/2012/187450

37.

Shek

D. T. L.

Sun

R. C. F.

(Eds.). (2013). Development and evaluation of Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.). Singapore, Singapore: Springer.

38.

Shek

D. T. L.

Sun

R. C. F.

Lam

C. M.

Lung

D. W.

S. C.

(2008). Evaluation of project P.A.T.H.S. in Hong Kong: Utilization of student weekly diary. The Scientific World Journal, 8, 13–21. doi:10.1100/tsw.2008.2

39.

Shek

D. T. L.

Tang

V. M. Y.

Han

X. Y.

(2005). Evaluation of evaluation studies using qualitative research methods in the social work literature (1990–2003): Evidence that constitutes a wake-up call. Research on Social Work Practice, 15, 180–194. doi:10.1177/1049731504271603

40.

Shek

D. T. L.

F. K. Y.

(2016). The project P.A.T.H.S. in Hong Kong: Work done and lessons learned in a decade. Journal of Pediatric and Adolescent Gynecology, 29, S3–S11. doi:10.1016/j.jpag.2015.10.003

41.

Shek

D. T. L.

Law

M. Y. M

. (in press). Subjective outcome evaluation of the Tin Ka Ping P.A.T.H.S. project: Views of senior secondary school students. International Journal of Child and Adolescent Health.

42.

Shek

D. T. L.

(2011). A review of validated youth prevention and positive youth development programs in Asia. International Journal of Adolescent Medicine and Health, 23, 317–324. doi:10.1515/IJAMH.2011.028

43.

Shek

D. T. L.

(2013). Impact of the project P.A.T.H.S. on adolescent risk behavior: A five-year longitudinal study. In Shek

D. T. L.

Sun

R. C. F.

(Eds.), Development and evaluation of Positive Adolescent Training through Holistic Social programs (P.A.T.H.S.) (pp. 85–106). Singapore, Singapore: Springer.

44.

Shek

D. T. L.

Sun

R. C. F.

Lee

T. Y.

Han

X. Y.

X. X.

Zhao

(2014). Objective outcome evaluation of a positive youth development program in China. International Journal on Disability and Human Development, 13, 255–265. doi:10.1515/ijdhd-2014-0311

45.

Shek

D. T. L.

Zhu

Leung

J. T. Y

. (2018). Evaluation findings of Tin Ka Ping P.A.T.H.S. project implemented in junior secondary schools: Implementers’ views. International Journal of Child and Adolescent Health, 1(1).

46.

Shek

D. T. L.

Zhu

Leung

J. T. Y

. (2017). Subjective outcome evaluation of the Tin Ka Ping P.A.T.H.S. project training program in mainland China. International Journal of Child and Adolescent Health, 10, 201–211.

47.

Siu

A. M. H.

Chan

C. C. H.

Poon

P. K. K.

Chui

D. Y. Y.

Chan

S. C. C.

(2007). Evaluation of the chronic disease self-management program in a Chinese population. Patient Education and Counseling, 65, 42–50. doi:10.1016/j.pec.2006.04.013

48.

Swim

J. K.

Hyers

L. L.

Cohen

L. L.

Ferguson

M. J.

(2001). Everyday sexism: Evidence for its incidence, nature, and psychological impact from three daily diary studies. Journal of Social Issues, 57, 31–53. doi:10.1111/0022-4537.00200

49.

Travers

(2011). Unveiling a reflective diary methodology for exploring the lived experiences of stress and coping. Journal of Vocational Behavior, 79, 204–216. doi:10.1016/j.jvb.2010.11.007

50.

Travers

C. J.

Morisano

Locke

E. A.

(2015). Self-reflection, growth goals, and academic outcomes: A qualitative study. British Journal of Educational Psychology, 85, 224–241. doi:10.1111/bjep.12059

51.

Wolcott

H. F.

(1994). Transforming qualitative data: Description, analysis, and interpretation. Thousand Oaks, CA: Sage.

52.

World Health Organization. (2016). Inspire: Seven strategies for ending violence against children. Geneva, Switzerland. Retrieved from http://apps.who.int/iris/bitstream/10665/207717/1/9789241565356-eng.pdf?ua=1