Building Program Evaluation Capacity of Non-Governmental Organizations Practitioners in Hong Kong

Abstract

Purpose: Despite the growing interest in program evaluation capacity building (PECB), its impacts have been rarely studied in the existing literature. This study aims to evaluate a PECB initiative in Hong Kong called the Jockey Club MEL Institute Project. Methods: A pretest–posttest quasi-experimental design was adopted to compare the changes between the training (N = 226) and comparison groups (N = 213). Focus groups with training participants, mentors, and trainers were also conducted. Results: The quantitative results reported that the training group showed greater improvements in all three core aspects of program evaluation capacity—evaluation mindset, evaluation implementation, and evaluation communication—than the comparison group. The qualitative findings further indicated how NGO practitioners benefited from the project. Discussion: The findings provide support for the positive impacts of PECB on NGO practitioners. As such, implications for researchers and evaluators in effectively conducting PECB activities are also discussed.

Keywords

program evaluation capacity building non-governmental organization Hong Kong

Facilitating the capacity of non-governmental organizations (NGOs) to conduct program evaluations has been recognized as an undeniably critical issue, especially since NGOs are under pressure to be accountable to funding agencies and promote program improvements (García-Iriarte, Suarez-Balcazar, Taylor-Ritzler, & Luna, 2010; Kettner, Moroney, & Martin, 2016). Program evaluation refers to a broad range of activities in the NGO sector to systematically investigate the effectiveness of social intervention projects and designed to assess organizational performance, meet the needs of diverse stakeholders, and promote project improvements (Suárez & Marshall, 2014). Importantly, it is worth noting that program evaluation consists of more than evaluating final outcomes but also focuses on the need for a program (needs assessment), its design (assessment of program theory and design), and its operation and service delivery (monitoring the process) (Mitchell, 2017; Rossi, Lipsey, & Henry, 2019). As most NGOs receive a significant portion of their funding from external sources, program evaluation has been advocated as an essential strategy for NGOs to demonstrate the effectiveness and social impacts of their services to funding agencies (Humphries, Gomez, & Hartwig, 2011). Furthermore, neoliberal ideology, with its emphasis on “accountable and countable,” puts extra stress on NGOs to be seen as accountable organizations for them to survive in the marketization of social services under globalization (Burton & van den Broek, 2009). Presenting the outcome through program evaluation becomes one of the accounting strategies that would best satisfy neoliberal ideology (McCoy, Rose, & Connolly, 2014; Yan, Cheung, Tsui, & Chu, 2015). Altogether, program evaluation practices have become increasingly common in the NGO sector to attract potential funders and enhance competitive advantages (García-Iriarte et al., 2010). Additionally, researchers have noticed that growing numbers of NGOs are embracing the potential of program evaluation as an important means for improving program performance, in that NGOs proactively use evaluation results to make programs more evidence-based and effective (Suárez & Marshall, 2014).

Despite the growing emphasis on program evaluation, the capacity of NGOs to undertake program evaluation has not kept pace with this trend in the literature (Carman & Fredericks, 2008). Therefore, program evaluation capacity building (PECB) has captured worldwide interest as it has been proposed as an effective approach to strengthen NGOs’ capacity for conducting rigorous program evaluation practices (García-Iriarte et al., 2010; Preskill & Boyle, 2008a). Although various PECB programs have mushroomed worldwide, research on the effects of PECB is still nascent (Morkel & Ramasobama, 2017). In particular, scholars have called for additional research that comprehensively describes the process of building program evaluation capacity and the impacts of such efforts (García-Iriarte et al., 2010). In recognition of this research gap, this study seeks to contribute to the research agenda by presenting a mixed-method evaluation of a PECB initiative in Hong Kong called the Jockey Club Monitoring, Evaluation, and Learning (MEL) Institute Project (hereafter the MEL project) to build three core aspects of program evaluation capacity among NGO practitioners: evaluation mindset, evaluation implementation, and evaluation communication.

Program Evaluation Capacity of NGO Practitioners

The capacity for program evaluation has been increasingly prioritized at the core of the capacity developments of NGOs (Hailey & James, 2003; Suárez & Marshall, 2014). Scholars have defined the program evaluation capacity of NGO practitioners as the ability to conduct effective evaluation of their service programs, using results generated from the evaluation to further improve their service delivery and organizational performance (e.g., Preskill & Boyle, 2008a). Importantly, growing research has suggested that evaluation capacity is multi-dimensional and extends beyond specific aspects (e.g., Doherty, Eccleston, Hansen, Natalier, & Churchill, 2015). For most NGOs, the dimension of capacities for program evaluators often underlines the knowledge of selecting or developing methods for evaluating programs (Brown & Kelsey, 2016) or the extent to which NGO practitioners are positive about using program evaluation principles and practices (Preskill & Boyle, 2008a). Apart from the preceding mindset dimension (Bourgeois & Cousins, 2013), scholars have pointed out that program evaluation capacity should also include the ability to transfer individual learning and expertise in program evaluation knowledge and skills into organizational evaluation processes and practices, that is, the dimension of evaluation implementation (Taylor-Ritzler, Suarez-Balcazar, Garcia-Iriarte, Henry, & Balcazar, 2013). Besides, to respond to the increasing expectations for NGOs to visualize and expand the social impacts of their projects, evaluation communication has recently been suggested as an important aspect of evaluation capacity and emphasizes NGO practitioners’ ability to leverage communication channels and opportunities to collect and disseminate evaluation information (Mitchell, 2017).

According to existing literature, numerous capacity deficits for undertaking program evaluation exercises have been identified in NGO sector. For instance, it has been found that NGOs that struggle the most with program evaluation commonly lack systematic evaluative knowledge (Carman & Fredericks, 2008). Furthermore, NGOs often suffer from a dearth of staff members with evaluation experience; such a lack is particularly challenging for smaller NGOs as they have limited funding to hire external evaluation experts for assistance (Andrews, Motes, Floyd, Flerx & Fede, 2005). For NGO practitioners, it is typical for them to find program evaluation procedures difficult to understand and implement, especially when they are not familiar with the evaluation frameworks and tools (Kegeles, Rebchook, & Tebbetts, 2005). NGO practitioners have also frequently reported that use of program evaluations, driven by external pressures, has increased their level of frustration and despair (Bornstein, 2006). Furthermore, as NGO culture tends to emphasize “doers” more than “evaluators,” adverse attitudes held by staff members toward program evaluation have been found to negatively impact the program evaluation process (Leviton, 2013).

Collectively, it is clear that solely underscoring one dimension is not sufficient for a PECB program to effectively fulfill its mission, and three core aspects of evaluation capacity have been identified for PECB interventions: evaluation mindset, evaluation implementation, and evaluation communication (Bourgeois & Cousins, 2013; Preskill & Boyle, 2008a; Taylor-Ritzler et al., 2013). Hence, built upon extant studies, the following definition of program evaluation capacity is constructed for the present MEL project to promote NGO practitioners’ holistic evaluation capacity: namely, program evaluation capacity refers to multidimensional competencies for NGO practitioners to conduct effective program evaluation, which involves evaluation mindset, evaluation implementation, and evaluation communication.

Contextualizing NGO Program Evaluation in Hong Kong

First, like many other parts of the world, the neoliberal discourse of accountability has been deeply embedded in the social service fields in Hong Kong, with the introduction of the Lump Sum Grant Subvention System (LSGSS) being the foremost piece of evidence (Tsui, 2004; Yan et al., 2015). In 2001, embracing the neoliberal ideology and its principles of marketization and competition, the Hong Kong government adopted the LSGSS, a subvention reform package through which the government allocates resources to NGOs to provide “more efficient, customer-focused, accountable and output-driven welfare services” (Social Welfare Department of Hong Kong, 2021). Based on the logic of competition and cost-efficiency, LSGSS has also induced a business-like model in the social service sector by generating a competitive contract culture (Lam & Nie, 2020). Under LSGSS, the marketization of social services and contracting out government services increase challenges for NGOs in Hong Kong, since there is the growing uncertainty surrounding NGOs’ financial sustainability. To survive, NGOs must strive for revenue stability and bid for new government contracts. More importantly, it is under the performance-based funding system of LSGSS that the standard for being a valuable program is largely based on performance indicators regarding whether or not to achieve the targeted service outputs and outcomes (Nip, 2010). Also, according to the survey of Hong Kong Jockey Club Charities Trust, 76% of their grantees believed that systematic monitoring and evaluation added value to NGO projects and increased their social impacts (Hong Kong Jockey Club Charities Trust, 2017).

Consequently, as NGOs in Hong Kong heavily depend on government subsidies and external funding agencies, presenting the outcomes and proving accountability through program evaluations becomes an increasingly important strategy for them in fundraising. For example, NGOs have used homegrown program evaluation frameworks, such as the Social Impact Measurement Model (Kwan, Kee, Chan, & Ng, 2016), or assessment tools from the business sector for program evaluation, including social return on investment (SROI) (Cheung, 2016). However, barriers to conducting program evaluation for NGOs in Hong Kong are never negligible. In Hong Kong, program evaluation still serves as a substantial obstacle that poses challenges to NGO staff, especially in terms of survey design, measurement tools, and its perceived helpfulness; NGOs, as a whole, possess a greatly limited pool of knowledge and resources to carry out service and program evaluation (Hong Kong Jockey Club Charities Trust, 2018).

Conceptual Framework of the MEL Project

Given the considerations mentioned above, including the multidimensional aspects of program evaluation capacity, the evaluation capacity deficits of NGO practitioners, and the complex challenges to NGOs' survival and fundraising, it is necessary for PECB practices to establish a comprehensive framework that contributes to building NGO practitioners’ multidimensional competencies for conducting program evaluation. One possible framework is the effectiveness-based framework, which has been suggested as helpful in facilitating NGO practitioners in developing a thorough understanding of effectiveness principles in program design and evaluation (Kettner et al., 2016; Whatley, 2013). Specifically, the effectiveness-based framework provides the step-by-step process necessary for a program to reach effectiveness, which mainly includes: (1) producing a clear understanding of social problems through problem analysis and needs assessment; (2) planning and designing the program with the best available evidence; (3) monitoring the program and conducting periodic checkups; (4) evaluating the program and determining the short- and long-term outcomes; and (5) disseminating and sharing program evaluation results with different stakeholders (Kettner et al., 2016). This framework allows NGO practitioners to see the rational flow of addressing a problem to accomplish social impacts while maintaining a focus on recording the data necessary for performance measurement and program evaluation (Grinnell, Gabor, & Unrau, 2019).

As the effectiveness-based framework emphasizes the effectiveness of program evaluation through promoting ongoing monitoring, rigorous evaluation, and holistic learning, it can assist the MEL project in building three core aspects of program evaluation capacity reviewed above: evaluation mindset, evaluation implementation, and evaluation communication. First, this framework includes a full range of learning about the logical and conceptual sequences of program-evaluation planning. Thus, it can facilitate the cultivation of the evaluation mindset in participants, namely, being familiar with the methods of program evaluation and holding positive attitudes about using them (Kegeles et al., 2005; Morkel & Ramasobama, 2017). Second, the effectiveness-based framework provides a clear flow of the phases of the program evaluation, allowing NGO practitioners who might lack evaluative knowledge and related experience to follow and operationalize the framework, which, in turn, would promote their capacity of evaluation implementation (García-Iriarte et al., 2010). Third, it emphasizes incorporating the dissemination and sharing of program evaluation results with diverse stakeholders, which could improve the capacity of evaluation communication among participants (Preskill & Boyle, 2008a).

Practical Framework of the MEL Project

As indicated above, the “MEL” in the present project signifies Monitoring, Evaluation, and Learning, which represents a wide spectrum of activities involved in program evaluation (Mitchell, 2017). Built on the effectiveness-based framework, the MEL project incorporated multiple effective approaches, including a certificate training course, a follow-up mentored practicum, an e-learning knowledge hub, and a final pitching day.

First, a certificate training course was developed to enable participants to comprehensively acquire knowledge, skills, and attitudes conducive to effective implementation of program evaluation. The course incorporated a wide range of elements proposed by the effectiveness-based framework, systematically covering four major areas: “service development and monitoring,” “resource and planning management,” “media and communications,” and “program evaluation and impact assessment,” comprising 14 sessions. Local and overseas experts from universities and internationally renowned institutions in program evaluation were invited to serve as trainers. The area “service development and monitoring” provided evaluation tools such as the logic model for participants to conduct problem analysis and needs assessment, review their existing services, and formulate new services. After the lessons, participants could have a clear picture of the inputs, activities, expected outputs, and outcomes of their projects, which are a premise of conducting program evaluation. Subsequently, the area “resource and planning management” covered topics regarding institutional arrangements for gathering and managing project data, such as establishing a data management system. Participants could acquire practical knowledge and skills to collect and manage data effectively, thereby improving the monitoring and evaluation of their projects. In addition, to help participants promote the outcome/impact dissemination of their projects, public presentation skills and digital storytelling techniques in sharing evidence-based practices were provided in the “Media and Communications” section. Participants also learned how to use social media and the internet in participant recruitment and data collection. The last course, “program evaluation and impact assessment,” offered step-by-step training for conducting program evaluation, covering topics including research designs for outcome/impact program evaluation, sampling and data collection, measurement tools, and using statistics in program evaluation and impact assessment.

Second, a mentored practicum, relying on the logic of the effectiveness-based framework, was delivered immediately following the training to walk participants through transferring their acquired knowledge into practice within the organizations, thereby improving evaluation implementation capacity and increasing evaluation experiences. Every two groups of participants shared one mentoring group formed by one expert from academia and one experienced practitioner in the field of business or information technology. Over a 4-month period, participants regularly reported on the progress, challenges met, and successful experiences, and received ongoing and tailored coaching from mentors, such as how to use the logic model to design strategies and deploy resources for service implementation, or how to disseminate good practices for mutual learning in the NGO sector.

Third, an e-learning knowledge hub was built on the MEL website to increase NGO practitioners’ accessibility to e-learning resources of program evaluation, including web-based audio-visual learning materials in four areas consistent with the training scheme (Jockey Club MEL Institute Project, 2021), various validated measurement tools, and an online sharing space for participants. The flipped classroom was implemented among participants with the audio-visual learning materials, that is, they could preview the course content in the e-learning knowledge hub before the training course and review it after the course. They were invited to answer a set of multiple-choice questions before and after watching the videos, and explanations for the correct answers were provided to facilitate the learning progress. In particular, the project provided training to two cohorts of NGO practitioners, namely, Cohort 2019–2020 (hereafter Cohort 1) and Cohort 2020–2021 (hereafter Cohort 2). Due to the outbreak of the COVID-19 pandemic and social distancing measures in Hong Kong, Cohort 2 received the training and mentoring via the online Zoom platform while Cohort 1 accepted face-to-face training and mentoring. Training videos and related materials were also put on the e-learning knowledge hub so that participants (including Cohort 1) could assess the resources anytime through the MEL website and review what they had learned.

Finally, a final pitching day was prepared for Cohorts 1 and 2, respectively, to help participants gain hands-on experience in evaluation and further transfer their learning into practice. Participants were required to work in teams and design a rigorous program evaluation based on their services and to carry out the evaluation, if possible. A total of 10 winning teams were awarded MEL best practices and were sponsored to present their evaluation practices at international conferences.

Altogether, aiming to promote participants' holistic evaluation capacity, the practical framework of the MEL project has several unique features. First, the framework is progressive in nature: at the beginning, through the systematic training, participants gained more capacity for an evaluation mindset, that is, they became familiar with evaluation and understanding its benefits. Then, through ongoing and tailored coaching from mentors, participants were supported in engaging with evaluation practices and tackling challenging evaluation processes, which in turn developed their capacity for evaluation implementation. On a pitching day, the focus was placed on consolidating the outcomes through personally designing a program of evaluation. Second, the capacity for evaluation communication was continually emphasized in both training and mentoring sessions. Even more, the preparation and implementation of a final pitching day also fostered evaluation communication capacity in nature, that is, participants were required to showcase their projects publicly with learned communication skills. Finally, the components of the MEL project intersected and complemented one another to build the three core aspects of program evaluation capacity. For example, diverse measurement tools in the e-learning knowledge hub not only directly improved participants' ability to implement a rigorous evaluation but also indirectly promoted their evaluation mindset by strengthening their confidence in engaging with evaluation.

Method

Participants and Data Collection

Adopting the train-the-trainer approach, the project aims to motivate participating NGO staff members to serve as resource persons and put the acquired knowledge and skills into practice within their own organizations to achieve change at service and/or organizational levels. Accordingly, the project targeted staff members at supervisory or above grades in NGOs or social enterprises. The participants should have a bachelor’s degree and at least one year’s working experience at an NGO or in a related sector. Invitation letters were sent to NGOs in Hong Kong, and participants were nominated by their organizations with a written organizational agreement to release the participants from their jobs during work hours to join the project. A pretest–posttest quasi-experimental design was employed in the present study. Hence, each participant was asked to coordinate another staff member who had similar job duties but did not join the project to join the comparison group. Both the participants (training group) and their non-participating colleagues (comparison group) were invited at pretest and posttest to complete a self-report questionnaire. Ultimately, 439 NGO practitioners responded to our survey questionnaire before and after the project, with 226 in the training group and 213 in the comparison group. Specifically, 217 respondents were from Cohort 1, with 114 in the training group and 103 in the comparison group. In addition, there were 222 respondents in Cohort 2, with 112 in the training group and 110 in the comparison group.

Table 1 shows the socio-demographic characteristics of the participants. Regarding respondents in Cohort 1 in the training group, over half were female (63.2%), and the mean age was 40.55 years (SD = 8.22). With a multiple-option item, we also collected information about respondents’ service areas, which included family and child services (52.6%), community development (51.7%), and social disadvantage and social security (36.1%). In the comparison group, 68.0% were female, and the mean age was 37.14 years (SD = 8.55). The comparison group respondents also served in different NGO services, including family and child services (38.8%), community development (37.8%), and youth services (32.0%). For respondents in Cohort 2 in the training group, 62.5% were female, and the mean age was 38.50 years (SD = 8.45). In addition, 64.5% were female in the comparison group, with an average age of 36.65 years (SD = 9.02). Respondents in both the training group and comparison group served in different NGO services, including family and child services (50.9% and 26.3%, respectively), community development (49.1% and 37.3%, respectively), and rehabilitation and medical social services (38.4% and 33.6%, respectively). Overall, in the training group, 62.8% were female, and the mean age was 39.53 years (SD = 8.38). In addition, 66.2% were female in the comparison group, with an average age of 36.89 years (SD = 8.78). The scope of services provided by the training and comparison group respondents included family and child services (51.8% and 32.4%, respectively), community development (50.5% and 37.6%, respectively), and rehabilitation and medical social services (35.8% and 28.1%, respectively).

Table 1.

Profile of Participants in Quantitative Study.

Variable		Cohort 1		Cohort 2		Overall
Variable		Training Group (%)	Comparison Group (%)	Training Group (%)	Comparison Group (%)	Training Group (%)	Comparison Group (%)
Gender
Male		36.8	32.0	37.5	35.5	37.2	33.8
Female		63.2	68.0	62.5	64.5	62.8	66.2
Educational level
Sub-degree/diploma		0.9	6.8	0.9	12.7	0.9	9.9
Bachelor’s degree		27.2	45.6	30.4	37.3	28.8	41.3
Master’s degree		68.4	47.6	68.8	50.0	68.6	48.8
Doctoral degree		3.5	0.0	0.0	0.0	1.8	0.0
Job position
Social worker		36.0	58.3	45.5	60.0	40.7	59.2
Healthcare professional		3.5	1.9	2.7	3.6	3.1	2.8
Manager/administrator		41.2	28.2	27.7	19.1	34.5	23.5
Social entrepreneur		4.4	0.0	2.7	0.9	3.5	0.5
Therapist		0.9	0.0	4.5	2.7	2.7	1.4
Others		14.0	11.7	17.0	13.6	15.5	12.7
Work areas (multiple choices)
Family and children services		52.6	38.8	50.9	26.3	51.8	32.4
Youth services		29.8	32.0	26.8	15.5	28.3	23.5
Older adult services		21.1	12.6	35.7	25.5	28.3	19.2
Community development		51.7	37.8	49.1	37.3	50.5	37.6
Social disadvantage and social security		36.1	15.6	26.8	6.3	31.4	10.7
Rehabilitation and medical social services		33.3	22.4	38.4	33.6	35.8	28.1
Age (years)	Mean	40.55	37.14	38.50	36.65	39.53	36.89
Age (years)	SD	8.22	8.55	8.45	9.02	8.38	8.78

Apart from surveys, nine focus groups were organized, of which five had training participants discuss how they benefited from the MEL project, two consisted of mentors and two had trainers explain their views on the effectiveness of the project. Invitation letters were sent to training participants, mentors, and trainers. Those who were willing to participate were interviewed. A total of 15 training participants, six trainers, and seven mentors joined the focus group interviews. Among training participants, seven were social workers (46.6%), ten had more than 5 years of experience (66.7%), and their work areas covered all six areas mentioned above. Among trainers and mentors, all of them had more than 5 years of experience and were active researchers or practitioners in the field of program evaluation, social services, business, or information technology. Table 2 shows the background information of the focus group members.

Table 2.

Background Information of the Focus Group Members (N = 28).

Code	Job Position	Work Areas (Main)	Educational Level	Years of Experience
Trainer 1	Professor, Foundation director	Evaluation of children and family service	Doctoral	≥10
Trainer 2	Professor	Evaluation of youth services	Doctoral	≥10
Trainer 3	Research associate	Evaluation of youth services	Doctoral	≥10
Trainer 4	Professor	Evaluation of complex interventions	Doctoral	≥10
Trainer 5	Adjunct associate professor	Media and communications	Doctoral	≥10
Trainer 6	Assistant professor	Evaluation of child welfare services	Doctoral	≥10
Mentor 1	Director	Design thinking	Master	≥10
Mentor 2	Associate professor	New media	Doctoral	≥10
Mentor 3	Vice-chairman	Social enterprises	Master	≥10
Mentor 4	Chairman	Interactive marketing	Master	≥10
Mentor 5	Director	Community development	Master	≥10
Mentor 6	Assistant professor	Rehabilitation and medical social services	Doctoral	≥5
Mentor 7	Assistant professor	Rehabilitation and medical social services	Doctoral	≥5
Participant 1	Manager/administrator	Rehabilitation and medical social services	Master	≥5
Participant 2	Therapist	Family and children services	Master	≥5
Participant 3	Social worker	Rehabilitation and medical social services	Bachelor	≥5
Participant 4	Social worker	Family and children services	Master	3–4
Participant 5	Manager/administrator	Community development	Master	≥5
Participant 6	Manager/administrator	Family and children services	Master	≥5
Participant 7	Healthcare professional	Older adult services	Master	1–2
Participant 8	Social worker	Older adult services	Bachelor	<1
Participant 9	Social entrepreneur	Rehabilitation and medical social services	Master	≥5
Participant 10	Social worker	Social disadvantage	Bachelor	≥5
Participant 11	Social worker	Older adult services	Bachelor	1–2
Participant 12	Manager/administrator	Youth services	Master	≥5
Participant 13	Social worker	Social disadvantage and social security	Master	≥5
Participant 14	Social entrepreneur	Community development	Bachelor	≥5
Participant 15	Social worker	Community development	Master	3–4

All focus groups were led by trained interviewers who had not participated in the implementation of the project, with groups of trainers and mentors being conducted in English, and groups of training participants being conducted in Cantonese, a dialect in China. The meeting for each focus group lasted one hour. Sample guiding questions for training participants included, “What knowledge and/or skills learned from the training sessions can be applied to your agency?” “How did you benefit from the mentored practicum?” and “How did you benefit from the e-learning knowledge hub?” Sample guiding questions for mentors included, “What knowledge and/or skills have the participants learned from the training sessions, based on your observations?” “What improvements have the participants achieved after the mentored practicum?” and “How did you find the e-learning knowledge hub useful in helping the MEL participants?” Sample guiding questions for trainers included, “How have the training sessions helped facilitate positive changes in the participants in relation to MEL?” “How did you find the e-learning knowledge hub to be useful for the MEL participants?” and “What do you think are the key achievements of the MEL training scheme?” Ethical approval had been obtained from an ethical review committee prior to the commencement of the project.

Measures

The program evaluation capacity of NGO practitioners was measured by the Program Evaluation Capacity Scale (PECS) developed by our research team, consisting of three important components—evaluation mindset, evaluation implementation, and evaluation communication. After reviewing the theoretical and empirical literature, we developed 17 items across these three dimensions. A group of 10 experts in the field of NGO development independently reviewed the items’ structure for conciseness, clarity, and developmental appropriateness. Following the suggestions and feedback of these experts, we made some minor revisions on the scale items. Participants responded to the 17 items on a 5-point Likert scale from “1 = none” to “5 = very much.”

Evaluation mindset refers to the extent to which NGO practitioners have systematically understanding and positive attitudes toward program evaluation. The Evaluation Mindset Subscale consisted of seven items. Three sample items were “Having an understanding of program evaluation,” “Having awareness of available research tools and technological resources for conducting program evaluation,” and “Appreciating program evaluation knowledge in informing service delivery.” The reliability estimates of this subscale in the baseline and follow-up tests for Cohort 1, Cohort 2, and the full sample were high (see Table 3; Cronbach’s α of this subscale ranged from 0.854 to 0.980).

Table 3.

Reliability Estimates of Measures.

Evaluation Scale	Reliability
	Cohort 1				Cohort 2				Overall
	Training Group (N = 114)		Comparison Group (N = 103)		Training Group (N = 112)		Comparison Group (N = 110)		Training Group (N = 226)		Comparison Group (N = 213)
	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post
Mindset	0.930	0.854	0.971	0.973	0.941	0.911	0.971	0.980	0.936	0.887	0.971	0.977
Implementation	0.827	0.817	0.803	0.899	0.819	0.844	0.826	0.880	0.822	0.833	0.819	0.891
Communication	0.698	0.776	0.779	0.870	0.709	0.788	0.762	0.840	0.703	0.784	0.780	0.857

Evaluation implementation refers to the extent to which NGO practitioners could conduct program evaluations and use them within organizations. The Evaluation Implementation Subscale was composed of six items. Sample items included “Developing performance indicators for service development and monitoring,” “Practicing the effectiveness-based framework of monitoring, evaluation, and learning,” and “Conducting problem analyses and needs assessments.” The reliability estimates of this subscale in the baseline and follow-up tests for Cohort 1, Cohort 2, and the full sample were high (see Table 3; Cronbach’s α of this subscale ranged from 0.803 to 0.899).

Evaluation communication refers to the extent to which NGO practitioners could disseminate and share program evaluation results with diverse stakeholders. The Evaluation Communication Subscale included four items. Three sample items were “Using digital storytelling techniques in sharing evidence-based practices,” “Using social media and the Internet in participant recruitment and data collection,” and “Conducting media and press interviews or conferences to build the brand, disseminate outcomes, and share impact.” The reliability estimates of this subscale in the baseline and follow-up tests for Cohort 1, Cohort 2, and the full sample were high (see Table 3; Cronbach’s α of this subscale ranged from 0.698 to 0.870).

Data Analysis

For the quantitative section, multivariate repeated-measures analyses of covariance (MANCOVA) including the scores of evaluation communication, evaluation implementation, and evaluation mindset at the baseline and follow-up as dependent variables tested the effectiveness of the MEL project. First, we conducted repeated-measures MANCOVA based on the first and second cohorts of the MEL project separately and then combined the two cohorts to test the overall effectiveness of the MEL project. The factors in these analyses were group (training or comparison), time (baseline and follow-up), and potential categorical variables to correct for (gender, education, and job positions), while potentially confounding continuous variables (age) were entered as covariates.

For the qualitative section, since focus groups of trainers and mentors were conducted in English and focus groups of training participants were conducted in Cantonese; hence, the interviews of training participants were first translated into English by research assistants who are native Cantonese speakers. To assure translation validity, the transcripts were carefully checked by the principal investigator before being used as the basis of data analyses. After that, thematic analysis was conducted and guided by the steps described in Braun and Clarke (2006). The initial steps were to identify and group all patterned meaning units from the transcripts. After that, related patterns were combined and cataloged into themes. Focusing on the NGO practitioners’ learning experiences in the MEL project, themes in the three domains of perceived outcomes that emerged from the participants’ responses were pieced together to form a comprehensive picture of how the MEL project promoted such changes. Data were coded and mapped by members of the research team and then cross-checked by the principal investigator.

Results

Quantitative Findings

The repeated-measures MANCOVA involving baseline and follow-up scores was an omnibus test of the three indicators of program evaluation capacity: evaluation communication, evaluation implementation, and evaluation mindset. Based on the Cohort 1 data, we found there were significant main effects of time on evaluation communication (F = 19.34, p < 0.001, partial η² = 0.08), evaluation implementation (F = 19.75, p < 0.001, partial η² = 0.08), and evaluation mindset (F = 95.73, p < 0.001, partial η² = 0.31), indicating that the three abovementioned indicators of program evaluation capacity increased from baseline to follow-up for both training and comparison groups. Furthermore, we also found a significant interaction of time × group for evaluation communication (F = 13.69, p < 0.001, partial η² = 0.06), evaluation implementation (F = 13.81, p < 0.001, partial η² = 0.06), and evaluation mindset (F = 11.94, p < 0.01, partial η² = 0.05), showing that the training group experienced a larger increase of program evaluation capacity than the comparison group. The results are shown in Table 4.

Table 4.

Means and Standard Deviations of the Main Variables at Pre- and Post-Training and the Repeated Measure MANCOVA Results (Cohort 1, Cohort 2, and Overall Sample).

Cohorts	Evaluation Variable	Training Group				Comparison Group				Repeated Measure MANCOVA
		Mean		SD		Mean		SD		Main Effects			Time × Group
		Pre	Post	Pre	Post	Pre	Post	Pre	Post	F	p	η²	F	p	η²
1	Mindset	2.38	3.41	0.87	0.58	2.13	2.63	0.97	1.03	95.73	<0.001	0.31	11.94	<0.01	0.05
	Implementation	2.79	3.32	0.72	0.63	3.03	3.08	0.68	0.89	19.75	<0.001	0.08	13.81	<0.001	0.06
	Communication	2.39	2.98	0.74	0.80	2.76	2.81	0.86	1.02	19.34	<0.001	0.08	13.69	<0.001	0.06
2	Mindset	2.19	3.39	0.91	0.71	1.90	2.76	0.94	1.09	175.29	<0.001	0.44	4.56	<0.05	0.02
	Implementation	2.78	3.39	0.72	0.72	2.82	3.20	0.83	0.79	69.74	<0.001	0.24	3.81	>0.05	0.02
	Communication	2.35	3.15	0.77	0.79	2.40	2.96	0.85	0.87	130.51	<0.001	0.37	4.17	<0.05	0.02
Overall	Mindset	2.29	3.40	0.89	0.64	2.01	2.69	0.96	1.06	263.85	<0.001	0.38	14.99	<0.001	0.03
	Implementation	2.79	3.36	0.72	0.68	2.92	3.14	0.76	0.84	80.00	<0.001	0.16	15.72	<0.001	0.04
	Communication	2.37	3.07	0.75	0.80	2.57	2.89	0.87	0.96	111.63	<0.001	0.20	16.01	<0.001	0.04

Note. The scores were on a 1–5 scale.

Based on the Cohort 2 data, we found the main effects of time were significant for evaluation communication (F = 130.51, p < 0.001, partial η² = 0.372), evaluation implementation (F = 69.74, p < 0.001, partial η² = 0.24), and evaluation mindset (F = 175.29, p < 0.001, partial η² = 0.44), revealing that the three indicators of program evaluation capacity increased from baseline to follow-up for both training and comparison groups. Moreover, the interaction of time × group was significant for evaluation communication (F = 4.17, p < 0.05, partial η² = 0.02) and evaluation mindset (F = 4.56, p < 0.05, partial η² = 0.02), but non-significant for evaluation implementation (F = 3.81, p > 0.05, partial η² = 0.02). This result suggests the training group experienced a larger increase of evaluation communication and evaluation mindset than the comparison group; however, there was no significant difference in the changes in evaluation implementation between the training and comparison groups. The results are presented in Table 4.

When combining the first and second cohorts of data, a significant main effect of time showed that evaluation communication (F = 111.63, p < 0.001, partial η² = 0.20), evaluation implementation (F = 80.00, p < 0.001, partial η² = 0.16), and evaluation mindset (F = 263.85, p < 0.001, partial η² = 0.38) increased from baseline to follow-up for both training and comparison groups. In addition, a significant interaction of time × group emerged for evaluation communication (F = 16.01, p < 0.001, partial η² = 0.04), evaluation implementation (F = 15.72, p < 0.001, partial η² = 0.04), and evaluation mindset (F = 14.99, p < 0.001, partial η² = 0.03), showing the training group experienced a larger increase of program evaluation capacity than the comparison group. The results are shown in Table 4.

Qualitative Findings

The thematic analysis of qualitative interviews resulted in three key themes surrounding the participants’ perceived outcomes of the MEL project and how the project promoted such positive changes. These themes are summarized and described below.

Improving Mindset of Program Evaluation

The qualitative findings first highlighted the participants’ positive changes in the evaluation mindset of program evaluation, which consists of knowledge, awareness, and attitudes toward program evaluation. Specifically, the participants experienced an improvement in evaluation mindset through acquiring systematic and practical evaluation knowledge and realizing the significance of program evaluation.

Acquiring systematic and practical evaluation knowledge. Nearly all participants emphasized that the systematic training in the MEL project has enriched their knowledge and skills in program evaluation, thereby helping them respond to increasingly complex external demands. The following account of Participant 10, who provided services of social disadvantage and social security, is a typical example of the remarks from other participants:

The course is very comprehensive and provides us with a big picture of how we should use these concepts step-by-step and how we should link input, activity, project implementation, expected output, and expected outcome together. This will definitely be helpful to us in implementing a systematic program evaluation in the future! And if we had a systematic evaluation, we would be more able to convince our funders! (Participant 10)

Many participants also reported that they greatly benefited from the conceptual inputs in the training course. Participant 1, who provided services in rehabilitation and medical social services, recalled, “I found the concepts of logic model and theory of change very useful. The course trainer used a real example from his anti-drug project to illustrate these concepts to help us understand better!”

Realizing the significance of program evaluation. Notably, some participants brought out another theme: increased awareness of the significance of program evaluation. That is, the MEL training scheme led them to clearly realize the importance of applying program evaluation to prove a project’s effectiveness, thereby achieving broader social impacts. Participant 13, who provided services in social disadvantage and social security, stated the following:

I realized that it was not enough for us to have good projects without anyone knowing how good they are! If we had a more systematic way to evaluate our projects and visualize program effectiveness, more people would know about our projects, and we could achieve better outcomes when promoting similar events. (Participant 13)

Trainers in the MEL project also noticed that participants had a significant mindset change on program evaluation. Some trainers particularly mentioned that the participants were increasingly making evaluation an integral part of their everyday practices. Trainer 1, an overseas expert in the field of evaluation of children and family service, indicated that “The key achievement is that participants realize that this is something they can do in their daily activities. There is something that they can keep in their daily practice.”

Transferring Individual Learning into Organizational Practice

Moreover, after participating in the MEL project, participants not only improved their mindset of program evaluation but also transferred it into behavioral changes. As revealed by the interviews, participants had enhanced evaluation implementation through actively applying the MEL knowledge and skills into practice and promoting evaluative culture in their organizations.

Applying MEL knowledge and skills into practice. Participants highlighted that the MEL knowledge and skills were practical enough to be directly applied to their program evaluation practices. For instance, Participant 3, who provided services in rehabilitation and medical social services, vividly shared her experience of adopting three scales learned from the MEL courses to measure the effectiveness of her team’s project. Some participants also gave accounts of how they applied the MEL knowledge when designing their services, including analyzing problems and conducting needs assessments. The following quotation from Participant 9, who provided services in rehabilitation and medical social services, is a typical example:

I have learned some concepts and theories, such as the logic model from the training sessions, and used them in our team’s final project. Whenever I need to design new services, I think about the needs assessment and problem analysis according to the logic model. (Participant 9)

Promoting evaluative culture in their organizations. Notably, participants were eager to share the knowledge they learned from the MEL project with colleagues, thereby promoting an evaluative culture at the organizational level. The following quotation provided by Participant 14, who provided services in community development, is a typical example: “After learning about concepts such as the logic model, I shared them with colleagues. We will have regular sharing sessions to share MEL knowledge and skills to bring some new things to our organization.” Trainers also indicated that participants had applied the MEL knowledge and skills into practice and made efforts to change the culture of their organization. As Trainer 3, our local trainer in the field of evaluation of youth services, illustrated:

They (participants) actually go back to their own organizations and try to change the organizational culture. They start to use the logic model that we proposed and try to share with colleagues how to use it and how to monitor their service programs. (Trainer 3)

Learning Communication Tools for Evaluation Dissemination

Participants unanimously shared their enhanced capacity for disseminating evaluation information with various learned communication tools. According to the interview, the participants greatly improved their evaluation communication through adopting advanced communication tools for data management and acquiring presentation and storytelling skills for impact dissemination.

Adopting advanced communication tools for data management. The participants gave many accounts of how helpful they found the advanced communication tools introduced by the trainers in facilitating project data management. The introduction of some digital tools, such as Airtable and Google Forms, was described by participants as an eye-opening experience. The recurring themes extracted from the narratives revealed that such tools were highly practical, and that the participants soon adopted them in daily practice, thus greatly contributing to the communication and coordination in their organizations and the data collection of their projects. Participant 4, who provided older adult services, expressed the following remark:

There was one session that introduced Airtable, which has the function of data storage and promotion. It is particularly useful for me. After the training, I tried using Airtable to aid in my work. It is being proposed and tested to see if we can use it as a platform for the database in the future. (Participant 4)

Acquiring presentation and storytelling skills for impact dissemination. Moreover, the participants mentioned that they had acquired various knowledge about presentation and storytelling skills, which were regarded as useful strategies to tell a good story and, thus, share their program evaluation with the public. Such tools also inspired participants to creatively integrate the samples learned from MEL training with their own projects. Participant 11, who provided services in social disadvantage and social security, mentioned that she was highly impressed by the concept and methods of digital storytelling.

I like the training session on digital storytelling very much! There are many strategies that we can use to tell a good story, such as using emotional engagement to attract the audience. Today, it is very important for NGOs to learn how to use media to let other people know about your projects and services and to bring about more social impacts in society. (Participant 11)

Discussion and Applications to Practice

Although theoretical and practical interest in PECB is currently high, evidence testing the effectiveness of PECB activities, processes, and outcomes has been suggested as limited (Morkel & Ramasobana, 2017; Preskill & Boyle, 2008a), with a small number of studies and some even showing no capacity gains for participants (e.g., Tarsilla, 2014). The current study fills the gap in program evaluation literature through reporting on the evaluation results of the MEL project. Using a pretest–posttest quasi-experimental design, the quantitative results demonstrated that NGO practitioners joining the MEL project (training group) showed higher levels of improvement in all three core aspects of program evaluation capacity—evaluation mindset, evaluation implementation, and evaluation communication—compared with those who did not join the project (comparison group). The qualitative findings further indicated that the MEL project helped the participants acquire systematic evaluation knowledge, underlined the significance of program evaluation, fostered the capacity for program evaluation dissemination, and subsequently transferred individual learning into organizational practice. The results are aligned with the previously reported positive evaluation findings of other PECB programs conducted in Western contexts (Despard, 2016; García-Iriarte et al., 2010) and added empirical evidence to strengthen efforts to pursue PECB.

In the present MEL project, significant differences between pretest and posttest were found in participants’ scores for evaluation mindset, which is consistent with past research (e.g., Preskill & Boyle, 2008b). In particular, results showed that participants gained improvements with more evaluative knowledge and skills, awareness of the significance and benefits of program evaluation, and motivation to conduct rigorous program evaluations. This type of evaluation mindset is critical in program evaluation capacity because it integrates evaluative knowledge in its varied forms in all phases of the evaluation process with practitioners’ readiness and confidence to use the acquired information (Kegeles et al., 2005; Morkel & Ramasobama, 2017). The positive changes in the evaluation mindset might be attributed to the multi-strategy intervention adopted by the MEL project, which includes training, mentoring, an e-learning knowledge hub, and a final pitching day. This echoes the empirical finding reporting that training, despite work as a significant part of PECB practices, has been found insufficient to invoke changes on its own; therefore, many scholars have suggested a combination of multiple interventions for conducting PECB practices (Preskill & Boyle, 2008a; Tarsilla, 2014). A multi-strategy intervention could also be the critical element to produce positive impacts on evaluation implementation and evaluation communication in the MEL project.

The benefit of the MEL project in promoting evaluation implementation among NGO practitioners was another significant finding in the present study. PECB practices usually provide knowledge-based interventions, such as training and workshops that simply focus on increasing the participants’ understanding of program evaluation (e.g., Chauveron et al., 2021; Preskill & Boyle, 2008b). Yet, the increasing emphasis on program evaluation challenges the capacity of NGO practitioners to not only conduct a quality program evaluation but to also apply what they learned about program evaluation to their daily work within the organization (Taylor-Ritzler et al., 2013). In response, the MEL project innovatively incorporated a mentorship scheme into intensive training courses to create opportunities for learning-by-doing for participants, with a final project pitching competition being simultaneously designed as a behavioral-level intervention to promote transferable learning. As a result, both quantitative and qualitative findings provided support for the beneficial effects of the MEL project on participants’ capacity to transfer their individual learning into organizational practice. One notable result is that despite an upward trend of the scores of evaluation implementation among participants in Cohort 2, there was no statistically significant difference in the score changes between the training and comparison groups. A plausible explanation is that, influenced by the COVID-19 pandemic, participants in Cohort 2 joined the MEL project mainly through the online Zoom platform, which weakened the project’s impacts in strengthening evaluation implementation.

The results also indicated a significantly higher increase in evaluation communication scores among the participants, compared with their comparison counterparts. Researchers and evaluators in previous studies have commonly targeted improving general communication skills to work with co-workers or policymakers (e.g., DeCorby-Watson et al., 2018), whereas evaluation communication has been increasingly recognized as a key component of program evaluation capacity and high-quality evaluation practice (Mitchell, 2017; Preskill & Boyle, 2008a). To our knowledge, the impacts of PECB programs on enhancing evaluation communication have not been thoroughly investigated in previous studies. The present MEL project underscores the importance of accessing and disseminating evaluation information, with various approaches designed to cultivate and enhance NGO practitioners’ awareness and abilities for evaluation dissemination. The positive evaluation findings imply that the MEL project can facilitate participants to develop the specific communication knowledge for program evaluation, including adopting advanced communication tools and acquiring storytelling and presentation skills, which can thereby promote their holistic development of program evaluation capacity. As indicated by the qualitative findings, with the acquired ability of evaluation communication, participants also gained more strength to attract potential funders through visualizing the social impacts of their projects and communicating the benefits of the service. Strong evaluation communication capacities have been highly recommended by scholars to ensure NGO survival in the prevailing competitive contract culture (García-Iriarte et al., 2010).

Furthermore, this study has several theoretical contributions. Theoretically, based on the existing literature, direct evidence for the effectiveness of the effectiveness-based framework is still lacking, and no study has attempted to design a corresponding PECB program. Our study is a first attempt at assessing a PECB program that adopts the effectiveness-based framework to build NGO practitioners’ evaluation capacity at multidimensional levels. Additionally, the MEL project offers academics and practitioners an overview of the dimensional aspects for assessing the capacity of program evaluation for NGO practitioners. Despite the increasing recognition of a multidimensional understanding of program evaluation capacity, existing research still tends to focus solely on one dimension, such as assessing evaluators’ knowledge level or behaviors (Medina et al., 2015; Morkel & Ramasobana, 2017). Hence, our study fills the gap by providing a guiding conceptual and practical framework for both evaluation study and PECB intervention. The MEL project reflects a comprehensive perspective with multidimensional focuses on understanding evaluation capacity, which further underscores that researchers and practitioners should simultaneously consider the evaluation mindset, evaluation implementation, and evaluation communication of NGO practitioners. Most importantly, the measures developed in the present study could be employed to conduct a pretest–posttest assessment of program evaluation capacity, and the results could be used to focus on their PECB efforts, as well as to assess participants’ changes that reflect the outcomes of PECB activities after the completion of the intervention.

Taken together, the MEL project offers a possible comprehensive intervention framework for researchers and evaluators in effectively conducting PECB efforts. Specifically, the following are key strengths for adopting the effectiveness-based framework in future PECB activities. First, while it becomes apparent that the capacity for NGO practitioners to conduct quality evaluations involves multiple dimensions, PECB programs need to identify a theory-based framework that would contribute to building multidimensional competencies. Guided by the effectiveness-based framework, we designed a corresponding PECB program to promote NGO practitioners' holistic evaluation capacity across three key aspects. Essentially, the critical component of the effectiveness-based framework is to develop a thorough understanding of effectiveness principles in every step of the program evaluation progress. With our results, this framework has proven beneficial to systematically strengthen the evaluation capacity of NGO practitioners who might lack evaluation literacy and related experiences. Second, based on the effectiveness-based framework, four interrelated sections were designed in the training courses: service development, service monitoring, program evaluation, and impact assessment. The positive outcomes of this study indicated these four areas contributed greatly to giving participants a comprehensive picture of the program evaluation at the mindset level, which is a premise for them to implement evaluations themselves. Hence, future PECB might consider involving these four topics in their design of training courses. Third, another strength of the effectiveness-based framework is that it incorporated multiple effective approaches arising from previous PECB practices (e.g., DeCorby-Watson et al., 2018). The combination of training courses and mentorship is advisable for future PECB activities: while a training scheme enables participants to gain knowledge and attitudes, a focus on linking evaluation mindset with implementation through mentoring would be crucial for enhancing participants' engagements with program evaluation, as well the sustainability of behavioral changes. Moreover, the effectiveness-based framework suggests PECB projects may consider providing technical support for front-line workers and establishing an online knowledge hub, if possible. Fourth, the significance of promoting hands-on experience in conducting program evaluation is repeatedly highlighted in the effectiveness-based framework. Besides the mentoring and a final pitching day, the framework additionally adopts the train-the-trainer approach, which would be adaptable in future PECB projects, especially for those targeting small NGOs. This implication corresponds with previous findings as NGOs often have limited funding, staff, and resources available to develop their program evaluation capacity (Carman, 2007; Connolly & York, 2002). Future PECB activities could focus on designated staff persons to promote the diffusion of program evaluation activities within the organization (García-Iriarte et al., 2010). Fifth, the effectiveness-based framework contributes by demonstrating that internet-based instruction is an alternative when conducting PECB programs, especially under special circumstances like the current COVID-19 pandemic. The benefits of the online intervention (including online training and mentoring), suggested by our results, are consistent with the conclusion of Cook et al. (2008). However, it cannot be ignored that online intervention might have limited impacts on promoting the capacity of evaluation implementation among NGO practitioners. Herewith, additional individualized and customized assistance should be provided. Finally, participants of the MEL project came from a diverse background of service areas, which further indicates the applicability of the effectiveness-based framework to NGO practitioners from different contexts, which serves as an advantage that should not be ignored.

Although the present findings are significant, there are several limitations in this study. First, the present findings are limited to one posttest immediately after the training and mentorship scheme. To test the long-term impact of the programs, future research could have more follow-up tests over a longer period. Second, without a randomized controlled trial, the generalizability of the findings from the present study is uncertain. Therefore, future research should randomly assign the participants to the control and treatment groups and compare the changes before and after their participation in a PECB project. Additionally, although all the main effects are significant, no main effects of this study reached large effects and only some items reached medium effects. Similarly, though most of the interaction effects are significant, the effect sizes are at the small effect size levels (Ferguson, 2009). The limitation in training effect sizes of this study may be derived from the quasi-experimental design. Hence, future research needs to consider conducting more rigorous research design studies (e.g., randomized trial design) with follow-up tests to get more rigorous training effects. Finally, the present study did not offer continuous support to the participants after the completion of the project, limiting the sustainability of the participants’ program evaluation practices; PECB is an ongoing process (Beere, 2005), and it is through continuous practice and reflection that evaluation becomes integral to organizational activities (Doherty et al., 2015). Future studies could provide ongoing support, such as individualized and customized technical assistance, to train participants to enhance their program evaluation capacity and the evaluative culture within their organizations.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study is supported by The Hong Kong Jockey Club Charities Trust (2019-0022).

ORCID iDs

Steven Sek-yum Ngai

Chau-kiu Cheung

References

Andrews

A. B.

Motes

P. S.

Floyd

A. G.

Flerx

V. C.

Fede

A. L. D.

(2005). Building evaluation capacity in community-based organizations. Journal of Community Practice, 13(4), 85–104. https://doi.org/10.1300/j125v13n04_06.

Beere

(2005). Evaluation capacity-building: A tale of value-adding. Evaluation Journal of Australasia, 5(2), 41–47. https://doi.org/10.1177/1035719x0500500207.

Bornstein

(2006). Systems of accountability, webs of deceit? Monitoring and evaluation in South African NGOs. Development, 49(2), 52–61. https://doi.org/10.1057/palgrave.development.1100261.

Bourgeois

Cousins

J. B.

(2013). Understanding dimensions of organizational evaluation capacity. American Journal of Evaluation, 34(3), 299–319. https://doi.org/10.1177/1098214013477235.

Braun

Clarke

(2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Brown

N. R.

Kelsey

K. D.

(2016). Prospect evaluation as an emerging pre-evaluation technique in the case of Great Plains wheat producers’ use of Web 2.0 Technology. Journal of Human Sciences and Extension, 4(2), 126–140.

Burton

van den Broek

(2009). Accountable and countable: Information management systems and the bureaucratization of social work. British Journal of Social Work, 39(7), 1326–1342. https://doi.org/10.1093/bjsw/bcn027.

Carman

J. G.

(2007). Evaluation practice among community-based organizations. American Journal of Evaluation, 28(1), 60–75. https://doi.org/10.1177/1098214006296245.

Carman

J. G.

Fredericks

K. A.

(2008). Nonprofits and evaluation: Empirical evidence from the field. New Directions for Evaluation, 2008(119), 51–71. https://doi.org/10.1002/ev.268.

10.

Chauveron

L. M.

Urban

J. B.

Samtani

Cox

Moorman

Hargraves

Buckley

Linver

M. R.

(2021). Promoting evaluation in youth character development through enhanced evaluation capacity building: Empirical findings from the PACE Project. New Directions for Evaluation, 2021(169), 79–95. https://doi.org/10.1002/ev.20447.

11.

Cheung

J. C. S.

(2016). A social work perspective on using Social Return on Investment (SROI) in humanistic social care. Australian Social Work, 70(4), 491–499. https://doi.org/10.1080/0312407x.2016.1189583.

12.

Connolly

York

(2002). Evaluating capacity-building efforts for nonprofit organizations. OD Practitioner, 34(4), 33–39.

13.

Cook

D. A.

Levinson

A. J.

Garside

Dupras

D. M.

Erwin

P. J.

Montori

V. M.

(2008). Internet-based learning in the health professions. JAMA, 300(10), 1181–1196. https://doi.org/10.1001/jama.300.10.1181.

14.

DeCorby-Watson

Mensah

Bergeron

Abdi

Rempel

Manson

(2018). Effectiveness of capacity building interventions relevant to public health practice: A systematic review. BMC Public Health, 18(1), 684. https://doi.org/10.1186/s12889-018-5591-6.

15.

Despard

M. R.

(2016). Strengthening evaluation in nonprofit human service organizations: Results of a capacity-building experiment. Human Service Organizations: Management, Leadership & Governance, 40(4), 352–368. https://doi.org/10.1080/23303131.2016.1140101.

16.

Doherty

Eccleston

Hansen

Natalier

Churchill

(2015). Building evaluation capacity in micro community organisations—more burden than benefit? Evaluation Journal of Australasia, 15(4), 29–37. https://doi.org/10.1177/1035719x1501500404.

17.

Ferguson

C. J.

(2009). An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice, 40(5), 532–538. https://doi.org/10.1037/a0015808.

18.

García-Iriarte

Suarez-Balcazar

Taylor-Ritzler

Luna

(2010). A catalyst-for-change approach to evaluation capacity building. American Journal of Evaluation, 32(2), 168–182. https://doi.org/10.1177/1098214010387114.

19.

Grinnell

R. M.

Gabor

P. A.

Unrau

Y. A.

(2019). Program evaluation for social workers: Foundations of evidence-based programs (8th ed.). Oxford University Press.

20.

Hailey

James

(2003, November). NGO capacity building: The challenge of impact assessment [Paper presentation]. The New Directions in Impact Assessment for Development. Methods & Practice Conference. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.535.7656&rep=rep1&type=pdf.

21.

Hong Kong Jockey Club Charities Trust . (2017, March). Outcome evaluation approach adopted by the Hong Kong Jockey Club Charities Trust [Poster presentation]. The 6th Conference on Practice and Research for Social Service Excellence. http://ebp.hkcss.org.hk/ppt/conference/2017_1/Session3/Bryan_and_Ada.pdf

22.

Hong Kong Jockey Club Charities Trust (2018). Thoughts on MEL capacity building. Hong Kong Jockey Club Charities Trust.

23.

Humphries

Gomez

Hartwig

(2011). Sustainability of NGO capacity building in southern Africa: Successes and opportunities. The International Journal of Health Planning and Management, 26(2), e85–e101. https://doi.org/10.1002/hpm.1029.–.

24.

Jockey Club MEL Institute Project (2021). Web-based audio visual learning materials. https://jcmel.swk.cuhk.edu.hk/web-based-audio-visual-learning-mate.

25.

Kegeles

S. M.

Rebchook

G. M.

Tebbetts

(2005). Challenges and facilitators to building program evaluation capacity among community–based organizations. AIDS Education and Prevention, 17(4), 284–299. https://doi.org/10.1521/aeap.2005.17.4.284.

26.

Kettner

P. M.

Moroney

R. M.

Martin

L. L.

(2016). Designing and managing programs: An effectiveness-based approach (5th ed.). SAGE Publications, Inc.

27.

Kwan

Kee

Chan

(2016). Introduction to social impact measurement: Hong Kong context. Fullness Social Enterprises Society Limited.

28.

Lam

W. F.

Nie

(2020). Online or offline? Nonprofits’ choice and use of social media in Hong Kong. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 31(1), 111–128. https://doi.org/10.1007/s11266-019-00128-1.

29.

Leviton

L. C.

(2013). Some underexamined aspects of evaluation capacity building. American Journal of Evaluation, 35(1), 90–94. https://doi.org/10.1177/1098214013502844.

30.

McCoy

Rose

Connolly

(2014). Approaches to evaluation in Australian child and family welfare organizations. Evaluation and Program Planning, 44, 68–74. https://doi.org/10.1016/j.evalprogplan.2014.02.004.

31.

Medina

Acosta-Perez

Velez

Martinez

Rivera

Sardinas

Pattatucci

(2015). Training and capacity building evaluation: Maximizing resources and results with Success Case Method. Evaluation and Program Planning, 52, 126–132. https://doi.org/10.1016/j.evalprogplan.2015.03.008.

32.

Mitchell

G. E.

(2017). Accounting for outcomes: Monitoring and evaluation in the transnational NGO sector. In Tirmizi

A. S.

Vogelsang

J. D.

(Eds.), Leading and Managing in the Social Sector: Strategies for Advancing Human Dignity and Social Justice (pp. 263–282). Springer.

33.

Morkel

Ramasobana

(2017). Measuring the effect of evaluation capacity building initiatives in Africa: A review. African Evaluation Journal, 5(1), 1–12. https://doi.org/10.4102/aej.v5i1.187.

34.

Nip

P. T.

(2010). Social welfare development in Hong Kong: Changes and challenges in building a caring and harmonious society. Asia Pacific Journal of Social Work and Development, 20(1), 65–81. https://doi.org/10.1080/21650993.2010.9756076.

35.

Preskill

Boyle

(2008a). A multidisciplinary model of evaluation capacity building. American Journal of Evaluation, 29(4), 443–459. https://doi.org/10.1177/1098214008324182.

36.

Preskill

Boyle

(2008b). Insights into evaluation capacity building: Motivations, strategies, outcomes, and lessons learned. The Canadian Journal of Program Evaluation, 23(3), 147–174.

37.

Rossi

P. H.

Lipsey

M. W.

Henry

G. T.

(2019). Evaluation: A systematic approach (8th ed.). SAGE Publications, Inc.

38.

Social Welfare Department of Hong Kong . (2021, June). Service performance section, service performance monitoring. Social Welfare Department Website. Retrieved November 2021, from https://www.swd.gov.hk/en/index/site_ngo/page_serviceper/sub_objectives/

39.

Suárez

Marshall

J. H.

(2014). Capacity in the NGO sector: Results from a national survey in Cambodia. VOLUNTAS: International Journal of Voluntary and Nonprofit Organizations, 25(1), 176–200. https://doi.org/10.1007/s11266-012-9331-8

40.

Tarsilla

(2014). Evaluation capacity development in Africa: Current landscape of international partners’ initiatives, lessons learned and the way forward. African Evaluation Journal, 2(1). https://doi.org/10.4102/aej.v2i1.89

41.

Taylor-Ritzler

Suarez-Balcazar

Garcia-Iriarte

Henry

D. B.

Balcazar

F. E.

(2013). Understanding and measuring evaluation capacity. American Journal of Evaluation, 34(2), 190–206. https://doi.org/10.1177/1098214012471421.

42.

Tsui

M. S.

(2004). Gone with the wind: The impacts of managerialism on human services. British Journal of Social Work, 34(3), 437–442. https://doi.org/10.1093/bjsw/bch046.

43.

Whatley

(2013). Improved learning for greater effectiveness in development NGOs. Development in Practice, 23(8), 963–976. https://doi.org/10.1080/09614524.2013.840563.

44.

Yan

M. C.

Cheung

J. C. S.

Tsui

M. S.

Chu

C. K.

(2015). Examining the neoliberal discourse of accountability: The case of Hong Kong’s social service sector. International Social Work, 60(4), 976–989. https://doi.org/10.1177/0020872815594229.