Abstract
Objective:
This study aims to evaluate the impact of a community development program meant to improve living standards of poor rural families through income generating activities (IGAs) based on conditional cash transfers (CCTs) in Doti, Nepal.
Method:
We use cross-sectional field data from a sample of 392 families representative of the village development committees of Pokhari, Ladagada, and Gajari. After running a propensity score analysis to increase comparability between the treatment and comparison groups, we compare mean scores on a series of chosen outcome variables via t-test analyses.
Results:
Results suggest that, although improvements in family income and living standards are felt subjectively, crop production might have worsened as a result of IGAs, suggesting the possibility of a trade-off and of long-term effects.
Discussion and Implications:
This article has implications for research and practice in community development programs and data collection and evaluation of such programs.
This study examines the impact of the Saemaul Zero Hunger Communities (SZHC) in community development program which took place in the district of Doti, in the Far-Western region of Nepal from the second half of 2012–2015. The program’s main goal was to ameliorate poverty problems at the village level (in the form of chronic hunger, lack of infrastructure, and high prevalence of diseases such as HIV), by engaging families into income generating activities (IGAs) based on conditional cash transfers (CCTs). Improvements on living conditions were expected over the short term (families’ need for food and for a decent level of livelihood) and the long term (strengthening livelihood assets and improving the efficiency of existing resources; Centre for Economic Development and Administration [CEDA], 2013). Since a sample of 392 families representative of the chosen areas has participated to surveys in 2013, after 1 year of the intervention, we focused on the short-term effects of the program only. These observational data have been statistically manipulated through propensity score analysis to assess whether the SZHC program had made a difference in improving levels of income and livelihood in the treatment group relative to the comparison group.
This study builds from a growing literature of intervention research, designed to examine a program’s purposeful impact on a specific target, at different levels (individual, family, group, community, etc.), in the presence of several intervention agents and contingencies (Fraser & Galinsky, 2010). In particular, community development programs involve “[…] collective social action toward solidarity and agency focused on a particular locality” (Green, 2016, p. 607) and aim for community change as “[…] the expected outcome of community development organizations (CDOs) programs and projects designed to improve the housing, employment, and health outcomes for people living in poor communities” (Dorius, 2011, p. 267).
Given the variety of programs that have been evaluated, ranging from interpersonal and psychological to psychosocial and community-based approaches in a variety of multidisciplinary sectors (Thyer, 2015), it is difficult to locate an overarching theory for evaluating all these programs. Instead, it is often the case that social and economic programs in developing countries tend not to pay attention to the real impact they have on their target populations, focusing instead on priorities set by donor agencies (such as nongovernment organizations [NGOs]; Bamberger, 2000), and CDOs might not comprehend the outcomes of their own interventions (Dorius, 2011). A shortage of theory-based evaluation and impact evaluation studies is indeed a challenge for social work program evaluation; nevertheless, assuming that a full understanding of the mechanisms of causality might just be an ideal to attain (white box), a focus on outcomes only, without knowing how causality works (black box), might be itself a worthy attempt (Scriven, 1999). Indeed, it is often the case that the evaluation of international development programs is conducted on a case-by-case basis, in the absence of a more general program theory (Dorius, 2011), and this trend is visible in many intervention programs led by social workers who have increasingly made use of the experimental method in recent years, resulting in several publications in academic journals from different disciplines, although randomization of program allocation has often not been possible, for practical and ethical reasons (Holosko, 2010; Thyer, 2015). A closer look at program evaluation from the interdisciplinary literature reveals that considerable experimental and quasi-experimental evidence on the effectiveness of international community development programs in poor communities has been published, especially for Latin America. These studies have shown that CCT programs providing incentives in undertaking health checkups and other desirable behaviors improve health outcomes and health prevention (Kumar et al., 2008; Lagarde, Haines, & Palmer, 2009) and children’s school attendance (Baird, Ferreira, Özler, & Woolcock, 2014; Benedetti, Ibarrarán, & McEwan, 2016; Diaz & Handa, 2006) and that microfinance programs strengthening families’ savings help to induce savings for the education of adolescent youth and in decreasing depression levels among AIDS-orphaned youth in Uganda (Ssewamala, Karimli, Han, & Ismayilova, 2010; Ssewamala, Neilands, Waldfogel, & Ismayilova, 2012). More crucially for our study, experimental and quasi-experimental evidence suggests that CCT programs and agricultural training help in increasing food expenditure (Diaz & Handa, 2006), investment in agricultural activity (Soares, Ribas, & Hirata, 2010), and household income (Becerril & Abdulai, 2010), thus contributing to reducing poverty in vulnerable communities.
This study aims to evaluate the impact that the SZHC development program has had for region of Doti, in Nepal, characterized by poverty and income vulnerability. Although experimental and quasi-experimental evidence is easily found for Latin American countries and health-related issues, welfare and community development in Asia are less covered topics, and we wish to fill this gap. Rural areas in Nepal are extremely vulnerable to the problem of food shortage, for which families’ coping strategies might result in seeking for loans or migrating to India for seasonal temporary work. Women might be particularly vulnerable in that they work considerably but they do not own land nor have strong decisional power within the family. In these villages, the factors that lead to food production deficits include lack of basic infrastructure, difficult access to agricultural input supplies, limited working capacity due to lack of farm equipment, lack of agricultural expertise, and frequent migration of manpower (CEDA, 2013). Since the SZHC program includes a CCT conditional on the participation to public work programs, and training for agricultural and livestock production activities, we expect that the villages which were part of the intervention group showed better posttest results for agricultural and livestock production, income from nonfarm work, and general income and living conditions, compared to the comparison group.
The relevance of this study can be described in several ways. First, by manipulating observational data with a quasi-experimental research design, it attempts to overcome the limits of internal validity common to many social work outcome studies (Holosko, 2010). Second, it addresses the problem of poverty and production sustainability in poor rural communities in Nepal, thus extending the evidence beyond health-related outcomes and Latin America. Finally, this study advances our understanding of the impact of the program by increasing the comparability of the outcomes of intervention and comparison group based on several specifications based on the propensity score (PS) technique including nearest-neighbor (NN) matching, specifications of the caliper, and the inverse probability weights, providing more potential evidence for the robustness of the results.
The following sections are organized as follows: first, a description of the SZHC development program is presented, followed by an overview of the research design. In the third section, results are presented, and in the final part, a discussion is presented, and implications for social work research discussed.
The SZHC Program
The SZHC project is a rural community development program conceived in collaboration between a United Nations (UN) agency and a South Korean governmental aid agency and implemented by several partner organizations with one international relief and development NGO as the main implementing partner. It has been implemented in Nepal as a 3-year project (2013–2015) assisting 1,772 households in two highly food insecure village development committees (VDCs), Ladagada and Pokhari of the Doti district located in the Far-Western Development Region of Nepal (CEDA, 2013).
The Far-Western region of Nepal is located higher in altitude than other parts of the country. Road accessibility is visibly lower in most parts of the Far-West than other regions, and the region is known to be lagging behind in socioeconomic standards. Food security issue especially in this region has repeatedly been on the country’s development agenda. Therefore, many development projects in Nepal, domestic and foreign, are targeting this region. The partners wanted the project to address the infamous food security issues in the remote areas of the country with the best resource each can contribute: the Korean donor agency with the knowledge of Saemaul Undong (New Village Movement), the UN agency with its cash transfer program, and the implementing NGO with its community development know-how. The agencies compared their selection criteria and agreed on selecting Doti district located in the Far-Western region, as the project location.
The primary feature of SZHC consists in its CCT programs, aiming to improve employability of residents by involving them in the construction of infrastructure to improve their livelihoods and income levels—an approach, this, that can be found in several other community development programs in low- to middle-income-level countries (Deeming, 2013; Lagarde et al., 2009). What makes SZHC an interesting case in the Asian region is the cooperation with the South Korea–based NGO and its reliance on a model of community development modeled after South Korea’s economic growth, also known as the Saemaul Undong (New Village Movement), which is based on ideas of self-reliance, diligence, and cooperation in the direct construction of infrastructures in rural, less developed areas of the country (Ringen, Kwon, Yi, Kim, & Lee, 2011). Among 13 food insecure VDCs in the region of Doti, Ladagada and Pokhari were chosen as at-risk target communities based on the donor agency and the partner UN agency’s selection criteria (food unavailability; lack of basic infrastructures; instable household income given by high rate of seasonal migration to India; high school dropout rate; prevalence of HIV, poor female health status-, gender-, and caste-based discrimination; geographical proximity of the two VDCs; and proximity to roads for future market promotion). Both VDCs became entirely eligible to the provision of the SZHC development program, due to their high vulnerability.
The needs assessment and feasibility studies in the project design phase revealed many socioeconomic problems the people of the two villages were facing. Other than the outmoded low-yield subsistence farming by which the great majority of the households depended on, few economic opportunities or social safety nets were available in the villages. Many households reported their agricultural production was not enough even to feed the family throughout the year. Most, if not all, households were suffering from protracted lack of staple grains and children were showing signs of chronic malnutrition. Most households had male members of their family seasonally working outside the district, mostly in India or overseas. Remittances sent by these family members were a crucial source of income that these families depended on.
Problems also persisted in other noneconomic aspects of the community life, conspicuously in education and public health, due mostly to insufficient provision of public services. Students were chronically underachieving in schools, and the proportion of school-age children discontinuing primary or secondary education was higher than the national average. The number of HIV/AIDS patients was proportionally very high in these communities. In a typical case, the husband would contract the virus from brothels while working overseas, mostly India, and spread it to his wife upon returning home. Inadequate provision of public services was failing to sufficiently address these issues in the communities. With most households living under the poverty line, it also seemed difficult to foster an alternative way, perhaps private service market, to substitute for the insufficient public provision.
With most households depending on agriculture, immediate objectives to deal with the problems in these communities would, therefore, involve strengthening agricultural production. Constructing drinking water facilities, irrigation canals, motorable roads, and so forth, would help improve the agricultural outputs immediately and give incentives for agricultural development and other ways to generate income.
In order to improve food shortage and living conditions in a composite way, SZHC consists of a series of programs ranging from the improvement of physical infrastructure (such as installation of basic water supply systems, construction of community centers, and improvement of farm-to-market roads) to the start of income-generating projects (aimed either at improving agricultural productivity or at increasing income through nonfarm-related activities) at the community level. The total number of programs on which take-up is measured is 19, 1 but the most popular ones, on which recorded participation score highest, were awareness training (on health, sanitation, and hygiene), education to improve seeds or crop varieties, and agricultural inputs such as introduction of fertilizer, provision of animals (such as livestock, poultry, or fish), health checkups, educational assistance for students and scholarships, building of handwashing and toilet facilities, and among all, food and cash for work programs (CCT), the most popular item. While the total number of residents is 9,157 individuals in 1,772 households, it has been estimated that the number of direct beneficiaries of the community development program is 8,917 (CEDA, 2013).
According to the intention of the community project, a positive impact of the program was originally expected in three areas of interest: – Outcome 1: Improved basic infrastructure and productive assets (in particular, schools and health institutions) – Outcome 2: Increased income status and improvements in living standards – Outcome 3: Improved access and delivery of basic services.
However, in consideration that field data have been collected when just 1 year had passed since the start of the community development program, we only focused on the attainment of the second outcome. The rationale for this is that a community development program focused on IGAs is expected to immediately bring relief to impoverished communities. This is more difficult to observe for Outcomes 1 and 3, since improved infrastructures and access to service facilities can only be observed within a longer span of time. More specifically, our research hypotheses are formulated as follows.
Data
The data used for this study were collected between May and July 2013 in Doti by an external agency for a final sample of 392 households: Of the 1,772 households in the selected areas for treatment (the VDCs of Ladagada and Pokhari), 298 were chosen via stratified systematic random sampling (16.7% of all beneficiaries), and 94 additional households were added to the sample from the neighboring VDC of Gajari as a comparison group. It is a natural grouping given by these communities’ geographical position, and this characteristic might somehow lessen the problem of self-selection, typically found in social work research (Barth, Guo, & McRae, 2008; see Figure 1).
Flow of participants. Treatment take-up for the treatment groups is calculated as the arithmetic mean of participation rates to the most popular programs. For a more comprehensive list of Saemaul Zero Hunger Communities programs, see Note 1.
The dependent variables observed were selected based on their relevance to the abovementioned community Outcome 2, relative to the improvement of income and living conditions. Specifically, included variables were the production of crops, vegetables, livestock, and animal products; the value generated from the sale of such products; perception of improvement in the production of each of these items; income generated from nonfarm work, perceived improvement in income related to nonfarm work, chance of finding employment; and total family income and perceptions of improvement in income and living conditions (CEDA, 2013). For a more detailed view, see Table 1.
Definition of Variables and Descriptive Statistics.a
aAll data are observed after the intervention, in that they have been collected only in one point in time, after 1 year had passed from the intervention. bRs.: Nepalese Rupee.
Analysis
Since data have been measured cross-sectionally, they correspond to a posttest only control group design (Campbell, Stanley, & Gage, 1963; Royse, Thyer, & Padgett, 2009), and PS analysis (Rosenbaum & Rubin, 1983) has been adopted as a conditioning strategy to improve comparability between the intervention and the comparison groups (Thoemmes & Kim, 2011). The representativeness of the samples to the VDCs was guaranteed by the stratified systematic random sampling procedure aimed to equally represent different populations from all wards and ethnic groups (CEDA, 2013). However, these observational survey data, originally collected for the purpose of reporting to the funding agency, could still be vulnerable to the selection bias problem (Barth et al., 2008; Lance, Guilkey, Hattori, & Angeles, 2014): Although participants’ self-selection was somehow less of an issue due to the geographical assignment of the community development intervention, and virtually no spillover according to preliminary data analysis, the problem of a higher likelihood for the more poverty-prone areas of Ladagada and Pokhari to be targeted for intervention, compared to the comparison group of Gajari, which might have been relatively more well-off, remained. In other words, it could have been problematic to assess the real impact of the program, even in the presence of representative data, in that intervention communities had to be compared to a neighboring community that was more affluent to start with, indicating the need to improve comparability between these two groups. We attempted to address the selection bias problem by directly modeling the probability of treatment assignment based on observable sociodemographic variables of our sample through a logit model; the logit score obtained (or propensity score of participation to the program) was then matched to the respondents in our sample to mitigate the selection bias problem (Barth et al., 2008; Rosenbaum & Rubin, 1985). It has been argued that these nonrandomized studies might still approximate the results of classic experimental studies for some quasi-experimental designs (Shadish, 2011); a good advantage of the propensity score matching method might improve the comparability of the treatment and comparison groups even in the absence of baseline data (Soares et al., 2010).
As suggested by Barth, Guo, and McRae (2008), we followed three steps for our propensity score matching (PSM) analysis. As a first step, we first modeled a logistic regression predicting the likelihood of being selected as a treatment group based on available sociodemographic characteristics: the age of the respondent, whether the observed family is nuclear or not, the type of caste, and the presence of washing soap in the vicinities. In the absence of longitudinal data, it was not possible to control for income levels at the baseline; therefore, the variables chosen for the calculation of the propensity score were as much as possible time invariant. The PSM was conducted using STATA 13/TEFFECTS based on NN matching within caliper with replacement. While the nearest neighbor method (NNM) consists of matching each treated individual with the control individual who has the closest propensity score, for the radius matching, a tolerance level on the maximum propensity score distance (caliper) is set to avoid the risk of bad matches (the radius matching is meant not only to use the closest NN within each caliper, but all the individuals in the control group within the caliper; Becerril & Abdulai, 2010; Caliendo & Kopeinig, 2005). As suggested by Austin (2011), the caliper width was set as .2 of the standard deviation of the logit of the propensity score. Since it is often pointed out that the best strategy is to use multiple confirmatory methods (Sosin, 2002), we also used a weighting strategy, namely, the inverse probability weights, which allow to keep observations in the sample by giving more weight to respondents with closer propensity scores. Standard errors are automatically corrected according to the Abadie–Imbens method (Abadie & Imbens, 2011).
Once the propensity score has been calculated, the next step consists in verifying whether the data are well balanced or not by checking the range of common support. The common support seems to be guaranteed well by our data, perhaps also due to the stratified sampling procedure taking into account the number of the wards and the ethnicity of the respondents during data collection. Our postestimation showed that there was no need to delete individuals outside of the range of common support, in that there were no observations falling outside of this area. If multiple tests indicate balance, there is a greater likelihood that covariates are balanced across treatment and comparison groups in the propensity score matched or weighted sample (Deheja & Wahba, 2002).
The final part of our analysis consisted in a series of t tests of mean differences between the treatment and comparison groups of our selected dependent variables. Considering the nature of the community development program, we aimed to measure the “intention to treat” and the differences between the targeted and comparison groups. To this purpose, we measured the average treatment effects (ATEs) for the sample within the range of common support, since it includes both the average treatment effect on the treated and the ATE on the untreated in the experimental group (Lance, Guilkey, Hattori, & Angeles, 2014). Preliminary analyses showed various degrees of participation to single programs and almost no spillover effects. Families also showed participation to several components of the SZHC program at once, so that belonging to the experimental group per se seemed to make a bigger difference rather than the participation to single programs alone.
Results
Following is a list of the dependent and independent variables used for this analysis. As displayed in Table 1, most dependent variables are related with the quantity and/or value of agricultural production (crops, vegetables and fruits, livestock, and animal products), nonfarm family income, total family income, and improvements of the situation measured as ordinal variables in three levels. Unadjusted data show that overall, the comparison group shows better performances in terms of crop and vegetable production; however, this difference might also be a consequence of the fact that the targeted VDCs for intervention were indeed the poorest and most vulnerable. In order to improve comparability between these two groups, a number of independent variables were chosen for the calculation of the propensity score from the sociodemographic variables available in the household data set: respondent’s age, family type, caste, and presence of soap in the vicinity. These variables were deemed to fit as criteria for discriminating the two groups, in that they were more likely to reflect preexisting differences between intervention and comparison communities, differently from income and production-related variables, which were more susceptible to change given by the community development program (see Table 1).
Among the chosen conditioning variables for the propensity score, caste and presence of soap in the vicinities were significantly different among the two groups at p < .05, and for whether the respondent belonged to a nuclear family at p < .1, showing that, among the chosen PS variables, caste and presence of washing soap were the ones for which the two groups differed the most. After running the PSM with the selected observable variables, the estimated bias was reduced by about 78%. More details on the PSM quality indicators before and after matching and the reduction of bias can be found in Table 2.
PSM Quality Indicators Before and After Matching.
Note. PSM = propensity score matching.
As the third and final step of our analysis, we ran a series of t tests for each outcome variable, after generating a propensity score using logistic regression. The outcome variables are all related to the short-term goals of reduced rural poverty and income insecurity: agricultural production, livestock, and animal products; nonfarm work; and general improvement in income and living conditions. All results seem to be quite robust across conditioning strategies (matching or weighting), indicating a certain consistency of findings. Following is a more detailed analysis of the results per research hypotheses.
Agricultural Production.
Note. NNM = nearest neighbor matching; radius = radius matching within a set caliper; IPW = inverse probability weights.
aGiven the general consistency of results across matching techniques, Cohen’s d has been calculated based on the NNM only.
*p < .1. **p < .05. ***p < .01.
Livestock and Animal Products.
Note. NNM = nearest-neighbor method; IPW = inverse probability weights.
*p < .1. **p < .05. ***p < .01.
Nonfarm Work.
Note. NNM = nearest-neighbor method; IPW = inverse probability weights.
aConvergence non achieved.
*p < .1. **p < .05. ***p < .01.
General Income and Living Conditions.
Note. NNM = nearest-neighbor method; IPW = inverse probability weights.
aIPW convergence non achieved.
*p < .1. **p < .05. ***p < .01.
Discussion
Overall our findings indicate that after controlling for the probability to be selected as an intervention group through calculation of propensity scores, short-term goals of reduced rural poverty and food insecurity have been only partially met and in a contradictory way. There is no evidence of improved crops’ production, with the comparison group actually faring better than the experimental group, both before and after matching; livestock and animal products also show modest results, although residents of intervention communities are strongly convinced of improvements in both sectors. On the other hand, it appeared that there were improvements related to nonfarm work, in that higher income and better chances of finding a job are reported for the experimental groups. However, total family revenue did not show statistically significant results, although, once again, respondents of the targeted communities reported higher levels of subjective improvement to this respect, in comparison to nonsubstantial change reported in the comparison group. It is quite likely that highly positive perceptions toward the community development program might have been driven by a desirability bias (Rubin & Babbie, 2013), in the hope that NGOs would keep delivering cash-for-work programs in the communities in the future as well. In our analysis, different specifications of the conditioning strategy (matching and weighting) resulted in coefficients and levels of statistical significance that are not too dissimilar from one another, which might provide some support for the robustness of the findings.
These contradictory findings might indicate a degree of trade-off that might have been generated from the introduction of CCT programs in Doti: Participation to IGA programs might have conflicted with previous farming activities, since the production and sale of crops showed poorer results for respondents in the experimental groups. A similar effect might have occurred for the sale of vegetables and fruits, although the zero productivity levels in this field reported from the comparison group of Gajari residents might reveal errors in measurement and, in any case, difficult comparability with our intervention communities. The difference seems to have been generated by participation to cash-for-work programs, which naturally resulted in higher nonfarm income; however, the ways in which this affects previous economic and farming activities, and how this program is going to impact the community over the long term, remain to be seen. In conclusion, results from the adjusted and unadjusted analyses suggest that the SZHC program did have an impact in positive perceptions from the intervention group and in improved nonfarm work; however, whether these works are going to benefit these rural communities in Doti for the future is yet to prove. This study is more of an illustrative one rather than a conclusive one, and a further analysis with follow-up data from the end of the program would be needed for a more thorough impact evaluation focusing on the programs’ long-term effects. Some implications for community program developers and professional evaluation might be summarized as follows.
First, the reasons for the occurrence of what seems to be a trade-off between the CCT and the production and sale of crops need to be further explored. Program planners should take into account a possible trade-off effect of participation to CCT programs and how this might infringe on preexisting production activities. Project reports show that the cash-for-asset activities focused mostly on constructing facilities to help increase agricultural production such as canals, ponds, and roads. It can be assumed that the program participants’ participation in the cash-for-asset activities might have interfered with their time spent on doing farm work, and at the onset of the project, the impact of the constructed facilities on the increase of the farm production is perhaps yet to be observed.
Second, when measuring income-related variables, loans and remittances should, whenever possible, be taken into account, for a fuller picture. Seasonal migration is a dominant phenomenon in the studied area, and household income relies heavily on the remittances sent by the family members working outside the country. Therefore, different income sources need to be separately studied to precisely explain the extent of the project’s effect on the increase in the household income.
Third, valid and reliable measurement for these community development programs should be implemented. It is quite possible that the real impact of the program has been bigger than what has been reported, in that there is a tendency for target populations of development programs to (1) have high expectations on the amount of help coming from the program (the already cited desirability bias) and (2) tend to systematically underreport their actual income values in order to be eligible to more benefits. Also, respondents of the samples on which our analysis was conducted appeared to be actively participating to only about half the SZHC programs. There might be a possibility of low take-up of the programs, which needs to be investigated and addressed. We tried to address this aspect by only looking at the differences between targeted and nontargeted communities, but we did not take into account the “dosing” of different degrees of participation to the single programs—a limit that is inherent to PSM analysis, which only models the likelihood of belonging to the targeted group, and does not take into account the dosage of the program (Barth et al., 2008). Another major limit of this study lays in the fact that the choice of variables to include for the calculation of the propensity score was less than optimal, in that we used available sociodemographic variables with primary field data, which were measured cross-sectionally, whereas a theoretically informed way of measuring participation to the program would have been desirable (Barth et al., 2008), and in general, the PSM technique requires a rich set of covariates (Diaz & Handa, 2006).
Implications for Social Work Practice
Community development programs are difficult to evaluate (Dorius, 2011), and data collection and analysis tend to be closer to the donors’ concerns rather than aiming to evaluate the impact of the program (Bamberger, 2000). Nevertheless, with the present study, we showed that it is still possible to analyze these field data with quasi-experimental adjustments, so to better address the problem of internal validity, which often limits inference in social work research designs (Holosko, 2010). The program assignment that characterized our data of poor rural communities in Doti has followed a series of geographically measured vulnerability characteristics. In consideration of this, although we analyzed observational data, the program assignment followed a nonrandom mechanism other than self-selection, resulting in a research design that offers a better approximation to experimental designs (Shadish, 2011). The sample we used was already representative of each VDC that has been selected according to a randomly stratified procedure in the absence of spillover, and, with our PS analysis, we attempted to improve comparability a step further.
The present study is particularly important in that it adds additional evidence on whether programs based on the CCT logic function differently in different regional and country contexts (Baird et al., 2014), making use of primary data which was originally collected without a specific evaluation design planned, as it often happens in field operations. This shows how the PSM method can be usefully applied with routinely collected field data, allowing researchers to achieve a greater degree of scientific rigor about program outcomes. We believe that such data utilization opens up promising ways for intervention evaluation in social work research. We also believe in the moral obligation to share research findings, even when results might be null or negative, in consideration of the importance of learning through evidence in order to improve social work practice (Royse, Thyer, & Padget, 2009, pp. 244–245) and of the diffused practice of this kind of reporting in experimental research (Des Jarlais, Lyles, Crepaz, & the Trend Group, 2004). Since causation of complex programs might be difficult to disentangle, it is important to at least distinguish the different aspects of the intervention that yielded effects in the expected direction.
An obvious limit of this study, and a task to address in future research, is the lack of a time dimension with our data, which has been measured only cross-sectionally, at one point in time. Taking into account the time dimension may allow researchers to make better informed inferences and more nuanced analyses on the direction of the causality between the treatment and the outcome, responding to one of the main criteria for determining causation (Hill, 1965). Ideally, an improvement of the research design for impact evaluation might be attained provided that a collaboration between professional evaluators and field practitioners is in place. Nevertheless, this study showed that even an analysis based on primary field data, which was not collected for evaluation purposes, is indeed possible.
Footnotes
Acknowledgments
We wish to thank Sojung Park, Yoonsun Han, Yiyoon Chung, and In Sik Min for their helpful comments on earlier versions of the article.
Declaration of Conflicting Interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF-2016S1A3A2923475).
