Mixed-method approaches in child-sensitive social protection evaluations: The promises and pitfalls

Introduction – impact evaluation

Over the past decade there has been a ‘sea change’ in the role of evidence informing social policy design and implementation. It is often considered that some research designs are more valid and reliable than others in their ability to answer evaluation questions on the effectiveness of interventions. Standard experimental approaches such as randomised control trails (RCTs) are often heralded as the ‘gold standard’ of social policy evaluation, and one of the most rigorous methods to draw causal inferences and inform social policy (Deaton and Cartwright, 2018; Samson et al., 2015) often encouraged by funding. While post-positivists recognise the risks of essentialism, they fail to address the complex and complicated nature of sociological research and programming resulting in a ‘soft essentialism’ (Holliday and Macdonald, 2019: 1).

There is often an underlying assumption that experimental, quantitative and positivist research methods are more internally valid and reliable than qualitative methods often neglecting alternative external factors that have an impact on programme results and outcomes (Samson et al., 2015). Experimental designs using quantitative methods are centred around culmination outcomes, where analysis is confined ‘to accounting for the consequences of determined inputs’ (Samson et al., 2015: 1). Such evaluation designs cannot fully grapple with the complexity of social contexts and the inherent heterogeneity and unpredictability of human behaviour. Even if we do accept a deterministic approach, there are compelling reasons to recognise that it is not only unreasonable but also impossible to make accurate predictions concerning human behaviours. There are external factors, such as political, geographical, social and economic, that shape individual lives and realities that are complex, multifaceted and multidimensional.

Child-sensitive social protection: complicated–complex nexus

Despite the limitations of experimental designs, based on our analysis, the majority of the child grant programmes evaluated use primarily quantitative approaches, specifically RCTs typically targeting households to assess impact on child indicators (Owusu-Addo et al., 2018). While these evaluations provide some important insights on programmatic impact, studies have shown that merely assessing household status does not necessarily account for the impact of the programme on children.

There is some evidence indicating that well-designed and targeted child-sensitive social protection (CSSP) programmes can equitably address many of the dimensions of children’s well-being. In order to unpack and address the multi-dimensionality of child poverty, CSSP programmes and evaluation frameworks must grapple with what is a simultaneously complex (multiple strands, multi-site and multi-level) and complicated (emergent and uncertain) set of problems (Glouberman and Zimmerman, 2002).

OPEN IN VIEWER

Complicated–complex dimensions of CSSP: 1. Heterogeneous target outcomes within individual interventions; 2. Highly varied partnerships across multiple sectors/organisations; 3. Time pressures to achieve short-term results while ensuring long-term impact; 4. Wide scope, often trying to drive changes in entrenched behaviours/norms simultaneously at the individual, household and systems level; 5. Different theories of change within one intervention/programme

CSSP: child-sensitive social protection.

This complicated–complex nexus means CSSP programming and evaluation is dealing with outcomes that are often embedded in highly nuanced relationships, such as those found between stunting and/or parental stress and households and children experiencing multiple deprivations. This multi-dimensionality of poverty and its evolving and changing relationship with a host of child-well-being outcomes is difficult to unpack and cannot easily be captured with any predesigned and prescribed methods. There is no ‘one-size-fits-all’ approach that can be employed for research and evaluation for CSSP programming. Human beings do not inhabit an ordered world, but instead human realities and experiences are subjective, messy and full of inequities. This is particularly important for children whose voice is often not heard and taken into account when assessing impact and developing future programme designs. Therefore, CSSP programming should incorporate and engage with the subjective experience and voice of children at all levels of design, implementation and evaluation.

Reflexivity and emotions in research

A demand to focus on subjective experience is not a new arena, several qualitative and feminist studies rely on heterogeneous experiences of the research subjects moving beyond incomes and narrow human development indicators such as the Human Development Index and focusing on individual’s experiences and feelings (Mahendru, 2010). Documenting and reporting children’s emotional experiences could help us unpack the power structures that exist in their lives and inform social protection designs so they are better tailored to meet their needs. Subjectivity assists in observing the epistemological and methodological assumptions and challenges associated with the idea of assumed objectivity in the field.

CSSP programme implementation and evaluations must be reflective processes that consider power relations as an important dimension for carrying out sociological inquiry affecting children’s realities. Giddens (1984) describes humans as reflexive beings capable of reflecting on their own actions and acting in the world to bring about both individual and social changes. This reflexive notion is the very means through which social structures could be elaborated and modified and can create opportunities for individual agents to define their new evolving realities. Qualitative research includes reflexivity and human interaction and provides honest accounts using thick description and a growing awareness of methodologies discussing the major challenges, approaches and stages of the evaluation. This reflexive approach to theory of change and evaluation in CSSP programming can provide space for children to directly inform the application of more child-sensitive processes and methodologies.

The complicated–complex nexus described above characterises CSSP programming and presents significant opportunities for reflexive evaluation. We can expect the socio-political system that encompasses CSSP programmes to change over time, and therefore the intervention logic, its components and potentially some of the goals may need to be adapted during programme implementation. However, linear logic models in traditional evaluations assume a relatively stable intervention process and context and single theory of change, and often do not account for the diverse perspectives of multiple stakeholders, including both adult and child beneficiaries who can co-create knowledge as collaborators rather than seeing them as passive subjects of a programme or an evaluation (Mahendru, 2010).

Mixed methods: adaptive and responsive evaluation methods

While Samson et al. (2015) emphasise that experimental evaluations are too narrow and limiting, and that they should be complemented with the inclusion of mixed-methods approaches, this runs the risk of being overly prescriptive and catering to satisfying donor demands rather than what the evaluation requires (Holliday and Macdonald, 2019). Holliday and Macdonald (2019) argue that a mixed-method approach is driven by a neoliberal agenda and ‘does not address the subjective implicatedness of the researcher’ (p. 1).

While there is ample literature critical of the limitations of quantitative methods, Samson et al. (2015) still emphasise experimental evaluations, which encourage safer, more quantifiable approaches to data collection and can serve a post-positivist’s agenda. This kind of purposive selection to research methods is cherry picking and undermines the legitimacy of the required research design and its outcomes. It is in danger of severing a neoliberal and post-positivist agenda, as there is no prescribed comprehensive evaluation framework that can predict outcomes and ensure a successful delivery mechanism.

This kind of purposive selection saves researchers from working out what they need from what emerges from the setting itself. All research should be able to employ whatever methods are required to get to the bottom of what is going on in a given setting or context. However, mixed methods as a separate and prescribed methodology is not only dangerous but also unjust. It can stifle certain voices or prevent the researcher from collecting rich data.

Contextually designed research methods have the ability to provide rich data that can provide a more grounded understanding of the various pathways and their outcomes to better inform CSSP programming and polices. For example, in her own work, Ritu used mixed methods in 2014 to research realities of girls who faced sexual and gender-based violence in Afghanistan. Ritu used questionnaires to ask sensitive questions that also helped gain access to the community and build relationship with the participants because that was what the setting required rather than seeing quantitative methods as a requirement. There is no prescriptive and systematic approach to evaluation and research design that can provide methodological certainty. As practitioners, we need to be able to unpack the mechanisms behind a programme in order to understand why certain things manifest and individuals behave in certain ways. This can only be done by immersing yourself in the research setting and understanding what the research requires rather than suggesting that research is not sufficient without quantitative add-ons.