OPALesence: Epistemological pluralism in the evaluation of a systems-wide childhood obesity prevention program

Abstract

The evaluation of complex systems-wide public health interventions requires evaluation research that is underpinned by theory. This article presents and discusses the trans-disciplinary evaluation research framework developed to support the evaluation of a South Australian program called OPAL (Obesity Prevention and Lifestyle). The aim is to provide insights into the research design, methods and implementation of the evaluation and contribute to the debate on how to evaluate community-based interventions with complicated and complex aspects. In an attempt to capture the complexity of childhood obesity and the intervention, the OPAL evaluation research employs post positivist, interpretive and critical epistemologies, valuing epistemological pluralism. Each component of the multi-phase mixed methods evaluation captures different yet complementary information concerning the context, process, cost effectiveness and outcomes providing a more complete understanding of the impacts of the complex program. Evaluation research is not without challenges. Some of the tensions and challenges that arose in the establishment, planning and conduct of the OPAL program and evaluation are discussed.

Keywords

childhood obesity prevention complex systems epistemology evaluation research methodology

Introduction

Theory-oriented approaches to evaluation have emerged as a way to unpack the ‘black box’ of programs, as compared with their method-oriented counterparts (Stame, 2004). The black box is the space between the actual input and the expected output or outcomes of a program, explaining how programs work (Astbury and Leeuw, 2010). O’Brien et al. (2010) used the term ‘evaluation research’ to describe evaluations that are well-resourced, rigorous and lengthy compared with ‘quick and dirty’ evaluations. Evaluation research can be further distinguished from method-driven evaluation as it requires identification of the epistemology or theory of knowledge underpinning both the intervention program, or program theory, as well as the evaluation research.

Three problem typologies – simple, complicated and complex – have been used to explain differences between aspects of interventions and program theory in evaluation (Funnell and Rogers, 2011; Ling, 2012). Stame explains that:

complex programs are non-standardized and adaptive, ‘emergent’ in response to changing needs, opportunities and understandings of what is working; they are implemented by multiple organisations with emergent and unpredictable roles; generalizations rapidly decay; and results are sensitive to initial or starting conditions as well as to context. (2010: 374)

Logic models and systems dynamics models can visually represent these complex or emergent aspects of community-based programs (Funnell and Rogers, 2011). Rogers (2008) recognises the limits of using a linear logic model when applied to complicated (multi-level and multi-site) or complex programs. She suggests in complex community-based initiatives the need for a more flexible theory of change, where an initial logic model is developed and used to guide planning and implementation but revised as plans change, to accommodate emergent and adaptive components of complex programs (Rogers, 2008). Traditional logic models are useful when designing evaluations for simple or complicated programs, however complex programs, as Reed and Harvey argue, ‘defy standard positivist canons of description, prediction and explanation’ (Reed and Harvey, 1992: 359). Critical to these debates is the importance of ensuring that suitable responses in both intervention and evaluation are sought to match the complexity of the social problems (Finegood et al., 2010). Evaluations of complex real-time programs offer myriad challenges for evaluators, including making explicit the theory of change and uncertainties and collecting and analysing data on the uncertainties (Ling, 2012). As solutions to evaluation complexities, Finegood et al. (2010) suggest establishing cycles for continuous improvement, matching evaluation capacity to complexity and recognising context; and establishing multi-disciplinary, multi-sector teams.

The purpose of this article is to present and discuss the comprehensive evaluation research framework used to evaluate OPAL (Obesity Prevention and Lifestyle), a community-based childhood obesity prevention program in South Australia. Childhood overweight and obesity in Australia is a pressing public health concern (Olds et al., 2009). In 2011 the National Health Survey found that 25.3% of children, aged 5–17 years, were overweight or obese in Australia (Australian Bureau of Statistics, 2012). OPAL aims to increase the proportion of children and youth (0–18 years) in the healthy weight range by improving the eating and activity patterns of children. The program is delivered within a context of National and State level public policy and programming all of which work towards reducing obesity. These policies and programs include kilo-joule labelling of foods in food chains; approaches to cross government partnerships on the social determinants of health such as ‘Health in All Policies’; the National Partnership Agreement for Preventive Health funding for ‘healthy community’ and ‘healthy workers’ programs and the introduction of legislation such as the South Australian Public Health Act (2011) where local government is also responsible for public health. The Act establishes a system of public health planning, including the development of a State Public Health Plan, with complementary regional public health plans by Local Councils.

This article aims to contribute to the dialogue within the evaluation community on how to evaluate community-based interventions with complicated and complex aspects and strengthen the evidence base for such approaches. Epistemological pluralism and inter-disciplinary research are used to offer a more thorough understanding of the evaluation of childhood obesity prevention. This article has been written in the sixth year of an eight-year program and is positioned as a methods paper providing insights into the research design, methods and implementation of the evaluation.

This article has four sections. The first section discusses previous evaluations of childhood obesity prevention programs and identifies the implicit epistemology or theories of knowledge. The second section introduces the OPAL program and identifies its program theory. The third section describes the different components of the OPAL evaluation, making explicit the epistemologies. The final section discusses the challenges of epistemological pluralism within the OPAL evaluation, and draws on the benefits of adopting a dialectic stance (Greene, 2007) in which paradigmatic differences are brought together synergistically to provide a comprehensive evaluation of a program with complicated and complex aspects.

Evaluation of childhood obesity prevention programs

The Foresight maps of obesity systems have gone some way toward unpacking the complex multi-faceted aetiology of obesity (Butland et al., 2007). Responses have included a range of interventions (i.e. educational, health promotion, community-based, behaviour therapy/counselling/management strategies) based in settings such as family, schools, early childhood and community (Waters et al., 2011). The Cochrane review of evaluations of childhood obesity prevention interventions (primarily simple or complicated programs such as school-based, diet and activity focused and underpinned by behaviour change theory) reported modest effect sizes on body mass index (BMI) from these interventions (Waters et al., 2011). Waters et al. (2011) suggest that future evaluations should determine the effectiveness of environmental and population level interventions (found in programs with predominantly complicated and complex aspects), to assist in the development of the best possible portfolio of interventions at the population level.

A number of recommendations have been put forward to improve the evaluation design of childhood obesity population interventions. These include the need for evaluations to use both quantitative and qualitative methods to capture environments and behaviours together with objective outcomes such as BMI, and to be administered over long periods of time and across population groups (Lobstein et al., 2004). In addition, minimum requisites for increasing the quality of any evaluation have been identified: use of quasi-experimental designs with the inclusion of intervention and comparison groups; analyses of outcomes by demographic variables; detailed descriptions of key intervention strategies and their intensity (Summerbell et al., 2005; Swinburn et al., 2007; Thomas, 2006; World Health Organisation, 2008); inclusion of a cost-effectiveness component; focus on equity impacts, potential harm, sustainability and implementation factors; inclusion of process, output, impact and outcome evaluations and translational approaches (Pettman et al., 2012; Wang et al., 2015; Waters et al., 2011).

Evaluations should also be explicit about the epistemology used to understand, interpret and explain the outcomes of programs. The epistemologies or paradigms underpinning childhood obesity prevention research and/or evaluation are usually implicit in the type of evaluation design applied. While not explicitly stated, the majority of childhood obesity prevention programs evaluations could be classified as post positivist. Post positivism has been the dominant paradigm in evaluation research (Neuman, 2000) and arguably the evaluation of obesity prevention programs which emphasise experimental and well controlled quasi-experimental evaluation designs as reflected in current hierarchies of evidence (National Health and Medical Research Council, 2009).

OPAL intervention and program theory

OPAL is the acronym given to one of the largest financial investments in childhood preventive health in South Australia. OPAL is based on the French EPODE methodology which includes four pillars: political commitment; partnerships or resources; social marketing and support services; and scientific monitoring and evaluation (Borys et al., 2012; Van Koperen et al., 2011). Opal is a gemstone mined in Australia. Opals are found deep below the red earth and when mined display a richness and beauty and an array of colours from white, red, blue and brown to deep black. As an opal is viewed from differing angles opalescence occurs whereby the colours change and are dispersed. This is a useful metaphor for considering the ways in which ‘epistemological pluralism’ is informing both practice and evaluation.

The OPAL program brings together three theoretical approaches to social and behavioural change. In OPAL the social marketing theory of behaviour change (Williams and Weir, 2013) assists in bringing about behaviour change to the individual and family through adoption of the social marketing benchmark criteria of: customer orientation, behavioural goals, insight, theory, segmentation, exchange, competition and intervention methods mix (French and Blair-Stevens, 2006) produced at the State and delivered at the local levels. Community development theory of action and change engages the organisation/settings at the community level, with ‘community’ defined diversely as locality, setting, interest or age group, and through a process of becoming (Ife, 2002). Finally ecological systems theory (Bronfenbrenner, 1977) brings to the foreground multiple systems interacting across the social world including microsystem (relations between person and immediate environment e.g. home, school), mesosystem (system of microsystems particular to life-stage), exosystem (extension of mesosystem, embracing other social structures including major institutions of society e.g. neighbourhood, mass media, transportation) and macrosystems (overarching institutional patterns of the culture or subculture, such as the economic, social, educational, legal, and political systems).

An ecological systems approach recognises multi-causal roots to the issue and shifts the emphasis away from the individualistic and reductionist biomedical equation of ‘energy in equals energy out’ to recognise the interdependencies between eating and activity systems that traverse society, such as food production and food security, physical environments and the impacts of planning regimes and regulations. Ecological systems theory recognises the interdependencies, interrelationships and feedback loops between obesity systems (Davison and Birch, 2001). Importantly it is the concepts of reciprocity, multiple systems and sub-systems and relations between systems that provide complexity.

Whilst the three program theories are complementary in the level of the target of the proposed change, there are tensions in bringing them together in program delivery. The social marketing approach is implemented in the community at the micro and meso levels but is developed at the exo-system level by marketing and subject matter experts. This has resulted in disjuncture between the messages that were developed at the State level and their relevance with the local context as identified through the community development approach. For example the social marketing theme message ‘Water. The original cool drink’ promoting tap water consumption as an alternative to consuming sugar sweetened beverages did not resonate with one regional community for whom their tap water supply is piped into the community. At the height of summer the water from the tap arrives hot from the cold tap, not ‘cool’ as the message suggested. Further, in another community the messages about consuming fluoridated tap water did not resonate as their water supply was at the time not fluoridated, which was a point of community sensitivity.

The theory and method of intervention is summarised in the OPAL program logic (Figure 1). While presented linearly for ease of engaging with stakeholders and the community about the intent of the program, the change processes are not considered to be linear. At the local level, the planning, implementation and evaluation cycle is best represented as a structured process with ongoing feedback and critical reflection built into the cycle to strengthen practice. The logic model depicts the direction in which the resources, inputs, activities and outputs are hypothesised to bring about changes in the environment, behaviour and body weight of children. It provides a road map for program stakeholders to understand in a simple and clear way how program inputs and activities translate into outputs, impacts and outcomes, over time. It also provides staff with shared clarity; a set of principles and processes to follow in planning, implementing and evaluating local projects (OPAL Collective, 2015).

Figure 1.

OPAL logic model.

While the OPAL intervention and evaluation were originally envisaged as a large, complicated intervention, in reality OPAL has taken on many characteristics of a complex intervention without being explicitly framed as such from the outset. The complex systems features of OPAL include feedback loops, emergence and a large number of parts. The complexity of the model is also captured in its interface with the OPAL ‘single platform’, an online real time data repository providing project management and an evaluation database where each component of the model is defined and embedded in the program planning, implementation and evaluation cycle. Further, three annual week-long training sessions were held with all staff and organised by the OPAL State Coordination Unit including staff reflections, content knowledge (eg evidence-based theme materials), evaluation and cultural fitness training.

The broader OPAL program recognises that whilst the planning processes by which inputs (i.e. funding, governance structures, EPODE methodology) are translated into community-level activities and outputs are common across the 21 OPAL communities, the nature of intervention activities implemented will vary by community. Therefore, the program will ‘look different’ across the 21 communities because of local variations in community characteristics and readiness to mobilise around obesity prevention, and in the social and built environmental factors that contribute to the development of childhood obesity. At a conceptual level the logic model lays out what the program is about and what needs to be done to achieve program goals.

OPAL program design and structure

OPAL has been funded by three tiers of government – Local, South Australian and National. OPAL has a federated structure with its maximum 45.5 full-time equivalent (FTE) positions. The State Manager, Evaluation Manager, Social Marketing Manager and project and administration support are located centrally at State level with Local Council Managers and Support Officers located in local government across the State. The intake of local council communities into the program was staggered and reached a maximum of 20 South Australian and one Northern Territory community. The first six communities were handpicked into the program based on need (high levels of social disadvantage, high rates of child obesity, high population of families and local government support), while the following 14 communities in local government areas applied and were selected to be OPAL communities based on the above criteria. Each community has a five-year commitment of staffing (teams of two – senior strategic manager and support project officer) and AUD$75,000/year operating budget (plus AUD$25,000 in kind support from local councils). Due to the size of some rural and remote communities in SA, communities with a total population of less than 10,000 people had commensurate staff (e.g. 0.8FTE) and operating budget. The larger metropolitan communities of population in excess of 30,000 people could apply to have a second site. The second sites (population 15,000–20,000) received the same operating budget with a full-time project officer and junior project officer managed by the senior strategic officer. As a result of National funding cuts in 2014, program duration has been reduced to 4.75 years, local staffing 1.5 FTE and local budget AUD$25,000 (with local variations to this base model). The OPAL program provides an example of a real-time complex program.

OPAL evaluation

The OPAL evaluation embraces epistemological pluralism to provide a more thorough understanding of obesity prevention from respective epistemologies. Mixing epistemologies is about more than mixing methods within a given epistemology. Any method (qualitative or quantitative) can be used, however they will be used differently and will ask differing research questions depending upon the choice of epistemology and the underlying views on ontology or the nature of being. Miller and colleagues describe ‘epistemological pluralism’ as a practice that ‘recognizes that, in any given research context, there may be several valuable ways of knowing, and that accommodating this plurality can lead to more successful integrated study’ (Miller et al., 2008: 46). Epistemological pluralism, in their view, creates new ground leading to a more complete understanding of the complexity of the issue (Miller et al., 2008). They further argue:

various insights and knowledge that separate epistemologies bring to bear on the problem may lead to a more complete understanding of the complexity of the situation, but that we can expect that the context and problems will change, prompting differing mixes of epistemologies. (Miller et al., 2008: 49)

A driver of epistemological pluralism is the need for evaluations to address the interests of multiple stakeholder groups. As part of participatory (Cousins and Earl, 1992) and developmental (Patton, 1994) approaches to evaluation, key stakeholders provide input into shaping the evaluation. The evaluation objectives of OPAL vary according to the specific needs of different stakeholder groups and include: documenting the level of success in achieving the program aim to increase the proportion of children in the healthy weight range for State Government; increasing the proportion of children eating well and being active for National government; increasing the community capacity and wellbeing for local government; identifying areas for continuous program improvement; and measuring cost effectiveness to inform the strategic directions of local, State and National governments. Addressing these needs at the evaluation design level has necessitated invoking different methodologies supported by different epistemologies. With these stakeholders’ interests in mind, the result for the OPAL evaluation has been a multi-phase mixed methods approach, whereby several qualitative, quantitative and mixed methods projects are conducted over time, with the focus on a common purpose for the multiple projects (Creswell, 2014).

The OPAL evaluation includes both internal and external funded research components, including competitively funded research grants with universities that add considerable value and extend the evaluation scope. The evaluation is guided by a Ministerial appointed OPAL Scientific Advisory Committee (SAC) whose membership is comprised of experts from the areas of clinical nutrition, physical activity, education, epidemiology, paediatrics, public health, endocrinology, health economics, Aboriginal health, epidemiology, social marketing, community development and psychology. The evaluation manager, located within the OPAL State Coordination Unit, manages the OPAL evaluation and budget of 12 percent of overall program funding. The evaluation budget does not include the additional funds gained through external research grants. The role of the internal evaluation manager is to: develop the overall OPAL evaluation framework; conduct parts of the evaluation; manage the externally contracted components of the evaluation; train OPAL staff in evaluation building local evaluation capacity; facilitate and support data collection; supervise students; partner with Universities to seek further research and grant opportunities; and translate and disseminate research findings. PhD and Honours research projects further complement and extend the OPAL evaluation, building ongoing evaluation research capacity.

The OPAL evaluation framework is based on the previously mentioned OPAL program logic. It reflects on the input from experts from a number of science and social science disciplines and integrates multiple types of data. The primary types of data collected to suit the evaluation research design include: context, process, outcome and economic (see Table S1 in Supplemental Data on http://evi.sagepub.com/supplemental).

Context

The context component of the evaluation involves the examination of the cultural, built and political environments within which the intervention operates. The context provides complementary information on what factors affect program implementation and how space and the built environment influence the enhancement of the program (Trojan et al., 2009). Context is important as it provides insight into the other activities within the systems that may impact on the outcomes that are the target for change (Hawe et al., 2009). Consistent with the socio-ecological theoretical underpinning, and depending on the nature of the intervention, what is contextually significant may not only relate to place but also to systems of interpersonal and social relationships, and even to biology, technology, economic conditions and so on (Pawson and Tilley, 2004). As part of the evaluation of the healthy weight context, pre and post changes to the hard (infrastructure) and soft (policy and program) environments are monitored using geospatial information systems (GIS).

An assessment of community readiness is also being undertaken where subjective perceptions of community members are integrated with ‘harder’ objective indicators of readiness including number of FTE positions for obesity prevention in the five years preceding OPAL and the number of state-wide obesity prevention initiatives.

To evaluate stakeholder engagement and partnerships within the OPAL communities a series of qualitative focus groups and interviews are being undertaken in the first year and final year of program operation, using an interpretive approach (see Table 1). An interpretive approach supports the belief that ‘social life is based on social interactions and socially constructed meaning systems’ (Neuman, 1997: 69) and recognises that all stakeholders in a program may not experience it in the same way. The stakeholders are asked how they understand healthy eating and physical activity might be improved in their community, what they anticipate might be the local barriers and facilitators for change, which groups within their community might be hard to access and what worked for whom within their local community.

Table 1.

OPAL evaluation research designs – Epistemological pluralism.

Epistemology	Nature of Hypothesis / Research Question	Examples in OPAL evaluation	Methods used in OPAL	Nature of approach	Sampling	Nature of findings
Post positivist	Seeking to accept or reject scientific hypothesis	Use of repeat cross sectional measurements to assess intervention effectiveness within a quasi-experimental evaluation design	Child anthropometric measures	Objective	Representative sample	Representative
Interpretive	Seeking subjective meanings of the experience of the social world	Qualitative – pre and post intervention communities only	Focus groups and interviews with OPAL staff and stakeholders, using semi-structured questions in first and fifth year of program	Subjective	Purposefully selected	Generalisable
Critical	Seeking subjective meanings of the unique experience of the social world due to culture, gender or class	Qualitative – ethnographic study in one OPAL community examining the lived experience of OPAL program – class and gendered analysis	Participant observation in family and community settings, including participant’s homes, public transport, shopping centres, food courts, community centres; In-depth interviews and visual mapping techniques (including photo-elicitation)	Subjective	Purposefully selected	Generalisable

Obesity and obesity prevention extend across systems and across continents. The OPAL evaluation recognises the global reach of the issue and the global sharing of expertise for obesity prevention, especially through its program origins, EPODE. For OPAL, gaining understanding of the transferability of the program from a French to an Australian cultural context was critical to understanding how it might impact the success of the program in Australia. A critical sociological review of the French and Australian political, cultural and social contexts for obesity prevention was undertaken as an independent research project (Hartwick et al., 2014).

Process

An evaluation of process provides insights into how programs are implemented and how implementation influences program outcomes (Rossi et al., 2004). The OPAL evaluation is tracking what was implemented, how it was implemented and the degree to which it was implemented in order to understand what worked and why. There are multiple linked parts to the evaluation of process.

A core component of the OPAL evaluation of process is to assess whether OPAL resources and inputs were translated into ecological activities and outputs for each goal area (see Figure 1). Linking the inputs with activities and outputs provides the architecture for the process evaluation. Each project delivered locally in the OPAL program includes an assessment of the extent to which it is ecological using the ecological coding grid (Richard et al., 1996; Richards et al., 2014). Transparency and testing of program theory is an important evaluative component in the evaluation of OPAL. In addition to determining the extent to which it is ecological, the OPAL process evaluation includes an assessment of program fidelity (Richards et al., 2014), adaptation, integrity (program fidelity + local adaptation), reach, dose and sustainability (see Table S2 in Supplemental Data on http://evi.sagepub.com/supplemental).

OPAL is underpinned by a set of principles including equitable and culturally sensitive service delivery (see Figure 1). A practice and research based sub-committee of the OPAL SAC has been established to support staff to ensure mainstream health services are delivered in a culturally sensitive manner to Australian Aboriginal people. A component of the evaluation examines the local adaptations made by OPAL staff to projects delivered locally to ensure they meet the needs of the Australian Aboriginal community. This information is collected and entered by OPAL council teams into the OPAL ‘Single Platform’. These indicators are used to provide immediate feedback to staff and stakeholders about how much and what types of activities are going where.

The OPAL evaluation includes opportunities for program improvement and modification through the use of guided reflective practice for OPAL staff (OPAL Collective, 2015). A component of staff training is participation in reflective practice or praxis to make improvements to the program as part of an iterative process. Staff are encouraged to share experiences and find shared ground and/or points of difference in their independent field work. This engagement provides opportunities for reflexive learning, self-evaluation and iterative program development (OPAL Collective, 2015). Researchers working on discrete research projects also return their results back to OPAL staff at the training sessions to inform and strengthen the design and delivery of local projects.

An evaluation of the strength of network ties between the systems can track changes in relationships and focus on the distribution and transfer of resources, and provides a means of measuring the capacity building networks over longer time frames for follow-up, recognising that change in systems does not happen linearly (Hawe et al., 2009). Reflection sessions with OPAL staff have identified the importance of relationships and networks in bringing about change. For example OPAL staff valued the importance of community champions to make changes across systems. Analysis of network ties across all communities within OPAL examines the role of networks, partnerships and collaborations. In addition, an interpretive case study analysis of community capacity evaluates the intricacies and experiences of capacity building over time of local community members and stakeholders (MacLellan-Wright et al., 2007). The latter also sheds light on contextual factors.

Outcome

An evaluation of outcomes assesses progress on the sequence of outcomes that the program is designed to address (National Center for Chronic Disease Prevention and Health Promotion, 2011). The outcomes are often described in a time sequence using terms like short-, medium- and long-term, or proximal (close to the intervention) or distal (distant from the intervention).

The effectiveness of OPAL is being evaluated using a quasi-experimental research design with ‘matched’ comparison communities. The primary long term outcome for OPAL is to increase the proportion of children in the healthy weight range. The methods include objective anthropometric measures on children aged 4–5 and 9–11 years (Leslie et al., 2015). This part of the evaluation fits within a post positivist epistemology (see Table 1). A post positivist approach values experimental (i.e. randomised controlled trial) and well controlled quasi-experimental evaluation designs which are supported by systematically collected and measured empirical observations (Creswell, 2009).

Questionnaires are used to measure medium-term outcomes in children’s eating and activity patterns and changes to the eating and activity environments within families (e.g. rules), schools, organisations (e.g. policy), communities and environments that assist in reinforcing behaviour change (See Table S1 in Supplemental Data on http://evi.sagepub.com/supplemental). In OPAL the short-term outcomes include changes in knowledge, attitudes and skills of children, parents and stakeholders that influence the medium-term behavioural outcomes. Short-term outcomes were measured using annual computer assisted telephone interviews with parents. These interviews provided immediate insight into the family’s behaviour, attitude and confidence to change as a result of OPAL’s annual social marketing themes.

Research funding was gained to explore short-term family outcomes using ethnographic methods adopting a critical epistemology. Critical approaches to research explore and unpack power relations within society, often making the invisible visible or giving voice to what is often left unsaid (Neuman, 1997). This research project investigated how low-income families in one OPAL community understand the health risks associated with obesity, and whether gender and social class intersect to influence responses to obesity intervention strategies delivered through an equity lens (Warin et al., 2015).

Economic

The OPAL evaluation includes an economic assessment of the cost effectiveness of the OPAL program. In addition to assessing the proportion of children in the healthy weight range, the evaluation includes a self-reported indicator of health related quality of life (the Child Health Utility 9D instrument) (Stevens and Ratcliffe, 2012). The CHU9D is scored using an Australian-specific adolescent general population tariff of utility values, facilitating estimation of the total quality adjusted life years (QALY) gains for the OPAL intervention and comparison communities respectively (Chen et al., 2014). Together with healthy weight status data, the quality adjusted life years gained will be coupled with information on the total costs of the intervention to calculate the cost effectiveness of the OPAL program (Weinstein and Stason, 1977). The outcomes will be used to inform policy-making decisions on resource allocation for public health and health-care interventions to prevent childhood obesity.

Integrative context-process-outcome

Variables and processes from the context, process and outcomes evaluations highlighted above will be integrated into a theory-driven context-process-outcome evaluation (Chen, 2005). This will unpack the extent to which the intervention is successful, and why it is successful (i.e. by testing for mediation) and will identify in which contextual circumstances OPAL is more or less successful (i.e. by testing for moderation) (Chen, 2005). This integrative approach to evaluation will allow results to be interpreted in relation to the integrity of implementation and the underlying program model. Implementation success should correspond with the activation of changes in the environments and behaviours hypothesised to impact healthy body weight in children and youth residing in OPAL communities (Chen, 2005). If implementation fails or the program model is suspect, then one would not expect to observe short or medium-term changes between OPAL and comparison communities. Ongoing prospective interviews with program staff will be integrated into the analysis to strengthen the basis for interpreting intervention effectiveness.

Discussion

The OPAL evaluation is classified as ‘evaluation research’ where high-level research is being conducted with program evaluation goals and contributes to the development of new knowledge while simultaneously addressing the interests of multiple stakeholder groups. Evaluation research attends to theory-based approaches to evaluation but also pays careful attention to the importance of knowledge generation and consideration of epistemological debates. OPAL brings together three theoretical approaches to social change (i.e. social marketing, ecological systems theory and community development), each informed by varying theoretical positions. These have been mirrored in the adoption of different theoretical approaches in the evaluation. Using the idea of ‘opalescence’, whereby light changes when it hits an opal’s surface, we consider the ways in which ‘epistemological pluralism’ is informing both practice and evaluation.

Epistemological pluralism

There are multiple understandings on the possibilities of epistemological pluralism or mixing epistemologies (Greene, 2007). Purists argue that post positivism and interpretive and critical epistemologies are based on radically different assumptions, beliefs and stances and so are not able to be mixed (Lincoln and Guba, 2000). In contrast, the dialectic stance purports that epistemologies are made up of interconnected sets of philosophical assumptions and that they themselves are social constructions (Greene and Caracelli, 1997) and so the epistemological boundaries are permeable. In the OPAL evaluation similar methods have been used within different epistemologies to provide insights into the issue of obesity prevention. Greene has described this as a complementary strengths stance, where multiple paradigms run in parallel (see Figure 2).

Figure 2.

Complementary strengths stance.

The post positivist quasi-experimental evaluation design is the primary driver of the OPAL evaluation. Despite arguments for broadening the evidence base for public health evaluation research beyond post positivist research designs (Swinburn et al., 2005), government continues to privilege experimental and quasi-experimental evaluation designs for accountability and justification of program continuation. This was made explicit through a review of South Australia’s Non-Hospital Based Services (McCann, 2012) which called for ‘positive and measurable impacts on population health’ and highlighted the ‘shortage of research evidence demonstrating intervention effectiveness; and the limited availability of validated utilisation data’. Following this review the OPAL program was one of the few surviving health promotion programs.

While a post positivist evaluation approach is viewed by government as providing accurate, value-free and objective knowledge on program effectiveness, multiple challenges have been experienced in applying this approach in the OPAL evaluation. For example it was problematic in defining and allocating ‘matched’ intervention and comparison communities within a finite population and reaching planned sample size targets to gain sufficient statistical power to detect change at the population level. The lower than anticipated response rates for the quantitative evaluation means that a greater effect size is required for the change to be detectable and statistically significant. It is also unlikely for population level interventions to see large effect sizes. Response rates also hindered the nature of data that were able to be reported back at the local level. Further challenges include possible respondent bias suggesting the sample is not representative.

A further limitation of using only a post positivist approach is that there are gaps in understanding the meaning of the program for participants and stakeholders. There are political risks in detecting no change and so the interpretive and critical components of the evaluation provide insights into organisational and community change and explanations as to why the change may not have extended to behaviour change.

An interpretive approach values subjective experiences whereby the participants construct their meaning in a particular context and time. Rather than testing theory, theory is inductively developed through interpretation of meanings by the researchers who recognise their own social position. The interpretive evaluation for OPAL seeks to explore, understand and explain stakeholders’ engagement with and barriers to engagement with the OPAL program and community capacity. The methods involved in interpretive approaches are generally more time intensive but add value to the objective data by providing meaning, adding local insights and experience. This is particularly important when interpretive data seek to understand and give meaning to the dynamics and processes of higher level units such as communities.

In the experience of the OPAL evaluation, interpretive approaches have been used to both complement and extend the quantitative data collection and act as stand-alone evaluation research. As part of the assessment of program integrity (fidelity plus local adaptation) qualitative interviews were used to understand and explain findings within the quantitative analysis such as the organisational barriers to implementation with integrity (Richards et al., 2014).

Researchers using a critical approach view the world from a particular value-base (e.g. feminist) and look for patterns in social phenomena (such as social class, gender and ethnicity) that may account for inequities (Creswell, 2009; Neuman, 1997). Critical approaches may deal with conflicting stories or experiences due to the imbalance of the social positions of research participants (Neuman, 1997). In the context of OPAL a critical approach examines knowledge and intervention uptake, and how a community defined as ‘obesogenic’ responds to such an intervention. The ethnographic research has honed in on some of the problematic aspects of the commonly held, taken for granted assumptions of health promotion. Childhood obesity prevention is based on the long-term view of health, where an investment in eating healthily and being active is made in childhood for the long-term future healthy adulthood. The ethnographic research uncovered how this long-term vision contrasts with the immediacy of the everyday lives of low income families. Planning for the future refers to a short-term vision of today and tomorrow, rather than years in advance. Differences in temporal experiences have been identified between the ‘future’ premise of childhood obesity prevention and the everyday lives of low income families in an OPAL community (Warin et al., 2015). Furthermore, some of the OPAL health promotion messages were initially framed in a way that asked the parents to reflect on their childhood memories and re-create the freedoms of their childhood for their own children. Feedback from the stakeholders suggested that some parents have had a childhood dominated by violence and abuse and that they would not want to re-create this for their own children, rejecting the premise of the messages (Warin et al., 2015).

Like interpretive paradigms, critical research does not set out to be representative but its findings (both empirical and theoretical) can be extrapolated beyond individual research participants to other ‘similar’ communities (Popay and Williams, 1998). Further the resources required to conduct a representative, prospective multi-site ethnographic study across all intervention and comparison communities is prohibitive but would be ideal from the perspective of understanding the causal mechanisms and understanding contextual factors and how to target and tailor intervention strategies to all groups, including hard to reach groups, within the community. A critical approach importantly identifies what is likely to be effective (and ineffective) in particular populations and contexts, and helps to reshape new interventions that might work. The focus for exploration is on everyday experiences of the program within these challenging socio-economic structures. Evidence about social practices cannot be isolated from the social context and findings can be used and fed back to empower participants.

Methodological and political issues and challenges

An implication of evaluating complex programs using epistemological pluralism is the need for an increased trans-disciplinary focus on evaluation design rather than just on choice of methods (Sanderson, 2002) and epistemology. It also requires high-level coordination, communication and commitment. The governance structures, including the OPAL Scientific Advisory Committee, together with the Evaluation Manager have allowed for this coordination at both the State and local levels. It has been crucial to ensure that there is no double-up in research questions, to assess the possible research burden across OPAL intervention and comparison communities, including negative impacts on OPAL staff, and coordinate ethics applications across multiple government and university ethics committees.

Important has been the allocation of time to engage in trans-disciplinary researcher discussion on the co-contributions that the various epistemologies make to resolving or understanding the social issue of childhood obesity. For example both ethnographic and GIS data were used in a study of community gardens valuing both methods and providing ‘diverse interpretations of the roles and effects of the gardens’ (Knigge and Cope, 2006: 2035). Where feasible data-sharing processes have been established to ensure maximum use of the data collected; this has included the consideration of confidentiality and privacy issues for participants.

This opalescent evaluation research has not been without challenges, both within each evaluation approach and in integrating the whole. A challenge in bringing the differing methods together is that the unit of analysis varies across the various epistemologies. For example the definition of community varies according to epistemology. In the quasi-experimental evaluation, for the purposes of statistical power and to adequately capture heterogeneity in intervention exposures and outcomes in children’s BMI, intervention and comparison communities were sub-divided into suburbs. The ethnography and the sociological study of transferability of EPODE, however, looked at the culture and norms of the broader community which is not disaggregated to the suburb level. The ethnography and measures of community capacity are measured using the participants’ own definitions of ‘community’ as the unit of analysis.

Furthermore there were many political challenges and sensitivities in managing an evaluation of this scale and scope. The program and evaluation received Government Cabinet support and the evaluation specifically required inter-departmental support. A bi-partisan long-term commitment has been required to undertake evaluations of this nature. The OPAL program was funded for eight years across several political cycles hence the political environment was constantly changing at the local, State and National levels. For example in the third year of the program State government operational budget cuts were required; this resulted in a scaling down in the scope of the outcomes evaluation. The OPAL SAC provided advice to the Department on the best approach to minimise impact on the final evaluation. Further, in the fifth year National funding ceased, which required modifications to both the program and evaluation.

There were also limits in how OPAL adopted a dialectic stance (Figure 3) whereby paradigmatic differences are used together to engage meaningfully with difference where enhanced or richer understandings of childhood obesity prevention could emerge.

Figure 3.

Dialectic stance.

Conclusion

Childhood obesity is a complex social issue. The OPAL program is theory driven (social marketing, community development, ecological systems) with complicated and complex aspects and works to reduce childhood obesity within communities in South Australia. There is a disjuncture between two theoretical approaches which provide challenges for program delivery.

The OPAL evaluation has been a multi-phase mixed methods approach, whereby several qualitative, quantitative and mixed methods projects are conducted over time. The primary driver for the evaluation of OPAL is post positivism and the evaluation framework has involved inputs from multiple stakeholders. This has resulted in a complex evaluation framework with multiple data and methods to address the interests and needs of these stakeholder groups.

The OPAL evaluation is based on a pluralistic epistemological framework, where each epistemology offers a perspective. The OPAL evaluation framework values epistemological pluralism. Epistemological pluralism creates new ground that requires more than one epistemology to enrich the understanding of the issue, leading to a more complete understanding of the complexity of the issue (Miller et al., 2008).

Epistemological pluralism can also assist in mitigating the limitations of each approach. There is value in including interpretive and critical epistemologies as they provide explanations as to why change may not have extended to behaviour change; however there are challenges to the implementation of such a methodology. There is a need for explicit discussion of the epistemology; and increased trans-disciplinary focus on the evaluation design, rather than just the methods. To achieve epistemological pluralism requires high-level coordination, communication and commitment.

The combined insights from epistemological pluralism offer a more thorough understanding of obesity from respective epistemologies; like opalescence allowing different perspectives to shine through. Complex obesity systems require the need for complex evaluations – long-term outcomes evaluations alone are insufficient to measure change, especially when change may be small at the population level.

There is richness in bringing together differing epistemologies; it brings diverse disciplines together, allowing for an understanding of the complexity of obesity from different angles. Like looking at an opal from different angles, the opalescence provides a new and rich perspective – differing colours but equally valued from whatever angle it is viewed.

Footnotes

Acknowledgements

Thanks to the members of the OPAL Scientific Advisory Committee and sub-committees for providing advice and guidance to the OPAL evaluation and to the Flinders University OPAL Evaluation research team. Thanks also to the OPAL staff team and stakeholders for their input into the development of the OPAL evaluation.

Declaration of conflicting interests

Michelle Jones is an employee of the SA Department for Health and Ageing.

Funding

This work was supported by the SA Department for Health and Ageing.

Michelle Jones is a social worker and Principal Evaluation Coordinator in the Office of the Chief Public Health Officer, SA Health.

Fiona Verity is Professor of Social Work and Social Care, Swansea University, Wales. She is both a social worker and a sociologist.

Megan Warin is an anthropologist and Australian Research Council Future Fellow whose research focuses on the gendering of health and illness (including anorexia and obesity) and the anthropology of epigenetics.

Julie Ratcliffe is a health economist and the current Research Chair in Health Economics within the School of Medicine at Flinders University.

Lynne Cobiac was the Director of the OPAL Flinders University Evaluation Project and is the Science Director of CSIRO’s Food and Nutrition.

Boyd Swinburn is the Professor of Population Nutrition and Global Health at the University of Auckland and Alfred Deakin Professor and Co-Director of the World Health Organisation (WHO) Collaborating Centre for Obesity Prevention at Deakin University in Melbourne.

Margaret Cargo is Associate Professor in the School of Health Sciences at the University of South Australia. Her expertise is in the process evaluation of community-based interventions.

References

Astbury

Leeuw

(2010) Unpacking black boxes: Mechanisms and theory building in evaluation. American Journal of Evaluation 31: 363–381.

Australian Bureau of Statistics (2012) Australian Health Survey – First Results, 2011–12. Canberra: Australian Bureau of Statistics.

Borys

Le Bodo

Jebb

. (2012) EPODE approach for childhood obesity prevention methods, progress and international development. Obesity Reviews 13: 299–315.

Bronfenbrenner

(1977) Toward an experimental ecology of human development. American Psychologist 32: 513–531.

Butland

Jebb

Kopelman

. (2007) Foresight Tackling Obesities: Future Choices – Project report. London: UK Government Office for Science.

Chen

Ratcliffe

Olds

. (2014) BMI, health behaviours and quality of life in children and adolescents: A school-based study. Pediatrics 133: e868–e874.

Chen

(2005) Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks, CA: Sage Publications.

Cousins

Earl

(1992) The case for participatory evaluation. Educational Evaluation and Policy Analysis 14(4): 397–418.

Creswell

(2009) Reseach Design: Qualitative, Quantitative and Mixed Methods Approaches. Los Angeles, CA: Sage Publications.

10.

Creswell

(2014) Reseach Design: Qualitative, Quantitative and Mixed Methods Approaches. International student edition, 4th edition. Los Angeles, CA: Sage Publications.

11.

Davison

Birch

(2001) Childhood overweight: A contextual model and recommendations for future research. Obesity Reviews 2: 159–171.

12.

Finegood

Merth

Rutter

(2010) Implications of the foresight obesity system map for solutions to childhood obesity. Obesity 18 Suppl 1: s13–6.

13.

French

Blair-Stevens

(2006) From snake oil salesmen to trusted policy advisors: The development of a strategic approach to the application of social marketing in England. Social Marketing Quarterly 12(3): 29–40.

14.

Funnell

Rogers

(2011) Purposeful Program Theory. San Francisco, CA: Jossey-Bass.

15.

Greene

Caracelli

(1997) Defining and describing the paradigm issue in mixed methods evaluation. In: Greene

Caracelli

(eds) Advances in mixed methods evaluation: The challenges and benefits of integrating diverse paradigms. New Directions for Evaluation 74. San Francisco, CA: Jossey-Bass.

16.

Greene

(2007) Mixed Methods in Social Inquiry. San Francisco, CA: Jossey-Bass.

17.

Hartwick

Coveney

Cox

. (2014) Transferring an innovation in food and lifestyle education: Adaptation of a French obesity prevention methodology in Australia. The FASEB Journal 28(1): Supp 262.2.

18.

Hawe

Shiell

Riley

(2009) Theorising interventions as events in systems. American Journal of Community Psychology 43: 267–276.

19.

Ife

(2002) Community Development: Community-based Alternatives in an Age of Globalisation. Frenchs Forest: Pearson Education Australia.

20.

Knigge

Cope

(2006) Grounded visualization: Integrating the analysis of qualitative and quantitative data through grounded theory and visualization. Environment and Planning 38: 2021–2037.

21.

Leslie

Magarey

Olds

. (2015) Community-based obesity prevention in Australia: Background, methods and recruitment outcomes for the evaluation of the effectiveness of OPAL (Obesity Prevention and Lifestyle). Advances in Pediatric Research 2(23): 1–16.

22.

Lincoln

Guba

(2000) Paradigmatic controversies, contradictions and emerging confluences. In: Denzin

Lincoln

(eds) Handbook of Qualitative Research. 2nd edition. Thousand Oaks, CA: Sage, pp. 163–188.

23.

Ling

(2012) Evauating complex and unfolding interventions in real time. Evaluation 18: 79–91.

24.

Lobstein

Baur

Uauy

(2004) Obesity in children and young people: A crisis in public health. Obesity Review (Suppl 1): 4–85.

25.

McCann

(2012) Review of Non-Hospital Based Services. A report prepared by Warren McCann, Office of Public Employment and Review to the Hon John Hill, Minister for Health and Ageing. October.

26.

MacLellan-Wright

Andreson

Barber

. (2007) The development of measures of community capacity for community-based funding programs in Canada. Health Promotion International 4: 299–306.

27.

Miller

Baird

Littlefield

. (2008) Epistemological pluralism: Recognizing interdisciplinary research. Ecology and Society 13: 46.

28.

National Center for Chronic Disease Prevention and Health Promotion (2011) Developing an Effective Evaluation Plan. Atlanta, GA: Centers for Disease Control and Prevention, Office on Smoking and Health; Division of Nutrition, Physical Activity, and Obesity.

29.

National Health and Medical Research Council (2009) NHMRC Levels of Evidence and Grades for Recommendations for Guideline Developers. Canberra: National Health and Medical Research Council. Available at: https://www.nhmrc.gov.au/_files_nhmrc/file/guidelines/developers/nhmrc_levels_grades_evidence_120423.pdf.

30.

Neuman

(1997) Social Research Methods: Qualitative and Quantitative Approaches. 3rd Edit. ed. Needham Heights, MA: Allyn and Bacon.

31.

Neuman

(2000) Social Research Methods – Qualitative and Quantitative Approaches. 4th edition. Boston, MA: Allyn and Bacon

32.

O’Brien

Payne

Nolan

. (2010) Unpacking the politics of evaluation: A dramaturgical analysis. Evaluation 16(4): 431–444.

33.

Olds

Tomkinson

Ferrar

. (2009) Trends in the prevalence of childhood overweight and obesity in Australia between 1985 and 2008. International Journal of Obesity 34: 57–66.

34.

OPAL Collective (2015) Practitioner insights on obesity prevention – The voice of South Australian OPAL workers. Health Promotion International. Epub ahead of print. DOI: 10.1093/heapro/dau118.

35.

Patton

(1994) Developmental evaluation. Evaluation Practice 15: 311–319.

36.

Pawson

Tilley

(2004) Realist Evaluation. Available at: http://www.communitymatters.com.au/RE_chapter.pdf.

37.

Pettman

Armstrong

Doyle

. (2012) Strengthening evaluation to capture the breadth of public health practice: Ideal vs. real. Journal of Public Health 34: 151–155.

38.

Popay

J an

Williams

(1998) Qualitative research and evidence based healthcare. Journal of the Royal Society of Medicine 91: 32–37.

39.

Reed

Harvey

(1992) The new science and the old: Complexity and realism in the social sciences. Journal for the Theory of Social Behaviour 22: 353–380.

40.

Richard

Potvin

Kishchuk

. (1996) Assessment of the integration of the ecological approach in health promotion programs. American Journal of Health Promotion 10: 318–328.

41.

Richards

Kostadinov

Jones

. (2014) Assessing implementation fidelity and adaptation in a community-based childhood obesity prevention intervention. Health Education Research. Epub ahead of print. DOI: 10.1093/her/cyu053.

42.

Rogers

(2008) Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation 14(1): 29–48.

43.

Rossi

Lipsey

Freeman

(2004) Evaluation: A Systematic Approach. Thousand Oaks, CA: SAGE.

44.

Sanderson

(2002) Evaluating complexity: Theory, evidence and practical guidance. In: Symposium on ‘Theory-Based Evaluation’ European Evaluation Society: Fifth Conference, Three Movements in Contemporary Evaluation: Learning, Theory and Evidence, Seville, Spain.

45.

Stame

(2004) Theory-based evaluation and types of complexity. Evaluation 10: 20.

46.

Stame

(2010) What doesn’t work? Three failiures, many answers. Evaluation 16: 371–387.

47.

Stevens

Ratcliffe

(2012) Measuring and valuing health benefits for economic evaluation in adolescence: An assessment of the practicality and validity of the Child Health Utility 9D in the Australian adolescent population. Value in Health 15: 1092–1099.

48.

Summerbell

Waters

Edmunds

. (2005) Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 20(3): CD001871. DOI: 10.1002/14651858.CD001871.pub2.

49.

Swinburn

Gill

Kumanika

(2005) Obesity prevention: A proposed framework for translating evidence into action. Obesity Reviews 6(1): 23–33.

50.

Swinburn

Bell

King

. (2007) Obesity prevention programs demand high-quality evaluations. Australian and New Zealand Journal of Public Health 31: 305–307.

51.

Thomas

(2006) Obesity prevention programs for children and youth: Why are their results so modest? Health Educucation Research 21: 783–795.

52.

Trojan

Suter

Arthur

. (2009) Evaluation framework for a multi-site practice-based interprofessional education intervention. Journal of Interpersonal Care 23: 380–389.

53.

Van Koperen

Visscher

Seidell

(2011) Scientific Evaluation and Dissemination. In: Borys

J-M

Le Bodo

De Henauw

Moreno

Romon

Seidell

., (Eds) Preventing Childhood Obesity: EEN Recommendations. France: Lavoiser Publishing.

54.

Wang

Cai

. (2015) What childhood obesity prevention programmes work? A systematic review and meta-analysis. Obesity Reviews 16: 547–565

55.

Warin

Zivkovic

Moore

. (2015) Short horizons and obesity futures: Disjunctures between public health interventions and everyday temporalities. Social Science and Medicine 128: 309–315.

56.

Waters

Silva-Sanigorski

Hall

. (2011) Interventions for preventing obesity in children. Cochrane Database of Systematic Reviews 2011.

57.

Weinstein

Stason

(1977) Foundations of cost-effectiveness analysis for health and medical practices. The New England Journal of Medicine 296: 716–721.

58.

Williams

Weir

(2013) OPAL – Using a social marketing approach to reducing childhood obesity. In: Rundle-Thiele

(ed.) Contemporary Issues in Social Marketing. Cambridge: Cambridge Scholars Press.

59.

World Health Organisation (2008) A Framework to Monitor and Evaluate Implementation: WHO Global Strategy on Diet, Physical Activity and Health. Geneva: WHO.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB