Abstract
The importance of evaluation to demonstrate the effectiveness of policies, programmes and interventions is widely recognised. Evaluation in the context of public health and healthcare is viewed as a complicated exercise, particularly when dealing with complex interventions involving multiple partners, multiple components and multiple outcomes. Eliciting the programme theory is an important starting point of an evaluation process to enable the link between theory and action to be articulated. This article gives a pragmatic account of the practicalities of working with stakeholders as they embark on a formative evaluation of a complex public health initiative, using a using a theory-based approach. Drawing on the principles of Leeuw’s strategic assessment, we planned a workshop to reflect the four stages of this approach–group formation, assumption surfacing, dialectical debate and synthesis. Stakeholders took part in four activities–Free Listing, Sphere of Influence, Beattie’s Theoretical Framework and Programme Concept Mapping. We found that our elicitation approach was particularly suited to reconstructing the programme theory in a non-threatening and playful environment, bringing about an alignment of programme theories by consensus and reducing anxiety.
Introduction
The importance of evaluation to demonstrate the effectiveness of policies, programmes and interventions is widely recognised. Moreover, evaluation is now seen as an aid to organisational learning, development and supporting the implementation of interventions to improve quality of care and patient safety (De Silva et al., 2014; Morello et al., 2013). In practice however, evaluation in the context of public health and healthcare is a complicated exercise, particularly when dealing with complex interventions involving multiple partners, multiple components and multiple outcomes (De Silva et al., 2014). The use of theory-based approaches is growing in popularity within public health and healthcare practice, as there is growing acknowledgement of the benefits of these approaches in enabling a better understanding of programme processes and activities; implementation process and applicability in practice (Breuer et al., 2016; De Silva et al., 2014; Kelly et al., 2007).
Theory-driven evaluation, for example, focuses on making the links between theory and actions more explicit, importantly allowing for an exploration of the process of change in meeting the specified outcomes (Leeuw and Donaldson, 2015; Weiss et al., 1995). The ‘theory’ in these situations is the set of beliefs and assumptions that underpin the programme activities (Weiss, 2000). Leeuw (2003) points out that many evaluations focus on how a programme should work and very rarely on articulating and examining the theory underpinning the interventions and their anticipated impact. According to Leeuw (2003), the reconstruction of the programme theory is necessary part of the evaluation process for the following reasons: to make sense of the changes produced by the programme, to establish if activities will produce the desired changes and to make the processes of delivery more transparent.
How this is accomplished in practice is open to debate, as there appears to be many approaches for practitioners to draw on to engage stakeholders either prior to or during the programme, or as part of an evaluation (Christie, 2003; Leeuw, 2003; Rosas, 2005). The aim of this article is not to review or debate the pros and cons of the different approaches of eliciting programme theory, as there is a body of supporting literature on this. Instead, it aims to give a pragmatic account of the practicalites of working with stakeholders as they embark on a formative evaluation of a complex public health initiative that they are in the process of implementing, in order to demonstrate the value of eliciting programme theory as part of the evaluation process.
This article describes our experience as evaluators of using a combination of approaches which draws on aspects of Leeuw’s (2003) methodologies to elicit a programme theory. Leeuw (2003) proposed that these methodologies – policy scientific, strategic assessment and elicitation with their roots in organisational psychology and management – can be adapted to assess the underlying assumptions of complex public health initiatives. The policy scientific approach is best suited to ex-post evaluations and uses a combination of document analysis and interviews. The strategic assessment and elicitation methodologies provide more dialectical approach to the process and facilitates double loop learning for organisations (Leeuw, 2003). We decided to draw on aspects of Leeuw’s (2003) strategic assessment approach to inform our evaluation workshop designed to work with programme stakeholders to reconstruct their programme theory.
The evaluation context
As this is an account of our experience and reflections of working with stakeholders to elicit and reconstruct their programme theory, we do not go into much details about the intervention itself. Briefly, we were commissioned by a public health team (the internal stakeholders) from a regional health office in the United Kingdom to conduct a formative evaluation of an initiative they were introducing to reduce obesity in the local area. The primary aim of the initiative was to invite partners (the external stakeholders) from the public, private and third sectors organisations to make a commitment or a promise to make changes within their organisations and introduce an intervention or initiative to reduce obesity and increase physical inactivity.
Eliciting the programme theory
It was agreed with members of the public health team that a theory-driven approach would help shape and inform the implementation and subsequent evaluation of the initiative. We, the evaluators, started by creating a logframe matrix as described by Green and South (2006) to illustrate our understanding and interpretation of the programme theory and underlying assumptions gained from interviews with internal stakeholders. Programme documents, including the marketing strategy and various briefing documents, were reviewed and used to inform the logframe matrix. However, when we presented the stakeholders with the logframe matrix, they pointed out that we had misrepresented their programme theory. We realised we had been working on the assumption that there was agreement around the theoretical underpinnings of the programme among the various stakeholders. We also assumed that our representation of the linkages across programme’s aims, objectives and activities were correct. Importantly, what we had presented was based on our assumptions, knowledge and expertise. We had taken it upon ourselves to decide what the underlying theory of the programme should be. In effect, we had presented our logical representation of their programme.
The challenge for us now was to align the various programme theories that were brought to our attention by the stakeholders and to get some agreement on ‘whose programme theory’. It became clear that we needed to take a more active role in working with the stakeholders to help them to re-conceptualise and re-articulate their programme theory. We also recognised that a consensus view of the programme needed to be achieved for things to progress. To do this, we decided to hold a participatory workshop drawing on the principles of Leeuw’s (2003) strategic assessment approach. We planned the activities of our workshop to reflect the four stages of this approach – group formation, assumption surfacing, dialectical debate and synthesis. Stakeholders were asked to take part in four activities – Free Listing, Sphere of Influence, Beattie’s (1991), Theoretical Framework and Programme Concept Mapping (Figure 1) (Box 1).
Theoretical Framework and Programme Concept Map.
Workshop activities.
Evaluators’ reflections
Our initial failure to reconstruct a programme theory that provided an accurate representation of the programme made us reflect on our role as evaluators. How much direction do we as evaluators provide when this becomes apparent? Should we reconstruct the theory for the programme as we perceive it, or should we focus on using methods that will facilitate a process that will enable stakeholders to do so themselves? The issue of contamination was a major concern and we wished to remain objective, as we were aware that misrepresentation was a possibility. There was also an underlying sense that, as technical experts, we were expected to present all the answers and help to validate the direction of the intervention they had started to implement. It was these questions and reflections that made us decide to find a way to remove misrepresentation and be assured that the process of reconstructing the programme theory was both transparent and representative of stakeholders’ collective logic.
The participatory workshop gave us an opportunity to engage with the participants in a manner that would not have been possible if we had carried on developing the theory ourselves based on the available information. The exercise also gave the participants the confidence to articulate their assumptions and their theories of change. The approaches we used – combined with attentive facilitation – had created a safe environment in which a range of underlying assumptions were surfaced and a shared common understanding of what the programme’s main stakeholders were aiming to achieve was established. By the end of the workshop, stakeholders were collectively making theoretical connections between the proposed activities and anticipated outcomes.
The participants found aspects of the process of reconstructing the programme theory challenging. Completing Beattie’s framework was the one exercise that required participants to think more critically about the theoretical underpinnings of certain elements of the programme they wanted to include. Participants struggled with this exercise the most, as this was when they were required to debate and examine the theoretical merits and logical links of their actions with the outcomes they were hoping their programme would achieve. As a group, they encountered difficulties in agreeing what the modality and focus of their intervention should be when placed in the context of Beattie’s framework. It was also the exercise in which the different theoretical understandings among the group started to emerge, provoking a deeper discussion about the purpose and relevance of certain activities and components of the programme they were planning to put in place. This exercise required the most focussed facilitation. In contrast, stakeholders found the sphere of influence exercise illuminating, almost cathartic, openly discussing and exploring the nature of their control over key activities of the programme. As a group, this gave them a better idea of what they could realistically achieve and what they could set as measures of success.
Initially, we were met with resistance from senior members of the team who had commissioned the evaluation. When we first suggested we wished to run a workshop to involve all the internal stakeholders in the process of eliciting the programme theory, they were not sure this was necessary. They did not see the value of full team involvement in exploring or identifying the theoretical underpinnings of the programme or gaining their perspectives on its implementation. We got the sense that some members felt we were subjecting the logic of what they had planned to scrutiny, opening the possibilities of their efforts being criticised by us and the rest of the team. Eventually, they acquiesced, and it was agreed that we could run a half day workshop, all internal stakeholders were invited and did join in. During the workshop itself, the initial resistance was expressed more strongly in the form of animosity from some senior members of the team. It was what we can only describe as an expression of ‘evaluation anxiety’. To alleviate this, we continually reassured and clarified what we were aiming to do, which was to assist them to openly discuss the purpose and role of eliciting their programme theory. We were careful to maintain a ‘playful’ atmosphere to allow underlying assumptions to surface and for participants to become more active in the facilitation of the discussions to reduce their anxiety.
Discussion
Reconstructing the programme theory is a significant stage in theory-driven evaluations. Leeuw (2003), Leeuw and Donaldson (2015) and Weiss (2000) consider this an essential step in the process in which the activities and assumptions of the programme are made explicit and mapped out. Weiss (2000) views this as ‘paying attention to the intervening steps in the process of change’ (p. 103) and creating a shared understanding of the ‘theory of change’. We certainly found this to be an essential step in the evaluation process and one which can easily be dismissed.
We found that the process of eliciting and reconstructing the programme theory required more intense engagement with stakeholders than originally planned. In this scenario, although the programme’s goal was clear (reducing obesity at a regional level), it was challenging for the stakeholders to document the programme activities and make the logical links between what they were planning and how it would lead to change. One of the reasons for this inability to populate the programme’s logframe matrix was the nature of the programme itself, which had initially been conceptualised as a loose network of multi-sectoral partners. No attempt had been made by the internal stakeholders to map out the relationships among programme activities, inputs or outputs of the various initiatives that each partner was proposing or links between assumptions being made.
The resistance we experienced at different stages of the evaluation process was an illustration of how analytical processes designed to challenge assumptions can lead to a feeling of anxiety among stakeholders. Donaldson et al. (2002) highlighted ‘evaluation anxiety’ as a real issue which tends to be expressed in five common ways; conflict, withdrawal, resistance, shame and anger. Donaldson et al. (2002) points out that the process of eliciting the programme theory can be perceived as external criticism of the programme team’s efforts and thinking, leading to stakeholders feeling vulnerable or exposed. They propose a number of strategies that evaluators should consider using to counteract signs of evaluation of anxiety early on in the process, such as determining the programme psychologic, using multiple strategies to allow stakeholders to discuss the effect of the evaluation (Donaldson et al., 2002).
The participatory workshop illustrated that key aspects of Leeuw’s strategic assessment approach can be applied in practice. The evaluation did bring to the fore the difficulties of using a theory-driven approach in the context of complex programming in identifying appropriate approaches and strategies to support the process of eliciting the programme theory. Drawing on Leeuw’s (2003) strategic assessment approach allowed us to create a dialectical tension by encouraging individuals to take a global perspective of the programme and taking them out of their confined remits and roles. We felt that we were able to illustrate, as they became more engaged in dialectic debate the dynamics of the group started change. The participants who were most resistant at the beginning eventually engaged with the process, actively listened, taking the perspectives of others into account. It also allowed us, as evaluators to depart from a more formal evaluation approach in which the evaluators drive the process and present their interpretation of the programme theory. We managed to create a ‘playful’ atmosphere that allowed them to bring their own individual programme theories to the table (Van der Heijden and Eden, 1998). In doing so, we gave the stakeholders an opportunity to voice their perceptions and understandings of the programme, as well as to uncover areas of potential misalignment of programme theories among themselves. These activities we used not only helped in eliciting the programme theory but shared many similarities with the strategies that Donaldson suggest evaluators should consider including in evaluation approach to reduce evaluation anxiety.
Conclusion
In this article, we have presented practical approaches to eliciting programme theory in the case of a complex intervention with multiple stakeholders. Our elicitation approach was particularly suited to reconstructing the programme theory and in bringing about an alignment of programme theories by consensus in a non-threatening and ‘playful’ environment. The outcome of our evaluation workshop was the articulation of a programme theory that better reflected stakeholders’ shared understanding of the potential changes their programme could achieve in the context in which they were operating.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This evaluation was funded by the Department of Health East Midlands, Public Health Directorate. The employing organisations had no influence over the study design, conduct, analysis or interpretation of the findings.
