Facilitating the evaluation of complexity in the public sector: learning from the NHS in Scotland

Abstract

It is necessary for public managers to be able to evaluate programmes in the context of complexity. This article offers key learning and reflections based on the experience of facilitating the evaluation of complexity with a range of public sector partners in Scotland. There have been several articles that consider evaluating complexity and theory-based approaches to evaluation; however, the literature is scarce when it comes to actually facilitating such approaches within the public sector. The article discusses the main steps used when facilitating outcomes planning with stakeholders and identifies some key challenges and learning in relation to undertaking this work. The article will be of interest to those that want to know more about the core components of outcomes-based evaluation and, moreover, to those that will undertake this type of facilitation within the public sector as part of organisational development and professional development.

Keywords

complexity evaluation learning outcomes planning public sector

Since the early 1990s the public sector in Britain has moved increasingly towards emphasising the outcomes, impacts or effectiveness of programmes (Cabinet Office, 1999). Yet the difficulties associated with a move towards an outcomes approach (from output management) in public management lie principally in demonstrating impacts in complex environments. This is particularly the case when professionals in the public sector work within a multi-partnership policy landscape – such as in the public health area – whereby outcomes cannot only be reported in quantitative terms (i.e. not all outcomes are “hard” in that they can be measured by using official statistics on, say, changes in morbidity and mortality levels amongst population groups). Rather, public managers are also expected to be able to report on “softer” outcomes such as changes in skill sets, knowledge and increased organisational capacities. There are also challenges in terms of articulating the direct and indirect influence of particular programmes and initiatives in impacting on outcomes – known as the ‘attribution problem’ within partnership contexts (Wimbush, 2011: 216). This being the case, it is necessary for public managers to be able to evaluate programmes in the context of complexity. There is also a need to consider the fact that, traditionally, public managers have worked in an organisational context whereby, as custodians of the public purse, they have been held to account for the activities they have undertaken and what they and their departments have produced – meaning that prioritising the measurement and reporting of impact arguably requires a change in mind set. That is to say that public managers need to be able to say what resources are used when, where, why, as well as the difference these resources have made.

In Scotland, the National Performance Framework (NPF) and the emergence of Single Outcome Agreements (SOAs) for 32 local authorities (LAs) and community planning partnerships (CPPs) require such organisations and partnerships to demonstrate what difference programmes are making in achieving and/or contributing to outcomes. As a case in point, the new guidance to local authorities and CPPs from 2013 onwards indicates that the Scottish government requires CPPs to be able to demonstrate how successful they have been in reducing inequalities (in its various forms; e.g. health, social, economic). This requires public managers to be able to articulate an evidence-based strategy on how this can be undertaken.

Evaluation literature sources consider the facilitation of evaluation generally (Earl et al., 2001; Patton, 2008) and addresses the topic of outcomes/theory-based evaluation as an approach (e.g. Weiss, 1997; Blamey and Mackenzie, 2007; Institute for Research and Innovation in the Social Sciences [IRISS], 2010; Mayne, 2012; Wimbush, 2011; Wimbush et al., 2012). The literature also addresses the organisational challenges and lessons around implementation of an outcome-focused approach within the public sector (Poole et al., 2001; Wimbush, 2011; Wimbush et al., 2012). What emerges from this literature is that the implementation of an outcomes approach is depend on senior support/leadership within the implementation context, whether there is collective buy-in to a results-orientated culture, and when performance management systems are able to adequately support monitoring and evaluation – not just for accountability but also for learning (Mayne, 2007). This article aims to build on such work by pursuing a gap in the literature that has not been addressed for those that will be undertaking capacity-building activities in the public sector. However, it should be noted from the outset that the article considers capacity building to encompass both the practical aspects of how to facilitate outcomes planning and the challenges of sustaining the outcomes planning within the public sector context. In other words, it is important for those facilitating the process to be able to work in the knowledge that public managers are likely to face key challenges once they have left the workshop environment. The sustained example of health improvement is used throughout the article; however, the learning on facilitating outcomes planning is representative of other areas of the public sector that involve multidisciplinary partners from multi-agency groups.

It is within this context that this article identifies key learning on facilitating the use of outcomes planning and evaluation in the public sector in Scotland on behalf of NHS Health Scotland. A major function of NHS Health Scotland, as Scotland’s health improvement agency and specialist Health Board, is to provide support to territorial Health Boards, CPPs, LAs and other public and voluntary bodies. A key part this role is to take a lead in outcomes-focused planning and evaluation. In case study terms, the article represents a ‘thick narrative’ (Flyvbjerg, 2006: 237) based on considerable experience of evaluating complex programmes and capacity building in Scotland. The lessons around capacity building are of relevance to those working in other policy and territorial contexts, especially those evaluating programmes within dense multi-partnership environments. The article draws heavily on the qualitative experience of the authors in relation to the steps and challenges of facilitating outcomes planning in relation to various Scottish public health programmes, such as Obesity Routemap for Scotland, Scotland’s Alcohol Strategy, Mental Health, and Maternal and Infant Nutrition Strategy (Scottish Government, 2009, 2010, 2011a). Furthermore, the evidence that informs the article is drawn from post-workshop evaluations and from the proceedings from routinely collected post-workshop review meetings.

In terms of structure, the article introduces what outcomes planning is and the steps and material that make up typical outcomes-planning workshops. It then draws out key learning, challenges and reflections in relation to what is meant by outcomes planning. This merits the attention of those who facilitate, or plan to facilitate, outcomes-focused approaches within a multi-partnership public sector landscape. The article concludes by considering the implications of outcomes planning capacity building for organisational learning and development within the public sector.

Outcomes-focused planning – why now?

Evidencing the impact of programmes is now more important than ever. In Scotland, as in many countries in the Western world, words such as impact, outcomes, and contributions are now mainstays of discourse in public management and, as noted earlier in the article, this has been driven by the emergence of a NPF for the Scottish public sector. The austere economic climate means that being able to justify the use of monies by assessing gains is very important (especially in terms of securing future funding given that most government-funded programmes come under the budgetary spotlight at intermittent government spending reviews). The NPF and emergence of SOAs have been key drivers for public managers to take an outcomes approach in Scotland. The political motivation was evident when the SNP Scottish government’s finance minister noted, ‘When we came to power in May 2007, we were clear we wanted to introduce a fresh, more strategic approach to government with a focus on outcomes and a single overarching purpose’ (Swinney, 2009: 4).

This poses some key challenges for evaluation but also for facilitating evaluation; for example, specifying the problem that needs to be addressed, specifying a broad range of interventions, using evaluative approaches to deal with complexity, and meeting the demands of multiple stakeholders (Smith and Petticrew, 2010; Craig, 2013).

What is outcomes planning?

Outcomes planning in the public sector has been influenced by John Mayne’s work, which highlights the need to ensure cross-partner agreement of the problem that programmes exist address (e.g. reducing societal inequalities). This also involves working through some key steps to identify what needs to be changed (the outcomes) and the means by which outcomes will be achieved at different stages of programme development, based on the best available evidence. Mayne (2007: 1–2) notes:

Results-based management involves deliberately gathering empirical evidence in order to know the extent to which intended results are being achieved so that modifications to the design and delivery of activities can be made to improve and account for performance in achieving intended outcomes.

There are a range of tools to help with this, such as: logic models (which, in fact, have been around in some form for many years in public administration and are often called “log frames”); multiple results chains (which help to identify key partnership contributions/inputs required to achieving outcomes); and outcome triangles (which provide a visual articulation of the context of the programme; e.g. political, economic and social). Yet what underpins the approach and the use of the models is the importance of evidence and the need to be able to monitor and evaluate progress throughout the life of a programme.

The process of outcomes planning helps individuals and organisations to define problems and identify the means by which such problems can be addressed based on evidence and plausible theory. The approach focuses on the results that are within an organisation’s sphere of direct and indirect influence, and the intended beneficiaries and stakeholders/partners who need to be involved to achieve outcomes. What will be achieved by who and by when depends on the nature of the programme and the type of issues(s) that programmes seek to address. As a guide when facilitating learning around outcomes, short-term outcomes (e.g. changes in knowledge, skills awareness) are those outcomes that can be plausibility achieved within a 1 to 2 year period; medium or intermediate-term outcomes (e.g. practice-level/organisational changes) are those that can be achieved generally within 3 to 5 years; and long-term outcomes (strategic/population-level changes) are more likely to be achieved on a 5 years (+) time period. In short, the process involves:

having a clear vision of why a programme exists, what it seeks to achieve and how achievement can be monitored and measured

planning work with staff with a clear set of objectives, activities, outputs, outcomes and measures of success (and identifying risks and recognising what is within your direct control and influence)

being clear and understanding what amount of investment of resources (budget and staff time) is required to achieve the outcome.

Why facilitate outcomes planning?

Facilitating the process with stakeholders has several advantages:

The planning process is participative and allows for a sense of buy-in to be fostered amongst partners about the need to tackle shared problems. This is helped by undertaking a shared/collaborative analysis of the problem.

The process allows for partners to prioritise what elements of the programmes need to be performance managed.

Outcomes planning allows for outcomes attributable to the programme to be monitored and evaluated.

Outcome plans are adaptable in order to respond to changing contexts – that is, outcome plans are not set in stone and should be adapted based on contextual changes that have a bearing on the programme, such as political, economic and social changes, and in light of new and emerging evidence.

Building on the final bullet above, outcomes planning should be flexible enough to allow for the realignment of activities and programmes that best serve to maximise the plausibility that, for example, A will lead to C via B (i.e. there is a clear theory that shows the plausible links between aspects of a programme or intervention, such as activities and its outcomes). Put differently, theories of change take into account the context in which initiatives work and the best available evidence. It is also important to consider that, from an evidence-based policy-making perspective, the evidence base may not be sufficient to allow for any degree of confidence or statement to be made about what the outcome of complex programmes will be. Programmes need to be designed in the light of the best available evidence, and implemented and evaluated in ways that enable learning about what works in what circumstances to be identified (Sanderson, 2011; Craig, 2013).

Learning and reflections part 1: the workshop approach

A key pedagogical approach to facilitating the evaluation of complexity is via the workshop approach. Workshops are particularly effective formats for training professional groupings (Klatt, 1999). Outcomes planning workshops operate best when conducted with relatively small groups (between 10 and 25 participants). The reason for this is to ensure that the groups are manageable and that higher numbers of participants might increase the likelihood that participants will be more reluctant to express themselves orally as part of a group – with potential for some individuals to dominate discussions (as highlighted in the learning and development literature; e.g. Harrison, 2005; Hackett, 2003; Stewart and Riggs, 2010). In addition, from a practical point of view, the manageability of the number of participants becomes even more important given that there are no guarantees that co-facilitation will always be possible (due to lack of resources in-house) and having a relatively small group to work with is a contingency that can, and should, be built into the workshop planning process.

The skills of the facilitator are of key importance in that they will set the tone of the workshop discussions and should be clear about establishing ground rules with the group in order to ensure that the workshop is a safe environment. Discussions need to be open and frank but, at the same time, the proceedings should be underpinned by trust and mutual respect. Facilitators also need to have high levels of emotional intelligence and interpersonal skills, and be able to gauge the energy and atmosphere within sessions – this includes being conscious about power dynamics and how to draw together diverse opinions that may be present amongst participants. In terms of subject knowledge, the facilitator needs to be someone that is comfortable with the outcomes-planning approach but it is also useful for the individual(s) to be comfortable with evaluation broadly (including public management, organisational learning and change, complexity theory, and quantitative and qualitative research methodologies). NHS Health Scotland have developed a facilitators brief to help planners implement outcomes approaches to evaluation workshops.¹

The key steps that should be undertaken in the outcomes planning and evaluation process as part of workshop activity are as follows:

Figure 1.

Multiple results chain (partnership contributions).

1. Analysis of current problems, risks, needs and gaps in evidence

This stage involves contextualising the workshop by undertaking an assessment of the stakeholder landscape. This prompts some key questions for facilitator-led discussions. These include: who are the key organisational actors, both internally and externally, that will have a bearing on the success of the programme? What are the political, economic and social challenges and opportunities that have implications for the programme? It is at this stage that the fundamental question of outcomes planning is posed (and which everything else in the process depends on, and is constructed around) and that is, what is the problem/issue that needs to be addressed? A vision of success cannot be formulated without a definition of the problem. With regards to the public health sector in Scotland, it is often the case that responses to this question will include long-term population-based/societal outcomes such as reductions in health inequalities (for example, this might be through targeting particular societal groupings though programmes; e.g. children through healthy weight programmes, early years initiatives, targeting people living in deprived circumstances to engage with primary care services, to name a few). Micro-visions will need to be identified that contribute to the achievement of the macro-vision. This might, for example, be about contributing to clinically related outcomes such as reductions in morbidity and mortality (with the latter being relevant to programmes such as alcohol and cardiovascular disease interventions). Careful consideration needs to be given to the risks of not taking action to address the overall vision and, moreover, the foreseeable risks to programme success that face public managers (e.g. changes in political/policy priorities, change of government, cuts or redeployment of public expenditures, economic pressures, and short-term funding cycles).

When analysing the nature of the problem there is also an opportunity to reflect on what evidence (perhaps from other programmes/international state contexts) there is around the relationship between the programme and the problem identified in terms of the best ways to reach outcomes. For example, there may be a range of policies, initiatives and approaches to achieve reductions in health inequalities through the topic of alcohol. These include (and are not limited to) a combination of awareness-raising and support measures (such as increasing the public’s knowledge of the benefits of safe levels of drinking and providing appropriate/tailored support to individuals) and economic/policy levers (such as reducing the affordability and availability of alcohol). Once there is agreement (at least in principle at early stages) on the problem that needs to be addressed, this is when the outcomes chain can be constructed based on the definition of the programme.

2. Construct a logic model and monitoring and evaluation plan

The next stage would be to construct an outcomes chain, a logic model that will illustrate how the outcomes will be achieved. The logic model can be described as a key navigational tool for dealing with a complex landscape (Wimbush et al., 2012: 325). The type of advice provided to local partnerships when helping them to develop an outcomes chain include working from the longer term outcomes at the macro-level and then working backwards (i.e. what medium-term outcomes can plausibility lead to the longer-term outcomes?). A way to encourage participants to get out of their comfort zone and to penetrate any forms of short-termism, including supporting them to avoid espousing notions of received wisdom in their thinking, is to ask participants about what overall success would look like for their programme. A strategy to develop this further with participants is to ask them what would need to be achieved in order to address the ultimate problem. This process continues at the medium-term outcome level (e.g. what short-term outcomes will lead to behavioural changes at a medium-term level?). In a similar vein, to achieve short-term outcomes, what activities need to be undertaken in order to achieve short-term outcomes? And finally, what inputs/resources are required in order to undertake these activities? Working in this way enables public managers to hold back from the “doing state” (i.e. moving straight to activities) and remain in the “thinking state” (consider outcomes first and navigate backwards to activities; Wimbush et al., 2012: 325).

It is also important to note at this stage, especially when considering the reach and beneficiaries of programmes, that it is becoming increasingly important to incorporate the principles of health inequalities impact assessments as part of the outcomes planning. Participants are encouraged to consider whether their programme impacts on certain societal groups in adverse ways. Equalities legislation requires that public authorities take into account and mitigate any negative impacts in relation to protected characteristics of societal groups.² Indeed, the Equality Act 2010 came into force in April 2011 – this is often referred to as the ‘general duty’ Scottish public authorities must have in giving ‘due regard’ to the need to eliminate unlawful discrimination, advance equality of opportunity and foster good relations (The Equality Act, 2010). This, of course, has implications for outcomes planning (and its facilitation) across a range of policy areas, due to the general duty on public bodies.

An example of a results chain/logic model is provided in Figure 1. The illustration shows the relationships between elements of a programme that aims to address problems in relation to alcohol consumption in Scotland. It shows the outcomes to be achieved and the specific roles of sectors and/or organisations within a partnership context. This is a key output from workshop sessions.

Monitoring and evaluation underpins the outcomes chain through the development of outcome indicators as well as plans for data collection. The purpose of this is to allow for the assessment of the extent to which outcomes have been achieved. Data collection might be in the form of in-house or externally commissioned evaluation or the use of performance monitoring data. When it comes to indicator development there is some key advice given to participants as part of the workshops in terms of developing outcome indicators:

Measurable: There should be the ability to measure either quantitatively or qualitatively over time (e.g. through annual surveys such as the Scottish Health Survey).

Precise/specific: There should be no room for ambiguity and any measurements should clearly identify the denominator and numerator (e.g. percentage of patients from deprived areas attending a primary care consultation as a proportion of the eligible population).

Consistency/sensitivity: Indicators should not change over time in order to maximise consistency and validity. It would be difficult to draw inferences for indicators that are modified (unless the indicator has been adapted in line with changed outcomes). As a case in point, if an indicator was developed to support the measurement of an employability outcome, it would be invalid to change an indicator from, say, the percentage of 16-to-24-year-olds in full-time employment over a 12-month period to the percentage of 16-to-30-year-olds in part-time employment over 8 months (unless the measurement is responding to an adapted outcome). The indicator(s) must always be as closely aligned to an outcome as possible for it to be meaningful.

In terms of securing outputs from the workshops, facilitators need to be skilled at time management, consensus-building and capturing and summarising workshop proceedings. In order for useful outputs to emerge from the workshop the facilitators also need to be task-focused. The main outputs that emerge from outcomes planning workshops include:

a write-up of the logic of the high- or macro-level changes that the programme seeks to contribute to/address based on an understanding of the problem (including the vision and key stakeholders);

a basic first draft illustration of the programme theory or logic model that outlines the outcomes (theory of change) and the means by which this will be achieved (theory of action) (e.g. an early version of Table 1); and

an outcomes plan that provides an attempt to identify how outcomes will be monitored and evaluated (e.g. identification of outcome indicators and evidence sources), which also includes an assessment of the roles of partners in this process.

Table 1.

Outcomes plan.

Outcomes plan	Progress measurement strategy
	Outcome indicators (a specific, observable, and measurable characteristic or change that will represent achievement of the outcome)	Data source (the source that will evidence change; e.g. evaluation studies, routine monitoring/statistical data, programme evidence, surveys)	Responsibility for collection (it is importance to ascribe responsibility to organisations(s) to be responsible for collecting data in order to be able to adequately resource the monitoring and evaluation of the plan).
Long-term outcomes: the ultimate population-level problem/issue to be addressed; e.g. reduction in health inequalities
Medium-term outcomes: practice and behaviour change
Short-term outcomes: reactions of target group, changes in knowledge, understanding, ability, skills
Reach: this is the group of people who use our outputs – the immediate target group, our beneficiaries. Often this component is concerned with the values and aspirations people hold, participation and reactions.
Outputs: the tangible results of activities, for example, the number of training courses delivered, the numbers of leaflets printed and delivered. Outputs are not indicators of the success of a programme – for example, printing and delivering leaflets does not mean that they were read or understood, or even acted upon
Activities: these are what a programme does to try and achieve its goal and can include training, meetings, publicity, service delivery
Inputs: these are the investments in a programme and can include, money, time, equipment, staff, “know-how”, facilities

The outcomes plan is a workshop resource that brings these steps together and helps to facilitate group discussions on monitoring and evaluation. Importantly, it also allows for consideration of allocation of responsibility amongst partners for measurement and data collection. This aids discussions in relation to which activities should owned by particular partners and how they will contribute to evidencing impact.

3. Reporting on outcomes (also known as telling the performance “story”)

Once the outcomes plan is completed and the data is collected as part of the monitoring and evaluation process, there is usually a follow-up workshop in order to support participants to articulate the story of the programme by highlighting the learning that has been accrued, including the extent to which challenges (such as external factors) have had implications for the achievement of outcomes. The word “stories” is used in evaluation literature to describe this form of reporting due to the emphasis that is placed on the use of narratives and a range of evidence sources to tell a trustworthy story about the impact that a programme has made. If an outcomes chain for the programme is used, the story can be expressed as discussing the extent to which the results chain actually reflects reality, thus giving structure and logic to the story (Office of the Auditor General of Canada, 2013). Again, a mantra of outcomes-focused performance reporting is about articulating the difference made to programmes. Mayne’s (2004: 50) key discussion questions articulate the key components of an outcomes-focused performance report:

What is the context?

the overall setting of the programme (description, objectives, resources)

the results/outcomes chain (programme theory)

the risks faced

What was expected to be accomplished at what cost?

statement of the (clear and concrete) outputs and outcomes expected

planned spending

What was accomplished in light of these expectations?

the relevant outputs delivered at what cost

the outcomes realised related to the expectations

a discussion of the evidence available demonstrating the contribution made by the program to those outcomes

What was learned and what will be done next?

a discussion of what will be done differently as a result of what was achieved

What was done to assure quality data?

a description of what the organisation does to ensure the quality of the data and information reported.

The main story line of a performance story is how well the programme has performed in relation to what was expected and what will now be done differently to better ensure future performance.

The learning-based nature of outcomes planning means that this is not necessarily the final step in the process. The learning accrued from the programme should, in theory, feed back into a revised outcomes plan in order to aid continuous improvement. So, workshop participants are encouraged to consider how performance can be improved without learning. Indeed, outcomes/learning-based performance reporting is inextricably linked to organisational learning (Levitt and March, 1988; Argyris, 1999). The organisational learning literature recognises the need for feedback (Aldrich and Ruef, 2006: 47) in making links between the past and the present. Performance stories represent, and are a mechanism by which, such feedback can be captured in order to contribute towards experiential learning from the past; that is, by helping to ensure that lessons are accessible to key stakeholders (e.g. funders, partner organisations and programme beneficiaries). They help to link past events and current intentions, actions, and future behaviours through a process of self-evaluation.

This following section highlights the challenges that are specifically related to facilitating the outcomes planning approach.

Learning and reflections part 2: challenges of facilitating outcomes planning/evaluating complexity

Evidencing the change (with implications for measurement and attribution)

When facilitating outcomes planning, it is important to communicate to participants the need to take a pragmatic approach to measuring and attributing the difference that the programme has made to achieving outcome(s). This involves considering the range of qualitative and quantitative evidence sources that are available and how combinations of evidence help participants say something about programme success (or otherwise). Participants often find it difficult to move beyond the idea that they need a full-scale evaluation to tell them about success. This article does not argue against types of evaluations but it does argue for a realistic approach to programme evaluation that allows for the gathering of a range of evidence sources, in turn allowing for the outcome to be evidenced for the programme in question. This comes from the perspective that the process is about ‘viewing evidence in the context of what is required to make good judgements or decisions’ (Tannahill, 2008: 383) and to give due consideration to all sources that can be viewed as valid and relevant (Learmonth and Watson, 1999). Mayne notes that it is likely that no single piece of evidence gathered will be enough to build a credible case concerning a result achieved or a contribution made by a programme. He suggests that ‘while no one piece of evidence may be very convincing, a larger set of different and complementary evidence can become quite convincing’ (Mayne, 1999: 15). The context of public health is particularly relevant here in that it is ‘often likened to a jigsaw, with different pieces of action joining together to make up the big picture’ (Tannahill, 2008: 383). A further key issue to emerge from workshops with public managers is that they have come to expect that the provision of evidence to local senior officials or funders needs to be an independent study (and often commissioned) – warranting the perception that self-evaluation would be of poor quality. Yet, this is at odds with what outcomes planning can offer, given it is a participatory approach to critical self-evaluation in that programme stakeholders themselves have a responsibility for critically reviewing the content, evidence and the product of the outcomes plan.

A key challenge here, therefore, is the fact that there needs to be some discussion of the most appropriate type of evidence for the circumstances. The most worthwhile response from a facilitator’s perspective is that appropriateness is dependent on the extent to which evidence sources (or a combination of sources) are aligned to the outcomes. In other words, does it enable us to say something about the expected result or condition? As with any research process, triangulation of sources is most desirable. However, the key message in facilitation is to encourage managers to be more conscious in their business processes about gathering routine data throughout the life course of a programme. In reality, it is unlikely that every outcome will have a measurement aligned to it. There needs to be some prioritisation of what is measured (i.e. what are the key outcomes that you need to evidence?). The prioritisation of this will often be determined by what programmes are accountable for and which aspects of their work that future funding will be dependent on. In short, there is a need to consider how to take a pragmatic approach to measuring and attributing the difference that the programme has made.

Leadership to take forward locally

Experience of capacity building with local partners has suggested that the extent to which processes are reconfigured is heavily correlated with the level of enthusiasm of local leaders to take the outcomes planning approach forward (and this serves to validate the claims of others; e.g. Wimbush, 2011). Outcome evaluation processes cannot be seen as an ‘add-on activity’ (Poole et al., 2001: 407) but there needs to be firm commitment by top management (Carter, 1984: 10). Senior managers in public sector organisations do not always take too kindly to being advised on how to plan differently. Indeed Earl et al. (2001: 13) are correct when they identify that ‘for the outcomes of the programme to be relevant and lead to long-term…sustainable benefits…local ownership needs to become effective and dominant’. However, experience of facilitating outcomes planning in Scotland has indicated that a major challenge has been the lack of skills of participants to cascade learning, to enable the capacity of the workforce to be developed. It is often the case that public managers will attend outcomes planning workshops, and are generally enthusiastic about the potential of the approach, but will then have challenges when it comes to securing local buy-in from senior management in order to take it further. Even if senior managers are committed to programme evaluation, managers and staff may not have the requisite knowledge and skills to design or implement evaluation plans (Chelimsky, 1994: 496; Poole et al., 2001: 408).

These issues have had negative implications in terms of the lack of willingness to give their staff the learning and creative space to take forward an outcomes approach. This is particularly pertinent given that leadership has been held up as crucial to the success of an outcomes-focused approach (IRISS, 2010). An explanation for this is that risk cannot be separated from the outcomes planning process. Put differently, ‘in taking an outcomes approach, people’s desired outcomes may well involve them undertaking a certain amount of “risky” behaviours, which needs to be given consideration and supported by the practitioner’ (IRISS, 2010: 6). Yet, the beauty of outcomes planning is that there is the opportunity for risks to be calculated (as much as possible beforehand); therefore there is the potential to take a rational approach to risk-taking with the inclusion of corresponding mitigating actions.

Conceptual understandings: key terms and spheres of influence

A fundamental priority for facilitating the approach with practitioners is the need to be able to offer conceptual clarity. In particular, there is often confusion around the differences between outputs and outcomes (short, medium and long) and outcome indicators. There tends to be a general understanding of the terms inputs and activities, which could be explained by the fact that these are very traditional management terms and are fairly intuitive in meaning (i.e. it is about investments you make and what you do with your investments). Yet, the understanding of outputs versus outcomes versus outcome indicators can be problematic. The need to understand the conceptual distinctions between components of the outcomes plan is not purely a matter of semantics; rather, misunderstandings around what is the change that is desired and the means by which this will be achieved need to be clearly identified and formulated in order for the programme to plausibly make a difference. A further key complex idea that can be challenging to facilitate is the different spheres of influence that partners have along the logic chain. These spheres are direct control, direct influence and indirect influence.

The best way to navigate through the challenges of helping workshop participants to understand the different spheres of influence is to discuss the extent to which outcomes are visible to daily work of the programme. That is, whether what is produced and the reaction of customers or clients to this work is something that can be seen and observed as part of routine organisational practice. Outcomes that are not visible on a daily basis by public managers are usually those that they can only influence indirectly. Thus, the further down to the right of a logic chain you go, the less control you have as these are subject to many other external influences and possibly many other services, and can only be changed through collective effort (e.g. through partnership working) – which are often the focus of SOAs. Similarly, this process requires participants to challenge cause and effect thinking by facilitating the perspective that the relationship between means and ends is underpinned by complexity. The experience of working in the public health policy context (but this arguably applies equally to any other policy area that tries to address complex societal issues) is that the relationship between problems and solutions is not a linear process and that programme theory will have “boxes and arrows” between stages in the logic chain that feed back to, and crossover with, different outcomes.

Figure 2 shows a simple linear logic chain to illustrate spheres of influence. In addition, Figure 1 earlier in the article gives a more realistic illustration of the multi-linearity of logic chains, which involves multiple actors making different contributions to outcomes and also shows the cross overs between outcomes. Enabling people to see the relationships between outcomes is required, and this can be a shift in thinking for many. From the facilitator’s perspective, having some knowledge about the programme and the general subject field that you are facilitating is important in this instance in order to support participants to make connections between partner contributions and different outcomes.

Figure 2.

Outcomes chain: spheres of influence.

Short-term funding cycles and competing organisational priorities

A key finding from follow-up questionnaires with participants, which aimed to assess the sustainability of the outcomes planning content facilitated at the workshops, has been that the propensity for implementation is strongly associated with short-term funding cycles and competing priorities. The Healthy Weight Communities programme in Scotland can be used as a case in point here to show how local managers were not in a position to take the outcomes planning approach forward, despite the availability of national support on the facilitation of the approach. This example serves to highlight common challenges experienced across programmes based on facilitating outcomes planning in Scotland.

In June 2008 the Scottish government announced its intention to establish pathfinder healthy weight community programmes in a small number of areas across Scotland. This was part of their Healthy Eating, Active Living action plan (Scottish Government, 2008). Eight Healthy Weight Community (HWC) pathfinders were selected in May 2009 and their funding continued until March 2011. The purpose of the Healthy Weight Communities Programme was to ‘demonstrate the ways in which engaging communities in healthy eating, physical activity and healthy weight activities as part of a single coherent programme may have a greater impact on health outcomes than current discrete activities’ (Scottish Government, 2008: 1). The programme is distinctive in a number of ways:

It defined the objective clearly (i.e. to group, connect and promote existing local initiatives in ways which enhance their impact), but left it to pathfinders to design how to achieve the objective.

There was an emphasis on innovation and experimentation, with no specific demands on outputs or milestones that might discourage this.

There was an emphasis on process, ownership, partnership and leadership.

There was a requirement to connect into the community planning structure.

There was an emphasis on learning and sharing as the work progresses.

The Scottish government, as the funder of the projects, had a performance management role with the projects and, as part of this role, encouraged the use of outcomes planning, given the focus on reporting against impact. The move towards this direction seemed to represent a response to one of the reflections presented in the evaluation of HWCs, which noted that ‘projects find it helpful to describe the difference they are making and may wish to embed evaluation, review and refinement into their practices to enhance impact’ (Scottish Government, 2011b: 14). As a result, the Scottish government requested the support of NHS Health Scotland to help local projects take an outcomes approach to their programmes.

NHS Health Scotland invited all the projects from all geographical areas to attend an outcomes planning workshop that would also include the offer of follow-up support in local areas. A key focus of the workshop was to encourage the systematic use of programme evidence in the knowledge that resources for evaluation (such as commissioned work) were likely to be scarce. Without discouraging the use of evaluations, if available, local projects were encouraged to use qualitative evidence such as documentation, minutes of meetings, anecdotal evidence, diaries, correspondence, testimonials, photographs and so on, with regularly collected output and short-term outcome-orientated descriptive statistics; for example, numbers coming though the local initiatives, re-attendance figures, numbers reporting changes in knowledge, awareness and behavioural change (through routine surveys) and so on.

The workshop evaluated well with participants being generally positive and enthused about outcomes planning and its potential value (this data was gathered via end of workshop questionnaires). In the end, however, the approach was not taken forward by local areas despite the offer of further and on-going national support from NHS Health Scotland. There are two key reasons for this, and these point to more common experience in relation to other programmes that have received support in outcomes planning.

First, the amount of resources required to take an outcomes-focused approach was out of sync with the funding period that would allow for self-evaluation to become embedded. As the evaluation of the HWCs highlights,

the relatively short time scale of funding for the projects has created a range of issues around the sustainability of the approach – and so of the scale of the difference that it may make in the longer term.…There remains a risk that the period of funding will not be enough to build capacity and embed ownership, approaches and structures to ensure the sustainability of the approach beyond the end of funding. (Scottish Government, 2011b: 6)

There needs to be commitment to an outcomes-focused approach and a major indicator of this comes from the funder, in order to allow for the approach to be sustained and to enhance the scope for changes to manifest. This adds weight to the conclusions of Taylor and Sumariwalla (1993), who found in their study that the single greatest barrier in implementing outcome evaluation is the lack of financial support from funders. Second, programme staff who worked on the HWC largely worked on other local authority programmes, which meant that there were often competing priorities. This points to more classic organisational reasons for implementation becoming a problem when organisational adjustments may be required – funding cycles, scarcity of resources, and competing priorities (Mayne, 2007). Nevertheless, experience of facilitating outcomes planning in the public sector in Scotland has proven that the approach has considerable value as an organisational development tool. In this respect, comparisons can be drawn here with what the Chartered Institute for Personnel Development (CIPD) regard as important characteristics of organisational development. This includes planning in a systematic/participative way in order to take into account the whole organisation with a view to maximising engagement, ownership and contributions – which helps to enhance the quality and effectiveness of organisational activities (CIPD, 2012).

Conclusion

A major theme of this article has been to draw together the learning and reflections from those leading the capacity building and professional development programmes in NHS Health Scotland in relation to evaluating complexity, particularly in relation to public health policies and programmes. The article has highlighted key steps that make up the facilitation of outcomes planning within the workshop environment. This learning is grouped around an analysis of current problems, risks, needs and gaps in evidence, the construction of an outcomes plan, and outcomes–based performance reporting. This article is novel and fills a gap in the literature on outcomes planning in that it provides an accessible set of resources and learning around the evaluation process which can support public managers in their own professional development. Learning around the key challenges with outcomes planning includes: evidencing change (with implications for measurement and attribution); leadership to take forward locally; conceptual understandings in relation to key terms and spheres of influence; resource limitations (short-term funding cycles and competing organisational priorities); and identifying appropriate indicators for monitoring. The article also highlights that the workshops require the sourcing of facilitators who have a range of skills and expertise, specifically in relation to managing group dynamics, combined with knowledge of evaluation research methodologies (ideally supported by applied experience of monitoring and evaluation).

The parallels to outcomes-focused planning and organisational development theory are clear in that it allows the specification of the problem that need(s) to be addressed and a broad range of interventions using evaluative approaches to deal with complexity that meets the demands of multiple stakeholders (Smith and Petticrew, 2010; Craig, 2013). In short, this is with a view to maximise organisational effectiveness through evaluative cultures and practice. It is within this context that the article adds to the evaluation literature by providing learning from the point of view of facilitators working “at the coal face” with public managers. The article has shown that outcomes planning is an approach that helps public managers demonstrate the impact of their work within multi-partnership environments in the context of outcomes-focused accountability processes (in Scotland this is largely been driven by SOAs).

The process of facilitating the evaluation of complexity can be a rather positive and satisfying experience from a professional point of view, in that that the facilitator has a crucial role in providing guidance to programmes around effectiveness. The outputs (mainly the outcomes plan) from an outcomes planning session represent progress in that they provide stakeholders with products and tools to take forward and develop further (often with follow-up support). With this in mind, the challenges of facilitating this process need to be seen in the context of the value of undertaking the process in the first place – that public managers are being supported to work collaboratively with partners to address societal problems with the intention of doing some good in society (Tannahill, 2008).

Footnotes

Acknowledgement

The opinions expressed in this article are those of the authors and not necessarily those of NHS Health Scotland.

Notes

References

Aldrich

Ruef

(2006) Organizations Evolving. (2nd ed). London: SAGE.

Argyris

(1999) On Organizational Learning. Oxford: Blackwell.

Blamey

Mackenzie

(2007) Theories of change and realistic evaluation. Peas in a pod or apples and oranges? Evaluation 13(4): 439–455.

Cabinet Office (1999) Modernizing Government. Cm 4310. London: The Stationery Office.

Carter

(1984) Measuring up: fear of failure could give way to the surprise of success. Public Welfare 42: 6–13.

Chelimsky

(1994) Making evaluation units effective. In Wholey

Hatry

Newcomer

(eds), Handbook of Practical Programme Evaluation. San Francisco: Jossey Bass.

CIPD (2012) Organisational Development Factsheet. Available at http://www.cipd.co.uk/hr-resources/factsheets/organisation-development.aspx.

Craig

(2013) Seeing the wood and the trees: using outcomes frameworks to inform planning, monitoring and evaluation in public health. Journal of Public Health 35(2): 1–8.

Earl

Carden

Smutylo

(2001) Outcome Mapping: The Challenges of Assessing Development Impacts: Building Learning and Reflection into Development Programmes. Ottawa, Ontario, Canada: IDRC.

10.

Flyvbjerg

(2006) Five misunderstandings about case study research. Qualitative Inquiry 12(2): 219–245.

11.

Hackett

(2003) Training Practice. London: CIPD.

12.

Harrison

(2005) Learning and Development (5th ed). London: CIPD.

13.

Institute for Research and Innovation in the Social Sciences (IRISS) (2010) Leading for Outcomes: A Guide. Available at http://www.iriss.org.uk/resources/leading-outcomes-guide.

14.

Klatt

(1999) The Ultimate Training Workshop Handbook. New York: McGraw-Hill.

15.

Learmonth

Watson

(1999) Constructing evidence-based health promotion: perspectives from the field. Critical Public Health 9(4): 317–333.

16.

Levitt

March

(1988) Organizational learning. Annual Review of Sociology 14: 319–340.

17.

Mayne

(1999) Addressing Attribution through Contribution Analysis: Using Performance Measures Sensibly.Available at http://www.oag-bvg.gc.ca/internet/docs/99dp1_e.pdf.

18.

Mayne

(2004) Reporting on outcomes: setting performance expectations and telling performance stories. Canadian Journal of Program Evaluation 19(1): 31–60.

19.

Mayne

(2007) Evaluation for accountability: myth or reality? In Bemelmans-Videc

M-L

Lonsdale

Perrin

(eds), Making Accountability Work. New Brunswick, NJ: Transaction Publishers.

20.

Mayne

(2012) Contribution analysis: coming of age? Evaluation 18(3): 270–280.

21.

Office of the Auditor General of Canada (2013) Spring 2013 Report of the Auditor General of Canada. Chapter 1, Status report on evaluating the effectiveness of programmes. Available at http://www.oag-bvg.gc.ca/internet/English/parl_oag_201304_e_38212.html.

22.

Patton

(2008) Utilisation-Focused Evaluation. Thousand Oaks, CA: SAGE.

23.

Poole

Davis

Reisman

Nelson

(2001) Improving the quality of outcome evaluation plans. Nonprofit Management and Leadership 11(4): 405–421.

24.

Sanderson

(2011) Evidence-based policy or policy-based evidence? Reflections on Scottish experience. Evidence and Policy 7(1): 59–76.

25.

Scottish Government (2008) Healthy Eating, Active Living: An Action Plan to Improve Diet, Increase Physical Activity, and Tackle Obesity (2008-2011). Available at http://www.scotland.gov.uk/Resource/Doc/228860/0061963.pdf.

26.

Scottish Government (2009) Changing Scotland’s Relationship with Alcohol: A Framework for Action. Available at http://www.scotland.gov.uk/Publications/2009/03/04144703/0.

27.

Scottish Government (2010) Preventing Overweight and Obesity in Scotland: A Routemap Towards Healthy Weight. Available at http://www.scotland.gov.uk/Resource/Doc/302783/0094795.pdf.

28.

Scottish Government (2011a) Improving Maternal and Infant Nutrition: A Framework for Action. Available at http://www.scotland.gov.uk/Publications/2011/01/13095228/0.

29.

Scottish Government (2011b) Healthy Weight Communities Programme Evaluation. Available at http://www.scotland.gov.uk/Publications/2011/08/04154123/0.

30.

Smith

Petticrew

(2010) Public health evaluation in the twenty-first century: time to see the wood as well as the trees. Journal of Public Health 32(1): 2–7.

31.

Stewart

Riggs

(2010) Learning and Talent Development. London: CIPD.

32.

Swinney

(2009) Unchartered territory. Public Service Magazine (June).

33.

Tannahill

(2008) Beyond evidence–to ethics: a decision-making framework for health promotion. Health Promotion International 23(4): 380–390.

34.

Taylor

Sumariwalla

(1993) Evaluating non-profit effectiveness: overcoming the barriers. In Young

Hollister

Hodgkinson

(eds), Governing, Leading and Managing Nonprofit Organisations. San Franciso, CA: Jossey-Bass.

35.

The Equality Act 2010 (Specific Duties) (Scotland) Regulations 2012 No. 162.

36.

Weiss

(1997) How can theory-based evaluation make greater headway? Evaluation Review 21(4): 501–508.

37.

Wimbush

(2011) Implementing an outcomes approach to public management and accountability in the UK—are we learning the lessons? Public Money & Management 31(3): 211–218.

38.

Wimbush

Montague S Mulherin

(2012) Applications of contribution analysis to outcome planning and impact evaluation. Evaluation 18(3): 310–329.