Using developmental evaluation principles to build capacity for knowledge mobilisation in health and social care

Practical steps in creating EfC

Prior to EfC’s launch (June 2014–April 2015), policy documents, published literature and stakeholder feedback were gathered and analysed by the EfC developers to create a flexible capacity building framework for EfC. The framework was used to advertise EfC to local health and social care providers. Data from the completed application forms and team selection interviewers were analysed to refine the framework, develop workshop 1 and the online portal, and select and brief the EfC facilitators.

During EfC’s lifespan (May–December 2015), EfC was incrementally and conjointly developed together with the qualitative impact evaluation. EfC was tested and developed in real-time using feedback and feedforward (Ferrell and Gray, 2016) collected during and after each workshop from the participating teams, together with observations by the EfC facilitators and developers of the teams’ developments, discussions and dilemmas.

EfC team members and EfC facilitators were invited to take part in the pre-launch and lifespan stages of EfC’s development. The participants’ line managers and EfC funders were invited to contribute to the pre-launch stage of EfC’s development.

Using DE principles to create EfC

To ensure EfC was fit for purpose, it was piloted by the participating teams for whom it was designed and tested live within the complex environment within which the teams were operating. This enabled the EfC developers to understand and interpret development through the complexity lens (Patton, 2016).

The EfC developers had a non-standard DE role, while they were embedded within the design team, they were not independent evaluators focusing solely upon the evaluative elements of the developmental work. Instead, they held multifaceted roles, they were the EfC designers and leaders and as such, they were key decision makers in the development and delivery of EfC. As EfC is a capacity building initiative, the EfC developers were also strategic learning partners (Preskill and Beer, 2012) supporting the participating teams in evaluative thinking (Lam and Shulha, 2015). Finally, they were also insider researchers (Robson, 2000) responsible for evaluating the impacts of EfC. Due in part to this complex and potentially conflicted role and partly to support their own learning in relation to evaluation, the EfC developers enlisted a highly experienced evaluator from Lancaster University, as an evaluation mentor and critical friend (Costa and Killick, 1993).

The EfC developers were the intended users of the DE evaluation data; therefore, the timing and coherence of DE feedback needed to fit with their requirements and deadlines. However, as depicted in Figure 1, the participating teams were the intended users of EfC; therefore, the EfC developers needed to use data collection methods that would not disrupt the teams’ activities and would collect sufficient and timely data to enable the shaping of EfC so that it was fit for purpose. To achieve both goals, the EfC developers planned opportunities within the workshops to regularly ask probing questions and seek feedback. Workshop activities were designed to elicit feedback about team projects throughout the workshop day. At the end of the workshop, team members were asked for feedback on the workshop (what worked or did not?) and importantly feedforward (what would help next time?) (Ferrell and Gray, 2016). Observations made by EfC facilitators were utilised to elucidate and validate participants’ feedback and progress. The EfC developers’ meetings between each workshop provided an opportunity to discuss the evaluation feedback and reflect upon progress against the flexible capacity building framework. The findings informed the design of the forthcoming workshop and shaped facilitator support offered outside of the workshops. Therefore, the initiative and evaluation were both interwoven, interdependent, iterative and co-created and made use of the naturally occurring evidence available (Patton, 2016). As depicted in Figure 1, the systems thinking principle was applied to EfC’s design by the sequence of continually testing developments live in the field and utilising immediate feedback from both the EfC teams and their stakeholders.

OPEN IN VIEWER

Figure 1.

The continual sequence of action and feedback that took place between the EfC developers, the participating teams and the teams’ stakeholders.

Embedding DE principles into EfC’s learning approach

EfC’s design utilised elements of Blended Learning (Glazer, 2012), which combined online and workshop support for learning; Work-Based Learning (Basit et al., 2015; Roodhouse, 2010), which involved learning from and for work; and Experiential Learning (Kolb, 1984), which involved learning, adapting and developing through reflection on a sequence of learning experiences. Regular opportunities were also provided for Team Reflexivity (Schippers et al., 2015), which involved the team in reflecting on how they were working and acting together. As the teams directed and led their own evidence-informed change projects, elements of Participatory Action Research (Reason and Bradbury, 2008) and Collaborative Inquiry (Bray et al., 2000) were also incorporated into EfC. While elements of these teaching, learning and research approaches are reflected in the following narrative, the focus is on the DE principles that were embedded into EfC’s learning approach.

The teams participating in EfC were required to bring a key issue or challenge from their practice to address and this formed the basis of their live project. In keeping with DE, the teams’ projects were being developed step-by-step, in real-time as EfC unfolded.

Each team had a leader who was a health or social care practitioner responsible for the project. Teams were requested to be multidisciplinary and multiagency in which all members were equal peers including the public representatives. The members were selected on the basis of being intended service users (e.g. patients, health and social care practitioners, commissioners) who represented various levels of decision making from across the system. The teams were encouraged to think systemically, involving a cross section of stakeholders in order to consider the interrelationships, perspectives, boundaries and social systems (Patton, 2016) within which they wished to make impactful practice change.

To support systemic thinking, EfC was sequenced to regularly allow time in the workplace for teams to test in practice the ideas they developed in the workshops and receive ongoing timely feedback from a variety of stakeholders (Patton, 2016). To enable evaluation rigour and utilisation (Patton, 2016), time was provided in the next workshop to share feedback, to collectively reflect upon impacts and to inform team decisions. The practical exercises were designed to enable the taught content (theory) to be directly applied to the teams’ projects. At the end of the workshop day, each team devised an action plan to steer the next phase of their workplace activity. Individual team members also met together regularly between workshops to move their project forward.

Alongside identifying organisational, stakeholder and local evidence, the teams were supported to identify, analyse and evaluate scientific peer-reviewed evidence. The objective was that teams would develop their projects by continuously interrogating the evidence base, while taking into account the local context by testing out their designs in practice.

EfC focused on flexibility and learning and therefore, where necessary, teams were encouraged to change plans as they progressed, to ensure their projects and the project evaluation data they gathered were suitable for the intended use by the intended users (Patton, 2016). For example, the teams were not pushed to set aims and objectives too early. Instead, the teams were encouraged to have intended aims and a general sense of direction, to be open and receptive to their stakeholders’ feedback, as well as their own observations and be willing to adapt in order to ensure their projects were in-step with any changes in the environment. Teams therefore regularly reviewed and adapted their project aims and objectives to ensure they aligned with their changing environments.

The concluding dissemination event was as an opportunity for teams to consolidate their learning and to formally share their results with their stakeholders. The teams therefore created both an academic poster and a 40-minute presentation. In keeping with the innovation niche principle (Patton, 2016), the dissemination event provided the EfC teams with an opportunity to explain how the processes and concepts they were developing were innovative.

The EfC teams did not have an embedded evaluator as a part of their team, instead each team was assigned a facilitator, provided by CLAHRC NWC. The facilitators were academic subject experts and their role was to facilitate team learning. They supported team discussion and reflection within the workshops by asking strategic and evaluative questions (Lam and Shulha, 2015) and kept teams focused throughout the day. Facilitators were also responsible for providing information, network contacts and mentoring support outside of the workshops, as and when required. The facilitators supported the teams but were not embedded within them.

Qualitative impact evaluation methods

The qualitative impact evaluation, designed during EfC’s lifespan, was conducted post-initiative to determine EfC outcomes and impacts. Impact data were gathered, January to September 2016, from participant online questionnaires using open questions and Likert-type scales (piloted prior to issue by a cross sample of participants), individual team focus groups, EfC facilitator focus groups, structured interviews with participants’ line managers, team videos, posters, presentations and reports. (Focus groups and interviews were conducted by an interviewer and a note taker, audio taped, transcribed verbatim and anonymised). Learning from EfC was shared with teams, facilitators, the funder and wider audiences via reports, posters, a video, conference presentations and discussions.

Data analysis

Overarching indicators based on Saunders’ (2007, 2011) model were adapted and developed by the authors to measure impact arising from learning and change across five levels. The newly developed indicators were: Level 1 (Self), Level 2 (Personal Practice), Level 3 (Team or Department), Level 4 (Organisation or Local Community) and Level 5 (Professional Sector or Wider Society).

During the preformative phase of EfC, the five overarching indicators were used repeatedly by the developers to ascertain the effectiveness of EfC as it unfolded. This process informed the decisions about the next stage of EfC’s development. During the qualitative impact evaluation, the overarching indicators were used to aid the thematic analysis (Braun and Clarke, 2006; Ritchie et al., 2014) of the endpoint impact data.

Using an iterative process, the qualitative impact evaluation data gathered post-initiative were uploaded to Atlas.ti (2016–2017) by EfC developer (L.M.H.) and coded against the overarching indicators using deductive analysis (Braun and Clarke, 2006; Ritchie et al., 2014). This process enabled data to be assigned to each of the five levels. To explore the relationship within and between the five levels, the data were then refined further by inductive analysis (Braun and Clarke, 2006; Ritchie et al., 2014) to identify new codes within the data at each level. Using these new codes, the data within each level were then recoded and analysed to illuminate topics and sub-topics. Validity, accuracy and consistency of coding were checked by EfC developer (R.D.) and an evaluation mentor. The different perspectives provided by the four groups of participants (team members, facilitators, EfC developers and line managers) were used to elucidate and confirm meaning arising from the qualitative impact data.

Ethical considerations

Informed consent was gained from all those taking part in EfC. Ethical approval from the University of Liverpool was granted to subsequently involve the team members’ line managers (Reference Number IPHS-1516-147). All data relating to the evaluation are securely stored at the University of Liverpool and all quotes have been anonymised to protect the identity of the participants.

The outcomes of using DE to both create EfC and to underpin EfC’s learning approach

EfC was created from scratch and built incrementally over 8 months using a DE approach to guide the development work. Shaped by the analysis of the EfC pre-launch data, EfC’s flexible capacity building framework incorporated an allocation of 2 days per month in the workplace to conduct a project and consult with stakeholders, interspersed with 5 day-long workshops and a dissemination event. An online portal, housing workshop materials and university library access, was also provided. Each team was supported within and outside of the workshops by a facilitator, who was an academic subject expert. Using stakeholder feedback and observations during EfC’s lifecycle enabled the just-in-time development of the workshop content and the tailoring of support provided outside of the workshops.

Four teams came forward to take part in EfC. Each team brought with them a live project brief to work on, as follows: Team 1- Improve esteem for dyslexic teenagers in an area of high deprivation; Team 2- Change organisational policy and practice around self-injury harm minimisation; Team 3- Reduce premature death from bowel cancer in an inner city Black, Asian and Minority Ethnic community; and Team 4- Reduce ambulance callouts in elderly care.

None of the teams taking part in EfC dropped out; all four teams were newly formed and multidisciplinary and three were also multiagency. Three teams remained the same in composition and size, whereas one team steadily grew in size over the duration of the project. This is a significant outcome as capacity building and professional development programmes are known to experience dropout. Each team also accessed and analysed scientific and local evidence in order to make a successful evidence-informed practice change. This is also a significant outcome, because the mobilisation of knowledge into health and social care practice is commonly known to be problematic (Grimshaw et al., 2012; Lau et al., 2016; Ward et al., 2012).

Reported outcomes for each of the four teams were Team 1, joined forces with another team within their organisation conducting similar work, they contributed the evidence-base they had developed within their EfC project which helped to shape their organisation’s work going forward; Team 2, used the evidence-base generated within their EfC project to write an organisational policy and work procedure which they presented to their board for consideration; Team 3, used the evidence-base generated within their EfC project to gain funding for the continuation and evaluation of their project, they have since piloted their EfC project in 11 general practitioner (GP) surgeries; Team 4, were persuaded by their public adviser members to change the focus of their project, which they successfully achieved. They used the evidence-base generated within their EfC project to create a tool for supporting communication between care home residents, carers, family and practitioners which was adopted and used by their local community. The results therefore demonstrate that EfC was fit for purpose.

A significant non-linear outcome (Patton, 2011) of EfC was the capacity development of CLAHRC NWC through the development of the EfC facilitators. The facilitators reported developing a deeper understanding of the complexity of the health and social care environment in which CLAHRC NWC partners were operating, developing new networks and developing the ability to undertake future DE. The facilitators’ learning was also put to good use in incrementally developing a similar new initiative for CLAHRC NWC, named ‘the Partner Priority Programme’, as well as contributing to the development of public engagement policies and practices.

Results of the qualitative impact evaluation

The response rates to the online questionnaires were as follows: team members 50 per cent (n = 15), facilitators and subject experts 83 per cent (n = 5) and EfC developers plus an administrator 100 per cent (n = 3). Three of the four EfC teams took part in a team-specific focus group, one team declined due to logistical reasons; these focus groups excluded the EfC facilitators. The EfC facilitators and EfC developers took part in a focus group held specifically for them. Focus group sizes ranged from two to nine participants with 21 taking part in total. Each team contributed one line manager interview (four in total).

Qualitative impact findings

Analysis of the qualitative impact evaluation data highlighted five interconnected themes, developed through capacity building, which enabled successful EfC team outcomes. These were teamwork, involving public as equal peers, increased communication, learning and development, and increases in positive attitudes and emotions. In addition, the role of organisational support was acknowledged as being important.

Theme 1- teamwork. Team members identified that, while it was difficult to begin with, ‘chaotic with a lot of arguments’ (Team member) and ‘meandering around’ (Team member) to establish the project and develop roles within the team, members reported that, over time, their team bonded. They increased their awareness of each other’s skills, developed a shared understanding, shared tasks and responsibilities, felt supported by the team and stated that the team stuck together during difficult times. Reflected in this feedback is the reality of working developmentally (Patton, 2011), of not having rigidly set plans to work to and the complexity of involving stakeholders from across the system. However, over time, team members felt it was important to work as equal peers and that having a variety of views helped them to gain new insights, to think through ideas and to shape their projects.

… as a multiagency team, the fact that we were there from different backgrounds, different perspectives but we were equal on that team …, we were all coming at things from slightly different perspectives and it was really helpful just having that time and space to focus on a piece of work … (Team member)

… it was impressive that you know there were people from multiple levels in different organisations and they worked extremely collaboratively and extremely respectfully together, there didn’t seem to be a hierarchy as such within teams. (Facilitator)

The team leader’s role was highlighted as being central to the team’s success. Team leaders were seen as being responsible for selecting team members with relevant skills, developing and supporting the team, being open to new ideas, enabling flexibility and change within the project, and keeping the team moving forward. It is possible that the team leaders were acting as informal developmental evaluators, leading their teams through a cycle of action and reflection in order to incrementally develop their projects.

… he (the team leader) had thoughts about the whole programme, the overall tasks needed to be achieved to lead the programme and he chose the professionals or the people to be involved … (Team member)

… this project has certainly made me think, and I think others think around how important it is to get the right people involved and engaged right from the beginning. (Team member)

The diverse membership achieved within each team enabled the teams to connect with a wide range of stakeholders from across the system within which they operated, thus each team was able to communicate more efficiently and effectively with their stakeholders. Each team’s collegiality appeared to have a positive impact on team members’ motivation and productivity. Team members also reported cascading their learning to others outside of the team by using their teamwork skills, sharing information, raising awareness and challenging perceptions, which helped prepare the ground for change.

Theme 2- involving the public as equal peers. Each team took a different approach to public involvement. One team consulted with practitioners, parents and young people to gain a variety of service user perspectives; one team involved an expert by experience to guide their work, while another involved six public members in order to represent the multicultural local community and both, they and one other team, facilitated wider public engagement through their public members.

Evaluation feedback indicated that team members who were practitioners felt they had learnt about the role and the value of involving the public. Several practitioners reported becoming disconnected from their service users by moving up the management ladder and now understood the importance of being reconnected with them. They explained how valuable the public members’ contributions were in shaping the project and in helping the team to communicate with wider service user groups.

It was a good reminder of the benefits of going to care settings to see the work in practice and to feel what difference the work is making, as I sometimes feel removed from this. (Team member)

Overall I would say it was brilliant and especially the invaluable contribution of the public advisory panel members when they came on board … (Team member)

All practitioners stated how much they had enjoyed the public involvement aspect of the project and most stated they had plans to embed public involvement into their future practice.

In terms of public engagement I guess, it now seems preposterous to continue any kind of project that’s around healthcare provision without involving the people who use the service … (Team member)

Conversely, team members who represented the public explained that they were apprehensive to begin with and not sure of how they might contribute to the team. However, with the team’s support, they felt respected and listened to, which helped them to engage and contribute. They stated they had gained confidence in communicating with practitioners and learnt about the services they were supporting.

… I initially started apprehensive because I thought you know as a public member, not being a professional, am I able to contribute anything? What is it that I can contribute? And it wasn’t until I started on the project … one of the proudest moments was when I was at one of the workshops and … actually I did mention something and everyone turned round and said yeah you know that is something that we were thinking of as well … We think of professionals from a public point of view as hierarchy and you feel you can’t really communicate but I felt that, that barrier as well has gone down. (Team member)

Both parties felt that public involvement helped to shape their project so that it was more likely to be fit for purpose. One team completely redesigned their project and refocused their aims based upon their public member’s advice.

… I was surprised I think by the extent to which our PPI’s (public member’s) involvement shaped the project … I knew he would be a really key person in the team but I don’t think I realised how much it would shift what we did … (Team member)

I have learnt that the people you are trying to effect the change for can actually, with support, create the change themselves. (Team member)

Theme 3- increased communication (including networking and stakeholder feedback). Participants reported increased two-way communication between the team and a variety of stakeholders on many levels, including national government agencies, clinical commissioning groups, senior executive boards, local community members and family. Team members explained how some meetings about their projects were difficult and challenging with many questions asked. They also outlined the importance of gaining respect from community leaders and clinicians so that the door for future communication could be left open.

… well it’s got out to the community which I think is the most important part to influence you know … the community members involved, they’ve gone into their communities and spread the word, that’s probably going to be the most effective part of the whole thing, not the phone calls to the patients, probably word of mouth within the communities, so by getting, engaging with the community I think that could have the greatest long term impact really … (Team member)

Team members expressed how surprised and motivated they were when they heard positive feedback from senior managers and local communities about their project. They explained how this feedback helped them to reflect and shape their plans. They outlined how this impacted on their perceptions of their own abilities leading to an increase in confidence and explained how they used their learning to influence and challenge other people.

We have just organised a Health Awareness Event addressing, highlighting those inequalities and especially cancer awareness. We were overwhelmed, more than 230 attended, and the evaluation is great. (Team member)

Team members and line managers outlined how a variety of media enabled them to reach a wider audience; in particular, they noted the impact of short videos, posters, reports and community workshops.

… certainly I’ve had feedback from councillors, executive directors and head of safeguarding and the general feedback is this is fantastic, this is fabulous, you need to show this (the video), … so it’s starting to spread … I think films are very powerful … (Line manager)

Theme 4- learning and development (including skills and knowledge development). Team members and facilitators reported four key aspects to their learning:

Theme 5- increases in positive attitudes and emotions. Team members’ confidence, motivation and morale were reported to have increased, which appeared to have positive impacts on their ability to undertake actions and to challenge others. Team members and facilitators unanimously stated the enjoyment and satisfaction they gained from being involved in the team and in the project. There is evidence that team members, including the public members and facilitators, continued with their project work, their engagement and their networking after EfC concluded.

I now feel more confident about speaking about the topic area and engaging people in our project. (Team member)

… the staff that were involved in the programme (EfC), I think it strengthened their belief in what they were doing and now they have the evidence to say what they were doing, what they were thinking, what they were taking out into practice was right because actually some of this is going on in practice so they’re now kind of doing that with a lot more confidence … (Line manager)

I think they’ve (two team members) gained, certainly from talking to them, a lot of personal satisfaction about being involved in the programme (EfC) … (Line manager)

Organisational support was additionally seen as key by many team members. Of particular note was protected time to work together and undertake the project, line manager support, organisational buy-in and support from other collaborating organisations who provided resources.

I think in terms of authority to act though the fact that we have that level of buy-in from that level in the organisation, from your (line manager) level of seniority in the organisation, that has made the world of difference, because had I not had (line manager name removed for anonymity) to go to then I probably couldn’t make those things happen. (Team member)

In summary, the five themes reveal that the diverse membership of each team enabled the team to gain a shared understanding of the problem, to support each other, and to communicate with and gain feedback from a wide range of stakeholders from across the system where change was required. This helped to reduce communication barriers between stakeholders and led to an increase in networking and feedback. Team members were able to apply their learning directly within their EfC project, to learn by experience, gain new perspectives and even transfer their learning beyond their EfC project. This process led to an increase in positive attitudes and self-belief, which enabled team members to challenge others, take action and pave the way for change.

What do the findings mean for knowledge mobilisation in health and social care?

EfC brought together academics, health and social care practitioners and service users to co-produce (Batalden et al., 2015; Marshall et al., 2014; SCIE, 2015) evidence-informed practice change. This was in keeping with the requirements of health and social care reforms (National Health Service, 2014, 2017) and the aims of EfC’s funders (CLAHRC NWC). The qualitative impact evaluation demonstrated that by being involved in EfC, the teams were starting to reduce barriers between and across the professions and between professionals and service users (Grimshaw et al., 2012). In addition, by engaging their public members as equal peers, the multidisciplinary teams increased their ability to communicate with a cross section of stakeholders from within the system where change was required. By using evidence to inform their decisions, the teams also demonstrated that they were starting a journey towards reducing the evidence to practice gap (Grimshaw et al., 2012; Lau et al., 2016; Ward et al., 2012).

As the environment was both complex and shifting, the iterative DE process enabled team members to engage with key stakeholders, incrementally test out ideas in practice, receive timely feedback and adapt their plans in order to take account of newly emerging information and previously overlooked service user needs. Taking account of service user needs is important within health and social care as it helps to ensure services are fit for purpose and helps towards reducing health inequalities by creating services that are accessible and suitable for service users (Whitehead, 2014).

What has been learnt from the qualitative impact evaluation of EfC?

The results of the qualitative impact evaluation demonstrate that the teams upheld the co-creation, systems thinking and timely feedback principles as applied within their complex environments (Patton, 2016). The steps that supported the EfC teams’ positive outcomes are shown in Box 1.

The continual communication with multiple stakeholders at many levels, as shown in Box 1, enabled three interconnected processes to take place, that facilitated change:

What do the findings mean for capacity building?

It is feasible that the constructive alignment (Biggs, 2003) of EfC’s learning approach with the environment in which the teams were operating has supported the positive outcomes noted in the qualitative impact evaluation. The alignment was achieved in two ways. First, the DE approach enabled the EfC developers to explore stakeholder requirements, in order to create a flexible capacity building framework. The framework, together with team and facilitator feedback were used in real-time to create each workshop to suit the teams’ emerging needs. Second, by using DE principles to shape the workshop structure, influence team composition and guide the learning processes, the DE principles were embedded into EfC’s learning approach. This provided a guide to support the teams to develop their own projects. The key elements of DE that supported the teams to learn and adapt within their complex environments were: engaging teams of equal peers who represented stakeholders from across the system where change was required, thus enabling systems exploration and thinking; sequencing support for the teams around their real-time projects enabling them to test ideas and adjust plans as they went; actively encouraging and providing opportunities for the teams to gather evidence and reflect upon impacts from a variety of diverse perspectives, before planning the next step, thus encouraging co-creation by incorporating DE processes into the change process. Therefore, this article adds to modern learning theories by explaining how DE principles can be used to underpin learning and capacity building approaches to support teams to innovate and change within complex environments.

What do the findings mean for DE practice?

The role of DE is to support the ongoing development and change required to enable projects to adapt to emerging and complex environments (Dozois et al., 2010; Gamble, 2008; Patton, 2011). Due to the adaptation required within such complex environments, a standard DE approach, a recipe for how to conduct a perfect DE, is not feasible. As Patton (2011) outlines, the adaptive developmental evaluator is a bricoleur using whatever can be found to fit the job. In this vein, the EfC developers also created their own way forward in the development of EfC. For example, unlike the cases outlined by Lam and Shulha (2015) and Lawrence et al. (2018), the EfC developers were the evaluators, developers and initiative leads, they held multifaceted roles. Bias was limited and objectivity was provided by employing an experienced evaluation mentor to question assumptions, and check and validate the EfC developers’ findings. The process of using EfC facilitators also provided a level of crosschecking and exploration to balance the views of the EfC developers. Feedback, fact checking and evidence provided by the EfC teams and their line managers also supported the EfC developers to think and act objectively in relation to the development of EfC and in conducting the qualitative impact evaluation. It maybe that the role the EfC developers played is not standard practice for DE, but it has still proven to provide successful outcomes.

It is suggested that evaluation utility can be greatly enhanced by in-depth knowledge of both DE and the field being changed (Lam and Shulha, 2015). Given the lack of experienced DE experts and the limited funding available at the time, it was not feasible to embed a DE specialist within each team. The EfC teams, however, accessed DE expertise from the EfC developers and gained in-depth and previously unexplored knowledge of the field by involving public members together with a variety of practitioners. Considering the qualitative impact evaluation feedback and the positive outcomes achieved by the teams, it is possible that the EfC team leaders guided by the EfC initiative became informal developmental evaluators leading their own teams steadily forward in the incremental development of their evidence-informed projects, which raises the question of whether specialist DE experts need to be embedded within each project, or whether through a capacity building initiative such as EfC, they could be sufficiently effective while taking a more devolved role.

The four stages to the adaptive cycle outlined by Patton (2011) involve sequential periods of creation, improvement, conservation and finally termination. This sequence of a project’s life cycle is however questionable in an environment that is constantly adapting to meet the changing, growing, accelerating health and social care needs of a diverse population. In such circumstances, fixing the parameters and co-ordinates of a capacity building initiative like EfC would make the initiative inflexible and unable to respond to the needs of the participants. This accords with Patton’s (2016) later work, where he suggests that in order for the initiative to remain current in a changing world, the developers may opt for an ongoing cycle of creation. The EfC developers also argue that they will continue to need to use DE, especially given the following two key factors. First, EfC is a capacity building initiative designed to support health and social care teams to develop evidence-informed practice change in complex environments. The teams that apply to take part in EfC will always bring with them new change projects and for that reason, the DE principles will remain embedded within EfC’s learning approach. Second, as the needs of future EfC teams are unknown and are likely to change with each cohort of teams and because being immersed in DE is a part of the learning experience for the teams, the EfC developers intend to continue to use a DE approach to create and develop future iterations of EfC. This suggests therefore, that EfC will remain in the creation and development stage until no further teams come forward to take part, at which point it will simply cease without a conservation or termination phase. This factor is likely to need careful consideration in the scaling-up of EfC.

This article also provides a useful example of the benefits of assessing the short- to medium-term impacts of an initiative developed through DE. Fagen et al. (2011) warn that premature summative evaluation can suppress innovation but also suggest DE may be a precursor to a cycle of formative and summative evaluation undertaken once the initiative is established. However, by establishing the short- to medium-term impacts of EfC across five levels, the developers were able to explore the strengths, limitations and unexpected outcomes that unfolded from DE.