Real-time evaluation pros and cons: Lessons from the Gavi Full Country Evaluation in Zambia

Stakeholder buy-in at inception stage

Stakeholder buy-in is very important in real-time evaluation. It enables researchers to have unlimited access to the programme and associated activities and meetings. The evaluation made use of methodologies such as meeting observations and KIIs which are only practical with good buy-in from stakeholders. Therefore, selling the relevance of the evaluation helps prevent resentment which might induce unintended behavioural change. Furthermore, real-time evaluations must regularly provide feedback to the programme and there must be an uptake of recommendations by the implementor.

In order to secure such buy-in, formal correspondence was done and meetings were held at inception with the ministry and other EPI stakeholders, notably; the Child Health Unit (CHU) of the ministry of health (MOH), United Nations Children’s Fund (UNICEF), World Health Organization (WHO), Centre for Infectious Disease Research in Zambia (CIDRZ), Churches Health Association of Zambia (CHAZ), Central Statistical Office (CSO) and OXFAM. The engagement of stakeholders and buy-in was high from inception as a result of this early consultation and is a recommended approach by others who have carried out this form of evaluation (Marjanovic et al., 2017; Sandison, 2003; Saunders et al., 2005.) in order to ensure a common understanding of the evaluation approach and determine its feasibility from the beginning.

Further engagement during planning for buy-in at later stages

In the final year of the FCE, the EPI represented by CHU, participated in the preparation of the annual work plan and priority setting for the 2016 evaluation as a way of getting further buy-in and their input into the evaluation. This was important in the process evaluation to ensure that both the data collection and management burden for the evaluation team and time for providing data on the programme’s side were balanced. These cost implications and the fact that such evaluations are expensive is also reported by Marjanovic et al. (2017) who emphasise a balance between what is desirable and what is feasible in terms of information needs.

Managing expectations of stakeholders regarding evaluator roles

As the evaluation progressed there was misunderstanding and frustration by some stakeholders regarding what the FCE was mandated to do. Observation and interviews revealed that the relationship was seen by some stakeholders as being one of grading the EPI on their performance at the end of every year, rather than being useful to their ongoing planning and decision-making. The FCE was even perceived by some to be Gavi ‘spies’. Some stakeholders indicated that they would have liked the evaluation team to contribute to discussions and decisions at various technical working group (TWG) and the inter-agency coordinating committee (ICC) meetings. This misunderstanding affected notification of the FCE about the meetings that were being held and meant that the team had to work extra hard to ensure that they were aware of all meetings through regular follow up with CHU, a notable challenge to the evaluation during the inception phase.

In order to address these concerns, through several meetings with stakeholders, the FCE explained that as evaluators they were unable to make decisions affecting the same programme they were evaluating as this brought about an obvious conflict of interest (Sandison, 2003). A more appropriate approach that was adopted in response was to be a provider of additional data and information as obtained through the evaluation that would be of value to the programme at various meetings, without participating in decision-making. Sharing of lessons from other FCE countries was an added approach that proved to be of value to Zambia’s EPI.

Making information available to stakeholders in a timely manner for decision-making

In the early part of the FCE, evaluation results were shared through annual reports comprising a country report specific to Zambia and a cross-country report synthesising findings from all the FCE countries. A draft report was initially sent to EPI stakeholders through CHU at the end of the year for their comments before it could be finalised. PowerPoint presentations at TWG meetings were an additional means of getting feedback on the draft report. Stakeholders would then sit to provide written feedback or comments to the draft report. Revisions were made based on this feedback, without compromising the findings. The climax of the report dissemination was a 1-day dissemination workshop involving broader EPI stakeholders that was held in the first-quarter of the following year. This workshop was often attended by heads of institutions and hosted by the permanent secretary at MOH or his or her designate. The purpose of this workshop was to share findings with policy makers and allow all EPI stakeholders to have additional input into the final report.

In addition, rather than sharing findings only through annual reports, the FCE started to share its findings at the various TWG meetings as and when needed to aid ongoing discussions. Furthermore, as the evaluation progressed and the need for more timely sharing of findings emerged, written quarterly briefs were prepared, giving key findings or summaries of lessons from ongoing work and from other countries. A summarised version of the annual report was also prepared in later years of the evaluation that busy EPI stakeholders could quickly engage with. Thus, information sharing became an important aspect of FCE work and the information was more tailored to the needs of the stakeholders. Such information-sharing is also detailed by Marjanovic et al. (2017) in their evaluation of the African Institutions Initiative (AII) and reflects the need to be relevant and useful to the programme being evaluated (Marjanovic et al, 2017; Sandison, 2003).

However, balance had to be reached between sharing emerging findings that were needed for improving implementation versus sharing findings relating to effectiveness (outcomes) of the programme in order to not compromise the external validity of the evaluation (Moore et al., 2015). An additional low effort but highly beneficial way of being useful to the EPI was for the FCE to volunteer to take meeting notes and passing these on to relevant EPI staff for minute preparation and other records. This was also an efficient way of gathering notes to be utilised by the evaluation itself and to ensure active participation in meetings rather than being seen to be aloof observers.

Ensuring added value of the evaluation to programme information needs

In some instances, stakeholders felt that they were already aware of EPI programme challenges that the FCE was highlighting. In such cases, the added value of the FCE was to tease out and understand more deeply the relationships between the various known factors in order to determine what factors were most important to deal with in addressing these challenges and thus make more appropriate and realistic recommendations (Sandison, 2003). This was achieved through root cause analysis (RCA), which is a form of bottleneck analysis that seeks to investigate and categorise the root causes of events (Rooney and Vanden Heauvel, 2004). A ‘root cause’ is a key factor in a causal chain of events that, if removed from the sequence, would prevent the final undesirable or desirable event from occurring or recurring (Rooney and Vanden Heauvel, 2004). RCA was also used to understand which components of the interventions contributed to successes when interventions led to significant results, an important use of process evaluation (Rooney and Vanden Heauvel, 2004). This helped to identify the most important bottlenecks and success factors in addressing known challenges and was appreciated by stakeholders as an added benefit of the FCE. Figure 1 is an example of RCA done for understanding Zambia’s initial HSS application to Gavi in 2015, which required a resubmission at the time.

OPEN IN VIEWER

Figure 1.

Root cause analysis of Zambia’s initial HSS application weaknesses.

Evaluation integration in ongoing programme activities

The FCE normally conducted one round of KIIs and FGDs annually to gather additional data that it may not have been able to get through ongoing observations, FCIs and document review. This was done in order to avoid requesting multiple interviews of stakeholders annually. The process of obtaining permission to conduct the interviews was another consideration for having one round of KIIs annually as it required first having ethical approval from the UNZA research ethics board, administrative approval from the permanent secretary of the ministry and finally approval from relevant organisational heads after the ministerial approval was granted. Then scheduling of interviews was done. Clearly this was a heavy administrative burden that could not be done often.

It was, therefore, important in an evaluation of this nature to ensure that evaluation activities were integrated into programme activities and did not place an unnecessary additional time-burden on an already constrained system such as EPI that had limited human resources both in government and among stakeholder. Methods of gathering information and feedback effectively but with as little disruption to and time required by the programme are needed for such evaluations (Sandison, 2003). These included embedding evaluation activities into ongoing meetings and utilising every opportunity available to gather data and share information. Specific activities that required dedicated stakeholder time, such as KIIs and surveys, were minimised but used effectively.

Evaluation acceptance

As a result of the measures detailed in this section, the FCE was perceived to be part of the team rather than outsiders and was invited to all EPI meetings as part of the official mailing list, as well as included in the EPI WhatsApp group, and thus became more involved in attending programme activities, lessening the need for regular follow up to get meeting notices and gather data. Following these measures, the EPI would often call on the FCE to help understand some phenomenon or data that needed further investigation. Thus, these measures helped the evaluation to be accepted as a useful part of the programme rather than being there purely for the purpose of gathering information for its own reporting needs. This is evidenced in Zambia by the various FCE findings that have been cited in many EPI grant applications, reports and other documents.

Getting consensus and balancing reporting of negative and positive findings

Real-time evaluation presents challenges that are different from evaluating programmes retrospectively. One of these challenges is the need for the evaluator to continually engage with the programme and share evaluation findings regularly. It was the experience of the FCE team that stakeholders normally accepted positive findings better than negative findings. Negative findings presented a challenge given that the evaluation team still had to interact and engage with stakeholders who may have been shown in a negative manner in reports.

Thus, the FCE had to find means of presenting findings in such a manner that the phrasing of the findings, even if they were negative, was acceptable to stakeholders without changing the overall meaning of the findings. It was also important for the evaluation reports to have balanced approach of reporting both positive and negative findings as opposed to the usual evaluation temptation to be stronger on negative findings. The process of getting feedback on the annual report as outlined previously was useful in this aspect as stakeholders felt they were consulted and made contributions before the report was finalised. Thus, the reports and findings were perceived as a reflection of consensus among the various stakeholders in the EPI programme, rather than the potentially biased opinion of any single group of people or the FCE, even when negative findings were reported. Such consensus gathering is a key aspect of many Gavi processes (Gandhi, 2015), including this evaluation.

Weighing the need to report sensitive information

In the course of the real-time evaluation, the evaluators also came across information that was perceived to have been of a sensitive nature, especially with regard to political factors. In such instances, the FCE had to weigh the benefit of this information to both the country stakeholders and to Gavi as the two important users of the reports. The added value of such information to understanding the findings was weighed and such information was included where absolutely necessary, but otherwise excluded. Furthermore, with real-time evaluation the picture evolves quickly and challenges may be transient and not relevant to overall programme achievements in the long run (Sandison, 2003). The strength of the evidence for such sensitive information was thus also weighed and only included if this was very strong. This was important for ensuring the goodwill of the ministry and EPI to the FCE, and is an inevitable consideration in such evaluations. It may be easy for programmes to frustrate evaluation efforts in instances where they are perceived to be antagonistic to the programme, the country’s image and overall objectives and success.

Finding means of participating in the programme without compromising evaluator roles

There were other means of being relevant to a programme being evaluated that did not compromise the evaluator’s role and maintained evaluator independence (Moore et al., 2015), such as note-taking and sharing, providing information as needed and when called upon, and helping to trouble-shoot arising data challenges in the programme in order to understand them better and provide insight to the programme. The emphasis here was on the evaluator building relationships and rapport with people in the programme they were evaluating so that they were more receptive to sharing information and saw the evaluator as a useful partner rather than as an outsider seeking to grade their performance (Marjanovic et al., 2017). That good working relationships are essential to quality process evaluation is recognised in the Medical Research Council’s guidance on conducting process evaluation (Moore et al., 2015).

Avoiding bias

In such circumstances where evaluators are actively engaged at all stages of programme implementation, bias or systematic deviation of results from what they should be (Camfield et al., 2014), needs to be avoided. In particular, friendship bias, a form of contextual bias in which the evaluator may be wary of upsetting programme staff with a very critical report due to having developed a good relationship with them, must be avoided (Camfield et al., 2014; White and Phillips, 2012). This may be further complicated by more cognitive bias of giving more weight to findings or opinions of people the evaluators are most in contact with and thus identify more with (White and Phillips, 2012). Respondent bias, including political correctness and courtesy bias, where key informants responded according to what they thought was the desired answer during interviews (White and Phillips, 2012), sometimes arose as they felt that the FCE was representing Gavi, who were the funders of both the evaluation and a large part of EPI itself.

The FCE team avoided bias by relying on the appropriate TOC to guide investigation and findings, interviewing as wide a range of stakeholders as possible, and reporting both negative and positive findings as appropriate. An example of such a TOC utilised in the Measles/Rubella (MR) campaign is provided in the supplemental material. Accountability to a wider team of evaluators outside Zambia (IHME, PATH and other FCE counties), as well as Gavi as evaluation funders, as described in the section that follows, also helped to address this. In addition, the FCE developed a robustness ranking of evidence which considered triangulation of data (or the number of data sources), the continuum of fact to perception (the more the findings rest on objective factual information as opposed to perceptions, the more robust was the finding) and the quality of data from each source. This was used to grade each finding as outlined in Table 2. In addition, using a mixed-method approach integrating qualitative and quantitative methods allowed more detail about the programme and aided triangulation that neither method could achieve alone (Linnan and Steckler, 2002).

OPEN IN VIEWER

Table 2.

Gavi FCE robustness ranking for findings.

Ranking	Description
A	The finding is supported by multiple data sources (good triangulation) which are generally of good quality. Where fewer data sources exist, the supporting evidence is more factual than subjective.
B	The finding is supported by multiple data sources (good triangulation) of lesser quality, or the finding is supported by fewer data sources (limited triangulation) of good quality but perhaps more perception-based than factual.
C	The finding is supported by few data sources (limited triangulation) and is perception-based, or generally based on data that are viewed as being of lesser quality.
D	The finding is supported by very limited evidence (single source) or by incomplete or unreliable evidence. In the context of this prospective evaluation, findings with this ranking may be preliminary or emerging, with active and ongoing data collection to follow-up.

Source. Gavi (2015) Full Country Evaluation Report – Zambia.

FCE: Full Country Evaluation.

A case in point illustrating this avoidance of bias was the 2015 Gavi FCE annual report (Gavi, 2016) for Zambia which was initially perceived as having been largely negative by many EPI stakeholders. But through the consultative process of validating the report as outlined in the foregoing section, the FCE was able to get consensus from stakeholders on the findings and acknowledgement that 2015 had been a relatively tough year for the programme.

Being relevant to funder needs

As real-time evaluations of this nature are usually funded by donors, in this case Gavi, it was important to ensure that the evaluation also addressed donor needs and not just EPI needs. In order to achieve this, the FCE held monthly update conference calls with the Gavi monitoring and evaluation (M&E) team in order to update them and get their continued input into the work being done. This helped to ensure that evaluation work was on track to meeting funder needs. In addition, the evaluation findings had to be comparable across the four countries without compromising the unique features, context and needs of each country. This was achieved through several mechanisms, including the hosting of mid-year multi-partner meetings with the aim of sharing emerging findings and informing subsequent data collection. Common elements between countries that needed further investigation were identified in this way.

Annual report writing workshops were held, bringing together researchers from all the four countries in order to share and shape individual country annual reports. This was particularly useful in understanding findings from other country context and learning how other countries were dealing with given problems. In addition, these meeting presented an opportunity to compare and harmonise feedback to Gavi. A cross-country report, synthesising the findings emerging from all the countries into cross-country themes was also worked on at the writing workshop. Since IHME and PATH had the advantage of knowing in detail the findings from each country, they were responsible for handling this component of the annual reporting, with input from country teams. Sharing of relevant information and lessons learnt between countries through conference calls and email and sharing of information relevant to the different countries on Basecamp cloud platform were other means of ensuring continued collaboration and information sharing.

Making better use of theories of change, a missed opportunity

In the final year of the evaluation, it became apparent that there was a need to have developed an overall evaluation framework detailing a TOC for the entire EPI programme rather than just for the various Gavi funding streams. As FCE findings, particularly cross-country findings, became clearer year after year it was apparent what the main drivers for successful EPI implementation were. It was thought by both Gavi and FCE implementers that these should have been the basis for the overall TOC and utilised to guide data collection and reporting year by year. This presented a lost opportunity for the FCE to make important comparisons in these elements from year to year within each country and across countries. This points to a rather common trait of process evaluation efforts in that there is general lack of clear, consistent definitions for key process evaluation components and lack of a systematic process for planning and developing complex process evaluation efforts in available literature (Linnan and Steckler, 2002; Saunders et al., 2005), although Linnan and Steckler (2002) and Saunders et al. (2005) do attempt to define this.

It is, however, quite difficult to have a one-size-fits-all solution given the huge variety and complexity of programmes being evaluated. Much learning and adjustment thus occurs in the course of the evaluation. Although, the overarching TOC was developed in the final year and an attempt was made to analyse the data in this way, it was a lost opportunity for the FCE to have done this in a more comprehensive manner. However, it must be remembered that at the beginning of such an evaluation it may be difficult to have a well-described TOC due to lack of existing information and this may only become clearer as data are gathered during the evaluation itself, as was the case here. This difficulty of gathering theory validating evidence is one of the limitations of developing a TOC, particularly in early stages (Care International, 2012).

Process evaluation can, however, also be a means of understanding and improving the TOC once it has been developed and utilised (Linnan and Steckler, 2002; McIntyre et al., 2018). In this case, this was another missed opportunity because other than developing an overall TOC as described, the findings were not used to question or modify the initial assumptions upon which individual funding stream TOCs were based. Thus, the TOCs did not evolve over time through the use of FCE findings. McIntyre et al. (2018) conducted a systematic review of 123 process evaluations and found that none had built or created theory, illustrating that this was by no means a unique challenge to the FCE. However, given that the FCE was planned to continue at the time of writing this article, it is still possible to use the overall findings to adjust these theoretical constructs and outline which elements were more influential in affecting successful programme implementation in order to inform future theory-informed interventions (Linnan and Steckler, 2002).

Evaluation relevance and usefulness to programme implementers

Evaluator roles and balance

Meeting needs of evaluation funders and collaborating with partners