Evaluation during war: Current realities and future possibilities of Ukrainian monitoring and evaluation

Evaluation in wartime

This new reality is forcing all of the key players in political, economic and legal spheres to reconsider their well-planned and structured projects, adapting instead to chaos and uncertainty, establishing new approaches to policy-making processes. The sector of monitoring and evaluation is not an exception and is also undergoing a metamorphosis in order to ensure the accuracy of development vectors and mitigate the negative consequences whenever possible.

Given the scale of the conflict, the monitoring and evaluation sector is experiencing difficulties performing under the given circumstances, where decisions are to be made quickly ‘on the ground’. The dynamics and trends of the war are unclear, information is often difficult to verify and much of the data is inaccessible or entirely classified (confidential).

Shift from well-planned development to emergency activities of NGOs leads to serious challenges: (a) harsh constraints with time and human resources create a dilemma on either focusing on immediate lifesaving activities or on aligning with the project team’s new monitoring and evaluation framework and monitoring on the long term; (b) uncertain environment and uneven scenarios cause difficulties with establishing definite targets for many indicators; and (c) composing new programmes and their monitoring and evaluation systems is like gathering a picture from ‘broken pieces’ (staff and beneficiaries scattered across Ukraine and Europe) while simultaneously rapidly adding new pieces.

Humanitarian organisations have had to optimise their activities on saving human and financial resources, directing them to the most crucial areas and reducing the time of providing assistance, which is most important when in the phase of active hostilities.

Organisations have also had to resort to using simplified monitoring methods. Monitoring and evaluation activities are mostly based on numerical statistics of distributions, direct observation of the process of registration and the distribution, as well as Post-Distribution Monitoring through phone calls. At the moment, feedback and complaint mechanisms for the beneficiaries – like hotlines – are understandably loaded.

Documenting war crimes has become another important task for Ukrainian civil society during the Russian invasion. War crimes are numerous and occur mainly in occupied territory. For this reason, authorities cannot conduct documentation immediately after a crime has been committed. The civil society of Ukraine is actively involved in documenting war crimes.

One of the authors of this article, Mikhail Savva, was documenting war crimes in the Kyiv region in February–March 2022 as part of a coalition of public organisations Euromaidan-SOS. In the first weeks of the Russian invasion, they learned through hands-on experience that documenting war crimes requires the skills that evaluation activities develop in people. Documentation, like other types of monitoring, requires the development of a system of indicators (what actions constitute a war crime?), implementation of this system based on professional skills (e.g. the ability to conduct interviews) and external control over the documentation process.

The methodology for documenting war crimes by civilian volunteers was created taking into account the experience of monitoring and evaluation in Ukraine, including in the frontline zone in 2014–2021. This methodology included the following basic ideas:

The coalition has collected information on more than 15,000 war crimes.

The role of international and Ukrainian organisations in combating humanitarian crisis in Ukraine: Monitoring and evaluation perspective

In February 2022, ‘SOS Children’s villages Ukraine’ (SOS UA) ² was one of the organisations shifted from well-planned development activities to emergency procedures. Since 1993, the SOS UA has been comprehensively approaching the problem of orphanhood in Ukraine, working not only with the consequences but also with prevention by supporting vulnerable families. Where it is not possible to maintain the biological family unit, the SOS UA would provide an alternative family environment. During the first 3 months of the war, beneficiaries increased 10-fold from just 2000 to 20,000 vulnerable children and their caregivers due to newly established programmes: evacuation, assistance to injured children, programme of psychosocial support, cash and voucher assistance, establishing new centres of family support in safe regions, assistance to internally displaced families in shelter through implementing partners and so on. Main indicators changed from ‘% of sustainable families’ to ‘# of vulnerable families assisted with lifesaving activities’ and the registering model changed from well-developed software to simplistic excel sheets. Advocacy of child care reform and promotion of family-based care stays one of the main priorities of SOS UA, as the country has the highest level of institutionalisation in Europe. In total, 105,000 of Ukrainian children (1% of all children) are placed in more than 700 orphanages or ‘boarding schools’. Since 24 February, more than 96,000 of them have been returned to their parents or guardians, who often are not able to assure proper care without social support. Another 1900 children were evacuated to other orphanages in Ukraine. The destiny of 26,000 is still unknown, including 7000 children forcibly taken to Russia. In the long term, addressing the issues of orphanage raises the necessity to consider wider need to improve and prioritise childcare reform at the EU level with its proper reflection in Ukraine.

East Europe Foundation is a 100 percent Ukrainian organisation that had to adapt its well-planned pre-war existence to provide help to Ukrainians affected by the conflict. Established in 2008, the Foundation has a track record of implementing successful complex innovative programmes in the areas of the development of democracy and civil society, instituting e-government, developing small and medium enterprises and social entrepreneurship, supporting projects in energy efficiency, and much more. For instance, it contributed to the development of the DIIA Portal ³ that allows access of the Ukrainian citizens to over 90 electronic governmental services. During the Covid-19 pandemic, it launched several distance-learning courses and offered other activities to help other CSOs adapt to the lockdown and prevent burnout in volunteers. After 24 February, it reprogrammed its activities to help Ukrainians affected by war, including the internally displaced persons (IDPs). For the first 6 months of the conflict, it supported establishment and operation of 40 hubs for IDPs and delivered over 320 tonnes of humanitarian aid to affected populations in different regions of Ukraine, which also entailed the development of the in-house capacity to monitor the provision and delivery of humanitarian aid. As in the volatile situation of the unfolding crisis, the information about existing needs was not always readily available, the Foundation had to develop a capacity to quickly assess local needs benefitting from its network of over 400 partner local CSOs from all regions of Ukraine. To ensure quick and targeted response and support small Ukrainian CSOs, it also introduced simplified application and reporting requirements to its subgrantees. Since April, the Foundation also continues its development projects that are concentrated on safety, digital innovations and support to civil society.

Another example is Caritas Ukraine with its wide network of regional offices and nearly 1000 employees and volunteers helping thousands of people who have suffered from the Russian aggression. Since the war started, the number of projects and amount of services provided have increased significantly. The challenge for the employees is to report more often and, in more detail, process a large number of queries related to database maintenance, as well as to communicate with beneficiaries in a bad psychological state. Since the beginning of the war, the organisation has tripled its amount of local MEAL staff (up to 60 local and national specialists) due to deployed emergency response to humanitarian crises. The main fields of work in MEAL are humanitarian response monitoring and needs assessments in various aid clusters like cash interventions, food assistance, water and hygiene delivery. Assessments of needs are a crucial activity for planning any intervention in the dynamic situation of humanitarian crises.

Not only the MEAL staff but also the entirety of the staff of the organisation are facing extreme difficulties in the areas of ongoing hostilities. The organisation is doing everything in its power to support its personnel and the people in need in these areas. Psychological assistance consultations and support for relocation of the staff are the primary activities that the organisation needs to have in order to function properly. A Caritas staff survey recently showed that while 83 per cent reported increased difficulty in their work, 75 per cent of the reports signalled towards psychological and only 8.3 per cent physical hardships. Also, according to the survey, 95 per cent of the staff are really seeing their own efforts benefitting the war-torn society that surrounds them and literally 100 per cent feel the organisation is supporting their work to a satisfactory degree.

Most large humanitarian actors and charities have lost contact with some of their staff members after the occupation of large territories of Ukraine. Moreover, some organisations have faced human losses among their staff. Four regional offices of SOS UA in the Luhansk region ended up under occupation by Russian forces causing up to 20 per cent of SOS UA staff to be unable to continue working and 60 per cent of staff and beneficiaries in the region had to be relocated. Two members of Caritas Ukraine network staff were killed in Mariupol during Russian tank shelling of the local Caritas office. Forty per cent of initial MEAL personnel now are currently within Russian occupied territories with no established contact.

In general, Ukrainian CSOs play very important role in covering both urgent (humanitarian) and long-term (democracy and human rights protection) needs and it will be definitely engaged to the ‘Rebuild Ukraine’ processes. ⁴

Developing policy: Challenges and perspectives

The absence of a single National Evaluation Policy in Ukraine leads to a serious failure of the public policy implementation. There are positive examples of state policy monitoring. One such example is the online platform for public control of state procurements ‘Dozzoro’ ⁵ – a platform that allows businesses whose rights have been violated to leave structured feedback on the tender, buyer or another bidder, and the party to which the complaint is addressed to may respond. Regardless of the positive developments, all state policy monitoring and evaluation needs are currently not met. There are many significant gaps in monitoring and evaluation processes at the national and regional levels. One example is unclear indicators such as ‘Promotion of Social Equity’ and ‘Creation of a favourable business environment’ that are being used in strategic documents as regional development strategies or action plans.

Because of these shortcomings, every new cycle of public policy in most spheres is implemented without proper evaluation of the results achieved. Thus, the next cycle again starts with its core unclear and efficiency unproven.

Becoming EU candidate brings not only new possibilities but also new commitments to Ukraine. One of the serious requirements lies in the monitoring and evaluation domains as an inevitable part of ensuring transparency and accountability of public and international funds spending. Most likely that these processes will be EU-accession driven, although Ukrainian civil society will definitely play an important role here as CSOs already have solid experience in monitoring (and sometimes even watchdogging) authorities while there is still huge space for improvement in the evaluation domain.

Conclusion

Embedding monitoring and evaluation in public policy plays a crucial role in achieving tangible results and advancing the political, economic and legal development of Ukraine. The war has created new challenges in the national monitoring and evaluation policy. Coordination of donor support, proper priority setting and development of new policy-making culture, with integrated monitoring and evaluation, should be the main vectors of the Ukrainian policy-making process. These processes should be an integral part of Ukrainian Recovery Plan considered in the nearest time without waiting for the end of the war.

Living in new realities, Ukrainian CSOs have to develop new monitoring and evaluation capacities and skills enabling them to quickly assess changing humanitarian needs and act accordingly, as well as to combine humanitarian and development activities. Ukrainian Evaluation Association should be a headliner of the process and advocate importance of monitoring and evaluation in the state policy.