Abstract
ICT is being rapidly introduced into development evaluation. Evaluators need to keep abreast of these important evolutions in the field in order to stay current and continue shaping the future of development work through their practice. ICT is helping to resolve some age-old issues that evaluators face, especially in terms of time, resources and data quality constraints. At the same time, the introduction of ICTs into evaluation brings a series of new ethical, political and methodological challenges that evaluators need to address. The article also weighs the potential opportunities for ICT to strengthen equity and empowerment-focused evaluation against concerns that these new technologies could result in more ‘extractive’ approaches whereby it becomes easier for international agencies and donors to collect information remotely thereby reducing the ability of local communities and civil society to participate in the development dialogue.
Keywords
1. Introduction
Information and Communication Technologies (ICT) 1 such as the Internet, mobile phones, tablets and global positioning systems (GPS) 2 are enabling new methods of gathering and analyzing data and changing how evaluations are conducted. In parallel, the growing demand from national governments, funders, development organizations and local communities for improved data for quick decision-making continues to drive the development of new technology-based tools, systems, and applications. To keep up with the rapidly evolving ICT sector and its implications for development evaluation, evaluation professionals need to become fully conversant with these tools and approaches.
This increased use of ICT for program monitoring and evaluation (M&E) is in part spurred by an explosive global growth in access to ICT, including among the world’s poorest people, raising hope that ICT can contribute to making development processes more inclusive. Digital devices, especially mobile phones, are increasingly available to populations targeted by development funding. These devices have not only changed how people find and use information; they have also opened up new ways for individuals to express opinions, ideas and demands, and additional means for them to engage with and use products and services (GSMA, 2015a). These powerful tools provide marginalized sectors of society with greater opportunities for voice and input into discussions and processes from which they were once excluded.
In this article, we review opportunities and practical challenges in the application of ICT to international development program evaluation 3 with a specific focus on:
logistical and methodological evaluation issues to which ICT can contribute;
new logistical and methodological challenges that arise when incorporating ICT into evaluation;
ways that ICT might strengthen equity-focused, empowerment-focused and/or social justice-focused evaluation approaches.
We highlight ways that innovative technology-enabled data-collection, analysis, and dissemination tools can strengthen equity-focused evaluations in particular, and note methodological challenges that need to be addressed in order for digital tools, platforms and approaches to achieve their potential, including concerns and risks associated with the use of ICT to improve social inclusion. Finally, we present two cases showing how organizations have addressed issues of social inclusion and bias in their ICT-supported evaluation efforts. Throughout the article, we offer insight from the discussion ‘Does ICT make us more inclusive or more extractive?’ held during October 2015 at the MERL Tech Conference in Washington, DC. 4
ICT has tremendous potential to complement and invigorate current approaches to program evaluation, but a number of issues have emerged. A focus on the ability of ICT-supported approaches to generate and analyze data at a fraction of the cost and time required by conventional data collection, and a fascination with new-found capacities to generate volumes of data that would previously have been impossible has meant that issues such as selection bias and data quality and relevance have frequently been overlooked. Additionally, evaluation processes risk becoming more extractive 5 due to increased pressure to automate processes from afar using ICT. This may result in less participation of local communities, lower awareness of local context, and biased interpretation of data. There are also ethical implications concerning the security of remotely collected data and the extraction of data without the consent or even knowledge of the subjects.
While ICT tools are widely used in program design and monitoring, they are less common in program evaluation. To date, evaluation approaches that harness ICT applications have received little attention from the main evaluation journals. Given the speed at which the field is advancing, the authors consider it important for evaluators to modernize their approaches.
2. How is ICT used to strengthen evaluation design and methodology?
Great expectations surround ICT and its ability to improve the performance of M&E systems. At the same time, there are key unresolved challenges. In this section, we share insights related to two main areas:
the use of ICT for data collection and analysis; and
how the incorporation of ICT affects evaluation design.
ICT for data collection and analysis
ICT is helping address problems related to data collection and analysis in several ways as noted below and explored in detail in Table 1.
Mobile phones and similar hand-held devices can
Smartphone apps make it easier to
Smartphones are being used to collect audio and video footage that
ICT (sometimes combined with big data) is being tested for
Finally, ICT are being used to
New challenges that ICT brings to data collection and analysis
Although ICT offers new ways for improving data collection and analysis, it also creates some new challenges:
Development and implementation of an entirely new digital data-collection tool or system often requires
Time and cost pressures, combined with a
While many contextual clues can theoretically be captured on smart phones through audio or visual images, it is difficult to explain digital data gathering to those who have not experienced ICT, raising doubts about truly
While many evaluators claim to use mixed methods, in most cases they do not use a
ICT in evaluation design: Addressing existing challenges and creating new ones
ICT is increasingly used to address existing evaluation challenges, yet its integration into evaluation design creates a whole new set of challenges that must be considered. Careful design and attention to these aspects are required in order to ensure methodological rigor, as noted below. In addition, many of these potentially promising ICT approaches require further testing and evaluation in the field.
Selection bias
When using mobile phones and other ICT to sample populations, questions arise concerning how representative are these respondents of all phone users and of the total target population. This inquiry includes questions such as: Who has access to a mobile phone? Who can afford to respond to a phone-based survey? Who chooses to respond, and which factors affect response rates?
Characteristics such as geography, income, language, literacy, sex, ethnicity, culture and age normally factor into both access and use of ICT. One well-documented example is the gap between men’s and women’s access to mobile phones in some countries (GSMA, 2015b). Mobile phone surveys are often skewed towards males who own data-enabled phones or who have greater access to the Internet. 6 Even when women and girls have access to a mobile phone, it is common for a husband or boyfriend to monitor usage (for example, reviewing who his wife, girlfriend or daughter has called) and to prevent private calls. This influences and biases responses. 7 Another commonly noted challenge is that certain household members do not speak or write in their country’s official language, and so they may not be able to reply to nation-wide mobile surveys.
People with access to mobile phones tend to be younger, better off, more urban and more male and it is difficult and expensive to find representative samples that can be used to adjust for this bias. However, even with sophisticated technology, phone response rates are often very low meaning that only people with certain characteristics or motivations are included (Leo and Morello, 2015).
Some of these challenges can be overcome. For example, phone companies keep detailed records of mobile phone ownership and use. Resources and time permitting, it may be possible to approximate a random sample using phone company records. Another way of dealing with this challenge is to set up automated interactive voice calls in local languages to bridge the language and literacy gap in response rates as shown in Case Study 1 (below) on VOTO Mobile. Generating an increase in response rates frequently involves incorporating a human component such as preparatory or validation visits to communities or households. While these methods usually increase response rates, they also increase costs and time, thus reducing one of the advantages of the technology.
The fallacy of large numbers
New data-collection techniques like crowdsourcing and social media offer quicker and cheaper ways to gather information from thousands of respondents. With data collected from such large numbers, it is easy to conclude that results are broadly representative of people’s thoughts and priorities. Yet because access to the Internet and mobile phones is uneven and usually skewed towards more elite or privileged members of a community or society, there is danger that the results will not be representative of the population of interest.
Some evaluators using ICT-based surveys have attempted to address these biases by combining digital data collection with human intervention to motivate participation and increase response rates (see Case Study 2 below on Dimagi). The organization BRAC addressed this imbalance in rural Bangladesh by bringing mobile phones to community assemblies so then even those without access to a mobile phone could participate in the UN’s ‘World We Want’ global campaign aimed at gathering input to determine priorities for the Sustainable Development Goals ((May, 2013); Development Goals (May, 2013; Demoombynes, 2011; Demoombynes et al., 2013)).
Lack of a program theory in evaluation design
A theoretical framework such as a theory of change (TOC) helps evaluators define the logical processes through which intended outcomes can be achieved, articulates core indicators to measure, and identifies key assumptions to test. ICT-based evaluations that rely on operationally generated monitoring data (for example, number of mothers who receive health messages or who take children for health check-ups) often have not articulated a TOC. Consequently, there is a risk the evaluation will not focus on important contextual factors or behavioral change processes, thus limiting the interpretative value of the findings.
Establishing a TOC can be costly and it is often difficult to bring together multiple stakeholders to ensure processes are participatory and involve a wide range of stakeholders. In addition, while time and resources may be invested in the initial development of the TOC, it is frequently never updated and becomes increasingly irrelevant over time. A final problem, discussed below, is when the TOC only focuses on how well the project achieves its intended outcomes and does not seek to identify unintended outcomes.
ICT are contributing through user-friendly online TOC development platforms such as (TOCO), 8 which makes it easier to involve a wide range of stakeholders and less expensive to update the TOC because it is online. Mobile phone apps enable generation of real-time updates on project implementation and behavioral change, allowing for testing and updating of key assumptions of the TOC. As discussed below, tools are also available to track unintended outcomes. However, while on-line software can facilitate the TOC preparation process, the problem remains of time pressure during the project launch. Since TOC is often not perceived as providing immediate benefits to program management, it is often not prioritized.
Failure to identify unintended outcomes
Due to a combination of political and methodological factors, many evaluation designs fail to capture unintended consequences, some of which can be serious (Bamberger et al., 2016a). A common example we cited earlier in this article is the failure to detect and mitigate negative consequences of women’s access to mobile phones (Wakunuma, 2013). Evaluation designs such as RCTs, which focus on assessing how well a program has achieved its intended objectives, tend to leave aside the identification of unintended consequences.
ICT makes it possible to better detect unintended consequences; for example: by soliciting direct and near real-time feedback from different segments of the target population; via collection and combination of different information sources; or (if GPS is used) through identifying the location and source of a particular problem. Tools such as audio and video recording can also contribute to observation of behavioral processes that might indicate unintended outcomes. Agencies using ICT, however, are subject to the same temptation as other programs to focus on the ‘Does it work?’ question and often end up ignoring the broader outcomes and unintended consequences. Also, many users of ICT only collect basic quantitative information and do not use available powerful behavioral analysis applications.
Difficulties in evaluating complex development interventions
As development programs grow in size and complexity, evaluators are becoming aware of the limitations of conventional experimental and quasi-experimental designs in capturing how multiple interventions, operating in complex contexts, with complex interactions among multiple stakeholders, contribute to multiple intended and unintended outcomes (Bamberger et al., 2016b). As yet, no widely accepted approaches to the evaluation of complex development interventions are available, although some agencies are experimenting with new approaches (USAID, 2013)
ICT makes real-time feedback from multiple sources possible for tracking complex, non-linear causal paths and implementation processes. Examples include: immediate input via mobile devices on how intended beneficiaries respond to program interventions; data from sensors attached to products (for example, heat sensors to determine whether an improved stove was used) tracking behavioral responses (discussed above), feedback from social media platforms on trends in target populations, and feedback from satellite images and remote sensors on contextual factors such as population movements, economic trends and access to water and other services (UN Global Pulse, 2014; Zazueta, 2015].
Table 1 provides examples of how ICTs can contribute to resolving common issues regarding data collection and analysis (Table 1a), and evaluation design (Table 1b). (Some of the new challenges that ICT brings to evaluation design are discussed in Section 5.) 9
Evaluation issues that ICT-based evaluations must address.
Data Collection and Analysis Issues.
Evaluation design issues.
3. The role of ICT in strengthening equity-, social justice-, and social inclusion-focused evaluation
An equity focus implies opening up space to give voice to excluded groups and avoiding the traditional top-down approach where planners and ‘experts’ prioritize and determine how problems should be addressed. Even when efforts are made to capture a range of voices during the initial diagnostic stage, they are often disregarded in subsequent program planning and design phases. The same is often true with respect to gender. Many evaluations now include gender analysis in project design, but attention to gender declines during project implementation and evaluation (Bamberger, 2013).
Huge increases in global access to mobile phones and growing levels of access to other ICT, including the Internet, have brought with them hope that ICT can support greater inclusion of program participants, especially those from marginalized and remote populations, in international development programming, including in monitoring and evaluation processes.
An equity focus requires: (1) identifying vulnerable groups, (2) opening up space for them to make their voices heard through channels that are culturally responsive, accessible and safe, and (3) ensuring their views are communicated to decision-makers. For large-scale programs it may also be necessary to apply social exclusion methodology (World Health Organization, n.d.) to assess how different groups are affected by the interactions among factors such as age, sex, race and ethnicity, economic status, geographical location and civil status.
There is an important distinction between reaching groups that are excluded due to logistical reasons and groups who are socially marginalized and not normally given voice. In the first case the challenges are logistical whereas in the second case powerful mechanisms of social control and exclusion must be addressed.
Issues requiring careful consideration in an equity-focused evaluation include:
Data. Existing data may not include certain vulnerable or excluded groups. Evaluators may overlook these groups when collecting new data.
Political resistance. Governments or other power holders may not want to recognize or enhance the rights of marginalized groups, making it difficult to collect data that could identify them and contribute to a comprehensive evaluation analysis.
Identifying and capturing dimensions and processes of exclusion. Many dimensions of inequity have difficult-to-measure qualitative elements, such as mechanisms of social control or psychological violence. Processes of exclusion, often subtle and occurring over long time periods, may be so hidden and internalized that people do not recognize them. Consequently, capturing exclusionary processes may require continual observation over a longer period of time than is normally available to evaluators.
These challenges can make it difficult to obtain a clear idea of the nature and extent of vulnerability and, in turn, to propose a strategy to open space for the voices of vulnerable groups or to highlight that a problem exists at all.
With respect to the logistical challenges of opening space for the voices of difficult-to-reach groups, ICT can capture information in new and more cost-effective ways, from more people in more remote places than ever before – making evaluation and data gathering more inclusive. It is common to see evaluators using smart phones and tablets for data collection (Bruce and Koler, 2016). Some organizations give basic phones and airtime to people in hard-to-reach areas so that evaluators can send them survey questions regularly (Demombynes et al., 2013). Others employ SMS to remind interviewees of face-to-face survey appointments, saving both time and money (Kuruvilla, 2013).
Mapping tools, GPS-enabled cellphones (Azzam and Robinson, 2013) and satellite technology now make it possible to locate and track populations migrating due to drought, natural disasters, and war. Satellite images and social media analysis can also provide information on economic, political, demographic, climate, and environmental factors affecting communities and which previously could not be easily documented or understood (Letouzé, 2013). 10
Smart phones, Twitter, and social media provide mechanisms for the general public to express their views and for overcoming mechanisms of social control. The UN, as mentioned above, ran a campaign to gather global input into the priorities for the Sustainable Development Goals (SDGs) using a combination of in-person consultations, text messages, and Internet-based surveys and discussions. This public feedback was presented to high-level decision makers in easy-to-digest graphic formats (May, 2013). 11
Audio and visual recording, with phones or other devices, provides new ways to document processes of social control. In the United States, for example, cell phone video recording has become a commonplace way for minorities to document incidents of police abuse. Text-based messages have also been used to monitor and report violence against children in Benin (Raftree, 2013). These types of citizen reporting can be combined with mapping tools to plot specific locations of abuse, enabling real-time feedback that also informs preventive actions. Similar tools are extensively used for citizen reporting on corruption and bribery, with some success (Eaves, 2013; Nesh-Nash, 2013). 12
Most of the challenges discussed in Section 2 apply to equity-focused evaluation, particularly issues concerning selection bias, unintended outcomes and ethics. Additionally, there is the concern that rather than contributing to greater participation and inclusion, ICT may lead to extractive data-collection processes. Among the remaining challenges, with or without the use of ICT, is the lack of concern in many societies for vulnerable groups such as ethnic minorities, refugees or the homeless. Unless used in a very creative way, and until there is interest in hearing and responding to these marginalized voices, ICT will not make much difference.
4. Cases illustrating increased attention to methodological rigor in the application of ICT in program evaluation
The following two cases illustrate inclusive approaches by experienced ICT practitioners to test and strengthen ICT-based methodologies for collecting survey data through mobile phones. 13 In both cases, the evaluators have made an effort to design processes that improve inclusion and reduce potential data bias.
The first case describes what is involved in developing a nationally representative population sample using data available from mobile network operators. It shows that while it is often possible to generate a statistically representative population sample using phone data, the process is time consuming, complex, and expensive. However, even when sufficient budget and time are available, the ability to generate a representative national sample also depends on the mobile phone penetration rate and on who adopts mobile technology in a given country.
The second case illustrates the careful planning and evaluation required when using mobile devices to improve the delivery of mobile health services. It also stresses the importance of combining ICT with in-person meetings to introduce respondents to the new ICT technology and to increase response rates.
Case 1. Testing the feasibility of building nationally representative samples using mobile phones
VOTO Mobile’s web-based platform and services help partners distribute and collect information by engaging difficult-to-reach populations through mobile phones. Interactive voice calls (including IVR) and SMS in local languages help reach across distance and literacy barriers. VOTO and partners have reached over 2,000,000 people across 37 countries.
In 2014 VOTO collaborated with the Center for Global Development (CGD) and the World Bank to analyze whether mobile phone-based surveys are feasible and cost-effective for gathering statistically significant information in four low-income countries (Afghanistan, Ethiopia, Mozambique, and Zimbabwe).
Results showed that it is possible to reduce the cost of a typical household survey by more than 90 per cent and, with care, mobile surveys are a feasible tool to replace more time-consuming and expensive approaches. About 2000 completed surveys were obtained per country (the system ran through an automatically generated phone list until the desired number of responses was reached). Table 2 provides additional detail:
Required Observations to create Effective Sample Size with Estimated Cost.
Lessons
Mobile penetration rates appear to have a substantial effect on the ability to produce nationally representative samples but can still be done. The Zimbabwe survey sample, where mobile penetration is around 79 per cent broadly reflects the national population (+/- 2.8%); whereas in Mozambique and Afghanistan where mobile penetration rates are around 40 per cent and 59 per cent respectively, the results are more mixed and nuanced (+/- 5–7%), and the Ethiopia results, with mobile penetration below 20 per cent, are likely not nationally representative. Note that average adult mobile penetration rates across Africa is 73 per cent (and growing), making this technique a real possibility for the continent.
While penetration rates are clearly important, who adopts mobile technology also matters. Afghanistan’s mobile penetration rate is 22 percentage points higher than Mozambique. However, cultural norms have slowed the adoption and usage of mobile technology by rural Afghan women. This factor inflates the samples required to reflect the broader population. Reaching rural women in most countries remains a challenge, and using voice-based mobile surveys that do not require literacy skills, is one step towards their inclusion.
Without special attention to details, mobile survey response rates can be lower than expected and exclude the voice of poor and vulnerable groups. In response, VOTO Mobile has conducted many experiments to develop a list of best practices to reach marginalized groups – these include: using voice messaging over SMS, offering a choice of interaction languages, creating a 9x retry pattern to ensure receipt, providing a call back channel to capture those calls you missed, using a longer introduction to surveys to provide more context, choosing a female voice, and paying close attention to the time of day of calls.
Case 2. Lessons from using an ICT platform for both service delivery and evaluation
Dimagi is a social enterprise that developed the CommCare platform, a tool that can be configured as a mobile phone-based job aid and data-collection tool for frontline workers like Community Health Workers (CHWs). While CommCare is most frequently used as a workflow and service delivery support tool, increasingly, organizations have also used it to carry out surveys and collect data for evaluation.
Below are key lessons learned from a program in Northern India that has used CommCare for several years. The program uses CommCare both as a job aid for CHWs to assist with delivery of antenatal and counseling services, and also to deploy a survey to enumerators as an evaluation of the program. The deployment of these tools provided illustrative insight into the design and use of ICT.
Listen to and work with users to develop and deploy appropriate and inclusive technology
Almost 1/3 of the CHWs participating in the program are functionally illiterate. After user testing and piloting, the application was redesigned to include extensive multimedia, which allowed users of any literacy level to navigate and use CommCare, guided by images and audio prompts.
A research study at the site provided performance feedback to CHWs using both visual and audio channels to make it accessible to all users. The study found that users had diverse preferences and abilities with respect to which medium of information they preferred, suggesting that it may sometimes be preferable to provide multiple modes of interaction.
Adapt implementation to limit exclusion of users, beneficiaries, or study participants
The training team, realizing that the less experienced or lower literate CHWs were having trouble using the system, altered the training protocol so that CHWs could bring other family members to support and learn along with them.
A supervision protocol was developed which facilitated extra support to CHWs who had trouble using the technology, or had poor coverage of their areas. This ensured that limited supervision was targeted where it was most needed, and that coverage of clients was as high as possible across all communities.
Do not let technology distract from the key role of people in designing and executing successful programs
The frequency and types of data collected both on the service delivery and evaluation sides would not have been possible to collect and manage with conventional tools; however the key to success is making these tools useful to people. Technology amplified the role of actors in the system, instead of just running parallel to them.
Adding questions to the evaluation surveys and making changes is much easier and faster using a digital survey, compared to printed survey tools. However, there is a risk that the surveys can become too long with questions that are not properly validated; good survey design and sampling methodology must not be unique to paper-based systems.
5. Challenges for the wider field of ICT in Evaluation
In addition to the logistical and methodological challenges explored in the above sections, some broad issues need to be considered regarding relationships between evaluators, development agencies, communities and the private sector. These include aspects of data ownership, public–private partnerships, individual data privacy rights, and potential unintended consequences of the presence of digital devices in the hands of an evaluator or enumerator.
Accessibility to data
Data such as phone records, social media, digital maps and satellite images are proprietary and access is limited. For example, phone records can only be accessed with the authorization of the phone company and sometimes government. So while access may be provided to certain UN agencies and perhaps some bi-lateral agencies, access is severely limited for civil society and other national agencies. In some cases data can be purchased but at high cost. While many data companies make data freely available during emergencies and are experimenting with ways to make it more widely available, this is still a challenge. This also relates to the concern about new information technology becoming extractive in that it is more easily available to international agencies than to local organizations, and data access for a growing set of interested organizations brings with it serious concerns about data privacy, as noted below.
Safety, security, and confidentiality
The use of electronic data brings serious new confidentiality issues. For example, if information is uploaded or downloaded in the local Internet café – which may be the only place with Internet access – the screen with confidential information might be visible to others. Organizations may use simple programs to collect data without ensuring that data security protocols are in place at the level of the device, the network, and the place of data storage. Devices might be stolen or confiscated, enabling militia, or governments to access sensitive information. Information collected via mobile phone surveys may remain on the phone and be viewed by others who share the phone. Additional issues arise when it comes to national laws that restrict or provide specific standards and protocols around data and data storage, for example, prohibitions on storing national health data on servers outside of the country.
It is also difficult to know whether the person answering a mobile phone survey is alone or with others. This may alter how a person responds (Leo and Morello, 2015). However, ICT can create greater intimacy and confidentiality (Raftree, 2014). For example, when mobile survey polls were used to research sexual violence against men, admissions of male rape increased as compared to paper and pencil surveys in communities. The assumption is that men are not willing to reveal sexual violence in a public setting, and it can be difficult to conduct surveys without community and family gathering around. Men revealed the violence over a mobile survey because of the increased sense of privacy. Some evaluators use tablets to conduct household level surveys, and when sensitive questions arise, hand the tablet to the respondent to answer privately so that the survey moves to the next screen without the enumerator seeing the response.
Integrating commercial and social development priorities
Both social-development practitioners and commercial data-generation and analytics companies are wrestling with questions regarding the extent to which commercial and development concerns can be reconciled with respect to the use of data generated through ICT. Some development practitioners are concerned that commercial business models will inevitably result in the exploitation of developing countries, particularly the poorest and most vulnerable groups. For example, information may be collected from local communities, often without their knowledge and then sold to companies seeking to expand market penetration. There are also security and privacy concerns (discussed above) in countries where companies are required to provide individuals’ information to government. At the same time, development agencies are very interested in accessing these new sources of data and analytics.
On the other hand, many companies are trying to develop guidelines on appropriate pricing policies to different kinds of development actors and when and how to provide information pro bono. One of the concerns of many companies is reputational risk when information provided pro bono might result in negative outcomes that reflect badly on a company’s reputation (for example, practical jokers using earthquake maps to provide false information to emergency workers (Meier, 2015), or social media information getting into the hands of police security agencies). While significant progress is being made, the mutual lack of understanding will continue to create challenges.
Unintended consequences for evaluators
We have mentioned potential negative outcomes when women use mobile phones in some contexts. Consequently, evaluators using digital devices in their evaluations should consider what the implications might be, given that negative consequences can sometimes be severe. For example, evaluators may be unaware of the risks that come with digitizing information, and of the need for different kinds of safety and security protocols when using mobile devices to gather data. On a different level, digital data-collection devices may lead to shifts in responses that a surveyor or evaluator is not able to predict, and holding a device rather than a clipboard could change the way people answer questions or relate to a surveyor. Because there is little research on these areas, evaluators may be learning as they are doing, with little guidance or support from past experiences.
6. Conclusions
ICT is being rapidly introduced into development evaluation. If evaluators are to keep up with their profession and help shape the future of development work, it will be important for them to keep abreast of these important developments. Additionally, though ICT is helping to address some traditional challenges with evaluation, ICT brings a series of new logistical and methodological issues that need further exploration. Many techniques described here have not yet been applied extensively and rigorously, and there is a need for wide-spread testing and improvement before they can be applied on a wide scale.
ICT are making important contributions to analyses of social exclusion and equity issues that limit the ability of development programs to reach all of their intended target populations. ICT offers real possibilities for enhancing inclusion and equity-focused evaluation, however this potential will not be realized if ICT are mainly seen as tools for quick, top-down data extraction. One key way that ICT in evaluation might help to identify and address these issues is by identifying and involving groups that tend to be excluded because they live in remote areas that are difficult and expensive to reach and reducing the cost and time of data collection, making it possible to include vulnerable and other difficult to reach groups in evaluation processes. ICT (especially mobile phones and social media platforms) may also provide political space for vulnerable groups who are frequently excluded due to social, cultural and political pressures to have their say. ICT can also help to measure processes that create and perpetuate exclusion and inequality, and enhance understanding of the dynamics of vulnerability.
The challenges with introducing new ICT tools and platforms and approaches are just starting to be recognized. Exploration of methodological issues such as integrating conventional and new kinds of information into an evaluation design; addressing and reconciling different kinds of data bias; developing mixed methods approaches that avoid over-reliance on quantitative data; and understanding how to work with data originally collected for a different purpose; is only beginning.
Additionally, insufficient emphasis is being placed on consent, safety, security and confidentiality. The nature of digital data, that it is created and stored on local devices, transferred to central storage and easily duplicated and shared exponentially, opens up new risks for vulnerable populations whose data is collected by evaluators. There is a need for better guidelines and codes of conduct and greater awareness of the technical challenges. This, along with other major areas that may be ripe for negative unintended consequences, requires further attention.
In conclusion, dramatic advances in ICT access at the global level, including by the most vulnerable and marginalized, open opportunities for more inclusive and equitable evaluation. In addition, data collection and analysis offer great potential for new approaches to development evaluation. These new technologies are already being incorporated into program monitoring, but less progress has been made with respect to evaluation. While there are great opportunities, methodological and ethical challenges need to be addressed.
Footnotes
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.